Compare commits

95 Commits
Author SHA1 Message Date
alpandmottopanikeiku ff01a6b14f [Bugfix][Frontend] Stop beam search on generation config EOS tokens (#62072)
pre-commit / pre-run-check (push) Canceled after 0s
pre-commit / pre-commit (push) Canceled after 0s
Signed-off-by: mottopanikeiku <176798723+mottopanikeiku@users.noreply.github.com>
Co-authored-by: mottopanikeiku <176798723+mottopanikeiku@users.noreply.github.com>
2026-10-18 16:49:42 +00:00
aoshen02andClaude Opus 5.5 fe8f2140ee [Doc] Document /inference/v1/generate request/response fields and DP rank header (#62305)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 15:26:36 +00:00
e80b6c642f [ROCm][DSv4] Overlap the unfused shared expert with AITER MegaMoE (#61819)
Signed-off-by: Taoyu Zhu <zhutaoyu97@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
2026-10-18 22:39:38 +08:00
aoshen02andClaude Opus 5.5 371745ebd6 [Doc][RL] Document routed experts return for MoE routing replay (R3) (#62302)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 21:11:45 +08:00
André Lustosamergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Shengqi Chen
072c0886d5 [Build] Keep plain CUDA arch when its suffixed variant is also requested (#56549)
Signed-off-by: Andre Lustosa <alustosa@redhat.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Shengqi Chen <harry-chen@outlook.com>
2026-10-18 20:13:17 +08:00
6b24947628 [CI/Build] Build DeepEP for SM103 (B300/GB300) in the Docker image (#61829)
Signed-off-by: Muhammad Ibrahim <muhammad.ibrahim@multiversecomputing.com>
Co-authored-by: Muhammad Ibrahim <muhammad.ibrahim@multiversecomputing.com>
Co-authored-by: Shengqi Chen <harry-chen@outlook.com>
2026-10-18 20:12:34 +08:00
Harry HuangandThien Tran 494663940b [Qwen4Exp] Clean up HC and PLE implementations for Qwen3.8-Flash-Next (#61454)
Signed-off-by: huanghaoyan.hhy <huanghaoyan.hhy@alibaba-inc.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
2026-10-18 19:38:21 +08:00
Rémi Delhaye 3c13653738 [Frontend] Score with the chat template declared by Sentence Transformers CrossEncoders (#60877)
Signed-off-by: rdlhio <contact@rdlh.io>
2026-10-18 11:30:12 +00:00
Kevin H. LuuandClaude Opus 5.5 953fa442fa [CI] Per-test timeout for Language Models (Extended Pooling) (#62279)
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 04:00:16 -07:00
Isotr0py 3f1b04e045 [Bugfix] Fix Opencv video backend frames count missing for WebM video inputs (#61759)
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
2026-10-18 10:37:39 +00:00
aoshen02andClaude Opus 5.5 2365177ddd [Doc][RL] Document the weight-transfer HTTP protocol for trainers without vLLM (#62267)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-18 09:26:56 +00:00
Louie Tsai 1b2a666391 [Doc][Recipe] Fix for redhat fp8 xeon support issue (#62245) 2026-10-18 16:46:56 +08:00
83c9a547bf [CI][Release] Fix release wheel builds: stop excluding tracked recipes/ from Docker context (#62221)
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
2026-10-18 00:48:00 -07:00
b510857d37 [Bugfix][Build] Build fused GDN MTP decode for SM80-SM90 with CUDA < 13 (#60896)
Signed-off-by: kimseunghyun <incongnito12@gmail.com>
Signed-off-by: kimseunghyun <hellonypcteam@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Shengqi Chen <harry-chen@outlook.com>
Co-authored-by: kimseunghyun <hellonypcteam@gmail.com>
2026-10-18 15:14:06 +08:00
7c9042c05a [Bugfix][MRV2] Fix speculative warmup deadlock with pipeline parallelism (#62129)
Signed-off-by: chenjr97 <chenjr97@mail2.sysu.edu.cn>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
2026-10-17 23:51:32 -07:00
da9134bc89 [ROCm][CI] Give the MoE parallel-launch watchdog room for AITER JIT builds (#62161)
Signed-off-by: Andreas Karatzas <AndreasKaratzas@users.noreply.github.com>
Co-authored-by: Andreas Karatzas <AndreasKaratzas@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-17 21:49:27 -05:00
b2456c622b [ROCm][CI] Make ROCm 10.0 the default image; keep ROCm 7.2 as -rocm72 (#58761)
Signed-off-by: Rohan Potdar <rohan.potdar@amd.com>
Signed-off-by: Rohan138 <rohanpotdar138@gmail.com>
Signed-off-by: Micah Williamson <micah.williamson@amd.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Micah Williamson <micah.williamson@amd.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-10-17 21:43:59 -05:00
03ee2e134d [CI][ROCm] Fix order-dependent test_rocm_aiter_sampler_defers_import (CPU tensors hit Triton) (#62213)
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-10-18 01:10:08 +00:00
djramicandAndreas Karatzas 9c778e9ff8 [Bugfix][EPLB] Wait for pending expert buffer copies before NIXL READs (#62081)
Signed-off-by: Djordje Ramic <djoramic@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-10-17 18:18:02 -05:00
7879f3687b [NIXL] Drop unreachable producer-side notif counting in push mode (#62210)
Signed-off-by: David Cheung <d7cheung@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
2026-10-17 21:24:35 +00:00
Kevin H. LuuandClaude Opus 5.5 255f14599e [Bugfix][Rust Frontend] Fix macOS build: read inherited listener family from its address (#62177)
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 11:56:28 -07:00
03e8300d2c [Bugfix] Enable inductor batch-invariant mode under VLLM_BATCH_INVARIANT (#61118)
Signed-off-by: Yifan Feng <evanfengff@gmail.com>
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-17 14:53:06 -04:00
Afonso RibeiroandFlora Feng 5c1c903501 [Perf][Parser] Stream Qwen3 tool argument values incrementally (12 KB call: 2.1 s to 11 ms) (#62027)
Signed-off-by: Afonso Ribeiro <afonso.diamantino2@gmail.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-10-17 18:49:06 +00:00
Tyrone bafb263664 [Bugfix][Benchmarks] Name the real cause when the /tokenize alignment probe fails (#59888) 2026-10-17 14:47:52 -04:00
Nick HillandClaude Opus 5.5 0c5dd18e7a [MRV2] Collect KV transfer results before the drafter runs (#62041)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-17 10:27:57 -07:00
Ting SUNmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Wentao Ye
763e3f7c91 [Bugfix][Spec Decode] Disable dynamic speculative decoding for fixed-K proposers instead of crashing (#46399)
Signed-off-by: Ting Sun <suntcrick@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
2026-10-17 12:41:04 -04:00
AlecandClaude Opus 5.5 e744eacf68 [Rust Frontend] Validate bad_words limits during request lowering (#61933)
Signed-off-by: Alec Flowers <aflowers@nvidia.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 09:10:40 -07:00
c0b6f74bee [Benchmark] Add per-case kernel profiles to vllm bench forward (#61692)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
2026-10-17 08:27:11 -07:00
siyuandXun Sun c61a1e73c6 [Feature] Skip MLA weight absorption when weights are pre-processed (#57320)
Co-authored-by: Xun Sun <UNIDY2002@outlook.com>
2026-10-17 23:10:04 +08:00
aoshen02andClaude Opus 5.5 237d99b4af Revert "[Docs] Add train-inference consistency agent skill" (#61551) (#62186)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 13:54:30 +00:00
aoshen02andClaude Opus 5.5 2d8efee9a7 [Docs] Add train-inference consistency agent skill (#61551)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 20:49:50 +08:00
Roy WangandClaude Opus 5.5 21e09ea959 [Doc] Sync DeepSeek-V4.1-Flash recipe with vllm-project/recipes#1136, #1137 (#62179)
Signed-off-by: esmeetu <jasonailu87@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 12:27:52 +00:00
837e87e1c5 [Feature][LoRA][MoE] Expert LoRA for FlashInfer TRTLLM MXFP4 experts (#61979)
Signed-off-by: aoshen524 <aoshen524@gmail.com>
Co-authored-by: aoshen524 <aoshen524@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 20:17:56 +08:00
liuzhenweiandKunshang Ji 4e964b2623 [XPU] Fix Event init failure w/ external (#61743)
Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
2026-10-17 19:10:39 +08:00
Vadim GimpelsonandClaude ec8b20fbf0 [Bugfix] Fix Gemma 4 encoder CUDA graph capture OOM at startup (#61365)
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-17 11:04:49 +00:00
2b85d9cb65 [Perf][Bugfix] Prefer MNNVL for single-node FlashInfer allreduce again (#61989)
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com>
2026-10-17 02:55:28 -07:00
026f6ab268 [Model] DeepSeek-V4/V4.1 VL: stream language-model weights instead of sorting the checkpoint (#58857)
Signed-off-by: git bisector <gitbisector@gmail.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
2026-10-17 09:37:56 +00:00
Roy WangandClaude Opus 5.5 b515648641 [Doc] Host model recipes in recipes/ (#61302)
Signed-off-by: esmeetu <jasonailu87@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 09:14:36 +00:00
f98ae6a9fd [Bugfix][Core] Keep resumable sessions that stop while preempted (#58778)
Signed-off-by: David Cheung <d7cheung@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Yifan Zong <yzong@redhat.com>
2026-10-17 09:08:14 +00:00
2067297b2f [ROCm][Perf] Use AITER tuned configs to select a4w4 gemm kernel instead of env var (#61278)
Signed-off-by: Andy Friedrich <afriedri@amd.com>
Signed-off-by: afriedri <afriedri@amd.com>
Co-authored-by: Simon Danielsson <70206058+simondanielsson@users.noreply.github.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-10-17 04:00:28 -05:00
Aarushi JainandAndreas Karatzas 3bf25c5097 [CI][Bugfix] Ignore AMD CI webhook no-op builds in /amd-ci run and retry (#62087)
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-10-17 03:54:31 -05:00
Jee Jee Li e56b1603af [Kernel]Add norm fused kernel for Step-5 Preview and Muse Spark Open (#62132)
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
2026-10-17 01:53:00 -07:00
Ting SUN eb8de8d1a1 [Bugfix][OpenAI] Respect allowed_token_ids in beam search (#47630)
Signed-off-by: Ting Sun <suntcrick@gmail.com>
2026-10-17 08:50:40 +00:00
dskwe b05ede5616 [Bugfix][Frontend] Preserve token IDs in completion derender responses (#62114)
Signed-off-by: Yuan Li <dskwelmcy@163.com>
2026-10-17 08:27:48 +00:00
61bb590877 [Bugfix][Spec Decode] Clear stale shared block-table rows (#61174)
Signed-off-by: Cheng Rui <286040359@qq.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Giancarlo Delfin <32987265+TheEpicDolphin@users.noreply.github.com>
2026-10-17 01:25:02 -07:00
c3de0b37e2 [Core][Spec Decode] Replace EAGLE group heuristics with explicit draft-layer tags (#61455)
Signed-off-by: Dakai An <dakaian108@gmail.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 08:22:33 +00:00
08a65e287e [Bugfix][Kernel] hadacore: don't load past the input on a partial last warp (#61944)
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
2026-10-17 00:52:17 -07:00
znzrybandWoosuk Kwon ad2688ab30 [Bugfix] Validate draft-token capacity in forward benchmark (#61809)
Signed-off-by: znzryb <znzryb34@gmail.com>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
2026-10-17 00:51:36 -07:00
JovanandOpenAI Codex 5be4266f5c [Bugfix][Frontend] Preserve Cohere and Anthropic streaming errors (#61947)
Signed-off-by: Kevin <jia__kuan@163.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-10-17 06:10:16 +00:00
Wei Zhao 90d6d871d6 [Attention] Add CuTe DSL backend selection for MLA context prefill (#62069)
Signed-off-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com>
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
2026-10-17 01:21:55 -04:00
Cy ChengandCy d74f4180cd [Bugfix][Frontend] Detokenize echoed prompts after truncation (#61876)
Signed-off-by: Cy <15118312+cycancode@users.noreply.github.com>
Co-authored-by: Cy <15118312+cycancode@users.noreply.github.com>
2026-10-17 05:20:38 +00:00
Jee Jee Li 39983a5e7a [Kernel] Add FP32 router GEMM support for Step-5 Preview and Muse Spark Open (#62126)
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
2026-10-16 22:17:55 -07:00
41b8c15809 [Feature][MoRIIO] Support hybrid WRITE transfers with DSpark (#60544)
Signed-off-by: Yichao Zhu <Yichao.Zhu@amd.com>
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: tjtanaa <tunjian.tan@embeddedllm.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-17 04:57:58 +00:00
Micah WilliamsonandCursor a0e7137c4b [CI][ROCm][Bugfix] DeepSeek V4.1: don't pass expert intermediate padding to AITER (#61960)
Signed-off-by: Micah Williamson <micah.williamson@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
2026-10-17 04:25:39 +00:00
7b6310c109 [Perf][Sampler] Speed up the small-batch split top-p pipeline (#61996)
Signed-off-by: Bowen Wang <bowen@inferact.ai>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
2026-10-16 21:22:11 -07:00
Fadi Arafehandvllm-agent 556ac15eb9 [Bugfix] Remove extra args from CPU's compute slot mapping kernel (#61992)
Signed-off-by: Fadi Arafeh <fadi.arafeh@arm.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
2026-10-17 12:11:33 +08:00
Gabriel WuandClaude Opus 5.5 dcd07967ca [CPU] Keep the hybrid block size across the engine core's config rerun (#61805)
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 12:06:03 +08:00
ce49c47cbb [Bugfix][Sampler] Carry the boundary tie count across blocks in the split top-p mask kernel (#62046)
Signed-off-by: cipheraxat <cipheraxat@users.noreply.github.com>
Co-authored-by: cipheraxat <cipheraxat@users.noreply.github.com>
Co-authored-by: Bowen Wang <bowen@inferact.ai>
2026-10-17 02:26:57 +00:00
Sherif Walyandvllm-agent ce8853d27a [Bugfix][KV Offload] Allow explicit SimpleCPUOffload block counts (#61674)
Signed-off-by: Sherif Waly <sherif.waly@mistral.ai>
Co-authored-by: vllm-agent <claw@inferact.ai>
2026-10-17 02:11:12 +00:00
71c20f73f2 [FlashInfer] Migrate autotune warmup to flashinfer.autotune_v2 managed cache (#55671)
Signed-off-by: Yang Xu <yanxu@nvidia.com>
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
Signed-off-by: YangXu <yanxu@nvidia.com>
Co-authored-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
Co-authored-by: Kimi Code <noreply@moonshot.cn>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
2026-10-17 02:08:28 +00:00
Shuolei Wangandaoshen02 b3a7e2ff6e [RL][Frontend] Return weight-version spans on OpenAI completions and chat (#61347)
Signed-off-by: Shuolei Wang <shuoleiwang123@gmail.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
2026-10-17 02:08:07 +00:00
6b6a6c5c98 [RL] Support per-parameter source layouts for NCCL M2N (#60934)
Signed-off-by: pkousha <pkousha@nvidia.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
2026-10-16 21:41:17 -04:00
Harry MellorandClaude Opus 5.5 f8112704d5 [Chore] Use upstream Transformers InklingProcessor (#61883)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-17 01:04:54 +00:00
Omar MuhammadNicholas SusantoDouglas Lehrmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
f019e3e105 [ROCm] attn res + sigmoid_mul + conv fusions (#52968)
Signed-off-by: Omar Muhammad <omar.muhammad@amd.com>
Signed-off-by: Nicholas Susanto <nicholas.susanto@amd.com>
Co-authored-by: Nicholas Susanto <nicholas.susanto@amd.com>
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-17 00:54:54 +00:00
Vadim GimpelsonandClaude 783380af39 [Bugfix] Fix NaN output of the Mamba2 chunk scan kernel on sm90 (#61718)
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-16 17:24:51 -07:00
Wei ZhaoandCodex be25977532 [CI][Tests] Account for FP8 KV-cache rounding in MLA reference (#62067)
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-16 17:10:08 -07:00
523d74e2ac [Bugfix][LoRA] Fix garbage/NaN lora_expand output on SM90 with fp32 i… (#61906)
Signed-off-by: BeichenHuang <polarishuang0.0@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
2026-10-16 19:42:30 -04:00
a94c315711 [DP] Engines own wave wake-ups; drop the front-end FIRST_REQ (#59559)
Signed-off-by: fh3 <a@b>
Co-authored-by: fh3 <a@b>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 19:41:54 -04:00
b544587408 [Core][Feat] Pluggable KVCacheConfigBuilder for platform/model-specific KV cache planning (#53558)
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
2026-10-16 23:16:47 +00:00
alpandmottopanikeiku 2268d83bd9 [Bugfix][Multimodal] Honor RGB PNG transparency when loading images (#61986)
Signed-off-by: mottopanikeiku <176798723+mottopanikeiku@users.noreply.github.com>
Co-authored-by: mottopanikeiku <176798723+mottopanikeiku@users.noreply.github.com>
2026-10-16 23:09:12 +00:00
28e92eeff2 [ROCm][Perf][Kimi-K3] Use AITER Gluon chunk_kda for KDA prefill (#59552)
Signed-off-by: Omar Muhammad <omar.muhammad@amd.com>
Signed-off-by: Nicholas Susanto <nicholas.susanto@amd.com>
Co-authored-by: Nicholas Susanto <nicholas.susanto@amd.com>
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-16 22:58:56 +00:00
Afonso RibeiroandFlora Feng b088eba944 [Perf][Parser] Stream MiniMax M2 tool argument values incrementally (16 KB call: 5.9 s to 26 ms) (#62035)
Signed-off-by: Afonso Ribeiro <afonso.diamantino2@gmail.com>
Co-authored-by: Flora Feng <4florafeng@gmail.com>
2026-10-16 22:46:02 +00:00
596e530675 [Kernel] Add router GEMM kernels for (5120, 256) and BF16 output to ll_bf16_gemm (#62042)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Lain <fusiyuan2000@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
2026-10-16 15:39:46 -07:00
153a17166e [Frontend] Add the Beam reasoning and tool parsers (#62052)
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Yoni Gozlan <74535834+yonigozlan@users.noreply.github.com>
Co-authored-by: neilkg <33635204+neilkg@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 14:08:29 -07:00
Dan BlanaruandClaude Opus 5.5 9731f30a03 [Bugfix] Keep VLLM_RAY_BUNDLE_INDICES out of the compile cache key (#61211)
Signed-off-by: DanBlanaru <48605845+DanBlanaru@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 20:59:48 +00:00
68ebea789a [Perf][MoE] Skip padded work in SM100 FP4 DeepGEMM experts (#59044)
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
2026-10-16 16:20:52 -04:00
afriedrimergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Claude Opus 5.5
cc2e4fecf2 [ROCm] [fusion] support ROCm aiter fusion of per tensor quant + rmsnorm (#52831)
Signed-off-by: Andy Friedrich <afriedri@amd.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-16 15:19:07 -05:00
9eb7fb9157 [Build] Remove unused tools/install_deepgemm.sh (#61265)
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-16 15:57:49 -04:00
cjackal 663e535640 [Perf][Spec Decode] Skip bonus-logit reduction for short drafts in rejection sampling (#61451)
Signed-off-by: cjackal <44624812+cjackal@users.noreply.github.com>
2026-10-16 12:42:48 -07:00
Oxana KorzhandClaude Opus 5.5 5bf3d9f665 [ROCm][Kimi-K3] Support DSpark adaptive verification on MI355X (#59824)
Signed-off-by: Oxana Korzh <okorzh@amd.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
2026-10-16 13:53:49 -05:00
djramic 307c5844ad [ROCm][CI] Enable batch invariance for the ngram scheduling accuracy test (#62048)
Signed-off-by: Djordje Ramic <djoramic@amd.com>
2026-10-16 18:36:14 +00:00
Guan-Ming Chiuandvllm-agent 3dc6def360 [Attention][Spec Decode] Enable varlen Mamba2 decode for adaptive verification (#53036)
Signed-off-by: Guan-Ming (Wesley) Chiu <105915352+guan404ming@users.noreply.github.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
2026-10-16 11:23:59 -07:00
djramic e7a5bdb454 [ROCm][CI] Move Spec Decode Speculators + MTP to MI355 (#62053)
Signed-off-by: Djordje Ramic <djoramic@amd.com>
2026-10-16 18:22:38 +00:00
alberto ae8f41238c [Bugfix][Parser] Honor a prompt-closed reasoning block in non-streaming parse and reasoning token counts (#61857)
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
2026-10-16 12:54:42 -04:00
cffc0cfa31 [CI] Fix B200 FusedMoE Layer Kernels hang on NVSHMEM IB setup (#61695)
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-16 12:00:12 -04:00
Andreas KaratzasandClaude 706791d516 [CI][AMD] Move 17 test groups from MI300 to MI355 and test large DeepSeek/Nemotron evals on AITER (#61955)
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: Claude <noreply@anthropic.com>
2026-10-16 10:58:59 -05:00
Sawyer BowermanandCodex 76081b9ee4 [Bugfix][Model] Honor quantization config in Inkling MTP (#61921)
Signed-off-by: Sawyer Bowerman <sbowerma@redhat.com>
Co-authored-by: Codex <noreply@openai.com>
2026-10-16 15:49:26 +00:00
bnellnm 1a5b187066 Add @bnellnm to CODEOWNERS (#62025)
Signed-off-by: Bill Nell <bnell@redhat.com>
2026-10-16 11:16:03 -04:00
Wentao Ye 8e25bbca19 [Refactor] Remove dead code multiple places (#61843)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
2026-10-16 15:01:07 +00:00
8a6c0449b3 [Bugfix][ROCm][GLM-5.2] Stop forcing skip_final_all_reduce on AMD DSA model path (#58589)
Signed-off-by: avininjamay8 <Avinash.Paul@amd.com>
Signed-off-by: avininjamay8 <avpaul@amd.com>
Co-authored-by: avininjamay8 <avpaul@amd.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
Co-authored-by: vllm-agent <claw@inferact.ai>
2026-10-16 21:38:55 +08:00
Ayushman Singh de7eeaa87b [Perf] Share Triton JIT warmup compilation pool across registered kernels (#61422) 2026-10-16 08:59:42 -04:00
Martin Hickey 64175d3827 [Bugfix][Frontend] Keep the parser's tool call IDs in batch derender (#61852)
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
2026-10-16 10:27:17 +00:00
alpandmottopanikeiku 4ab99ff136 [Bugfix][Frontend] Honor the constructor chat template in offline scoring (#61990)
Signed-off-by: mottopanikeiku <176798723+mottopanikeiku@users.noreply.github.com>
Co-authored-by: mottopanikeiku <176798723+mottopanikeiku@users.noreply.github.com>
2026-10-16 10:20:46 +00:00
024c5188bf [Frontend] Resolve logprobs on the parsed streaming derender path (#60437)
Signed-off-by: Shimi Bandiel <shimib@google.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: wang.yuqi <yuqi.wang@daocloud.io>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
2026-10-16 18:04:15 +08:00
Hexiang Wang 9a16d43ef0 [Bugfix][Mooncake] Fix MultiConnector stats and proxy SSE responses (#61218)
Signed-off-by: whx-sjtu <xiaowang990929@gmail.com>
Signed-off-by: Nicolò Lucchesi <nicolo.lucchesi@gmail.com>
2026-10-16 17:54:48 +08:00
539 changed files with 67595 additions and 6180 deletions
@@ -31,6 +31,9 @@ Requests: `[N*]Q[@K][c][:D]`, comma-separated, per DP rank.
| `2048@10240` | prefill after an 8192-token prefix-cache hit | | `2048@10240` | prefill after an 8192-token prefix-cache hit |
| `2048@10240c` | chunked prefill: the prefix came from earlier chunks | | `2048@10240c` | chunked prefill: the prefix came from earlier chunks |
Verify cases require `--speculative-config` with `num_speculative_tokens`
at least as large as the case's draft count; insufficient capacity is rejected.
Hit vs chunk only differs on models with SWA bounded replay (DSV4.1): a hit Hit vs chunk only differs on models with SWA bounded replay (DSV4.1): a hit
also recomputes the last window (128 tokens) of the prefix. also recomputes the last window (128 tokens) of the prefix.
@@ -48,6 +51,46 @@ Key flags (plus all `EngineArgs`):
Read `median_us` (each replay is already the max over ranks). Read `median_us` (each replay is already the max over ranks).
## Kernel breakdown
Add `--profile-dir DIR` to see where the time goes. After timing, each case is
replayed 3 more times under `torch.profiler`; every rank writes
`DIR/<case>.rank<r>.kernels.json` (GPU time per kernel per forward, sorted)
and `DIR/<case>.rank<r>.trace.json.gz` (open in Perfetto). Rank 0's top
kernels are printed and included in `--output-json`.
```bash
python -m vllm.benchmarks.forward.profile DIR/decode.rank0.kernels.json
# Before/after a change, same case: per-kernel deltas, largest first.
python -m vllm.benchmarks.forward.profile base/decode.rank0.kernels.json new/decode.rank0.kernels.json
```
- Kernels are grouped by name without template or function arguments; the
JSON keeps one full name per group in `example`.
- Only the timed forward is included: the in-graph KV restore and rank
rendezvous are excluded (`pre_forward_us`). `span_us` (first to last
forward kernel) should be close to `median_us`, a bit higher from profiler
overhead; `busy_us` can exceed it when streams overlap.
- Compare the same rank across runs: under EP, ranks differ (e.g. collective
wait time shows up inside communication kernels).
For Nsight Systems (NVTX, NCCL, CPU side, GPU metrics), skip `--profile-dir`:
the two can't share CUPTI. Each case's 3 profiled replays run between
`cudaProfilerStart`/`Stop` in an NVTX range `forward/<case>`, so a capture range
records only them, one report per case (and rank):
```bash
nsys profile --capture-range=cudaProfilerApi --capture-range-end=repeat \
--cuda-graph-trace=node -o prof vllm bench forward ... # prof.1, prof.2, ...
torchrun --nproc-per-node=4 --no-python nsys profile ... -o prof_rank%q{RANK} \
python -m vllm.entrypoints.cli.main bench forward ...
nsys stats --report cuda_gpu_kern_sum --format csv -o kern prof.1.nsys-rep
```
Keep `--capture-range-end=repeat`: nsys's default stops after the first case
and leaves the bench hanging. `nsys stats` also counts the KV restore copies
that the JSON summary excludes.
## Caveats ## Caveats
- **Dummy weights lie for data-dependent kernels.** Dummy gates route every - **Dummy weights lie for data-dependent kernels.** Dummy gates route every
+11 -5
View File
@@ -5,9 +5,12 @@ description: Find vLLM deployment recipes and configs for LLMs and Hugging Face
# Model Recipes # Model Recipes
Model serving recipes are maintained in the Model serving recipes are maintained in this repository's
[models/ directory of vllm-project/recipes](https://github.com/vllm-project/recipes/tree/main/models). [recipes/models/ directory](../../../recipes/models). Omni (image, video,
The published catalog is at audio) recipes stay in vllm-project/recipes for now (moving to vllm-omni),
and Ascend NPU content is inline in these recipes (moving to vllm-ascend
overlays). The published catalog, rendered by
[vllm-project/recipes](https://github.com/vllm-project/recipes), is at
[recipes.vllm.ai](https://recipes.vllm.ai). [recipes.vllm.ai](https://recipes.vllm.ai).
## Usage notes ## Usage notes
@@ -146,6 +149,9 @@ uploads, untested hardware claims, or non-standard flags that only work for one
setup. Before opening a pull request, confirm that the user ran the setup. Before opening a pull request, confirm that the user ran the
configuration and can share the results. configuration and can share the results.
Follow the recipes repository's Edit the YAML under [recipes/models/](../../../recipes/models) in this
repository, following the recipes repository's
[add-recipe skill](https://github.com/vllm-project/recipes/blob/main/.agents/skills/add-recipe/SKILL.md) [add-recipe skill](https://github.com/vllm-project/recipes/blob/main/.agents/skills/add-recipe/SKILL.md)
in a checkout of `vllm-project/recipes` or the user's intended fork. for the schema. Validate and preview it from a checkout of
`vllm-project/recipes` pointed at this directory, as
[recipes/README.md](../../../recipes/README.md) describes.
+208 -473
View File
@@ -703,385 +703,6 @@ steps:
DOCKER_BUILDKIT: "1" DOCKER_BUILDKIT: "1"
DOCKERHUB_USERNAME: "vllmbot" DOCKERHUB_USERNAME: "vllmbot"
# =============================================================================
# ROCm Release Pipeline (x86_64 only)
# =============================================================================
#
# vLLM version is determined by the Buildkite checkout (like CUDA pipeline).
# To build a specific version, trigger the build from that branch/tag.
#
# Environment variables for ROCm builds (set via Buildkite UI or schedule):
#
# Note: ROCm version is determined by BASE_IMAGE in docker/Dockerfile.rocm_base
#
# =============================================================================
- group: "Build ROCm Wheel / Image "
key: "build-rocm-wheel-image"
depends_on: ~
steps:
# ROCm Job 1: Build ROCm Base Wheels (with S3 caching)
- label: ":rocm: Build ROCm Base Image & Wheels"
id: build-rocm-base-wheels
depends_on: ~
agents:
queue: cpu_queue_release
commands:
- |
set -euo pipefail
# Generate cache key
CACHE_KEY=$$(.buildkite/scripts/cache-rocm-base-wheels.sh key)
ECR_CACHE_TAG="public.ecr.aws/q9t5s3a7/vllm-release-repo:$${CACHE_KEY}-rocm-base"
echo "========================================"
echo "ROCm Base Build Configuration"
echo "========================================"
echo " CACHE_KEY: $${CACHE_KEY}"
echo " ECR_CACHE_TAG: $${ECR_CACHE_TAG}"
echo "========================================"
# Login to ECR
aws ecr-public get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
IMAGE_EXISTS=false
WHEELS_EXIST=false
# Check ECR for Docker image
if docker manifest inspect "$${ECR_CACHE_TAG}" > /dev/null 2>&1; then
IMAGE_EXISTS=true
echo "ECR image cache HIT"
fi
# Check S3 for wheels
WHEEL_CACHE_STATUS=$(.buildkite/scripts/cache-rocm-base-wheels.sh check)
if [ "$${WHEEL_CACHE_STATUS}" = "hit" ]; then
WHEELS_EXIST=true
echo "S3 wheels cache HIT"
fi
# Scenario 1: Both cached (best case)
if [ "$${IMAGE_EXISTS}" = "true" ] && [ "$${WHEELS_EXIST}" = "true" ]; then
echo ""
echo "FULL CACHE HIT - Reusing both image and wheels"
echo ""
# Download wheels
.buildkite/scripts/cache-rocm-base-wheels.sh download
# Save ECR tag for downstream jobs
buildkite-agent meta-data set "rocm-base-image-tag" "$${ECR_CACHE_TAG}"
# Scenario 2: Full rebuild needed
else
echo ""
echo " CACHE MISS - Building from scratch..."
echo ""
# Build full base image and push to ECR
DOCKER_BUILDKIT=1 docker buildx build \
--file docker/Dockerfile.rocm_base \
--tag "$${ECR_CACHE_TAG}" \
--build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 \
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
--push \
.
# Build wheel extraction stage
DOCKER_BUILDKIT=1 docker buildx build \
--file docker/Dockerfile.rocm_base \
--tag rocm-base-debs:$${BUILDKITE_BUILD_NUMBER} \
--target debs_wheel_release \
--build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 \
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
--load \
.
# Extract and upload wheels
mkdir -p artifacts/rocm-base-wheels
cid=$(docker create rocm-base-debs:$${BUILDKITE_BUILD_NUMBER})
docker cp $${cid}:/app/debs/. artifacts/rocm-base-wheels/
docker rm $${cid}
.buildkite/scripts/cache-rocm-base-wheels.sh upload
# Cache base docker image to ECR
docker push "$${ECR_CACHE_TAG}"
buildkite-agent meta-data set "rocm-base-image-tag" "$${ECR_CACHE_TAG}"
echo ""
echo " Build complete - Image and wheels cached"
fi
artifact_paths:
- "artifacts/rocm-base-wheels/*.whl"
env:
DOCKER_BUILDKIT: "1"
S3_BUCKET: "vllm-wheels"
# ROCm Job 2: Build vLLM ROCm Wheel
- label: ":python: Build vLLM ROCm Wheel - x86_64"
id: build-rocm-vllm-wheel
depends_on:
- step: build-rocm-base-wheels
allow_failure: false
agents:
queue: cpu_queue_release
timeout_in_minutes: 180
commands:
# Download artifacts and prepare Docker image
- |
set -euo pipefail
# Ensure git tags are up-to-date (Buildkite's default fetch doesn't update tags)
# This fixes version detection when tags are moved/force-pushed
echo "Fetching latest tags from origin..."
git fetch --tags --force origin
# Log tag information for debugging version detection
echo "========================================"
echo "Git Tag Verification"
echo "========================================"
echo "Current HEAD: $(git rev-parse HEAD)"
echo "git describe --tags: $(git describe --tags 2>/dev/null || echo 'No tags found')"
echo ""
echo "Recent tags (pointing to commits near HEAD):"
git tag -l --sort=-creatordate | head -5
echo "setuptools_scm version detection:"
pip install -q setuptools_scm 2>/dev/null || true
python3 -c "import setuptools_scm; print(' Detected version:', setuptools_scm.get_version())" 2>/dev/null || echo " (setuptools_scm not available in this environment)"
echo "========================================"
# Download wheel artifacts from current build
echo "Downloading wheel artifacts from current build"
buildkite-agent artifact download "artifacts/rocm-base-wheels/*.whl" .
# Get ECR image tag from metadata (set by build-rocm-base-wheels)
ECR_IMAGE_TAG="$$(buildkite-agent meta-data get rocm-base-image-tag 2>/dev/null || echo '')"
if [ -z "$${ECR_IMAGE_TAG}" ]; then
echo "ERROR: rocm-base-image-tag metadata not found"
echo "This should have been set by the build-rocm-base-wheels job"
exit 1
fi
echo "Pulling base Docker image from ECR: $${ECR_IMAGE_TAG}"
# Login to ECR
aws ecr-public get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
# Pull base Docker image from ECR
docker pull "$${ECR_IMAGE_TAG}"
echo "Loaded base image: $${ECR_IMAGE_TAG}"
# Prepare base wheels for Docker build context
mkdir -p docker/context/base-wheels
touch docker/context/base-wheels/.keep
cp artifacts/rocm-base-wheels/*.whl docker/context/base-wheels/
echo "Base wheels for vLLM build:"
ls -lh docker/context/base-wheels/
echo "========================================"
echo "Building vLLM wheel with:"
echo " BUILDKITE_COMMIT: $${BUILDKITE_COMMIT}"
echo " BUILDKITE_BRANCH: $${BUILDKITE_BRANCH}"
echo " BASE_IMAGE: $${ECR_IMAGE_TAG}"
echo "========================================"
# Build vLLM wheel using local checkout (REMOTE_VLLM=0)
DOCKER_BUILDKIT=1 docker build \
--file docker/Dockerfile.rocm \
--target export_vllm_wheel_release \
--output type=local,dest=rocm-dist \
--build-arg BASE_IMAGE="$${ECR_IMAGE_TAG}" \
--build-arg REMOTE_VLLM=0 \
--build-arg GIT_REPO_CHECK=1 \
--build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 \
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
.
echo "Built vLLM wheel:"
ls -lh rocm-dist/*.whl
# Copy wheel to artifacts directory
mkdir -p artifacts/rocm-vllm-wheel
cp rocm-dist/*.whl artifacts/rocm-vllm-wheel/
echo "Final vLLM wheel:"
ls -lh artifacts/rocm-vllm-wheel/
artifact_paths:
- "artifacts/rocm-vllm-wheel/*.whl"
env:
DOCKER_BUILDKIT: "1"
S3_BUCKET: "vllm-wheels"
# ROCm Job 3: Upload Wheels to S3
- label: ":s3: Upload ROCm Wheels to S3"
id: upload-rocm-wheels
depends_on:
- step: build-rocm-vllm-wheel
allow_failure: false
agents:
queue: cpu_queue_release
timeout_in_minutes: 60
commands:
# Download all wheel artifacts and run upload
- |
set -euo pipefail
# Download artifacts from current build
echo "Downloading artifacts from current build"
buildkite-agent artifact download "artifacts/rocm-base-wheels/*.whl" .
buildkite-agent artifact download "artifacts/rocm-vllm-wheel/*.whl" .
# # Run upload script
bash .buildkite/scripts/upload-rocm-wheels.sh
env:
DOCKER_BUILDKIT: "1"
S3_BUCKET: "vllm-wheels"
# ROCm Job 4: Annotate ROCm Wheel Release
- label: ":memo: Annotate ROCm wheel release"
id: annotate-rocm-release
depends_on:
- upload-rocm-wheels
agents:
queue: cpu_queue_release
commands:
- "bash .buildkite/scripts/annotate-rocm-release.sh"
env:
S3_BUCKET: "vllm-wheels"
# ROCm Job 5: Generate Root Index for ROCm Wheels (for release only)
# This is the job to create https://wheels.vllm.ai/rocm/ index allowing
# users to install with `uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/`
- block: "Generate Root Index for ROCm Wheels for Release"
key: block-generate-root-index-rocm-wheels
depends_on: upload-rocm-wheels
- label: ":package: Generate Root Index for ROCm Wheels for Release"
depends_on: block-generate-root-index-rocm-wheels
id: generate-root-index-rocm-wheels
agents:
queue: cpu_queue_release
commands:
- "bash tools/vllm-rocm/generate-rocm-wheels-root-index.sh"
env:
S3_BUCKET: "vllm-wheels"
VARIANT: "rocm723"
# ROCm Job 6: Build ROCm Release Docker Image
- label: ":docker: Build release image - x86_64 - ROCm"
id: build-rocm-release-image
depends_on:
- step: block-build-release-images
allow_failure: true
- step: build-rocm-base-wheels
allow_failure: false
agents:
queue: cpu_queue_release
timeout_in_minutes: 60
commands:
- |
set -euo pipefail
# Login to ECR
aws ecr-public get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
# Get ECR image tag from metadata (set by build-rocm-base-wheels)
ECR_IMAGE_TAG="$$(buildkite-agent meta-data get rocm-base-image-tag 2>/dev/null || echo '')"
if [ -z "$${ECR_IMAGE_TAG}" ]; then
echo "ERROR: rocm-base-image-tag metadata not found"
echo "This should have been set by the build-rocm-base-wheels job"
exit 1
fi
echo "Pulling base Docker image from ECR: $${ECR_IMAGE_TAG}"
# Pull base Docker image from ECR
docker pull "$${ECR_IMAGE_TAG}"
echo "Loaded base image: $${ECR_IMAGE_TAG}"
# Pass the base image ECR tag to downstream steps (nightly publish)
buildkite-agent meta-data set "rocm-base-ecr-tag" "$${ECR_IMAGE_TAG}"
echo "========================================"
echo "Building vLLM ROCm release image with:"
echo " BASE_IMAGE: $${ECR_IMAGE_TAG}"
echo " BUILDKITE_COMMIT: $${BUILDKITE_COMMIT}"
echo "========================================"
# Build vLLM ROCm release image using cached base
DOCKER_BUILDKIT=1 docker build \
--build-arg max_jobs=16 \
--build-arg BASE_IMAGE="$${ECR_IMAGE_TAG}" \
--build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 \
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
--build-arg INSTALL_LMCACHE=true \
--tag public.ecr.aws/q9t5s3a7/vllm-release-repo:$${BUILDKITE_COMMIT}-rocm \
--target vllm-openai \
--progress plain \
-f docker/Dockerfile.rocm .
# Push to ECR
docker push public.ecr.aws/q9t5s3a7/vllm-release-repo:$${BUILDKITE_COMMIT}-rocm
echo ""
echo " Successfully built and pushed ROCm release image"
echo " Image: public.ecr.aws/q9t5s3a7/vllm-release-repo:$${BUILDKITE_COMMIT}-rocm"
echo ""
env:
DOCKER_BUILDKIT: "1"
S3_BUCKET: "vllm-wheels"
- label: "Publish nightly XPU image to DockerHub"
depends_on:
- create-manifest-xpu
if: build.env("NIGHTLY") == "1"
agents:
queue: small_cpu_queue_release
commands:
- "bash .buildkite/scripts/xpu/push-nightly-builds-xpu.sh"
- "bash .buildkite/scripts/cleanup-nightly-builds.sh nightly- vllm/vllm-openai-xpu"
plugins:
- docker-login#v3.0.0:
username: vllmbot
password-env: DOCKERHUB_TOKEN
env:
DOCKER_BUILDKIT: "1"
DOCKERHUB_USERNAME: "vllmbot"
- label: "Publish nightly ROCm image to DockerHub"
depends_on:
- build-rocm-release-image
if: build.env("NIGHTLY") == "1"
agents:
queue: small_cpu_queue_release
commands:
- "bash .buildkite/scripts/push-nightly-builds-rocm.sh"
# Clean up old nightly builds (keep only last 14)
- "bash .buildkite/scripts/cleanup-nightly-builds.sh nightly- vllm/vllm-openai-rocm"
- "bash .buildkite/scripts/cleanup-nightly-builds.sh base-nightly- vllm/vllm-openai-rocm"
plugins:
- docker-login#v3.0.0:
username: vllmbot
password-env: DOCKERHUB_TOKEN
env:
DOCKER_BUILDKIT: "1"
DOCKERHUB_USERNAME: "vllmbot"
- trigger: "perf-eval" - trigger: "perf-eval"
label: "Trigger perf-eval (CUDA)" label: "Trigger perf-eval (CUDA)"
key: trigger-perf-eval-cuda key: trigger-perf-eval-cuda
@@ -1103,7 +724,7 @@ steps:
key: trigger-perf-eval-rocm key: trigger-perf-eval-rocm
if: build.env("NIGHTLY") == "1" || build.env("RUN_PERF_EVAL") == "1" if: build.env("NIGHTLY") == "1" || build.env("RUN_PERF_EVAL") == "1"
depends_on: depends_on:
- build-rocm-release-image - build-rocm100-release-image
async: true async: true
soft_fail: true soft_fail: true
build: build:
@@ -1112,138 +733,250 @@ steps:
env: env:
NIGHTLY: 1 # run all perf-eval workloads NIGHTLY: 1 # run all perf-eval workloads
VLLM_COMMIT: "$BUILDKITE_COMMIT" VLLM_COMMIT: "$BUILDKITE_COMMIT"
VLLM_IMAGE_ROCM: "public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-rocm" VLLM_IMAGE_ROCM: "public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-rocm100"
# ============================================================================= # =============================================================================
# ROCk Release Pipeline (x86_64 only) # ROCm Release Pipelines (x86_64 only)
# ============================================================================= # =============================================================================
# #
# Same shape as the ROCm pipeline above (Jobs 1 and 6 only), but built from # One group per ROCm stack, identified by its base Dockerfile (ROCM_BASE_DOCKERFILE).
# docker/Dockerfile.rock_base and docker/Dockerfile.rock, which install the # The scripts derive the ROCm version, wheel variant, cache keys and image tags from
# ROCm SDK from TheRock wheels instead of the ROCm apt repos. # it (.buildkite/scripts/rocm/stack.sh), so both groups share one set of step
# definitions, anchored on the ROCm 10.0 group.
# #
# Note: ROCm SDK version is determined by ROCM_SDK_VERSION in # - ROCm 10.0 (TheRock, docker/Dockerfile.rocm{,_base}) is the default: it owns the
# docker/Dockerfile.rock_base # :nightly / :latest images and the root wheel index.
# - ROCm 7.2 (docker/Dockerfile.rocm_72{,_base}) is kept for a transition period as
# rocm72 images and wheels.
# #
# vLLM version is determined by the Buildkite checkout (like the CUDA pipeline).
# ============================================================================= # =============================================================================
- group: "Build ROCk Image" - group: "Build ROCm 10.0 Wheel / Image"
key: "build-rock-image" key: "build-rocm100"
depends_on: ~ depends_on: ~
steps: steps:
# ROCk Job 1: Build ROCk Base Image (with ECR caching) - label: ":rocm: Build ROCm 10.0 Base Image & Wheels"
- label: ":rocm: Build ROCk Base Image" id: build-rocm100-base
id: build-rock-base-image
depends_on: ~ depends_on: ~
agents: agents: &rocm_build_agents
queue: cpu_queue_release queue: cpu_queue_release
timeout_in_minutes: 360 timeout_in_minutes: 360
commands: env: &rocm100_env
ROCM_BASE_DOCKERFILE: "docker/Dockerfile.rocm_base"
ROCM_DEFAULT_ARG: "--default"
DOCKER_BUILDKIT: "1"
S3_BUCKET: "vllm-wheels"
DOCKERHUB_USERNAME: "vllmbot"
commands: &rocm_base_commands
- | - |
set -euo pipefail set -euo pipefail
source .buildkite/scripts/rocm/stack.sh "$${ROCM_BASE_DOCKERFILE}"
# The SDK version, torch wheels and component branches all live in SCCACHE_ARGS=(--build-arg USE_SCCACHE=1 --build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache
# the Dockerfile, so its hash is the cache key. --build-arg SCCACHE_REGION_NAME=us-west-2 --build-arg SCCACHE_S3_NO_CREDENTIALS=0)
CACHE_KEY=$$(sha256sum docker/Dockerfile.rock_base | cut -c1-16)
ECR_CACHE_TAG="public.ecr.aws/q9t5s3a7/vllm-release-repo:$${CACHE_KEY}-rock-base"
echo "========================================"
echo "ROCk Base Build Configuration"
echo "========================================"
echo " CACHE_KEY: $${CACHE_KEY}"
echo " ECR_CACHE_TAG: $${ECR_CACHE_TAG}"
echo "========================================"
# Login to ECR
aws ecr-public get-login-password --region us-east-1 | \ aws ecr-public get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7 docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
if docker manifest inspect "$${ECR_CACHE_TAG}" > /dev/null 2>&1; then # Base image, cached in ECR by the base Dockerfile's hash
echo "ECR image cache HIT" if docker manifest inspect "$${ROCM_STACK_ECR_BASE}" > /dev/null 2>&1; then
echo "ECR image cache HIT: $${ROCM_STACK_ECR_BASE}"
else else
echo "CACHE MISS - Building from scratch..." docker buildx build --file "$${ROCM_BASE_DOCKERFILE}" --tag "$${ROCM_STACK_ECR_BASE}" \
"$${SCCACHE_ARGS[@]}" --push .
DOCKER_BUILDKIT=1 docker buildx build \
--file docker/Dockerfile.rock_base \
--tag "$${ECR_CACHE_TAG}" \
--build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 \
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
--push \
.
fi fi
# Save ECR tag for downstream jobs # Base wheels, cached in S3 (see cache-rocm-base-wheels.sh)
buildkite-agent meta-data set "rock-base-image-tag" "$${ECR_CACHE_TAG}" CACHE=.buildkite/scripts/cache-rocm-base-wheels.sh
env: if [ "$$($${CACHE} "$${ROCM_BASE_DOCKERFILE}" check)" = "hit" ]; then
DOCKER_BUILDKIT: "1" $${CACHE} "$${ROCM_BASE_DOCKERFILE}" download
else
DEBS="rocm-base-debs:$${BUILDKITE_BUILD_NUMBER}-$${ROCM_STACK_VARIANT}"
docker buildx build --file "$${ROCM_BASE_DOCKERFILE}" --tag "$${DEBS}" \
--target debs_wheel_release "$${SCCACHE_ARGS[@]}" --load .
mkdir -p "$${ROCM_STACK_BASE_WHEELS_DIR}"
cid=$$(docker create "$${DEBS}")
docker cp "$${cid}":/app/debs/. "$${ROCM_STACK_BASE_WHEELS_DIR}/"
docker rm "$${cid}"
$${CACHE} "$${ROCM_BASE_DOCKERFILE}" upload
fi
buildkite-agent artifact upload "$${ROCM_STACK_BASE_WHEELS_DIR}/*.whl"
# ROCk Job 2: Build ROCk Release Docker Image - label: ":python: Build vLLM ROCm 10.0 Wheel - x86_64"
- label: ":docker: Build release image - x86_64 - ROCk" id: build-rocm100-vllm-wheel
id: build-rock-release-image depends_on: build-rocm100-base
agents: *rocm_build_agents
timeout_in_minutes: 180
env: *rocm100_env
commands: &rocm_vllm_wheel_commands
- |
set -euo pipefail
source .buildkite/scripts/rocm/stack.sh "$${ROCM_BASE_DOCKERFILE}"
# Buildkite's default fetch doesn't update moved tags; version detection needs them
git fetch --tags --force origin
buildkite-agent artifact download "$${ROCM_STACK_BASE_WHEELS_DIR}/*.whl" .
aws ecr-public get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
docker pull "$${ROCM_STACK_ECR_BASE}"
mkdir -p docker/context/base-wheels
touch docker/context/base-wheels/.keep
cp "$${ROCM_STACK_BASE_WHEELS_DIR}"/*.whl docker/context/base-wheels/
docker build --file "$${ROCM_STACK_DOCKERFILE}" --target export_vllm_wheel_release \
--output type=local,dest=rocm-dist \
--build-arg BASE_IMAGE="$${ROCM_STACK_ECR_BASE}" --build-arg REMOTE_VLLM=0 \
--build-arg GIT_REPO_CHECK=1 --build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 --build-arg SCCACHE_S3_NO_CREDENTIALS=0 .
mkdir -p "$${ROCM_STACK_VLLM_WHEEL_DIR}"
cp rocm-dist/*.whl "$${ROCM_STACK_VLLM_WHEEL_DIR}/"
buildkite-agent artifact upload "$${ROCM_STACK_VLLM_WHEEL_DIR}/*.whl"
- label: ":s3: Upload ROCm 10.0 Wheels to S3"
id: upload-rocm100-wheels
depends_on: build-rocm100-vllm-wheel
agents: *rocm_build_agents
timeout_in_minutes: 60
env: *rocm100_env
commands: &rocm_upload_commands
- |
set -euo pipefail
source .buildkite/scripts/rocm/stack.sh "$${ROCM_BASE_DOCKERFILE}"
buildkite-agent artifact download "$${ROCM_STACK_BASE_WHEELS_DIR}/*.whl" .
buildkite-agent artifact download "$${ROCM_STACK_VLLM_WHEEL_DIR}/*.whl" .
bash .buildkite/scripts/upload-rocm-wheels.sh "$${ROCM_BASE_DOCKERFILE}"
- label: ":memo: Annotate ROCm 10.0 wheel release"
id: annotate-rocm100-release
depends_on: upload-rocm100-wheels
agents: *rocm_build_agents
env: *rocm100_env
commands: &rocm_annotate_commands
- bash .buildkite/scripts/annotate-rocm-release.sh "$${ROCM_BASE_DOCKERFILE}" $${ROCM_DEFAULT_ARG}
# Generate https://wheels.vllm.ai/rocm/ (the default stack's variant), so users can
# `uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/` (release only)
- block: "Generate Root Index for ROCm Wheels for Release"
key: block-generate-root-index-rocm-wheels
depends_on:
- upload-rocm100-wheels
- upload-rocm72-wheels
- label: ":package: Generate Root Index for ROCm Wheels for Release"
depends_on: block-generate-root-index-rocm-wheels
id: generate-root-index-rocm-wheels
agents: *rocm_build_agents
env: *rocm100_env
commands:
- "bash tools/vllm-rocm/generate-rocm-wheels-root-index.sh"
- label: ":docker: Build release image - x86_64 - ROCm 10.0"
id: build-rocm100-release-image
depends_on: depends_on:
- step: block-build-release-images - step: block-build-release-images
allow_failure: true allow_failure: true
- step: build-rock-base-image - step: build-rocm100-base
allow_failure: false allow_failure: false
agents: agents: *rocm_build_agents
queue: cpu_queue_release
timeout_in_minutes: 180 timeout_in_minutes: 180
commands: env: *rocm100_env
commands: &rocm_release_image_commands
- | - |
set -euo pipefail set -euo pipefail
source .buildkite/scripts/rocm/stack.sh "$${ROCM_BASE_DOCKERFILE}"
# Login to ECR
aws ecr-public get-login-password --region us-east-1 | \ aws ecr-public get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7 docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
docker pull "$${ROCM_STACK_ECR_BASE}"
ECR_IMAGE_TAG="$$(buildkite-agent meta-data get rock-base-image-tag 2>/dev/null || echo '')" docker build --file "$${ROCM_STACK_DOCKERFILE}" --target vllm-openai \
if [ -z "$${ECR_IMAGE_TAG}" ]; then --tag "$${ROCM_STACK_ECR_IMAGE}" --progress plain \
echo "ERROR: rock-base-image-tag metadata not found" --build-arg max_jobs=16 --build-arg BASE_IMAGE="$${ROCM_STACK_ECR_BASE}" \
echo "This should have been set by the build-rock-base-image job" --build-arg INSTALL_LMCACHE=true --build-arg USE_SCCACHE=1 \
exit 1
fi
docker pull "$${ECR_IMAGE_TAG}"
# Pass the base image ECR tag to downstream steps (nightly publish)
buildkite-agent meta-data set "rock-base-ecr-tag" "$${ECR_IMAGE_TAG}"
echo "========================================"
echo "Building vLLM ROCk release image with:"
echo " BASE_IMAGE: $${ECR_IMAGE_TAG}"
echo " BUILDKITE_COMMIT: $${BUILDKITE_COMMIT}"
echo "========================================"
DOCKER_BUILDKIT=1 docker build \
--build-arg max_jobs=16 \
--build-arg BASE_IMAGE="$${ECR_IMAGE_TAG}" \
--build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \ --build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 \ --build-arg SCCACHE_REGION_NAME=us-west-2 --build-arg SCCACHE_S3_NO_CREDENTIALS=0 .
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \ docker push "$${ROCM_STACK_ECR_IMAGE}"
--build-arg INSTALL_LMCACHE=true \
--tag public.ecr.aws/q9t5s3a7/vllm-release-repo:$${BUILDKITE_COMMIT}-rock \
--target vllm-openai \
--progress plain \
-f docker/Dockerfile.rock .
docker push public.ecr.aws/q9t5s3a7/vllm-release-repo:$${BUILDKITE_COMMIT}-rock - label: "Publish nightly ROCm 10.0 image to DockerHub"
env: id: publish-rocm100-nightly
depends_on: build-rocm100-release-image
if: build.env("NIGHTLY") == "1"
agents: &rocm_publish_agents
queue: small_cpu_queue_release
env: *rocm100_env
commands: &rocm_publish_nightly_commands
- bash .buildkite/scripts/push-nightly-builds-rocm.sh "$${ROCM_BASE_DOCKERFILE}" $${ROCM_DEFAULT_ARG}
plugins: &rocm_dockerhub_login
- docker-login#v3.0.0:
username: vllmbot
password-env: DOCKERHUB_TOKEN
- group: "Build ROCm 7.2 Wheel / Image"
key: "build-rocm72"
depends_on: ~
steps:
- label: ":rocm: Build ROCm 7.2 Base Image & Wheels"
id: build-rocm72-base
depends_on: ~
agents: *rocm_build_agents
timeout_in_minutes: 360
env: &rocm72_env
ROCM_BASE_DOCKERFILE: "docker/Dockerfile.rocm_72_base"
ROCM_DEFAULT_ARG: ""
DOCKER_BUILDKIT: "1" DOCKER_BUILDKIT: "1"
S3_BUCKET: "vllm-wheels"
DOCKERHUB_USERNAME: "vllmbot"
commands: *rocm_base_commands
- label: "Publish nightly ROCk image to DockerHub" - label: ":python: Build vLLM ROCm 7.2 Wheel - x86_64"
id: build-rocm72-vllm-wheel
depends_on: build-rocm72-base
agents: *rocm_build_agents
timeout_in_minutes: 180
env: *rocm72_env
commands: *rocm_vllm_wheel_commands
- label: ":s3: Upload ROCm 7.2 Wheels to S3"
id: upload-rocm72-wheels
depends_on: build-rocm72-vllm-wheel
agents: *rocm_build_agents
timeout_in_minutes: 60
env: *rocm72_env
commands: *rocm_upload_commands
- label: ":memo: Annotate ROCm 7.2 wheel release"
id: annotate-rocm72-release
depends_on: upload-rocm72-wheels
agents: *rocm_build_agents
env: *rocm72_env
commands: *rocm_annotate_commands
- label: ":docker: Build release image - x86_64 - ROCm 7.2"
id: build-rocm72-release-image
depends_on: depends_on:
- build-rock-release-image - step: block-build-release-images
allow_failure: true
- step: build-rocm72-base
allow_failure: false
agents: *rocm_build_agents
timeout_in_minutes: 180
env: *rocm72_env
commands: *rocm_release_image_commands
- label: "Publish nightly ROCm 7.2 image to DockerHub"
id: publish-rocm72-nightly
depends_on: build-rocm72-release-image
if: build.env("NIGHTLY") == "1"
agents: *rocm_publish_agents
env: *rocm72_env
commands: *rocm_publish_nightly_commands
plugins: *rocm_dockerhub_login
- label: "Publish nightly XPU image to DockerHub"
depends_on:
- create-manifest-xpu
if: build.env("NIGHTLY") == "1" if: build.env("NIGHTLY") == "1"
agents: agents:
queue: small_cpu_queue_release queue: small_cpu_queue_release
commands: commands:
- "bash .buildkite/scripts/push-nightly-builds-rocm.sh rock rocm100" - "bash .buildkite/scripts/xpu/push-nightly-builds-xpu.sh"
# Clean up old nightly builds (keep only last 14) - "bash .buildkite/scripts/cleanup-nightly-builds.sh nightly- vllm/vllm-openai-xpu"
- "bash .buildkite/scripts/cleanup-nightly-builds.sh nightly-rocm100- vllm/vllm-openai-rocm"
- "bash .buildkite/scripts/cleanup-nightly-builds.sh base-nightly-rocm100- vllm/vllm-openai-rocm"
plugins: plugins:
- docker-login#v3.0.0: - docker-login#v3.0.0:
username: vllmbot username: vllmbot
@@ -1264,7 +997,8 @@ steps:
- create-multi-arch-manifest-ubuntu2404 - create-multi-arch-manifest-ubuntu2404
- create-multi-arch-manifest-cuda-12-9-ubuntu2404 - create-multi-arch-manifest-cuda-12-9-ubuntu2404
- create-manifest-xpu - create-manifest-xpu
- build-rocm-release-image - build-rocm100-release-image
- build-rocm72-release-image
- input-release-version - input-release-version
# Wait for CPU builds if their block steps were unblocked, so publish # Wait for CPU builds if their block steps were unblocked, so publish
# doesn't race the in-progress CPU build. allow_failure lets publish # doesn't race the in-progress CPU build. allow_failure lets publish
@@ -1289,6 +1023,7 @@ steps:
- "cuda-13-0-ubuntu-24-04" - "cuda-13-0-ubuntu-24-04"
- "cuda-12-9-ubuntu-24-04" - "cuda-12-9-ubuntu-24-04"
- "rocm" - "rocm"
- "rocm72"
- "xpu" - "xpu"
- "cpu" - "cpu"
plugins: plugins:
+40 -66
View File
@@ -2,15 +2,23 @@
# SPDX-License-Identifier: Apache-2.0 # SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# #
# Generate Buildkite annotation for ROCm wheel release # Generate Buildkite annotation for a ROCm stack's wheel release.
#
# Usage: annotate-rocm-release.sh <base-dockerfile> [--default]
# base-dockerfile e.g. docker/Dockerfile.rocm_base or docker/Dockerfile.rocm_72_base
# --default the stack also owns the plain :latest / :v<ver> Docker Hub tags
set -ex set -ex
# Extract build configuration from Dockerfile.rocm_base (single source of truth) # shellcheck source=.buildkite/scripts/rocm/stack.sh
# Extract ROCm version dynamically from Dockerfile.rocm_base source "$(dirname "${BASH_SOURCE[0]}")/rocm/stack.sh" "${1:?Usage: $0 <base-dockerfile> [--default]}"
# BASE_IMAGE format: rocm/dev-ubuntu-22.04:7.0-complete -> extracts "7.0" TAG_VARIANTS=("-${ROCM_STACK_VARIANT}")
ROCM_VERSION=$(grep -E '^ARG BASE_IMAGE=' docker/Dockerfile.rocm_base | sed -E 's/.*:([0-9]+\.[0-9]+).*/\1/' || echo "unknown") if [[ "${2:-}" == "--default" ]]; then
PYTHON_VERSION=$(grep '^ARG PYTHON_VERSION=' docker/Dockerfile.rocm_base | sed 's/^ARG PYTHON_VERSION=//') TAG_VARIANTS=("" "-${ROCM_STACK_VARIANT}")
PYTORCH_ROCM_ARCH=$(grep '^ARG PYTORCH_ROCM_ARCH=' docker/Dockerfile.rocm_base | sed 's/^ARG PYTORCH_ROCM_ARCH=//') fi
ROCM_VERSION="$ROCM_STACK_VERSION"
VARIANT="$ROCM_STACK_VARIANT"
PYTHON_VERSION=$(sed -nE 's/^ARG PYTHON_VERSION="?([^" ]+)"?.*/\1/p' "$ROCM_STACK_BASE_DOCKERFILE")
PYTORCH_ROCM_ARCH=$(sed -nE 's/^ARG PYTORCH_ROCM_ARCH="?([^"]+)"?.*/\1/p' "$ROCM_STACK_BASE_DOCKERFILE")
# Get release version, default to 1.0.0.dev for nightly/per-commit builds # Get release version, default to 1.0.0.dev for nightly/per-commit builds
RELEASE_VERSION=$(buildkite-agent meta-data get release-version 2>/dev/null || echo "") RELEASE_VERSION=$(buildkite-agent meta-data get release-version 2>/dev/null || echo "")
@@ -18,22 +26,32 @@ if [ -z "${RELEASE_VERSION}" ]; then
RELEASE_VERSION="1.0.0.dev" RELEASE_VERSION="1.0.0.dev"
fi fi
ROCM_BASE_CACHE_KEY=$(.buildkite/scripts/cache-rocm-base-wheels.sh key)
# S3 URLs # S3 URLs
S3_BUCKET="${S3_BUCKET:-vllm-wheels}" S3_BUCKET="${S3_BUCKET:-vllm-wheels}"
S3_REGION="${AWS_DEFAULT_REGION:-us-west-2}" S3_REGION="${AWS_DEFAULT_REGION:-us-west-2}"
S3_URL="http://${S3_BUCKET}.s3-website-${S3_REGION}.amazonaws.com" S3_HOST="${S3_BUCKET}.s3-website-${S3_REGION}.amazonaws.com"
S3_URL="http://${S3_HOST}"
WHEEL_PATH="rocm/${BUILDKITE_COMMIT}/${VARIANT}-wheels/"
# Format ROCm version for path (e.g., "7.1" -> "rocm710") DOCKER_CMDS="docker pull ${ROCM_STACK_ECR_BASE}
ROCM_VERSION_PATH="rocm$(echo "${ROCM_VERSION}" | tr -d '.')" docker pull ${ROCM_STACK_ECR_IMAGE}"
ROCM_PATH="rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}" for suffix in "${TAG_VARIANTS[@]}"; do
buildkite-agent annotate --style 'success' --context 'rocm-release-workflow' << EOF for tag in "latest${suffix}" "v${RELEASE_VERSION}${suffix}"; do
## ROCm Wheel and Docker Image Releases DOCKER_CMDS+="
docker tag ${ROCM_STACK_ECR_BASE} vllm/vllm-openai-rocm:${tag}-base
docker tag ${ROCM_STACK_ECR_IMAGE} vllm/vllm-openai-rocm:${tag}
docker push vllm/vllm-openai-rocm:${tag}-base
docker push vllm/vllm-openai-rocm:${tag}"
done
done
buildkite-agent annotate --style 'success' --context "rocm-release-workflow-${VARIANT}" << EOF
## ROCm ${ROCM_VERSION} Wheel and Docker Image Releases
### Build Configuration ### Build Configuration
| Setting | Value | | Setting | Value |
|---------|-------| |---------|-------|
| **ROCm Version** | ${ROCM_VERSION} | | **ROCm Version** | ${ROCM_VERSION} |
| **Index Variant** | ${VARIANT} |
| **Python Version** | ${PYTHON_VERSION} | | **Python Version** | ${PYTHON_VERSION} |
| **GPU Architectures** | ${PYTORCH_ROCM_ARCH} | | **GPU Architectures** | ${PYTORCH_ROCM_ARCH} |
| **Branch** | \`${BUILDKITE_BRANCH}\` | | **Branch** | \`${BUILDKITE_BRANCH}\` |
@@ -41,49 +59,20 @@ buildkite-agent annotate --style 'success' --context 'rocm-release-workflow' <<
### :package: Installation ### :package: Installation
**Install from this build (by commit):**
\`\`\`bash \`\`\`bash
pip install vllm --extra-index-url ${S3_URL}/${ROCM_PATH}/ --trusted-host ${S3_BUCKET}.s3-website-${S3_REGION}.amazonaws.com # This build (by commit)
pip install vllm --extra-index-url ${S3_URL}/rocm/${BUILDKITE_COMMIT}/${VARIANT}/ --trusted-host ${S3_HOST}
# Example for ROCm ${ROCM_VERSION}: # Nightly (if published)
pip install vllm --extra-index-url ${S3_URL}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/ --trusted-host ${S3_BUCKET}.s3-website-${S3_REGION}.amazonaws.com pip install vllm --extra-index-url ${S3_URL}/rocm/nightly/${VARIANT}/ --trusted-host ${S3_HOST}
\`\`\`
**Install from nightly (if published):**
\`\`\`bash
pip install vllm --extra-index-url ${S3_URL}/rocm/nightly/ --trusted-host ${S3_BUCKET}.s3-website-${S3_REGION}.amazonaws.com
\`\`\` \`\`\`
### :floppy_disk: Download Wheels Directly ### :floppy_disk: Download Wheels Directly
\`\`\`bash \`\`\`bash
# List all ROCm wheels aws s3 ls s3://${S3_BUCKET}/${WHEEL_PATH}
aws s3 ls s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/ aws s3 cp --recursive --exclude "*" --include "*.whl" s3://${S3_BUCKET}/${WHEEL_PATH} .
# Download specific wheels
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/vllm-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/torch-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/triton-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/triton-kernels-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/torchvision-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/torchaudio-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/amdsmi-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/amd_aiter-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/flash-attn-*.whl .
\`\`\` \`\`\`
### :gear: Included Packages
- **vllm**: vLLM with ROCm support
- **torch**: PyTorch built for ROCm ${ROCM_VERSION}
- **triton**: Triton
- **triton-kernels**: Triton kernels
- **torchvision**: TorchVision for ROCm PyTorch
- **torchaudio**: Torchaudio for ROCm PyTorch
- **amdsmi**: AMD SMI Python bindings
- **amd_aiter**: Aiter for ROCm
- **flash-attn**: Flash Attention for ROCm
### :warning: Notes ### :warning: Notes
- These wheels are built for **ROCm ${ROCM_VERSION}** and will NOT work with CUDA GPUs - These wheels are built for **ROCm ${ROCM_VERSION}** and will NOT work with CUDA GPUs
- Supported GPU architectures: ${PYTORCH_ROCM_ARCH} - Supported GPU architectures: ${PYTORCH_ROCM_ARCH}
@@ -91,23 +80,8 @@ aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/flash-
### :package: Docker Image Release ### :package: Docker Image Release
To download and upload the image:
\`\`\` \`\`\`
docker pull public.ecr.aws/q9t5s3a7/vllm-release-repo:${BUILDKITE_COMMIT}-rocm-base ${DOCKER_CMDS}
docker pull public.ecr.aws/q9t5s3a7/vllm-release-repo:${BUILDKITE_COMMIT}-rocm
docker tag public.ecr.aws/q9t5s3a7/vllm-release-repo:${ROCM_BASE_CACHE_KEY}-rocm-base vllm/vllm-openai-rocm:${BUILDKITE_COMMIT}-base
docker tag vllm/vllm-openai-rocm:${BUILDKITE_COMMIT}-base vllm/vllm-openai-rocm:latest-base
docker tag vllm/vllm-openai-rocm:${BUILDKITE_COMMIT}-base vllm/vllm-openai-rocm:v${RELEASE_VERSION}-base
docker push vllm/vllm-openai-rocm:latest-base
docker push vllm/vllm-openai-rocm:v${RELEASE_VERSION}-base
docker tag public.ecr.aws/q9t5s3a7/vllm-release-repo:${BUILDKITE_COMMIT}-rocm vllm/vllm-openai-rocm:${BUILDKITE_COMMIT}
docker tag vllm/vllm-openai-rocm:${BUILDKITE_COMMIT} vllm/vllm-openai-rocm:latest
docker tag vllm/vllm-openai-rocm:${BUILDKITE_COMMIT} vllm/vllm-openai-rocm:v${RELEASE_VERSION}
docker push vllm/vllm-openai-rocm:latest
docker push vllm/vllm-openai-rocm:v${RELEASE_VERSION}
\`\`\` \`\`\`
EOF EOF
+28 -34
View File
@@ -5,39 +5,33 @@
# Cache helper for ROCm base wheels # Cache helper for ROCm base wheels
# #
# This script manages caching of pre-built ROCm base wheels (torch, triton, etc.) # This script manages caching of pre-built ROCm base wheels (torch, triton, etc.)
# to avoid rebuilding them when Dockerfile.rocm_base hasn't changed. # to avoid rebuilding them when the stack's base Dockerfile hasn't changed.
# #
# Usage: # Usage: cache-rocm-base-wheels.sh <base-dockerfile> <command>
# cache-rocm-base-wheels.sh check - Check if cache exists, outputs "hit" or "miss" # base-dockerfile e.g. docker/Dockerfile.rocm_base or docker/Dockerfile.rocm_72_base
# cache-rocm-base-wheels.sh upload - Upload wheels to cache # Commands:
# cache-rocm-base-wheels.sh download - Download wheels from cache # check - Check if cache exists, outputs "hit" or "miss"
# cache-rocm-base-wheels.sh key - Output the cache key # upload - Upload wheels to cache
# download - Download wheels from cache
# key - Output the cache key
# path - Output the full S3 cache path
# #
# Environment variables: # Environment variables:
# S3_BUCKET - S3 bucket name (default: vllm-wheels) # S3_BUCKET - S3 bucket name (default: vllm-wheels)
# #
# Note: ROCm version is determined by BASE_IMAGE in Dockerfile.rocm_base, # The cache key hashes the base Dockerfile and the repo files it COPYs in, which
# so changes to ROCm version are captured by the Dockerfile hash. # capture the ROCm version and every component pin (see rocm/stack.sh).
set -euo pipefail set -euo pipefail
BUCKET="${S3_BUCKET:-vllm-wheels}" BUCKET="${S3_BUCKET:-vllm-wheels}"
DOCKERFILE="docker/Dockerfile.rocm_base"
CACHE_PREFIX="rocm/cache" CACHE_PREFIX="rocm/cache"
# Generate hash from Dockerfile content + build args # shellcheck source=.buildkite/scripts/rocm/stack.sh
generate_cache_key() { source "$(dirname "${BASH_SOURCE[0]}")/rocm/stack.sh" "${1:?Usage: $0 <base-dockerfile> <command>}"
# Include Dockerfile content shift
if [[ ! -f "$DOCKERFILE" ]]; then CACHE_KEY="$ROCM_STACK_WHEEL_KEY"
echo "ERROR: Dockerfile not found: $DOCKERFILE" >&2 WHEELS_DIR="$ROCM_STACK_BASE_WHEELS_DIR"
exit 1
fi
local dockerfile_hash=$(sha256sum "$DOCKERFILE" | cut -c1-16)
echo "${dockerfile_hash}"
}
CACHE_KEY=$(generate_cache_key)
CACHE_PATH="s3://${BUCKET}/${CACHE_PREFIX}/${CACHE_KEY}/" CACHE_PATH="s3://${BUCKET}/${CACHE_PREFIX}/${CACHE_KEY}/"
case "${1:-}" in case "${1:-}" in
@@ -67,19 +61,19 @@ case "${1:-}" in
echo "Cache path: ${CACHE_PATH}" echo "Cache path: ${CACHE_PATH}"
echo "" echo ""
if [[ ! -d "artifacts/rocm-base-wheels" ]]; then if [[ ! -d "${WHEELS_DIR}" ]]; then
echo "ERROR: artifacts/rocm-base-wheels directory not found" >&2 echo "ERROR: ${WHEELS_DIR} directory not found" >&2
exit 1 exit 1
fi fi
WHEEL_COUNT=$(find artifacts/rocm-base-wheels -maxdepth 1 -name '*.whl' 2>/dev/null | wc -l) WHEEL_COUNT=$(find "${WHEELS_DIR}" -maxdepth 1 -name '*.whl' 2>/dev/null | wc -l)
if [[ "$WHEEL_COUNT" -eq 0 ]]; then if [[ "$WHEEL_COUNT" -eq 0 ]]; then
echo "ERROR: No wheels found in artifacts/rocm-base-wheels/" >&2 echo "ERROR: No wheels found in ${WHEELS_DIR}/" >&2
exit 1 exit 1
fi fi
echo "Uploading $WHEEL_COUNT wheels..." echo "Uploading $WHEEL_COUNT wheels..."
aws s3 cp --recursive artifacts/rocm-base-wheels/ "${CACHE_PATH}" aws s3 cp --recursive "${WHEELS_DIR}/" "${CACHE_PATH}"
echo "" echo ""
echo "Cache upload complete!" echo "Cache upload complete!"
@@ -93,17 +87,17 @@ case "${1:-}" in
echo "Cache key: ${CACHE_KEY}" echo "Cache key: ${CACHE_KEY}"
echo "Cache path: ${CACHE_PATH}" echo "Cache path: ${CACHE_PATH}"
echo "" echo ""
mkdir -p artifacts/rocm-base-wheels mkdir -p "${WHEELS_DIR}"
# Use sync with include/exclude to only download .whl files # Use sync with include/exclude to only download .whl files
aws s3 sync "${CACHE_PATH}" artifacts/rocm-base-wheels/ \ aws s3 sync "${CACHE_PATH}" "${WHEELS_DIR}/" \
--exclude "*" \ --exclude "*" \
--include "*.whl" --include "*.whl"
echo "" echo ""
echo "Downloaded wheels:" echo "Downloaded wheels:"
find artifacts/rocm-base-wheels -maxdepth 1 -name '*.whl' -exec ls -lh {} \; find "${WHEELS_DIR}" -maxdepth 1 -name '*.whl' -exec ls -lh {} \;
WHEEL_COUNT=$(find artifacts/rocm-base-wheels -maxdepth 1 -name '*.whl' 2>/dev/null | wc -l) WHEEL_COUNT=$(find "${WHEELS_DIR}" -maxdepth 1 -name '*.whl' 2>/dev/null | wc -l)
echo "" echo ""
echo "Total: $WHEEL_COUNT wheels" echo "Total: $WHEEL_COUNT wheels"
echo "========================================" echo "========================================"
@@ -118,12 +112,12 @@ case "${1:-}" in
;; ;;
*) *)
echo "Usage: $0 {check|upload|download|key|path}" >&2 echo "Usage: $0 <base-dockerfile> {check|upload|download|key|path}" >&2
echo "" >&2 echo "" >&2
echo "Commands:" >&2 echo "Commands:" >&2
echo " check - Check if cache exists, outputs 'hit' or 'miss'" >&2 echo " check - Check if cache exists, outputs 'hit' or 'miss'" >&2
echo " upload - Upload wheels from artifacts/rocm-base-wheels/ to cache" >&2 echo " upload - Upload wheels from ${WHEELS_DIR}/ to cache" >&2
echo " download - Download wheels from cache to artifacts/rocm-base-wheels/" >&2 echo " download - Download wheels from cache to ${WHEELS_DIR}/" >&2
echo " key - Output the cache key" >&2 echo " key - Output the cache key" >&2
echo " path - Output the full S3 cache path" >&2 echo " path - Output the full S3 cache path" >&2
exit 1 exit 1
+7 -54
View File
@@ -17,7 +17,7 @@ DEFAULT_REPO_SLUG="vllm-project/vllm"
DEFAULT_CI_HCL_SOURCE="docker/ci-rocm.hcl" DEFAULT_CI_HCL_SOURCE="docker/ci-rocm.hcl"
DEFAULT_CI_BASE_CONTENT_FILES=".dockerignore requirements/common.txt requirements/rocm.txt requirements/test/rocm.txt tools/install_torchcodec_rocm.sh rust-toolchain.toml tests/vllm_test_utils" DEFAULT_CI_BASE_CONTENT_FILES=".dockerignore requirements/common.txt requirements/rocm.txt requirements/test/rocm.txt tools/install_torchcodec_rocm.sh rust-toolchain.toml tests/vllm_test_utils"
DEFAULT_CI_BASE_DOCKERFILE="docker/Dockerfile.rocm" DEFAULT_CI_BASE_DOCKERFILE="docker/Dockerfile.rocm"
DEFAULT_CI_BASE_DOCKERFILE_STAGES="base rust_toolchain_input_0 rust-toolchain-input rust-toolchain build_nixl lmcache_source build_lmcache build_rocshmem build_deepep mori_base ci_base" DEFAULT_CI_BASE_DOCKERFILE_STAGES="base rust_toolchain_input_0 rust-toolchain-input rust-toolchain build_nixl lmcache_source build_lmcache build_deepep mori_base ci_base"
DEFAULT_CI_BASE_METADATA_VERSION="3" DEFAULT_CI_BASE_METADATA_VERSION="3"
# ROCm CI forces REMOTE_VLLM=0, so content identity covers only the selected # ROCm CI forces REMOTE_VLLM=0, so content identity covers only the selected
# local-source stages rather than unreachable remote-fetch alternatives. # local-source stages rather than unreachable remote-fetch alternatives.
@@ -555,9 +555,6 @@ configure_custom_rocm_stages() {
CI_BASE_DOCKERFILE_STAGES="${CI_BASE_DOCKERFILE_STAGES:-${stages}}" CI_BASE_DOCKERFILE_STAGES="${CI_BASE_DOCKERFILE_STAGES:-${stages}}"
ROCM_CSRC_DOCKERFILE_STAGES="${ROCM_CSRC_DOCKERFILE_STAGES:-${stages}}" ROCM_CSRC_DOCKERFILE_STAGES="${ROCM_CSRC_DOCKERFILE_STAGES:-${stages}}"
ROCM_RUST_DOCKERFILE_STAGES="${ROCM_RUST_DOCKERFILE_STAGES:-${stages}}" ROCM_RUST_DOCKERFILE_STAGES="${ROCM_RUST_DOCKERFILE_STAGES:-${stages}}"
if [[ " ${stages} " != *" build_rocshmem "* ]]; then
unset ROCSHMEM_BRANCH ROCSHMEM_CACHE_KEY DEEPEP_CACHE_KEY
fi
} }
compose_dependency_cache_key() { compose_dependency_cache_key() {
@@ -1724,14 +1721,11 @@ ci_base_metadata_pairs() {
metadata_pair "vllm.rocm.nixl_commit" "${NIXL_BRANCH:-$(resolve_dockerfile_arg_value "${dockerfile}" "NIXL_BRANCH")}" metadata_pair "vllm.rocm.nixl_commit" "${NIXL_BRANCH:-$(resolve_dockerfile_arg_value "${dockerfile}" "NIXL_BRANCH")}"
metadata_pair "vllm.rocm.ucx_repo" "$(resolve_dockerfile_arg_value "${dockerfile}" "UCX_REPO")" metadata_pair "vllm.rocm.ucx_repo" "$(resolve_dockerfile_arg_value "${dockerfile}" "UCX_REPO")"
metadata_pair "vllm.rocm.ucx_commit" "${UCX_BRANCH:-$(resolve_dockerfile_arg_value "${dockerfile}" "UCX_BRANCH")}" metadata_pair "vllm.rocm.ucx_commit" "${UCX_BRANCH:-$(resolve_dockerfile_arg_value "${dockerfile}" "UCX_BRANCH")}"
metadata_pair "vllm.rocm.rocshmem_repo" "$(resolve_dockerfile_arg_value "${dockerfile}" "ROCSHMEM_REPO")"
metadata_pair "vllm.rocm.rocshmem_commit" "${ROCSHMEM_BRANCH:-$(resolve_dockerfile_arg_value "${dockerfile}" "ROCSHMEM_BRANCH")}"
metadata_pair "vllm.rocm.deepep_repo" "$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_REPO")" metadata_pair "vllm.rocm.deepep_repo" "$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_REPO")"
metadata_pair "vllm.rocm.deepep_commit" "${DEEPEP_BRANCH:-$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_BRANCH")}" metadata_pair "vllm.rocm.deepep_commit" "${DEEPEP_BRANCH:-$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_BRANCH")}"
metadata_pair "vllm.rocm.deepep_nic" "$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_NIC")" metadata_pair "vllm.rocm.deepep_nic" "$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_NIC")"
metadata_pair "vllm.rocm.deepep_rocm_arch" "$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_ROCM_ARCH")" metadata_pair "vllm.rocm.deepep_rocm_arch" "$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_ROCM_ARCH")"
metadata_pair "vllm.rocm.nixl_cache_key" "${NIXL_CACHE_KEY:-}" metadata_pair "vllm.rocm.nixl_cache_key" "${NIXL_CACHE_KEY:-}"
metadata_pair "vllm.rocm.rocshmem_cache_key" "${ROCSHMEM_CACHE_KEY:-}"
metadata_pair "vllm.rocm.deepep_cache_key" "${DEEPEP_CACHE_KEY:-}" metadata_pair "vllm.rocm.deepep_cache_key" "${DEEPEP_CACHE_KEY:-}"
} }
@@ -2314,7 +2308,7 @@ extract_dependency_pins() {
return 0 return 0
fi fi
for var in NIXL_BRANCH UCX_BRANCH ROCSHMEM_BRANCH DEEPEP_BRANCH; do for var in NIXL_BRANCH UCX_BRANCH DEEPEP_BRANCH; do
if [[ -n "${!var:-}" ]]; then if [[ -n "${!var:-}" ]]; then
echo "Using provided ${var}: ${!var}" echo "Using provided ${var}: ${!var}"
continue continue
@@ -2336,17 +2330,14 @@ compute_dependency_cache_keys() {
local dockerfile_rocm="" local dockerfile_rocm=""
local nixl_branch="" local nixl_branch=""
local ucx_branch="" local ucx_branch=""
local rocshmem_branch=""
local deepep_branch="" local deepep_branch=""
local nixl_material="" local nixl_material=""
local rocshmem_material=""
local deepep_material="" local deepep_material=""
bake_dir=$(dirname "${VLLM_BAKE_FILE}") bake_dir=$(dirname "${VLLM_BAKE_FILE}")
dockerfile_rocm="${CI_BASE_DOCKERFILE:-${bake_dir}/Dockerfile.rocm}" dockerfile_rocm="${CI_BASE_DOCKERFILE:-${bake_dir}/Dockerfile.rocm}"
nixl_branch=$(resolve_dockerfile_arg_value "${dockerfile_rocm}" "NIXL_BRANCH") nixl_branch=$(resolve_dockerfile_arg_value "${dockerfile_rocm}" "NIXL_BRANCH")
ucx_branch=$(resolve_dockerfile_arg_value "${dockerfile_rocm}" "UCX_BRANCH") ucx_branch=$(resolve_dockerfile_arg_value "${dockerfile_rocm}" "UCX_BRANCH")
rocshmem_branch=$(resolve_dockerfile_arg_value "${dockerfile_rocm}" "ROCSHMEM_BRANCH")
deepep_branch=$(resolve_dockerfile_arg_value "${dockerfile_rocm}" "DEEPEP_BRANCH") deepep_branch=$(resolve_dockerfile_arg_value "${dockerfile_rocm}" "DEEPEP_BRANCH")
if [[ -n "${nixl_branch}" && -n "${ucx_branch}" ]]; then if [[ -n "${nixl_branch}" && -n "${ucx_branch}" ]]; then
@@ -2360,22 +2351,11 @@ compute_dependency_cache_keys() {
echo "NIXL dependency cache key: ${NIXL_CACHE_KEY}" echo "NIXL dependency cache key: ${NIXL_CACHE_KEY}"
fi fi
if [[ -n "${rocshmem_branch}" ]]; then if [[ -n "${deepep_branch}" ]]; then
rocshmem_material=$(compose_stage_cache_material "${dockerfile_rocm}" "base build_rocshmem") deepep_material=$(compose_stage_cache_material "${dockerfile_rocm}" "base build_deepep")
ROCSHMEM_CACHE_KEY=$(
compose_dependency_cache_key \
"${rocshmem_branch}" \
"${rocshmem_material}"
)
export ROCSHMEM_CACHE_KEY
echo "ROCShmem dependency cache key: ${ROCSHMEM_CACHE_KEY}"
fi
if [[ -n "${deepep_branch}" && -n "${rocshmem_branch}" ]]; then
deepep_material=$(compose_stage_cache_material "${dockerfile_rocm}" "base build_rocshmem build_deepep")
DEEPEP_CACHE_KEY=$( DEEPEP_CACHE_KEY=$(
compose_dependency_cache_key \ compose_dependency_cache_key \
"${deepep_branch}-rocshmem-${rocshmem_branch}" \ "${deepep_branch}" \
"${deepep_material}" "${deepep_material}"
) )
export DEEPEP_CACHE_KEY export DEEPEP_CACHE_KEY
@@ -2415,18 +2395,11 @@ dependency_cache_ref_for_target() {
printf '%s\n' "${cache_repo}:nixl-rocm-${NIXL_BRANCH}-ucx-${UCX_BRANCH:-}" printf '%s\n' "${cache_repo}:nixl-rocm-${NIXL_BRANCH}-ucx-${UCX_BRANCH:-}"
fi fi
;; ;;
rocshmem-rocm-ci)
if [[ -n "${ROCSHMEM_CACHE_KEY:-}" ]]; then
printf '%s\n' "${cache_repo}:rocshmem-rocm-${ROCSHMEM_CACHE_KEY}"
elif [[ -n "${ROCSHMEM_BRANCH:-}" ]]; then
printf '%s\n' "${cache_repo}:rocshmem-rocm-${ROCSHMEM_BRANCH}"
fi
;;
deepep-rocm-ci) deepep-rocm-ci)
if [[ -n "${DEEPEP_CACHE_KEY:-}" ]]; then if [[ -n "${DEEPEP_CACHE_KEY:-}" ]]; then
printf '%s\n' "${cache_repo}:deepep-rocm-${DEEPEP_CACHE_KEY}" printf '%s\n' "${cache_repo}:deepep-rocm-${DEEPEP_CACHE_KEY}"
elif [[ -n "${DEEPEP_BRANCH:-}" ]]; then elif [[ -n "${DEEPEP_BRANCH:-}" ]]; then
printf '%s\n' "${cache_repo}:deepep-rocm-${DEEPEP_BRANCH}-rocshmem-${ROCSHMEM_BRANCH:-}" printf '%s\n' "${cache_repo}:deepep-rocm-${DEEPEP_BRANCH}"
fi fi
;; ;;
esac esac
@@ -2444,7 +2417,6 @@ add_dependency_cache_target() {
resolve_ci_base_dependency_targets() { resolve_ci_base_dependency_targets() {
local mode="${ROCM_DEP_CACHE_EXPORT_MODE:-missing}" local mode="${ROCM_DEP_CACHE_EXPORT_MODE:-missing}"
local nixl_ref="" local nixl_ref=""
local rocshmem_ref=""
local deepep_ref="" local deepep_ref=""
[[ "${TARGET}" == "ci-base-rocm-ci-with-deps" ]] || return 0 [[ "${TARGET}" == "ci-base-rocm-ci-with-deps" ]] || return 0
@@ -2452,7 +2424,7 @@ resolve_ci_base_dependency_targets() {
case "${mode}" in case "${mode}" in
always) always)
echo "ROCM_DEP_CACHE_EXPORT_MODE=always; exporting all dependency caches serially" echo "ROCM_DEP_CACHE_EXPORT_MODE=always; exporting all dependency caches serially"
for target in nixl-rocm-ci rocshmem-rocm-ci deepep-rocm-ci; do for target in nixl-rocm-ci deepep-rocm-ci; do
if [[ -n "$(dependency_cache_ref_for_target "${target}")" ]]; then if [[ -n "$(dependency_cache_ref_for_target "${target}")" ]]; then
add_dependency_cache_target "${target}" add_dependency_cache_target "${target}"
fi fi
@@ -2482,16 +2454,6 @@ resolve_ci_base_dependency_targets() {
fi fi
fi fi
if [[ "${mode}" != "always" && -n "${ROCSHMEM_CACHE_KEY:-}" ]]; then
rocshmem_ref=$(dependency_cache_ref_for_target "rocshmem-rocm-ci")
if dependency_cache_ref_exists "${rocshmem_ref}"; then
echo "ROCShmem dependency cache exists: ${rocshmem_ref}"
else
echo "ROCShmem dependency cache missing; will seed: ${rocshmem_ref}"
add_dependency_cache_target "rocshmem-rocm-ci"
fi
fi
if [[ "${mode}" != "always" && -n "${DEEPEP_CACHE_KEY:-}" ]]; then if [[ "${mode}" != "always" && -n "${DEEPEP_CACHE_KEY:-}" ]]; then
deepep_ref=$(dependency_cache_ref_for_target "deepep-rocm-ci") deepep_ref=$(dependency_cache_ref_for_target "deepep-rocm-ci")
if dependency_cache_ref_exists "${deepep_ref}"; then if dependency_cache_ref_exists "${deepep_ref}"; then
@@ -2502,15 +2464,6 @@ resolve_ci_base_dependency_targets() {
fi fi
fi fi
# DeepEP inherits from ROCShmem. If ROCShmem is being seeded, seed DeepEP too
# so the pair stays consistent for future ci_base rebuilds.
if printf '%s\n' "${DEPENDENCY_CACHE_TARGETS[@]}" | grep -qx "rocshmem-rocm-ci" \
&& ! printf '%s\n' "${DEPENDENCY_CACHE_TARGETS[@]}" | grep -qx "deepep-rocm-ci" \
&& [[ -n "${DEEPEP_BRANCH:-}" ]]; then
echo "ROCShmem cache is missing; also seeding DeepEP cache"
add_dependency_cache_target "deepep-rocm-ci"
fi
BAKE_TARGETS=("ci-base-rocm-ci") BAKE_TARGETS=("ci-base-rocm-ci")
if [[ ${#DEPENDENCY_CACHE_TARGETS[@]} -eq 0 ]]; then if [[ ${#DEPENDENCY_CACHE_TARGETS[@]} -eq 0 ]]; then
echo "All dependency caches exist; building ci_base without dependency cache exports" echo "All dependency caches exist; building ci_base without dependency cache exports"
+5 -2
View File
@@ -5,14 +5,17 @@ set -ex
# Clean up old nightly builds from DockerHub, keeping only the last 14 builds # Clean up old nightly builds from DockerHub, keeping only the last 14 builds
# This script uses DockerHub API to list and delete old tags with specified prefix # This script uses DockerHub API to list and delete old tags with specified prefix
# Tags starting with "nightly-dev" are always excluded from deletion # Tags starting with "nightly-dev" are always excluded from deletion
# Usage: cleanup-nightly-builds.sh [TAG_PREFIX] [REPO] # Usage: cleanup-nightly-builds.sh [TAG_PREFIX] [REPO] [EXCLUDE_PREFIX]
# Example: cleanup-nightly-builds.sh "nightly-" # Example: cleanup-nightly-builds.sh "nightly-"
# Example: cleanup-nightly-builds.sh "cu130-nightly-" # Example: cleanup-nightly-builds.sh "cu130-nightly-"
# Example: cleanup-nightly-builds.sh "nightly-" "vllm/vllm-openai-rocm" # Example: cleanup-nightly-builds.sh "nightly-" "vllm/vllm-openai-rocm"
# Example: cleanup-nightly-builds.sh "nightly-" "vllm/vllm-openai-rocm" "nightly-rocm"
# (EXCLUDE_PREFIX skips tags owned by another variant sharing TAG_PREFIX)
# Get tag prefix and repo from arguments # Get tag prefix and repo from arguments
TAG_PREFIX="${1:-nightly-}" TAG_PREFIX="${1:-nightly-}"
REPO="${2:-vllm/vllm-openai}" REPO="${2:-vllm/vllm-openai}"
EXCLUDE_PREFIX="${3:-}"
echo "Cleaning up tags with prefix: $TAG_PREFIX in repository: $REPO" echo "Cleaning up tags with prefix: $TAG_PREFIX in repository: $REPO"
@@ -57,7 +60,7 @@ get_all_tags() {
# Get both last_updated timestamp and tag name, separated by | # Get both last_updated timestamp and tag name, separated by |
# Exclude mutable architecture tags and nightly-dev tags from cleanup # Exclude mutable architecture tags and nightly-dev tags from cleanup
local tags=$(echo "$response" | jq -r --arg prefix "$TAG_PREFIX" '.results[] | select(.name | startswith($prefix)) | select(.name != ($prefix + "x86_64") and .name != ($prefix + "aarch64")) | select(.name | startswith("nightly-dev") | not) | "\(.last_updated)|\(.name)"') local tags=$(echo "$response" | jq -r --arg prefix "$TAG_PREFIX" --arg exclude "$EXCLUDE_PREFIX" '.results[] | select(.name | startswith($prefix)) | select($exclude == "" or (.name | startswith($exclude) | not)) | select(.name != ($prefix + "x86_64") and .name != ($prefix + "aarch64")) | select(.name | startswith("nightly-dev") | not) | "\(.last_updated)|\(.name)"')
if [ -z "$tags" ]; then if [ -z "$tags" ]; then
break break
+63 -9
View File
@@ -12,7 +12,7 @@ from dataclasses import asdict, dataclass
from datetime import datetime from datetime import datetime
from pathlib import Path from pathlib import Path
from typing import Any from typing import Any
from urllib.parse import quote from urllib.parse import quote, unquote, urlparse
import regex as re import regex as re
@@ -51,6 +51,8 @@ class WheelFileInfo:
platform_tag: str platform_tag: str
variant: str | None variant: str | None
filename: str filename: str
# Absolute URL for files hosted elsewhere (see --external-links)
url: str | None = None
def parse_from_filename(file: str) -> WheelFileInfo: def parse_from_filename(file: str) -> WheelFileInfo:
@@ -110,6 +112,29 @@ def parse_from_filename(file: str) -> WheelFileInfo:
) )
def parse_external_url(url: str) -> WheelFileInfo:
"""Parse an absolute wheel or sdist URL hosted on another index."""
filename = unquote(urlparse(url).path.rsplit("/", 1)[-1])
if filename.endswith(".whl"):
info = parse_from_filename(filename)
else:
match = re.match(r"^(?P<name>.+)-(?P<version>[^-]+)\.(tar\.gz|zip)$", filename)
if not match:
raise ValueError(f"Unsupported external file: {filename}")
info = WheelFileInfo(
package_name=match.group("name"),
version=match.group("version"),
build_tag=None,
python_tag="source",
abi_tag="none",
platform_tag="any",
variant=None,
filename=filename,
)
info.url = url
return info
def generate_project_list(subdir_names: list[str], comment: str = "") -> str: def generate_project_list(subdir_names: list[str], comment: str = "") -> str:
"""Generate project list HTML content linking to each project & variant subdirectory.""" """Generate project list HTML content linking to each project & variant subdirectory."""
href_tags = [] href_tags = []
@@ -129,14 +154,19 @@ def generate_package_index_and_metadata(
href_tags = [] href_tags = []
metadata = [] metadata = []
for file in sorted(wheel_files, key=lambda x: x.filename): for file in sorted(wheel_files, key=lambda x: x.filename):
relative_path = ( if file.url:
wheel_base_dir.relative_to(index_base_dir, walk_up=True) / file.filename file_path_quoted = file.url
) else:
# handle with '+' in URL, and avoid double-encoding '/' and already-encoded '%2B' relative_path = (
# NOTE: this is AWS S3 specific behavior! wheel_base_dir.relative_to(index_base_dir, walk_up=True) / file.filename
file_path_quoted = quote(relative_path.as_posix(), safe=":%/") )
# handle with '+' in URL, and avoid double-encoding '/' and already-encoded '%2B'
# NOTE: this is AWS S3 specific behavior!
file_path_quoted = quote(relative_path.as_posix(), safe=":%/")
href_tags.append(f' <a href="{file_path_quoted}">{file.filename}</a><br/>') href_tags.append(f' <a href="{file_path_quoted}">{file.filename}</a><br/>')
file_meta = asdict(file) file_meta = asdict(file)
if file_meta["url"] is None:
del file_meta["url"]
file_meta["path"] = file_path_quoted file_meta["path"] = file_path_quoted
metadata.append(file_meta) metadata.append(file_meta)
index_str = INDEX_HTML_TEMPLATE.format(items="\n".join(href_tags), comment=comment) index_str = INDEX_HTML_TEMPLATE.format(items="\n".join(href_tags), comment=comment)
@@ -151,6 +181,7 @@ def generate_index_and_metadata(
default_variant: str | None = None, default_variant: str | None = None,
alias_to_default: str | None = None, alias_to_default: str | None = None,
comment: str = "", comment: str = "",
external_urls: list[str] | None = None,
): ):
"""Generate index for all wheel files. """Generate index for all wheel files.
@@ -160,6 +191,8 @@ def generate_index_and_metadata(
index_base_dir (Path): Base directory to store index files. index_base_dir (Path): Base directory to store index files.
default_variant (str | None): The default variant name, if any. default_variant (str | None): The default variant name, if any.
alias_to_default (str | None): Alias variant name for the default variant, if any. alias_to_default (str | None): Alias variant name for the default variant, if any.
external_urls (list[str] | None): Absolute URLs of files hosted on another index,
listed in the generated indices without being copied.
comment (str | None): Optional comment to include in the generated HTML files. comment (str | None): Optional comment to include in the generated HTML files.
First, parse all wheel files to extract metadata. First, parse all wheel files to extract metadata.
@@ -214,6 +247,7 @@ def generate_index_and_metadata(
""" """
parsed_files = [parse_from_filename(f) for f in whl_files] parsed_files = [parse_from_filename(f) for f in whl_files]
parsed_files += [parse_external_url(u) for u in external_urls or []]
if not parsed_files: if not parsed_files:
print("No wheel files found, skipping index generation.") print("No wheel files found, skipping index generation.")
@@ -232,10 +266,14 @@ def generate_index_and_metadata(
print(f"Detected ROCm variant from vllm: {rocm_variant}") print(f"Detected ROCm variant from vllm: {rocm_variant}")
break break
# Apply ROCm variant to all wheels without a variant # Apply ROCm variant to all wheels without a variant. Dependencies with their
# own ROCm local version (e.g. TheRock's torch-2.12.0+rocm10.0.0) belong to
# the same variant rather than one of their own.
if rocm_variant: if rocm_variant:
for file in parsed_files: for file in parsed_files:
if file.variant is None: if file.variant is None or (
file.variant.startswith("rocm") and file.variant != rocm_variant
):
file.variant = rocm_variant file.variant = rocm_variant
print(f"Inherited variant '{rocm_variant}' for {file.filename}") print(f"Inherited variant '{rocm_variant}' for {file.filename}")
@@ -366,6 +404,13 @@ if __name__ == "__main__":
default=None, default=None,
help="Alias variant name for the default variant", help="Alias variant name for the default variant",
) )
parser.add_argument(
"--external-links",
type=str,
default=None,
help="Optional file with one absolute URL per line for packages hosted on "
"another index (listed alongside, but not copied into, this index)",
)
parser.add_argument( parser.add_argument(
"--comment", "--comment",
type=str, type=str,
@@ -463,5 +508,14 @@ if __name__ == "__main__":
default_variant=None, default_variant=None,
alias_to_default=args.alias_to_default, alias_to_default=args.alias_to_default,
comment=args.comment.strip(), comment=args.comment.strip(),
external_urls=(
[
line.strip()
for line in Path(args.external_links).read_text().splitlines()
if line.strip() and not line.startswith("#")
]
if args.external_links
else None
),
) )
print(f"Successfully generated index and metadata in {output_dir}") print(f"Successfully generated index and metadata in {output_dir}")
+28 -16
View File
@@ -11,9 +11,9 @@ set -euo pipefail
TARGET="${1:-all}" TARGET="${1:-all}"
case "${TARGET}" in case "${TARGET}" in
cuda-13-0 | cuda-12-9 | cuda-13-0-ubuntu-24-04 | \ cuda-13-0 | cuda-12-9 | cuda-13-0-ubuntu-24-04 | \
cuda-12-9-ubuntu-24-04 | rocm | xpu | cpu | all) ;; cuda-12-9-ubuntu-24-04 | rocm | rocm72 | xpu | cpu | all) ;;
*) *)
echo "Usage: $0 {cuda-13-0|cuda-12-9|cuda-13-0-ubuntu-24-04|cuda-12-9-ubuntu-24-04|rocm|xpu|cpu|all}" echo "Usage: $0 {cuda-13-0|cuda-12-9|cuda-13-0-ubuntu-24-04|cuda-12-9-ubuntu-24-04|rocm|rocm72|xpu|cpu|all}"
exit 2 exit 2
;; ;;
esac esac
@@ -143,23 +143,35 @@ if target_enabled cuda-12-9-ubuntu-24-04; then
fi fi
# ---- ROCm ---- # ---- ROCm ----
# Each ROCm stack is identified by its base Dockerfile (see rocm/stack.sh). The
# default stack also owns the plain :latest / :v<ver> tags.
publish_rocm_stack() {
local base_dockerfile="$1" default="$2" suffix
(
# shellcheck source=.buildkite/scripts/rocm/stack.sh
source .buildkite/scripts/rocm/stack.sh "$base_dockerfile"
local suffixes=("-${ROCM_STACK_VARIANT}")
[[ "$default" == "1" ]] && suffixes=("" "-${ROCM_STACK_VARIANT}")
docker pull "$ROCM_STACK_ECR_IMAGE"
docker pull "$ROCM_STACK_ECR_BASE"
for suffix in "${suffixes[@]}"; do
for tag in "latest${suffix}" "v${RELEASE_VERSION}${suffix}"; do
docker tag "$ROCM_STACK_ECR_IMAGE" "vllm/vllm-openai-rocm:${tag}"
docker tag "$ROCM_STACK_ECR_BASE" "vllm/vllm-openai-rocm:${tag}-base"
docker push "vllm/vllm-openai-rocm:${tag}"
docker push "vllm/vllm-openai-rocm:${tag}-base"
done
done
)
}
if target_enabled rocm; then if target_enabled rocm; then
ROCM_BASE_CACHE_KEY=$(.buildkite/scripts/cache-rocm-base-wheels.sh key) publish_rocm_stack docker/Dockerfile.rocm_base 1
echo "ROCm base cache key: ${ROCM_BASE_CACHE_KEY}" fi
docker pull "public.ecr.aws/q9t5s3a7/vllm-release-repo:${COMMIT}-rocm" if target_enabled rocm72; then
docker pull "public.ecr.aws/q9t5s3a7/vllm-release-repo:${ROCM_BASE_CACHE_KEY}-rocm-base" publish_rocm_stack docker/Dockerfile.rocm_72_base 0
docker tag "public.ecr.aws/q9t5s3a7/vllm-release-repo:${COMMIT}-rocm" vllm/vllm-openai-rocm:latest
docker tag "public.ecr.aws/q9t5s3a7/vllm-release-repo:${COMMIT}-rocm" "vllm/vllm-openai-rocm:v${RELEASE_VERSION}"
docker push vllm/vllm-openai-rocm:latest
docker push "vllm/vllm-openai-rocm:v${RELEASE_VERSION}"
docker tag "public.ecr.aws/q9t5s3a7/vllm-release-repo:${ROCM_BASE_CACHE_KEY}-rocm-base" vllm/vllm-openai-rocm:latest-base
docker tag "public.ecr.aws/q9t5s3a7/vllm-release-repo:${ROCM_BASE_CACHE_KEY}-rocm-base" "vllm/vllm-openai-rocm:v${RELEASE_VERSION}-base"
docker push vllm/vllm-openai-rocm:latest-base
docker push "vllm/vllm-openai-rocm:v${RELEASE_VERSION}-base"
fi fi
# ---- XPU ---- # ---- XPU ----
+42 -65
View File
@@ -2,42 +2,36 @@
# SPDX-License-Identifier: Apache-2.0 # SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project # SPDX-FileCopyrightText: Copyright contributors to the vLLM project
# #
# Push a ROCm-family nightly base image and nightly image from ECR to Docker Hub # Push a ROCm stack's nightly base image and nightly image from ECR to Docker Hub
# under vllm/vllm-openai-rocm, as base-nightly[-<variant>], nightly[-<variant>] # under vllm/vllm-openai-rocm, as base-nightly-<variant>, nightly-<variant> and their
# and their -<commit> forms. # -<commit> forms, then prune old nightly tags. With --default, also push the plain
# Run when NIGHTLY=1 after the matching build-*-release-image step has pushed to ECR. # base-nightly and nightly tags for the same images.
# Run when NIGHTLY=1 after the stack's release image has been pushed to ECR.
# #
# Usage: push-nightly-builds-rocm.sh [ECR_TAG_SUFFIX TAG_VARIANT] # Usage: push-nightly-builds-rocm.sh <base-dockerfile> [--default]
# Pass both arguments or neither. With none, the default ROCm apt build is # e.g. "docker/Dockerfile.rocm_base --default" (ROCm 10.0, also :nightly)
# published as :nightly. The TheRock build passes "rock" (the source ECR tag # and "docker/Dockerfile.rocm_72_base" (ROCm 7.2, :nightly-rocm72).
# suffix) and "rocm714" (the Docker Hub tag flavor), giving :nightly-rocm714.
# #
# Local testing (no push to Docker Hub): # Local testing (no push to Docker Hub):
# BASE_ECR_IMAGE=<full-base-image-reference> \ # BUILDKITE_COMMIT=<commit-with-image-in-ecr> DRY_RUN=1 \
# BUILDKITE_COMMIT=<commit-with-image-in-ecr> DRY_RUN=1 \ # bash .buildkite/scripts/push-nightly-builds-rocm.sh docker/Dockerfile.rocm_base --default
# bash .buildkite/scripts/push-nightly-builds-rocm.sh
# Requires: AWS CLI configured (for ECR public login), Docker. For full run: Docker Hub login. # Requires: AWS CLI configured (for ECR public login), Docker. For full run: Docker Hub login.
set -euxo pipefail set -euxo pipefail
usage() { usage() {
echo "Usage: $0 [ECR_TAG_SUFFIX TAG_VARIANT]" >&2 echo "Usage: $0 <base-dockerfile> [--default]" >&2
} }
DOCKERHUB_REPO="vllm/vllm-openai-rocm" PUSH_DEFAULT=0
case "$#" in case "$#" in
0) 1) ;;
ECR_TAG_SUFFIX="rocm"
TAG_VARIANT=""
;;
2) 2)
if [[ -z "$1" || -z "$2" ]]; then if [[ "$2" != "--default" ]]; then
usage usage
exit 2 exit 2
fi fi
ECR_TAG_SUFFIX="$1" PUSH_DEFAULT=1
TAG_VARIANT="$2"
;; ;;
*) *)
usage usage
@@ -49,61 +43,44 @@ esac
BUILDKITE_COMMIT="${BUILDKITE_COMMIT:?Set BUILDKITE_COMMIT to the commit SHA that has the image in ECR (e.g. from a previous release pipeline run)}" BUILDKITE_COMMIT="${BUILDKITE_COMMIT:?Set BUILDKITE_COMMIT to the commit SHA that has the image in ECR (e.g. from a previous release pipeline run)}"
DRY_RUN="${DRY_RUN:-0}" DRY_RUN="${DRY_RUN:-0}"
ECR_REPO="public.ecr.aws/q9t5s3a7/vllm-release-repo" # shellcheck source=.buildkite/scripts/rocm/stack.sh
source "$(dirname "${BASH_SOURCE[0]}")/rocm/stack.sh" "$1"
DOCKERHUB_REPO="vllm/vllm-openai-rocm"
VARIANT="$ROCM_STACK_VARIANT"
# Get the base image ECR tag (set by the build-*-release-image pipeline step) BASE_TAGS=("base-nightly-${VARIANT}" "base-nightly-${VARIANT}-${BUILDKITE_COMMIT}")
BASE_METADATA_KEY="${ECR_TAG_SUFFIX}-base-ecr-tag" TAGS=("nightly-${VARIANT}" "nightly-${VARIANT}-${BUILDKITE_COMMIT}")
if [[ -n "${BASE_ECR_IMAGE:-}" ]]; then if [[ "$PUSH_DEFAULT" == "1" ]]; then
if [[ "$DRY_RUN" != "1" ]]; then BASE_TAGS+=("base-nightly" "base-nightly-${BUILDKITE_COMMIT}")
echo "ERROR: BASE_ECR_IMAGE may only be used with DRY_RUN=1" >&2 TAGS+=("nightly" "nightly-${BUILDKITE_COMMIT}")
exit 1
fi
BASE_ORIG_TAG="$BASE_ECR_IMAGE"
elif ! BASE_ORIG_TAG="$(buildkite-agent meta-data get "$BASE_METADATA_KEY")"; then
echo "ERROR: Failed to read required Buildkite metadata '$BASE_METADATA_KEY'" >&2
exit 1
fi
if [[ -z "$BASE_ORIG_TAG" ]]; then
echo "ERROR: Required Buildkite metadata '$BASE_METADATA_KEY' is empty" >&2
echo "Set BASE_ECR_IMAGE to the full ECR base image reference for local testing" >&2
exit 1
fi fi
ORIG_TAG="${BUILDKITE_COMMIT}-${ECR_TAG_SUFFIX}" echo "Pushing base image $ROCM_STACK_ECR_BASE as: ${BASE_TAGS[*]}"
VARIANT_SUFFIX="" echo "Pushing release image $ROCM_STACK_ECR_IMAGE as: ${TAGS[*]}"
[[ -n "$TAG_VARIANT" ]] && VARIANT_SUFFIX="-$TAG_VARIANT"
BASE_TAG_NAME="base-nightly${VARIANT_SUFFIX}"
TAG_NAME="nightly${VARIANT_SUFFIX}"
BASE_TAG_NAME_COMMIT="base-nightly${VARIANT_SUFFIX}-${BUILDKITE_COMMIT}"
TAG_NAME_COMMIT="nightly${VARIANT_SUFFIX}-${BUILDKITE_COMMIT}"
echo "Pushing base image from ECR: $BASE_ORIG_TAG" # Login to ECR and pull the stack's images
echo "Pushing release image from ECR tag: $ORIG_TAG to Docker Hub as $DOCKERHUB_REPO:$TAG_NAME and $DOCKERHUB_REPO:$TAG_NAME_COMMIT"
[[ "$DRY_RUN" == "1" ]] && echo "[DRY_RUN] Skipping push to Docker Hub"
# Login to ECR and pull the images built by build-*-release-image
aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7 aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
docker pull "$BASE_ORIG_TAG" docker pull "$ROCM_STACK_ECR_BASE"
docker pull "${ECR_REPO}:${ORIG_TAG}" docker pull "$ROCM_STACK_ECR_IMAGE"
# Tag for Docker Hub (base-nightly and base-nightly-<commit>, nightly and nightly-<commit>) for tag in "${BASE_TAGS[@]}"; do docker tag "$ROCM_STACK_ECR_BASE" "$DOCKERHUB_REPO:$tag"; done
docker tag "$BASE_ORIG_TAG" "$DOCKERHUB_REPO":"$BASE_TAG_NAME" for tag in "${TAGS[@]}"; do docker tag "$ROCM_STACK_ECR_IMAGE" "$DOCKERHUB_REPO:$tag"; done
docker tag "$BASE_ORIG_TAG" "$DOCKERHUB_REPO":"$BASE_TAG_NAME_COMMIT"
docker tag "${ECR_REPO}:${ORIG_TAG}" "$DOCKERHUB_REPO":"$TAG_NAME"
docker tag "${ECR_REPO}:${ORIG_TAG}" "$DOCKERHUB_REPO":"$TAG_NAME_COMMIT"
if [[ "$DRY_RUN" == "1" ]]; then if [[ "$DRY_RUN" == "1" ]]; then
echo "[DRY_RUN] Would push $DOCKERHUB_REPO:$BASE_TAG_NAME and $DOCKERHUB_REPO:$BASE_TAG_NAME_COMMIT"
echo "[DRY_RUN] Would push $DOCKERHUB_REPO:$TAG_NAME and $DOCKERHUB_REPO:$TAG_NAME_COMMIT"
echo "[DRY_RUN] Local tags created. Exiting without push." echo "[DRY_RUN] Local tags created. Exiting without push."
exit 0 exit 0
fi fi
# Push to Docker Hub (docker-login plugin runs before this step in CI) # Push to Docker Hub (docker-login plugin runs before this step in CI)
docker push "$DOCKERHUB_REPO":"$BASE_TAG_NAME" for tag in "${BASE_TAGS[@]}" "${TAGS[@]}"; do docker push "$DOCKERHUB_REPO:$tag"; done
docker push "$DOCKERHUB_REPO":"$BASE_TAG_NAME_COMMIT" echo "Pushed $DOCKERHUB_REPO: ${BASE_TAGS[*]} ${TAGS[*]}"
docker push "$DOCKERHUB_REPO":"$TAG_NAME"
docker push "$DOCKERHUB_REPO":"$TAG_NAME_COMMIT"
echo "Pushed $DOCKERHUB_REPO:$BASE_TAG_NAME and $DOCKERHUB_REPO:$BASE_TAG_NAME_COMMIT" # Keep only the last 14 builds of each tag family. The plain nightly- families
echo "Pushed $DOCKERHUB_REPO:$TAG_NAME and $DOCKERHUB_REPO:$TAG_NAME_COMMIT" # exclude the variant ones sharing their prefix; commit SHAs never start with "rocm".
CLEANUP=.buildkite/scripts/cleanup-nightly-builds.sh
bash "$CLEANUP" "nightly-${VARIANT}-" "$DOCKERHUB_REPO"
bash "$CLEANUP" "base-nightly-${VARIANT}-" "$DOCKERHUB_REPO"
if [[ "$PUSH_DEFAULT" == "1" ]]; then
bash "$CLEANUP" "nightly-" "$DOCKERHUB_REPO" "nightly-rocm"
bash "$CLEANUP" "base-nightly-" "$DOCKERHUB_REPO" "base-nightly-rocm"
fi
+2 -2
View File
@@ -1,8 +1,8 @@
#!/usr/bin/env bash #!/usr/bin/env bash
# Shared Dockerfile selection for the normal AMD CI build chain. # Shared Dockerfile selection for the normal AMD CI build chain.
# CI_ROCM_DOCKERFILE_BASE and CI_ROCM_DOCKERFILE accept repository-relative paths. # CI_ROCM_DOCKERFILE_BASE and CI_ROCM_DOCKERFILE accept repository-relative paths.
# For Rock, set CI_ROCM_DOCKERFILE_BASE=docker/Dockerfile.rock_base and # For the legacy ROCm 7.2 stack, set CI_ROCM_DOCKERFILE_BASE=docker/Dockerfile.rocm_72_base
# CI_ROCM_DOCKERFILE=docker/Dockerfile.rock. # and CI_ROCM_DOCKERFILE=docker/Dockerfile.rocm_72.
# Custom recipes must preserve the CI targets and artifact layout. Additional # Custom recipes must preserve the CI targets and artifact layout. Additional
# COPY inputs also need the corresponding *_CONTENT_FILES overrides. # COPY inputs also need the corresponding *_CONTENT_FILES overrides.
+78
View File
@@ -0,0 +1,78 @@
#!/usr/bin/env bash
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
#
# Derive everything the release pipeline needs about a ROCm stack from its base
# Dockerfile, so the stack's image is the single input to every ROCm script.
#
# Usage: source .buildkite/scripts/rocm/stack.sh <base-dockerfile>
# e.g. docker/Dockerfile.rocm_base (ROCm 10.0) or docker/Dockerfile.rocm_72_base
#
# Exports:
# ROCM_STACK_BASE_DOCKERFILE the base Dockerfile
# ROCM_STACK_DOCKERFILE the matching vLLM Dockerfile (base name minus _base)
# ROCM_STACK_THEROCK 1 if the ROCm SDK comes from TheRock wheels
# ROCM_STACK_VERSION ROCm version, e.g. 10.0.0 or 7.2.3
# ROCM_STACK_VARIANT wheel variant and Docker Hub tag flavor from major.minor,
# same rule as setup.py (rocm100, rocm72)
# ROCM_STACK_IMAGE_KEY hash of the base Dockerfile (base image cache key)
# ROCM_STACK_WHEEL_KEY hash of the base Dockerfile plus files it COPYs from
# the repo (base wheel cache key)
# ROCM_STACK_ECR_BASE ECR tag of the cached base image
# ROCM_STACK_ECR_IMAGE ECR tag of this commit's release image
# ROCM_STACK_BASE_WHEELS_DIR local dir for base wheels (also the artifact path)
# ROCM_STACK_VLLM_WHEEL_DIR local dir for the vLLM wheel (also the artifact path)
rocm_stack_init() {
local base="$1"
if [[ ! -f "$base" ]]; then
echo "ROCm base Dockerfile not found: $base" >&2
return 2
fi
ROCM_STACK_BASE_DOCKERFILE="$base"
ROCM_STACK_DOCKERFILE="${base%_base}"
local sdk
sdk=$(sed -nE 's/^ARG ROCM_SDK_VERSION=([0-9.]+).*/\1/p' "$base" | head -1)
if [[ -n "$sdk" ]]; then
ROCM_STACK_THEROCK=1
ROCM_STACK_VERSION="$sdk"
else
ROCM_STACK_THEROCK=0
# BASE_IMAGE format: rocm/dev-ubuntu-22.04:7.2.3-complete -> 7.2.3
ROCM_STACK_VERSION=$(sed -nE 's/^ARG BASE_IMAGE=.*:([0-9]+(\.[0-9]+)+).*/\1/p' "$base" | head -1)
fi
if [[ -z "$ROCM_STACK_VERSION" ]]; then
echo "Could not determine the ROCm version from $base" >&2
return 2
fi
local major minor
IFS=. read -r major minor _ <<< "$ROCM_STACK_VERSION"
ROCM_STACK_VARIANT="rocm${major}${minor}"
ROCM_STACK_IMAGE_KEY=$(sha256sum "$base" | cut -c1-16)
local inputs
inputs=$(grep -E '^COPY ' "$base" | grep -v -- '--from' | awk '{print $2}' || true)
if [[ -z "$inputs" ]]; then
ROCM_STACK_WHEEL_KEY="$ROCM_STACK_IMAGE_KEY"
else
# shellcheck disable=SC2086
ROCM_STACK_WHEEL_KEY=$(sha256sum "$base" $inputs | sha256sum | cut -c1-16)
fi
local ecr="public.ecr.aws/q9t5s3a7/vllm-release-repo"
ROCM_STACK_ECR_BASE="${ecr}:${ROCM_STACK_IMAGE_KEY}-${ROCM_STACK_VARIANT}-base"
ROCM_STACK_ECR_IMAGE="${ecr}:${BUILDKITE_COMMIT:-unknown}-${ROCM_STACK_VARIANT}"
ROCM_STACK_BASE_WHEELS_DIR="artifacts/${ROCM_STACK_VARIANT}-base-wheels"
ROCM_STACK_VLLM_WHEEL_DIR="artifacts/${ROCM_STACK_VARIANT}-vllm-wheel"
export ROCM_STACK_BASE_DOCKERFILE ROCM_STACK_DOCKERFILE ROCM_STACK_THEROCK \
ROCM_STACK_VERSION ROCM_STACK_VARIANT ROCM_STACK_IMAGE_KEY \
ROCM_STACK_WHEEL_KEY ROCM_STACK_ECR_BASE ROCM_STACK_ECR_IMAGE \
ROCM_STACK_BASE_WHEELS_DIR ROCM_STACK_VLLM_WHEEL_DIR
}
if [[ $# -gt 0 ]]; then
rocm_stack_init "$1"
fi
+66 -20
View File
@@ -4,21 +4,31 @@
# #
# Upload ROCm wheels to S3 with proper index generation # Upload ROCm wheels to S3 with proper index generation
# #
# Usage: upload-rocm-wheels.sh <base-dockerfile>
# base-dockerfile the ROCm stack, e.g. docker/Dockerfile.rocm_base (see rocm/stack.sh)
#
# Required environment variables: # Required environment variables:
# AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY (or IAM role) # AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY (or IAM role)
# S3_BUCKET (default: vllm-wheels) # S3_BUCKET (default: vllm-wheels)
# #
# S3 path structure: # S3 path structure:
# s3://vllm-wheels/rocm/{commit}/ - All wheels for this commit # s3://vllm-wheels/rocm/{commit}/<variant>-wheels/ - This stack's wheels for this commit
# s3://vllm-wheels/rocm/nightly/ - Index pointing to latest nightly # s3://vllm-wheels/rocm/{commit}/<variant>/ - Index per ROCm variant (e.g. rocm100)
# s3://vllm-wheels/rocm/{version}/ - Index for release versions # s3://vllm-wheels/rocm/nightly/<variant>/ - Index pointing to latest nightly
# s3://vllm-wheels/rocm/{version}/<variant>/ - Index for release versions
# The top-level index.html at each prefix lists every variant present, since
# several ROCm stacks publish into the same prefixes.
set -ex set -ex
# ======== Configuration ======== # ======== Configuration ========
BUCKET="${S3_BUCKET:-vllm-wheels}" BUCKET="${S3_BUCKET:-vllm-wheels}"
# shellcheck source=.buildkite/scripts/rocm/stack.sh
source "$(dirname "${BASH_SOURCE[0]}")/rocm/stack.sh" "${1:?Usage: $0 <base-dockerfile>}"
ROCM_SUBPATH="rocm/${BUILDKITE_COMMIT}" ROCM_SUBPATH="rocm/${BUILDKITE_COMMIT}"
S3_COMMIT_PREFIX="s3://$BUCKET/$ROCM_SUBPATH/" WHEEL_DIRS="$ROCM_STACK_BASE_WHEELS_DIR $ROCM_STACK_VLLM_WHEEL_DIR"
WHEEL_SUBPATH="$ROCM_SUBPATH/${ROCM_STACK_VARIANT}-wheels"
S3_WHEEL_PREFIX="s3://$BUCKET/$WHEEL_SUBPATH/"
INDICES_OUTPUT_DIR="rocm-indices" INDICES_OUTPUT_DIR="rocm-indices"
echo "========================================" echo "========================================"
@@ -49,9 +59,11 @@ source .buildkite/scripts/lib/manylinux.sh
# ======== Part 1: Collect and prepare wheels ======== # ======== Part 1: Collect and prepare wheels ========
# Collect all wheels # Collect all wheels
rm -rf all-rocm-wheels "$INDICES_OUTPUT_DIR"
mkdir -p all-rocm-wheels mkdir -p all-rocm-wheels
cp artifacts/rocm-base-wheels/*.whl all-rocm-wheels/ 2>/dev/null || true for dir in $WHEEL_DIRS; do
cp artifacts/rocm-vllm-wheel/*.whl all-rocm-wheels/ 2>/dev/null || true cp "$dir"/*.whl all-rocm-wheels/ 2>/dev/null || true
done
WHEEL_COUNT=$(find all-rocm-wheels -maxdepth 1 -name '*.whl' 2>/dev/null | wc -l) WHEEL_COUNT=$(find all-rocm-wheels -maxdepth 1 -name '*.whl' 2>/dev/null | wc -l)
echo "Total wheels to upload: $WHEEL_COUNT" echo "Total wheels to upload: $WHEEL_COUNT"
@@ -84,9 +96,9 @@ ls -lh all-rocm-wheels/
# ======== Part 2: Upload wheels to S3 ======== # ======== Part 2: Upload wheels to S3 ========
echo "" echo ""
echo "Uploading wheels to $S3_COMMIT_PREFIX" echo "Uploading wheels to $S3_WHEEL_PREFIX"
for wheel in all-rocm-wheels/*.whl; do for wheel in all-rocm-wheels/*.whl; do
aws s3 cp "$wheel" "$S3_COMMIT_PREFIX" aws s3 cp "$wheel" "$S3_WHEEL_PREFIX"
done done
# ======== Part 3: Generate and upload indices ======== # ======== Part 3: Generate and upload indices ========
@@ -95,28 +107,63 @@ done
echo "" echo ""
echo "Generating indices..." echo "Generating indices..."
obj_json="rocm-objects.json" obj_json="rocm-objects.json"
aws s3api list-objects-v2 --bucket "$BUCKET" --prefix "$ROCM_SUBPATH/" --delimiter / --output json > "$obj_json" aws s3api list-objects-v2 --bucket "$BUCKET" --prefix "$WHEEL_SUBPATH/" --delimiter / --output json > "$obj_json"
mkdir -p "$INDICES_OUTPUT_DIR" mkdir -p "$INDICES_OUTPUT_DIR"
# Use the existing generate-nightly-index.py # Use the existing generate-nightly-index.py
# HACK: Replace regex module with stdlib re (same as CUDA script) # HACK: Replace regex module with stdlib re (same as CUDA script)
sed -i 's/import regex as re/import re/g' .buildkite/scripts/generate-nightly-index.py sed -i 's/import regex as re/import re/g' .buildkite/scripts/generate-nightly-index.py
sed -i 's/import regex as re/import re/g' tools/vllm-rocm/therock_wheels.py
sed -i 's/import pybase64 as base64/import base64/g' tools/vllm-rocm/therock_wheels.py
INDEX_ARGS=(--version "$ROCM_SUBPATH" --wheel-dir "${WHEEL_SUBPATH#rocm/}")
# TheRock's SDK, device kernels and torchvision/torchaudio are linked, not hosted
if [[ "$ROCM_STACK_THEROCK" == "1" ]]; then
$PYTHON tools/vllm-rocm/therock_wheels.py external-links \
--dockerfile "$ROCM_STACK_BASE_DOCKERFILE" \
--torch-wheel "$(ls "$ROCM_STACK_BASE_WHEELS_DIR"/torch-*.whl)" > therock-external-links.txt
cat therock-external-links.txt
INDEX_ARGS+=(--external-links therock-external-links.txt)
fi
$PYTHON .buildkite/scripts/generate-nightly-index.py \ $PYTHON .buildkite/scripts/generate-nightly-index.py \
--version "$ROCM_SUBPATH" \ "${INDEX_ARGS[@]}" \
--current-objects "$obj_json" \ --current-objects "$obj_json" \
--output-dir "$INDICES_OUTPUT_DIR" \ --output-dir "$INDICES_OUTPUT_DIR" \
--comment "ROCm commit $BUILDKITE_COMMIT" --comment "ROCm commit $BUILDKITE_COMMIT"
# Upload indices to commit directory # Upload this build's variant indices under an S3 prefix (e.g. rocm/nightly), then
echo "Uploading indices to $S3_COMMIT_PREFIX" # rebuild that prefix's top-level project list from every variant now present.
aws s3 cp --recursive "$INDICES_OUTPUT_DIR/" "$S3_COMMIT_PREFIX" publish_indices() {
local prefix="$1"
local listing variants merged
echo "Uploading indices to s3://$BUCKET/$prefix/"
aws s3 cp --recursive "$INDICES_OUTPUT_DIR/" "s3://$BUCKET/$prefix/" --exclude "index.html"
listing=$(aws s3api list-objects-v2 --bucket "$BUCKET" --prefix "$prefix/" \
--delimiter / --query 'CommonPrefixes[].Prefix' --output text)
variants=$(tr '\t' '\n' <<< "$listing" | sed -n "s|^$prefix/\(rocm[0-9]*\)/$|\1|p" | sort -u)
merged=$(mktemp)
{
echo '<!DOCTYPE html>'
echo '<html>'
echo ' <meta name="pypi:repository-version" content="1.0">'
echo ' <body>'
for variant in $variants; do
echo " <a href=\"$variant/\">$variant/</a><br/>"
done
echo ' </body>'
echo '</html>'
} > "$merged"
echo "Variants under $prefix/: $(echo "$variants" | tr '\n' ' ')"
aws s3 cp "$merged" "s3://$BUCKET/$prefix/index.html"
rm -f "$merged"
}
publish_indices "$ROCM_SUBPATH"
# Only scheduled nightly builds should update the moving nightly index. # Only scheduled nightly builds should update the moving nightly index.
if [[ "${NIGHTLY:-0}" == "1" ]]; then if [[ "${NIGHTLY:-0}" == "1" ]]; then
echo "Updating rocm/nightly/ index..." publish_indices "rocm/nightly"
aws s3 cp --recursive "$INDICES_OUTPUT_DIR/" "s3://$BUCKET/rocm/nightly/"
fi fi
# Extract version from vLLM wheel and update version-specific index # Extract version from vLLM wheel and update version-specific index
@@ -129,8 +176,7 @@ if [ -n "$VLLM_WHEEL" ]; then
echo "Pure version: $PURE_VERSION" echo "Pure version: $PURE_VERSION"
if [[ "$VERSION" != *"dev"* ]]; then if [[ "$VERSION" != *"dev"* ]]; then
echo "Updating rocm/$PURE_VERSION/ index..." publish_indices "rocm/$PURE_VERSION"
aws s3 cp --recursive "$INDICES_OUTPUT_DIR/" "s3://$BUCKET/rocm/$PURE_VERSION/"
fi fi
fi fi
@@ -142,14 +188,14 @@ echo "ROCm Wheel Upload Complete!"
echo "========================================" echo "========================================"
echo "" echo ""
echo "Wheels available at:" echo "Wheels available at:"
echo " s3://$BUCKET/$ROCM_SUBPATH/" echo " s3://$BUCKET/$WHEEL_SUBPATH/"
echo "" echo ""
echo "Install command (by commit):" echo "Install command (by commit):"
echo " pip install vllm --extra-index-url https://${BUCKET}.s3.amazonaws.com/$ROCM_SUBPATH/" echo " pip install vllm --extra-index-url https://${BUCKET}.s3.amazonaws.com/$ROCM_SUBPATH/$ROCM_STACK_VARIANT/"
echo "" echo ""
if [[ "${NIGHTLY:-0}" == "1" ]]; then if [[ "${NIGHTLY:-0}" == "1" ]]; then
echo "Install command (nightly):" echo "Install command (nightly):"
echo " pip install vllm --extra-index-url https://${BUCKET}.s3.amazonaws.com/rocm/nightly/" echo " pip install vllm --extra-index-url https://${BUCKET}.s3.amazonaws.com/rocm/nightly/$ROCM_STACK_VARIANT/"
fi fi
echo "" echo ""
echo "Wheel count: $WHEEL_COUNT" echo "Wheel count: $WHEEL_COUNT"
+212 -300
View File
@@ -399,27 +399,6 @@ steps:
commands: commands:
- pytest -v -s v1/e2e/spec_decode/draft_model/ - pytest -v -s v1/e2e/spec_decode/draft_model/
- label: ":amd: (MI250) Spec Decode Speculators + MTP" # TBD
timeout_in_minutes: 75
mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]
dind: false
agent_pool: mi250_1
optional: true
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/
- vllm/model_executor/model_loader/
- vllm/v1/sample/
- vllm/model_executor/layers/
- vllm/transformers_utils/configs/speculators/
- tests/v1/e2e/spec_decode/
- vllm/platforms/rocm.py
- vllm/v1/attention/backends/
commands:
- pytest -v -s v1/e2e/spec_decode/speculators/
- pytest -v -s v1/e2e/spec_decode/mtp/
#------------------------------------------------------------- mi250 · misc ------------------------------------------------------------# #------------------------------------------------------------- mi250 · misc ------------------------------------------------------------#
- label: ":amd: (MI300) Python-only Installation" # TBD - label: ":amd: (MI300) Python-only Installation" # TBD
@@ -1094,25 +1073,6 @@ steps:
#----------------------------------------------------------- mi300 · evals -----------------------------------------------------------# #----------------------------------------------------------- mi300 · evals -----------------------------------------------------------#
- label: ":amd: (MI300) LM Eval Small Models" # TBD
timeout_in_minutes: 45
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
dind: false
agent_pool: mi300_1
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- csrc/
- vllm/model_executor/layers/quantization
- vllm/model_executor/models/
- vllm/model_executor/model_loader/
- vllm/v1/attention/backends/
- vllm/v1/attention/selector.py
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
optional: true
commands:
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-small.txt
- label: ":amd: (MI300) DeepSeek V2-Lite Prefetch Offload Accuracy" # TBD - label: ":amd: (MI300) DeepSeek V2-Lite Prefetch Offload Accuracy" # TBD
timeout_in_minutes: 30 timeout_in_minutes: 30
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300] mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
@@ -1153,23 +1113,6 @@ steps:
commands: commands:
- pytest -s -v evals/mrcr/test_mrcr_correctness.py --config-list-file=evals/mrcr/configs/models-small.txt - pytest -s -v evals/mrcr/test_mrcr_correctness.py --config-list-file=evals/mrcr/configs/models-small.txt
- label: ":amd: (MI300) LM Eval KV-Offload" # TBD
timeout_in_minutes: 55
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
dind: false
agent_pool: mi300_1
optional: true
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/distributed/kv_transfer/kv_connector/v1/offloading/
- vllm/distributed/kv_transfer/kv_connector/v1/simple_cpu_offload_connector.py
- vllm/v1/kv_offload/
- vllm/v1/simple_kv_offload/
- tests/evals/gsm8k/test_gsm8k_offloading.py
- vllm/platforms/rocm.py
commands:
- pytest -s -v evals/gsm8k/test_gsm8k_offloading.py -k "nemotron-h-8b or gemma-4-e4b-it"
- label: ":amd: (MI300) LM Eval KV-Offload Medium" # TBD - label: ":amd: (MI300) LM Eval KV-Offload Medium" # TBD
timeout_in_minutes: 70 timeout_in_minutes: 70
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300] mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
@@ -1188,24 +1131,6 @@ steps:
commands: commands:
- pytest -s -v evals/gsm8k/test_gsm8k_offloading.py -k "qwen3.5-35b or deepseek-v2-lite" - pytest -s -v evals/gsm8k/test_gsm8k_offloading.py -k "qwen3.5-35b or deepseek-v2-lite"
- label: ":amd: (MI300) LM Eval KV-Offload Large" # TBD
timeout_in_minutes: 65
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
dind: false
agent_pool: mi300_4
num_gpus: 4
optional: true
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/distributed/kv_transfer/kv_connector/v1/offloading/
- vllm/distributed/kv_transfer/kv_connector/v1/simple_cpu_offload_connector.py
- vllm/v1/kv_offload/
- vllm/v1/simple_kv_offload/
- tests/evals/gsm8k/test_gsm8k_offloading.py
- vllm/platforms/rocm.py
commands:
- VLLM_ROCM_USE_AITER=1 pytest -s -v evals/gsm8k/test_gsm8k_offloading.py -k "deepseek-v4-flash"
- label: ":amd: (MI300) LM Eval Small Models Harness" # TBD - label: ":amd: (MI300) LM Eval Small Models Harness" # TBD
timeout_in_minutes: 35 timeout_in_minutes: 35
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300] mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
@@ -1383,52 +1308,6 @@ steps:
commands: commands:
- VLLM_ENGINE_READY_TIMEOUT_S=1800 bash .buildkite/scripts/scheduled_integration_test/qwen3_next_mtp_async_eplb.sh 0.8 1319 8040 - VLLM_ENGINE_READY_TIMEOUT_S=1800 bash .buildkite/scripts/scheduled_integration_test/qwen3_next_mtp_async_eplb.sh 0.8 1319 8040
- label: ":amd: (MI300) LM Eval Large Models" # TBD
timeout_in_minutes: 65
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
dind: false
agent_pool: mi300_8
optional: true
num_gpus: 8
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/model_executor/models/
- vllm/model_executor/model_loader/
- vllm/model_executor/layers/quantization/
- vllm/v1/attention/backends/
- vllm/v1/attention/selector.py
- vllm/model_executor/layers/layernorm.py
- csrc/
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
- tests/evals/
commands:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- export PYTORCH_ROCM_ARCH=gfx942
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi3xx.txt
- label: ":amd: (MI300) LM Eval Large Models ROCm Harness" # TBD
timeout_in_minutes: 40
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
dind: false
agent_pool: mi300_8
optional: true
num_gpus: 8
working_dir: "/vllm-workspace/.buildkite/lm-eval-harness"
source_file_dependencies:
- vllm/model_executor/models/
- vllm/model_executor/model_loader/
- vllm/model_executor/layers/quantization/
- vllm/v1/attention/backends/
- vllm/v1/attention/selector.py
- vllm/model_executor/layers/layernorm.py
- csrc/
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
commands:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pytest -s -v test_lm_eval_correctness.py --config-list-file=configs/models-large-rocm.txt --tp-size=8 --timeout=5400
- label: ":amd: (MI300) LM Eval Large Models FP8" # TBD - label: ":amd: (MI300) LM Eval Large Models FP8" # TBD
timeout_in_minutes: 40 timeout_in_minutes: 40
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300] mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
@@ -1939,20 +1818,6 @@ steps:
commands: commands:
- pytest -v -s models/language/pooling_mteb_test - pytest -v -s models/language/pooling_mteb_test
- label: ":amd: (MI300) Language Models (Extended Pooling) Shard %N"
timeout_in_minutes: 95
parallelism: 4
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
dind: false
agent_pool: mi300_1
optional: true
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/
- tests/models/language/pooling
commands:
- pytest -v -s models/language/pooling -m 'not core_model' --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB
- label: ":amd: (MI300) Language Models (Extended Generation)" # TBD - label: ":amd: (MI300) Language Models (Extended Generation)" # TBD
timeout_in_minutes: 105 timeout_in_minutes: 105
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300] mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
@@ -1970,21 +1835,6 @@ steps:
#---------------------------------------------------- mi300 · models / multimodal ----------------------------------------------------# #---------------------------------------------------- mi300 · models / multimodal ----------------------------------------------------#
- label: ":amd: (MI300) Multimodal Models (Extended Generation 1)" # TBD
timeout_in_minutes: 100
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
dind: false
agent_pool: mi300_1
optional: true
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/
- tests/models/multimodal/generation
- tests/models/multimodal/test_mapping.py
commands:
- pytest -v -s models/multimodal/generation -m 'not core_model' --ignore models/multimodal/generation/test_common.py
- pytest -v -s models/multimodal/test_mapping.py
- label: ":amd: (MI300) Multimodal Models (Extended Generation 2) Shard %N" - label: ":amd: (MI300) Multimodal Models (Extended Generation 2) Shard %N"
timeout_in_minutes: 100 timeout_in_minutes: 100
parallelism: 4 parallelism: 4
@@ -2013,19 +1863,6 @@ steps:
commands: commands:
- pytest -v -s models/multimodal/generation/test_common.py -m 'split(group=1) and not core_model' - pytest -v -s models/multimodal/generation/test_common.py -m 'split(group=1) and not core_model'
- label: ":amd: (MI300) Multimodal Models (PPL)"
timeout_in_minutes: 90
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
dind: false
agent_pool: mi300_1
optional: true
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/
- tests/models/multimodal/generation_ppl_test/
commands:
- pytest -v -s models/multimodal/generation_ppl_test/
- label: ":amd: (MI300) Multimodal Models (Standard) 3: llava + qwen2_vl" # TBD - label: ":amd: (MI300) Multimodal Models (Standard) 3: llava + qwen2_vl" # TBD
timeout_in_minutes: 55 timeout_in_minutes: 55
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300] mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
@@ -2057,19 +1894,6 @@ steps:
commands: commands:
- pytest -v -s models/multimodal/processing/test_tensor_schema.py --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB - pytest -v -s models/multimodal/processing/test_tensor_schema.py --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB
- label: ":amd: (MI300) Multimodal Models (Extended Pooling)" # TBD
timeout_in_minutes: 75
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
dind: false
agent_pool: mi300_1
optional: true
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/
- tests/models/multimodal/pooling
commands:
- pytest -v -s models/multimodal/pooling -m 'not core_model'
#----------------------------------------------------- mi300 · models / quantized -----------------------------------------------------# #----------------------------------------------------- mi300 · models / quantized -----------------------------------------------------#
- label: ":amd: (MI300) Quantized Models" # TBD - label: ":amd: (MI300) Quantized Models" # TBD
@@ -2533,27 +2357,6 @@ steps:
- VLLM_TP_SIZE=4 VLLM_NEEDLE_TRIALS=2 pytest -v -s v1/determinism/test_batch_invariance.py::test_v1_generation_is_deterministic_across_batch_sizes_with_needle -k TRITON_ATTN - VLLM_TP_SIZE=4 VLLM_NEEDLE_TRIALS=2 pytest -v -s v1/determinism/test_batch_invariance.py::test_v1_generation_is_deterministic_across_batch_sizes_with_needle -k TRITON_ATTN
- VLLM_TEST_MODEL=allenai/OLMoE-1B-7B-0924 VLLM_TEST_SERVER_ARGS="--data-parallel-size 4 --enable-expert-parallel --max-model-len 4096" pytest -v -s v1/determinism/test_online_batch_invariance.py - VLLM_TEST_MODEL=allenai/OLMoE-1B-7B-0924 VLLM_TEST_SERVER_ARGS="--data-parallel-size 4 --enable-expert-parallel --max-model-len 4096" pytest -v -s v1/determinism/test_online_batch_invariance.py
- label: ":amd: (MI300) Spec Decode Speculators + MTP" # TBD
timeout_in_minutes: 75
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
dind: false
agent_pool: mi300_1
optional: true
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/
- vllm/model_executor/model_loader/
- vllm/v1/sample/
- vllm/model_executor/layers/
- vllm/transformers_utils/configs/speculators/
- tests/v1/e2e/spec_decode/
- vllm/platforms/rocm.py
- vllm/v1/attention/backends/
commands:
- pytest -v -s v1/e2e/spec_decode/speculators/
- pytest -v -s v1/e2e/spec_decode/mtp/
- label: ":amd: (MI300) Spec Decode DeepSeek MTP Parallel Load" - label: ":amd: (MI300) Spec Decode DeepSeek MTP Parallel Load"
timeout_in_minutes: 45 timeout_in_minutes: 45
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300] mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
@@ -2581,60 +2384,6 @@ steps:
commands: commands:
- pytest -v -s v1/e2e/spec_decode/test_mtp_parallel_load.py - pytest -v -s v1/e2e/spec_decode/test_mtp_parallel_load.py
- label: ":amd: (MI300) Spec Decode AL DFlash Nightly"
timeout_in_minutes: 120
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
dind: false
agent_pool: mi300_1
optional: true
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/
- vllm/model_executor/models/qwen3_dflash.py
- vllm/model_executor/models/qwen3_dflash2.py
- vllm/model_executor/models/qwen3_5.py
- vllm/model_executor/models/laguna_dflash.py
- tests/v1/e2e/spec_decode/
- vllm/v1/attention/
- vllm/v1/sample/
- vllm/model_executor/model_loader/
- vllm/model_executor/layers/
- vllm/model_executor/models/qwen2.py
- vllm/model_executor/models/qwen3.py
- vllm/model_executor/models/laguna.py
- vllm/platforms/rocm.py
commands:
- pytest -v -s v1/e2e/spec_decode/acceptance_rates/dflash/ -k "not dflash2"
- label: ":amd: (MI300) Spec Decode AL DSpark Nightly"
timeout_in_minutes: 90
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
dind: false
agent_pool: mi300_1
optional: true
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/
- vllm/model_executor/models/qwen3_dspark.py
- vllm/model_executor/models/gemma4_dspark.py
- tests/v1/e2e/spec_decode/
- vllm/v1/attention/
- vllm/v1/sample/
- vllm/model_executor/model_loader/
- vllm/model_executor/layers/
- vllm/model_executor/models/qwen2.py
- vllm/model_executor/models/qwen3.py
- vllm/model_executor/models/qwen3_dflash.py
- vllm/model_executor/models/gemma4.py
- vllm/model_executor/models/gemma4_mm.py
- vllm/model_executor/models/gemma4_unified.py
- vllm/model_executor/models/gemma4_mtp.py
- vllm/platforms/rocm.py
commands:
- pytest -v -s v1/e2e/spec_decode/acceptance_rates/dspark/
- label: ":amd: (MI300) V1 Spec Decode" # TBD - label: ":amd: (MI300) V1 Spec Decode" # TBD
timeout_in_minutes: 50 timeout_in_minutes: 50
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300] mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
@@ -2719,39 +2468,6 @@ steps:
- pytest -v -s v1/engine --ignore v1/engine/test_preprocess_error_handling.py - pytest -v -s v1/engine --ignore v1/engine/test_preprocess_error_handling.py
- pytest -v -s v1/test_tensor_ipc_queue.py - pytest -v -s v1/test_tensor_ipc_queue.py
- label: ":amd: (MI300) Speculators Correctness Nightly"
timeout_in_minutes: 40
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
dind: false
agent_pool: mi300_1
optional: true
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/config/speculative.py
- vllm/engine/arg_utils.py
- vllm/transformers_utils/config.py
- vllm/transformers_utils/configs/speculators/
- vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/
- vllm/v1/worker/gpu_model_runner.py
- vllm/v1/sample/
- vllm/v1/attention/backends/
- vllm/v1/attention/selector.py
- vllm/model_executor/model_loader/
- vllm/model_executor/layers/
- vllm/model_executor/models/llama_eagle3.py
- vllm/model_executor/models/qwen3.py
- vllm/model_executor/models/qwen3_dflash.py
- vllm/model_executor/models/registry.py
- vllm/_aiter_ops.py
- tests/evals/gsm8k/
- tests/v1/spec_decode/test_speculators_correctness.py
- vllm/platforms/rocm.py
- vllm/v1/worker/gpu/spec_decode/dflash/
commands:
- export VLLM_ALLOW_INSECURE_SERIALIZATION=1
- pytest -v -s v1/spec_decode/test_speculators_correctness.py -m slow_test
- label: ":amd: (MI300) Extract Hidden States Integration" # TBD - label: ":amd: (MI300) Extract Hidden States Integration" # TBD
timeout_in_minutes: 30 timeout_in_minutes: 30
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300] mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
@@ -3106,22 +2822,6 @@ steps:
- uv pip install --system -r /vllm-workspace/requirements/kv_connectors_rocm.txt - uv pip install --system -r /vllm-workspace/requirements/kv_connectors_rocm.txt
- ATTENTION_BACKEND=TRITON_ATTN bash v1/kv_connector/nixl_integration/run_multi_connector_edge_case_test.sh - ATTENTION_BACKEND=TRITON_ATTN bash v1/kv_connector/nixl_integration/run_multi_connector_edge_case_test.sh
- label: ":amd: (MI300) V1 E2E Hybrid Chunked Prefill" # TBD
timeout_in_minutes: 35
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
dind: false
agent_pool: mi300_4
num_gpus: 4
optional: true
source_file_dependencies:
- vllm/v1/attention/backends/utils.py
- vllm/v1/worker/gpu_model_runner.py
- tests/v1/e2e/test_hybrid_chunked_prefill.py
- vllm/v1/core/
working_dir: /vllm-workspace/tests
commands:
- pytest -v -s v1/e2e/test_hybrid_chunked_prefill.py
######################################################################################################################################### #########################################################################################################################################
# # # #
# MI355 (gfx950) tests # # MI355 (gfx950) tests #
@@ -4604,6 +4304,82 @@ steps:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn - export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi355-nemotron-3-super.txt - pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi355-nemotron-3-super.txt
- label: ":amd: (MI355) LM Eval Small Models" # TBD
timeout_in_minutes: 45
mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]
dind: false
agent_pool: mi355_dpx
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- csrc/
- vllm/model_executor/layers/quantization
- vllm/model_executor/models/
- vllm/model_executor/model_loader/
- vllm/v1/attention/backends/
- vllm/v1/attention/selector.py
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
optional: true
commands:
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-small.txt
- label: ":amd: (MI355 DPX) LM Eval KV-Offload" # TBD
timeout_in_minutes: 55
mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]
dind: false
agent_pool: mi355_dpx
optional: true
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/distributed/kv_transfer/kv_connector/v1/offloading/
- vllm/distributed/kv_transfer/kv_connector/v1/simple_cpu_offload_connector.py
- vllm/v1/kv_offload/
- vllm/v1/simple_kv_offload/
- tests/evals/gsm8k/test_gsm8k_offloading.py
- vllm/platforms/rocm.py
commands:
- pytest -s -v evals/gsm8k/test_gsm8k_offloading.py -k "nemotron-h-8b or gemma-4-e4b-it"
- label: ":amd: (MI355) LM Eval KV-Offload Large" # TBD
timeout_in_minutes: 65
mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]
dind: false
agent_pool: mi355_4
num_gpus: 4
optional: true
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/distributed/kv_transfer/kv_connector/v1/offloading/
- vllm/distributed/kv_transfer/kv_connector/v1/simple_cpu_offload_connector.py
- vllm/v1/kv_offload/
- vllm/v1/simple_kv_offload/
- tests/evals/gsm8k/test_gsm8k_offloading.py
- vllm/platforms/rocm.py
commands:
- VLLM_ROCM_USE_AITER=1 pytest -s -v evals/gsm8k/test_gsm8k_offloading.py -k "deepseek-v4-flash"
- label: ":amd: (MI355) LM Eval Large Models ROCm Harness" # TBD
timeout_in_minutes: 40
mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]
dind: false
agent_pool: mi355_8
optional: true
num_gpus: 8
working_dir: "/vllm-workspace/.buildkite/lm-eval-harness"
source_file_dependencies:
- vllm/model_executor/models/
- vllm/model_executor/model_loader/
- vllm/model_executor/layers/quantization/
- vllm/v1/attention/backends/
- vllm/v1/attention/selector.py
- vllm/model_executor/layers/layernorm.py
- csrc/
- vllm/_aiter_ops.py
- vllm/platforms/rocm.py
commands:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pytest -s -v test_lm_eval_correctness.py --config-list-file=configs/models-large-rocm.txt --tp-size=8 --timeout=5400
#--------------------------------------------------------- mi355 · examples ----------------------------------------------------------# #--------------------------------------------------------- mi355 · examples ----------------------------------------------------------#
- label: ":amd: (MI355 DPX) Examples" # TBD - label: ":amd: (MI355 DPX) Examples" # TBD
@@ -5421,6 +5197,19 @@ steps:
- pytest models/multimodal/generation/test_memory_leak.py -m core_model - pytest models/multimodal/generation/test_memory_leak.py -m core_model
- cd .. && VLLM_WORKER_MULTIPROC_METHOD=spawn pytest -v -s tests/models/multimodal/generation/test_whisper.py -m core_model - cd .. && VLLM_WORKER_MULTIPROC_METHOD=spawn pytest -v -s tests/models/multimodal/generation/test_whisper.py -m core_model
- label: ":amd: (MI355 DPX) Multimodal Models (PPL)"
timeout_in_minutes: 90
mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]
dind: false
agent_pool: mi355_dpx
optional: true
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/
- tests/models/multimodal/generation_ppl_test/
commands:
- pytest -v -s models/multimodal/generation_ppl_test/
#----------------------------------------------------- mi355 · models / quantized -----------------------------------------------------# #----------------------------------------------------- mi355 · models / quantized -----------------------------------------------------#
- label: ":amd: (MI355 DPX) Quantized Models" # TBD - label: ":amd: (MI355 DPX) Quantized Models" # TBD
@@ -5565,6 +5354,27 @@ steps:
commands: commands:
- pytest -v -s v1/e2e/spec_decode/draft_model/ - pytest -v -s v1/e2e/spec_decode/draft_model/
- label: ":amd: (MI355 DPX) Spec Decode Speculators + MTP" # TBD
timeout_in_minutes: 75
mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]
dind: false
agent_pool: mi355_dpx
optional: true
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/
- vllm/model_executor/model_loader/
- vllm/v1/sample/
- vllm/model_executor/layers/
- vllm/transformers_utils/configs/speculators/
- tests/v1/e2e/spec_decode/
- vllm/platforms/rocm.py
- vllm/v1/attention/backends/
commands:
- pytest -v -s v1/e2e/spec_decode/speculators/
- pytest -v -s v1/e2e/spec_decode/mtp/
- label: ":amd: (MI355 DPX) Spec Decode Eagle 1: DeepSeek + Qwen" - label: ":amd: (MI355 DPX) Spec Decode Eagle 1: DeepSeek + Qwen"
timeout_in_minutes: 60 timeout_in_minutes: 60
mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355] mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]
@@ -6041,6 +5851,108 @@ steps:
- if [ "$$BUILDKITE_PARALLEL_JOB" = "1" ]; then pytest -v -s "v1/e2e/spec_decode/acceptance_rates/mtp_other/test_mtp.py::test_gemma4_mtp_acceptance_lengths[False]"; fi - if [ "$$BUILDKITE_PARALLEL_JOB" = "1" ]; then pytest -v -s "v1/e2e/spec_decode/acceptance_rates/mtp_other/test_mtp.py::test_gemma4_mtp_acceptance_lengths[False]"; fi
- if [ "$$BUILDKITE_PARALLEL_JOB" = "2" ]; then pytest -v -s "v1/e2e/spec_decode/acceptance_rates/mtp_other/test_mtp.py::test_gemma4_mtp_acceptance_lengths[True]"; fi - if [ "$$BUILDKITE_PARALLEL_JOB" = "2" ]; then pytest -v -s "v1/e2e/spec_decode/acceptance_rates/mtp_other/test_mtp.py::test_gemma4_mtp_acceptance_lengths[True]"; fi
- label: ":amd: (MI355 DPX) V1 E2E Hybrid Chunked Prefill" # TBD
timeout_in_minutes: 35
mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]
dind: false
agent_pool: mi355_dpx
optional: true
source_file_dependencies:
- vllm/v1/attention/backends/utils.py
- vllm/v1/worker/gpu_model_runner.py
- tests/v1/e2e/test_hybrid_chunked_prefill.py
- vllm/v1/core/
working_dir: /vllm-workspace/tests
commands:
- pytest -v -s v1/e2e/test_hybrid_chunked_prefill.py
- label: ":amd: (MI355 DPX) Speculators Correctness Nightly"
timeout_in_minutes: 40
mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]
dind: false
agent_pool: mi355_dpx
optional: true
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/config/speculative.py
- vllm/engine/arg_utils.py
- vllm/transformers_utils/config.py
- vllm/transformers_utils/configs/speculators/
- vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/
- vllm/v1/worker/gpu_model_runner.py
- vllm/v1/sample/
- vllm/v1/attention/backends/
- vllm/v1/attention/selector.py
- vllm/model_executor/model_loader/
- vllm/model_executor/layers/
- vllm/model_executor/models/llama_eagle3.py
- vllm/model_executor/models/qwen3.py
- vllm/model_executor/models/qwen3_dflash.py
- vllm/model_executor/models/registry.py
- vllm/_aiter_ops.py
- tests/evals/gsm8k/
- tests/v1/spec_decode/test_speculators_correctness.py
- vllm/platforms/rocm.py
- vllm/v1/worker/gpu/spec_decode/dflash/
commands:
- export VLLM_ALLOW_INSECURE_SERIALIZATION=1
- pytest -v -s v1/spec_decode/test_speculators_correctness.py -m slow_test
- label: ":amd: (MI355 DPX) Spec Decode AL DFlash Nightly"
timeout_in_minutes: 120
mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]
dind: false
agent_pool: mi355_dpx
optional: true
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/
- vllm/model_executor/models/qwen3_dflash.py
- vllm/model_executor/models/qwen3_dflash2.py
- vllm/model_executor/models/qwen3_5.py
- vllm/model_executor/models/laguna_dflash.py
- tests/v1/e2e/spec_decode/
- vllm/v1/attention/
- vllm/v1/sample/
- vllm/model_executor/model_loader/
- vllm/model_executor/layers/
- vllm/model_executor/models/qwen2.py
- vllm/model_executor/models/qwen3.py
- vllm/model_executor/models/laguna.py
- vllm/platforms/rocm.py
commands:
- pytest -v -s v1/e2e/spec_decode/acceptance_rates/dflash/ -k "not dflash2"
- label: ":amd: (MI355 DPX) Spec Decode AL DSpark Nightly"
timeout_in_minutes: 90
mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]
dind: false
agent_pool: mi355_dpx
optional: true
working_dir: "/vllm-workspace/tests"
source_file_dependencies:
- vllm/v1/spec_decode/
- vllm/v1/worker/gpu/spec_decode/
- vllm/model_executor/models/qwen3_dspark.py
- vllm/model_executor/models/gemma4_dspark.py
- tests/v1/e2e/spec_decode/
- vllm/v1/attention/
- vllm/v1/sample/
- vllm/model_executor/model_loader/
- vllm/model_executor/layers/
- vllm/model_executor/models/qwen2.py
- vllm/model_executor/models/qwen3.py
- vllm/model_executor/models/qwen3_dflash.py
- vllm/model_executor/models/gemma4.py
- vllm/model_executor/models/gemma4_mm.py
- vllm/model_executor/models/gemma4_unified.py
- vllm/model_executor/models/gemma4_mtp.py
- vllm/platforms/rocm.py
commands:
- pytest -v -s v1/e2e/spec_decode/acceptance_rates/dspark/
#----------------------------------------------------------- mi355 · misc ------------------------------------------------------------# #----------------------------------------------------------- mi355 · misc ------------------------------------------------------------#
- label: ":amd: (MI355 DPX) Regression" # TBD - label: ":amd: (MI355 DPX) Regression" # TBD
+2 -2
View File
@@ -238,9 +238,9 @@ steps:
- pytest -v -s v1/e2e/test_hybrid_chunked_prefill.py - pytest -v -s v1/e2e/test_hybrid_chunked_prefill.py
mirror: mirror:
amd: amd:
label: ":amd: (MI300) V1 E2E Hybrid Chunked Prefill" label: ":amd: (MI355 DPX) V1 E2E Hybrid Chunked Prefill"
dind: false dind: false
device: mi300_1 device: mi355_dpx
timeout_in_minutes: 35 timeout_in_minutes: 35
depends_on: depends_on:
- image-build-amd - image-build-amd
+1 -2
View File
@@ -545,7 +545,6 @@ steps:
num_devices: 1 num_devices: 1
source_file_dependencies: source_file_dependencies:
- cmake/external_projects/deepgemm.cmake - cmake/external_projects/deepgemm.cmake
- tools/install_deepgemm.sh
- tools/build_deepgemm_C.py - tools/build_deepgemm_C.py
- tools/setup_deepgemm_pythons.sh - tools/setup_deepgemm_pythons.sh
- tools/check_wheel_deepgemm.py - tools/check_wheel_deepgemm.py
@@ -939,7 +938,7 @@ steps:
- label: ":nvidia: (B200) FusedMoE Layer Kernels" - label: ":nvidia: (B200) FusedMoE Layer Kernels"
key: kernels-fusedmoe-layer-test-2-b200s key: kernels-fusedmoe-layer-test-2-b200s
timeout_in_minutes: 90 timeout_in_minutes: 45
device: b200-k8s device: b200-k8s
num_devices: 2 num_devices: 2
source_file_dependencies: source_file_dependencies:
+11 -12
View File
@@ -15,9 +15,9 @@ steps:
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-small.txt - pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-small.txt
mirror: mirror:
amd: amd:
label: ":amd: (MI300) LM Eval Small Models" label: ":amd: (MI355) LM Eval Small Models"
dind: false dind: false
device: mi300_1 device: mi355_dpx
timeout_in_minutes: 45 timeout_in_minutes: 45
depends_on: depends_on:
- image-build-amd - image-build-amd
@@ -707,18 +707,17 @@ steps:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn - export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi355-nemotron-3-super.txt - pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi355-nemotron-3-super.txt
- label: ":amd: (MI300) LM Eval Large Models" - label: ":amd: (MI355) LM Eval Large Models"
key: amd-lm-eval-large-models key: amd-lm-eval-large-models
dind: false dind: false
device: mi300_8 device: mi355_8
optional: true optional: true
timeout_in_minutes: 65 timeout_in_minutes: 65
depends_on: depends_on:
- image-build-amd - image-build-amd
commands: commands:
- export VLLM_WORKER_MULTIPROC_METHOD=spawn - export VLLM_WORKER_MULTIPROC_METHOD=spawn
- export PYTORCH_ROCM_ARCH=gfx942 # Limit Quark compilation to save time - pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-gfx950-large.txt
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi3xx.txt
source_file_dependencies: source_file_dependencies:
- vllm/_aiter_ops.py - vllm/_aiter_ops.py
- vllm/platforms/rocm.py - vllm/platforms/rocm.py
@@ -1298,9 +1297,9 @@ steps:
- pytest -s -v evals/gsm8k/test_gsm8k_offloading.py -k "nemotron-h-8b or gemma-4-e4b-it" - pytest -s -v evals/gsm8k/test_gsm8k_offloading.py -k "nemotron-h-8b or gemma-4-e4b-it"
mirror: mirror:
amd: amd:
label: ":amd: (MI300) LM Eval KV-Offload" label: ":amd: (MI355 DPX) LM Eval KV-Offload"
dind: false dind: false
device: mi300_1 device: mi355_dpx
timeout_in_minutes: 55 timeout_in_minutes: 55
depends_on: depends_on:
- image-build-amd - image-build-amd
@@ -1360,9 +1359,9 @@ steps:
- pytest -s -v evals/gsm8k/test_gsm8k_offloading.py -k "deepseek-v4-flash" - pytest -s -v evals/gsm8k/test_gsm8k_offloading.py -k "deepseek-v4-flash"
mirror: mirror:
amd: amd:
label: ":amd: (MI300) LM Eval KV-Offload Large" label: ":amd: (MI355) LM Eval KV-Offload Large"
dind: false dind: false
device: mi300_4 device: mi355_4
timeout_in_minutes: 65 timeout_in_minutes: 65
depends_on: depends_on:
- image-build-amd - image-build-amd
@@ -1427,10 +1426,10 @@ steps:
# Qwen2-57B has 28 attention heads and cannot run at TP8. # Qwen2-57B has 28 attention heads and cannot run at TP8.
- pytest -s -v test_lm_eval_correctness.py --config-list-file=configs/models-large.txt --tp-size=4 - pytest -s -v test_lm_eval_correctness.py --config-list-file=configs/models-large.txt --tp-size=4
- label: ":amd: (MI300) LM Eval Large Models ROCm Harness" - label: ":amd: (MI355) LM Eval Large Models ROCm Harness"
key: amd-lm-eval-large-models-rocm-harness key: amd-lm-eval-large-models-rocm-harness
dind: false dind: false
device: mi300_8 device: mi355_8
optional: true optional: true
timeout_in_minutes: 40 timeout_in_minutes: 40
depends_on: depends_on:
+54
View File
@@ -164,6 +164,59 @@ steps:
depends_on: depends_on:
- image-build-amd - image-build-amd
# The shared KV Connectors steps above exclude vllm/distributed/kv_transfer/ from
# their triggers, and CUDA skips MoRIIO's mori/aiter-gated tests. Run the whole
# MoRIIO suite on ROCm whenever the connector or its tests change.
- label: ":amd: (MI355 DPX) MoRIIO Connector Unit Tests"
key: amd-moriio-connector-unit-mi355
dind: false
device: mi355_dpx
num_devices: 1
timeout_in_minutes: 30
working_dir: /vllm-workspace/tests
depends_on:
- image-build-amd
source_file_dependencies:
- vllm/distributed/kv_transfer/kv_connector/v1/moriio/
- vllm/distributed/kv_transfer/kv_connector/v1/base.py
- vllm/distributed/kv_transfer/kv_connector/v1/ssm_conv_transfer_utils.py
- examples/disaggregated/disaggregated_serving/moriio_toy_proxy_server.py
- tests/v1/kv_connector/unit/test_moriio_
- tests/v1/kv_connector/unit/utils.py
- requirements/kv_connectors_rocm.txt
commands:
- bash /vllm-workspace/.buildkite/scripts/install-kv-connectors.sh
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pytest -v -s -rs v1/kv_connector/unit/test_moriio_*.py
# Single-node MoRIIO WRITE P/D on dummy weights: prefill and decode on separate
# GPUs over xGMI, compared exactly against a direct serve.
- label: ":amd: (MI355) MoRIIO WRITE P/D Dummy-Weight E2E Shard %N"
key: amd-moriio-write-dummy-e2e-mi355
dind: false
device: mi355_2
num_devices: 2
# ~3.5-5 min per case; pytest-shard hashes test ids, so with 20 cases the
# largest of 6 shards holds 5 (~22 min on MI355). Every case picks free ports
# for its servers and proxy and its own compile-cache dir.
parallelism: 6
timeout_in_minutes: 45
working_dir: /vllm-workspace/tests
depends_on:
- image-build-amd
source_file_dependencies:
- vllm/distributed/kv_transfer/kv_connector/v1/moriio/
- vllm/distributed/kv_transfer/kv_connector/v1/base.py
- vllm/distributed/kv_transfer/kv_connector/v1/ssm_conv_transfer_utils.py
- vllm/model_executor/model_loader/dummy_loader.py
- examples/disaggregated/disaggregated_serving/moriio_toy_proxy_server.py
- tests/v1/kv_connector/moriio_integration/
- requirements/kv_connectors_rocm.txt
commands:
- bash /vllm-workspace/.buildkite/scripts/install-kv-connectors.sh
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
- pytest -v -s -rs v1/kv_connector/moriio_integration/test_write_dummy_e2e_rocm.py --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
- label: ":nvidia: (H200 MIG 35GB) V1 Metrics + LM Eval" - label: ":nvidia: (H200 MIG 35GB) V1 Metrics + LM Eval"
device: h200_35gb device: h200_35gb
key: v1-metrics-lmeval key: v1-metrics-lmeval
@@ -214,6 +267,7 @@ steps:
- vllm/v1/ - vllm/v1/
- tests/v1 - tests/v1
- tests/watermarking - tests/watermarking
- examples/disaggregated/mooncake_connector/mooncake_connector_proxy.py
device: cpu-small device: cpu-small
commands: commands:
# split the test to avoid interference # split the test to avoid interference
+2 -1
View File
@@ -72,8 +72,9 @@ steps:
- tests/models/test_deepseek_v41_pcp.py - tests/models/test_deepseek_v41_pcp.py
- tests/models/transformers/test_layer_registry_e2e.py - tests/models/transformers/test_layer_registry_e2e.py
- tests/models/test_deepseek_v41_replay_start.py - tests/models/test_deepseek_v41_replay_start.py
- tests/models/test_deepseek_v4_vl_stream_weights.py
commands: commands:
- pytest -v -s models/test_terratorch.py models/transformers/test_backend.py models/test_registry.py models/test_deepseek_v4_vl_rocm.py models/transformers/test_layer_registry_e2e.py models/test_deepseek_v41_replay_start.py models/test_deepseek_v41_decoder_replay_layers.py models/test_deepseek_v41_replay_batch.py models/test_deepseek_v41_pcp.py - pytest -v -s models/test_terratorch.py models/transformers/test_backend.py models/test_registry.py models/test_deepseek_v4_vl_rocm.py models/transformers/test_layer_registry_e2e.py models/test_deepseek_v41_replay_start.py models/test_deepseek_v41_decoder_replay_layers.py models/test_deepseek_v41_replay_batch.py models/test_deepseek_v41_pcp.py models/test_deepseek_v4_vl_stream_weights.py
mirror: mirror:
amd: amd:
label: ":amd: (MI355) Basic Models (Other)" label: ":amd: (MI355) Basic Models (Other)"
+6 -3
View File
@@ -210,15 +210,18 @@ steps:
- "!vllm/distributed/kv_transfer/" - "!vllm/distributed/kv_transfer/"
- tests/models/language/pooling - tests/models/language/pooling
commands: commands:
- pytest -v -s models/language/pooling -m 'not core_model' # Per-test timeout: a hung test fails with a stack dump instead of holding
# the job until Buildkite cancels it, which also blocks the full run's
# coverage collection. A whole shard takes 12-21 minutes.
- pytest -v -s models/language/pooling -m 'not core_model' --timeout=900 --timeout-method=thread
mirror: mirror:
amd: amd:
parallelism: 4 parallelism: 4
commands: commands:
- "pytest -v -s models/language/pooling -m 'not core_model' --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB" - "pytest -v -s models/language/pooling -m 'not core_model' --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB"
label: ":amd: (MI300) Language Models (Extended Pooling) Shard %N" label: ":amd: (MI355 DPX) Language Models (Extended Pooling) Shard %N"
dind: false dind: false
device: mi300_1 device: mi355_dpx
timeout_in_minutes: 95 timeout_in_minutes: 95
depends_on: depends_on:
- image-build-amd - image-build-amd
+6 -6
View File
@@ -163,9 +163,9 @@ steps:
- pytest -v -s models/multimodal/test_mapping.py - pytest -v -s models/multimodal/test_mapping.py
mirror: mirror:
amd: amd:
label: ":amd: (MI300) Multimodal Models (Extended Generation 1)" label: ":amd: (MI355 DPX) Multimodal Models (Extended Generation 1)"
dind: false dind: false
device: mi300_1 device: mi355_dpx
timeout_in_minutes: 100 timeout_in_minutes: 100
depends_on: depends_on:
- image-build-amd - image-build-amd
@@ -182,9 +182,9 @@ steps:
- pytest -v -s models/multimodal/generation_ppl_test/ - pytest -v -s models/multimodal/generation_ppl_test/
mirror: mirror:
amd: amd:
label: ":amd: (MI300) Multimodal Models (PPL)" label: ":amd: (MI355 DPX) Multimodal Models (PPL)"
dind: false dind: false
device: mi300_1 device: mi355_dpx
timeout_in_minutes: 90 timeout_in_minutes: 90
depends_on: depends_on:
- image-build-amd - image-build-amd
@@ -240,9 +240,9 @@ steps:
- pytest -v -s models/multimodal/pooling -m 'not core_model' - pytest -v -s models/multimodal/pooling -m 'not core_model'
mirror: mirror:
amd: amd:
label: ":amd: (MI300) Multimodal Models (Extended Pooling)" label: ":amd: (MI355 DPX) Multimodal Models (Extended Pooling)"
dind: false dind: false
device: mi300_1 device: mi355_dpx
timeout_in_minutes: 75 timeout_in_minutes: 75
depends_on: depends_on:
- image-build-amd - image-build-amd
+14 -14
View File
@@ -137,9 +137,9 @@ steps:
- pytest -v -s v1/e2e/spec_decode/speculators/ - pytest -v -s v1/e2e/spec_decode/speculators/
mirror: mirror:
amd: amd:
label: ":amd: (MI300) Spec Decode Speculators" label: ":amd: (MI355 DPX) Spec Decode Speculators"
dind: false dind: false
device: mi300_1 device: mi355_dpx
timeout_in_minutes: 40 timeout_in_minutes: 40
depends_on: depends_on:
- image-build-amd - image-build-amd
@@ -167,9 +167,9 @@ steps:
- pytest -v -s v1/e2e/spec_decode/mtp/gemma4/ - pytest -v -s v1/e2e/spec_decode/mtp/gemma4/
mirror: mirror:
amd: amd:
label: ":amd: (MI300) Spec Decode MTP Gemma4" label: ":amd: (MI355 DPX) Spec Decode MTP Gemma4"
dind: false dind: false
device: mi300_1 device: mi355_dpx
timeout_in_minutes: 40 timeout_in_minutes: 40
depends_on: depends_on:
- image-build-amd - image-build-amd
@@ -198,9 +198,9 @@ steps:
- pytest -v -s v1/e2e/spec_decode/mtp/mimo/ - pytest -v -s v1/e2e/spec_decode/mtp/mimo/
mirror: mirror:
amd: amd:
label: ":amd: (MI300) Spec Decode MTP DeepSeek + MiMo" label: ":amd: (MI355 DPX) Spec Decode MTP DeepSeek + MiMo"
dind: false dind: false
device: mi300_1 device: mi355_dpx
timeout_in_minutes: 40 timeout_in_minutes: 40
depends_on: depends_on:
- image-build-amd - image-build-amd
@@ -228,9 +228,9 @@ steps:
- pytest -v -s v1/e2e/spec_decode/mtp/qwen3_5/ - pytest -v -s v1/e2e/spec_decode/mtp/qwen3_5/
mirror: mirror:
amd: amd:
label: ":amd: (MI300) Spec Decode MTP Qwen3.5" label: ":amd: (MI355 DPX) Spec Decode MTP Qwen3.5"
dind: false dind: false
device: mi300_1 device: mi355_dpx
timeout_in_minutes: 40 timeout_in_minutes: 40
depends_on: depends_on:
- image-build-amd - image-build-amd
@@ -387,9 +387,9 @@ steps:
- pytest -v -s v1/spec_decode/test_speculators_correctness.py -m slow_test - pytest -v -s v1/spec_decode/test_speculators_correctness.py -m slow_test
mirror: mirror:
amd: amd:
label: ":amd: (MI300) Speculators Correctness Nightly" label: ":amd: (MI355 DPX) Speculators Correctness Nightly"
dind: false dind: false
device: mi300_1 device: mi355_dpx
timeout_in_minutes: 40 timeout_in_minutes: 40
depends_on: depends_on:
- image-build-amd - image-build-amd
@@ -452,9 +452,9 @@ steps:
- pytest -v -s v1/e2e/spec_decode/acceptance_rates/dflash/ -k "not dflash2" - pytest -v -s v1/e2e/spec_decode/acceptance_rates/dflash/ -k "not dflash2"
mirror: mirror:
amd: amd:
label: ":amd: (MI300) Spec Decode AL DFlash Nightly" label: ":amd: (MI355 DPX) Spec Decode AL DFlash Nightly"
dind: false dind: false
device: mi300_1 device: mi355_dpx
timeout_in_minutes: 120 timeout_in_minutes: 120
commands: commands:
- pytest -v -s v1/e2e/spec_decode/acceptance_rates/dflash/ -k "not dflash2" - pytest -v -s v1/e2e/spec_decode/acceptance_rates/dflash/ -k "not dflash2"
@@ -531,9 +531,9 @@ steps:
- pytest -v -s v1/e2e/spec_decode/acceptance_rates/dspark/ - pytest -v -s v1/e2e/spec_decode/acceptance_rates/dspark/
mirror: mirror:
amd: amd:
label: ":amd: (MI300) Spec Decode AL DSpark Nightly" label: ":amd: (MI355 DPX) Spec Decode AL DSpark Nightly"
dind: false dind: false
device: mi300_1 device: mi355_dpx
timeout_in_minutes: 90 timeout_in_minutes: 90
depends_on: depends_on:
- image-build-amd - image-build-amd
+5 -1
View File
@@ -8,7 +8,7 @@
/vllm/distributed/kv_transfer/kv_connector/v1/hisparse @MatthewBonanni @LucasWilkinson /vllm/distributed/kv_transfer/kv_connector/v1/hisparse @MatthewBonanni @LucasWilkinson
/vllm/lora @jeejeelee /vllm/lora @jeejeelee
/vllm/model_executor/layers/attention @LucasWilkinson @MatthewBonanni /vllm/model_executor/layers/attention @LucasWilkinson @MatthewBonanni
/vllm/model_executor/layers/fused_moe @mgoin @pavanimajety @zyongye @wzhao18 /vllm/model_executor/layers/fused_moe @mgoin @pavanimajety @zyongye @wzhao18 @bnellnm
/vllm/model_executor/layers/quantization @mgoin @robertgshaw2-redhat @tlrmchlsmth @yewentao256 @pavanimajety @zyongye /vllm/model_executor/layers/quantization @mgoin @robertgshaw2-redhat @tlrmchlsmth @yewentao256 @pavanimajety @zyongye
/vllm/model_executor/layers/mamba @tdoublep @tomeras91 @ZJY0516 /vllm/model_executor/layers/mamba @tdoublep @tomeras91 @ZJY0516
/vllm/model_executor/layers/mamba/gdn/qwen_gdn_linear_attn.py @tdoublep @ZJY0516 @vadiklyutiy /vllm/model_executor/layers/mamba/gdn/qwen_gdn_linear_attn.py @tdoublep @ZJY0516 @vadiklyutiy
@@ -138,6 +138,7 @@
/tests/evals @mgoin @vadiklyutiy @AndreasKaratzas @ZJY0516 /tests/evals @mgoin @vadiklyutiy @AndreasKaratzas @ZJY0516
/tests/kernels @mgoin @tlrmchlsmth @WoosukKwon @yewentao256 @zyongye @AndreasKaratzas /tests/kernels @mgoin @tlrmchlsmth @WoosukKwon @yewentao256 @zyongye @AndreasKaratzas
/tests/kernels/ir @ProExpertProg @tjtanaa /tests/kernels/ir @ProExpertProg @tjtanaa
/tests/kernels/moe @bnellnm
/tests/models @DarkLight1337 @ywang96 @AndreasKaratzas /tests/models @DarkLight1337 @ywang96 @AndreasKaratzas
/tests/multimodal @DarkLight1337 @ywang96 @NickLucche @shen-shanshan /tests/multimodal @DarkLight1337 @ywang96 @NickLucche @shen-shanshan
/tests/multimodal/media @Isotr0py /tests/multimodal/media @Isotr0py
@@ -164,6 +165,9 @@
.readthedocs.yaml @hmellor .readthedocs.yaml @hmellor
mkdocs.yaml @hmellor mkdocs.yaml @hmellor
# Model recipes
/recipes @esmeetu
# Linting # Linting
/tools/pre_commit/.markdownlint.yaml @hmellor /tools/pre_commit/.markdownlint.yaml @hmellor
.pre-commit-config.yaml @hmellor .pre-commit-config.yaml @hmellor
+28
View File
@@ -0,0 +1,28 @@
name: Recipes
# Validate recipe changes against the renderer that publishes them
# (recipes.vllm.ai). The check logic lives in vllm-project/recipes and tracks
# its main branch, so a PR is held to the same validation that runs when
# vllm-project/recipes picks up this directory after merge.
on:
pull_request:
paths:
- 'recipes/**'
- '.github/workflows/recipes.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
validate:
name: Recipes
if: github.repository_owner == 'vllm-project'
uses: vllm-project/recipes/.github/workflows/validate-recipes.yml@main
with:
source: vllm
path: recipes
+12 -2
View File
@@ -621,6 +621,16 @@ def blocks_new_run(command: str, build: Mapping[str, Any]) -> bool:
return is_active_build(build) return is_active_build(build)
def list_commit_builds(
buildkite: BuildkiteClient, command: str, commit: str
) -> list[dict[str, Any]]:
builds = buildkite.list_builds(commit)
if command not in AMD_CI_COMMANDS:
return builds
# bootstrap-amd.sh exits before uploading any test steps for PR webhook builds.
return [build for build in builds if build.get("source") != "webhook"]
def select_latest_build( def select_latest_build(
builds: Sequence[dict[str, Any]], builds: Sequence[dict[str, Any]],
pr_number: int, pr_number: int,
@@ -968,7 +978,7 @@ def handle_run_ci(
f"{ci_name} was already requested by this comment: {duplicate['web_url']}" f"{ci_name} was already requested by this comment: {duplicate['web_url']}"
) )
current_builds = buildkite.list_builds(pr["head"]["sha"]) current_builds = list_commit_builds(buildkite, command, pr["head"]["sha"])
active_build = next( active_build = next(
( (
build build
@@ -1023,7 +1033,7 @@ def handle_retry_failed(
ci_name = ci_name_for_command(command) ci_name = ci_name_for_command(command)
run_command = run_command_for_command(command) run_command = run_command_for_command(command)
retry_command = retry_command_for_command(command) retry_command = retry_command_for_command(command)
builds = buildkite.list_builds(pr["head"]["sha"]) builds = list_commit_builds(buildkite, command, pr["head"]["sha"])
build = select_latest_build(builds, pr["number"]) build = select_latest_build(builds, pr["number"])
if build: if build:
metadata = build.get("meta_data") or {} metadata = build.get("meta_data") or {}
@@ -883,6 +883,42 @@ class RunCiCommandTest(unittest.TestCase):
self.assertEqual(buildkite.created_builds, []) self.assertEqual(buildkite.created_builds, [])
self.assertIn("AMD CI is already running", github.comments[0]) self.assertIn("AMD CI is already running", github.comments[0])
def test_amd_run_ignores_finished_webhook_build_for_author(self) -> None:
webhook_build = {
"created_at": "2026-08-18T01:00:00Z",
"meta_data": {},
"number": 122,
"pull_request": {"id": 42},
"source": "webhook",
"state": "passed",
"web_url": "https://buildkite.example/amd-ci/builds/122",
}
ui_build = {**webhook_build, "source": "ui"}
command_build = {
**webhook_build,
"meta_data": {"github-comment-id": "98"},
"source": "api",
}
for build, allowed in (
(webhook_build, True),
(ui_build, False),
(command_build, False),
):
with self.subTest(source=build["source"]):
github = FakeGitHub(
permission="read",
pr=make_pr(labels=[{"name": "ready"}]),
)
buildkite = FakeBuildkite([[], [build]])
run(make_event(COMMAND_RUN_AMD_CI, "author"), github, buildkite)
self.assertEqual(bool(buildkite.created_builds), allowed)
if not allowed:
self.assertIn(
"AMD CI already ran on this commit", github.comments[0]
)
def test_run_all_sets_buildkite_environment(self) -> None: def test_run_all_sets_buildkite_environment(self) -> None:
github = FakeGitHub() github = FakeGitHub()
buildkite = FakeBuildkite([[], []]) buildkite = FakeBuildkite([[], []])
@@ -1229,6 +1265,29 @@ class RunCiCommandTest(unittest.TestCase):
) )
self.assertIn("Use `/amd-ci run`", github.comments[0]) self.assertIn("Use `/amd-ci run`", github.comments[0])
def test_amd_ci_retry_ignores_webhook_build_for_the_current_head(self) -> None:
github = FakeGitHub(
permission="read",
pr=make_pr(labels=[{"name": "ready"}]),
)
webhook_build = {
"created_at": "2026-08-18T01:00:00Z",
"meta_data": {},
"number": 122,
"pull_request": {"id": 42},
"source": "webhook",
"state": "passed",
"web_url": "https://buildkite.example/amd-ci/builds/122",
}
buildkite = FakeBuildkite([[webhook_build]])
run(make_event(COMMAND_RETRY_AMD_FAILED, "author"), github, buildkite)
self.assertEqual(buildkite.retry_calls, [])
self.assertIn(
"No AMD CI build exists for the current PR head", github.comments[0]
)
def test_ci_retry_creates_filtered_build_for_new_head(self) -> None: def test_ci_retry_creates_filtered_build_for_new_head(self) -> None:
github = FakeGitHub( github = FakeGitHub(
behind=100, behind=100,
+6
View File
@@ -277,6 +277,12 @@ repos:
files: ^docker/(Dockerfile|versions\.json)$ files: ^docker/(Dockerfile|versions\.json)$
pass_filenames: false pass_filenames: false
additional_dependencies: [dockerfile-parse] additional_dependencies: [dockerfile-parse]
- id: check-dockerignore-tracked
name: Check .dockerignore does not exclude tracked files
entry: python tools/pre_commit/check_dockerignore_tracked.py
language: python
always_run: true
pass_filenames: false
- id: check-boolean-context-manager - id: check-boolean-context-manager
name: Check for boolean ops in with-statements name: Check for boolean ops in with-statements
entry: python tools/pre_commit/check_boolean_context_manager.py entry: python tools/pre_commit/check_boolean_context_manager.py
+12
View File
@@ -415,6 +415,7 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
"csrc/cuda_view.cu" "csrc/cuda_view.cu"
"csrc/cuda_utils_kernels.cu" "csrc/cuda_utils_kernels.cu"
"csrc/activation_kernels.cu" "csrc/activation_kernels.cu"
"csrc/muse_spark_mul_tanh_soft_cap_kernel.cu"
"csrc/ngram_embedding_kernels.cu" "csrc/ngram_embedding_kernels.cu"
"csrc/vocab_parallel_embedding_kernels.cu" "csrc/vocab_parallel_embedding_kernels.cu"
"csrc/quantization/activation_kernels.cu" "csrc/quantization/activation_kernels.cu"
@@ -423,6 +424,7 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
"csrc/quantization/w8a8/fp8/per_token_group_quant.cu" "csrc/quantization/w8a8/fp8/per_token_group_quant.cu"
"csrc/quantization/gptq/q_gemm.cu" "csrc/quantization/gptq/q_gemm.cu"
"csrc/pos_encoding_kernels.cu" "csrc/pos_encoding_kernels.cu"
"csrc/fused_muse_spark_qknorm_rope_kv_insert_kernel.cu"
"csrc/fused_qknorm_rope_kernel.cu" "csrc/fused_qknorm_rope_kernel.cu"
"csrc/fused_minimax_m3_qknorm_rope_kv_insert_kernel.cu" "csrc/fused_minimax_m3_qknorm_rope_kv_insert_kernel.cu"
"csrc/layernorm_kernels.cu" "csrc/layernorm_kernels.cu"
@@ -441,6 +443,11 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
"csrc/fused_deepseek_v4_qnorm_rope_kv_insert_kernel.cu" "csrc/fused_deepseek_v4_qnorm_rope_kv_insert_kernel.cu"
"csrc/fused_kimi_k3_mla_key_concat_kv_cache_kernel.cu") "csrc/fused_kimi_k3_mla_key_concat_kv_cache_kernel.cu")
if(VLLM_GPU_LANG STREQUAL "CUDA")
list(APPEND VLLM_STABLE_EXT_SRC
"csrc/fused_qknorm_rope_cache_kernel.cu")
endif()
if(VLLM_GPU_LANG STREQUAL "CUDA" AND if(VLLM_GPU_LANG STREQUAL "CUDA" AND
DEFINED CMAKE_CUDA_COMPILER_VERSION AND DEFINED CMAKE_CUDA_COMPILER_VERSION AND
CMAKE_CUDA_COMPILER_VERSION VERSION_GREATER_EQUAL 12.0) CMAKE_CUDA_COMPILER_VERSION VERSION_GREATER_EQUAL 12.0)
@@ -1112,6 +1119,11 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
"9.0a;10.0f;12.0f" "${CUDA_ARCHS}") "9.0a;10.0f;12.0f" "${CUDA_ARCHS}")
cuda_archs_loose_intersection(FUSED_GDN_DECODE_ARCHS cuda_archs_loose_intersection(FUSED_GDN_DECODE_ARCHS
"8.0;8.6;8.9;9.0a;10.0f;11.0f;12.0f" "${CUDA_ARCHS}") "8.0;8.6;8.9;9.0a;10.0f;11.0f;12.0f" "${CUDA_ARCHS}")
else()
# The fused GDN decode kernel needs only cp.async and BF16 math. Only the
# family-specific targets (10.0f, 11.0f, 12.0f) need CUDA >= 13.0.
cuda_archs_loose_intersection(FUSED_GDN_DECODE_ARCHS
"8.0;8.6;8.9;9.0a" "${CUDA_ARCHS}")
endif() endif()
if(FUSED_KDA_DECODE_ARCHS) if(FUSED_KDA_DECODE_ARCHS)
set(FUSED_KDA_DECODE_SRC set(FUSED_KDA_DECODE_SRC
@@ -0,0 +1,326 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Benchmark the fused Muse-Spark QK-norm + RoPE + KV-insert kernel vs unfused.
``tuned_us`` forces the cp.async schedule ``fused_qk_norm_rope`` only auto-selects
on SM 9.0; ``unfused_us`` is what production runs.
"""
import argparse
import itertools
import torch
import vllm._custom_ops as ops
HEAD_DIM = 64
DEFAULT_TOKENS = [1, 8, 64, 256, 1024, 4096, 8192, 16384, 32768, 65536]
def make_cos_sin_cache(max_pos, rotary_dim, base, device):
inv_freq = 1.0 / (
base
** (
torch.arange(0, rotary_dim, 2, dtype=torch.float32, device=device)
/ rotary_dim
)
)
t = torch.arange(max_pos, dtype=torch.float32, device=device)
freqs = torch.einsum("i,j->ij", t, inv_freq)
return torch.cat((freqs.cos(), freqs.sin()), dim=-1)
class Case:
"""Everything a single (num_tokens, config) measurement needs."""
def __init__(self, num_tokens, nq, nkv, dtype, kv_cache_dtype, device):
self.num_tokens = num_tokens
self.nq, self.nkv = nq, nkv
self.dtype = dtype
self.kv_cache_dtype = kv_cache_dtype
self.eps = 1e-5
self.block_size = 128
self.rotary_dim = HEAD_DIM
self.qsz, self.kvsz = nq * HEAD_DIM, nkv * HEAD_DIM
self.row = self.qsz + 2 * self.kvsz + self.qsz # [q | k | v | o_gate]
self.qkv = torch.randn(num_tokens, self.row, dtype=dtype, device=device)
self.pristine = self.qkv.clone()
max_pos = 32768
self.cos_sin = make_cos_sin_cache(max_pos, self.rotary_dim, 500_000.0, device)
self.positions = torch.randint(
0, max_pos, (num_tokens,), dtype=torch.int64, device=device
)
num_blocks = (num_tokens + self.block_size - 1) // self.block_size + 1
storage = torch.uint8 if kv_cache_dtype != "auto" else dtype
self.kv_cache = torch.zeros(
num_blocks,
nkv,
self.block_size,
2 * HEAD_DIM,
dtype=storage,
device=device,
)
self.k_cache, self.v_cache = self.kv_cache.transpose(1, 2).split(
HEAD_DIM, dim=-1
)
self.slot_mapping = torch.randperm(
num_blocks * self.block_size, dtype=torch.int64, device=device
)[:num_tokens]
self.k_scale = torch.tensor(1.0, dtype=torch.float32, device=device)
self.v_scale = torch.tensor(1.0, dtype=torch.float32, device=device)
self.ones = torch.ones(HEAD_DIM, dtype=dtype, device=device)
self.q_fp8 = torch.empty(
num_tokens, self.qsz, dtype=torch.float8_e4m3fn, device=device
)
self.q_scale = torch.tensor(0.37, dtype=torch.float32, device=device)
def views(self):
q, k, v, _ = self.qkv.split([self.qsz, self.kvsz, self.kvsz, self.qsz], dim=-1)
return q, k, v
def fused(self, use_rope=True, quant_query=False):
ops.fused_muse_spark_qknorm_rope_kv_insert(
self.qkv,
self.positions,
self.nq,
self.nkv,
HEAD_DIM,
self.eps,
cos_sin_cache=self.cos_sin if use_rope else None,
rotary_dim=self.rotary_dim if use_rope else 0,
slot_mapping=self.slot_mapping,
kv_cache=self.kv_cache,
block_size=self.block_size,
kv_cache_dtype=self.kv_cache_dtype,
k_scale=self.k_scale if self.kv_cache_dtype != "auto" else None,
v_scale=self.v_scale if self.kv_cache_dtype != "auto" else None,
q_fp8_out=self.q_fp8 if quant_query else None,
q_scale=self.q_scale if quant_query else None,
)
def separate_query_quant(self, use_rope=True):
self.fused(use_rope)
ops.scaled_fp8_quant(self.qkv[:, : self.qsz], scale=self.q_scale)
def _insert(self):
_, k, v = self.views()
ops.reshape_and_cache_flash(
k.view(-1, self.nkv, HEAD_DIM),
v.view(-1, self.nkv, HEAD_DIM),
self.k_cache,
self.v_cache,
self.slot_mapping,
self.kv_cache_dtype,
self.k_scale,
self.v_scale,
)
def unfused_rope(self, token_heads_per_warp: int = -1):
ops.fused_qk_norm_rope(
self.qkv,
self.nq,
self.nkv,
self.nkv + self.nq, # v heads plus the o-gate
HEAD_DIM,
self.eps,
self.ones,
self.ones,
self.cos_sin,
False,
self.positions,
token_heads_per_warp,
)
self._insert()
def fused_bytes(self):
el = self.qkv.element_size()
cache_el = self.kv_cache.element_size()
read = (self.nq + 2 * self.nkv) * HEAD_DIM * el
write = (self.nq + self.nkv) * HEAD_DIM * el
cache = 2 * self.nkv * HEAD_DIM * cache_el
return self.num_tokens * (read + write + cache)
def _l2_bytes(device) -> int:
return torch.cuda.get_device_properties(device).L2_cache_size
def _peak_bandwidth(device) -> float:
"""Measured large-copy bandwidth, in GB/s -- the practical ceiling."""
n = 1 << 28 # 256 Mi elements = 512 MiB each way at 2 bytes
a = torch.empty(n, dtype=torch.bfloat16, device=device)
b = torch.empty_like(a)
for _ in range(3):
b.copy_(a)
torch.accelerator.synchronize()
start, end = torch.cuda.Event(True), torch.cuda.Event(True)
start.record()
iters = 20
for _ in range(iters):
b.copy_(a)
end.record()
torch.accelerator.synchronize()
ms = start.elapsed_time(end) / iters
return (2 * a.numel() * a.element_size()) / (ms * 1e-3) / 1e9
def time_eager(fn, warmup=20, iters=100) -> float:
for _ in range(warmup):
fn()
torch.accelerator.synchronize()
start, end = torch.cuda.Event(True), torch.cuda.Event(True)
start.record()
for _ in range(iters):
fn()
end.record()
torch.accelerator.synchronize()
return start.elapsed_time(end) / iters * 1e3 # us
def time_graph(fn, warmup=10, iters=50, inner=32) -> float:
"""GPU time per call; ``inner`` calls per graph amortize the ~2 us replay."""
side = torch.cuda.Stream()
side.wait_stream(torch.cuda.current_stream())
with torch.cuda.stream(side):
for _ in range(warmup):
fn()
torch.cuda.current_stream().wait_stream(side)
graph = torch.cuda.CUDAGraph()
with torch.cuda.graph(graph):
for _ in range(inner):
fn()
for _ in range(3):
graph.replay()
torch.accelerator.synchronize()
start, end = torch.cuda.Event(True), torch.cuda.Event(True)
start.record()
for _ in range(iters):
graph.replay()
end.record()
torch.accelerator.synchronize()
return start.elapsed_time(end) / (iters * inner) * 1e3 # us
def main():
p = argparse.ArgumentParser()
p.add_argument("--nq", type=int, default=32, help="query heads per rank")
p.add_argument("--nkv", type=int, default=4, help="kv heads per rank")
p.add_argument("--kv-cache-dtype", default="auto", choices=["auto", "fp8"])
p.add_argument("--dtype", default="bfloat16", choices=["bfloat16", "float16"])
p.add_argument("--tokens", type=int, nargs="*", default=DEFAULT_TOKENS)
p.add_argument("--csv", default=None)
p.add_argument("--no-graph", action="store_true")
p.add_argument(
"--query-fp8", action="store_true", help="compare fused Q quantization"
)
args = p.parse_args()
device = "cuda"
dtype = getattr(torch, args.dtype)
if args.query_fp8:
if args.kv_cache_dtype != "fp8":
p.error("--query-fp8 requires --kv-cache-dtype fp8")
timer = time_eager if args.no_graph else time_graph
print(f"device: {torch.cuda.get_device_name()}; nq={args.nq}, nkv={args.nkv}")
print("layer,tokens,separate_us,fused_us,speedup")
rows = []
for use_rope, nt in itertools.product([True, False], args.tokens):
case = Case(nt, args.nq, args.nkv, dtype, args.kv_cache_dtype, device)
separate_us = timer(lambda c=case, r=use_rope: c.separate_query_quant(r))
case.qkv.copy_(case.pristine)
fused_us = timer(lambda c=case, r=use_rope: c.fused(r, quant_query=True))
row = (
"rope" if use_rope else "nope",
nt,
separate_us,
fused_us,
separate_us / fused_us,
)
rows.append(row)
print(f"{row[0]},{nt},{separate_us:.3f},{fused_us:.3f},{row[4]:.3f}")
if args.csv:
import csv
with open(args.csv, "w", newline="") as f:
writer = csv.writer(f)
writer.writerow(
["layer", "tokens", "separate_us", "fused_us", "speedup"]
)
writer.writerows(rows)
return
peak = _peak_bandwidth(device)
print(
f"device: {torch.cuda.get_device_name()} measured copy peak: {peak:.0f} GB/s"
)
print(
f"config: nq={args.nq} nkv={args.nkv} head_dim={HEAD_DIM} "
f"dtype={args.dtype} kv_cache={args.kv_cache_dtype}\n"
)
modes = ["rope"]
l2 = _l2_bytes(device)
print(
f"rows marked * have a working set inside the {l2 / 2**20:.0f} MiB L2, "
"so their bandwidth is not an HBM number\n"
)
header = (
f"{'layer':>5} {'tokens':>7} {'fused_us':>9} {'unfused_us':>11} "
f"{'tuned_us':>9} {'speedup':>8} {'GB/s':>8} {'SOL%':>6}"
)
rows = []
for mode, use_graph in itertools.product(modes, [False, True]):
if use_graph and args.no_graph:
continue
label = "graph" if use_graph else "eager"
print(f"── {mode} layer, {label} ──")
print(header)
timer = time_graph if use_graph else time_eager
for nt in args.tokens:
case = Case(nt, args.nq, args.nkv, dtype, args.kv_cache_dtype, device)
use_rope = mode == "rope"
fused_us = timer(lambda c=case, r=use_rope: c.fused(r))
baseline = case.unfused_rope
unfused_us = timer(baseline)
tuned_us = (
min(timer(lambda c=case, t=t: c.unfused_rope(t)) for t in (4, 8))
if use_rope
else unfused_us
)
nbytes = case.fused_bytes()
gbps = nbytes / (fused_us * 1e-6) / 1e9
sol = 100.0 * gbps / peak
resident = "*" if nbytes < l2 else " "
print(
f"{mode:>5} {nt:>7} {fused_us:>9.2f} {unfused_us:>11.2f} "
f"{tuned_us:>9.2f} {tuned_us / fused_us:>7.2f}x {gbps:>8.0f} "
f"{sol:>5.1f}%{resident}"
)
rows.append((mode, label, nt, fused_us, unfused_us, tuned_us, gbps, sol))
print()
if args.csv:
import csv
with open(args.csv, "w", newline="") as f:
w = csv.writer(f)
w.writerow(
[
"layer",
"timing",
"tokens",
"fused_us",
"unfused_us",
"tuned_us",
"gbps",
"sol",
]
)
w.writerows(rows)
print(f"wrote {args.csv}")
if __name__ == "__main__":
main()
-1
View File
@@ -27,7 +27,6 @@ if(DEEPGEMM_SRC_DIR)
set(deepgemm_SOURCE_DIR "${DEEPGEMM_SRC_DIR}") set(deepgemm_SOURCE_DIR "${DEEPGEMM_SRC_DIR}")
message(STATUS "DeepGEMM using local DEEPGEMM_SRC_DIR: ${deepgemm_SOURCE_DIR}") message(STATUS "DeepGEMM using local DEEPGEMM_SRC_DIR: ${deepgemm_SOURCE_DIR}")
else() else()
# Keep in sync with tools/install_deepgemm.sh
set(_DEEPGEMM_UPSTREAM_REPO "https://github.com/vllm-project/DeepGEMM.git") set(_DEEPGEMM_UPSTREAM_REPO "https://github.com/vllm-project/DeepGEMM.git")
# Pinned to the fork's dev branch: upstream's 26/09/30 release (2.8.1) # Pinned to the fork's dev branch: upstream's 26/09/30 release (2.8.1)
# plus the SM120 port, the SM90 paged-MQA kv_block=32/next_n=4 port, # plus the SM120 port, the SM90 paged-MQA kv_block=32/next_n=4 port,
+4 -1
View File
@@ -425,6 +425,7 @@ function(cuda_archs_loose_intersection OUT_CUDA_ARCHS SRC_CUDA_ARCHS TGT_CUDA_AR
if("${_base}" IN_LIST _TGT_CUDA_ARCHS) if("${_base}" IN_LIST _TGT_CUDA_ARCHS)
list(REMOVE_ITEM _SRC_CUDA_ARCHS "${_arch}") list(REMOVE_ITEM _SRC_CUDA_ARCHS "${_arch}")
list(REMOVE_ITEM _TGT_CUDA_ARCHS "${_base}") list(REMOVE_ITEM _TGT_CUDA_ARCHS "${_base}")
list(REMOVE_ITEM _TGT_CUDA_ARCHS "${_base}a" "${_base}f")
list(APPEND _CUDA_ARCHS "${_arch}") list(APPEND _CUDA_ARCHS "${_arch}")
endif() endif()
endif() endif()
@@ -464,8 +465,10 @@ function(cuda_archs_loose_intersection OUT_CUDA_ARCHS SRC_CUDA_ARCHS TGT_CUDA_AR
string(REGEX REPLACE "[af]$" "" _base "${_arch}") string(REGEX REPLACE "[af]$" "" _base "${_arch}")
if ("${_base}" IN_LIST _SRC_CUDA_ARCHS) if ("${_base}" IN_LIST _SRC_CUDA_ARCHS)
list(REMOVE_ITEM _TGT_CUDA_ARCHS "${_arch}") list(REMOVE_ITEM _TGT_CUDA_ARCHS "${_arch}")
list(REMOVE_ITEM _SRC_CUDA_ARCHS "${_base}")
list(APPEND _CUDA_ARCHS "${_arch}") list(APPEND _CUDA_ARCHS "${_arch}")
if (NOT "${_base}" IN_LIST _TGT_CUDA_ARCHS)
list(REMOVE_ITEM _SRC_CUDA_ARCHS "${_base}")
endif()
endif() endif()
endif() endif()
endforeach() endforeach()
-27
View File
@@ -1059,33 +1059,6 @@ void situ_and_mul(torch::stable::Tensor& out, // [..., d]
}); });
} }
void masked_situ_and_mul(torch::stable::Tensor& out, // [E, T, d]
torch::stable::Tensor& input, // [E, T, 2 * d]
const torch::stable::Tensor& expert_num_tokens,
double beta, double linear_beta) {
int num_experts = input.size(0);
int max_num_tokens = input.size(1);
int d = input.size(2) / 2;
if (num_experts == 0 || max_num_tokens == 0 || d == 0) {
return;
}
constexpr int block_size = 256;
dim3 grid((d + block_size - 1) / block_size, num_experts);
dim3 block(block_size);
const torch::stable::accelerator::DeviceGuard device_guard(
input.get_device_index());
const cudaStream_t stream = get_current_cuda_stream();
VLLM_STABLE_DISPATCH_FLOATING_TYPES(
input.scalar_type(), "masked_situ_and_mul_kernel", [&] {
vllm::masked_situ_and_mul_kernel<scalar_t, true>
<<<grid, block, 0, stream>>>(
out.mutable_data_ptr<scalar_t>(),
input.const_data_ptr<scalar_t>(),
expert_num_tokens.const_data_ptr<int>(), max_num_tokens, d,
(float)beta, (float)linear_beta);
});
}
#define LAUNCH_MASKED_ACT_AND_MUL(KERNEL, ACT_FIRST, HAS_CLAMP, LIMIT, ALPHA, \ #define LAUNCH_MASKED_ACT_AND_MUL(KERNEL, ACT_FIRST, HAS_CLAMP, LIMIT, ALPHA, \
BETA) \ BETA) \
vllm::masked_act_and_mul_kernel<scalar_t, KERNEL<scalar_t>, ACT_FIRST, \ vllm::masked_act_and_mul_kernel<scalar_t, KERNEL<scalar_t>, ACT_FIRST, \
+76 -5
View File
@@ -1,7 +1,7 @@
// SPDX-License-Identifier: Apache-2.0 // SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright contributors to the vLLM project // SPDX-FileCopyrightText: Copyright contributors to the vLLM project
// //
// Router GEMM: activation(T) x weight(fp32) -> fp32, M<=32, for the // Router GEMM: activation(T) x weight(fp32) -> fp32, M<=48, for the
// supported (E, H) pairs listed at the bottom of this file. // supported (E, H) pairs listed at the bottom of this file.
// Supports bf16 or fp32 activation; weight is always fp32. // Supports bf16 or fp32 activation; weight is always fp32.
@@ -214,8 +214,26 @@ void invokeFp32RouterGemm(float* output, InputT const* mat_a,
// production-fidelity harness (CUDA-graph replay, per-layer cold weights). // production-fidelity harness (CUDA-graph replay, per-layer cold weights).
// Only enabled on the Blackwell family where it was validated; Hopper and // Only enabled on the Blackwell family where it was validated; Hopper and
// other shapes / fp32 activation keep the legacy geometry. // other shapes / fp32 activation keep the legacy geometry.
if constexpr (std::is_same_v<InputT, __nv_bfloat16> && kNumExperts == 128 && if constexpr (std::is_same_v<InputT, __nv_bfloat16> && kNumExperts == 256 &&
kHiddenDim == 6144) { kHiddenDim == 5120) {
if (!isBlackwellFamily()) {
launchFp32RouterGemm<InputT, 128, 1, kNumTokens, kNumExperts, kHiddenDim>(
output, mat_a, mat_b, stream);
return;
}
if constexpr (kNumTokens > 16 && kNumTokens % 8 == 0) {
constexpr int block_size = kNumTokens == 24 ? 160 : 128;
launchFp32RouterGemm<InputT, block_size, 2, kNumTokens, kNumExperts,
kHiddenDim, 4>(output, mat_a, mat_b, stream);
} else {
constexpr int block_size = kNumTokens == 1 ? 64 : 128;
constexpr int token_groups = kNumTokens % 2 == 0 ? 2 : 1;
launchFp32RouterGemm<InputT, block_size, 1, kNumTokens, kNumExperts,
kHiddenDim, token_groups>(output, mat_a, mat_b,
stream);
}
} else if constexpr (std::is_same_v<InputT, __nv_bfloat16> &&
kNumExperts == 128 && kHiddenDim == 6144) {
// MiniMax-M3. Legacy 128/1 only fills 128 blocks and pays the same // MiniMax-M3. Legacy 128/1 only fills 128 blocks and pays the same
// accumulator register cliffs; B300 sweep: // accumulator register cliffs; B300 sweep:
// even M in [6, 10] : BS=384, EPB=1, 2 token groups (1.26-1.43x) // even M in [6, 10] : BS=384, EPB=1, 2 token groups (1.26-1.43x)
@@ -261,6 +279,27 @@ void invokeFp32RouterGemm(float* output, InputT const* mat_a,
launchFp32RouterGemm<InputT, 128, 1, kNumTokens, kNumExperts, kHiddenDim>( launchFp32RouterGemm<InputT, 128, 1, kNumTokens, kNumExperts, kHiddenDim>(
output, mat_a, mat_b, stream); output, mat_a, mat_b, stream);
} }
} else if constexpr (std::is_same_v<InputT, __nv_bfloat16> &&
kNumExperts == 256 && kHiddenDim == 8192) {
// MuseSpark1x: the 8MB weight makes this shape FFMA-bound past M~10, so
// the wider block wins at every M and token groups clear the M >= 28
// accumulator cliff. Thresholds are from a B300 sweep. BS=192/384 are not
// instantiable: VPT=8 requires kBlockSize to divide kHiddenDim / 8 = 1024.
if (!isBlackwellFamily()) {
launchFp32RouterGemm<InputT, 128, 1, kNumTokens, kNumExperts, kHiddenDim>(
output, mat_a, mat_b, stream);
return;
}
if constexpr (kNumTokens >= 10 && kNumTokens % 2 == 0) {
launchFp32RouterGemm<InputT, 256, 2, kNumTokens, kNumExperts, kHiddenDim,
2>(output, mat_a, mat_b, stream);
} else if constexpr (kNumTokens >= 13) {
launchFp32RouterGemm<InputT, 256, 2, kNumTokens, kNumExperts, kHiddenDim>(
output, mat_a, mat_b, stream);
} else {
launchFp32RouterGemm<InputT, 256, 1, kNumTokens, kNumExperts, kHiddenDim>(
output, mat_a, mat_b, stream);
}
} else { } else {
launchFp32RouterGemm<InputT, 128, 1, kNumTokens, kNumExperts, kHiddenDim>( launchFp32RouterGemm<InputT, 128, 1, kNumTokens, kNumExperts, kHiddenDim>(
output, mat_a, mat_b, stream); output, mat_a, mat_b, stream);
@@ -268,8 +307,9 @@ void invokeFp32RouterGemm(float* output, InputT const* mat_a,
} }
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
// Explicit instantiations: M=1..32, for both input types, for the supported // Explicit instantiations for both input types.
// (E, H) pairs: (256, 3072) [MiniMax-M2/M2.5] and (128, 6144) [MiniMax-M3]. // Supported (E, H) pairs: (256, 3072), (128, 6144), (256, 5120),
// (352, 4096) [Step-5], and (256, 8192) [Muse Spark].
// --------------------------------------------------------------------------- // ---------------------------------------------------------------------------
#define INSTANTIATE(T, M, E, H) \ #define INSTANTIATE(T, M, E, H) \
@@ -315,5 +355,36 @@ INSTANTIATE_ALL(__nv_bfloat16, 256, 3072)
INSTANTIATE_ALL(float, 128, 6144) INSTANTIATE_ALL(float, 128, 6144)
INSTANTIATE_ALL(__nv_bfloat16, 128, 6144) INSTANTIATE_ALL(__nv_bfloat16, 128, 6144)
INSTANTIATE_ALL(float, 256, 5120)
INSTANTIATE_ALL(__nv_bfloat16, 256, 5120)
INSTANTIATE_ALL(float, 352, 4096)
INSTANTIATE_ALL(__nv_bfloat16, 352, 4096)
INSTANTIATE_ALL(float, 256, 8192)
INSTANTIATE_ALL(__nv_bfloat16, 256, 8192)
#define INSTANTIATE_LARGE(T, E, H) \
INSTANTIATE(T, 33, E, H) \
INSTANTIATE(T, 34, E, H) \
INSTANTIATE(T, 35, E, H) \
INSTANTIATE(T, 36, E, H) \
INSTANTIATE(T, 37, E, H) \
INSTANTIATE(T, 38, E, H) \
INSTANTIATE(T, 39, E, H) \
INSTANTIATE(T, 40, E, H) \
INSTANTIATE(T, 41, E, H) \
INSTANTIATE(T, 42, E, H) \
INSTANTIATE(T, 43, E, H) \
INSTANTIATE(T, 44, E, H) \
INSTANTIATE(T, 45, E, H) \
INSTANTIATE(T, 46, E, H) \
INSTANTIATE(T, 47, E, H) \
INSTANTIATE(T, 48, E, H)
INSTANTIATE_LARGE(float, 256, 5120)
INSTANTIATE_LARGE(__nv_bfloat16, 256, 5120)
#undef INSTANTIATE_LARGE
#undef INSTANTIATE_ALL #undef INSTANTIATE_ALL
#undef INSTANTIATE #undef INSTANTIATE
+20 -7
View File
@@ -25,10 +25,13 @@ inline int getSMVersion() {
static constexpr int FP32_MAX_TOKENS = 32; static constexpr int FP32_MAX_TOKENS = 32;
// Supported (hidden_dim, num_experts) pairs (must match the instantiations in // Supported (hidden_dim, num_experts) pairs (must match the instantiations in
// fp32_router_gemm.cu): (3072, 256) for MiniMax-M2/M2.5, (6144, 128) for M3. // fp32_router_gemm.cu).
static inline bool fp32_router_gemm_supported(int hidden_dim, int num_experts) { static inline bool fp32_router_gemm_supported(int hidden_dim, int num_experts) {
return (hidden_dim == 3072 && num_experts == 256) || return (hidden_dim == 3072 && num_experts == 256) ||
(hidden_dim == 6144 && num_experts == 128); (hidden_dim == 6144 && num_experts == 128) ||
(hidden_dim == 5120 && num_experts == 256) ||
(hidden_dim == 4096 && num_experts == 352) ||
(hidden_dim == 8192 && num_experts == 256);
} }
// Forward declarations — 4 template params must match fp32_router_gemm.cu // Forward declarations — 4 template params must match fp32_router_gemm.cu
@@ -60,8 +63,7 @@ struct Fp32LoopUnroller<InputT, kNumExperts, kHiddenDim, kEnd, kEnd> {
invokeFp32RouterGemm<InputT, kEnd, kNumExperts, kHiddenDim>( invokeFp32RouterGemm<InputT, kEnd, kNumExperts, kHiddenDim>(
output, mat_a, mat_b, stream); output, mat_a, mat_b, stream);
} else { } else {
throw std::invalid_argument( throw std::invalid_argument("fp32_router_gemm: unsupported num_tokens");
"fp32_router_gemm: num_tokens must be in [1, 32]");
} }
} }
}; };
@@ -77,6 +79,15 @@ void dispatchFp32RouterGemm(int num_experts, int hidden_dim, int num_tokens,
} else if (num_experts == 128 && hidden_dim == 6144) { } else if (num_experts == 128 && hidden_dim == 6144) {
Fp32LoopUnroller<InputT, 128, 6144, 1, FP32_MAX_TOKENS>::unroll( Fp32LoopUnroller<InputT, 128, 6144, 1, FP32_MAX_TOKENS>::unroll(
num_tokens, output, mat_a, mat_b, stream); num_tokens, output, mat_a, mat_b, stream);
} else if (num_experts == 256 && hidden_dim == 5120) {
Fp32LoopUnroller<InputT, 256, 5120, 1, 48>::unroll(num_tokens, output,
mat_a, mat_b, stream);
} else if (num_experts == 352 && hidden_dim == 4096) {
Fp32LoopUnroller<InputT, 352, 4096, 1, FP32_MAX_TOKENS>::unroll(
num_tokens, output, mat_a, mat_b, stream);
} else if (num_experts == 256 && hidden_dim == 8192) {
Fp32LoopUnroller<InputT, 256, 8192, 1, FP32_MAX_TOKENS>::unroll(
num_tokens, output, mat_a, mat_b, stream);
} else { } else {
throw std::invalid_argument( throw std::invalid_argument(
"fp32_router_gemm: unsupported (hidden_dim, num_experts) pair"); "fp32_router_gemm: unsupported (hidden_dim, num_experts) pair");
@@ -111,9 +122,11 @@ void fp32_router_gemm(
STD_TORCH_CHECK( STD_TORCH_CHECK(
fp32_router_gemm_supported(hidden_dim, num_experts), fp32_router_gemm_supported(hidden_dim, num_experts),
"fp32_router_gemm: supported (hidden_dim, num_experts) pairs are " "fp32_router_gemm: supported (hidden_dim, num_experts) pairs are "
"(3072, 256) and (6144, 128)"); "(3072, 256), (6144, 128), (5120, 256), (4096, 352), and (8192, 256)");
STD_TORCH_CHECK(num_tokens <= FP32_MAX_TOKENS, const int max_tokens =
"fp32_router_gemm: num_tokens must be in [0, 32]"); hidden_dim == 5120 && num_experts == 256 ? 48 : FP32_MAX_TOKENS;
STD_TORCH_CHECK(num_tokens <= max_tokens,
"fp32_router_gemm: num_tokens exceeds supported range");
STD_TORCH_CHECK( STD_TORCH_CHECK(
mat_a.scalar_type() == torch::headeronly::ScalarType::Float || mat_a.scalar_type() == torch::headeronly::ScalarType::Float ||
mat_a.scalar_type() == torch::headeronly::ScalarType::BFloat16, mat_a.scalar_type() == torch::headeronly::ScalarType::BFloat16,
@@ -0,0 +1,752 @@
/*
* SPDX-License-Identifier: Apache-2.0
* SPDX-FileCopyrightText: Copyright contributors to the vLLM project
*
* Fused Muse Spark attention pre-processing: weightless QK RMSNorm, GPT-J
* interleaved RoPE (skipped on NoPE layers, ``cos_sin_cache == None``) and
* the K/V insert of ``reshape_and_cache_flash``, over ``[q | k | v | o_gate]``
* rows.
*
* A group of ``head_dim * sizeof(scalar_t) / 16`` lanes owns one head (8 lanes
* for bf16 head_dim=64), so each lane moves one 16-byte vector, the rotated
* pair (2i, 2i+1) is lane-local, and the norm reduction is 3 shuffles.
*
* Slot enumeration per token:
* [0, nq) Q heads -> norm (+RoPE), in place or to
* ``q_fp8_out``
* [nq, nq+nkv) K heads -> norm (+RoPE), in place and into
* the K cache
* [nq+nkv, nq+2*nkv) V heads -> copied into the V cache (no norm)
* V slots exist only when a KV cache is bound.
*
* Norm and rotation run in fp32 and round once at the store. Both fp8 paths
* quantize after rounding to the model dtype, the same boundary
* ``reshape_and_cache_flash`` and ``scaled_fp8_quant`` use.
*/
#include <algorithm>
#include <cmath>
#include <cuda_runtime.h>
#include <type_traits>
#include "torch_utils.h"
#include "core/cuda_compat.h"
#include "type_convert.cuh"
#include "core/vec_dtypes/dtype_fp8.cuh"
#include "dispatch_utils.h"
#ifndef USE_ROCM
#include <cuda_fp8.h>
#include "quantization/w8a8/fp8/nvidia/quant_utils.cuh"
#else
#include <hip/hip_fp8.h>
#include "quantization/w8a8/fp8/amd/quant_utils.cuh"
#endif
#ifndef FINAL_MASK
#ifdef USE_ROCM
#define FINAL_MASK 0xffffffffffffffffULL
#else
#define FINAL_MASK 0xffffffffu
#endif
#endif
namespace vllm {
namespace muse_spark_fused_ops {
namespace {
inline int getSMVersion() {
auto* props = get_device_prop();
return props->major * 10 + props->minor;
}
inline int getSMCount() {
auto* props = get_device_prop();
return props->multiProcessorCount;
}
} // namespace
// 16 bytes per lane: the widest vector access, and the reason a head is owned
// by a group of lanes rather than a whole warp.
constexpr int kBytesPerLane = 16;
#ifdef USE_ROCM
using WarpMask = uint64_t;
constexpr unsigned kWarpSize = 64;
#else
using WarpMask = unsigned;
constexpr unsigned kWarpSize = 32;
#endif
// Slot -> (token, head) needs a division by a value only known at launch. The
// kernel is issue-bound, and a 32-bit hardware division is ~20 instructions of
// it, so the reciprocal is computed once on the host instead.
struct FastDiv {
uint32_t mul;
uint32_t shr;
int div;
};
inline FastDiv makeFastDiv(int denom) {
// The reciprocal form needs denom >= 2; denom == 1 is carried as a flag the
// device side selects on. Dividends stay below 2^31, which covers every slot
// index this kernel can form.
if (denom <= 1) return FastDiv{0u, 0u, 1};
int lg = 31 - __builtin_clz(denom);
if ((1 << lg) < denom) lg++;
uint32_t const p = 31 + static_cast<uint32_t>(lg);
uint64_t const m = ((1ull << p) + static_cast<uint64_t>(denom) - 1) /
static_cast<uint64_t>(denom);
return FastDiv{static_cast<uint32_t>(m), p - 32, denom};
}
__device__ __forceinline__ void fastDivmod(int& quo, int& rem, int src,
FastDiv const fd) {
int const q =
static_cast<int>(__umulhi(static_cast<uint32_t>(src), fd.mul) >> fd.shr);
// A select, not a branch: fd.div is uniform across the launch.
quo = (fd.div == 1) ? src : q;
rem = src - quo * fd.div;
}
// Sum across the LANES contiguous lanes that share one head. The group is
// LANES-aligned inside the warp, so an XOR butterfly over masks < LANES never
// leaves it. `group_mask` names exactly the group: whole groups are uniformly
// active or inactive, so a warp whose other groups bailed out still syncs.
template <int LANES>
__device__ __forceinline__ float groupReduceSum(float val,
WarpMask const group_mask) {
#pragma unroll
for (int mask = LANES >> 1; mask > 0; mask >>= 1) {
val += __shfl_xor_sync(group_mask, val, mask, 32);
}
return val;
}
// 16 contiguous bytes -> kElems fp32 registers, keeping the raw vector so a
// pass-through slot (V) can be re-stored without a conversion round trip.
template <typename scalar_t, int kElems>
__device__ __forceinline__ uint4 loadElems(scalar_t const* __restrict__ src,
float (&elems)[kElems]) {
using Converter = vllm::_typeConvert<scalar_t>;
using T2 = typename Converter::packed_hip_type;
uint4 raw = *reinterpret_cast<uint4 const*>(src);
auto const* p = reinterpret_cast<T2 const*>(&raw);
#pragma unroll
for (int i = 0; i < kElems / 2; i++) {
float2 f2 = Converter::convert(p[i]);
elems[2 * i] = f2.x;
elems[2 * i + 1] = f2.y;
}
return raw;
}
// The lane's slice of the rope table: kPairs cos and kPairs sin values, each
// run contiguous, so each is one vector load (16 bytes for the fp32 table, 8
// for a model-dtype one) instead of kPairs scalar ones. rotary_dim being a
// multiple of 16 is what keeps both runs naturally aligned.
template <typename rope_t, int kPairs>
__device__ __forceinline__ void loadRopeVec(rope_t const* __restrict__ cos_ptr,
rope_t const* __restrict__ sin_ptr,
float (&c)[kPairs],
float (&s)[kPairs]) {
static_assert(kPairs == 4, "rope vector load assumes 4 pairs per lane");
if constexpr (std::is_same_v<rope_t, float>) {
float4 const cv = VLLM_LDG(reinterpret_cast<float4 const*>(cos_ptr));
float4 const sv = VLLM_LDG(reinterpret_cast<float4 const*>(sin_ptr));
c[0] = cv.x, c[1] = cv.y, c[2] = cv.z, c[3] = cv.w;
s[0] = sv.x, s[1] = sv.y, s[2] = sv.z, s[3] = sv.w;
} else {
using Converter = vllm::_typeConvert<rope_t>;
using T2 = typename Converter::packed_hip_type;
uint2 const cv = VLLM_LDG(reinterpret_cast<uint2 const*>(cos_ptr));
uint2 const sv = VLLM_LDG(reinterpret_cast<uint2 const*>(sin_ptr));
auto const* cp = reinterpret_cast<T2 const*>(&cv);
auto const* sp = reinterpret_cast<T2 const*>(&sv);
#pragma unroll
for (int i = 0; i < kPairs / 2; i++) {
float2 const cf = Converter::convert(cp[i]);
float2 const sf = Converter::convert(sp[i]);
c[2 * i] = cf.x, c[2 * i + 1] = cf.y;
s[2 * i] = sf.x, s[2 * i + 1] = sf.y;
}
}
}
// kElems fp32 registers -> the 16-byte scalar_t vector they round to. Returned
// rather than stored so the fp8 cache path can reuse the *rounded* values,
// which is what reshape_and_cache_flash quantizes from.
template <typename scalar_t, int kElems>
__device__ __forceinline__ uint4 packElems(float const (&elems)[kElems]) {
using Converter = vllm::_typeConvert<scalar_t>;
using T2 = typename Converter::packed_hip_type;
uint4 raw;
auto* p = reinterpret_cast<T2*>(&raw);
#pragma unroll
for (int i = 0; i < kElems / 2; i++) {
p[i] = Converter::convert(make_float2(elems[2 * i], elems[2 * i + 1]));
}
return raw;
}
// Scatter one lane's 16 bytes into the KV cache. kAuto stores the model dtype
// verbatim; the fp8 dtypes quantize the value *after* it has been rounded to
// scalar_t, which is the boundary reshape_and_cache_flash uses.
//
// `inv_scale` is taken once per thread: per-element fp32 divisions would cost
// more than the rest of the kernel. The extra rounding is 2^-24 relative.
template <typename scalar_t, typename cache_t, Fp8KVCacheDataType kv_dt,
int kElems>
__device__ __forceinline__ void storeCache(cache_t* __restrict__ dst,
uint4 const& packed,
float const inv_scale) {
if constexpr (kv_dt == Fp8KVCacheDataType::kAuto) {
*reinterpret_cast<uint4*>(dst) = packed;
} else {
using Converter = vllm::_typeConvert<scalar_t>;
using HipT = typename Converter::hip_type;
auto const* rounded = reinterpret_cast<HipT const*>(&packed);
#ifndef USE_ROCM
constexpr __nv_fp8_interpretation_t kInterp =
(kv_dt == Fp8KVCacheDataType::kFp8E4M3) ? __NV_E4M3 : __NV_E5M2;
__nv_fp8x2_storage_t out[kElems / 2];
#pragma unroll
for (int i = 0; i < kElems / 2; i++) {
float2 v =
make_float2(Converter::convert(rounded[2 * i]) * inv_scale,
Converter::convert(rounded[2 * i + 1]) * inv_scale);
out[i] = __nv_cvt_float2_to_fp8x2(v, __NV_SATFINITE, kInterp);
}
// kElems fp8 bytes: 8 for the 2-byte dtypes this kernel serves.
static_assert(kElems == 8, "fp8 store assumes 8 elements per lane");
*reinterpret_cast<uint2*>(dst) = *reinterpret_cast<uint2 const*>(out);
#else
#pragma unroll
for (int i = 0; i < kElems; i++) {
float const v = Converter::convert(rounded[i]) * inv_scale;
dst[i] = fp8::scaled_convert<cache_t, float, kv_dt>(v, 1.0f);
}
#endif
}
}
// Grid is 1D over (token, head-chunk) pairs. A chunk is HEADS_PER_GROUP
// consecutive head slots of one token, served by kLanes contiguous lanes; the
// whole chunk shares the token's position and its slice of the rope table, so
// the slot division, the position load and the cos/sin load are paid once per
// chunk. `slots_per_token % HEADS_PER_GROUP == 0` keeps a chunk inside one
// token. `rope_t` is the cos/sin cache dtype, independent of the qkv dtype.
template <typename scalar_t, typename cache_t, Fp8KVCacheDataType kv_dt,
typename rope_t, int HEAD_DIM, int HEADS_PER_GROUP, bool kDoRope,
bool kInsertKV>
__global__
__launch_bounds__(256, 6) void fusedMuseSparkQKNormRopeKVInsertKernel(
scalar_t* __restrict__ qkv, // [N, qkv_stride] in/out
rope_t const* __restrict__ cos_sin_cache, // [max_pos, rotary_dim]
int64_t const* __restrict__ positions, // [N] i64
int64_t const* __restrict__ slot_mapping, // [N] i64 or nullptr
cache_t* __restrict__ kv_cache, // [nb, nkv, bs, 2*HEAD_DIM]
float const* __restrict__ k_scale, // [1] or nullptr
float const* __restrict__ v_scale, // [1] or nullptr
uint8_t* __restrict__ q_fp8_out, // [N, nq * HEAD_DIM] or nullptr
float const* __restrict__ q_scale, // [1] or nullptr
float const eps, int const rotary_dim, int const num_tokens, int const nq,
int const nkv, FastDiv const chunks_div, int64_t const qkv_stride,
FastDiv const block_div, int64_t const kv_s_block, int64_t const kv_s_head,
int64_t const kv_s_token) {
#if (!defined(__CUDA_ARCH__) || __CUDA_ARCH__ < 800) && !defined(USE_ROCM)
// _typeConvert<BFloat16> is unavailable on pre-Ampere; this kernel only ever
// runs with bf16/fp16 inputs. Discard the bf16 body there.
if constexpr (std::is_same_v<scalar_t, c10::BFloat16>) {
return;
} else {
#endif
constexpr int kElems = kBytesPerLane / sizeof(scalar_t);
constexpr int kLanes = HEAD_DIM / kElems;
constexpr int kPairs = kElems / 2;
constexpr int kH = HEADS_PER_GROUP;
static_assert(kLanes > 0 && kLanes <= 32 && (32 % kLanes) == 0,
"head_dim must give a power-of-two lane group <= 32");
int const tid = blockIdx.x * blockDim.x + threadIdx.x;
int const groupIdx = tid / kLanes;
int const dimBase = (tid % kLanes) * kElems;
int tokenIdx, chunk;
fastDivmod(tokenIdx, chunk, groupIdx, chunks_div);
int const firstSlot = chunk * kH;
// Out-of-range groups must not exit before the reductions below: the XOR
// butterfly needs every lane the mask names, and the mask covers the whole
// group. They ride along and skip every memory access instead.
bool const active = tokenIdx < num_tokens;
constexpr WarpMask kGroupBits =
(kLanes == 32) ? static_cast<WarpMask>(0xffffffffu)
: ((WarpMask{1} << kLanes) - WarpMask{1});
unsigned const group_shift =
(threadIdx.x & (kWarpSize - 1)) & ~static_cast<unsigned>(kLanes - 1);
WarpMask const group_mask = kGroupBits << group_shift;
// PDL: wait for the qkv-projection GEMM that produces `qkv`. No-op when
// PDL is not enabled on the launch.
#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900)
cudaGridDependencySynchronize();
#endif
// Q, K and V are contiguous head slots at the head of the row, so the slot
// index doubles as the head offset. The trailing o-gate is skipped by the
// row stride alone.
scalar_t* const base = qkv + static_cast<int64_t>(tokenIdx) * qkv_stride +
static_cast<int64_t>(firstSlot) * HEAD_DIM + dimBase;
// Issue every load in the chunk before touching any of it, so the group has
// kH loads in flight instead of one. An inactive group leaves these unset;
// nothing downstream reads them, since every store is gated on `active`.
uint4 raw[kH];
if (active) {
#pragma unroll
for (int h = 0; h < kH; h++) {
raw[h] = *reinterpret_cast<uint4 const*>(base + h * HEAD_DIM);
}
}
float cosv[kPairs], sinv[kPairs];
bool const laneRotates = kDoRope && dimBase < rotary_dim;
if constexpr (kDoRope) {
if (active && laneRotates) {
int64_t const pos = positions[tokenIdx];
rope_t const* cache_ptr = cos_sin_cache + pos * rotary_dim;
loadRopeVec<rope_t, kPairs>(cache_ptr + dimBase / 2,
cache_ptr + rotary_dim / 2 + dimBase / 2,
cosv, sinv);
}
}
int64_t sm = -1;
if constexpr (kInsertKV) {
if (active) sm = slot_mapping[tokenIdx];
}
int cache_block = 0, cache_tok = 0;
float k_inv = 1.0f, v_inv = 1.0f;
float const q_inv = q_fp8_out ? 1.0f / *q_scale : 1.0f;
if constexpr (kInsertKV) {
if (sm >= 0) {
fastDivmod(cache_block, cache_tok, static_cast<int>(sm), block_div);
}
if constexpr (kv_dt != Fp8KVCacheDataType::kAuto) {
k_inv = 1.0f / *k_scale;
v_inv = 1.0f / *v_scale;
}
}
#pragma unroll
for (int h = 0; h < kH; h++) {
int const slot = firstSlot + h;
bool const isV = kInsertKV && slot >= nq + nkv;
bool const isK = slot >= nq && slot < nq + nkv;
bool const isQK = active && !isV;
float elems[kElems];
uint4 packed = raw[h];
if (isQK) {
using Converter = vllm::_typeConvert<scalar_t>;
using T2 = typename Converter::packed_hip_type;
auto const* p = reinterpret_cast<T2 const*>(&raw[h]);
#pragma unroll
for (int i = 0; i < kElems / 2; i++) {
float2 const f2 = Converter::convert(p[i]);
elems[2 * i] = f2.x;
elems[2 * i + 1] = f2.y;
}
float sumsq = 0.0f;
#pragma unroll
for (int i = 0; i < kElems; i++) sumsq += elems[i] * elems[i];
sumsq = groupReduceSum<kLanes>(sumsq, group_mask);
float const rms_rcp =
rsqrtf(sumsq / static_cast<float>(HEAD_DIM) + eps);
#pragma unroll
for (int i = 0; i < kElems; i++) elems[i] *= rms_rcp;
if constexpr (kDoRope) {
if (laneRotates) {
#pragma unroll
for (int i = 0; i < kPairs; i++) {
float const v0 = elems[2 * i];
float const v1 = elems[2 * i + 1];
elems[2 * i] = v0 * cosv[i] - v1 * sinv[i];
elems[2 * i + 1] = v0 * sinv[i] + v1 * cosv[i];
}
}
}
// Rounding to scalar_t happens exactly once, here, and both the
// Q output and the fp8 cache quantization consume the same
// rounded value. V passes through with the vector it was loaded with.
packed = packElems<scalar_t, kElems>(elems);
if (q_fp8_out && slot < nq) {
int64_t const q_off =
(static_cast<int64_t>(tokenIdx) * nq + slot) * HEAD_DIM + dimBase;
storeCache<scalar_t, uint8_t, Fp8KVCacheDataType::kFp8E4M3, kElems>(
q_fp8_out + q_off, packed, q_inv);
} else {
*reinterpret_cast<uint4*>(base + h * HEAD_DIM) = packed;
}
}
if constexpr (kInsertKV) {
if (sm >= 0 && (isK || isV)) { // sm < 0 marks a padded token
int const head = isK ? (slot - nq) : (slot - nq - nkv);
int64_t const off = static_cast<int64_t>(cache_block) * kv_s_block +
head * kv_s_head +
static_cast<int64_t>(cache_tok) * kv_s_token +
(isK ? 0 : HEAD_DIM) + dimBase;
storeCache<scalar_t, cache_t, kv_dt, kElems>(kv_cache + off, packed,
isK ? k_inv : v_inv);
}
}
}
// PDL: let a dependent successor launch early.
#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900)
cudaTriggerProgrammaticLaunchCompletion();
#endif
#if (!defined(__CUDA_ARCH__) || __CUDA_ARCH__ < 800) && !defined(USE_ROCM)
}
#endif
}
template <typename scalar_t, typename cache_t, Fp8KVCacheDataType kv_dt,
typename rope_t, int HEAD_DIM, int HEADS_PER_GROUP>
void launchFusedMuseSparkTyped(
scalar_t* qkv, rope_t const* cos_sin_cache, int64_t const* positions,
int64_t const* slot_mapping, cache_t* kv_cache, float const* k_scale,
float const* v_scale, uint8_t* q_fp8_out, float const* q_scale,
float const eps, int const rotary_dim, int const num_tokens, int const nq,
int const nkv, int64_t const qkv_stride, int const block_size,
int64_t const kv_s_block, int64_t const kv_s_head, int64_t const kv_s_token,
bool const do_rope, bool const insert_kv, cudaStream_t stream) {
FastDiv const block_div = makeFastDiv(block_size);
constexpr int kElems = kBytesPerLane / sizeof(scalar_t);
constexpr int kLanes = HEAD_DIM / kElems;
int const slots_per_token = nq + nkv + (insert_kv ? nkv : 0);
int const chunks_per_token = slots_per_token / HEADS_PER_GROUP;
FastDiv const chunks_div = makeFastDiv(chunks_per_token);
int64_t const total_lanes =
static_cast<int64_t>(num_tokens) * chunks_per_token * kLanes;
if (total_lanes == 0) return;
constexpr int kBlockSize = 256;
int const grid =
static_cast<int>((total_lanes + kBlockSize - 1) / kBlockSize);
#ifndef USE_ROCM
// PDL wherever the hardware supports it (SM90+); a plain launch elsewhere.
static int const sm_version = getSMVersion();
cudaLaunchConfig_t config;
config.gridDim = dim3(grid);
config.blockDim = dim3(kBlockSize);
config.dynamicSmemBytes = 0;
config.stream = stream;
cudaLaunchAttribute attrs[1];
attrs[0].id = cudaLaunchAttributeProgrammaticStreamSerialization;
attrs[0].val.programmaticStreamSerializationAllowed = 1;
config.attrs = attrs;
config.numAttrs = (sm_version >= 90) ? 1 : 0;
#define LAUNCH_MUSE(DO_ROPE, INSERT) \
cudaLaunchKernelEx(&config, \
fusedMuseSparkQKNormRopeKVInsertKernel< \
scalar_t, cache_t, kv_dt, rope_t, HEAD_DIM, \
HEADS_PER_GROUP, DO_ROPE, INSERT>, \
qkv, cos_sin_cache, positions, slot_mapping, kv_cache, \
k_scale, v_scale, q_fp8_out, q_scale, eps, rotary_dim, \
num_tokens, nq, nkv, chunks_div, qkv_stride, block_div, \
kv_s_block, kv_s_head, kv_s_token)
#else
// clang-format off
#define LAUNCH_MUSE(DO_ROPE, INSERT) \
fusedMuseSparkQKNormRopeKVInsertKernel< \
scalar_t, cache_t, kv_dt, rope_t, HEAD_DIM, HEADS_PER_GROUP, DO_ROPE, \
INSERT><<<grid, kBlockSize, 0, stream>>>( \
qkv, cos_sin_cache, positions, slot_mapping, kv_cache, k_scale, \
v_scale, q_fp8_out, q_scale, eps, rotary_dim, num_tokens, nq, nkv, \
chunks_div, qkv_stride, block_div, kv_s_block, kv_s_head, kv_s_token)
// clang-format on
#endif
if (do_rope) {
if (insert_kv) {
LAUNCH_MUSE(true, true);
} else {
LAUNCH_MUSE(true, false);
}
} else {
if (insert_kv) {
LAUNCH_MUSE(false, true);
} else {
LAUNCH_MUSE(false, false);
}
}
#undef LAUNCH_MUSE
}
template <typename scalar_t, typename cache_t, Fp8KVCacheDataType kv_dt>
void launchFusedMuseSpark(
scalar_t* qkv, void const* cos_sin_cache, bool const rope_is_fp32,
int64_t const* positions, int64_t const* slot_mapping, cache_t* kv_cache,
float const* k_scale, float const* v_scale, uint8_t* q_fp8_out,
float const* q_scale, float const eps, int const rotary_dim,
int const num_tokens, int const nq, int const nkv, int const head_dim,
int64_t const qkv_stride, int const block_size, int64_t const kv_s_block,
int64_t const kv_s_head, int64_t const kv_s_token, bool const do_rope,
bool const insert_kv, cudaStream_t stream) {
// The group width must divide the slot count. Widen only while the grid
// still covers a full wave; decode-sized batches keep the narrow schedule.
int const slots_per_token = nq + nkv + (insert_kv ? nkv : 0);
constexpr int kElemsSel = kBytesPerLane / sizeof(scalar_t);
constexpr int kBlockSel = 256;
int const lanes_per_head = head_dim / kElemsSel;
int64_t const full_wave =
static_cast<int64_t>(getSMCount()) * (2048 / kBlockSel);
int heads_per_group = 1;
for (int h : {4, 2}) {
if (slots_per_token % h != 0) continue;
int64_t const blocks = (static_cast<int64_t>(num_tokens) *
(slots_per_token / h) * lanes_per_head +
kBlockSel - 1) /
kBlockSel;
if (blocks >= full_wave) {
heads_per_group = h;
break;
}
}
// TP4/TP8 amortize head addressing before filling a full CTA wave. RoPE
// benefits earlier; FP8 conversion keeps wider groups expensive for longer.
if (insert_kv && head_dim == 64 &&
((nq == 16 && nkv == 2) || (nq == 32 && nkv == 4))) {
int64_t const query_heads = static_cast<int64_t>(num_tokens) * nq;
if (do_rope && query_heads >= 24576) {
heads_per_group = std::max(heads_per_group, 2);
if constexpr (kv_dt == Fp8KVCacheDataType::kAuto) {
if (query_heads >= 49152) heads_per_group = 4;
}
} else if (!do_rope && query_heads >= 32768) {
if constexpr (kv_dt == Fp8KVCacheDataType::kAuto) {
heads_per_group = std::max(heads_per_group, 2);
}
}
}
#define DISPATCH_ARGS(RT, HD, HPG) \
launchFusedMuseSparkTyped<scalar_t, cache_t, kv_dt, RT, HD, HPG>( \
qkv, reinterpret_cast<RT const*>(cos_sin_cache), positions, \
slot_mapping, kv_cache, k_scale, v_scale, q_fp8_out, q_scale, eps, \
rotary_dim, num_tokens, nq, nkv, qkv_stride, block_size, kv_s_block, \
kv_s_head, kv_s_token, do_rope, insert_kv, stream)
#define DISPATCH_HPG(RT, HD) \
switch (heads_per_group) { \
case 4: \
DISPATCH_ARGS(RT, HD, 4); \
break; \
case 2: \
DISPATCH_ARGS(RT, HD, 2); \
break; \
default: \
DISPATCH_ARGS(RT, HD, 1); \
break; \
}
// A NoPE launch never reads the table, so it can share the fp32 branch.
#define DISPATCH_ROPE_T(HD) \
if (!do_rope || rope_is_fp32) { \
DISPATCH_HPG(float, HD); \
} else { \
DISPATCH_HPG(scalar_t, HD); \
}
switch (head_dim) {
case 64:
DISPATCH_ROPE_T(64);
break;
case 128:
DISPATCH_ROPE_T(128);
break;
case 256:
DISPATCH_ROPE_T(256);
break;
default:
STD_TORCH_CHECK(false, "unsupported head_dim: ", head_dim);
}
#undef DISPATCH_ROPE_T
#undef DISPATCH_HPG
#undef DISPATCH_ARGS
}
} // namespace muse_spark_fused_ops
} // namespace vllm
// clang-format off
#define CALL_FUSED_MUSE_SPARK(_RAW_T, CACHE_T, KV_DTYPE) \
vllm::muse_spark_fused_ops::launchFusedMuseSpark<st, CACHE_T, KV_DTYPE>( \
reinterpret_cast<st*>(qkv.data_ptr()), \
do_rope ? cos_sin_cache->data_ptr() : nullptr, rope_is_fp32, \
reinterpret_cast<int64_t const*>(positions.data_ptr()), \
insert_kv ? reinterpret_cast<int64_t const*>(slot_mapping->data_ptr()) \
: nullptr, \
insert_kv ? reinterpret_cast<CACHE_T*>(kv_cache->data_ptr()) : nullptr, \
needs_scales ? reinterpret_cast<float const*>(k_scale->data_ptr()) \
: nullptr, \
needs_scales ? reinterpret_cast<float const*>(v_scale->data_ptr()) \
: nullptr, \
q_fp8_out ? reinterpret_cast<uint8_t*>(q_fp8_out->data_ptr()) : nullptr, \
q_scale ? reinterpret_cast<float const*>(q_scale->data_ptr()) : nullptr, \
static_cast<float>(eps), static_cast<int>(rotary_dim), num_tokens, nq, \
nkv, hd, qkv_stride, static_cast<int>(block_size), kv_s_block, \
kv_s_head, kv_s_token, do_rope, insert_kv, stream)
// clang-format on
void fused_muse_spark_qknorm_rope_kv_insert(
torch::stable::Tensor& qkv, // [N, (nq + 2*nkv + gate)*hd]
torch::stable::Tensor const& positions, // [N] i64
int64_t num_heads, int64_t num_kv_heads, int64_t head_dim, double eps,
std::optional<torch::stable::Tensor> cos_sin_cache, // None => NoPE layer
int64_t rotary_dim,
std::optional<torch::stable::Tensor> slot_mapping, // [N] i64
std::optional<torch::stable::Tensor> kv_cache, // [nb, nkv, bs, 2*hd]
int64_t block_size, const std::string& kv_cache_dtype,
std::optional<torch::stable::Tensor> k_scale, // [1] fp32, fp8 cache only
std::optional<torch::stable::Tensor> v_scale,
std::optional<torch::stable::Tensor> q_fp8_out,
std::optional<torch::stable::Tensor> q_scale) {
STD_TORCH_CHECK(qkv.is_cuda() && qkv.dim() == 2 && qkv.stride(1) == 1,
"qkv must be a 2-D CUDA tensor with a contiguous row");
STD_TORCH_CHECK(
qkv.scalar_type() == torch::headeronly::ScalarType::Half ||
qkv.scalar_type() == torch::headeronly::ScalarType::BFloat16,
"qkv must be float16 or bfloat16");
STD_TORCH_CHECK(
positions.is_cuda() &&
positions.scalar_type() == torch::headeronly::ScalarType::Long,
"positions must be int64 CUDA");
int const num_tokens = static_cast<int>(qkv.size(0));
int const nq = static_cast<int>(num_heads);
int const nkv = static_cast<int>(num_kv_heads);
int const hd = static_cast<int>(head_dim);
int64_t const qkv_stride = qkv.stride(0);
STD_TORCH_CHECK(q_fp8_out.has_value() == q_scale.has_value(),
"q_fp8_out and q_scale must be provided together");
if (q_fp8_out) {
STD_TORCH_CHECK(
q_fp8_out->is_cuda() &&
q_fp8_out->get_device_index() == qkv.get_device_index() &&
q_fp8_out->scalar_type() ==
torch::headeronly::ScalarType::Float8_e4m3fn &&
q_fp8_out->dim() == 2 && q_fp8_out->is_contiguous() &&
q_fp8_out->size(0) == num_tokens &&
q_fp8_out->size(1) == static_cast<int64_t>(nq) * hd,
"q_fp8_out must be contiguous E4M3 [num_tokens, num_heads * head_dim] "
"on the qkv device");
STD_TORCH_CHECK(
q_scale->is_cuda() &&
q_scale->get_device_index() == qkv.get_device_index() &&
q_scale->scalar_type() == torch::headeronly::ScalarType::Float &&
q_scale->numel() == 1,
"q_scale must be a single fp32 value on the qkv device");
}
STD_TORCH_CHECK(hd == 64 || hd == 128 || hd == 256,
"head_dim must be 64, 128 or 256");
STD_TORCH_CHECK(qkv.size(1) >= static_cast<int64_t>(nq + 2 * nkv) * hd,
"qkv row must hold at least [q|k|v]");
// A lane moves 16 bytes, so every row start and every head start must land
// on a 16-byte boundary. Every supported head_dim is a multiple of 8 at 2
// bytes, so heads are fine; the row stride is the one that can break it.
STD_TORCH_CHECK(qkv_stride % 8 == 0,
"qkv row stride must be a multiple of 8 elements, got ",
qkv_stride);
bool const do_rope = cos_sin_cache.has_value();
bool rope_is_fp32 = true;
if (do_rope) {
STD_TORCH_CHECK(cos_sin_cache->is_cuda() && cos_sin_cache->dim() == 2 &&
cos_sin_cache->is_contiguous(),
"cos_sin_cache must be a contiguous 2-D CUDA tensor");
STD_TORCH_CHECK(cos_sin_cache->size(1) == rotary_dim,
"cos_sin_cache shape must be [max_pos, rotary_dim]");
STD_TORCH_CHECK(
cos_sin_cache->scalar_type() == torch::headeronly::ScalarType::Float ||
cos_sin_cache->scalar_type() == qkv.scalar_type(),
"cos_sin_cache must be float32 or match the qkv dtype");
rope_is_fp32 =
cos_sin_cache->scalar_type() == torch::headeronly::ScalarType::Float;
// A lane owns 16 bytes = 8 elements of the head and must be wholly inside
// or wholly outside the rotated prefix.
STD_TORCH_CHECK(rotary_dim > 0 && rotary_dim <= hd && rotary_dim % 16 == 0,
"rotary_dim must be a positive multiple of 16 and <= "
"head_dim");
}
vllm::Fp8KVCacheDataType const kv_dt =
vllm::get_fp8_kv_cache_data_type(kv_cache_dtype);
bool const insert_kv = kv_cache.has_value() && kv_cache->numel() > 0;
bool const needs_scales =
insert_kv && kv_dt != vllm::Fp8KVCacheDataType::kAuto;
int64_t kv_s_block = 0, kv_s_head = 0, kv_s_token = 0;
if (insert_kv) {
STD_TORCH_CHECK(
slot_mapping.has_value() && slot_mapping->is_cuda() &&
slot_mapping->scalar_type() == torch::headeronly::ScalarType::Long,
"insert mode requires an int64 CUDA slot_mapping");
STD_TORCH_CHECK(slot_mapping->numel() >= num_tokens,
"slot_mapping must cover every token");
STD_TORCH_CHECK(kv_cache->dim() == 4 && kv_cache->stride(3) == 1,
"kv_cache must be [nb, nkv, bs, 2*head_dim] with a "
"contiguous content dim");
STD_TORCH_CHECK(kv_cache->size(3) == 2 * hd,
"kv_cache content dim must be 2*head_dim");
STD_TORCH_CHECK(kv_cache->size(1) == nkv,
"kv_cache head dim must equal num_kv_heads");
STD_TORCH_CHECK(kv_cache->size(2) == block_size,
"kv_cache token dim must equal block_size");
if (kv_dt == vllm::Fp8KVCacheDataType::kAuto) {
STD_TORCH_CHECK(kv_cache->scalar_type() == qkv.scalar_type(),
"auto kv_cache dtype must match qkv");
} else {
STD_TORCH_CHECK(
kv_cache->scalar_type() == torch::headeronly::ScalarType::Byte ||
kv_cache->scalar_type() ==
torch::headeronly::ScalarType::Float8_e4m3fn,
"fp8 kv_cache must use uint8 or float8_e4m3fn storage");
STD_TORCH_CHECK(k_scale.has_value() && v_scale.has_value(),
"fp8 kv cache requires k_scale and v_scale");
STD_TORCH_CHECK(
k_scale->numel() == 1 && v_scale->numel() == 1 &&
k_scale->scalar_type() == torch::headeronly::ScalarType::Float &&
v_scale->scalar_type() == torch::headeronly::ScalarType::Float,
"k_scale and v_scale must be single fp32 values");
}
kv_s_block = kv_cache->stride(0);
kv_s_head = kv_cache->stride(1);
kv_s_token = kv_cache->stride(2);
}
const torch::stable::accelerator::DeviceGuard device_guard(
qkv.get_device_index());
auto stream = get_current_cuda_stream(qkv.get_device_index());
VLLM_STABLE_DISPATCH_HALF_TYPES(
qkv.scalar_type(), "fused_muse_spark_qknorm_rope_kv_insert", [&] {
using st = scalar_t;
DISPATCH_BY_KV_CACHE_DTYPE(qkv.scalar_type(), kv_cache_dtype,
CALL_FUSED_MUSE_SPARK);
});
}
#undef CALL_FUSED_MUSE_SPARK
+567
View File
@@ -0,0 +1,567 @@
// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright contributors to the vLLM project
// Step5 fused QK norm, RoPE, and KV-cache update kernel.
// This kernel is built into `_C_stable_libtorch` and uses the libtorch stable
// ABI (torch::stable::Tensor / STD_TORCH_CHECK) instead of ATen and c10.
#include "torch_utils.h"
#include "dispatch_utils.h"
#include "type_convert.cuh"
#include "core/vec_dtypes/dtype_fp8.cuh"
#include "quantization/w8a8/fp8/nvidia/quant_utils.cuh"
#include <torch/csrc/stable/macros.h>
#include <cuda_runtime.h>
#include <algorithm>
#include <type_traits>
namespace vllm {
namespace {
constexpr int kWarpSize = 32;
constexpr int kThreadsPerBlock = 128;
template <typename scalar_t, typename weight_t, typename rope_t, int HEAD_DIM,
bool WIDE_192_ROTARY = false, bool FP8_CACHE = false>
__global__ void qknorm_rope_cache_kernel(
const void* __restrict__ qkv_void, void* __restrict__ q_out_void,
void* __restrict__ k_out_void, void* __restrict__ v_out_void,
const void* __restrict__ q_weight_void,
const void* __restrict__ k_weight_void, const void* __restrict__ cos_void,
const void* __restrict__ sin_void, const int64_t* __restrict__ positions,
const int64_t* __restrict__ slot_mapping, void* __restrict__ k_cache_void,
void* __restrict__ v_cache_void, int64_t qkv_token_stride,
int64_t q_token_stride, int64_t k_token_stride, int64_t v_token_stride,
int64_t cos_token_stride, int64_t sin_token_stride,
int64_t cache_block_stride, int64_t cache_page_stride,
int64_t cache_head_stride, int num_tokens, int num_slots, int num_q_heads,
int num_kv_heads, int rotary_pairs, int block_size, float epsilon,
int block_size_shift, float norm_weight_bias, const float* k_scale,
const float* v_scale) {
#if !defined(__CUDA_ARCH__) || __CUDA_ARCH__ < 800
if constexpr (std::is_same_v<scalar_t, torch::headeronly::BFloat16> ||
std::is_same_v<weight_t, torch::headeronly::BFloat16> ||
std::is_same_v<rope_t, torch::headeronly::BFloat16>) {
return;
} else {
#endif
using ScalarConverter = _typeConvert<scalar_t>;
using WeightConverter = _typeConvert<weight_t>;
using RopeConverter = _typeConvert<rope_t>;
using Scalar = typename ScalarConverter::hip_type;
using Weight = typename WeightConverter::hip_type;
using Rope = typename RopeConverter::hip_type;
// Preserve the original per-head reduction and activation rounding.
static_assert(!WIDE_192_ROTARY || HEAD_DIM == 192);
constexpr int kThreadsPerHead =
WIDE_192_ROTARY ? 32
: (HEAD_DIM <= 64 ? 8 : (HEAD_DIM <= 192 ? 16 : 32));
constexpr int kValuesPerVector =
WIDE_192_ROTARY ? 8 : (HEAD_DIM == 192 ? 4 : 8);
constexpr int kValuesPerTile = kThreadsPerHead * kValuesPerVector;
constexpr int kNumVectors =
(HEAD_DIM + kValuesPerTile - 1) / kValuesPerTile;
constexpr int kValuesPerThread = kValuesPerVector * kNumVectors;
constexpr int kTileSize = kValuesPerTile * kNumVectors;
constexpr int kHeadsPerBlock = kThreadsPerBlock / kThreadsPerHead;
const auto* qkv = static_cast<const Scalar*>(qkv_void);
auto* q_out = static_cast<Scalar*>(q_out_void);
auto* k_out = static_cast<Scalar*>(k_out_void);
auto* v_out = static_cast<Scalar*>(v_out_void);
const auto* q_weight = static_cast<const Weight*>(q_weight_void);
const auto* k_weight = static_cast<const Weight*>(k_weight_void);
const auto* cos = static_cast<const Rope*>(cos_void);
const auto* sin = static_cast<const Rope*>(sin_void);
auto* k_cache = static_cast<Scalar*>(k_cache_void);
auto* v_cache = static_cast<Scalar*>(v_cache_void);
const int head_in_block = threadIdx.x / kThreadsPerHead;
const int lane = threadIdx.x % kThreadsPerHead;
const int total_heads = num_q_heads + 2 * num_kv_heads;
int token;
int local_head;
const int64_t global_head =
static_cast<int64_t>(blockIdx.x) * kHeadsPerBlock + head_in_block;
token = global_head / total_heads;
// Reuse the quotient instead of spelling this as global_head %
// total_heads. nvcc otherwise emits a second runtime integer-division
// sequence for the flattened token/head mapping.
local_head = static_cast<int>(global_head -
static_cast<int64_t>(token) * total_heads);
if (token >= num_tokens) return;
const bool is_q = local_head < num_q_heads;
const bool is_k = !is_q && local_head < num_q_heads + num_kv_heads;
const int head = is_q ? local_head
: (is_k ? local_head - num_q_heads
: local_head - num_q_heads - num_kv_heads);
int64_t input_offset = token * qkv_token_stride;
Scalar* output;
int64_t output_offset;
if (is_q) {
input_offset += static_cast<int64_t>(head) * HEAD_DIM;
output = q_out;
output_offset =
token * q_token_stride + static_cast<int64_t>(head) * HEAD_DIM;
} else if (is_k) {
input_offset += static_cast<int64_t>(num_q_heads + head) * HEAD_DIM;
output = k_out;
output_offset =
token * k_token_stride + static_cast<int64_t>(head) * HEAD_DIM;
} else {
input_offset +=
static_cast<int64_t>(num_q_heads + num_kv_heads + head) * HEAD_DIM;
output = v_out;
output_offset =
token * v_token_stride + static_cast<int64_t>(head) * HEAD_DIM;
}
bool write_cache = false;
int64_t cache_offset = 0;
if (!is_q && token < num_slots) {
const int64_t slot = slot_mapping[token];
if (slot >= 0) {
int64_t block;
int64_t page;
if (block_size_shift >= 0) {
// vLLM production cache blocks are powers of two. Avoid the costly
// runtime int64 division on that path while retaining generic block
// sizes through the fallback below.
block = slot >> block_size_shift;
page = slot & (block_size - 1);
} else {
block = slot / block_size;
page = slot - block * block_size;
}
cache_offset = block * cache_block_stride + page * cache_page_stride +
static_cast<int64_t>(head) * cache_head_stride;
write_cache = true;
}
}
if (!is_q && !is_k) {
#pragma unroll
for (int block = 0; block < kNumVectors; ++block) {
#pragma unroll
for (int element = 0; element < kValuesPerVector; ++element) {
const int dim = block * kThreadsPerHead * kValuesPerVector +
lane * kValuesPerVector + element;
if constexpr (kTileSize > HEAD_DIM) {
if (dim >= HEAD_DIM) continue;
}
const Scalar value = qkv[input_offset + dim];
output[output_offset + dim] = value;
if (write_cache) {
if constexpr (FP8_CACHE) {
reinterpret_cast<uint8_t*>(v_cache_void)[cache_offset + dim] =
fp8::scaled_convert<uint8_t, float,
Fp8KVCacheDataType::kFp8E4M3>(
ScalarConverter::convert(value), *v_scale);
} else {
v_cache[cache_offset + dim] = value;
}
}
}
}
return;
}
float values[kValuesPerThread];
float sum_squares = 0.0f;
#pragma unroll
for (int block = 0; block < kNumVectors; ++block) {
#pragma unroll
for (int element = 0; element < kValuesPerVector; ++element) {
const int value_index = block * kValuesPerVector + element;
const int dim = block * kThreadsPerHead * kValuesPerVector +
lane * kValuesPerVector + element;
float value = 0.0f;
if constexpr (kTileSize > HEAD_DIM) {
if (dim < HEAD_DIM) {
value = ScalarConverter::convert(qkv[input_offset + dim]);
}
} else {
value = ScalarConverter::convert(qkv[input_offset + dim]);
}
values[value_index] = value;
// CuTeDSL lowers its fragment reduction to a sequential fma.rn
// chain. Keep this explicit: changing it changes RMSNorm bits.
sum_squares = __fmaf_rn(value, value, sum_squares);
}
}
const int warp_lane = threadIdx.x % kWarpSize;
const int subwarp_base = (warp_lane / kThreadsPerHead) * kThreadsPerHead;
constexpr uint32_t kSubwarpBits =
kThreadsPerHead == kWarpSize ? 0xffffffffu
: ((uint32_t{1} << kThreadsPerHead) - 1);
const uint32_t subwarp_mask = kSubwarpBits << subwarp_base;
#pragma unroll
for (int offset = 1; offset < kThreadsPerHead; offset <<= 1) {
// CuTeDSL's warp_reduce uses ascending XOR offsets (1, 2, 4, ...).
sum_squares +=
__shfl_xor_sync(subwarp_mask, sum_squares, offset, kWarpSize);
}
const float mean_square =
__fdiv_rn(sum_squares, static_cast<float>(HEAD_DIM));
const float inverse_rms = rsqrtf(mean_square + epsilon);
const Weight* weight = is_q ? q_weight : k_weight;
// CuTeDSL writes RMSNorm results to activation-typed shared memory and
// reloads them for RoPE. This intermediate FP16/BF16 rounding boundary
// is intentional and is necessary for bitwise output parity.
__shared__ Scalar normalized[kHeadsPerBlock * HEAD_DIM];
Scalar* normalized_head = normalized + head_in_block * HEAD_DIM;
#pragma unroll
for (int block = 0; block < kNumVectors; ++block) {
#pragma unroll
for (int element = 0; element < kValuesPerVector; ++element) {
const int value_index = block * kValuesPerVector + element;
const int dim = block * kThreadsPerHead * kValuesPerVector +
lane * kValuesPerVector + element;
if constexpr (kTileSize > HEAD_DIM) {
if (dim >= HEAD_DIM) continue;
}
const float x_hat = values[value_index] * inverse_rms;
const float norm_weight =
WeightConverter::convert(weight[dim]) + norm_weight_bias;
normalized_head[dim] = ScalarConverter::convert(x_hat * norm_weight);
}
}
__syncwarp(subwarp_mask);
// Apply NeoX-style RoPE. CuTeDSL assigns four rotary pairs to each
// participating lane and reloads activation-typed normalized values.
constexpr int kRotaryPairsPerThread = 4;
if (lane < rotary_pairs / kRotaryPairsPerThread) {
const int64_t position = positions[token];
const Rope* token_cos = cos + position * cos_token_stride;
const Rope* token_sin = sin + position * sin_token_stride;
#pragma unroll
for (int element = 0; element < kRotaryPairsPerThread; ++element) {
const int pair = lane * kRotaryPairsPerThread + element;
const int dim0 = pair;
const int dim1 = pair + rotary_pairs;
const float value0 = ScalarConverter::convert(normalized_head[dim0]);
const float value1 = ScalarConverter::convert(normalized_head[dim1]);
const float cos_value = RopeConverter::convert(token_cos[pair]);
const float sin_value = RopeConverter::convert(token_sin[pair]);
// Match CuTeDSL PTX exactly. The first half is mul + mul + sub; the
// second is one mul followed by one fused multiply-add.
const float rotated0 = __fsub_rn(__fmul_rn(value0, cos_value),
__fmul_rn(value1, sin_value));
const float rotated1 =
__fmaf_rn(value0, sin_value, __fmul_rn(value1, cos_value));
normalized_head[dim0] = ScalarConverter::convert(rotated0);
normalized_head[dim1] = ScalarConverter::convert(rotated1);
}
}
__syncwarp(subwarp_mask);
#pragma unroll
for (int block = 0; block < kNumVectors; ++block) {
#pragma unroll
for (int element = 0; element < kValuesPerVector; ++element) {
const int dim = block * kThreadsPerHead * kValuesPerVector +
lane * kValuesPerVector + element;
if constexpr (kTileSize > HEAD_DIM) {
if (dim >= HEAD_DIM) continue;
}
const Scalar value = normalized_head[dim];
output[output_offset + dim] = value;
if (write_cache) {
if constexpr (FP8_CACHE) {
reinterpret_cast<uint8_t*>(k_cache_void)[cache_offset + dim] =
fp8::scaled_convert<uint8_t, float,
Fp8KVCacheDataType::kFp8E4M3>(
ScalarConverter::convert(value), *k_scale);
} else {
k_cache[cache_offset + dim] = value;
}
}
}
}
#if !defined(__CUDA_ARCH__) || __CUDA_ARCH__ < 800
}
#endif
}
template <typename scalar_t, typename weight_t, typename rope_t, bool FP8_CACHE>
void launch_qknorm_rope_cache(
const torch::stable::Tensor& qkv, torch::stable::Tensor& q_out,
torch::stable::Tensor& k_out, torch::stable::Tensor& v_out,
const torch::stable::Tensor& q_weight,
const torch::stable::Tensor& k_weight, const torch::stable::Tensor& cos,
const torch::stable::Tensor& sin, const torch::stable::Tensor& positions,
const torch::stable::Tensor& slot_mapping, torch::stable::Tensor& k_cache,
torch::stable::Tensor& v_cache, int head_dim, int num_q_heads,
int num_kv_heads, int rotary_pairs, float epsilon, float norm_weight_bias,
cudaStream_t stream, const float* k_scale, const float* v_scale) {
const int total_heads = num_q_heads + 2 * num_kv_heads;
const int64_t total_work = qkv.size(0) * total_heads;
const int block_size = static_cast<int>(k_cache.size(1));
const int block_size_shift =
(block_size & (block_size - 1)) == 0
? __builtin_ctz(static_cast<unsigned int>(block_size))
: -1;
#define LAUNCH_QKNORM_ROPE_CACHE(HEAD_DIM, WIDE_192_ROTARY) \
do { \
constexpr int threads_per_head = \
WIDE_192_ROTARY ? 32 \
: (HEAD_DIM <= 64 ? 8 : (HEAD_DIM <= 192 ? 16 : 32)); \
constexpr int heads_per_block = kThreadsPerBlock / threads_per_head; \
const dim3 grid((total_work + heads_per_block - 1) / heads_per_block); \
const dim3 block(kThreadsPerBlock); \
qknorm_rope_cache_kernel<scalar_t, weight_t, rope_t, HEAD_DIM, \
WIDE_192_ROTARY, FP8_CACHE> \
<<<grid, block, 0, stream>>>( \
qkv.data_ptr(), q_out.data_ptr(), k_out.data_ptr(), \
v_out.data_ptr(), q_weight.data_ptr(), k_weight.data_ptr(), \
cos.data_ptr(), sin.data_ptr(), \
reinterpret_cast<const int64_t*>(positions.data_ptr()), \
reinterpret_cast<const int64_t*>(slot_mapping.data_ptr()), \
k_cache.data_ptr(), v_cache.data_ptr(), qkv.stride(0), \
q_out.stride(0), k_out.stride(0), v_out.stride(0), cos.stride(0), \
sin.stride(0), k_cache.stride(0), k_cache.stride(1), \
k_cache.stride(2), static_cast<int>(qkv.size(0)), \
static_cast<int>(slot_mapping.numel()), num_q_heads, num_kv_heads, \
rotary_pairs, block_size, epsilon, block_size_shift, \
norm_weight_bias, k_scale, v_scale); \
} while (false)
switch (head_dim) {
case 64:
LAUNCH_QKNORM_ROPE_CACHE(64, false);
break;
case 128:
LAUNCH_QKNORM_ROPE_CACHE(128, false);
break;
case 192:
if (rotary_pairs > 64) {
LAUNCH_QKNORM_ROPE_CACHE(192, true);
} else {
LAUNCH_QKNORM_ROPE_CACHE(192, false);
}
break;
case 256:
LAUNCH_QKNORM_ROPE_CACHE(256, false);
break;
default:
STD_TORCH_CHECK(false, "Unsupported head_dim: ", head_dim);
}
#undef LAUNCH_QKNORM_ROPE_CACHE
}
void check_cuda_tensor(const torch::stable::Tensor& tensor, const char* name) {
STD_TORCH_CHECK(tensor.is_cuda(), name, " must be a CUDA tensor");
}
void check_same_device(const torch::stable::Tensor& reference,
const torch::stable::Tensor& tensor, const char* name) {
STD_TORCH_CHECK(tensor.get_device_index() == reference.get_device_index(),
name, " must be on the same CUDA device as qkv");
}
} // namespace
void fused_qknorm_rope_cache(
torch::stable::Tensor& q_out, torch::stable::Tensor& k_out,
torch::stable::Tensor& v_out, torch::stable::Tensor& qkv,
torch::stable::Tensor& q_weight, torch::stable::Tensor& k_weight,
torch::stable::Tensor& cos, torch::stable::Tensor& sin,
torch::stable::Tensor& positions, torch::stable::Tensor& slot_mapping,
torch::stable::Tensor& k_cache, torch::stable::Tensor& v_cache,
int64_t head_dim, int64_t num_q_heads, int64_t num_kv_heads,
int64_t rotary_pairs, double epsilon, double norm_weight_bias,
const std::string& kv_cache_dtype,
std::optional<torch::stable::Tensor> k_scale,
std::optional<torch::stable::Tensor> v_scale) {
check_cuda_tensor(qkv, "qkv");
check_cuda_tensor(q_out, "q_out");
check_cuda_tensor(k_out, "k_out");
check_cuda_tensor(v_out, "v_out");
check_cuda_tensor(q_weight, "q_weight");
check_cuda_tensor(k_weight, "k_weight");
check_cuda_tensor(cos, "cos");
check_cuda_tensor(sin, "sin");
check_cuda_tensor(positions, "positions");
check_cuda_tensor(slot_mapping, "slot_mapping");
check_cuda_tensor(k_cache, "k_cache");
check_cuda_tensor(v_cache, "v_cache");
check_same_device(qkv, q_out, "q_out");
check_same_device(qkv, k_out, "k_out");
check_same_device(qkv, v_out, "v_out");
check_same_device(qkv, q_weight, "q_weight");
check_same_device(qkv, k_weight, "k_weight");
check_same_device(qkv, cos, "cos");
check_same_device(qkv, sin, "sin");
check_same_device(qkv, positions, "positions");
check_same_device(qkv, slot_mapping, "slot_mapping");
check_same_device(qkv, k_cache, "k_cache");
check_same_device(qkv, v_cache, "v_cache");
STD_TORCH_CHECK(qkv.dim() == 2 && qkv.stride(1) == 1,
"qkv must be a 2D tensor contiguous in its last dimension");
STD_TORCH_CHECK(q_out.dim() == 2 && q_out.is_contiguous(),
"q_out must be contiguous and 2D");
STD_TORCH_CHECK(k_out.dim() == 2 && k_out.is_contiguous(),
"k_out must be contiguous and 2D");
STD_TORCH_CHECK(v_out.dim() == 2 && v_out.is_contiguous(),
"v_out must be contiguous and 2D");
STD_TORCH_CHECK(q_weight.dim() == 1 && q_weight.is_contiguous(),
"q_weight must be contiguous and 1D");
STD_TORCH_CHECK(k_weight.dim() == 1 && k_weight.is_contiguous(),
"k_weight must be contiguous and 1D");
STD_TORCH_CHECK(cos.dim() == 2 && cos.stride(1) == 1,
"cos must be 2D and contiguous in its last dimension");
STD_TORCH_CHECK(sin.dim() == 2 && sin.stride(1) == 1,
"sin must be 2D and contiguous in its last dimension");
STD_TORCH_CHECK(
positions.scalar_type() == torch::headeronly::ScalarType::Long,
"positions must be an int64 tensor");
STD_TORCH_CHECK(positions.is_contiguous(), "positions must be contiguous");
STD_TORCH_CHECK(
slot_mapping.scalar_type() == torch::headeronly::ScalarType::Long,
"slot_mapping must be an int64 tensor");
STD_TORCH_CHECK(slot_mapping.is_contiguous(),
"slot_mapping must be contiguous");
STD_TORCH_CHECK(k_cache.dim() == 4 && v_cache.dim() == 4,
"K/V cache must have shape [blocks, block_size, heads, dim]");
for (int64_t d = 0; d < 4; ++d) {
STD_TORCH_CHECK(k_cache.size(d) == v_cache.size(d) &&
k_cache.stride(d) == v_cache.stride(d),
"K/V cache must have identical shapes and strides");
}
STD_TORCH_CHECK(k_cache.stride(3) == 1,
"K/V cache head dimension must be contiguous");
STD_TORCH_CHECK(num_q_heads > 0 && num_kv_heads > 0,
"Q/KV head counts must be positive");
STD_TORCH_CHECK(
head_dim == 64 || head_dim == 128 || head_dim == 192 || head_dim == 256,
"Unsupported head_dim");
STD_TORCH_CHECK(cos.size(0) == sin.size(0),
"cos/sin cache must have the same number of positions");
const int64_t num_tokens = qkv.size(0);
const int64_t packed_width = (num_q_heads + 2 * num_kv_heads) * head_dim;
STD_TORCH_CHECK(qkv.size(1) == packed_width,
"qkv width must equal (num_q_heads + 2 * num_kv_heads) * "
"head_dim");
STD_TORCH_CHECK(
q_out.size(0) == num_tokens && q_out.size(1) == num_q_heads * head_dim,
"q_out has an invalid shape");
STD_TORCH_CHECK(
k_out.size(0) == num_tokens && k_out.size(1) == num_kv_heads * head_dim,
"k_out has an invalid shape");
STD_TORCH_CHECK(
v_out.size(0) == k_out.size(0) && v_out.size(1) == k_out.size(1),
"v_out has an invalid shape");
STD_TORCH_CHECK(q_weight.numel() == head_dim && k_weight.numel() == head_dim,
"Q/K norm weights must have head_dim elements");
STD_TORCH_CHECK(positions.numel() == num_tokens,
"positions must have one entry per qkv token");
STD_TORCH_CHECK(rotary_pairs >= 0 && rotary_pairs * 2 <= head_dim &&
rotary_pairs % 4 == 0,
"rotary_pairs must be a multiple of 4 in [0, head_dim / 2]");
STD_TORCH_CHECK(cos.size(1) >= rotary_pairs && sin.size(1) >= rotary_pairs,
"cos/sin cache is smaller than rotary_pairs");
STD_TORCH_CHECK(
k_cache.size(2) == num_kv_heads && k_cache.size(3) == head_dim,
"K/V cache head shape does not match the kernel arguments");
STD_TORCH_CHECK(k_cache.size(1) > 0, "K/V cache block_size must be positive");
STD_TORCH_CHECK(slot_mapping.numel() <= num_tokens,
"slot_mapping cannot contain more entries than qkv tokens");
STD_TORCH_CHECK(qkv.scalar_type() == q_out.scalar_type() &&
qkv.scalar_type() == k_out.scalar_type() &&
qkv.scalar_type() == v_out.scalar_type(),
"qkv and Q/K/V outputs must have the same dtype");
const bool fp8_cache =
kv_cache_dtype == "fp8" || kv_cache_dtype == "fp8_e4m3";
STD_TORCH_CHECK(fp8_cache || kv_cache_dtype == "auto" ||
kv_cache_dtype == "bfloat16" ||
kv_cache_dtype == "float16",
"Unsupported KV cache dtype: ", kv_cache_dtype);
STD_TORCH_CHECK(k_cache.scalar_type() == v_cache.scalar_type(),
"K/V caches must have the same dtype");
if (fp8_cache) {
STD_TORCH_CHECK(
k_cache.scalar_type() == torch::headeronly::ScalarType::Byte ||
k_cache.scalar_type() ==
torch::headeronly::ScalarType::Float8_e4m3fn,
"FP8 cache requires uint8 or float8_e4m3fn storage");
STD_TORCH_CHECK(k_scale.has_value() && v_scale.has_value(),
"FP8 cache requires K/V scales");
for (const auto* scale : {&*k_scale, &*v_scale}) {
check_cuda_tensor(*scale, "KV scale");
check_same_device(qkv, *scale, "KV scale");
STD_TORCH_CHECK(
scale->numel() == 1 &&
scale->scalar_type() == torch::headeronly::ScalarType::Float,
"KV scales must be single FP32 values");
}
} else {
STD_TORCH_CHECK(qkv.scalar_type() == k_cache.scalar_type(),
"Unquantized cache must match QKV dtype");
}
STD_TORCH_CHECK(q_weight.scalar_type() == k_weight.scalar_type(),
"Q/K norm weights must have the same dtype");
STD_TORCH_CHECK(cos.scalar_type() == sin.scalar_type(),
"cos and sin must have the same dtype");
if (num_tokens == 0) return;
const torch::stable::accelerator::DeviceGuard device_guard(
qkv.get_device_index());
const cudaStream_t stream = get_current_cuda_stream(qkv.get_device_index());
VLLM_STABLE_DISPATCH_HALF_TYPES(
qkv.scalar_type(), "qknorm_rope_cache_input", [&] {
using input_t = scalar_t;
VLLM_STABLE_DISPATCH_FLOATING_TYPES(
q_weight.scalar_type(), "qknorm_rope_cache_weight", [&] {
using norm_weight_t = scalar_t;
VLLM_STABLE_DISPATCH_FLOATING_TYPES(
cos.scalar_type(), "qknorm_rope_cache_rope", [&] {
using rope_cache_t = scalar_t;
auto launch = [&](auto fp8_tag) {
launch_qknorm_rope_cache<input_t, norm_weight_t,
rope_cache_t,
decltype(fp8_tag)::value>(
qkv, q_out, k_out, v_out, q_weight, k_weight, cos,
sin, positions, slot_mapping, k_cache, v_cache,
static_cast<int>(head_dim),
static_cast<int>(num_q_heads),
static_cast<int>(num_kv_heads),
static_cast<int>(rotary_pairs),
static_cast<float>(epsilon),
static_cast<float>(norm_weight_bias), stream,
fp8_cache
? static_cast<const float*>(k_scale->data_ptr())
: nullptr,
fp8_cache
? static_cast<const float*>(v_scale->data_ptr())
: nullptr);
};
if (fp8_cache) {
launch(std::true_type{});
} else {
launch(std::false_type{});
}
});
});
});
STD_CUDA_KERNEL_LAUNCH_CHECK();
}
} // namespace vllm
+100
View File
@@ -0,0 +1,100 @@
// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright contributors to the vLLM project
#include <cuda_runtime.h>
#include <torch/csrc/stable/accelerator.h>
#include <torch/csrc/stable/tensor.h>
#include <torch/headeronly/core/ScalarType.h>
#include <algorithm>
#include "dispatch_utils.h"
#include "quantization/vectorization_utils.cuh"
#include "torch_utils.h"
namespace vllm {
// 16 B of output per thread; low-precision inputs are widened in registers.
static constexpr int kMuseSoftCapVecSize = 4;
// blockIdx.x walks rows, blockIdx.y splits a row. Rows are the sampled tokens
// (often one, at decode) while a row is the whole vocab, so splitting the row
// is what keeps the GPU busy; a row-per-block grid would leave a single-token
// decode on one SM.
template <typename scalar_t>
__global__ void muse_spark_mul_tanh_soft_cap_kernel(
float* __restrict__ out, const scalar_t* __restrict__ input, int d,
float multiplier, float soft_cap_temp) {
const int64_t row_offset = static_cast<int64_t>(blockIdx.x) * d;
const scalar_t* __restrict__ in_row = input + row_offset;
float* __restrict__ out_row = out + row_offset;
const int tid = blockIdx.y * blockDim.x + threadIdx.x;
const int stride = gridDim.y * blockDim.x;
vectorize_with_alignment<kMuseSoftCapVecSize>(
in_row, out_row, d, tid, stride,
[=] __device__(float& dst, const scalar_t& src) {
// Multiply then divide, as MulTanhSoftCap.forward does, so fp32
// rounding matches the eager path.
dst = soft_cap_temp *
tanhf(static_cast<float>(src) * multiplier / soft_cap_temp);
});
}
} // namespace vllm
// out = soft_cap_temp * tanh(input * multiplier / soft_cap_temp), fp32.
void muse_spark_mul_tanh_soft_cap(torch::stable::Tensor& out,
torch::stable::Tensor& input,
double multiplier, double soft_cap_temp) {
STD_TORCH_CHECK(
input.scalar_type() == torch::headeronly::ScalarType::Float ||
input.scalar_type() == torch::headeronly::ScalarType::Half ||
input.scalar_type() == torch::headeronly::ScalarType::BFloat16,
"muse_spark_mul_tanh_soft_cap: input must be float32, float16 or "
"bfloat16");
STD_TORCH_CHECK(out.scalar_type() == torch::headeronly::ScalarType::Float,
"muse_spark_mul_tanh_soft_cap: out must be float32");
STD_TORCH_CHECK(input.is_contiguous(),
"muse_spark_mul_tanh_soft_cap: input must be contiguous");
STD_TORCH_CHECK(out.is_contiguous(),
"muse_spark_mul_tanh_soft_cap: out must be contiguous");
STD_TORCH_CHECK(out.numel() == input.numel(),
"muse_spark_mul_tanh_soft_cap: out and input must have the "
"same number of elements");
STD_TORCH_CHECK(
soft_cap_temp != 0.0,
"muse_spark_mul_tanh_soft_cap: soft_cap_temp must be nonzero");
if (input.numel() == 0) {
return;
}
const int d = static_cast<int>(input.size(input.dim() - 1));
const int64_t num_tokens = input.numel() / d;
constexpr int kThreads = 128;
const int64_t vecs_per_row =
(d + vllm::kMuseSoftCapVecSize - 1) / vllm::kMuseSoftCapVecSize;
const int64_t blocks_to_cover = (vecs_per_row + kThreads - 1) / kThreads;
constexpr int64_t kTargetBlocks = 32768;
const int64_t blocks_per_row = std::clamp<int64_t>(
kTargetBlocks / num_tokens, 1, std::max<int64_t>(blocks_to_cover, 1));
dim3 grid(static_cast<unsigned>(num_tokens),
static_cast<unsigned>(blocks_per_row));
dim3 block(kThreads);
const torch::stable::accelerator::DeviceGuard device_guard(
input.get_device_index());
const cudaStream_t stream = get_current_cuda_stream();
VLLM_STABLE_DISPATCH_FLOATING_TYPES(
input.scalar_type(), "muse_spark_mul_tanh_soft_cap", [&] {
vllm::muse_spark_mul_tanh_soft_cap_kernel<scalar_t>
<<<grid, block, 0, stream>>>(out.mutable_data_ptr<float>(),
input.const_data_ptr<scalar_t>(), d,
static_cast<float>(multiplier),
static_cast<float>(soft_cap_temp));
});
}
+33 -4
View File
@@ -363,6 +363,35 @@ minimax_allreduce_rms_qk(torch::stable::Tensor qkv,
int64_t const nranks, double const eps); int64_t const nranks, double const eps);
#endif #endif
#ifndef USE_ROCM
namespace vllm {
void fused_qknorm_rope_cache(
torch::stable::Tensor& q_out, torch::stable::Tensor& k_out,
torch::stable::Tensor& v_out, torch::stable::Tensor& qkv,
torch::stable::Tensor& q_weight, torch::stable::Tensor& k_weight,
torch::stable::Tensor& cos, torch::stable::Tensor& sin,
torch::stable::Tensor& positions, torch::stable::Tensor& slot_mapping,
torch::stable::Tensor& k_cache, torch::stable::Tensor& v_cache,
int64_t head_dim, int64_t num_q_heads, int64_t num_kv_heads,
int64_t rotary_pairs, double epsilon, double norm_weight_bias,
const std::string& kv_cache_dtype,
std::optional<torch::stable::Tensor> k_scale,
std::optional<torch::stable::Tensor> v_scale);
} // namespace vllm
#endif
void fused_muse_spark_qknorm_rope_kv_insert(
torch::stable::Tensor& qkv, torch::stable::Tensor const& positions,
int64_t num_heads, int64_t num_kv_heads, int64_t head_dim, double eps,
std::optional<torch::stable::Tensor> cos_sin_cache, int64_t rotary_dim,
std::optional<torch::stable::Tensor> slot_mapping,
std::optional<torch::stable::Tensor> kv_cache, int64_t block_size,
const std::string& kv_cache_dtype,
std::optional<torch::stable::Tensor> k_scale,
std::optional<torch::stable::Tensor> v_scale,
std::optional<torch::stable::Tensor> q_fp8_out,
std::optional<torch::stable::Tensor> q_scale);
// Horizontally-fused MiniMax-M3 QK-norm + partial NeoX RoPE (+ optional KV / // Horizontally-fused MiniMax-M3 QK-norm + partial NeoX RoPE (+ optional KV /
// index-cache insert). Dense layer: norm+RoPE only; sparse layer: also packs // index-cache insert). Dense layer: norm+RoPE only; sparse layer: also packs
// the index branch and scatters k/v/index_k into their paged caches. // the index branch and scatters k/v/index_k into their paged caches.
@@ -546,6 +575,10 @@ std::tuple<int64_t, torch::stable::Tensor> allocate_shared_buffer_and_handle(
int64_t open_mem_handle(torch::stable::Tensor& mem_handle); int64_t open_mem_handle(torch::stable::Tensor& mem_handle);
void free_shared_buffer(int64_t buffer); void free_shared_buffer(int64_t buffer);
void muse_spark_mul_tanh_soft_cap(torch::stable::Tensor& out,
torch::stable::Tensor& input,
double multiplier, double soft_cap_temp);
// Activation kernels (shared CUDA/ROCm) // Activation kernels (shared CUDA/ROCm)
void silu_and_mul(torch::stable::Tensor& out, torch::stable::Tensor& input); void silu_and_mul(torch::stable::Tensor& out, torch::stable::Tensor& input);
void silu_and_mul_clamp(torch::stable::Tensor& out, void silu_and_mul_clamp(torch::stable::Tensor& out,
@@ -579,10 +612,6 @@ void situ_and_mul_quant(
int64_t group_size = 0, int64_t group_size = 0,
std::optional<torch::stable::Tensor> num_valid_tokens = std::nullopt, std::optional<torch::stable::Tensor> num_valid_tokens = std::nullopt,
int64_t topk = 1); int64_t topk = 1);
void masked_situ_and_mul(torch::stable::Tensor& out,
torch::stable::Tensor& input,
const torch::stable::Tensor& expert_num_tokens,
double beta = 1.0, double linear_beta = -1.0);
void masked_moe_activation(torch::stable::Tensor& out, void masked_moe_activation(torch::stable::Tensor& out,
torch::stable::Tensor& input, torch::stable::Tensor& input,
const torch::stable::Tensor& valid_token_counts, const torch::stable::Tensor& valid_token_counts,
@@ -135,6 +135,13 @@ hadamard_transform_kernel(b16* a, b16* out, int total_num_chunks) {
#pragma unroll #pragma unroll
for (int64_t k = 0; k < num_chunks; k++) { for (int64_t k = 0; k < num_chunks; k++) {
// The last warp may own fewer chunks than num_chunks: loading the
// rest would read past the end of `a`. The cp.async.wait_group counts
// below already assume only real_num_chunks groups were committed.
if constexpr(enable_mask) {
if (k >= real_num_chunks)
break;
}
size_t shared_ptr = __cvta_generic_to_shared(b_frag_ptr); size_t shared_ptr = __cvta_generic_to_shared(b_frag_ptr);
#if (__CUDA_ARCH__ >= 900) // SM90 #if (__CUDA_ARCH__ >= 900) // SM90
asm volatile( asm volatile(
+31 -4
View File
@@ -501,6 +501,27 @@ STABLE_TORCH_LIBRARY_FRAGMENT(_C, ops) {
"float eps) -> (Tensor, Tensor)"); "float eps) -> (Tensor, Tensor)");
#endif #endif
#ifndef USE_ROCM
ops.def(
"fused_qknorm_rope_cache("
"Tensor! q_out, Tensor! k_out, Tensor! v_out, Tensor qkv, "
"Tensor q_weight, Tensor k_weight, Tensor cos, Tensor sin, "
"Tensor positions, Tensor slot_mapping, Tensor! k_cache, "
"Tensor! v_cache, int head_dim, int num_q_heads, int num_kv_heads, "
"int rotary_pairs, float epsilon, float norm_weight_bias, "
"str kv_cache_dtype=\"auto\", Tensor? k_scale=None, Tensor? "
"v_scale=None) -> ()");
#endif
ops.def(
"fused_muse_spark_qknorm_rope_kv_insert("
"Tensor! qkv, Tensor positions, int num_heads, int num_kv_heads, "
"int head_dim, float eps, Tensor? cos_sin_cache, int rotary_dim, "
"Tensor? slot_mapping, Tensor!? kv_cache, int block_size, "
"str kv_cache_dtype, Tensor? k_scale=None, Tensor? v_scale=None, "
"Tensor!? q_fp8_out=None, Tensor? q_scale=None) "
"-> ()");
// Horizontally-fused MiniMax-M3 QK-norm + partial NeoX RoPE + KV-insert. // Horizontally-fused MiniMax-M3 QK-norm + partial NeoX RoPE + KV-insert.
ops.def( ops.def(
"fused_minimax_m3_qknorm_rope_kv_insert(" "fused_minimax_m3_qknorm_rope_kv_insert("
@@ -598,6 +619,10 @@ STABLE_TORCH_LIBRARY_FRAGMENT(_C, ops) {
"y_q, Tensor! y_s, bool use_ue8m0) -> ()"); "y_q, Tensor! y_s, bool use_ue8m0) -> ()");
ops.def("weak_ref_tensor(Tensor input) -> Tensor"); ops.def("weak_ref_tensor(Tensor input) -> Tensor");
ops.def(
"muse_spark_mul_tanh_soft_cap(Tensor! out, Tensor input, "
"float multiplier, float soft_cap_temp) -> ()");
// Activation function used in SwiGLU. // Activation function used in SwiGLU.
ops.def("silu_and_mul(Tensor! result, Tensor input) -> ()"); ops.def("silu_and_mul(Tensor! result, Tensor input) -> ()");
@@ -640,9 +665,6 @@ STABLE_TORCH_LIBRARY_FRAGMENT(_C, ops) {
"situ_and_mul_quant(Tensor! out, Tensor! scale, Tensor input, " "situ_and_mul_quant(Tensor! out, Tensor! scale, Tensor input, "
"float beta=1.0, float linear_beta=-1.0, int group_size=0, " "float beta=1.0, float linear_beta=-1.0, int group_size=0, "
"Tensor? num_valid_tokens=None, int topk=1) -> ()"); "Tensor? num_valid_tokens=None, int topk=1) -> ()");
ops.def(
"masked_situ_and_mul(Tensor! out, Tensor input, Tensor "
"expert_num_tokens, float beta=1.0, float linear_beta=-1.0) -> ()");
ops.def( ops.def(
"masked_moe_activation(Tensor! out, Tensor input, Tensor " "masked_moe_activation(Tensor! out, Tensor input, Tensor "
"valid_token_counts, str activation, float clamp_limit=0.0, float " "valid_token_counts, str activation, float clamp_limit=0.0, float "
@@ -831,7 +853,11 @@ STABLE_TORCH_LIBRARY_IMPL(_C, CUDA, ops) {
TORCH_BOX(&fused_kimi_k3_mla_decode_q_concat_ds_mla_insert)); TORCH_BOX(&fused_kimi_k3_mla_decode_q_concat_ds_mla_insert));
#ifndef USE_ROCM #ifndef USE_ROCM
ops.impl("minimax_allreduce_rms_qk", TORCH_BOX(&minimax_allreduce_rms_qk)); ops.impl("minimax_allreduce_rms_qk", TORCH_BOX(&minimax_allreduce_rms_qk));
ops.impl("fused_qknorm_rope_cache",
TORCH_BOX(&vllm::fused_qknorm_rope_cache));
#endif #endif
ops.impl("fused_muse_spark_qknorm_rope_kv_insert",
TORCH_BOX(&fused_muse_spark_qknorm_rope_kv_insert));
ops.impl("fused_minimax_m3_qknorm_rope_kv_insert", ops.impl("fused_minimax_m3_qknorm_rope_kv_insert",
TORCH_BOX(&fused_minimax_m3_qknorm_rope_kv_insert)); TORCH_BOX(&fused_minimax_m3_qknorm_rope_kv_insert));
#ifdef VLLM_ENABLE_FUSED_KDA_DECODE #ifdef VLLM_ENABLE_FUSED_KDA_DECODE
@@ -867,6 +893,8 @@ STABLE_TORCH_LIBRARY_IMPL(_C, CUDA, ops) {
TORCH_BOX(&persistent_masked_m_silu_mul_quant)); TORCH_BOX(&persistent_masked_m_silu_mul_quant));
ops.impl("weak_ref_tensor", TORCH_BOX(&weak_ref_tensor)); ops.impl("weak_ref_tensor", TORCH_BOX(&weak_ref_tensor));
ops.impl("silu_and_mul_quant", TORCH_BOX(&silu_and_mul_quant)); ops.impl("silu_and_mul_quant", TORCH_BOX(&silu_and_mul_quant));
ops.impl("muse_spark_mul_tanh_soft_cap",
TORCH_BOX(&muse_spark_mul_tanh_soft_cap));
ops.impl("silu_and_mul", TORCH_BOX(&silu_and_mul)); ops.impl("silu_and_mul", TORCH_BOX(&silu_and_mul));
ops.impl("mul_and_silu", TORCH_BOX(&mul_and_silu)); ops.impl("mul_and_silu", TORCH_BOX(&mul_and_silu));
ops.impl("gelu_and_mul", TORCH_BOX(&gelu_and_mul)); ops.impl("gelu_and_mul", TORCH_BOX(&gelu_and_mul));
@@ -875,7 +903,6 @@ STABLE_TORCH_LIBRARY_IMPL(_C, CUDA, ops) {
ops.impl("swigluoai_and_mul", TORCH_BOX(&swigluoai_and_mul)); ops.impl("swigluoai_and_mul", TORCH_BOX(&swigluoai_and_mul));
ops.impl("situ_and_mul", TORCH_BOX(&situ_and_mul)); ops.impl("situ_and_mul", TORCH_BOX(&situ_and_mul));
ops.impl("situ_and_mul_quant", TORCH_BOX(&situ_and_mul_quant)); ops.impl("situ_and_mul_quant", TORCH_BOX(&situ_and_mul_quant));
ops.impl("masked_situ_and_mul", TORCH_BOX(&masked_situ_and_mul));
ops.impl("masked_moe_activation", TORCH_BOX(&masked_moe_activation)); ops.impl("masked_moe_activation", TORCH_BOX(&masked_moe_activation));
ops.impl("gelu_new", TORCH_BOX(&gelu_new)); ops.impl("gelu_new", TORCH_BOX(&gelu_new));
ops.impl("gelu_fast", TORCH_BOX(&gelu_fast)); ops.impl("gelu_fast", TORCH_BOX(&gelu_fast));
+5 -1
View File
@@ -502,7 +502,11 @@ RUN --mount=type=cache,target=/opt/uv/cache \
> /tmp/nccl-override.txt && \ > /tmp/nccl-override.txt && \
export UV_OVERRIDE=/tmp/nccl-override.txt; \ export UV_OVERRIDE=/tmp/nccl-override.txt; \
fi && \ fi && \
export TORCH_CUDA_ARCH_LIST='9.0a 10.0a' && \ DEEPEP_ARCH_LIST='9.0a 10.0a' && \
if [ "$(printf '%s\n' 12.9 "$CUDA_VERSION" | sort -V | head -n1)" = 12.9 ]; then \
DEEPEP_ARCH_LIST="$DEEPEP_ARCH_LIST 10.3a"; \
fi && \
export TORCH_CUDA_ARCH_LIST="$DEEPEP_ARCH_LIST" && \
/tmp/install_python_libraries.sh \ /tmp/install_python_libraries.sh \
--workspace /tmp/ep_kernels_workspace \ --workspace /tmp/ep_kernels_workspace \
--mode wheel \ --mode wheel \
-453
View File
@@ -1,453 +0,0 @@
ARG BASE_IMAGE=ubuntu:22.04
ARG PYTHON_VERSION=3.12
ARG TORCH_VERSION=2.12.0+rocm10.0.0
ARG TORCHVISION_VERSION=0.27.0+rocm10.0.0
ARG TORCHAUDIO_VERSION=2.11.0+rocm10.0.0
ARG TRITON_BRANCH="669b31ac" # release/internal/3.8.x as of 09/29/2026
ARG TRITON_VERSION=3.8.0+git669b31ac
ARG TRITON_REPO="https://github.com/ROCm/triton.git"
ARG FA_BRANCH="0e60e394"
ARG FA_REPO="https://github.com/Dao-AILab/flash-attention.git"
ARG AITER_BRANCH="v0.1.25"
ARG AITER_REPO="https://github.com/ROCm/aiter.git"
ARG MORI_BRANCH="v1.2.3.post1"
ARG MORI_REPO="https://github.com/ROCm/mori.git"
ARG ROCM_SYSTEMS_REPO="https://github.com/ROCm/rocm-systems.git"
ARG ROCM_RUNTIME_COMMIT="25e14349f2606a6b95c4ccd45eb92a1835646cf2"
# TheRock configuration
ARG SITE_PACKAGES=/usr/local/lib/python${PYTHON_VERSION}/dist-packages
ARG ROCM_RELEASE_WHEELS_MULTIARCH_URL="https://stable.repo.amd.com/rocm/whl-next/" # For pre-built GPU images, packages, etc.
ARG ROCM_NIGHTLY_WHEELS_MULTIARCH_URL="https://nightly.repo.amd.com/rocm/whl-next/" # For torch, triton, etc. wheels with latest fixes.
ARG ROCM_SDK_VERSION=10.0.0
# Build configuration
ARG PYTORCH_ROCM_ARCH="gfx90a;gfx942;gfx950;gfx1100;gfx1101;gfx1200;gfx1201;gfx1150;gfx1151"
ARG AITER_ROCM_ARCH="gfx942;gfx950"
# Sccache configuration (only used in release pipeline)
ARG USE_SCCACHE
ARG SCCACHE_DOWNLOAD_URL
ARG SCCACHE_ENDPOINT
ARG SCCACHE_BUCKET_NAME=vllm-build-sccache
ARG SCCACHE_REGION_NAME=us-west-2
ARG SCCACHE_S3_NO_CREDENTIALS=0
FROM ${BASE_IMAGE} AS base
ARG PYTORCH_ROCM_ARCH
ENV PYTORCH_ROCM_ARCH="${PYTORCH_ROCM_ARCH}"
ARG AITER_ROCM_ARCH
ENV AITER_ROCM_ARCH="${AITER_ROCM_ARCH}"
ARG SITE_PACKAGES
ENV SITE_PACKAGES=${SITE_PACKAGES}
ENV SDK_DEV="$SITE_PACKAGES/_rocm_sdk_devel" \
SDK_CORE="$SITE_PACKAGES/_rocm_sdk_core"
ENV ROCM_PATH="$SDK_DEV" \
ROCM_HOME="$SDK_DEV" \
HIP_PATH="$SDK_DEV" \
HIP_CLANG_PATH="$SDK_DEV/lib/llvm/bin" \
HIP_DEVICE_LIB_PATH="$SDK_CORE/lib/llvm/amdgcn/bitcode"
ENV LD_LIBRARY_PATH="$SDK_DEV/lib:$SDK_DEV/lib/rocm_sysdeps/lib:$SDK_CORE/lib:$SDK_CORE/lib/rocm_sysdeps/lib"
ENV PATH="$SDK_DEV/bin:$SDK_DEV/lib/llvm/bin:$SDK_CORE/bin:$PATH"
# Required for RCCL in ROCm7.1
ENV HSA_NO_SCRATCH_RECLAIM=1
ARG PYTHON_VERSION
ENV PYTHON_VERSION=${PYTHON_VERSION}
RUN mkdir -p /app
WORKDIR /app
ENV DEBIAN_FRONTEND=noninteractive
# Install Python and other dependencies
RUN apt-get update -y \
&& apt-get install -y software-properties-common build-essential git curl sudo vim less libgfortran5 libopenmpi-dev libpci-dev liblzma-dev pkg-config libdrm-dev libnuma-dev kmod \
&& for i in 1 2 3; do \
add-apt-repository -y ppa:deadsnakes/ppa && break || \
{ echo "Attempt $i failed, retrying in 5s..."; sleep 5; }; \
done \
&& apt-get update -y \
&& apt-get install -y python${PYTHON_VERSION} python${PYTHON_VERSION}-dev python${PYTHON_VERSION}-venv \
python${PYTHON_VERSION}-lib2to3 python-is-python3 \
&& update-alternatives --install /usr/bin/python3 python3 /usr/bin/python${PYTHON_VERSION} 1 \
&& update-alternatives --set python3 /usr/bin/python${PYTHON_VERSION} \
&& ln -sf /usr/bin/python${PYTHON_VERSION}-config /usr/bin/python3-config \
&& curl -sS https://bootstrap.pypa.io/get-pip.py | python${PYTHON_VERSION} \
&& python3 --version && python3 -m pip --version
RUN pip install -U packaging 'cmake<4' ninja wheel 'setuptools<80' pybind11==3.0.4 Cython
RUN apt-get update && apt-get install -y libjpeg-dev libsox-dev libsox-fmt-all sox && rm -rf /var/lib/apt/lists/*
# Install ROCm SDK via TheRock
ARG ROCM_RELEASE_WHEELS_MULTIARCH_URL
ARG ROCM_NIGHTLY_WHEELS_MULTIARCH_URL
ARG ROCM_SDK_VERSION
RUN pip install --index-url ${ROCM_RELEASE_WHEELS_MULTIARCH_URL} \
--extra-index-url ${ROCM_NIGHTLY_WHEELS_MULTIARCH_URL} \
"rocm[libraries,devel]${ROCM_SDK_VERSION:+==${ROCM_SDK_VERSION}}" \
&& rocm-sdk init
# Install PyTorch wheels (nightly for latest torch, release as fallback)
ARG TORCH_VERSION
ARG TORCHAUDIO_VERSION
ARG TORCHVISION_VERSION
RUN pip install \
--index-url ${ROCM_RELEASE_WHEELS_MULTIARCH_URL} \
--extra-index-url ${ROCM_NIGHTLY_WHEELS_MULTIARCH_URL} \
"torch==${TORCH_VERSION}" \
"torchaudio==${TORCHAUDIO_VERSION}" \
"torchvision==${TORCHVISION_VERSION}"
# Install the AMD-compiled device kernels corresponding to PYTORCH_ROCM_ARCH:
# Possible values that can be in PYTORCH_ROCM_ARCH:
#
# gfx90a (MI210/250), gfx942 (MI300), gfx950 (MI350), gfx1100 (7900XTX),
# gfx1101 (7800XT), gfx1200 (9070XT), gfx1201 (9070), gfx1150/1151 (Strix APU)
# For each arch in PYTORCH_ROCM_ARCH, install the precompiled GPU code that
# PyTorch needs to run on that specific chip.
SHELL ["/bin/bash", "-c"]
RUN TORCH_VER=$(python3 -c "import torch; print(torch.__version__)") \
&& for arch in $(echo "${PYTORCH_ROCM_ARCH}" | tr ';' ' '); do \
echo "Installing device packages for ${arch}" \
&& pip install --index-url ${ROCM_RELEASE_WHEELS_MULTIARCH_URL} \
--extra-index-url ${ROCM_NIGHTLY_WHEELS_MULTIARCH_URL} \
amd-torch-device-${arch}==${TORCH_VER} \
rocm-sdk-device-${arch}==${ROCM_SDK_VERSION}; \
done
# Remove gfx90a xnack kpacks (ASAN artifacts that cause hipErrorInvalidImage on MI250)
RUN rm -f "${SITE_PACKAGES}/torch/.kpack/torch_gfx90a:xnack+.kpack" \
"${SITE_PACKAGES}/torch/.kpack/torch_gfx90a:xnack-.kpack"
RUN cd $SITE_PACKAGES/_rocm_sdk_core/share/amd_smi && pip install .
ENV PYTHONPATH=$SITE_PACKAGES/_rocm_sdk_core/share/amd_smi
# TheRock runtime fixes
RUN if [ -f "${SITE_PACKAGES}/rocm_sdk/__init__.py" ]; then \
sed -i 's/rtld_global: bool = True/rtld_global: bool = False/g' "${SITE_PACKAGES}/rocm_sdk/__init__.py"; \
fi \
&& if [ -f "${SITE_PACKAGES}/_rocm_sdk_core/lib/librocm_smi64.so.1" ] \
&& [ ! -e "${ROCM_PATH}/lib/librocm_smi64.so.7" ]; then \
ln -sfn "${SITE_PACKAGES}/_rocm_sdk_core/lib/librocm_smi64.so.1" "${ROCM_PATH}/lib/librocm_smi64.so.7"; \
fi \
&& printf '%s\n%s\n' "${ROCM_PATH}/lib" "${SITE_PACKAGES}/_rocm_sdk_core/lib" > /etc/ld.so.conf.d/rocm-pip.conf \
&& for f in ${SITE_PACKAGES}/_rocm_sdk_core/lib/libamdhip64.so*; do \
bn=$(basename "$f"); \
if [ -f "${ROCM_PATH}/lib/${bn}" ] && [ -f "$f" ] && [ ! -L "$f" ]; then \
rm -f "$f"; \
ln -sfn "${ROCM_PATH}/lib/${bn}" "$f"; \
fi; \
done \
&& for f in librocprofiler-sdk.so.1 librocprofiler-register.so.0 \
librocprofiler-sdk-roctx.so.1; do \
if [ -e "${SITE_PACKAGES}/_rocm_sdk_core/lib/${f}" ] \
&& [ -e "${SITE_PACKAGES}/_rocm_sdk_devel/lib/${f}" ]; then \
rm -f "${SITE_PACKAGES}/_rocm_sdk_core/lib/${f}"; \
ln -sfn "${SITE_PACKAGES}/_rocm_sdk_devel/lib/${f}" \
"${SITE_PACKAGES}/_rocm_sdk_core/lib/${f}"; \
fi; \
done \
&& ldconfig
# Install sccache if USE_SCCACHE is enabled (for release builds)
ARG USE_SCCACHE
ARG SCCACHE_DOWNLOAD_URL
ARG SCCACHE_ENDPOINT
ARG SCCACHE_BUCKET_NAME
ARG SCCACHE_REGION_NAME
ARG SCCACHE_S3_NO_CREDENTIALS
RUN if [ "$USE_SCCACHE" = "1" ]; then \
echo "Installing sccache..." \
&& SCCACHE_ARCH="x86_64" \
&& SCCACHE_VERSION="v0.8.1" \
&& SCCACHE_DL_URL="${SCCACHE_DOWNLOAD_URL:-https://github.com/mozilla/sccache/releases/download/${SCCACHE_VERSION}/sccache-${SCCACHE_VERSION}-${SCCACHE_ARCH}-unknown-linux-musl.tar.gz}" \
&& curl -L -o /tmp/sccache.tar.gz ${SCCACHE_DL_URL} \
&& tar -xzf /tmp/sccache.tar.gz -C /tmp \
&& mv /tmp/sccache-${SCCACHE_VERSION}-${SCCACHE_ARCH}-unknown-linux-musl/sccache /usr/bin/sccache \
&& chmod +x /usr/bin/sccache \
&& rm -rf /tmp/sccache.tar.gz /tmp/sccache-${SCCACHE_VERSION}-${SCCACHE_ARCH}-unknown-linux-musl \
&& sccache --version; \
fi
# Setup sccache for HIP compilation via HIP_CLANG_PATH
# This creates wrapper scripts in a separate directory and points HIP to use them
# This avoids modifying the original ROCm binaries which can break detection
# NOTE: HIP_CLANG_PATH is NOT set as ENV to avoid affecting downstream images (Dockerfile.rocm)
# Instead, each build stage should export HIP_CLANG_PATH=/opt/sccache-wrappers if USE_SCCACHE=1
RUN if [ "$USE_SCCACHE" = "1" ]; then \
echo "Setting up sccache wrappers for HIP compilation..." \
&& mkdir -p /opt/sccache-wrappers \
&& printf '#!/bin/bash\nexec sccache %s/lib/llvm/bin/clang++ "$@"\n' "${ROCM_PATH}" > /opt/sccache-wrappers/clang++ \
&& chmod +x /opt/sccache-wrappers/clang++ \
&& printf '#!/bin/bash\nexec sccache %s/lib/llvm/bin/clang "$@"\n' "${ROCM_PATH}" > /opt/sccache-wrappers/clang \
&& chmod +x /opt/sccache-wrappers/clang \
&& echo "sccache wrappers created in /opt/sccache-wrappers"; \
fi
ENV SCCACHE_BUCKET=${USE_SCCACHE:+${SCCACHE_BUCKET_NAME}}
ENV SCCACHE_REGION=${USE_SCCACHE:+${SCCACHE_REGION_NAME}}
ENV SCCACHE_S3_NO_CREDENTIALS=${USE_SCCACHE:+${SCCACHE_S3_NO_CREDENTIALS}}
ENV SCCACHE_IDLE_TIMEOUT=${USE_SCCACHE:+0}
###
### MORI Build
###
FROM base AS build_mori
ARG MORI_BRANCH
ARG MORI_REPO
RUN git clone ${MORI_REPO}
RUN cd mori \
&& git checkout ${MORI_BRANCH} \
&& git submodule update --init --recursive \
&& ln -sfn ${ROCM_PATH} /opt/rocm \
&& pip install "setuptools_scm>=6.2" \
&& CMAKE_PREFIX_PATH="${ROCM_PATH}:${ROCM_PATH}/lib/rocm_sysdeps" \
python3 setup.py bdist_wheel --dist-dir=dist && ls /app/mori/dist/*.whl
RUN mkdir -p /app/install && cp /app/mori/dist/*.whl /app/install
###
### FlashAttention Build
###
FROM base AS build_fa
ARG FA_BRANCH
ARG FA_REPO
ARG USE_SCCACHE
RUN git clone ${FA_REPO}
RUN cd flash-attention \
&& git checkout ${FA_BRANCH} \
&& git submodule update --init \
&& if [ "$USE_SCCACHE" = "1" ]; then \
export HIP_CLANG_PATH=/opt/sccache-wrappers \
&& sccache --show-stats; \
fi \
&& GPU_ARCHS=$(echo ${PYTORCH_ROCM_ARCH} | sed -e 's/;gfx1[0-9]\{3\}//g') python3 setup.py bdist_wheel --dist-dir=dist \
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi
RUN mkdir -p /app/install && cp /app/flash-attention/dist/*.whl /app/install
###
### AITER Build
###
FROM base AS build_aiter
ARG AITER_BRANCH
ARG AITER_REPO
ARG USE_SCCACHE
RUN git clone --recursive --branch ${AITER_BRANCH} ${AITER_REPO}
RUN cd aiter \
&& git submodule update --init --recursive \
&& pip install -r requirements.txt
# AITER_USE_SYSTEM_TRITON keeps setup.py from resolving its own triton, which would
# displace the pinned one already installed in base.
RUN pip install pyyaml && cd aiter \
&& if [ "$USE_SCCACHE" = "1" ]; then \
export HIP_CLANG_PATH=/opt/sccache-wrappers \
&& sccache --show-stats; \
fi \
&& PREBUILD_KERNELS=1 AITER_USE_SYSTEM_TRITON=1 GPU_ARCHS=${AITER_ROCM_ARCH} python3 setup.py bdist_wheel --dist-dir=dist \
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi \
&& ls /app/aiter/dist/*.whl
RUN mkdir -p /app/install && cp /app/aiter/dist/*.whl /app/install
###
### Triton Build
###
FROM base AS build_triton
ARG TRITON_BRANCH
ARG TRITON_REPO
RUN git clone ${TRITON_REPO}
RUN cd triton \
&& git checkout ${TRITON_BRANCH} \
&& if [ ! -f setup.py ]; then cd python; fi \
&& python3 setup.py bdist_wheel --dist-dir=dist \
&& mkdir -p /app/install && cp dist/*.whl /app/install
###
### ROCr + CLR build
###
# Brings in multi-stream perf improvements; drop after these arrive in an official release
FROM base AS build_rocm_runtime
ARG ROCM_SYSTEMS_REPO
ARG ROCM_RUNTIME_COMMIT
# libgl-dev: rocclr does find_package(OpenGL REQUIRED) unconditionally and the wheel image has
# no GL headers. The link must not pick up a libGL DT_NEEDED, which is asserted after the build.
RUN apt-get update \
&& apt-get install -y --no-install-recommends g++ libdw-dev xxd libgl-dev libglx-dev \
&& rm -rf /var/lib/apt/lists/*
RUN pip install --no-cache-dir CppHeaderParser
RUN git init -q /src && cd /src \
&& git remote add origin ${ROCM_SYSTEMS_REPO} \
&& git config core.sparseCheckout true \
&& git config remote.origin.promisor true \
&& git config remote.origin.partialclonefilter blob:none \
&& git sparse-checkout init --cone \
&& git sparse-checkout set projects/rocr-runtime projects/clr projects/hip shared cmake \
&& git fetch --filter=blob:none --depth 1 origin ${ROCM_RUNTIME_COMMIT} \
&& git checkout -q FETCH_HEAD
# libelf is taken from the SDK's own vendored sysdeps rather than apt -- the wheel image has no
# /usr/include/libelf.h -- so ROCr links against the copy the wheel itself was built against.
ENV SYSDEPS="$SDK_DEV/lib/rocm_sysdeps"
# ROCr first, and installed into $SDK_DEV, because the clr build below needs its headers.
RUN cd /src/projects/rocr-runtime \
&& cmake -B build -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_PREFIX_PATH="$SDK_DEV;$SYSDEPS" \
-DCMAKE_INSTALL_PREFIX="$SDK_DEV" \
-DLIBELF_INCLUDE_DIRS="$SYSDEPS/include" \
-DLIBELF_LIBRARIES="$SYSDEPS/lib/libelf.so" \
&& cmake --build build --parallel 16 \
&& cmake --install build \
&& cmake --install build --prefix /rocr-install --strip
RUN cd /src/projects/clr \
&& cmake -B build -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCLR_BUILD_HIP=ON \
-DCLR_BUILD_OCL=OFF \
-DROCM_KPACK_ENABLED=ON \
-DHIP_COMMON_DIR=/src/projects/hip \
-DROCM_PATH="$SDK_DEV" \
-DCMAKE_PREFIX_PATH="$SDK_DEV;$SYSDEPS" \
-DCMAKE_INSTALL_PREFIX="$SDK_DEV" \
-DHIP_LLVM_ROOT="$SDK_DEV/lib/llvm" \
&& cmake --build build --parallel 16 \
&& cmake --install build --prefix /clr-install --strip
RUN mkdir -p /staging/lib \
&& cp -P /rocr-install/lib/libhsa-runtime64.so* /staging/lib/ \
&& cp -P /clr-install/lib/libamdhip64.so* /staging/lib/ \
&& ls -l /staging/lib
###
### Final Build
###
# Wheel release stage -
# only includes dependencies used by wheel release pipeline
FROM base AS debs_wheel_release
RUN mkdir /app/debs
RUN --mount=type=bind,from=build_triton,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_fa,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_aiter,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
FROM base AS debs
RUN mkdir /app/debs
RUN --mount=type=bind,from=build_triton,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_fa,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_aiter,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_mori,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
FROM base AS final
ARG TRITON_VERSION
# torch 2.12 implicitly attaches rocprofiler-sdk; its queue interposition hangs at HSA
# teardown, so disable it. TODO: drop once ROCm's stock rocprofiler carries the fix.
ENV ROCPROFILER_QUEUE_INTERPOSITION=0
COPY --from=build_rocm_runtime /staging/lib/ ${SDK_DEV}/lib/
# Hard link the patched library onto every stock file name, in both SDK trees, so that the
# SONAME lookup and the hard coded paths all resolve to it. Hard links rather than symlinks
# because ldconfig rewrites SONAME symlinks but cannot re-point an inode. $SDK_CORE keeps its
# own libhsa-runtime64 copy, which is on LD_LIBRARY_PATH and in ld.so.conf; relinking it is
# belt and braces rather than strictly required, but it leaves no stock ROCr in the image.
RUN set -eux; \
cd "$SDK_DEV/lib"; \
ours_hsa="$(readlink -f libhsa-runtime64.so.1)"; \
ours_hip="$(readlink -f libamdhip64.so.7)"; \
for f in "$SDK_DEV"/lib/libhsa-runtime64.so.1* "$SDK_CORE"/lib/libhsa-runtime64.so.1* \
"$SDK_DEV"/lib/libamdhip64.so.7* "$SDK_CORE"/lib/libamdhip64.so.7*; do \
[ -f "$f" ] && [ ! -L "$f" ] || continue; \
case "${f##*/}" in libhsa*) ours="$ours_hsa";; *) ours="$ours_hip";; esac; \
[ "$f" -ef "$ours" ] || ln -f "$ours" "$f"; \
done
RUN ldconfig
# This is to prevent resolvers from checking METADATA and RECORD and clobbering
# the Triton source build we installed. Constraints files are unable to help
# with this. However, these entries are optional and can be blanked which
# solves the problem.
RUN TORCH_DIST_INFO=$(ls -d ${SITE_PACKAGES}/torch-*.dist-info) \
&& grep -q '^Requires-Dist: triton' "$TORCH_DIST_INFO/METADATA" \
&& sed -i '/^Requires-Dist: triton/d' "$TORCH_DIST_INFO/METADATA" \
&& sed -i "s|^\(${TORCH_DIST_INFO##*/}/METADATA\),.*|\1,,|" "$TORCH_DIST_INFO/RECORD"
RUN --mount=type=bind,from=debs,src=/app/debs,target=/install \
pip install /install/*.whl \
&& ! pip check 2>&1 | grep -iE '^(torch|triton) ' \
&& test "$(pip show triton | awk '/^Version:/ {print $2}')" = "${TRITON_VERSION}" \
# Create a constraints file to protect the versions we need, especially
# triton, requirements will try to reinstall repeatedly.
&& pip list --format=freeze \
| grep -E '^(torch|triton|torchaudio|torchvision|pybind11)==' > /etc/rocm-constraints.txt \
&& echo "ROCm constraints:" && cat /etc/rocm-constraints.txt \
&& test "$(wc -l < /etc/rocm-constraints.txt)" -eq 5 \
&& grep -q '^pybind11==3\.0\.' /etc/rocm-constraints.txt \
&& python3 -c "import torch; assert torch.version.hip, f'non-ROCm torch: {torch.__version__}'; print(f'torch={torch.__version__}')" \
&& python3 -c "import triton; print(f'triton={triton.__version__}')" \
&& pip list 2>/dev/null | grep -iE '^(torch|triton) '
ARG BASE_IMAGE
ARG PYTHON_VERSION
ARG ROCM_SDK_VERSION
ARG TORCH_VERSION
ARG TORCHAUDIO_VERSION
ARG TORCHVISION_VERSION
ARG PYTORCH_ROCM_ARCH
ARG AITER_ROCM_ARCH
ARG TRITON_BRANCH
ARG TRITON_REPO
ARG FA_BRANCH
ARG FA_REPO
ARG AITER_BRANCH
ARG AITER_REPO
ARG MORI_BRANCH
ARG MORI_REPO
ARG ROCM_SYSTEMS_REPO
ARG ROCM_RUNTIME_COMMIT
ARG TRITON_VERSION
# Record hashes for Triton and pybind11 so we can verify later.
RUN echo "BASE_IMAGE: ${BASE_IMAGE}" > /app/versions.txt \
&& echo "PYTHON_VERSION: ${PYTHON_VERSION}" >> /app/versions.txt \
&& echo "ROCM_SDK_VERSION: ${ROCM_SDK_VERSION}" >> /app/versions.txt \
&& echo "TORCH_VERSION: ${TORCH_VERSION}" >> /app/versions.txt \
&& echo "TORCHAUDIO_VERSION: ${TORCHAUDIO_VERSION}" >> /app/versions.txt \
&& echo "TORCHVISION_VERSION: ${TORCHVISION_VERSION}" >> /app/versions.txt \
&& echo "PYTORCH_ROCM_ARCH: ${PYTORCH_ROCM_ARCH}" >> /app/versions.txt \
&& echo "AITER_ROCM_ARCH: ${AITER_ROCM_ARCH}" >> /app/versions.txt \
&& echo "TRITON_BRANCH: ${TRITON_BRANCH}" >> /app/versions.txt \
&& echo "TRITON_REPO: ${TRITON_REPO}" >> /app/versions.txt \
&& echo "FA_BRANCH: ${FA_BRANCH}" >> /app/versions.txt \
&& echo "FA_REPO: ${FA_REPO}" >> /app/versions.txt \
&& echo "AITER_BRANCH: ${AITER_BRANCH}" >> /app/versions.txt \
&& echo "AITER_REPO: ${AITER_REPO}" >> /app/versions.txt \
&& echo "MORI_BRANCH: ${MORI_BRANCH}" >> /app/versions.txt \
&& echo "MORI_REPO: ${MORI_REPO}" >> /app/versions.txt \
&& echo "ROCM_SYSTEMS_REPO: ${ROCM_SYSTEMS_REPO}" >> /app/versions.txt \
&& echo "ROCM_RUNTIME_COMMIT: ${ROCM_RUNTIME_COMMIT}" >> /app/versions.txt \
# Fingerprint our triton so Dockerfile.rocm can prove it survived the build.
&& echo "TRITON_VERSION: $(pip show triton | awk '/^Version:/ {print $2}')" >> /app/versions.txt \
&& echo "TRITON_SHA256: $(find $(python3 -c 'import triton, os; print(os.path.dirname(triton.__file__))') \
-type f \( -name '*.py' -o -name '*.so' \) | sort | xargs sha256sum | sha256sum | cut -d' ' -f1)" \
>> /app/versions.txt \
&& echo "PYBIND11_VERSION: $(pip show pybind11 | awk '/^Version:/ {print $2}')" >> /app/versions.txt \
&& echo "PYBIND11_INTERNALS_VERSION: $(grep -rhE '^#[[:space:]]*define PYBIND11_INTERNALS_VERSION' \
$(python3 -c 'import pybind11; print(pybind11.get_include())') | awk '{print $NF}')" \
>> /app/versions.txt \
&& echo "PYBIND11_SHA256: $(find $(python3 -c 'import pybind11; print(pybind11.get_include())') \
-type f -name '*.h' | sort | xargs sha256sum | sha256sum | cut -d' ' -f1)" \
>> /app/versions.txt
+41 -41
View File
@@ -280,8 +280,9 @@ COPY --from=fetch_vllm \
${COMMON_WORKDIR}/vllm/requirements/rocm.txt \ ${COMMON_WORKDIR}/vllm/requirements/rocm.txt \
requirements/ requirements/
# Use constraints file to protect triton, pybind11, and torch dependencies.
RUN --mount=type=cache,id=vllm-rocm-uv,target=/root/.cache/uv \ RUN --mount=type=cache,id=vllm-rocm-uv,target=/root/.cache/uv \
uv pip install --system -r requirements/rocm.txt uv pip install --system -c /etc/rocm-constraints.txt -r requirements/rocm.txt
# Fetch only the pinned ROCm Triton kernels package. Keeping this before the # Fetch only the pinned ROCm Triton kernels package. Keeping this before the
# vLLM source COPYs makes the download reusable when native sources change. # vLLM source COPYs makes the download reusable when native sources change.
@@ -322,8 +323,9 @@ ENV SETUPTOOLS_SCM_PRETEND_VERSION="0.0.0+rocm.csrc.build"
RUN --mount=type=bind,source=pyproject.toml,target=${COMMON_WORKDIR}/vllm/pyproject.toml \ RUN --mount=type=bind,source=pyproject.toml,target=${COMMON_WORKDIR}/vllm/pyproject.toml \
--mount=type=cache,id=vllm-rocm-ccache,target=/root/.cache/ccache \ --mount=type=cache,id=vllm-rocm-ccache,target=/root/.cache/ccache \
grep -Fq "set(TRITON_KERNELS_TAG \"${ROCM_TRITON_KERNELS_COMMIT}\")" \ { grep -Fq "set(TRITON_KERNELS_TAG \"${ROCM_TRITON_KERNELS_COMMIT}\")" \
cmake/external_projects/triton_kernels.cmake \ cmake/external_projects/triton_kernels.cmake \
|| echo "WARNING: Dockerfile.rocm ROCM_TRITON_KERNELS_COMMIT (${ROCM_TRITON_KERNELS_COMMIT}) does not match cmake TRITON_KERNELS_TAG; the Docker build uses the Dockerfile pin, cmake governs only non-Docker builds"; } \
&& export CCACHE_BASEDIR="$PWD" \ && export CCACHE_BASEDIR="$PWD" \
&& echo "=== ccache stats before ROCm native build ===" \ && echo "=== ccache stats before ROCm native build ===" \
&& (ccache --show-stats || true) \ && (ccache --show-stats || true) \
@@ -366,7 +368,8 @@ COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/requirements /requirements
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/benchmarks /benchmarks COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/benchmarks /benchmarks
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/tests /tests COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/tests /tests
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/examples /examples COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/examples /examples
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/tools /tools COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/tools/benchmark_helion_kernels.py /tools/benchmark_helion_kernels.py
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/tools/install_torchcodec_rocm.sh /tools/install_torchcodec_rocm.sh
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile /docker/Dockerfile COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile /docker/Dockerfile
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile.cpu /docker/Dockerfile.cpu COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile.cpu /docker/Dockerfile.cpu
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm /docker/ COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm /docker/
@@ -386,7 +389,6 @@ ARG NIXL_BRANCH="1683cf3b7f3d11674c03c5e861cea22339876c96"
ARG NIXL_REPO="https://github.com/ai-dynamo/nixl.git" ARG NIXL_REPO="https://github.com/ai-dynamo/nixl.git"
ARG UCX_BRANCH="96e58a16039f6d7d213bc967b8069238742c5194" ARG UCX_BRANCH="96e58a16039f6d7d213bc967b8069238742c5194"
ARG UCX_REPO="https://github.com/openucx/ucx.git" ARG UCX_REPO="https://github.com/openucx/ucx.git"
ENV ROCM_PATH=/opt/rocm
ENV UCX_HOME=/usr/local/ucx ENV UCX_HOME=/usr/local/ucx
ENV NIXL_HOME=/usr/local/nixl ENV NIXL_HOME=/usr/local/nixl
@@ -541,7 +543,7 @@ RUN --mount=type=tmpfs,target=/tmp/lmcache-wheel \
c_ops="$(find /tmp/lmcache-wheel -type f -name 'c_ops*.so')"; \ c_ops="$(find /tmp/lmcache-wheel -type f -name 'c_ops*.so')"; \
test "$(find /tmp/lmcache-wheel -type f -name 'c_ops*.so' | wc -l)" -eq 1; \ test "$(find /tmp/lmcache-wheel -type f -name 'c_ops*.so' | wc -l)" -eq 1; \
expected="$(printf '%s' "${LMCACHE_ROCM_ARCH}" | tr ';' '\n' | sort -u)"; \ expected="$(printf '%s' "${LMCACHE_ROCM_ARCH}" | tr ';' '\n' | sort -u)"; \
actual="$(/opt/rocm/llvm/bin/llvm-objdump --offloading "${c_ops}" \ actual="$(llvm-objdump --offloading "${c_ops}" \
| grep -oE 'gfx[0-9a-z]+' | sort -u)"; \ | grep -oE 'gfx[0-9a-z]+' | sort -u)"; \
if [ "${actual}" != "${expected}" ]; then \ if [ "${actual}" != "${expected}" ]; then \
printf 'LMCache code objects mismatch\nexpected:\n%s\nactual:\n%s\n' \ printf 'LMCache code objects mismatch\nexpected:\n%s\nactual:\n%s\n' \
@@ -549,32 +551,10 @@ RUN --mount=type=tmpfs,target=/tmp/lmcache-wheel \
exit 1; \ exit 1; \
fi fi
# ROCShmem build stage - split from DeepEP so changing DEEPEP_BRANCH does not # DeepEP build stage. ROCShmem is provided by TheRock SDK
# invalidate the slow ROCShmem build. # ROCSHMEM_DIR set to the SDK install path so DeepEP finds the headers, etc.
FROM base AS build_rocshmem FROM base AS build_deepep
ARG ROCSHMEM_BRANCH="f0acb0c6" ENV ROCSHMEM_DIR=${ROCM_PATH}
ARG ROCSHMEM_REPO="https://github.com/ROCm/rocm-systems.git"
# DeepEP only supports gfx942 and gfx950; build ROCShmem for the same set so
# it can be linked against DeepEP without arch mismatches.
ARG DEEPEP_ROCM_ARCH="gfx942;gfx950"
ENV ROCM_PATH=/opt/rocm
ENV ROCSHMEM_DIR=/opt/rocshmem
RUN --mount=type=cache,target=/root/.cache/ccache \
git clone --no-checkout --filter=blob:none ${ROCSHMEM_REPO} \
&& cd rocm-systems \
&& git sparse-checkout set --cone projects/rocshmem \
&& git checkout ${ROCSHMEM_BRANCH} \
&& mkdir -p projects/rocshmem/build \
&& cd projects/rocshmem/build \
&& CC="ccache gcc" CXX="ccache g++" INSTALL_PREFIX=${ROCSHMEM_DIR} \
bash ../scripts/build_configs/all_backends \
-DROCM_PATH=${ROCM_PATH} \
-DGPU_TARGETS="${DEEPEP_ROCM_ARCH}" \
-DUSE_EXTERNAL_MPI=OFF
# DeepEP build stage - depends on ROCShmem, builds the HIP kernel wheel.
FROM build_rocshmem AS build_deepep
ARG DEEPEP_BRANCH="0f63d3e9" ARG DEEPEP_BRANCH="0f63d3e9"
ARG DEEPEP_REPO="https://github.com/ROCm/DeepEP.git" ARG DEEPEP_REPO="https://github.com/ROCm/DeepEP.git"
ARG DEEPEP_NIC="cx7" ARG DEEPEP_NIC="cx7"
@@ -693,8 +673,9 @@ RUN if [ "$GIT_REPO_CHECK" != "0" ]; then \
# Install the packaging helpers before modifying requirements/rocm.txt. # Install the packaging helpers before modifying requirements/rocm.txt.
RUN --mount=type=cache,target=/root/.cache/uv \ RUN --mount=type=cache,target=/root/.cache/uv \
grep -Fq "set(TRITON_KERNELS_TAG \"${ROCM_TRITON_KERNELS_COMMIT}\")" \ { grep -Fq "set(TRITON_KERNELS_TAG \"${ROCM_TRITON_KERNELS_COMMIT}\")" \
vllm/cmake/external_projects/triton_kernels.cmake \ vllm/cmake/external_projects/triton_kernels.cmake \
|| echo "WARNING: Dockerfile.rocm ROCM_TRITON_KERNELS_COMMIT (${ROCM_TRITON_KERNELS_COMMIT}) does not match cmake TRITON_KERNELS_TAG; the Docker build uses the Dockerfile pin, cmake governs only non-Docker builds"; } \
&& cd vllm \ && cd vllm \
&& uv pip install --system setuptools_scm regex && uv pip install --system setuptools_scm regex
@@ -750,7 +731,8 @@ COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/requirements /requir
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/benchmarks /benchmarks COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/benchmarks /benchmarks
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/tests /tests COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/tests /tests
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/examples /examples COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/examples /examples
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/tools /tools COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/tools/benchmark_helion_kernels.py /tools/benchmark_helion_kernels.py
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/tools/install_torchcodec_rocm.sh /tools/install_torchcodec_rocm.sh
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile /docker/Dockerfile COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile /docker/Dockerfile
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile.cpu /docker/Dockerfile.cpu COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile.cpu /docker/Dockerfile.cpu
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm /docker/ COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm /docker/
@@ -795,9 +777,6 @@ RUN --mount=type=bind,from=build_nixl,src=/app/install,target=/nixl_install \
--mount=type=bind,from=build_deepep,src=/app/deep_install,target=/deep_install \ --mount=type=bind,from=build_deepep,src=/app/deep_install,target=/deep_install \
uv pip install --system /nixl_install/*.whl /deep_install/*.whl uv pip install --system /nixl_install/*.whl /deep_install/*.whl
# Copy ROCShmem runtime libraries.
COPY --from=build_rocshmem /opt/rocshmem /opt/rocshmem
# RDMA userspace libraries plus FFmpeg dev libs needed by torchcodec. # RDMA userspace libraries plus FFmpeg dev libs needed by torchcodec.
RUN timeout --kill-after=30s 5m apt-get update -q -y --error-on=any \ RUN timeout --kill-after=30s 5m apt-get update -q -y --error-on=any \
-o Acquire::Retries=3 \ -o Acquire::Retries=3 \
@@ -830,7 +809,8 @@ RUN --mount=type=cache,target=/root/.cache/uv \
# Pre-install shared ROCm runtime dependencies. # Pre-install shared ROCm runtime dependencies.
COPY requirements/common.txt requirements/rocm.txt /tmp/ci-base-requirements/ COPY requirements/common.txt requirements/rocm.txt /tmp/ci-base-requirements/
RUN --mount=type=cache,target=/root/.cache/uv \ RUN --mount=type=cache,target=/root/.cache/uv \
uv pip install --system -r /tmp/ci-base-requirements/rocm.txt \ uv pip install --system -c /etc/rocm-constraints.txt \
-r /tmp/ci-base-requirements/rocm.txt \
&& rm -rf /tmp/ci-base-requirements && rm -rf /tmp/ci-base-requirements
# Enable fast and less brittle model downloads in tests. # Enable fast and less brittle model downloads in tests.
@@ -908,8 +888,6 @@ RUN mkdir src && mv vllm src/vllm
# Catch GPU<->CPU syncs in execute_model/sample_tokens during tests. # Catch GPU<->CPU syncs in execute_model/sample_tokens during tests.
ENV VLLM_GPU_SYNC_CHECK=error ENV VLLM_GPU_SYNC_CHECK=error
# Run the structural smoke test inside BuildKit, where the test image layers
# are already available. Export only a build-scoped success marker.
FROM test AS test_smoke FROM test AS test_smoke
ARG ROCM_SMOKE_ID ARG ROCM_SMOKE_ID
RUN --network=none \ RUN --network=none \
@@ -956,15 +934,37 @@ RUN --mount=type=bind,from=export_vllm,src=/,target=/install \
--mount=type=cache,target=/root/.cache/uv \ --mount=type=cache,target=/root/.cache/uv \
cd /install \ cd /install \
&& echo "opentelemetry-api<=1.40.0" > /tmp/otel-constraints.txt \ && echo "opentelemetry-api<=1.40.0" > /tmp/otel-constraints.txt \
&& uv pip install --system -c /tmp/otel-constraints.txt -r requirements/rocm.txt \ && uv pip install --system -c /etc/rocm-constraints.txt -c /tmp/otel-constraints.txt -r requirements/rocm.txt \
&& pip uninstall -y vllm \ && pip uninstall -y vllm \
&& uv pip install --system -c /tmp/otel-constraints.txt *.whl \ && uv pip install --system -c /etc/rocm-constraints.txt -c /tmp/otel-constraints.txt *.whl \
&& rm /tmp/otel-constraints.txt && rm /tmp/otel-constraints.txt
# Install NIXL ROCm wheel # Install NIXL ROCm wheel
RUN --mount=type=bind,from=build_nixl,src=/app/install,target=/nixl_install \ RUN --mount=type=bind,from=build_nixl,src=/app/install,target=/nixl_install \
uv pip install --system /nixl_install/*.whl uv pip install --system /nixl_install/*.whl
# Recompute the base's triton fingerprint to detect resolver interference.
RUN EXPECTED=$(grep '^TRITON_SHA256: ' /app/versions.txt | cut -d' ' -f2) \
&& ACTUAL=$(find $(python3 -c 'import triton, os; print(os.path.dirname(triton.__file__))') \
-type f \( -name '*.py' -o -name '*.so' \) | sort | xargs sha256sum | sha256sum | cut -d' ' -f1) \
&& echo "triton sha256 expected=${EXPECTED} actual=${ACTUAL}" \
&& test -n "${EXPECTED}" \
&& test "${EXPECTED}" = "${ACTUAL}" \
&& python3 -c "import torch; assert torch.version.hip, f'non-ROCm torch: {torch.__version__}'; print(f'torch={torch.__version__}')"
# Recompute the base's pybind11 fingerprint to detect resolver interference.
RUN EXPECTED=$(grep '^PYBIND11_SHA256: ' /app/versions.txt | cut -d' ' -f2) \
&& EXPECTED_ABI=$(grep '^PYBIND11_INTERNALS_VERSION: ' /app/versions.txt | cut -d' ' -f2) \
&& INC=$(python3 -c 'import pybind11; print(pybind11.get_include())') \
&& ACTUAL=$(find "${INC}" -type f -name '*.h' | sort | xargs sha256sum | sha256sum | cut -d' ' -f1) \
&& ACTUAL_ABI=$(grep -rhE '^#[[:space:]]*define PYBIND11_INTERNALS_VERSION' "${INC}" | awk '{print $NF}') \
&& echo "pybind11 $(pip show pybind11 | awk '/^Version:/ {print $2}')" \
"sha256 expected=${EXPECTED} actual=${ACTUAL}" \
"internals expected=${EXPECTED_ABI} actual=${ACTUAL_ABI}" \
&& test -n "${EXPECTED}" \
&& test "${EXPECTED}" = "${ACTUAL}" \
&& test "${EXPECTED_ABI}" = "${ACTUAL_ABI}"
ARG COMMON_WORKDIR ARG COMMON_WORKDIR
ARG BASE_IMAGE ARG BASE_IMAGE
ARG NIC_BACKEND ARG NIC_BACKEND
@@ -3,7 +3,7 @@ ARG REMOTE_VLLM="0"
ARG VLLM_REPO="https://github.com/vllm-project/vllm.git" ARG VLLM_REPO="https://github.com/vllm-project/vllm.git"
ARG VLLM_BRANCH="main" ARG VLLM_BRANCH="main"
ARG COMMON_WORKDIR=/app ARG COMMON_WORKDIR=/app
ARG BASE_IMAGE=rocm/vllm-dev:base ARG BASE_IMAGE=vllm/vllm-openai-rocm:base-nightly-rocm72
ARG CI_BASE_IMAGE=rocm/vllm-dev:ci_base ARG CI_BASE_IMAGE=rocm/vllm-dev:ci_base
ARG ROCM_TRITON_KERNELS_COMMIT=669b31acc1dd1b3fd93286afd5db67f65d9f7557 ARG ROCM_TRITON_KERNELS_COMMIT=669b31acc1dd1b3fd93286afd5db67f65d9f7557
ARG INSTALL_LMCACHE=false ARG INSTALL_LMCACHE=false
@@ -46,7 +46,7 @@ RUN --mount=type=cache,target=/root/.cache/pip \
# Note: mold is installed but not set as the system default linker because # Note: mold is installed but not set as the system default linker because
# some packages use JIT compilation at runtime with flags mold does not support. # some packages use JIT compilation at runtime with flags mold does not support.
# Build stages opt in via LDFLAGS="-fuse-ld=mold". # Build stages opt in via LDFLAGS="-fuse-ld=mold".
# Remove sccache only if not using sccache (it exists in base image from Dockerfile.rocm_base) # Remove sccache only if not using sccache (it exists in base image from Dockerfile.rocm_72_base)
ARG USE_SCCACHE ARG USE_SCCACHE
RUN if [ "$USE_SCCACHE" != "1" ]; then \ RUN if [ "$USE_SCCACHE" != "1" ]; then \
apt-get purge -y sccache || true; \ apt-get purge -y sccache || true; \
@@ -280,9 +280,8 @@ COPY --from=fetch_vllm \
${COMMON_WORKDIR}/vllm/requirements/rocm.txt \ ${COMMON_WORKDIR}/vllm/requirements/rocm.txt \
requirements/ requirements/
# Use constraints file to protect triton, pybind11, and torch dependencies.
RUN --mount=type=cache,id=vllm-rocm-uv,target=/root/.cache/uv \ RUN --mount=type=cache,id=vllm-rocm-uv,target=/root/.cache/uv \
uv pip install --system -c /etc/rocm-constraints.txt -r requirements/rocm.txt uv pip install --system -r requirements/rocm.txt
# Fetch only the pinned ROCm Triton kernels package. Keeping this before the # Fetch only the pinned ROCm Triton kernels package. Keeping this before the
# vLLM source COPYs makes the download reusable when native sources change. # vLLM source COPYs makes the download reusable when native sources change.
@@ -323,9 +322,8 @@ ENV SETUPTOOLS_SCM_PRETEND_VERSION="0.0.0+rocm.csrc.build"
RUN --mount=type=bind,source=pyproject.toml,target=${COMMON_WORKDIR}/vllm/pyproject.toml \ RUN --mount=type=bind,source=pyproject.toml,target=${COMMON_WORKDIR}/vllm/pyproject.toml \
--mount=type=cache,id=vllm-rocm-ccache,target=/root/.cache/ccache \ --mount=type=cache,id=vllm-rocm-ccache,target=/root/.cache/ccache \
{ grep -Fq "set(TRITON_KERNELS_TAG \"${ROCM_TRITON_KERNELS_COMMIT}\")" \ grep -Fq "set(TRITON_KERNELS_TAG \"${ROCM_TRITON_KERNELS_COMMIT}\")" \
cmake/external_projects/triton_kernels.cmake \ cmake/external_projects/triton_kernels.cmake \
|| echo "WARNING: Dockerfile.rock ROCM_TRITON_KERNELS_COMMIT (${ROCM_TRITON_KERNELS_COMMIT}) does not match cmake TRITON_KERNELS_TAG; the Docker build uses the Dockerfile pin, cmake governs only non-Docker builds"; } \
&& export CCACHE_BASEDIR="$PWD" \ && export CCACHE_BASEDIR="$PWD" \
&& echo "=== ccache stats before ROCm native build ===" \ && echo "=== ccache stats before ROCm native build ===" \
&& (ccache --show-stats || true) \ && (ccache --show-stats || true) \
@@ -368,12 +366,11 @@ COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/requirements /requirements
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/benchmarks /benchmarks COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/benchmarks /benchmarks
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/tests /tests COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/tests /tests
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/examples /examples COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/examples /examples
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/tools/benchmark_helion_kernels.py /tools/benchmark_helion_kernels.py COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/tools /tools
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/tools/install_torchcodec_rocm.sh /tools/install_torchcodec_rocm.sh
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile /docker/Dockerfile COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile /docker/Dockerfile
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile.cpu /docker/Dockerfile.cpu COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile.cpu /docker/Dockerfile.cpu
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm /docker/ COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm_72 /docker/
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm_base /docker/Dockerfile.rocm_base COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm_72_base /docker/
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/ci-rocm.hcl /docker/ci-rocm.hcl COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/ci-rocm.hcl /docker/ci-rocm.hcl
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/docker-bake.hcl /docker/docker-bake.hcl COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/docker-bake.hcl /docker/docker-bake.hcl
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/docker-bake-rocm.hcl /docker/docker-bake-rocm.hcl COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/docker-bake-rocm.hcl /docker/docker-bake-rocm.hcl
@@ -389,6 +386,7 @@ ARG NIXL_BRANCH="1683cf3b7f3d11674c03c5e861cea22339876c96"
ARG NIXL_REPO="https://github.com/ai-dynamo/nixl.git" ARG NIXL_REPO="https://github.com/ai-dynamo/nixl.git"
ARG UCX_BRANCH="96e58a16039f6d7d213bc967b8069238742c5194" ARG UCX_BRANCH="96e58a16039f6d7d213bc967b8069238742c5194"
ARG UCX_REPO="https://github.com/openucx/ucx.git" ARG UCX_REPO="https://github.com/openucx/ucx.git"
ENV ROCM_PATH=/opt/rocm
ENV UCX_HOME=/usr/local/ucx ENV UCX_HOME=/usr/local/ucx
ENV NIXL_HOME=/usr/local/nixl ENV NIXL_HOME=/usr/local/nixl
@@ -543,7 +541,7 @@ RUN --mount=type=tmpfs,target=/tmp/lmcache-wheel \
c_ops="$(find /tmp/lmcache-wheel -type f -name 'c_ops*.so')"; \ c_ops="$(find /tmp/lmcache-wheel -type f -name 'c_ops*.so')"; \
test "$(find /tmp/lmcache-wheel -type f -name 'c_ops*.so' | wc -l)" -eq 1; \ test "$(find /tmp/lmcache-wheel -type f -name 'c_ops*.so' | wc -l)" -eq 1; \
expected="$(printf '%s' "${LMCACHE_ROCM_ARCH}" | tr ';' '\n' | sort -u)"; \ expected="$(printf '%s' "${LMCACHE_ROCM_ARCH}" | tr ';' '\n' | sort -u)"; \
actual="$(llvm-objdump --offloading "${c_ops}" \ actual="$(/opt/rocm/llvm/bin/llvm-objdump --offloading "${c_ops}" \
| grep -oE 'gfx[0-9a-z]+' | sort -u)"; \ | grep -oE 'gfx[0-9a-z]+' | sort -u)"; \
if [ "${actual}" != "${expected}" ]; then \ if [ "${actual}" != "${expected}" ]; then \
printf 'LMCache code objects mismatch\nexpected:\n%s\nactual:\n%s\n' \ printf 'LMCache code objects mismatch\nexpected:\n%s\nactual:\n%s\n' \
@@ -551,10 +549,32 @@ RUN --mount=type=tmpfs,target=/tmp/lmcache-wheel \
exit 1; \ exit 1; \
fi fi
# DeepEP build stage. ROCShmem is provided by TheRock SDK # ROCShmem build stage - split from DeepEP so changing DEEPEP_BRANCH does not
# ROCSHMEM_DIR set to the SDK install path so DeepEP finds the headers, etc. # invalidate the slow ROCShmem build.
FROM base AS build_deepep FROM base AS build_rocshmem
ENV ROCSHMEM_DIR=${ROCM_PATH} ARG ROCSHMEM_BRANCH="f0acb0c6"
ARG ROCSHMEM_REPO="https://github.com/ROCm/rocm-systems.git"
# DeepEP only supports gfx942 and gfx950; build ROCShmem for the same set so
# it can be linked against DeepEP without arch mismatches.
ARG DEEPEP_ROCM_ARCH="gfx942;gfx950"
ENV ROCM_PATH=/opt/rocm
ENV ROCSHMEM_DIR=/opt/rocshmem
RUN --mount=type=cache,target=/root/.cache/ccache \
git clone --no-checkout --filter=blob:none ${ROCSHMEM_REPO} \
&& cd rocm-systems \
&& git sparse-checkout set --cone projects/rocshmem \
&& git checkout ${ROCSHMEM_BRANCH} \
&& mkdir -p projects/rocshmem/build \
&& cd projects/rocshmem/build \
&& CC="ccache gcc" CXX="ccache g++" INSTALL_PREFIX=${ROCSHMEM_DIR} \
bash ../scripts/build_configs/all_backends \
-DROCM_PATH=${ROCM_PATH} \
-DGPU_TARGETS="${DEEPEP_ROCM_ARCH}" \
-DUSE_EXTERNAL_MPI=OFF
# DeepEP build stage - depends on ROCShmem, builds the HIP kernel wheel.
FROM build_rocshmem AS build_deepep
ARG DEEPEP_BRANCH="0f63d3e9" ARG DEEPEP_BRANCH="0f63d3e9"
ARG DEEPEP_REPO="https://github.com/ROCm/DeepEP.git" ARG DEEPEP_REPO="https://github.com/ROCm/DeepEP.git"
ARG DEEPEP_NIC="cx7" ARG DEEPEP_NIC="cx7"
@@ -571,8 +591,8 @@ RUN --mount=type=cache,target=/root/.cache/ccache \
&& git checkout ${DEEPEP_BRANCH} \ && git checkout ${DEEPEP_BRANCH} \
&& LDFLAGS="-fuse-ld=mold" MAX_JOBS="${MAX_JOBS:-$(nproc)}" python3 setup.py --variant rocm --rocm-explicit-ctx --nic ${DEEPEP_NIC} --rocm-gfx942-fp8-fnuz-max 224 bdist_wheel --dist-dir=/app/deep_install && LDFLAGS="-fuse-ld=mold" MAX_JOBS="${MAX_JOBS:-$(nproc)}" python3 setup.py --variant rocm --rocm-explicit-ctx --nic ${DEEPEP_NIC} --rocm-gfx942-fp8-fnuz-max 224 bdist_wheel --dist-dir=/app/deep_install
# MoRI runtime dependencies live in Dockerfile.rocm so NIC backend changes do # MoRI runtime dependencies live in Dockerfile.rocm_72 so NIC backend changes do
# not force users to rebuild the long-lived Dockerfile.rocm_base image. # not force users to rebuild the long-lived Dockerfile.rocm_72_base image.
FROM base AS mori_base FROM base AS mori_base
ARG NIC_BACKEND ARG NIC_BACKEND
ARG AINIC_VERSION ARG AINIC_VERSION
@@ -673,9 +693,8 @@ RUN if [ "$GIT_REPO_CHECK" != "0" ]; then \
# Install the packaging helpers before modifying requirements/rocm.txt. # Install the packaging helpers before modifying requirements/rocm.txt.
RUN --mount=type=cache,target=/root/.cache/uv \ RUN --mount=type=cache,target=/root/.cache/uv \
{ grep -Fq "set(TRITON_KERNELS_TAG \"${ROCM_TRITON_KERNELS_COMMIT}\")" \ grep -Fq "set(TRITON_KERNELS_TAG \"${ROCM_TRITON_KERNELS_COMMIT}\")" \
vllm/cmake/external_projects/triton_kernels.cmake \ vllm/cmake/external_projects/triton_kernels.cmake \
|| echo "WARNING: Dockerfile.rock ROCM_TRITON_KERNELS_COMMIT (${ROCM_TRITON_KERNELS_COMMIT}) does not match cmake TRITON_KERNELS_TAG; the Docker build uses the Dockerfile pin, cmake governs only non-Docker builds"; } \
&& cd vllm \ && cd vllm \
&& uv pip install --system setuptools_scm regex && uv pip install --system setuptools_scm regex
@@ -731,12 +750,11 @@ COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/requirements /requir
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/benchmarks /benchmarks COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/benchmarks /benchmarks
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/tests /tests COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/tests /tests
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/examples /examples COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/examples /examples
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/tools/benchmark_helion_kernels.py /tools/benchmark_helion_kernels.py COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/tools /tools
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/tools/install_torchcodec_rocm.sh /tools/install_torchcodec_rocm.sh
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile /docker/Dockerfile COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile /docker/Dockerfile
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile.cpu /docker/Dockerfile.cpu COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile.cpu /docker/Dockerfile.cpu
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm /docker/ COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm_72 /docker/
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm_base /docker/Dockerfile.rocm_base COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm_72_base /docker/
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/ci-rocm.hcl /docker/ci-rocm.hcl COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/ci-rocm.hcl /docker/ci-rocm.hcl
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/docker-bake.hcl /docker/docker-bake.hcl COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/docker-bake.hcl /docker/docker-bake.hcl
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/docker-bake-rocm.hcl /docker/docker-bake-rocm.hcl COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/docker-bake-rocm.hcl /docker/docker-bake-rocm.hcl
@@ -777,6 +795,9 @@ RUN --mount=type=bind,from=build_nixl,src=/app/install,target=/nixl_install \
--mount=type=bind,from=build_deepep,src=/app/deep_install,target=/deep_install \ --mount=type=bind,from=build_deepep,src=/app/deep_install,target=/deep_install \
uv pip install --system /nixl_install/*.whl /deep_install/*.whl uv pip install --system /nixl_install/*.whl /deep_install/*.whl
# Copy ROCShmem runtime libraries.
COPY --from=build_rocshmem /opt/rocshmem /opt/rocshmem
# RDMA userspace libraries plus FFmpeg dev libs needed by torchcodec. # RDMA userspace libraries plus FFmpeg dev libs needed by torchcodec.
RUN timeout --kill-after=30s 5m apt-get update -q -y --error-on=any \ RUN timeout --kill-after=30s 5m apt-get update -q -y --error-on=any \
-o Acquire::Retries=3 \ -o Acquire::Retries=3 \
@@ -809,8 +830,7 @@ RUN --mount=type=cache,target=/root/.cache/uv \
# Pre-install shared ROCm runtime dependencies. # Pre-install shared ROCm runtime dependencies.
COPY requirements/common.txt requirements/rocm.txt /tmp/ci-base-requirements/ COPY requirements/common.txt requirements/rocm.txt /tmp/ci-base-requirements/
RUN --mount=type=cache,target=/root/.cache/uv \ RUN --mount=type=cache,target=/root/.cache/uv \
uv pip install --system -c /etc/rocm-constraints.txt \ uv pip install --system -r /tmp/ci-base-requirements/rocm.txt \
-r /tmp/ci-base-requirements/rocm.txt \
&& rm -rf /tmp/ci-base-requirements && rm -rf /tmp/ci-base-requirements
# Enable fast and less brittle model downloads in tests. # Enable fast and less brittle model downloads in tests.
@@ -888,6 +908,8 @@ RUN mkdir src && mv vllm src/vllm
# Catch GPU<->CPU syncs in execute_model/sample_tokens during tests. # Catch GPU<->CPU syncs in execute_model/sample_tokens during tests.
ENV VLLM_GPU_SYNC_CHECK=error ENV VLLM_GPU_SYNC_CHECK=error
# Run the structural smoke test inside BuildKit, where the test image layers
# are already available. Export only a build-scoped success marker.
FROM test AS test_smoke FROM test AS test_smoke
ARG ROCM_SMOKE_ID ARG ROCM_SMOKE_ID
RUN --network=none \ RUN --network=none \
@@ -934,37 +956,15 @@ RUN --mount=type=bind,from=export_vllm,src=/,target=/install \
--mount=type=cache,target=/root/.cache/uv \ --mount=type=cache,target=/root/.cache/uv \
cd /install \ cd /install \
&& echo "opentelemetry-api<=1.40.0" > /tmp/otel-constraints.txt \ && echo "opentelemetry-api<=1.40.0" > /tmp/otel-constraints.txt \
&& uv pip install --system -c /etc/rocm-constraints.txt -c /tmp/otel-constraints.txt -r requirements/rocm.txt \ && uv pip install --system -c /tmp/otel-constraints.txt -r requirements/rocm.txt \
&& pip uninstall -y vllm \ && pip uninstall -y vllm \
&& uv pip install --system -c /etc/rocm-constraints.txt -c /tmp/otel-constraints.txt *.whl \ && uv pip install --system -c /tmp/otel-constraints.txt *.whl \
&& rm /tmp/otel-constraints.txt && rm /tmp/otel-constraints.txt
# Install NIXL ROCm wheel # Install NIXL ROCm wheel
RUN --mount=type=bind,from=build_nixl,src=/app/install,target=/nixl_install \ RUN --mount=type=bind,from=build_nixl,src=/app/install,target=/nixl_install \
uv pip install --system /nixl_install/*.whl uv pip install --system /nixl_install/*.whl
# Recompute the base's triton fingerprint to detect resolver interference.
RUN EXPECTED=$(grep '^TRITON_SHA256: ' /app/versions.txt | cut -d' ' -f2) \
&& ACTUAL=$(find $(python3 -c 'import triton, os; print(os.path.dirname(triton.__file__))') \
-type f \( -name '*.py' -o -name '*.so' \) | sort | xargs sha256sum | sha256sum | cut -d' ' -f1) \
&& echo "triton sha256 expected=${EXPECTED} actual=${ACTUAL}" \
&& test -n "${EXPECTED}" \
&& test "${EXPECTED}" = "${ACTUAL}" \
&& python3 -c "import torch; assert torch.version.hip, f'non-ROCm torch: {torch.__version__}'; print(f'torch={torch.__version__}')"
# Recompute the base's pybind11 fingerprint to detect resolver interference.
RUN EXPECTED=$(grep '^PYBIND11_SHA256: ' /app/versions.txt | cut -d' ' -f2) \
&& EXPECTED_ABI=$(grep '^PYBIND11_INTERNALS_VERSION: ' /app/versions.txt | cut -d' ' -f2) \
&& INC=$(python3 -c 'import pybind11; print(pybind11.get_include())') \
&& ACTUAL=$(find "${INC}" -type f -name '*.h' | sort | xargs sha256sum | sha256sum | cut -d' ' -f1) \
&& ACTUAL_ABI=$(grep -rhE '^#[[:space:]]*define PYBIND11_INTERNALS_VERSION' "${INC}" | awk '{print $NF}') \
&& echo "pybind11 $(pip show pybind11 | awk '/^Version:/ {print $2}')" \
"sha256 expected=${EXPECTED} actual=${ACTUAL}" \
"internals expected=${EXPECTED_ABI} actual=${ACTUAL_ABI}" \
&& test -n "${EXPECTED}" \
&& test "${EXPECTED}" = "${ACTUAL}" \
&& test "${EXPECTED_ABI}" = "${ACTUAL_ABI}"
ARG COMMON_WORKDIR ARG COMMON_WORKDIR
ARG BASE_IMAGE ARG BASE_IMAGE
ARG NIC_BACKEND ARG NIC_BACKEND
+479
View File
@@ -0,0 +1,479 @@
ARG BASE_IMAGE=rocm/dev-ubuntu-22.04:7.2.3-complete
ARG TRITON_BRANCH="669b31a" # release/internal/3.8.x as of 09/17
ARG TRITON_REPO="https://github.com/ROCm/triton.git"
ARG PYTORCH_BRANCH="733fca1" # release/2.13 as of 09/17
ARG PYTORCH_REPO="https://github.com/ROCm/pytorch.git"
ARG PYTORCH_VISION_BRANCH="v0.28.0"
ARG PYTORCH_VISION_REPO="https://github.com/pytorch/vision.git"
ARG PYTORCH_AUDIO_BRANCH="v2.11.0"
ARG PYTORCH_AUDIO_REPO="https://github.com/pytorch/audio.git"
ARG FA_BRANCH="0e60e394"
ARG FA_REPO="https://github.com/Dao-AILab/flash-attention.git"
ARG AITER_BRANCH="v0.1.25"
ARG AITER_REPO="https://github.com/ROCm/aiter.git"
ARG MORI_BRANCH="v1.2.3.post1"
ARG MORI_REPO="https://github.com/ROCm/mori.git"
ARG ROCM_SYSTEMS_REPO="https://github.com/ROCm/rocm-systems.git"
ARG ROCPROFILER_SDK_COMMIT="2b22ab0195cc1461cd9abf3b969e9dd7c10af350"
ARG ROCM_RUNTIME_COMMIT="af70edc91f1dc5d228fd4c076df99e65e49c4521"
# Sccache configuration (only used in release pipeline)
ARG USE_SCCACHE
ARG SCCACHE_DOWNLOAD_URL
ARG SCCACHE_ENDPOINT
ARG SCCACHE_BUCKET_NAME=vllm-build-sccache
ARG SCCACHE_REGION_NAME=us-west-2
ARG SCCACHE_S3_NO_CREDENTIALS=0
FROM ${BASE_IMAGE} AS base
ENV PATH=/opt/rocm/llvm/bin:/opt/rocm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ENV ROCM_PATH=/opt/rocm
ENV LD_LIBRARY_PATH=/opt/rocm/lib:/usr/local/lib:
ARG PYTORCH_ROCM_ARCH=gfx90a;gfx942;gfx950;gfx1100;gfx1101;gfx1200;gfx1201;gfx1150;gfx1151
ENV PYTORCH_ROCM_ARCH=${PYTORCH_ROCM_ARCH}
ENV AITER_ROCM_ARCH=gfx942;gfx950
# Note: Do not set MORI_GPU_ARCHS here, it is automatically inferred at runtime
# Required for RCCL in ROCm7.1
ENV HSA_NO_SCRATCH_RECLAIM=1
ARG PYTHON_VERSION=3.12
ENV PYTHON_VERSION=${PYTHON_VERSION}
RUN mkdir -p /app
WORKDIR /app
ENV DEBIAN_FRONTEND=noninteractive
# Install Python and other dependencies (plus gcc-13, selected later in build_pytorch).
RUN apt-get update -y \
&& apt-get install -y software-properties-common git curl sudo vim less libgfortran5 libopenmpi-dev libpci-dev liblzma-dev pkg-config \
&& for i in 1 2 3; do \
add-apt-repository -y ppa:deadsnakes/ppa && add-apt-repository -y ppa:ubuntu-toolchain-r/test && break || \
{ echo "Attempt $i failed, retrying in 5s..."; sleep 5; }; \
done \
&& apt-get update -y \
&& apt-get install -y python${PYTHON_VERSION} python${PYTHON_VERSION}-dev python${PYTHON_VERSION}-venv \
python${PYTHON_VERSION}-lib2to3 python-is-python3 gcc-13 g++-13 \
&& update-alternatives --install /usr/bin/python3 python3 /usr/bin/python${PYTHON_VERSION} 1 \
&& update-alternatives --set python3 /usr/bin/python${PYTHON_VERSION} \
&& ln -sf /usr/bin/python${PYTHON_VERSION}-config /usr/bin/python3-config \
&& curl -sS https://bootstrap.pypa.io/get-pip.py | python${PYTHON_VERSION} \
&& python3 --version && python3 -m pip --version
RUN pip install -U packaging 'cmake<4' ninja wheel 'setuptools<80' 'pybind11<3.1.0' Cython
RUN apt-get update && apt-get install -y libjpeg-dev libsox-dev libsox-fmt-all sox && rm -rf /var/lib/apt/lists/*
# Install sccache if USE_SCCACHE is enabled (for release builds)
ARG USE_SCCACHE
ARG SCCACHE_DOWNLOAD_URL
ARG SCCACHE_ENDPOINT
ARG SCCACHE_BUCKET_NAME
ARG SCCACHE_REGION_NAME
ARG SCCACHE_S3_NO_CREDENTIALS
RUN if [ "$USE_SCCACHE" = "1" ]; then \
echo "Installing sccache..." \
&& SCCACHE_ARCH="x86_64" \
&& SCCACHE_VERSION="v0.8.1" \
&& SCCACHE_DL_URL="${SCCACHE_DOWNLOAD_URL:-https://github.com/mozilla/sccache/releases/download/${SCCACHE_VERSION}/sccache-${SCCACHE_VERSION}-${SCCACHE_ARCH}-unknown-linux-musl.tar.gz}" \
&& curl -L -o /tmp/sccache.tar.gz ${SCCACHE_DL_URL} \
&& tar -xzf /tmp/sccache.tar.gz -C /tmp \
&& mv /tmp/sccache-${SCCACHE_VERSION}-${SCCACHE_ARCH}-unknown-linux-musl/sccache /usr/bin/sccache \
&& chmod +x /usr/bin/sccache \
&& rm -rf /tmp/sccache.tar.gz /tmp/sccache-${SCCACHE_VERSION}-${SCCACHE_ARCH}-unknown-linux-musl \
&& sccache --version; \
fi
# Setup sccache for HIP compilation via HIP_CLANG_PATH
# This creates wrapper scripts in a separate directory and points HIP to use them
# This avoids modifying the original ROCm binaries which can break detection
# NOTE: HIP_CLANG_PATH is NOT set as ENV to avoid affecting downstream images (Dockerfile.rocm_72)
# Instead, each build stage should export HIP_CLANG_PATH=/opt/sccache-wrappers if USE_SCCACHE=1
RUN if [ "$USE_SCCACHE" = "1" ]; then \
echo "Setting up sccache wrappers for HIP compilation..." \
&& mkdir -p /opt/sccache-wrappers \
&& printf '#!/bin/bash\nexec sccache /opt/rocm/lib/llvm/bin/clang++ "$@"\n' > /opt/sccache-wrappers/clang++ \
&& chmod +x /opt/sccache-wrappers/clang++ \
&& printf '#!/bin/bash\nexec sccache /opt/rocm/lib/llvm/bin/clang "$@"\n' > /opt/sccache-wrappers/clang \
&& chmod +x /opt/sccache-wrappers/clang \
&& echo "sccache wrappers created in /opt/sccache-wrappers"; \
fi
# Set sccache environment variables only when USE_SCCACHE=1
# This prevents S3 config from leaking into images when sccache is not used
ARG USE_SCCACHE
ENV SCCACHE_BUCKET=${USE_SCCACHE:+${SCCACHE_BUCKET_NAME}}
ENV SCCACHE_REGION=${USE_SCCACHE:+${SCCACHE_REGION_NAME}}
ENV SCCACHE_S3_NO_CREDENTIALS=${USE_SCCACHE:+${SCCACHE_S3_NO_CREDENTIALS}}
ENV SCCACHE_IDLE_TIMEOUT=${USE_SCCACHE:+0}
###
### Triton Build
###
FROM base AS build_triton
ARG TRITON_BRANCH
ARG TRITON_REPO
RUN git clone ${TRITON_REPO}
RUN cd triton \
&& git checkout ${TRITON_BRANCH} \
&& git config --global user.email "you@example.com" && git config --global user.name "Your Name" \
&& if [ ! -f setup.py ]; then cd python; fi \
&& MAX_JOBS=16 python3 setup.py bdist_wheel --dist-dir=dist \
&& mkdir -p /app/install && cp dist/*.whl /app/install
###
### AMD SMI Build
###
FROM base AS build_amdsmi
RUN cd /opt/rocm/share/amd_smi \
&& pip wheel . --wheel-dir=dist
RUN mkdir -p /app/install && cp /opt/rocm/share/amd_smi/dist/*.whl /app/install
###
### Pytorch build
###
FROM base AS build_pytorch
ARG PYTORCH_BRANCH
ARG PYTORCH_REPO
ARG USE_SCCACHE
RUN apt-get update && apt-get install -y pkg-config liblzma-dev
# torch 2.12 emits float->__bf16 libcalls (__truncsfbf2) absent from gcc-11 libgcc; build with gcc-13.
RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-13 130 --slave /usr/bin/g++ g++ /usr/bin/g++-13
RUN git clone ${PYTORCH_REPO} pytorch
RUN cd pytorch && git checkout ${PYTORCH_BRANCH}
RUN cd pytorch \
&& pip install -r requirements.txt && git submodule update --init --recursive
RUN cd pytorch && python3 tools/amd_build/build_amd.py \
&& if [ "$USE_SCCACHE" = "1" ]; then \
export HIP_CLANG_PATH=/opt/sccache-wrappers \
&& export CMAKE_C_COMPILER_LAUNCHER=sccache \
&& export CMAKE_CXX_COMPILER_LAUNCHER=sccache \
&& sccache --show-stats; \
fi \
&& CMAKE_PREFIX_PATH=$(python3 -c 'import sys; print(sys.prefix)') python3 setup.py bdist_wheel --dist-dir=dist \
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi
RUN mkdir -p /app/install && cp /app/pytorch/dist/*.whl /app/install
###
### Shared PyTorch runtime for dependent builds
###
FROM base AS build_pytorch_runtime
RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-13 130 --slave /usr/bin/g++ g++ /usr/bin/g++-13
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install,ro \
pip install /install/*.whl
###
### TorchVision build
###
FROM build_pytorch_runtime AS build_torchvision
ARG PYTORCH_VISION_BRANCH
ARG PYTORCH_VISION_REPO
ARG USE_SCCACHE
RUN git clone ${PYTORCH_VISION_REPO} vision
RUN cd vision && git checkout ${PYTORCH_VISION_BRANCH} \
&& if [ "$USE_SCCACHE" = "1" ]; then \
export HIP_CLANG_PATH=/opt/sccache-wrappers \
&& export CMAKE_C_COMPILER_LAUNCHER=sccache \
&& export CMAKE_CXX_COMPILER_LAUNCHER=sccache; \
fi \
&& python3 setup.py bdist_wheel --dist-dir=dist \
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi
RUN mkdir -p /app/install && cp /app/vision/dist/*.whl /app/install
###
### TorchAudio build
###
FROM build_pytorch_runtime AS build_torchaudio
ARG PYTORCH_AUDIO_BRANCH
ARG PYTORCH_AUDIO_REPO
ARG USE_SCCACHE
RUN git clone ${PYTORCH_AUDIO_REPO} audio
RUN cd audio && git checkout ${PYTORCH_AUDIO_BRANCH} \
&& git submodule update --init --recursive
RUN cd audio && pip install -r requirements.txt
RUN cd audio \
&& if [ "$USE_SCCACHE" = "1" ]; then \
export HIP_CLANG_PATH=/opt/sccache-wrappers \
&& export CMAKE_C_COMPILER_LAUNCHER=sccache \
&& export CMAKE_CXX_COMPILER_LAUNCHER=sccache; \
fi \
&& python3 setup.py bdist_wheel --dist-dir=dist \
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi
RUN mkdir -p /app/install && cp /app/audio/dist/*.whl /app/install
###
### MORI Build
###
FROM base AS build_mori
ARG MORI_BRANCH
ARG MORI_REPO
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
pip install /install/*.whl
RUN git clone ${MORI_REPO}
RUN cd mori \
&& git checkout ${MORI_BRANCH} \
&& git submodule update --init --recursive \
&& python3 setup.py bdist_wheel --dist-dir=dist && ls /app/mori/dist/*.whl
RUN mkdir -p /app/install && cp /app/mori/dist/*.whl /app/install
###
### FlashAttention Build
###
FROM base AS build_fa
ARG FA_BRANCH
ARG FA_REPO
ARG USE_SCCACHE
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
pip install /install/*.whl
RUN git clone ${FA_REPO}
RUN cd flash-attention \
&& git checkout ${FA_BRANCH} \
&& git submodule update --init \
&& if [ "$USE_SCCACHE" = "1" ]; then \
export HIP_CLANG_PATH=/opt/sccache-wrappers \
&& sccache --show-stats; \
fi \
&& GPU_ARCHS=$(echo ${PYTORCH_ROCM_ARCH} | sed -e 's/;gfx1[0-9]\{3\}//g') python3 setup.py bdist_wheel --dist-dir=dist \
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi
RUN mkdir -p /app/install && cp /app/flash-attention/dist/*.whl /app/install
###
### AITER Build
###
FROM base AS build_aiter
ARG AITER_BRANCH
ARG AITER_REPO
ARG USE_SCCACHE
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
pip install /install/*.whl
RUN git clone --recursive --branch ${AITER_BRANCH} ${AITER_REPO}
RUN cd aiter \
&& git submodule update --init --recursive \
&& pip install -r requirements.txt
RUN pip install pyyaml && cd aiter \
&& if [ "$USE_SCCACHE" = "1" ]; then \
export HIP_CLANG_PATH=/opt/sccache-wrappers \
&& sccache --show-stats; \
fi \
&& PREBUILD_KERNELS=1 AITER_USE_SYSTEM_TRITON=1 GPU_ARCHS=${AITER_ROCM_ARCH} python3 setup.py bdist_wheel --dist-dir=dist \
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi \
&& ls /app/aiter/dist/*.whl
RUN mkdir -p /app/install && cp /app/aiter/dist/*.whl /app/install
###
### Final Build
###
# Wheel release stage -
# only includes dependencies used by wheel release pipeline
FROM base AS debs_wheel_release
RUN mkdir /app/debs
RUN --mount=type=bind,from=build_triton,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_fa,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_amdsmi,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_torchvision,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_torchaudio,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_aiter,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
# Full debs stage - includes Mori (used by Docker releases)
FROM base AS debs
RUN mkdir /app/debs
RUN --mount=type=bind,from=build_triton,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_fa,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_amdsmi,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_torchvision,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_torchaudio,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_aiter,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_mori,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
###
### ROCr + CLR build
###
# Pull in updates to ROCr + clr from ROCm 7.2.4 and beyond.
FROM base AS build_rocm_runtime
ARG ROCM_SYSTEMS_REPO
ARG ROCM_RUNTIME_COMMIT
RUN apt-get update \
&& apt-get install -y --no-install-recommends g++ \
libelf-dev libdrm-dev libnuma-dev libdw-dev xxd \
&& rm -rf /var/lib/apt/lists/*
RUN pip install --no-cache-dir CppHeaderParser
RUN git init -q /src && cd /src \
&& git remote add origin ${ROCM_SYSTEMS_REPO} \
&& git config core.sparseCheckout true \
&& git config remote.origin.promisor true \
&& git config remote.origin.partialclonefilter blob:none \
&& git sparse-checkout init --cone \
&& git sparse-checkout set projects/rocr-runtime projects/clr projects/hip shared cmake \
&& git fetch --filter=blob:none --depth 1 origin ${ROCM_RUNTIME_COMMIT} \
&& git checkout -q FETCH_HEAD
# ROCr's trap_handler and blit_shaders call find_package(Clang/LLVM REQUIRED) only to get
# IMPORTED executables. The ROCm dev image ships the binaries but no CMake package config, so
# supply one pointing at them.
RUN mkdir -p /opt/rocm-cmake-shim \
&& printf 'if(NOT TARGET clang)\n add_executable(clang IMPORTED GLOBAL)\n set_target_properties(clang PROPERTIES IMPORTED_LOCATION "/opt/rocm/llvm/bin/clang")\nendif()\nset(Clang_PACKAGE_VERSION "rocm-image-shim")\n' > /opt/rocm-cmake-shim/ClangConfig.cmake \
&& printf 'if(NOT TARGET llvm-objcopy)\n add_executable(llvm-objcopy IMPORTED GLOBAL)\n set_target_properties(llvm-objcopy PROPERTIES IMPORTED_LOCATION "/opt/rocm/llvm/bin/llvm-objcopy")\nendif()\nset(LLVM_FOUND TRUE)\nset(LLVM_PACKAGE_VERSION "rocm-image-shim")\n' > /opt/rocm-cmake-shim/LLVMConfig.cmake
# ROCr first, and installed into /opt/rocm, because the clr build below needs its headers
RUN cd /src/projects/rocr-runtime \
&& cmake -B build -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_PREFIX_PATH=/opt/rocm \
-DCMAKE_INSTALL_PREFIX=/opt/rocm \
-DClang_DIR=/opt/rocm-cmake-shim \
-DLLVM_DIR=/opt/rocm-cmake-shim \
&& cmake --build build --parallel 16 \
&& cmake --install build \
&& cmake --install build --prefix /rocr-install --strip
RUN cd /src/projects/clr \
&& cmake -B build -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCLR_BUILD_HIP=ON \
-DCLR_BUILD_OCL=OFF \
-DHIP_COMMON_DIR=/src/projects/hip \
-DROCM_PATH=/opt/rocm \
-DCMAKE_PREFIX_PATH=/opt/rocm \
-DCMAKE_INSTALL_PREFIX=/opt/rocm \
-DLLVM_DIR=/opt/rocm-cmake-shim \
-DHIP_LLVM_ROOT=/opt/rocm/llvm \
&& cmake --build build --parallel 16 \
&& cmake --install build --prefix /clr-install --strip
RUN mkdir -p /staging/lib \
&& cp -P /rocr-install/lib/libhsa-runtime64.so* /staging/lib/ \
&& cp -P /clr-install/lib/libamdhip64.so* /staging/lib/ \
&& ls -l /staging/lib
# rocprofiler-sdk 1.3.2 fixes torch 2.12 profiler on ROCm 7.2.3
FROM base AS build_rocprofiler_sdk
ARG ROCM_SYSTEMS_REPO
ARG ROCPROFILER_SDK_COMMIT
ARG USE_SCCACHE
RUN apt-get update \
&& apt-get install -y --no-install-recommends libdw-dev libelf-dev libsqlite3-dev libzstd-dev \
&& rm -rf /var/lib/apt/lists/*
RUN git init -q /src && cd /src \
&& git remote add origin ${ROCM_SYSTEMS_REPO} \
&& git config core.sparseCheckout true \
&& git config remote.origin.promisor true \
&& git config remote.origin.partialclonefilter blob:none \
&& git sparse-checkout init --cone \
&& git sparse-checkout set projects/rocprofiler-sdk shared cmake \
&& git fetch --filter=blob:none --depth 1 origin ${ROCPROFILER_SDK_COMMIT} \
&& git checkout -q FETCH_HEAD
# Patch the ROCm 7.2.3 build of rocprofiler-sdk to include queue-interposition fixes that are
# absent from the pinned 7.14 build. This resolves hangs on torch.profiler on ROCm.
# TODO: Drop these once ${ROCPROFILER_SDK_COMMIT} is bumped to a build that already
# contains them, such as ROCm 10.0. See https://github.com/vllm-project/vllm/issues/54087.
ARG ROCPROFILER_SDK_PR_7796=fae7fd57ffb8a76e811d6410b8a0cee1e94d710a
ARG ROCPROFILER_SDK_PR_7924=f8db22adbe17686a9b05ae03367dd1ec73011653
RUN cd /src \
&& HSA_PATHS='projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/hsa/*' \
&& curl -fsSL -o /tmp/qi-7796.patch "${ROCM_SYSTEMS_REPO%.git}/commit/${ROCPROFILER_SDK_PR_7796}.patch" \
&& curl -fsSL -o /tmp/qi-7924.patch "${ROCM_SYSTEMS_REPO%.git}/commit/${ROCPROFILER_SDK_PR_7924}.patch" \
&& git apply -p1 --include="$HSA_PATHS" --verbose /tmp/qi-7796.patch \
&& git apply -p1 --include="$HSA_PATHS" --verbose /tmp/qi-7924.patch
RUN cd /src/projects/rocprofiler-sdk \
&& if [ "$USE_SCCACHE" = "1" ]; then \
export CMAKE_C_COMPILER_LAUNCHER=sccache \
&& export CMAKE_CXX_COMPILER_LAUNCHER=sccache; \
fi \
&& cmake -B build -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_PREFIX_PATH=/opt/rocm \
-DCMAKE_INSTALL_PREFIX=/opt/rocm \
-DROCPROFILER_BUILD_TESTS=OFF \
-DROCPROFILER_BUILD_SAMPLES=OFF \
-DROCPROFILER_BUILD_DOCS=OFF \
&& cmake --build build --parallel "$(nproc)" \
&& cmake --install build --prefix /staging --strip
FROM base AS final
RUN --mount=type=bind,from=debs,src=/app/debs,target=/install \
pip install /install/*.whl
COPY --from=build_rocprofiler_sdk /staging/ /opt/rocm/
ENV ROCPROFILER_QUEUE_INTERPOSITION=0
COPY --from=build_rocm_runtime /staging/ /opt/rocm/
# Hard link the patched library onto every stock file name so that both the
# SONAME lookup and the hard coded paths resolve to it.
RUN set -eux; \
cd /opt/rocm/lib; \
ours_hsa="$(readlink -f libhsa-runtime64.so.1)"; \
ours_hip="$(readlink -f libamdhip64.so.7)"; \
for f in libhsa-runtime64.so.1.*; do \
[ "$f" = "${ours_hsa##*/}" ] || ln -f "$ours_hsa" "$f"; \
done; \
for f in libamdhip64.so.7.*; do \
[ "$f" = "${ours_hip##*/}" ] || ln -f "$ours_hip" "$f"; \
done
RUN ldconfig
ARG BASE_IMAGE
ARG TRITON_BRANCH
ARG TRITON_REPO
ARG PYTORCH_BRANCH
ARG PYTORCH_VISION_BRANCH
ARG PYTORCH_REPO
ARG PYTORCH_VISION_REPO
ARG PYTORCH_AUDIO_BRANCH
ARG PYTORCH_AUDIO_REPO
ARG FA_BRANCH
ARG FA_REPO
ARG AITER_BRANCH
ARG AITER_REPO
ARG MORI_BRANCH
ARG MORI_REPO
ARG ROCM_SYSTEMS_REPO
ARG ROCPROFILER_SDK_COMMIT
ARG ROCM_RUNTIME_COMMIT
RUN echo "BASE_IMAGE: ${BASE_IMAGE}" > /app/versions.txt \
&& echo "ROCM_SYSTEMS_REPO: ${ROCM_SYSTEMS_REPO}" >> /app/versions.txt \
&& echo "ROCPROFILER_SDK_COMMIT: ${ROCPROFILER_SDK_COMMIT}" >> /app/versions.txt \
&& echo "ROCM_RUNTIME_COMMIT: ${ROCM_RUNTIME_COMMIT}" >> /app/versions.txt \
&& echo "TRITON_BRANCH: ${TRITON_BRANCH}" >> /app/versions.txt \
&& echo "TRITON_REPO: ${TRITON_REPO}" >> /app/versions.txt \
&& echo "PYTORCH_BRANCH: ${PYTORCH_BRANCH}" >> /app/versions.txt \
&& echo "PYTORCH_VISION_BRANCH: ${PYTORCH_VISION_BRANCH}" >> /app/versions.txt \
&& echo "PYTORCH_REPO: ${PYTORCH_REPO}" >> /app/versions.txt \
&& echo "PYTORCH_VISION_REPO: ${PYTORCH_VISION_REPO}" >> /app/versions.txt \
&& echo "PYTORCH_AUDIO_BRANCH: ${PYTORCH_AUDIO_BRANCH}" >> /app/versions.txt \
&& echo "PYTORCH_AUDIO_REPO: ${PYTORCH_AUDIO_REPO}" >> /app/versions.txt \
&& echo "FA_BRANCH: ${FA_BRANCH}" >> /app/versions.txt \
&& echo "FA_REPO: ${FA_REPO}" >> /app/versions.txt \
&& echo "AITER_BRANCH: ${AITER_BRANCH}" >> /app/versions.txt \
&& echo "AITER_REPO: ${AITER_REPO}" >> /app/versions.txt \
&& echo "MORI_BRANCH: ${MORI_BRANCH}" >> /app/versions.txt \
&& echo "MORI_REPO: ${MORI_REPO}" >> /app/versions.txt
+283 -286
View File
@@ -1,12 +1,11 @@
ARG BASE_IMAGE=rocm/dev-ubuntu-22.04:7.2.3-complete ARG BASE_IMAGE=ubuntu:22.04
ARG TRITON_BRANCH="669b31a" # release/internal/3.8.x as of 09/17 ARG PYTHON_VERSION=3.12
ARG TORCH_VERSION=2.12.0+rocm10.0.0
ARG TORCHVISION_VERSION=0.27.0+rocm10.0.0
ARG TORCHAUDIO_VERSION=2.11.0+rocm10.0.0
ARG TRITON_BRANCH="669b31ac" # release/internal/3.8.x as of 09/29/2026
ARG TRITON_VERSION=3.8.0+git669b31ac
ARG TRITON_REPO="https://github.com/ROCm/triton.git" ARG TRITON_REPO="https://github.com/ROCm/triton.git"
ARG PYTORCH_BRANCH="733fca1" # release/2.13 as of 09/17
ARG PYTORCH_REPO="https://github.com/ROCm/pytorch.git"
ARG PYTORCH_VISION_BRANCH="v0.28.0"
ARG PYTORCH_VISION_REPO="https://github.com/pytorch/vision.git"
ARG PYTORCH_AUDIO_BRANCH="v2.11.0"
ARG PYTORCH_AUDIO_REPO="https://github.com/pytorch/audio.git"
ARG FA_BRANCH="0e60e394" ARG FA_BRANCH="0e60e394"
ARG FA_REPO="https://github.com/Dao-AILab/flash-attention.git" ARG FA_REPO="https://github.com/Dao-AILab/flash-attention.git"
ARG AITER_BRANCH="v0.1.25" ARG AITER_BRANCH="v0.1.25"
@@ -14,8 +13,18 @@ ARG AITER_REPO="https://github.com/ROCm/aiter.git"
ARG MORI_BRANCH="v1.2.3.post1" ARG MORI_BRANCH="v1.2.3.post1"
ARG MORI_REPO="https://github.com/ROCm/mori.git" ARG MORI_REPO="https://github.com/ROCm/mori.git"
ARG ROCM_SYSTEMS_REPO="https://github.com/ROCm/rocm-systems.git" ARG ROCM_SYSTEMS_REPO="https://github.com/ROCm/rocm-systems.git"
ARG ROCPROFILER_SDK_COMMIT="2b22ab0195cc1461cd9abf3b969e9dd7c10af350" ARG ROCM_RUNTIME_COMMIT="25e14349f2606a6b95c4ccd45eb92a1835646cf2"
ARG ROCM_RUNTIME_COMMIT="af70edc91f1dc5d228fd4c076df99e65e49c4521"
# TheRock configuration
ARG SITE_PACKAGES=/usr/local/lib/python${PYTHON_VERSION}/dist-packages
ARG ROCM_RELEASE_WHEELS_MULTIARCH_URL="https://stable.repo.amd.com/rocm/whl-next/" # For pre-built GPU images, packages, etc.
ARG ROCM_NIGHTLY_WHEELS_MULTIARCH_URL="https://nightly.repo.amd.com/rocm/whl-next/" # For torch, triton, etc. wheels with latest fixes.
ARG ROCM_SDK_VERSION=10.0.0
# Build configuration
ARG PYTORCH_ROCM_ARCH="gfx90a;gfx942;gfx950;gfx1100;gfx1101;gfx1200;gfx1201;gfx1150;gfx1151"
ARG AITER_ROCM_ARCH="gfx942;gfx950"
# Sccache configuration (only used in release pipeline) # Sccache configuration (only used in release pipeline)
ARG USE_SCCACHE ARG USE_SCCACHE
@@ -25,47 +34,128 @@ ARG SCCACHE_BUCKET_NAME=vllm-build-sccache
ARG SCCACHE_REGION_NAME=us-west-2 ARG SCCACHE_REGION_NAME=us-west-2
ARG SCCACHE_S3_NO_CREDENTIALS=0 ARG SCCACHE_S3_NO_CREDENTIALS=0
FROM ${BASE_IMAGE} AS base FROM ${BASE_IMAGE} AS base
ENV PATH=/opt/rocm/llvm/bin:/opt/rocm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin ARG PYTORCH_ROCM_ARCH
ENV ROCM_PATH=/opt/rocm ENV PYTORCH_ROCM_ARCH="${PYTORCH_ROCM_ARCH}"
ENV LD_LIBRARY_PATH=/opt/rocm/lib:/usr/local/lib: ARG AITER_ROCM_ARCH
ENV AITER_ROCM_ARCH="${AITER_ROCM_ARCH}"
ARG PYTORCH_ROCM_ARCH=gfx90a;gfx942;gfx950;gfx1100;gfx1101;gfx1200;gfx1201;gfx1150;gfx1151 ARG SITE_PACKAGES
ENV PYTORCH_ROCM_ARCH=${PYTORCH_ROCM_ARCH} ENV SITE_PACKAGES=${SITE_PACKAGES}
ENV AITER_ROCM_ARCH=gfx942;gfx950 ENV SDK_DEV="$SITE_PACKAGES/_rocm_sdk_devel" \
# Note: Do not set MORI_GPU_ARCHS here, it is automatically inferred at runtime SDK_CORE="$SITE_PACKAGES/_rocm_sdk_core"
ENV ROCM_PATH="$SDK_DEV" \
ROCM_HOME="$SDK_DEV" \
HIP_PATH="$SDK_DEV" \
HIP_CLANG_PATH="$SDK_DEV/lib/llvm/bin" \
HIP_DEVICE_LIB_PATH="$SDK_CORE/lib/llvm/amdgcn/bitcode"
ENV LD_LIBRARY_PATH="$SDK_DEV/lib:$SDK_DEV/lib/rocm_sysdeps/lib:$SDK_CORE/lib:$SDK_CORE/lib/rocm_sysdeps/lib"
ENV PATH="$SDK_DEV/bin:$SDK_DEV/lib/llvm/bin:$SDK_CORE/bin:$PATH"
# Required for RCCL in ROCm7.1 # Required for RCCL in ROCm7.1
ENV HSA_NO_SCRATCH_RECLAIM=1 ENV HSA_NO_SCRATCH_RECLAIM=1
ARG PYTHON_VERSION=3.12 ARG PYTHON_VERSION
ENV PYTHON_VERSION=${PYTHON_VERSION} ENV PYTHON_VERSION=${PYTHON_VERSION}
RUN mkdir -p /app RUN mkdir -p /app
WORKDIR /app WORKDIR /app
ENV DEBIAN_FRONTEND=noninteractive ENV DEBIAN_FRONTEND=noninteractive
# Install Python and other dependencies (plus gcc-13, selected later in build_pytorch). # Install Python and other dependencies
RUN apt-get update -y \ RUN apt-get update -y \
&& apt-get install -y software-properties-common git curl sudo vim less libgfortran5 libopenmpi-dev libpci-dev liblzma-dev pkg-config \ && apt-get install -y software-properties-common build-essential git curl sudo vim less libgfortran5 libopenmpi-dev libpci-dev liblzma-dev pkg-config libdrm-dev libnuma-dev kmod \
&& for i in 1 2 3; do \ && for i in 1 2 3; do \
add-apt-repository -y ppa:deadsnakes/ppa && add-apt-repository -y ppa:ubuntu-toolchain-r/test && break || \ add-apt-repository -y ppa:deadsnakes/ppa && break || \
{ echo "Attempt $i failed, retrying in 5s..."; sleep 5; }; \ { echo "Attempt $i failed, retrying in 5s..."; sleep 5; }; \
done \ done \
&& apt-get update -y \ && apt-get update -y \
&& apt-get install -y python${PYTHON_VERSION} python${PYTHON_VERSION}-dev python${PYTHON_VERSION}-venv \ && apt-get install -y python${PYTHON_VERSION} python${PYTHON_VERSION}-dev python${PYTHON_VERSION}-venv \
python${PYTHON_VERSION}-lib2to3 python-is-python3 gcc-13 g++-13 \ python${PYTHON_VERSION}-lib2to3 python-is-python3 \
&& update-alternatives --install /usr/bin/python3 python3 /usr/bin/python${PYTHON_VERSION} 1 \ && update-alternatives --install /usr/bin/python3 python3 /usr/bin/python${PYTHON_VERSION} 1 \
&& update-alternatives --set python3 /usr/bin/python${PYTHON_VERSION} \ && update-alternatives --set python3 /usr/bin/python${PYTHON_VERSION} \
&& ln -sf /usr/bin/python${PYTHON_VERSION}-config /usr/bin/python3-config \ && ln -sf /usr/bin/python${PYTHON_VERSION}-config /usr/bin/python3-config \
&& curl -sS https://bootstrap.pypa.io/get-pip.py | python${PYTHON_VERSION} \ && curl -sS https://bootstrap.pypa.io/get-pip.py | python${PYTHON_VERSION} \
&& python3 --version && python3 -m pip --version && python3 --version && python3 -m pip --version
RUN pip install -U packaging 'cmake<4' ninja wheel 'setuptools<80' 'pybind11<3.1.0' Cython RUN pip install -U packaging 'cmake<4' ninja wheel 'setuptools<80' pybind11==3.0.4 Cython
RUN apt-get update && apt-get install -y libjpeg-dev libsox-dev libsox-fmt-all sox && rm -rf /var/lib/apt/lists/* RUN apt-get update && apt-get install -y libjpeg-dev libsox-dev libsox-fmt-all sox && rm -rf /var/lib/apt/lists/*
# Install ROCm SDK via TheRock
ARG ROCM_RELEASE_WHEELS_MULTIARCH_URL
ARG ROCM_NIGHTLY_WHEELS_MULTIARCH_URL
ARG ROCM_SDK_VERSION
RUN pip install --index-url ${ROCM_RELEASE_WHEELS_MULTIARCH_URL} \
--extra-index-url ${ROCM_NIGHTLY_WHEELS_MULTIARCH_URL} \
"rocm[libraries,devel]${ROCM_SDK_VERSION:+==${ROCM_SDK_VERSION}}" \
&& rocm-sdk init
# Install PyTorch wheels (nightly for latest torch, release as fallback)
ARG TORCH_VERSION
ARG TORCHAUDIO_VERSION
ARG TORCHVISION_VERSION
RUN pip install \
--index-url ${ROCM_RELEASE_WHEELS_MULTIARCH_URL} \
--extra-index-url ${ROCM_NIGHTLY_WHEELS_MULTIARCH_URL} \
"torch==${TORCH_VERSION}" \
"torchaudio==${TORCHAUDIO_VERSION}" \
"torchvision==${TORCHVISION_VERSION}"
# Install the AMD-compiled device kernels corresponding to PYTORCH_ROCM_ARCH:
# Possible values that can be in PYTORCH_ROCM_ARCH:
#
# gfx90a (MI210/250), gfx942 (MI300), gfx950 (MI350), gfx1100 (7900XTX),
# gfx1101 (7800XT), gfx1200 (9070XT), gfx1201 (9070), gfx1150/1151 (Strix APU)
# For each arch in PYTORCH_ROCM_ARCH, install the precompiled GPU code that
# PyTorch needs to run on that specific chip.
SHELL ["/bin/bash", "-c"]
RUN TORCH_VER=$(python3 -c "import torch; print(torch.__version__)") \
&& for arch in $(echo "${PYTORCH_ROCM_ARCH}" | tr ';' ' '); do \
echo "Installing device packages for ${arch}" \
&& pip install --index-url ${ROCM_RELEASE_WHEELS_MULTIARCH_URL} \
--extra-index-url ${ROCM_NIGHTLY_WHEELS_MULTIARCH_URL} \
"torch[device-${arch}]==${TORCH_VER}" \
"torchvision[device-${arch}]==${TORCHVISION_VERSION}" \
"rocm[device-${arch}]==${ROCM_SDK_VERSION}"; \
done
# Remove gfx90a xnack kpacks (ASAN artifacts that cause hipErrorInvalidImage on MI250)
RUN rm -f "${SITE_PACKAGES}/torch/.kpack/torch_gfx90a:xnack+.kpack" \
"${SITE_PACKAGES}/torch/.kpack/torch_gfx90a:xnack-.kpack"
RUN cd $SITE_PACKAGES/_rocm_sdk_core/share/amd_smi && pip install .
ENV PYTHONPATH=$SITE_PACKAGES/_rocm_sdk_core/share/amd_smi
# TheRock runtime fixes
RUN if [ -f "${SITE_PACKAGES}/rocm_sdk/__init__.py" ]; then \
sed -i 's/rtld_global: bool = True/rtld_global: bool = False/g' "${SITE_PACKAGES}/rocm_sdk/__init__.py"; \
fi \
&& if [ -f "${SITE_PACKAGES}/_rocm_sdk_core/lib/librocm_smi64.so.1" ] \
&& [ ! -e "${ROCM_PATH}/lib/librocm_smi64.so.7" ]; then \
ln -sfn "${SITE_PACKAGES}/_rocm_sdk_core/lib/librocm_smi64.so.1" "${ROCM_PATH}/lib/librocm_smi64.so.7"; \
fi \
&& printf '%s\n%s\n' "${ROCM_PATH}/lib" "${SITE_PACKAGES}/_rocm_sdk_core/lib" > /etc/ld.so.conf.d/rocm-pip.conf \
&& for f in ${SITE_PACKAGES}/_rocm_sdk_core/lib/libamdhip64.so*; do \
bn=$(basename "$f"); \
if [ -f "${ROCM_PATH}/lib/${bn}" ] && [ -f "$f" ] && [ ! -L "$f" ]; then \
rm -f "$f"; \
ln -sfn "${ROCM_PATH}/lib/${bn}" "$f"; \
fi; \
done \
&& for f in librocprofiler-sdk.so.1 librocprofiler-register.so.0 \
librocprofiler-sdk-roctx.so.1; do \
if [ -e "${SITE_PACKAGES}/_rocm_sdk_core/lib/${f}" ] \
&& [ -e "${SITE_PACKAGES}/_rocm_sdk_devel/lib/${f}" ]; then \
rm -f "${SITE_PACKAGES}/_rocm_sdk_core/lib/${f}"; \
ln -sfn "${SITE_PACKAGES}/_rocm_sdk_devel/lib/${f}" \
"${SITE_PACKAGES}/_rocm_sdk_core/lib/${f}"; \
fi; \
done \
&& ldconfig
# Install sccache if USE_SCCACHE is enabled (for release builds) # Install sccache if USE_SCCACHE is enabled (for release builds)
ARG USE_SCCACHE ARG USE_SCCACHE
ARG SCCACHE_DOWNLOAD_URL ARG SCCACHE_DOWNLOAD_URL
@@ -94,141 +184,35 @@ RUN if [ "$USE_SCCACHE" = "1" ]; then \
RUN if [ "$USE_SCCACHE" = "1" ]; then \ RUN if [ "$USE_SCCACHE" = "1" ]; then \
echo "Setting up sccache wrappers for HIP compilation..." \ echo "Setting up sccache wrappers for HIP compilation..." \
&& mkdir -p /opt/sccache-wrappers \ && mkdir -p /opt/sccache-wrappers \
&& printf '#!/bin/bash\nexec sccache /opt/rocm/lib/llvm/bin/clang++ "$@"\n' > /opt/sccache-wrappers/clang++ \ && printf '#!/bin/bash\nexec sccache %s/lib/llvm/bin/clang++ "$@"\n' "${ROCM_PATH}" > /opt/sccache-wrappers/clang++ \
&& chmod +x /opt/sccache-wrappers/clang++ \ && chmod +x /opt/sccache-wrappers/clang++ \
&& printf '#!/bin/bash\nexec sccache /opt/rocm/lib/llvm/bin/clang "$@"\n' > /opt/sccache-wrappers/clang \ && printf '#!/bin/bash\nexec sccache %s/lib/llvm/bin/clang "$@"\n' "${ROCM_PATH}" > /opt/sccache-wrappers/clang \
&& chmod +x /opt/sccache-wrappers/clang \ && chmod +x /opt/sccache-wrappers/clang \
&& echo "sccache wrappers created in /opt/sccache-wrappers"; \ && echo "sccache wrappers created in /opt/sccache-wrappers"; \
fi fi
# Set sccache environment variables only when USE_SCCACHE=1
# This prevents S3 config from leaking into images when sccache is not used
ARG USE_SCCACHE
ENV SCCACHE_BUCKET=${USE_SCCACHE:+${SCCACHE_BUCKET_NAME}} ENV SCCACHE_BUCKET=${USE_SCCACHE:+${SCCACHE_BUCKET_NAME}}
ENV SCCACHE_REGION=${USE_SCCACHE:+${SCCACHE_REGION_NAME}} ENV SCCACHE_REGION=${USE_SCCACHE:+${SCCACHE_REGION_NAME}}
ENV SCCACHE_S3_NO_CREDENTIALS=${USE_SCCACHE:+${SCCACHE_S3_NO_CREDENTIALS}} ENV SCCACHE_S3_NO_CREDENTIALS=${USE_SCCACHE:+${SCCACHE_S3_NO_CREDENTIALS}}
ENV SCCACHE_IDLE_TIMEOUT=${USE_SCCACHE:+0} ENV SCCACHE_IDLE_TIMEOUT=${USE_SCCACHE:+0}
###
### Triton Build
###
FROM base AS build_triton
ARG TRITON_BRANCH
ARG TRITON_REPO
RUN git clone ${TRITON_REPO}
RUN cd triton \
&& git checkout ${TRITON_BRANCH} \
&& git config --global user.email "you@example.com" && git config --global user.name "Your Name" \
&& if [ ! -f setup.py ]; then cd python; fi \
&& MAX_JOBS=16 python3 setup.py bdist_wheel --dist-dir=dist \
&& mkdir -p /app/install && cp dist/*.whl /app/install
###
### AMD SMI Build
###
FROM base AS build_amdsmi
RUN cd /opt/rocm/share/amd_smi \
&& pip wheel . --wheel-dir=dist
RUN mkdir -p /app/install && cp /opt/rocm/share/amd_smi/dist/*.whl /app/install
###
### Pytorch build
###
FROM base AS build_pytorch
ARG PYTORCH_BRANCH
ARG PYTORCH_REPO
ARG USE_SCCACHE
RUN apt-get update && apt-get install -y pkg-config liblzma-dev
# torch 2.12 emits float->__bf16 libcalls (__truncsfbf2) absent from gcc-11 libgcc; build with gcc-13.
RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-13 130 --slave /usr/bin/g++ g++ /usr/bin/g++-13
RUN git clone ${PYTORCH_REPO} pytorch
RUN cd pytorch && git checkout ${PYTORCH_BRANCH}
RUN cd pytorch \
&& pip install -r requirements.txt && git submodule update --init --recursive
RUN cd pytorch && python3 tools/amd_build/build_amd.py \
&& if [ "$USE_SCCACHE" = "1" ]; then \
export HIP_CLANG_PATH=/opt/sccache-wrappers \
&& export CMAKE_C_COMPILER_LAUNCHER=sccache \
&& export CMAKE_CXX_COMPILER_LAUNCHER=sccache \
&& sccache --show-stats; \
fi \
&& CMAKE_PREFIX_PATH=$(python3 -c 'import sys; print(sys.prefix)') python3 setup.py bdist_wheel --dist-dir=dist \
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi
RUN mkdir -p /app/install && cp /app/pytorch/dist/*.whl /app/install
###
### Shared PyTorch runtime for dependent builds
###
FROM base AS build_pytorch_runtime
RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-13 130 --slave /usr/bin/g++ g++ /usr/bin/g++-13
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install,ro \
pip install /install/*.whl
###
### TorchVision build
###
FROM build_pytorch_runtime AS build_torchvision
ARG PYTORCH_VISION_BRANCH
ARG PYTORCH_VISION_REPO
ARG USE_SCCACHE
RUN git clone ${PYTORCH_VISION_REPO} vision
RUN cd vision && git checkout ${PYTORCH_VISION_BRANCH} \
&& if [ "$USE_SCCACHE" = "1" ]; then \
export HIP_CLANG_PATH=/opt/sccache-wrappers \
&& export CMAKE_C_COMPILER_LAUNCHER=sccache \
&& export CMAKE_CXX_COMPILER_LAUNCHER=sccache; \
fi \
&& python3 setup.py bdist_wheel --dist-dir=dist \
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi
RUN mkdir -p /app/install && cp /app/vision/dist/*.whl /app/install
###
### TorchAudio build
###
FROM build_pytorch_runtime AS build_torchaudio
ARG PYTORCH_AUDIO_BRANCH
ARG PYTORCH_AUDIO_REPO
ARG USE_SCCACHE
RUN git clone ${PYTORCH_AUDIO_REPO} audio
RUN cd audio && git checkout ${PYTORCH_AUDIO_BRANCH} \
&& git submodule update --init --recursive
RUN cd audio && pip install -r requirements.txt
RUN cd audio \
&& if [ "$USE_SCCACHE" = "1" ]; then \
export HIP_CLANG_PATH=/opt/sccache-wrappers \
&& export CMAKE_C_COMPILER_LAUNCHER=sccache \
&& export CMAKE_CXX_COMPILER_LAUNCHER=sccache; \
fi \
&& python3 setup.py bdist_wheel --dist-dir=dist \
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi
RUN mkdir -p /app/install && cp /app/audio/dist/*.whl /app/install
### ###
### MORI Build ### MORI Build
### ###
FROM base AS build_mori FROM base AS build_mori
ARG MORI_BRANCH ARG MORI_BRANCH
ARG MORI_REPO ARG MORI_REPO
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
pip install /install/*.whl
RUN git clone ${MORI_REPO} RUN git clone ${MORI_REPO}
RUN cd mori \ RUN cd mori \
&& git checkout ${MORI_BRANCH} \ && git checkout ${MORI_BRANCH} \
&& git submodule update --init --recursive \ && git submodule update --init --recursive \
&& python3 setup.py bdist_wheel --dist-dir=dist && ls /app/mori/dist/*.whl && ln -sfn ${ROCM_PATH} /opt/rocm \
&& pip install "setuptools_scm>=6.2" \
&& CMAKE_PREFIX_PATH="${ROCM_PATH}:${ROCM_PATH}/lib/rocm_sysdeps" \
python3 setup.py bdist_wheel --dist-dir=dist && ls /app/mori/dist/*.whl
RUN mkdir -p /app/install && cp /app/mori/dist/*.whl /app/install RUN mkdir -p /app/install && cp /app/mori/dist/*.whl /app/install
### ###
### FlashAttention Build ### FlashAttention Build
### ###
@@ -236,8 +220,6 @@ FROM base AS build_fa
ARG FA_BRANCH ARG FA_BRANCH
ARG FA_REPO ARG FA_REPO
ARG USE_SCCACHE ARG USE_SCCACHE
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
pip install /install/*.whl
RUN git clone ${FA_REPO} RUN git clone ${FA_REPO}
RUN cd flash-attention \ RUN cd flash-attention \
&& git checkout ${FA_BRANCH} \ && git checkout ${FA_BRANCH} \
@@ -258,12 +240,12 @@ FROM base AS build_aiter
ARG AITER_BRANCH ARG AITER_BRANCH
ARG AITER_REPO ARG AITER_REPO
ARG USE_SCCACHE ARG USE_SCCACHE
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
pip install /install/*.whl
RUN git clone --recursive --branch ${AITER_BRANCH} ${AITER_REPO} RUN git clone --recursive --branch ${AITER_BRANCH} ${AITER_REPO}
RUN cd aiter \ RUN cd aiter \
&& git submodule update --init --recursive \ && git submodule update --init --recursive \
&& pip install -r requirements.txt && pip install -r requirements.txt
# AITER_USE_SYSTEM_TRITON keeps setup.py from resolving its own triton, which would
# displace the pinned one already installed in base.
RUN pip install pyyaml && cd aiter \ RUN pip install pyyaml && cd aiter \
&& if [ "$USE_SCCACHE" = "1" ]; then \ && if [ "$USE_SCCACHE" = "1" ]; then \
export HIP_CLANG_PATH=/opt/sccache-wrappers \ export HIP_CLANG_PATH=/opt/sccache-wrappers \
@@ -275,6 +257,75 @@ RUN pip install pyyaml && cd aiter \
RUN mkdir -p /app/install && cp /app/aiter/dist/*.whl /app/install RUN mkdir -p /app/install && cp /app/aiter/dist/*.whl /app/install
###
### Triton Build
###
FROM base AS build_triton
ARG TRITON_BRANCH
ARG TRITON_REPO
RUN git clone ${TRITON_REPO}
RUN cd triton \
&& git checkout ${TRITON_BRANCH} \
&& if [ ! -f setup.py ]; then cd python; fi \
&& python3 setup.py bdist_wheel --dist-dir=dist \
&& mkdir -p /app/install && cp dist/*.whl /app/install
###
### ROCr + CLR build
###
# Brings in multi-stream perf improvements; drop after these arrive in an official release
FROM base AS build_rocm_runtime
ARG ROCM_SYSTEMS_REPO
ARG ROCM_RUNTIME_COMMIT
# libgl-dev: rocclr does find_package(OpenGL REQUIRED) unconditionally and the wheel image has
# no GL headers. The link must not pick up a libGL DT_NEEDED, which is asserted after the build.
RUN apt-get update \
&& apt-get install -y --no-install-recommends g++ libdw-dev xxd libgl-dev libglx-dev \
&& rm -rf /var/lib/apt/lists/*
RUN pip install --no-cache-dir CppHeaderParser
RUN git init -q /src && cd /src \
&& git remote add origin ${ROCM_SYSTEMS_REPO} \
&& git config core.sparseCheckout true \
&& git config remote.origin.promisor true \
&& git config remote.origin.partialclonefilter blob:none \
&& git sparse-checkout init --cone \
&& git sparse-checkout set projects/rocr-runtime projects/clr projects/hip shared cmake \
&& git fetch --filter=blob:none --depth 1 origin ${ROCM_RUNTIME_COMMIT} \
&& git checkout -q FETCH_HEAD
# libelf is taken from the SDK's own vendored sysdeps rather than apt -- the wheel image has no
# /usr/include/libelf.h -- so ROCr links against the copy the wheel itself was built against.
ENV SYSDEPS="$SDK_DEV/lib/rocm_sysdeps"
# ROCr first, and installed into $SDK_DEV, because the clr build below needs its headers.
RUN cd /src/projects/rocr-runtime \
&& cmake -B build -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_PREFIX_PATH="$SDK_DEV;$SYSDEPS" \
-DCMAKE_INSTALL_PREFIX="$SDK_DEV" \
-DLIBELF_INCLUDE_DIRS="$SYSDEPS/include" \
-DLIBELF_LIBRARIES="$SYSDEPS/lib/libelf.so" \
&& cmake --build build --parallel 16 \
&& cmake --install build \
&& cmake --install build --prefix /rocr-install --strip
RUN cd /src/projects/clr \
&& cmake -B build -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCLR_BUILD_HIP=ON \
-DCLR_BUILD_OCL=OFF \
-DROCM_KPACK_ENABLED=ON \
-DHIP_COMMON_DIR=/src/projects/hip \
-DROCM_PATH="$SDK_DEV" \
-DCMAKE_PREFIX_PATH="$SDK_DEV;$SYSDEPS" \
-DCMAKE_INSTALL_PREFIX="$SDK_DEV" \
-DHIP_LLVM_ROOT="$SDK_DEV/lib/llvm" \
&& cmake --build build --parallel 16 \
&& cmake --install build --prefix /clr-install --strip
RUN mkdir -p /staging/lib \
&& cp -P /rocr-install/lib/libhsa-runtime64.so* /staging/lib/ \
&& cp -P /clr-install/lib/libamdhip64.so* /staging/lib/ \
&& ls -l /staging/lib
### ###
### Final Build ### Final Build
### ###
@@ -287,169 +338,102 @@ RUN --mount=type=bind,from=build_triton,src=/app/install/,target=/install \
cp /install/*.whl /app/debs cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_fa,src=/app/install/,target=/install \ RUN --mount=type=bind,from=build_fa,src=/app/install/,target=/install \
cp /install/*.whl /app/debs cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_amdsmi,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_torchvision,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_torchaudio,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_aiter,src=/app/install/,target=/install \ RUN --mount=type=bind,from=build_aiter,src=/app/install/,target=/install \
cp /install/*.whl /app/debs cp /install/*.whl /app/debs
# The wheel index also serves a torch repacked to require our triton instead of
# the one TheRock built it with, and amdsmi, which TheRock ships only as sources
# inside rocm-sdk-core.
ARG TORCH_VERSION
ARG ROCM_RELEASE_WHEELS_MULTIARCH_URL
ARG ROCM_NIGHTLY_WHEELS_MULTIARCH_URL
COPY tools/vllm-rocm/therock_wheels.py /tmp/therock_wheels.py
# regex and pybase64 aren't installed here; fall back to the stdlib modules.
RUN sed -i -e 's/import regex as re/import re/' \
-e 's/import pybase64 as base64/import base64/' /tmp/therock_wheels.py \
&& pip download --no-deps --dest /tmp/torch-dl \
--index-url ${ROCM_RELEASE_WHEELS_MULTIARCH_URL} \
--extra-index-url ${ROCM_NIGHTLY_WHEELS_MULTIARCH_URL} \
"torch==${TORCH_VERSION}" \
&& TRITON_VER=$(basename /app/debs/triton-*.whl | cut -d- -f2) \
&& python3 /tmp/therock_wheels.py repack-torch /tmp/torch-dl/torch-*.whl \
--triton-version "${TRITON_VER}" --out-dir /app/debs \
&& python3 /tmp/therock_wheels.py patch-amdsmi "${SDK_CORE}/share/amd_smi" \
--out-dir /tmp/amd_smi \
&& pip wheel --no-deps --wheel-dir /app/debs /tmp/amd_smi \
&& rm -rf /tmp/torch-dl /tmp/amd_smi /tmp/therock_wheels.py \
&& ls -l /app/debs
# Full debs stage - includes Mori (used by Docker releases)
FROM base AS debs FROM base AS debs
RUN mkdir /app/debs RUN mkdir /app/debs
RUN --mount=type=bind,from=build_triton,src=/app/install/,target=/install \ RUN --mount=type=bind,from=build_triton,src=/app/install/,target=/install \
cp /install/*.whl /app/debs cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_fa,src=/app/install/,target=/install \ RUN --mount=type=bind,from=build_fa,src=/app/install/,target=/install \
cp /install/*.whl /app/debs cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_amdsmi,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_torchvision,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_torchaudio,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_aiter,src=/app/install/,target=/install \ RUN --mount=type=bind,from=build_aiter,src=/app/install/,target=/install \
cp /install/*.whl /app/debs cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_mori,src=/app/install/,target=/install \ RUN --mount=type=bind,from=build_mori,src=/app/install/,target=/install \
cp /install/*.whl /app/debs cp /install/*.whl /app/debs
###
### ROCr + CLR build
###
# Pull in updates to ROCr + clr from ROCm 7.2.4 and beyond.
FROM base AS build_rocm_runtime
ARG ROCM_SYSTEMS_REPO
ARG ROCM_RUNTIME_COMMIT
RUN apt-get update \
&& apt-get install -y --no-install-recommends g++ \
libelf-dev libdrm-dev libnuma-dev libdw-dev xxd \
&& rm -rf /var/lib/apt/lists/*
RUN pip install --no-cache-dir CppHeaderParser
RUN git init -q /src && cd /src \
&& git remote add origin ${ROCM_SYSTEMS_REPO} \
&& git config core.sparseCheckout true \
&& git config remote.origin.promisor true \
&& git config remote.origin.partialclonefilter blob:none \
&& git sparse-checkout init --cone \
&& git sparse-checkout set projects/rocr-runtime projects/clr projects/hip shared cmake \
&& git fetch --filter=blob:none --depth 1 origin ${ROCM_RUNTIME_COMMIT} \
&& git checkout -q FETCH_HEAD
# ROCr's trap_handler and blit_shaders call find_package(Clang/LLVM REQUIRED) only to get
# IMPORTED executables. The ROCm dev image ships the binaries but no CMake package config, so
# supply one pointing at them.
RUN mkdir -p /opt/rocm-cmake-shim \
&& printf 'if(NOT TARGET clang)\n add_executable(clang IMPORTED GLOBAL)\n set_target_properties(clang PROPERTIES IMPORTED_LOCATION "/opt/rocm/llvm/bin/clang")\nendif()\nset(Clang_PACKAGE_VERSION "rocm-image-shim")\n' > /opt/rocm-cmake-shim/ClangConfig.cmake \
&& printf 'if(NOT TARGET llvm-objcopy)\n add_executable(llvm-objcopy IMPORTED GLOBAL)\n set_target_properties(llvm-objcopy PROPERTIES IMPORTED_LOCATION "/opt/rocm/llvm/bin/llvm-objcopy")\nendif()\nset(LLVM_FOUND TRUE)\nset(LLVM_PACKAGE_VERSION "rocm-image-shim")\n' > /opt/rocm-cmake-shim/LLVMConfig.cmake
# ROCr first, and installed into /opt/rocm, because the clr build below needs its headers
RUN cd /src/projects/rocr-runtime \
&& cmake -B build -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_PREFIX_PATH=/opt/rocm \
-DCMAKE_INSTALL_PREFIX=/opt/rocm \
-DClang_DIR=/opt/rocm-cmake-shim \
-DLLVM_DIR=/opt/rocm-cmake-shim \
&& cmake --build build --parallel 16 \
&& cmake --install build \
&& cmake --install build --prefix /rocr-install --strip
RUN cd /src/projects/clr \
&& cmake -B build -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCLR_BUILD_HIP=ON \
-DCLR_BUILD_OCL=OFF \
-DHIP_COMMON_DIR=/src/projects/hip \
-DROCM_PATH=/opt/rocm \
-DCMAKE_PREFIX_PATH=/opt/rocm \
-DCMAKE_INSTALL_PREFIX=/opt/rocm \
-DLLVM_DIR=/opt/rocm-cmake-shim \
-DHIP_LLVM_ROOT=/opt/rocm/llvm \
&& cmake --build build --parallel 16 \
&& cmake --install build --prefix /clr-install --strip
RUN mkdir -p /staging/lib \
&& cp -P /rocr-install/lib/libhsa-runtime64.so* /staging/lib/ \
&& cp -P /clr-install/lib/libamdhip64.so* /staging/lib/ \
&& ls -l /staging/lib
# rocprofiler-sdk 1.3.2 fixes torch 2.12 profiler on ROCm 7.2.3
FROM base AS build_rocprofiler_sdk
ARG ROCM_SYSTEMS_REPO
ARG ROCPROFILER_SDK_COMMIT
ARG USE_SCCACHE
RUN apt-get update \
&& apt-get install -y --no-install-recommends libdw-dev libelf-dev libsqlite3-dev libzstd-dev \
&& rm -rf /var/lib/apt/lists/*
RUN git init -q /src && cd /src \
&& git remote add origin ${ROCM_SYSTEMS_REPO} \
&& git config core.sparseCheckout true \
&& git config remote.origin.promisor true \
&& git config remote.origin.partialclonefilter blob:none \
&& git sparse-checkout init --cone \
&& git sparse-checkout set projects/rocprofiler-sdk shared cmake \
&& git fetch --filter=blob:none --depth 1 origin ${ROCPROFILER_SDK_COMMIT} \
&& git checkout -q FETCH_HEAD
# Patch the ROCm 7.2.3 build of rocprofiler-sdk to include queue-interposition fixes that are
# absent from the pinned 7.14 build. This resolves hangs on torch.profiler on ROCm.
# TODO: Drop these once ${ROCPROFILER_SDK_COMMIT} is bumped to a build that already
# contains them, such as ROCm 10.0. See https://github.com/vllm-project/vllm/issues/54087.
ARG ROCPROFILER_SDK_PR_7796=fae7fd57ffb8a76e811d6410b8a0cee1e94d710a
ARG ROCPROFILER_SDK_PR_7924=f8db22adbe17686a9b05ae03367dd1ec73011653
RUN cd /src \
&& HSA_PATHS='projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/hsa/*' \
&& curl -fsSL -o /tmp/qi-7796.patch "${ROCM_SYSTEMS_REPO%.git}/commit/${ROCPROFILER_SDK_PR_7796}.patch" \
&& curl -fsSL -o /tmp/qi-7924.patch "${ROCM_SYSTEMS_REPO%.git}/commit/${ROCPROFILER_SDK_PR_7924}.patch" \
&& git apply -p1 --include="$HSA_PATHS" --verbose /tmp/qi-7796.patch \
&& git apply -p1 --include="$HSA_PATHS" --verbose /tmp/qi-7924.patch
RUN cd /src/projects/rocprofiler-sdk \
&& if [ "$USE_SCCACHE" = "1" ]; then \
export CMAKE_C_COMPILER_LAUNCHER=sccache \
&& export CMAKE_CXX_COMPILER_LAUNCHER=sccache; \
fi \
&& cmake -B build -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_PREFIX_PATH=/opt/rocm \
-DCMAKE_INSTALL_PREFIX=/opt/rocm \
-DROCPROFILER_BUILD_TESTS=OFF \
-DROCPROFILER_BUILD_SAMPLES=OFF \
-DROCPROFILER_BUILD_DOCS=OFF \
&& cmake --build build --parallel "$(nproc)" \
&& cmake --install build --prefix /staging --strip
FROM base AS final FROM base AS final
RUN --mount=type=bind,from=debs,src=/app/debs,target=/install \ ARG TRITON_VERSION
pip install /install/*.whl
COPY --from=build_rocprofiler_sdk /staging/ /opt/rocm/ # torch 2.12 implicitly attaches rocprofiler-sdk; its queue interposition hangs at HSA
# teardown, so disable it. TODO: drop once ROCm's stock rocprofiler carries the fix.
ENV ROCPROFILER_QUEUE_INTERPOSITION=0 ENV ROCPROFILER_QUEUE_INTERPOSITION=0
COPY --from=build_rocm_runtime /staging/ /opt/rocm/ COPY --from=build_rocm_runtime /staging/lib/ ${SDK_DEV}/lib/
# Hard link the patched library onto every stock file name so that both the # Hard link the patched library onto every stock file name, in both SDK trees, so that the
# SONAME lookup and the hard coded paths resolve to it. # SONAME lookup and the hard coded paths all resolve to it. Hard links rather than symlinks
# because ldconfig rewrites SONAME symlinks but cannot re-point an inode. $SDK_CORE keeps its
# own libhsa-runtime64 copy, which is on LD_LIBRARY_PATH and in ld.so.conf; relinking it is
# belt and braces rather than strictly required, but it leaves no stock ROCr in the image.
RUN set -eux; \ RUN set -eux; \
cd /opt/rocm/lib; \ cd "$SDK_DEV/lib"; \
ours_hsa="$(readlink -f libhsa-runtime64.so.1)"; \ ours_hsa="$(readlink -f libhsa-runtime64.so.1)"; \
ours_hip="$(readlink -f libamdhip64.so.7)"; \ ours_hip="$(readlink -f libamdhip64.so.7)"; \
for f in libhsa-runtime64.so.1.*; do \ for f in "$SDK_DEV"/lib/libhsa-runtime64.so.1* "$SDK_CORE"/lib/libhsa-runtime64.so.1* \
[ "$f" = "${ours_hsa##*/}" ] || ln -f "$ours_hsa" "$f"; \ "$SDK_DEV"/lib/libamdhip64.so.7* "$SDK_CORE"/lib/libamdhip64.so.7*; do \
done; \ [ -f "$f" ] && [ ! -L "$f" ] || continue; \
for f in libamdhip64.so.7.*; do \ case "${f##*/}" in libhsa*) ours="$ours_hsa";; *) ours="$ours_hip";; esac; \
[ "$f" = "${ours_hip##*/}" ] || ln -f "$ours_hip" "$f"; \ [ "$f" -ef "$ours" ] || ln -f "$ours" "$f"; \
done done
RUN ldconfig RUN ldconfig
# This is to prevent resolvers from checking METADATA and RECORD and clobbering
# the Triton source build we installed. Constraints files are unable to help
# with this. However, these entries are optional and can be blanked which
# solves the problem.
RUN TORCH_DIST_INFO=$(ls -d ${SITE_PACKAGES}/torch-*.dist-info) \
&& grep -q '^Requires-Dist: triton' "$TORCH_DIST_INFO/METADATA" \
&& sed -i '/^Requires-Dist: triton/d' "$TORCH_DIST_INFO/METADATA" \
&& sed -i "s|^\(${TORCH_DIST_INFO##*/}/METADATA\),.*|\1,,|" "$TORCH_DIST_INFO/RECORD"
RUN --mount=type=bind,from=debs,src=/app/debs,target=/install \
pip install /install/*.whl \
&& ! pip check 2>&1 | grep -iE '^(torch|triton) ' \
&& test "$(pip show triton | awk '/^Version:/ {print $2}')" = "${TRITON_VERSION}" \
# Create a constraints file to protect the versions we need, especially
# triton, requirements will try to reinstall repeatedly.
&& pip list --format=freeze \
| grep -E '^(torch|triton|torchaudio|torchvision|pybind11)==' > /etc/rocm-constraints.txt \
&& echo "ROCm constraints:" && cat /etc/rocm-constraints.txt \
&& test "$(wc -l < /etc/rocm-constraints.txt)" -eq 5 \
&& grep -q '^pybind11==3\.0\.' /etc/rocm-constraints.txt \
&& python3 -c "import torch; assert torch.version.hip, f'non-ROCm torch: {torch.__version__}'; print(f'torch={torch.__version__}')" \
&& python3 -c "import triton; print(f'triton={triton.__version__}')" \
&& pip list 2>/dev/null | grep -iE '^(torch|triton) '
ARG BASE_IMAGE ARG BASE_IMAGE
ARG PYTHON_VERSION
ARG ROCM_SDK_VERSION
ARG TORCH_VERSION
ARG TORCHAUDIO_VERSION
ARG TORCHVISION_VERSION
ARG PYTORCH_ROCM_ARCH
ARG AITER_ROCM_ARCH
ARG TRITON_BRANCH ARG TRITON_BRANCH
ARG TRITON_REPO ARG TRITON_REPO
ARG PYTORCH_BRANCH
ARG PYTORCH_VISION_BRANCH
ARG PYTORCH_REPO
ARG PYTORCH_VISION_REPO
ARG PYTORCH_AUDIO_BRANCH
ARG PYTORCH_AUDIO_REPO
ARG FA_BRANCH ARG FA_BRANCH
ARG FA_REPO ARG FA_REPO
ARG AITER_BRANCH ARG AITER_BRANCH
@@ -457,23 +441,36 @@ ARG AITER_REPO
ARG MORI_BRANCH ARG MORI_BRANCH
ARG MORI_REPO ARG MORI_REPO
ARG ROCM_SYSTEMS_REPO ARG ROCM_SYSTEMS_REPO
ARG ROCPROFILER_SDK_COMMIT
ARG ROCM_RUNTIME_COMMIT ARG ROCM_RUNTIME_COMMIT
ARG TRITON_VERSION
# Record hashes for Triton and pybind11 so we can verify later.
RUN echo "BASE_IMAGE: ${BASE_IMAGE}" > /app/versions.txt \ RUN echo "BASE_IMAGE: ${BASE_IMAGE}" > /app/versions.txt \
&& echo "ROCM_SYSTEMS_REPO: ${ROCM_SYSTEMS_REPO}" >> /app/versions.txt \ && echo "PYTHON_VERSION: ${PYTHON_VERSION}" >> /app/versions.txt \
&& echo "ROCPROFILER_SDK_COMMIT: ${ROCPROFILER_SDK_COMMIT}" >> /app/versions.txt \ && echo "ROCM_SDK_VERSION: ${ROCM_SDK_VERSION}" >> /app/versions.txt \
&& echo "ROCM_RUNTIME_COMMIT: ${ROCM_RUNTIME_COMMIT}" >> /app/versions.txt \ && echo "TORCH_VERSION: ${TORCH_VERSION}" >> /app/versions.txt \
&& echo "TORCHAUDIO_VERSION: ${TORCHAUDIO_VERSION}" >> /app/versions.txt \
&& echo "TORCHVISION_VERSION: ${TORCHVISION_VERSION}" >> /app/versions.txt \
&& echo "PYTORCH_ROCM_ARCH: ${PYTORCH_ROCM_ARCH}" >> /app/versions.txt \
&& echo "AITER_ROCM_ARCH: ${AITER_ROCM_ARCH}" >> /app/versions.txt \
&& echo "TRITON_BRANCH: ${TRITON_BRANCH}" >> /app/versions.txt \ && echo "TRITON_BRANCH: ${TRITON_BRANCH}" >> /app/versions.txt \
&& echo "TRITON_REPO: ${TRITON_REPO}" >> /app/versions.txt \ && echo "TRITON_REPO: ${TRITON_REPO}" >> /app/versions.txt \
&& echo "PYTORCH_BRANCH: ${PYTORCH_BRANCH}" >> /app/versions.txt \
&& echo "PYTORCH_VISION_BRANCH: ${PYTORCH_VISION_BRANCH}" >> /app/versions.txt \
&& echo "PYTORCH_REPO: ${PYTORCH_REPO}" >> /app/versions.txt \
&& echo "PYTORCH_VISION_REPO: ${PYTORCH_VISION_REPO}" >> /app/versions.txt \
&& echo "PYTORCH_AUDIO_BRANCH: ${PYTORCH_AUDIO_BRANCH}" >> /app/versions.txt \
&& echo "PYTORCH_AUDIO_REPO: ${PYTORCH_AUDIO_REPO}" >> /app/versions.txt \
&& echo "FA_BRANCH: ${FA_BRANCH}" >> /app/versions.txt \ && echo "FA_BRANCH: ${FA_BRANCH}" >> /app/versions.txt \
&& echo "FA_REPO: ${FA_REPO}" >> /app/versions.txt \ && echo "FA_REPO: ${FA_REPO}" >> /app/versions.txt \
&& echo "AITER_BRANCH: ${AITER_BRANCH}" >> /app/versions.txt \ && echo "AITER_BRANCH: ${AITER_BRANCH}" >> /app/versions.txt \
&& echo "AITER_REPO: ${AITER_REPO}" >> /app/versions.txt \ && echo "AITER_REPO: ${AITER_REPO}" >> /app/versions.txt \
&& echo "MORI_BRANCH: ${MORI_BRANCH}" >> /app/versions.txt \ && echo "MORI_BRANCH: ${MORI_BRANCH}" >> /app/versions.txt \
&& echo "MORI_REPO: ${MORI_REPO}" >> /app/versions.txt && echo "MORI_REPO: ${MORI_REPO}" >> /app/versions.txt \
&& echo "ROCM_SYSTEMS_REPO: ${ROCM_SYSTEMS_REPO}" >> /app/versions.txt \
&& echo "ROCM_RUNTIME_COMMIT: ${ROCM_RUNTIME_COMMIT}" >> /app/versions.txt \
# Fingerprint our triton so Dockerfile.rocm can prove it survived the build.
&& echo "TRITON_VERSION: $(pip show triton | awk '/^Version:/ {print $2}')" >> /app/versions.txt \
&& echo "TRITON_SHA256: $(find $(python3 -c 'import triton, os; print(os.path.dirname(triton.__file__))') \
-type f \( -name '*.py' -o -name '*.so' \) | sort | xargs sha256sum | sha256sum | cut -d' ' -f1)" \
>> /app/versions.txt \
&& echo "PYBIND11_VERSION: $(pip show pybind11 | awk '/^Version:/ {print $2}')" >> /app/versions.txt \
&& echo "PYBIND11_INTERNALS_VERSION: $(grep -rhE '^#[[:space:]]*define PYBIND11_INTERNALS_VERSION' \
$(python3 -c 'import pybind11; print(pybind11.get_include())') | awk '{print $NF}')" \
>> /app/versions.txt \
&& echo "PYBIND11_SHA256: $(find $(python3 -c 'import pybind11; print(pybind11.get_include())') \
-type f -name '*.h' | sort | xargs sha256sum | sha256sum | cut -d' ' -f1)" \
>> /app/versions.txt
+5 -29
View File
@@ -83,10 +83,6 @@ variable "UCX_BRANCH" {
default = "" default = ""
} }
variable "ROCSHMEM_BRANCH" {
default = ""
}
variable "DEEPEP_BRANCH" { variable "DEEPEP_BRANCH" {
default = "" default = ""
} }
@@ -95,10 +91,6 @@ variable "NIXL_CACHE_KEY" {
default = "" default = ""
} }
variable "ROCSHMEM_CACHE_KEY" {
default = ""
}
variable "DEEPEP_CACHE_KEY" { variable "DEEPEP_CACHE_KEY" {
default = "" default = ""
} }
@@ -236,7 +228,7 @@ function "get_cache_to_rocm_rust" {
]) ])
} }
# Cache functions for upstream dependency stages (NIXL/UCX, ROCShmem, DeepEP). # Cache functions for upstream dependency stages (NIXL/UCX, DeepEP).
# These stages are pinned to specific upstream commit hashes, so cache keys use # These stages are pinned to specific upstream commit hashes, so cache keys use
# those hashes rather than the Buildkite commit. This means the cache persists # those hashes rather than the Buildkite commit. This means the cache persists
# across all vLLM commits as long as the upstream dependency pins don't change. # across all vLLM commits as long as the upstream dependency pins don't change.
@@ -245,8 +237,7 @@ function "get_cache_from_rocm_deps" {
params = [] params = []
result = compact([ result = compact([
NIXL_CACHE_KEY != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:nixl-rocm-${NIXL_CACHE_KEY}" : (NIXL_BRANCH != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:nixl-rocm-${NIXL_BRANCH}-ucx-${UCX_BRANCH}" : ""), NIXL_CACHE_KEY != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:nixl-rocm-${NIXL_CACHE_KEY}" : (NIXL_BRANCH != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:nixl-rocm-${NIXL_BRANCH}-ucx-${UCX_BRANCH}" : ""),
ROCSHMEM_CACHE_KEY != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:rocshmem-rocm-${ROCSHMEM_CACHE_KEY}" : (ROCSHMEM_BRANCH != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:rocshmem-rocm-${ROCSHMEM_BRANCH}" : ""), DEEPEP_CACHE_KEY != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:deepep-rocm-${DEEPEP_CACHE_KEY}" : (DEEPEP_BRANCH != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:deepep-rocm-${DEEPEP_BRANCH}" : ""),
DEEPEP_CACHE_KEY != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:deepep-rocm-${DEEPEP_CACHE_KEY}" : (DEEPEP_BRANCH != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:deepep-rocm-${DEEPEP_BRANCH}-rocshmem-${ROCSHMEM_BRANCH}" : ""),
]) ])
} }
@@ -257,17 +248,10 @@ function "get_cache_to_rocm_nixl" {
]) ])
} }
function "get_cache_to_rocm_rocshmem" {
params = []
result = compact([
ROCSHMEM_CACHE_KEY != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:rocshmem-rocm-${ROCSHMEM_CACHE_KEY},mode=min" : (ROCSHMEM_BRANCH != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:rocshmem-rocm-${ROCSHMEM_BRANCH},mode=min" : ""),
])
}
function "get_cache_to_rocm_deepep" { function "get_cache_to_rocm_deepep" {
params = [] params = []
result = compact([ result = compact([
DEEPEP_CACHE_KEY != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:deepep-rocm-${DEEPEP_CACHE_KEY},mode=min" : (DEEPEP_BRANCH != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:deepep-rocm-${DEEPEP_BRANCH}-rocshmem-${ROCSHMEM_BRANCH},mode=min" : ""), DEEPEP_CACHE_KEY != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:deepep-rocm-${DEEPEP_CACHE_KEY},mode=min" : (DEEPEP_BRANCH != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:deepep-rocm-${DEEPEP_BRANCH},mode=min" : ""),
]) ])
} }
@@ -385,14 +369,6 @@ target "nixl-rocm-ci" {
output = ["type=cacheonly"] output = ["type=cacheonly"]
} }
target "rocshmem-rocm-ci" {
inherits = ["_common-rocm", "_ci-rocm"]
target = "build_rocshmem"
cache-from = get_cache_from_rocm_deps()
cache-to = get_cache_to_rocm_rocshmem()
output = ["type=cacheonly"]
}
target "deepep-rocm-ci" { target "deepep-rocm-ci" {
inherits = ["_common-rocm", "_ci-rocm"] inherits = ["_common-rocm", "_ci-rocm"]
target = "build_deepep" target = "build_deepep"
@@ -418,7 +394,7 @@ target "ci-base-rocm-ci" {
CI_BASE_IMAGE_TAG != "" ? "type=registry,ref=${CI_BASE_IMAGE_TAG}" : "", CI_BASE_IMAGE_TAG != "" ? "type=registry,ref=${CI_BASE_IMAGE_TAG}" : "",
CI_BASE_TRUSTED_CONTENT_REF != "" ? "type=registry,ref=${CI_BASE_TRUSTED_CONTENT_REF}" : "", CI_BASE_TRUSTED_CONTENT_REF != "" ? "type=registry,ref=${CI_BASE_TRUSTED_CONTENT_REF}" : "",
]), ]),
# Import upstream dependency caches so NIXL/ROCShmem/DeepEP stages # Import upstream dependency caches so NIXL/DeepEP stages
# are cache hits even when ci_base itself needs rebuilding. # are cache hits even when ci_base itself needs rebuilding.
get_cache_from_rocm_deps(), get_cache_from_rocm_deps(),
) )
@@ -431,5 +407,5 @@ target "ci-base-rocm-ci" {
# Group for ci_base builds -- exports dependency stage caches alongside the # Group for ci_base builds -- exports dependency stage caches alongside the
# ci_base image so future rebuilds can reuse them independently. # ci_base image so future rebuilds can reuse them independently.
group "ci-base-rocm-ci-with-deps" { group "ci-base-rocm-ci-with-deps" {
targets = ["nixl-rocm-ci", "rocshmem-rocm-ci", "deepep-rocm-ci", "ci-base-rocm-ci"] targets = ["nixl-rocm-ci", "deepep-rocm-ci", "ci-base-rocm-ci"]
} }
-4
View File
@@ -65,10 +65,6 @@ variable "UCX_BRANCH" {
default = "" default = ""
} }
variable "ROCSHMEM_BRANCH" {
default = ""
}
variable "DEEPEP_BRANCH" { variable "DEEPEP_BRANCH" {
default = "" default = ""
} }
+1 -1
View File
@@ -136,7 +136,7 @@ vllm bench {forward, latency, serve, throughput}
### forward ### forward
Benchmark the model forward of exact batches: each case lists its requests as `[N*]Q[@K][c][:D]` (query length Q, KV length K including the query, D draft tokens). A prefill's prefix is a prefix-cache hit unless `c` marks a chunked prefill; they differ only on models with SWA bounded replay, which recompute the last window of a hit. Each request's prefix length varies by ±50% by default (`--kv-len-jitter`), identically on every DP rank. Each forward is captured as a one-shot CUDA graph; sampling and CPU overhead aren't measured. Use `torchrun`, one process per GPU, for DP/TP/EP. CUDA only for now. Benchmark the model forward of exact batches: each case lists its requests as `[N*]Q[@K][c][:D]` (query length Q, KV length K including the query, D draft tokens). A prefill's prefix is a prefix-cache hit unless `c` marks a chunked prefill; they differ only on models with SWA bounded replay, which recompute the last window of a hit. Each request's prefix length varies by ±50% by default (`--kv-len-jitter`), identically on every DP rank. Each forward is captured as a one-shot CUDA graph; sampling and CPU overhead aren't measured. Use `torchrun`, one process per GPU, for DP/TP/EP. CUDA only for now. Add `--profile-dir DIR` to write a Perfetto trace and a per-kernel GPU time summary of each case; `python -m vllm.benchmarks.forward.profile A.kernels.json [B.kernels.json]` prints a summary or compares two. Under Nsight Systems, `nsys profile --capture-range=cudaProfilerApi --capture-range-end=repeat --cuda-graph-trace=node` records just those replays, one report per case.
```bash ```bash
vllm bench forward \ vllm bench forward \
+29
View File
@@ -411,6 +411,35 @@ Example (OpenAI-compatible completions request):
} }
``` ```
## Explicit pool sizing for `SimpleCPUOffloadConnector`
`SimpleCPUOffloadConnector` accepts an optional `num_cpu_blocks` in
`kv_connector_extra_config`. The scheduler and every worker use this exact
number of offload blocks instead of automatic sizing from
`cpu_bytes_to_use_per_rank` or `cpu_bytes_to_use / world_size`.
Use the same positive integer on all ranks, chosen
to fit the smallest per-worker capacity. This supports pipeline stages with
different KV cache sizes without adding a collective during initialization.
```json
{
"kv_connector": "SimpleCPUOffloadConnector",
"kv_role": "kv_both",
"kv_connector_extra_config": {
"cpu_bytes_to_use_per_rank": 1073741824,
"num_cpu_blocks": 64,
"lazy_offload": false
}
}
```
The example count must be adjusted for the model, KV layout, and parallelism.
The configured byte budget remains an upper bound: startup fails if the
requested count exceeds a worker's capacity. For the `disk` backend the count
selects disk slots and is checked against `disk_capacity_bytes`. Omitting
`num_cpu_blocks` preserves automatic per-worker sizing. This option applies
to `SimpleCPUOffloadConnector`, separately from `OffloadingConnector` above.
## Further Reading ## Further Reading
- [vLLM blog: KV Offloading Connector](https://vllm.ai/blog/2026-01-08-kv-offloading-connector) — motivation, architecture (DMA-based async transfer), and benchmarks (TTFT and throughput). - [vLLM blog: KV Offloading Connector](https://vllm.ai/blog/2026-01-08-kv-offloading-connector) — motivation, architecture (DMA-based async transfer), and benchmarks (TTFT and throughput).
+8
View File
@@ -136,6 +136,14 @@ WRITE mode is used by default. READ mode can be configured by setting `--kv-tran
!!! note !!! note
`notify_port` is used as a *base* port: each (DP rank, TP rank) pair within an instance uses `notify_port + offset` where the offset is based on the rank. Make sure the range starting at `notify_port` is free on the host. `notify_port` is used as a *base* port: each (DP rank, TP rank) pair within an instance uses `notify_port + offset` where the offset is based on the rank. Make sure the range starting at `notify_port` is free on the host.
### Hybrid attention and recurrent state
WRITE mode supports hybrid attention plus Mamba/KDA models, including DSpark, with equal producer and consumer TP sizes. Use matching cache layouts on both instances, `--mamba-cache-mode align`, and `read_mode: false` in each instance's `kv_connector_extra_config`. Both peers must support hybrid WRITE. Sliding-window WRITE and hybrid speculative methods other than DSpark remain unsupported.
For DSpark, configure the same draft model on both instances through `--speculative-config`. Target and draft attention groups retain their own block IDs and token spans; only the running recurrent state is transferred, not speculative scratch slots. The producer transfers the state after the first N-1 prompt tokens, and the consumer recomputes the last prompt token. Convolution and SSM transfers must both finish before the layer is complete.
Hybrid READ keeps its existing single-transferable-attention-group requirement. Plain-attention WRITE retains its existing notification format.
### Transport configuration ### Transport configuration
MoRI has two transport backends: RDMA and xGMI. You can select backend using `--kv-transfer-config.kv_connector_extra_config.backend $BACKEND`, with `$BACKEND` being `rdma` or `xgmi`. RDMA is the default backend and should be used in multi-node deployments. MoRI has two transport backends: RDMA and xGMI. You can select backend using `--kv-transfer-config.kv_connector_extra_config.backend $BACKEND`, with `$BACKEND` being `rdma` or `xgmi`. RDMA is the default backend and should be used in multi-node deployments.
+3 -3
View File
@@ -187,7 +187,7 @@ Multi-image input can be extended to perform video captioning. We show this with
#### Custom RGBA Background Color #### Custom RGBA Background Color
When loading RGBA images (images with transparency), vLLM converts them to RGB format. By default, transparent pixels are replaced with white background. You can customize this background color using the `rgba_background_color` parameter in `media_io_kwargs`. When loading images with transparency (including RGBA images and PNGs with a tRNS chunk), vLLM converts them to RGB format. By default, transparent pixels are replaced with white background. You can customize this background color using the `rgba_background_color` parameter in `media_io_kwargs`.
??? code ??? code
@@ -212,7 +212,7 @@ When loading RGBA images (images with transparency), vLLM converts them to RGB f
!!! note !!! note
- The `rgba_background_color` accepts RGB values as a list `[R, G, B]` or tuple `(R, G, B)` where each value is 0-255 - The `rgba_background_color` accepts RGB values as a list `[R, G, B]` or tuple `(R, G, B)` where each value is 0-255
- This setting only affects RGBA images with transparency; RGB images are unchanged - This setting affects images with transparency, including RGB PNGs with a transparent color; opaque RGB images are unchanged
- If not specified, the default white background `(255, 255, 255)` is used for backward compatibility - If not specified, the default white background `(255, 255, 255)` is used for backward compatibility
#### Moondream3 Prompt Recipes { #moondream3-prompt-recipes } #### Moondream3 Prompt Recipes { #moondream3-prompt-recipes }
@@ -1089,7 +1089,7 @@ By passing this metadata, the model can better understand the temporal distribut
#### Custom RGBA Background Color #### Custom RGBA Background Color
To use a custom background color for RGBA images, pass the `rgba_background_color` parameter via `--media-io-kwargs`: To use a custom background color for images with transparency, pass the `rgba_background_color` parameter via `--media-io-kwargs`:
```bash ```bash
# Example: Black background for dark theme # Example: Black background for dark theme
+24
View File
@@ -201,6 +201,30 @@ checkpoint's output format.
!!! note !!! note
With `tool_choice="auto"`, structural-tag constraints require both `VLLM_ENFORCE_STRICT_TOOL_CALLING=true` (the default) and at least one tool with `strict: true`, or a server-side floor set via `--tool-strict-level`. When these conditions are met and the selected parser supports structural tags, vLLM constrains the tool-call envelope and pins the argument schema of each tool that sets `strict: true` (or of every tool under `--tool-strict-level parameter`). Otherwise, vLLM extracts tool calls from raw text, so arguments may occasionally be malformed or violate the function's parameter schema. With `tool_choice="auto"`, structural-tag constraints require both `VLLM_ENFORCE_STRICT_TOOL_CALLING=true` (the default) and at least one tool with `strict: true`, or a server-side floor set via `--tool-strict-level`. When these conditions are met and the selected parser supports structural tags, vLLM constrains the tool-call envelope and pins the argument schema of each tool that sets `strict: true` (or of every tool under `--tool-strict-level parameter`). Otherwise, vLLM extracts tool calls from raw text, so arguments may occasionally be malformed or violate the function's parameter schema.
### Beam Models (`beam`)
Beam checkpoints describe their output format with `response_template` metadata, so the `beam` parser reads it like the `hf` parser does:
```bash
vllm serve <model> \
--enable-auto-tool-choice \
--tool-call-parser beam \
--reasoning-parser beam
```
Unlike `hf`, the `beam` parser also constrains its output. For strict tools, required or named tool choice, and structured outputs, it derives a
structural-tag grammar from the template. The grammar follows the template's tool call format, pins the arguments of strict tools when their
encoding can be derived, allows a single call with `parallel_tool_calls=false`, and frames a structured output as the answer. Raw string arguments
keep the `format` or `pattern` of their schema, as XGrammar keeps them in JSON strings, unless a match could end the value early. With
`tool_choice="none"`, the grammar allows only reasoning and an answer, so a model that still sees the tools in its prompt cannot call them. When no
grammar can be derived, required and named tool choice are rejected, and strict tools under `tool_choice="auto"` are left unconstrained with a
warning.
It also constrains the output of requests without strict tools, forced tool choice, or structured outputs to the template's format: tool calls with
free arguments or an answer, or just the rest of an answer the prompt already opened. This always-on grammar needs the `xgrammar` structured
outputs backend, the default, and is disabled with `VLLM_BEAM_ALWAYS_CONSTRAIN_OUTPUT=0`. From the message start, every `beam` grammar opens with
reasoning that holds visible text, unless `enable_thinking` is false.
### Hermes Models (`hermes`) ### Hermes Models (`hermes`)
All Nous Research Hermes-series models newer than Hermes 2 Pro should be supported. All Nous Research Hermes-series models newer than Hermes 2 Pro should be supported.
@@ -1,7 +1,7 @@
<!-- markdownlint-disable MD041 MD051 --> <!-- markdownlint-disable MD041 MD051 -->
--8<-- [start:installation] --8<-- [start:installation]
vLLM supports AMD GPUs with ROCm 6.3 or above. Pre-built wheels are available for ROCm 7.0 and ROCm 7.2.1. vLLM supports AMD GPUs with ROCm 6.3 or above. Pre-built wheels are available for ROCm 10.0 (the default) and ROCm 7.2.
#### Prebuilt Wheels #### Prebuilt Wheels
@@ -9,6 +9,8 @@ vLLM supports AMD GPUs with ROCm 6.3 or above. Pre-built wheels are available fo
| ------------ | -------------- | ------------ | ----------------- | ------------------ | | ------------ | -------------- | ------------ | ----------------- | ------------------ |
| `rocm700` | 3.12 | 7.0 | >= 2.35 | `0.14.0` to `0.18.0` | | `rocm700` | 3.12 | 7.0 | >= 2.35 | `0.14.0` to `0.18.0` |
| `rocm721` | 3.12 | 7.2.1 | >= 2.35 | Nightly releases after commit `171775f306a333a9cf105bfd533bf3e113d401d9` | | `rocm721` | 3.12 | 7.2.1 | >= 2.35 | Nightly releases after commit `171775f306a333a9cf105bfd533bf3e113d401d9` |
| `rocm72` | 3.12 | 7.2 | >= 2.39 | Nightly releases on the legacy ROCm 7.2 stack |
| `rocm100` | 3.12 | 10.0 | >= 2.35 | Nightly releases (default) |
--8<-- [end:installation] --8<-- [end:installation]
--8<-- [start:requirements] --8<-- [start:requirements]
@@ -40,12 +42,17 @@ If you need a different ROCm version or want to use an existing PyTorch installa
source .venv/bin/activate source .venv/bin/activate
``` ```
To install the latest version of vLLM for Python 3.12, ROCm 7.0 and `glibc >= 2.35`. To install the latest version of vLLM for Python 3.12, ROCm 10.0 and `glibc >= 2.35`:
```bash ```bash
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/ --upgrade uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/ --upgrade
``` ```
!!! tip
The ROCm 10.0 wheels install the ROCm SDK from pip ([TheRock](https://github.com/ROCm/TheRock)), including the GPU kernels for every supported architecture.
Wheels for the legacy ROCm 7.2 stack, which use a system ROCm installation, remain available at `https://wheels.vllm.ai/rocm/${VLLM_VERSION}/rocm72`.
!!! tip !!! tip
You can find out about which ROCm version the latest vLLM supports by checking the `vllm` package in index in extra-index-url <https://wheels.vllm.ai/rocm/> at [https://wheels.vllm.ai/rocm/vllm](https://wheels.vllm.ai/rocm/vllm) . You can find out about which ROCm version the latest vLLM supports by checking the `vllm` package in index in extra-index-url <https://wheels.vllm.ai/rocm/> at [https://wheels.vllm.ai/rocm/vllm](https://wheels.vllm.ai/rocm/vllm) .
@@ -195,9 +202,9 @@ install it without re-running the full installation:
and place it in `vllm/vllm-rs`. and place it in `vllm/vllm-rs`.
If you see an error about a wheel not being found, the wheel for your base If you see an error about a wheel not being found, the wheel for your base
commit and ROCm patch version might not be available. Check the available commit and ROCm version might not be available. Check the available
variants under `https://wheels.vllm.ai/rocm/<commit>/`. For example, ROCm 7.2.1 variants under `https://wheels.vllm.ai/rocm/<commit>/`. Variants are named by ROCm
uses the `rocm721` variant. major and minor version, e.g. ROCm 7.2.3 uses the `rocm72` variant.
There are more environment variables to control the behavior of Python-only There are more environment variables to control the behavior of Python-only
build: build:
@@ -208,7 +215,7 @@ build:
- `VLLM_PRECOMPILED_WHEEL_COMMIT`: override the full commit hash used to - `VLLM_PRECOMPILED_WHEEL_COMMIT`: override the full commit hash used to
download the pre-compiled wheel. download the pre-compiled wheel.
- `VLLM_PRECOMPILED_WHEEL_VARIANT`: specify the ROCm variant subdirectory, e.g., - `VLLM_PRECOMPILED_WHEEL_VARIANT`: specify the ROCm variant subdirectory, e.g.,
`rocm700` or `rocm721`. If not specified, the variant is auto-detected based `rocm72` or `rocm100`. If not specified, the variant is auto-detected based
on your system's ROCm version. An explicitly specified variant must match the on your system's ROCm version. An explicitly specified variant must match the
detected environment. detected environment.
@@ -226,23 +233,34 @@ You can find more information about vLLM's wheels in
#### Full build (with compilation) {#full-build} #### Full build (with compilation) {#full-build}
!!! tip !!! tip
- If you found that the following installation step does not work for you, please refer to [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base). Dockerfile is a form of installation steps. - The steps below follow [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base), which pins the validated version of every component. If a step does not work for you, refer to it; a Dockerfile is a form of installation steps.
- The legacy ROCm 7.2 stack uses a system ROCm installation instead; its steps are in [docker/Dockerfile.rocm_72_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_72_base).
0. Install prerequisites (skip if you are already in an environment/docker with the following installed): 0. Install the ROCm SDK and PyTorch (skip if you are already in an environment or docker image with them installed, e.g. `vllm/vllm-openai-rocm:base-nightly`).
- [ROCm](https://rocm.docs.amd.com/en/latest/deploy/linux/index.html) ROCm 10.0 is installed from pip ([TheRock](https://github.com/ROCm/TheRock)), together with PyTorch and the GPU kernels for your GPU's architecture. In a Python 3.12 environment:
- [PyTorch](https://pytorch.org/)
For installing PyTorch, you can start from a fresh docker image, e.g, `rocm/pytorch:rocm7.0_ubuntu22.04_py3.10_pytorch_release_2.8.0`, `rocm/pytorch-nightly`. If you are using docker image, you can skip to Step 3.
Alternatively, you can install PyTorch using PyTorch wheels. You can check PyTorch installation guide in PyTorch [Getting Started](https://pytorch.org/get-started/locally/). Example:
```bash ```bash
# Install PyTorch # Your GPU's architecture, e.g. gfx942 for MI300 or gfx950 for MI350
pip uninstall torch -y export GPU_ARCH=$(uvx --from rocm-bootstrap rocm-bootstrap-detect --unique)
pip install --no-cache-dir torch torchvision --index-url https://download.pytorch.org/whl/nightly/rocm7.0 export ROCM_INDEX=https://stable.repo.amd.com/rocm/whl-next/
pip install --index-url ${ROCM_INDEX} \
"rocm[libraries,devel,device-${GPU_ARCH}]==10.0.0" \
"torch[device-${GPU_ARCH}]==2.12.0+rocm10.0.0" \
"torchvision[device-${GPU_ARCH}]==0.27.0+rocm10.0.0"
rocm-sdk init
# Point builds at the pip-installed ROCm SDK
export ROCM_PATH=$(rocm-sdk path --root)
export ROCM_HOME=${ROCM_PATH} HIP_PATH=${ROCM_PATH}
export PATH=$(rocm-sdk path --bin):${PATH}
``` ```
!!! note
- The validated `ROCM_SDK_VERSION`, `TORCH_VERSION` and `TORCHVISION_VERSION` can be found in [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base).
- To support several GPU architectures, add a `device-<arch>` extra for each of them.
1. Install [Triton for ROCm](https://github.com/ROCm/triton.git) 1. Install [Triton for ROCm](https://github.com/ROCm/triton.git)
Install ROCm's Triton following the instructions from [ROCm/triton](https://github.com/ROCm/triton.git) Install ROCm's Triton following the instructions from [ROCm/triton](https://github.com/ROCm/triton.git)
@@ -252,8 +270,7 @@ You can find more information about vLLM's wheels in
pip uninstall -y triton pip uninstall -y triton
git clone https://github.com/ROCm/triton.git git clone https://github.com/ROCm/triton.git
cd triton cd triton
# git checkout $TRITON_BRANCH git checkout $TRITON_BRANCH
git checkout f9e5bf54
if [ ! -f setup.py ]; then cd python; fi if [ ! -f setup.py ]; then cd python; fi
python3 setup.py install python3 setup.py install
cd ../.. cd ../..
@@ -265,17 +282,14 @@ You can find more information about vLLM's wheels in
2. Optionally, if you choose to use CK flash attention, you can install [flash attention for ROCm](https://github.com/Dao-AILab/flash-attention.git) 2. Optionally, if you choose to use CK flash attention, you can install [flash attention for ROCm](https://github.com/Dao-AILab/flash-attention.git)
Install ROCm's flash attention (v2.8.0) following the instructions from [ROCm/flash-attention](https://github.com/Dao-AILab/flash-attention#amd-rocm-support) Install ROCm's flash attention following the instructions from [ROCm/flash-attention](https://github.com/Dao-AILab/flash-attention#amd-rocm-support)
For example, for ROCm 7.0, suppose your gfx arch is `gfx942`. To get your gfx architecture, run `rocminfo |grep gfx`.
```bash ```bash
git clone https://github.com/Dao-AILab/flash-attention.git git clone https://github.com/Dao-AILab/flash-attention.git
cd flash-attention cd flash-attention
# git checkout $FA_BRANCH git checkout $FA_BRANCH
git checkout 0e60e394
git submodule update --init git submodule update --init
GPU_ARCHS="gfx942" python3 setup.py install GPU_ARCHS="${GPU_ARCH}" python3 setup.py install
cd .. cd ..
``` ```
@@ -290,7 +304,8 @@ You can find more information about vLLM's wheels in
cd aiter cd aiter
git checkout $AITER_BRANCH_OR_COMMIT git checkout $AITER_BRANCH_OR_COMMIT
git submodule sync; git submodule update --init --recursive git submodule sync; git submodule update --init --recursive
python3 setup.py develop # Keep the Triton installed in step 1
AITER_USE_SYSTEM_TRITON=1 python3 setup.py develop
``` ```
!!! note !!! note
@@ -300,32 +315,34 @@ You can find more information about vLLM's wheels in
4. Optionally, if you want to use MORI for EP or PD disaggregation, you can install [MORI](https://github.com/ROCm/mori) using the following steps: 4. Optionally, if you want to use MORI for EP or PD disaggregation, you can install [MORI](https://github.com/ROCm/mori) using the following steps:
```bash ```bash
pip install "setuptools_scm>=6.2"
git clone https://github.com/ROCm/mori.git git clone https://github.com/ROCm/mori.git
cd mori cd mori
git checkout $MORI_BRANCH_OR_COMMIT git checkout $MORI_BRANCH_OR_COMMIT
git submodule sync; git submodule update --init --recursive git submodule sync; git submodule update --init --recursive
MORI_GPU_ARCHS="gfx942;gfx950" python3 setup.py install # The ROCm SDK keeps its CMake configs for system dependencies (e.g. NUMA) under rocm_sysdeps
CMAKE_PREFIX_PATH="${ROCM_PATH}:${ROCM_PATH}/lib/rocm_sysdeps" \
MORI_GPU_ARCHS="gfx942;gfx950" python3 setup.py install
``` ```
!!! note !!! note
- You will need to config the `$MORI_BRANCH_OR_COMMIT` for your purpose. - You will need to config the `$MORI_BRANCH_OR_COMMIT` for your purpose.
- The validated `$MORI_BRANCH_OR_COMMIT` can be found in the [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base). - The validated `$MORI_BRANCH_OR_COMMIT` can be found in the [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base).
5. Build vLLM. For example, vLLM on ROCM 7.0 can be built with the following steps: 5. Build vLLM. For example, vLLM on ROCm 10.0 can be built with the following steps:
???+ console "Commands" ???+ console "Commands"
```bash ```bash
pip install --upgrade pip pip install --upgrade pip
# Build & install AMD SMI # Install AMD SMI, which the ROCm SDK ships as sources
pip install /opt/rocm/share/amd_smi SDK_CORE=$(python3 -c "import _rocm_sdk_core, os; print(os.path.dirname(_rocm_sdk_core.__file__))")
pip install ${SDK_CORE}/share/amd_smi
# It loads the SDK's libamd_smi relative to its own directory
export PYTHONPATH=${SDK_CORE}/share/amd_smi
# Install dependencies # Install dependencies
pip install --upgrade numba \
scipy \
huggingface-hub \
setuptools_scm
pip install -r requirements/rocm.txt pip install -r requirements/rocm.txt
# To build for a single architecture (e.g., MI300) for faster installation (recommended): # To build for a single architecture (e.g., MI300) for faster installation (recommended):
@@ -352,8 +369,9 @@ You can find more information about vLLM's wheels in
vLLM offers official Docker images for deployment. vLLM offers official Docker images for deployment.
The images can be used to run OpenAI compatible server and are available on Docker Hub as [vllm/vllm-openai-rocm](https://hub.docker.com/r/vllm/vllm-openai-rocm/tags). The images can be used to run OpenAI compatible server and are available on Docker Hub as [vllm/vllm-openai-rocm](https://hub.docker.com/r/vllm/vllm-openai-rocm/tags).
- `vllm/vllm-openai-rocm:latest` — stable release - `vllm/vllm-openai-rocm:latest` — stable release, built on ROCm 10.0 (also tagged `latest-rocm100`)
- `vllm/vllm-openai-rocm:nightly` — preview build from the latest development branch, use this if you want the latest features and fixes - `vllm/vllm-openai-rocm:nightly` — preview build from the latest development branch on ROCm 10.0 (also tagged `nightly-rocm100`), use this if you want the latest features and fixes
- `vllm/vllm-openai-rocm:latest-rocm72` / `nightly-rocm72` — the same builds on the legacy ROCm 7.2 stack, kept for a transition period
```bash ```bash
docker run --rm \ docker run --rm \
@@ -409,7 +427,7 @@ You can build and run vLLM from source via the provided [docker/Dockerfile.rocm]
??? info "(Optional) Build an image with ROCm software stack" ??? info "(Optional) Build an image with ROCm software stack"
Build a docker image from [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base) which setup ROCm software stack needed by the vLLM. Build a docker image from [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base) which setup ROCm software stack needed by the vLLM.
**This step is optional as this rocm_base image is usually prebuilt and store at [Docker Hub](https://hub.docker.com/r/rocm/vllm-dev) under tag `rocm/vllm-dev:base` to speed up user experience.** **This step is optional as this rocm_base image is usually prebuilt and store at [Docker Hub](https://hub.docker.com/r/rocm/vllm-dev) under tag `rocm/vllm-dev:base` to speed up user experience.** For the legacy ROCm 7.2 stack, the base image is built from [docker/Dockerfile.rocm_72_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_72_base) and published as `vllm/vllm-openai-rocm:base-nightly-rocm72`.
If you choose to build this rocm_base image yourself, the steps are as follows. If you choose to build this rocm_base image yourself, the steps are as follows.
It is important that the user kicks off the docker build using buildkit. Either the user put `DOCKER_BUILDKIT=1` as environment variable when calling docker build command, or the user needs to set up buildkit in the docker daemon configuration `/etc/docker/daemon.json` as follows and restart the daemon: It is important that the user kicks off the docker build using buildkit. Either the user put `DOCKER_BUILDKIT=1` as environment variable when calling docker build command, or the user needs to set up buildkit in the docker daemon configuration `/etc/docker/daemon.json` as follows and restart the daemon:
@@ -422,7 +440,7 @@ You can build and run vLLM from source via the provided [docker/Dockerfile.rocm]
} }
``` ```
To build vllm on ROCm 7.0 for MI200 and MI300 series, you can use the default: To build vllm on ROCm 10.0 for all supported GPUs, you can use the default:
```bash ```bash
DOCKER_BUILDKIT=1 docker build \ DOCKER_BUILDKIT=1 docker build \
@@ -441,15 +459,15 @@ It is important that the user kicks off the docker build using buildkit. Either
} }
``` ```
[docker/Dockerfile.rocm](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm) uses ROCm 7.0 by default, but also supports ROCm 5.7, 6.0, 6.1, 6.2, 6.3, and 6.4, in older vLLM branches. [docker/Dockerfile.rocm](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm) uses ROCm 10.0 (installed from TheRock wheels) by default. The legacy ROCm 7.2 stack is available as [docker/Dockerfile.rocm_72](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_72) with [docker/Dockerfile.rocm_72_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_72_base); older vLLM branches support ROCm 5.7 through 7.0.
It provides flexibility to customize the build of docker image using the following arguments: It provides flexibility to customize the build of docker image using the following arguments:
- `BASE_IMAGE`: specifies the base image used when running `docker build`. The default value `rocm/vllm-dev:base` is an image published and maintained by AMD. It is being built using [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base) - `BASE_IMAGE`: specifies the base image used when running `docker build`. The default value `rocm/vllm-dev:base` is an image published and maintained by AMD. It is being built using [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base). For [docker/Dockerfile.rocm_72](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_72), the default is `vllm/vllm-openai-rocm:base-nightly-rocm72`
- `ARG_PYTORCH_ROCM_ARCH`: Allows to override the gfx architecture values from the base docker image - `ARG_PYTORCH_ROCM_ARCH`: Allows to override the gfx architecture values from the base docker image
Their values can be passed in when running `docker build` with `--build-arg` options. Their values can be passed in when running `docker build` with `--build-arg` options.
To build vllm on ROCm 7.0 for MI200 and MI300 series, you can use the default (which build a docker image with `vllm serve` as entrypoint): To build vllm on ROCm 10.0 for all supported GPUs, you can use the default (which build a docker image with `vllm serve` as entrypoint):
```bash ```bash
DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.rocm -t vllm/vllm-openai-rocm . DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.rocm -t vllm/vllm-openai-rocm .
+27
View File
@@ -455,6 +455,8 @@ Score templates are supported for **cross-encoder** models only. If you are usin
Some scoring models require a specific prompt format to work correctly. You can specify a custom score template using the `--chat-template` parameter (see [Chat Template](../../serving/online_serving/README.md#chat-template)). Some scoring models require a specific prompt format to work correctly. You can specify a custom score template using the `--chat-template` parameter (see [Chat Template](../../serving/online_serving/README.md#chat-template)).
For offline scoring, pass `chat_template` to the `LLM` constructor to use it for every `LLM.score` call. A `chat_template` passed to `LLM.score` overrides the constructor's template for that call.
Like chat templates, the score template receives a `messages` list. For scoring, each message has a `role` attribute—either `"query"` or `"document"`. For the usual kind of point-wise cross-encoder, you can expect exactly two messages: one query and one document. To access the query and document content, use Jinja's `selectattr` filter: Like chat templates, the score template receives a `messages` list. For scoring, each message has a `role` attribute—either `"query"` or `"document"`. For the usual kind of point-wise cross-encoder, you can expect exactly two messages: one query and one document. To access the query and document content, use Jinja's `selectattr` filter:
- **Query**: `{{ (messages | selectattr("role", "eq", "query") | first).content }}` - **Query**: `{{ (messages | selectattr("role", "eq", "query") | first).content }}`
@@ -464,6 +466,31 @@ This approach is more robust than index-based access (`messages[0]`, `messages[1
Example template file: [examples/pooling/score/template/nemotron-rerank.jinja](../../../examples/pooling/score/template/nemotron-rerank.jinja) Example template file: [examples/pooling/score/template/nemotron-rerank.jinja](../../../examples/pooling/score/template/nemotron-rerank.jinja)
#### Templates declared by Sentence Transformers checkpoints
vLLM does not apply a checkpoint's chat template by default, since many rerankers inherit an unrelated template from their base LLM. The exception is a Sentence Transformers (v5.4+) `CrossEncoder` with a single `sequence-classification` Transformer module whose `sentence_bert_config.json` declares the `message` modality. Sentence Transformers formats the pairs of such checkpoints as `query`/`document` messages, so vLLM does the same with the saved template:
```json
{
"transformer_task": "sequence-classification",
"modality_config": {
"text": {"method": "forward", "method_output_name": "logits"},
"message": {"method": "forward", "method_output_name": "logits", "format": "flat"}
},
"module_output_name": "scores",
"processing_kwargs": {"chat_template": {"add_generation_prompt": true}}
}
```
- The default chat template is used, unless `processing_kwargs.chat_template.chat_template` names another template saved with the tokenizer (e.g. `additional_chat_templates/score.jinja` for `"score"`).
- The other `processing_kwargs.chat_template` settings, such as `add_generation_prompt`, are passed to the template; the request's `chat_template_kwargs` override them.
- Message contents are passed as strings, as with `--chat-template`.
- vLLM does not restore the template suffix after truncation (`restore_suffix`). Use `max_tokens_per_doc` rather than `truncate_prompt_tokens` to truncate long documents.
- `--chat-template` (or the `chat_template` argument of `LLM.score`) takes precedence over the saved template and its settings.
- Models with a built-in score template keep using it.
The template in use is logged at startup.
### Enable/disable activation ### Enable/disable activation
You can enable or disable activation via `use_activation` only works for cross-encoder models. You can enable or disable activation via `use_activation` only works for cross-encoder models.
@@ -313,14 +313,12 @@ Start the server:
```shell ```shell
vllm serve nvidia/llama-nemotron-rerank-vl-1b-v2 \ vllm serve nvidia/llama-nemotron-rerank-vl-1b-v2 \
--runner pooling \ --runner pooling \
--trust-remote-code \ --trust-remote-code
--chat-template examples/pooling/score/template/nemotron-vl-rerank.jinja
``` ```
!!! note !!! note
The chat template bundled with this checkpoint's tokenizer is not suitable The checkpoint declares its score template in its Sentence Transformers config, so vLLM applies it automatically (see [Templates declared by Sentence Transformers checkpoints](scoring.md#templates-declared-by-sentence-transformers-checkpoints)).
for the Score/Rerank APIs. Use the provided override template when serving: Revisions older than October 2026 need `--chat-template examples/pooling/score/template/nemotron-vl-rerank.jinja`.
`examples/pooling/score/template/nemotron-vl-rerank.jinja`.
Score a text query against an image document: Score a text query against an image document:
+2 -2
View File
@@ -105,7 +105,7 @@ The server keeps no state between calls. Everything the next call needs is in `s
- **Send `prompt_token_ids` on the first chunk, even without a parser.** It seeds detokenization so the first token's leading space matches the coupled endpoint (see [Request format](#request-format)). Later chunks already carry that context in `stream_state`. - **Send `prompt_token_ids` on the first chunk, even without a parser.** It seeds detokenization so the first token's leading space matches the coupled endpoint (see [Request format](#request-format)). Later chunks already carry that context in `stream_state`.
- **Don't forward `[DONE]`.** It marks the end of the generate stream and isn't a chunk. - **Don't forward `[DONE]`.** It marks the end of the generate stream and isn't a chunk.
`/inference/v1/generate` returns `GenerateLogProbs` with integer `token_id`s (the generate server has no tokenizer), and derender turns those into `ChatCompletionLogProbs` / `CompletionLogProbs`, filling `token` and `bytes` from the tokenizer with the usual U+FFFD byte-fallback correction. Generate returns every top-k candidate the engine produced; derender keeps the ones the coupled endpoint would, with the same helpers: the first `top_logprobs` on chat (all of them for `top_logprobs=-1`) and the first `logprobs + 1` on completions (all of them for `logprobs=-1`). It reads those values from `chat_request` / `completion_request`, so without the original request, or with one that does not ask for logprobs, it returns every candidate. Requests that use `logprob_token_ids` get no output logprobs from generate (their `sampling_params.logprobs` is unset), so derender has none to return for them either. Streaming derender does the same per chunk on the plain detokenization path (see [Streaming state and logprobs](#streaming-state-and-logprobs)). The parser path doesn't resolve them yet and drops them. `/inference/v1/generate` returns `GenerateLogProbs` with integer `token_id`s (the generate server has no tokenizer), and derender turns those into `ChatCompletionLogProbs` / `CompletionLogProbs`, filling `token` and `bytes` from the tokenizer with the usual U+FFFD byte-fallback correction. Generate returns every top-k candidate the engine produced; derender keeps the ones the coupled endpoint would, with the same helpers: the first `top_logprobs` on chat (all of them for `top_logprobs=-1`) and the first `logprobs + 1` on completions (all of them for `logprobs=-1`). It reads those values from `chat_request` / `completion_request`, so without the original request, or with one that does not ask for logprobs, it returns every candidate. Requests that use `logprob_token_ids` get no output logprobs from generate (their `sampling_params.logprobs` is unset), so derender has none to return for them either. Streaming derender does the same per chunk on both the plain detokenization path and the parser path (see [Streaming state and logprobs](#streaming-state-and-logprobs)). As on `/v1/chat/completions`, a request with `include_reasoning: false` and a parser configured gets no logprobs at all, batch or streaming, since the decoded logprob tokens would reveal the hidden reasoning.
## Streaming cost ## Streaming cost
@@ -129,7 +129,7 @@ Streaming derender is stateless on the server side: all mutable state lives in t
- `logprob_context_token_ids`: the trailing sampled token IDs (at most 4) from previous chunks, used to seed byte-fallback (U+FFFD) correction so multi-byte characters whose tokens split across chunk boundaries still resolve to real strings - `logprob_context_token_ids`: the trailing sampled token IDs (at most 4) from previous chunks, used to seed byte-fallback (U+FFFD) correction so multi-byte characters whose tokens split across chunk boundaries still resolve to real strings
- `logprob_text_offset`: the cumulative emitted text length, so `text_offset` in completion streaming logprobs stays absolute across chunks instead of restarting at 0 - `logprob_text_offset`: the cumulative emitted text length, so `text_offset` in completion streaming logprobs stays absolute across chunks instead of restarting at 0
When a streamed `GenerateTokensStreamChoice` carries `logprobs`, the integer `token_id`s are decoded per chunk and the resolved logprobs are attached to the corresponding streamed choice. For chat it is as `ChatCompletionLogProbs` and for completions it is converted to the flat `CompletionLogProbs` lists. Chunks without `logprobs` produce choices with `logprobs: null`. When a streamed `GenerateTokensStreamChoice` carries `logprobs`, the integer `token_id`s are decoded per chunk and the resolved logprobs are attached to the corresponding streamed choice. For chat it is as `ChatCompletionLogProbs` and for completions it is converted to the flat `CompletionLogProbs` lists. Chunks without `logprobs` produce choices with `logprobs: null`. On the parser path the chunk's logprobs are attached to whatever delta the parser emits for that chunk (reasoning, content or a tool call delta, or an empty delta while the parser is still buffering markup), which is what the coupled endpoint does; replay never touches them, since each chunk's `logprobs` cover exactly that chunk's tokens.
## Example ## Example
@@ -1,8 +1,50 @@
# Tokens In <> Tokens Out API # Tokens In <> Tokens Out API
`/inference/v1/generate` takes prompt token IDs and returns generated token IDs. It is the generation step of the `render → generate → derender` pipeline used in disaggregated serving (see the [Renderer APIs](renderer.md) and [Derenderer APIs](derenderer.md)). `POST /inference/v1/generate` is the token-in/token-out (TITO) HTTP endpoint: it takes prompt token IDs and returns generated token IDs, logprobs keyed by integer token ID, and `finish_reason`/`stop_reason`, with no detokenization. Use it when the caller does its own tokenization and chat templating, such as an RL rollout worker. It is also the generation step of the `render → generate → derender` pipeline used in disaggregated serving (see the [Renderer APIs](renderer.md) and [Derenderer APIs](derenderer.md)).
The endpoint is registered on `vllm serve` with `--enable-scale-out`, and always with `--tokens-only`. `--tokens-only` also skips loading the tokenizer, so the server can't turn token IDs back into text. ## Enabling the endpoint (`--enable-scale-out`, `--tokens-only`)
- `--enable-scale-out` registers it, with `/render`, `/derender` and `/inference/v1/abort_requests`.
- `--tokens-only` always registers it, skips loading the tokenizer and forces `detokenize=False`. Stop strings (`sampling_params.stop`) are then rejected with a 400; use `stop_token_ids`.
- With neither flag, it returns 403 `Scale-out endpoints are disabled. Restart vllm serve with --enable-scale-out to enable them.`
```bash
vllm serve Qwen/Qwen3-0.6B --enable-scale-out
curl -X POST http://localhost:8000/inference/v1/generate \
-H "Content-Type: application/json" \
-d '{"token_ids": [151644, 872, 198], "sampling_params": {"max_tokens": 32, "logprobs": 1}}'
```
A full client is in [examples/scale_out/token_generation_client.py](../../../examples/scale_out/token_generation_client.py).
## Request fields
| Field | Notes |
| --- | --- |
| `token_ids` | Required `list[int]`: the already-templated prompt. Non-empty, no negative IDs. |
| `sampling_params` | Required. Any [SamplingParams](../../api/README.md#inference-parameters) field, e.g. `max_tokens`, `temperature`, `logprobs`, `stop_token_ids`. Without `max_tokens`, the server applies its own default (as on the OpenAI endpoints), not 16. |
| `output_mode` | `"tokens"` (default) or `"text"`, see [Output modes](#output-modes). |
| `stream`, `stream_options` | SSE chunks ending with `data: [DONE]`. |
| `return_token_ids` | Echo the final prompt IDs as `prompt_token_ids`. |
| `features` / `content_parts` | Multimodal input, mutually exclusive. |
| `request_id`, `model`, `cache_salt`, `priority`, `kv_transfer_params`, `ec_transfer_params` | Optional. |
## Response fields
| `choices[i]` field | Notes |
| --- | --- |
| `token_ids` | Generated token IDs. |
| `finish_reason` | `stop`, `length` or `abort`. |
| `stop_reason` | Stop token ID or string; `null` on EOS, length and abort. |
| `logprobs` | `{"content": [{"token_id", "logprob", "rank", "top_logprobs": [...]}]}`, one entry per token, sampled token first. `token_id` is an integer, not the older `"token_id:<id>"` string. Needs `sampling_params.logprobs`. |
| `weight_versions`, `routed_experts`, `sampling_mask` | Optional, see [Weight versions](#weight-versions); the last two need `--enable-return-routed-experts` / `--return-sampling-mask`. |
The top level adds `request_id`, `usage`, `prompt_token_ids` and `prompt_logprobs`. Stream chunks carry the same per-choice fields for the new tokens.
## Targeting a data-parallel rank (`X-data-parallel-rank`)
Send the header `X-data-parallel-rank: <int>` to pin a request to one data-parallel (DP) engine rank; there is no body field. Without the header, or with a non-integer value, the request is load balanced. A rank outside `[0, data_parallel_size)` is rejected.
## Output modes ## Output modes
+4
View File
@@ -76,6 +76,10 @@ for span in completion.weight_versions:
The offsets cover the whole output, so with streamed delta outputs, collect the tokens before slicing. `/inference/v1/generate` returns the same list on each choice; see [Tokens In <> Tokens Out API](../serving/online_serving/token_in_token_out.md#weight-versions). The offsets cover the whole output, so with streamed delta outputs, collect the tokens before slicing. `/inference/v1/generate` returns the same list on each choice; see [Tokens In <> Tokens Out API](../serving/online_serving/token_in_token_out.md#weight-versions).
`/v1/completions` and `/v1/chat/completions` return it on each choice when the request sets `return_weight_versions: true`. The offsets count the choice's output tokens, which `return_token_ids` returns as `token_ids`, and a streamed choice gets the list on its last chunk.
An aborted request returns spans over its partial output. If you resubmit that output as part of a new prompt, the new request's spans start again at 0, so shift them by the number of tokens already generated before joining the two lists.
## Example ## Example
The [async RLHF example](../../examples/rl/rlhf_async_new_apis.py) demonstrates this pattern with `vllm.AsyncLLMEngine`, NCCL weight transfer, and mid-flight pause/resume with validation. The [async RLHF example](../../examples/rl/rlhf_async_new_apis.py) demonstrates this pattern with `vllm.AsyncLLMEngine`, NCCL weight transfer, and mid-flight pause/resume with validation.
+2
View File
@@ -18,6 +18,8 @@ The following open-source RL libraries use vLLM for fast rollouts (sorted alphab
For weight synchronization between training and inference, see the [Weight Transfer](weight_transfer/README.md) documentation, which covers the pluggable backend system with [NCCL](weight_transfer/nccl.md) (multi-GPU) and [IPC](weight_transfer/ipc.md) (same-GPU) engines. To verify that an update landed on every rank, see the [Weight Checker](weight_checker.md). For weight synchronization between training and inference, see the [Weight Transfer](weight_transfer/README.md) documentation, which covers the pluggable backend system with [NCCL](weight_transfer/nccl.md) (multi-GPU) and [IPC](weight_transfer/ipc.md) (same-GPU) engines. To verify that an update landed on every rank, see the [Weight Checker](weight_checker.md).
For MoE policies, see [Routed Experts Return](routed_experts.md) (`--enable-return-routed-experts`) to get the per-token expert routing of each rollout for routing replay (R3) in the trainer.
For pipelining generation and training to improve GPU utilization and throughput, see the [Async Reinforcement Learning](async_rl.md) guide, which covers the pause/resume API for safely updating weights mid-flight. For pipelining generation and training to improve GPU utilization and throughput, see the [Async Reinforcement Learning](async_rl.md) guide, which covers the pause/resume API for safely updating weights mid-flight.
See the following notebooks showing how to use vLLM for GRPO: See the following notebooks showing how to use vLLM for GRPO:
+54
View File
@@ -0,0 +1,54 @@
# Routed Experts Return (MoE Routing Replay / R3)
`--enable-return-routed-experts` makes vLLM return the routed experts (the expert IDs each MoE layer's router selected) for every token of a request. RL trainers use it for MoE routing replay (R3): the training forward pass reuses the rollout's routing.
```bash
vllm serve Qwen/Qwen3-30B-A3B --enable-return-routed-experts
# offline: LLM("Qwen/Qwen3-30B-A3B", enable_return_routed_experts=True)
```
The flag sets `AuxOutputConfig.enable_return_routed_experts` (full form: `--aux-output-config '{"enable_return_routed_experts": true, "max_bytes": N}'`). There is no per-request on/off switch.
## Requirements
Startup fails with a `ValueError` unless:
- **Model Runner V2** is used (`VLLM_USE_V2_MODEL_RUNNER=1` if vLLM falls back to V1);
- the runner is `generate` and the model is **MoE**;
- **prefix caching is enabled** (the default). `--no-enable-prefix-caching` fails; the old v0.21 advice to disable it is wrong.
Rejected at startup: adaptive speculative verification, PP > 1, DCP/PCP > 1, and the `NixlConnector`, `NixlPullConnector`, `NixlPushConnector`, `MoRIIOConnector`, `MooncakeConnector` KV connectors. Regular speculative decoding works.
### How it works: aux output connector
Routing is stored by the aux output connector (`vllm/distributed/aux_output_connector/`, configured via `--aux-output-config`), keyed by KV block hash and shared with prefix caching, which is why prefix caching is required. `max_bytes` is its host-memory LRU capacity (default: derived from KV cache capacity). If a prefix-cache hit needs evicted rows, the worker raises `BlockObjectStoreError`; increase `max_bytes`.
## Output format
| API | Field |
| --- | --- |
| Offline (`LLM`, `AsyncLLM`) | `CompletionOutput.routed_experts`: `np.ndarray` or `None` |
| `/v1/completions`, `/v1/chat/completions` | `choices[i].routed_experts`: base64 `.npy`, **non-streaming only** |
| `/inference/v1/generate` | `routed_experts` per choice (base64 `.npy`); with `stream: true`, on the final chunk |
- **Shape:** `(num_tokens - 1 - routed_experts_prompt_start, num_moe_layers, num_experts_per_tok)`, `num_tokens` = prompt + output. The last sampled token has no row.
- **Rows:** row `i` is position `routed_experts_prompt_start + i` of `prompt_token_ids + output_token_ids`.
- **dtype:** `uint8` for at most 256 experts, else `uint16`; logical expert IDs.
- Filled only when the request finishes. `None` if the flag is off or the request aborted before any forward pass.
```python
import base64, io
import numpy as np
experts = np.load(io.BytesIO(base64.b64decode(choice["routed_experts"])))
```
## `routed_experts_prompt_start`
Per-request, default `0`: drops the first N prompt-token rows (e.g. routing already received in earlier turns). Must be `0 <= value <= prompt_len`, else HTTP 400. Pass it top-level on `/v1/*completions`, inside `sampling_params` on `/inference/v1/generate`, or as `SamplingParams(routed_experts_prompt_start=...)`.
## Limitations
- The Rust HTTP frontend never returns `routed_experts` and always sends `routed_experts_prompt_start = 0`. Use the Python API server.
Example: [examples/rl/routed_experts_e2e.py](../../examples/rl/routed_experts_e2e.py).
+3 -2
View File
@@ -139,8 +139,9 @@ Each position then carries the sampled token's logprob and the top-128
drop them. The finite entries are the sampling mask when it has at most `k` drop them. The finite entries are the sampling mask when it has at most `k`
tokens. tokens.
- The sampled token is always the first entry and may repeat inside the top-k. - The sampled token is always the first entry and may repeat inside the top-k.
- `/inference/v1/generate` encodes IDs as `token_id:<id>`; the OpenAI - The `token_id:<id>` strings come from the OpenAI endpoints with
endpoints need `return_tokens_as_token_ids`. `return_tokens_as_token_ids`; `/inference/v1/generate` returns an integer
`token_id` instead.
## Limitations ## Limitations
+8 -1
View File
@@ -145,7 +145,7 @@ engine.send_weights() # called on every rank
## API Endpoints ## API Endpoints
When running vLLM as an HTTP server, the following endpoints are available for weight transfer. `HTTPVLLMWeightSyncClient` speaks the first four for you. When running vLLM as an HTTP server, the following endpoints are available for weight transfer. `HTTPVLLMWeightSyncClient` speaks the first five for you. A trainer may also implement the protocol directly without importing vLLM: see the wire contracts in [NCCL](nccl.md#trainer-without-vllm-over-http) and [IPC](ipc.md#trainer-without-vllm-over-http). This keeps the trainer decoupled from vLLM: it needs no vLLM install or version pin, only this documented HTTP and wire contract, so trainer and server can upgrade independently (non-PyTorch trainers such as JAX can use it too).
| Endpoint | Method | Description | | Endpoint | Method | Description |
| -------- | ------ | ----------- | | -------- | ------ | ----------- |
@@ -153,6 +153,7 @@ When running vLLM as an HTTP server, the following endpoints are available for w
| `/start_weight_update` | POST | Start a weight update | | `/start_weight_update` | POST | Start a weight update |
| `/update_weights` | POST | Transfer a batch of weights with backend-specific metadata | | `/update_weights` | POST | Transfer a batch of weights with backend-specific metadata |
| `/finish_weight_update` | POST | Finish the update and optionally commit its `weight_version` | | `/finish_weight_update` | POST | Finish the update and optionally commit its `weight_version` |
| `/shutdown_weight_transfer_engine` | POST | Release the transfer channel without stopping inference |
| `/update_weight_version` | POST | Update `weight_version` without changing model weights | | `/update_weight_version` | POST | Update `weight_version` without changing model weights |
| `/weight_info` | GET | Get the latest committed weight version | | `/weight_info` | GET | Get the latest committed weight version |
| `/pause` | POST | Pause generation before weight sync to handle inflight requests | | `/pause` | POST | Pause generation before weight sync to handle inflight requests |
@@ -162,6 +163,12 @@ When running vLLM as an HTTP server, the following endpoints are available for w
!!! note !!! note
The HTTP weight transfer endpoints require `VLLM_SERVER_DEV_MODE=1` to be set. The HTTP weight transfer endpoints require `VLLM_SERVER_DEV_MODE=1` to be set.
!!! warning
If `/update_weights` fails, the model is left mid-reload while `/health`
stays 200, and `/finish_weight_update` is rejected. Before generating again,
resend a full `/start_weight_update` → `/update_weights` →
`/finish_weight_update` (see [#57854](https://github.com/vllm-project/vllm/pull/57854)).
The Rust frontend's optional gRPC `Control` service exposes the same pause, sleep, weight-transfer, and weight-version lifecycle for trusted sidecars. The `ServerInfo.rl_capabilities` response reports whether weight transfer and sleep mode were configured. Backend-specific `init_info` and `update_info` remain JSON metadata; model tensors continue to move over the configured NCCL, IPC, sparse-NCCL, or sharded-RDT transport. The Rust frontend's optional gRPC `Control` service exposes the same pause, sleep, weight-transfer, and weight-version lifecycle for trusted sidecars. The `ServerInfo.rl_capabilities` response reports whether weight transfer and sleep mode were configured. Backend-specific `init_info` and `update_info` remain JSON metadata; model tensors continue to move over the configured NCCL, IPC, sparse-NCCL, or sharded-RDT transport.
## Extending the System ## Extending the System
+38
View File
@@ -110,6 +110,43 @@ mode carries a per-chunk barrier across ranks: without it, a rank could overwrit
its buffer while its colocated worker is still reading the current chunk. This is its buffer while its colocated worker is still reading the current chunk. This is
handled inside `send_weights()`. handled inside `send_weights()`.
## Trainer Without vLLM over HTTP
A colocated trainer can speak the [HTTP endpoints](README.md#api-endpoints)
directly, with only PyTorch and an HTTP client. The server needs
`VLLM_ALLOW_INSECURE_SERIALIZATION=1`; without it, init and start return 200 and
the first `/update_weights` returns 500. Reference:
[`rlhf_http_ipc_no_vllm.py`](../../../examples/rl/rlhf_http_ipc_no_vllm.py).
`POST /init_weight_transfer_engine` body: `{"init_info": {"packed": false}}`.
`POST /update_weights` body `{"update_info": {...}}`:
| Field | Meaning |
| ----- | ------- |
| `names`, `dtype_names`, `shapes` | Checkpoint names, `torch` dtype attribute names (`"bfloat16"`), full shapes |
| `ipc_handles_pickled` | `base64(pickle.dumps(handles))`. Unpacked: a list with one dict per name. Packed: one dict for the shared buffer |
| `tensor_sizes` | Packed only: byte size of each tensor in the chunk, in `names` order |
- A handle dict is `{gpu_uuid: reduce_tensor(t)[1]}` with one entry per GPU the
server uses (`torch.multiprocessing.reductions.reduce_tensor`).
`gpu_uuid = str(torch.cuda.get_device_properties(i).uuid)`, **without** the
`GPU-` prefix that `nvidia-smi` prints. The worker overwrites `args[6]`
(device index) with its own.
- `ipc_handles` (unpickled) is not JSON-serializable; over HTTP use
`ipc_handles_pickled`.
- Packed: export one `uint8` buffer once and reuse it for every chunk. Copy
`t.contiguous().view(-1).view(torch.uint8)` in `names` order and close a chunk
**before** the next tensor would overflow it (unlike NCCL, which closes after).
A tensor larger than the buffer is an error. Synchronize before each
`/update_weights`; do not overwrite the buffer until the call returns.
- Keep every exported tensor alive until `/finish_weight_update` returns, then
drop the references and call `torch.cuda.ipc_collect()`.
- Multi-rank trainers: each rank exports handles for its own GPU, the ranks
gather the `{uuid: args}` dicts to rank 0 over their own `torch.distributed`
group and merge them per index, and rank 0 sends all HTTP. In packed mode,
barrier before and after each `/update_weights`.
## Rank-Local Updates ## Rank-Local Updates
For different parameter subsets per worker, `update_info` may be a list indexed For different parameter subsets per worker, `update_info` may be a list indexed
@@ -119,4 +156,5 @@ worker to participate.
## Examples ## Examples
- [RLHF with IPC weight syncing (`vllm serve`, HTTP)](../../../examples/rl/rlhf_http_ipc.py) - **Start here.** Server and training model share a single GPU; HTTP control plane, CUDA IPC data plane. Launches and tears down its own server - [RLHF with IPC weight syncing (`vllm serve`, HTTP)](../../../examples/rl/rlhf_http_ipc.py) - **Start here.** Server and training model share a single GPU; HTTP control plane, CUDA IPC data plane. Launches and tears down its own server
- [IPC weight syncing from a trainer that does not import vLLM](../../../examples/rl/rlhf_http_ipc_no_vllm.py) - `torch` + `requests` only, `ipc_handles_pickled` over HTTP; see [Trainer Without vLLM over HTTP](#trainer-without-vllm-over-http)
- [RLHF with IPC + FSDP2 and expert parallelism](../../../examples/rl/rlhf_ipc_fsdp_ep.py) - Multi-rank trainer colocated with a `--data-parallel-size 4` server on the same 4 GPUs: every FSDP rank builds an engine and joins the handle all-gather, with packed chunking and sleep/wake around the transfer - [RLHF with IPC + FSDP2 and expert parallelism](../../../examples/rl/rlhf_ipc_fsdp_ep.py) - Multi-rank trainer colocated with a `--data-parallel-size 4` server on the same 4 GPUs: every FSDP rank builds an engine and joins the handle all-gather, with packed chunking and sleep/wake around the transfer
+18 -13
View File
@@ -10,18 +10,14 @@ The `nccl_m2n` weight transfer engine uses [NCCL M2N](https://github.com/NVIDIA/
## Requirements ## Requirements
This backend depends on the `nccl-extensions` Python package (and its `nccl4py` dependency), which vLLM does not install. It needs NCCL 2.30.5 or newer, and `libnccl_m2n.so` is linked against a specific NCCL build, so vLLM must load the same library: This backend depends on the `nccl-extensions` Python package (and its `nccl4py` dependency), which vLLM does not install. Install both against a compatible NCCL 2.30.5 or newer runtime. The backend creates its data-plane communicator through `nccl4py` and passes that communicator directly to `nccl.m2n`.
```bash
export VLLM_NCCL_SO_PATH=/path/to/the/same/libnccl.so
```
The backend imports the runtime lazily, so vLLM is unaffected unless you select it. The backend imports the runtime lazily, so vLLM is unaffected unless you select it.
## How It Works ## How It Works
1. The trainer ranks and all inference workers join **one** NCCL communicator via `StatelessProcessGroup`. Trainer ranks occupy `[0, T)` and workers `[T, T + N)` — m2n meshes are contiguous rank intervals, which is exactly what vLLM's existing `rank_offset` convention produces. 1. A `StatelessProcessGroup` exchanges metadata and the NCCL unique ID. The trainer ranks and all inference workers then create **one** `nccl.core.Communicator`: trainer ranks occupy `[0, T)` and workers `[T, T + N)`. The communicator is passed directly to each M2N collective.
2. At the init handshake the trainer ships the full transfer plan: both meshes — one per side, shared by every parameter — then each parameter's name, dtype, shape and placement. The workers resolve the matching **destination placements** and publish them back over the shared group; the meshes themselves are declared, not negotiated. 2. At the init handshake the trainer ships the full transfer plan: the session-wide destination mesh, then each parameter's name, dtype, shape, source mesh, and source placements. The workers resolve the matching **destination placements** and publish them over the data-plane communicator; the meshes themselves are declared, not negotiated.
3. Each round, both sides issue one `reshard` per parameter in the same order. Every rank in the communicator participates in every reshard, so — as with the NCCL engine — the worker must be inside `update_weights` while the trainer sends. The trainer engine handles that concurrency itself. 3. Each round, both sides issue one `reshard` per parameter in the same order. Every rank in the communicator participates in every reshard, so — as with the NCCL engine — the worker must be inside `update_weights` while the trainer sends. The trainer engine handles that concurrency itself.
### Destination resolution ### Destination resolution
@@ -83,19 +79,28 @@ engine.send_weights()
`send_weights()` drives the whole round — `start_weight_update`, `update_weights`, `finish_weight_update`. Every rank in the communicator must be inside the same reshard, so `update_weights` has to be in flight while the trainer sends; the engine runs it on a helper thread so the trainer loop does not have to open-code that. `send_weights()` drives the whole round — `start_weight_update`, `update_weights`, `finish_weight_update`. Every rank in the communicator must be inside the same reshard, so `update_weights` has to be in flight while the trainer sends; the engine runs it on a helper thread so the trainer loop does not have to open-code that.
Trainers with a custom producer (a Megatron export, MoE re-fusing) subclass `M2NWeightSource` and supply layouts explicitly. Trainers with a custom producer (a Megatron export, MoE re-fusing) subclass
`M2NWeightSource` and supply layouts explicitly. A source may yield any
nccl4py-compatible CUDA buffer: a torch tensor, a CUDA Array Interface object,
or a raw device pointer. vLLM keeps buffer owners alive through stream
completion, using allocator stream tracking when the buffer provides it.
Sources must provide buffers on the communicator's current device; nccl4py
resolves their pointers but does not verify device ownership. Raw-pointer
sources are additionally responsible for pointer lifetime and matching their
declared dtype and local shape.
See [`examples/rl/rlhf_m2n.py`](../../../examples/rl/rlhf_m2n.py) for a runnable FSDP → TP example. See [`examples/rl/rlhf_m2n.py`](../../../examples/rl/rlhf_m2n.py) for a runnable FSDP → TP example.
## Limitations ## Limitations
- Tensor rank 1..3; 4-D parameters are rejected. - vLLM validates tensor-rank and dtype support while agreeing on the complete
- dtypes: int8/uint8, fp8 e4m3/e5m2, fp16, bf16, int32/uint32, fp32, int64/uint64, fp64. No fp4. plan; the installed M2N runtime validates them again for each reshard.
- The trainer's device mesh must cover a contiguous rank interval starting at 0, and be 1-D or 2-D. - Each parameter's source mesh must cover the same contiguous trainer-rank interval starting at 0, and be 1-D or 2-D.
- `Partial` placements are not supported. - `Partial` placements are not supported.
- Models with known model-level checkpoint preprocessing use the full-tensor - Models with known model-level checkpoint preprocessing use the full-tensor
fallback for the affected module. fallback for the affected module.
- A reshard cannot be issued from inside a CUDA graph capture (weight updates run outside capture, so this only matters for custom callers). - A reshard cannot be issued from inside a CUDA graph capture (weight updates run outside capture, so this only matters for custom callers).
- The reshard plan is capped per *shard*, not per mesh: at most 16 source shards may feed one destination shard, and one source shard may feed at most 64 destination shards (`MAX_SOURCES` / `MAX_TARGETS` in the m2n build; raising either needs a rebuild). - vLLM checks every parameter's shard overlap against conservative 16-source
- A directly-resharded parameter spreads over `tp_size` destination shards, so each one is fed by fewer source shards. A parameter on the fallback path is replicated, i.e. a single destination shard fed by every source shard, so a trainer that **sharded** its parameters is still limited to 16 ranks whenever any parameter falls back — which fused parameters always do. A replicated trainer is unaffected. and 64-target capacities before committing the plan. The installed M2N
runtime remains authoritative and validates each reshard again.
- Each communicator holds a cached staging pool (~256 MiB by default: 4 channels x 64 MiB). - Each communicator holds a cached staging pool (~256 MiB by default: 4 channels x 64 MiB).
+46
View File
@@ -130,6 +130,51 @@ pass — this is the invariant to test first.
IPC does not read `metadata()` at all: it derives the update info from IPC does not read `metadata()` at all: it derives the update info from
iteration as it goes, so it cannot observe a divergence. iteration as it goes, so it cannot observe a divergence.
## Trainer Without vLLM over HTTP
A trainer can speak the [HTTP endpoints](README.md#api-endpoints) directly, with
only PyTorch and an HTTP client. It mints the `ncclUniqueId` itself and is NCCL
rank 0. Reference: [`rlhf_http_nccl_no_vllm.py`](../../../examples/rl/rlhf_http_nccl_no_vllm.py).
`POST /init_weight_transfer_engine` body `{"init_info": {...}}`:
| Field | Default | Meaning |
| ----- | ------- | ------- |
| `nccl_unique_id_b64` | — | Standard (not URL-safe) base64 of the 128-byte `ncclUniqueId`. Replaces `master_address` / `master_port` |
| `rank_offset` | — | First worker rank; `1` for one trainer rank |
| `world_size` | — | Trainer ranks + `GET /get_world_size` |
| `packed` | `false` | Packed broadcasting |
| `packed_buffer_size_bytes` | 1 GiB | Must match the trainer's chunking |
| `packed_num_buffers` | 2 | Worker-side receive buffers |
- Worker rank = `rank_offset + dp_index * (TP * PP) + rank`.
- `/init_weight_transfer_engine` returns only after all ranks join: send it in
the background while the trainer calls `ncclCommInitRank(world_size, uid, 0)`.
There is no timeout; if the trainer never joins, the workers wait forever.
- Right after init every worker issues a warm-up `all_reduce` (sum) of
`torch.zeros(1)` (one float32 element). The trainer must issue the same one
before any other collective.
Each sync: `POST /start_weight_update`, then `POST /update_weights` with
`{"update_info": {"names", "dtype_names", "shapes"}}` **in the background**
while rank 0 broadcasts in `names` order, then `POST /finish_weight_update`.
`dtype_names` are `torch` attribute names (`"bfloat16"`); `shapes` are full
checkpoint shapes.
- Unpacked: one broadcast per tensor.
- Packed: concatenate `t.contiguous().view(-1).view(torch.uint8)` in `names`
order; broadcast a chunk as soon as its size **exceeds**
`packed_buffer_size_bytes` (the crossing tensor stays in the chunk), then the
remainder.
Trainer-side communicators without vLLM:
| Communicator | Status |
| ------------ | ------ |
| `torch.distributed.ProcessGroupNCCL` on a private `TCPStore` + `eager_connect_single_device` | Works; the unique id is read from store key `"0"`, a torch implementation detail (used by the example) |
| `ctypes` against torch's bundled `libnccl.so.2` | Works |
| `torch.cuda.nccl.init_rank` | Fails on Python 3.10+ (`PY_SSIZE_T_CLEAN` `SystemError`) |
## Sparse NCCL ## Sparse NCCL
Sparse, flat-index weight patches use Sparse, flat-index weight patches use
@@ -196,6 +241,7 @@ Sparse NCCL sends only `O(nnz)` indices and values. Checkpoint application still
## Examples ## Examples
- [RLHF with NCCL weight syncing (`vllm serve`, HTTP)](../../../examples/rl/rlhf_http_nccl.py) - **Start here.** Trainer on one GPU, 2x tensor-parallel fp8 server on two others; HTTP control plane, NCCL data plane. Launches and tears down its own server - [RLHF with NCCL weight syncing (`vllm serve`, HTTP)](../../../examples/rl/rlhf_http_nccl.py) - **Start here.** Trainer on one GPU, 2x tensor-parallel fp8 server on two others; HTTP control plane, NCCL data plane. Launches and tears down its own server
- [NCCL weight syncing from a trainer that does not import vLLM](../../../examples/rl/rlhf_http_nccl_no_vllm.py) - `torch` + `requests` only, `nccl_unique_id_b64` rendezvous; see [Trainer Without vLLM over HTTP](#trainer-without-vllm-over-http)
- [RLHF with NCCL + FSDP2 and expert parallelism](../../../examples/rl/rlhf_nccl_fsdp_ep.py) - Multi-rank trainer: every FSDP rank builds an engine and joins the `full_tensor()` gather while only rank 0 touches the wire - [RLHF with NCCL + FSDP2 and expert parallelism](../../../examples/rl/rlhf_nccl_fsdp_ep.py) - Multi-rank trainer: every FSDP rank builds an engine and joins the `full_tensor()` gather while only rank 0 touches the wire
- [RLHF with sparse NCCL weight syncing (offline, Ray)](../../../examples/rl/rlhf_sparse_nccl.py) - Qwen3 MoE per-expert checkpoint updates with one trainer GPU and two TP2/EP2 inference GPUs - [RLHF with sparse NCCL weight syncing (offline, Ray)](../../../examples/rl/rlhf_sparse_nccl.py) - Qwen3 MoE per-expert checkpoint updates with one trainer GPU and two TP2/EP2 inference GPUs
- [RLHF with async weight syncing (offline, Ray)](../../../examples/rl/rlhf_async_new_apis.py) - Async generation with mid-flight pause, weight sync, resume, and validation against a fresh model; uses an in-process `AsyncLLMEngine` with `RayVLLMWeightSyncClient` - [RLHF with async weight syncing (offline, Ray)](../../../examples/rl/rlhf_async_new_apis.py) - Async generation with mid-flight pause, weight sync, resume, and validation against a fresh model; uses an in-process `AsyncLLMEngine` with `RayVLLMWeightSyncClient`
+2 -2
View File
@@ -423,9 +423,9 @@ Most cache paths default to subdirectories under a single root. Changing `VLLM_C
| Environment Variable | Default | Description | | Environment Variable | Default | Description |
| --- | --- | --- | | --- | --- | --- |
| `VLLM_CACHE_ROOT` | `~/.cache/vllm` | Base cache directory. Respects `XDG_CACHE_HOME` if set. All paths below inherit from this unless explicitly overridden. | | `VLLM_CACHE_ROOT` | `~/.cache/vllm` | Base directory for vLLM-owned caches. Respects `XDG_CACHE_HOME` if set. |
| *(torch.compile)* | `$VLLM_CACHE_ROOT/torch_compile_cache/` | Compilation cache for AOT-compiled models, Inductor graphs, and Triton kernels. Controlled by `VLLM_DISABLE_COMPILE_CACHE` (set to `1` to disable). | | *(torch.compile)* | `$VLLM_CACHE_ROOT/torch_compile_cache/` | Compilation cache for AOT-compiled models, Inductor graphs, and Triton kernels. Controlled by `VLLM_DISABLE_COMPILE_CACHE` (set to `1` to disable). |
| `VLLM_FLASHINFER_AUTOTUNE_CACHE_DIR` | `$VLLM_CACHE_ROOT/flashinfer_autotune_cache/<flashinfer-version>/<arch>/<cache-hash>/` | FlashInfer autotune config cache. | | `VLLM_FLASHINFER_AUTOTUNE_CACHE_DIR` | Managed: `~/.cache/flashinfer/autotune/`; legacy: `$VLLM_CACHE_ROOT/flashinfer_autotune_cache/<flashinfer-version>/<arch>/<cache-hash>/` | Overrides cache placement for both FlashInfer autotune cache APIs. |
| `VLLM_ASSETS_CACHE` | `$VLLM_CACHE_ROOT/assets/` | Downloaded assets (e.g., tokenizer files). | | `VLLM_ASSETS_CACHE` | `$VLLM_CACHE_ROOT/assets/` | Downloaded assets (e.g., tokenizer files). |
| `VLLM_XLA_CACHE_PATH` | `$VLLM_CACHE_ROOT/xla_cache/` | XLA/TPU compilation cache. | | `VLLM_XLA_CACHE_PATH` | `$VLLM_CACHE_ROOT/xla_cache/` | XLA/TPU compilation cache. |
| `VLLM_MEDIA_CACHE` | *(disabled)* | Optional cache for downloaded media (images, video, audio). Not enabled unless explicitly set. | | `VLLM_MEDIA_CACHE` | *(disabled)* | Optional cache for downloaded media (images, video, audio). Not enabled unless explicitly set. |
@@ -473,9 +473,11 @@ async def stop_profile():
if __name__ == "__main__": if __name__ == "__main__":
parser = argparse.ArgumentParser() parser = argparse.ArgumentParser()
parser.add_argument("--port", type=int, default=10001) parser.add_argument("--port", type=int, default=10001)
# Port the prefill/decode servers register on (their proxy_ping_port).
parser.add_argument("--service-discovery-port", type=int, default=36367)
args = parser.parse_args() args = parser.parse_args()
t = start_service_discovery("0.0.0.0", 36367) t = start_service_discovery("0.0.0.0", args.service_discovery_port)
# High-concurrency hardening. Quart's app.run() uses a shallow listen # High-concurrency hardening. Quart's app.run() uses a shallow listen
# backlog (100) and, with app.debug=True, adds per-request overhead that # backlog (100) and, with app.debug=True, adds per-request overhead that
# starves the single accept loop. Under a burst of ~512 simultaneous client # starves the single accept loop. Under a burst of ~512 simultaneous client
@@ -340,7 +340,10 @@ async def _handle_completions(api: str, request: Request):
): ):
yield chunk yield chunk
return StreamingResponse(generate_stream(), media_type="application/json") media_type = (
"text/event-stream" if req_data.get("stream") else "application/json"
)
return StreamingResponse(generate_stream(), media_type=media_type)
except Exception as e: except Exception as e:
import sys import sys
+168
View File
@@ -0,0 +1,168 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""CUDA IPC weight sync over HTTP from a trainer that never imports vLLM.
The trainer uses only `torch`, `requests`, `safetensors`, `pybase64` and the
stdlib, and speaks the wire protocol in
`docs/training/weight_transfer/ipc.md` ("Trainer Without vLLM over HTTP"):
it exports its tensors with `torch.multiprocessing.reductions.reduce_tensor`
and sends the handles, pickled and base64-encoded, in `ipc_handles_pickled`.
IPC requires colocation: this process holds a full copy of the weights on every
GPU the server uses, and keys each handle by that GPU's UUID. A multi-process
trainer would instead gather the per-rank `{uuid: args}` dicts to one rank.
Start a server; IPC handles are pickled, so it needs
`VLLM_ALLOW_INSECURE_SERIALIZATION=1` (the weights it starts with are discarded):
VLLM_SERVER_DEV_MODE=1 VLLM_ALLOW_INSECURE_SERIALIZATION=1 \
CUDA_VISIBLE_DEVICES=0 vllm serve Qwen/Qwen3-0.6B \
--load-format dummy --enforce-eager --gpu-memory-utilization 0.4 \
--weight-transfer-config '{"backend": "ipc"}'
Then run the trainer on the same GPU(s) (`--packed` for chunked transfer):
CUDA_VISIBLE_DEVICES=0 python examples/rl/rlhf_http_ipc_no_vllm.py \
--checkpoint /path/to/Qwen3-0.6B
For a TP=2 server on GPUs 0-1, use `CUDA_VISIBLE_DEVICES=0,1` on both sides and
pass `--num-gpus 2` to the trainer.
"""
import sys
class _BlockVLLM:
def find_spec(self, name, path=None, target=None):
if name.split(".")[0] == "vllm":
raise ImportError(f"this trainer must not import {name!r}")
return None
sys.meta_path.insert(0, _BlockVLLM())
import argparse # noqa: E402
import glob # noqa: E402
import pickle # noqa: E402
import pybase64 as base64 # noqa: E402
import requests # noqa: E402
import torch # noqa: E402
from safetensors.torch import load_file # noqa: E402
from torch.multiprocessing.reductions import reduce_tensor # noqa: E402
parser = argparse.ArgumentParser()
parser.add_argument("--url", default="http://localhost:8000")
parser.add_argument("--checkpoint", required=True, help="dir with *.safetensors")
parser.add_argument("--num-gpus", type=int, default=1)
parser.add_argument("--packed", action="store_true")
parser.add_argument("--packed-buffer-size-bytes", type=int, default=512 << 20)
args = parser.parse_args()
def post(path: str, body: dict | None = None) -> dict:
r = requests.post(args.url + path, json=body or {}, timeout=600)
r.raise_for_status()
return r.json()
def generate(prompts: list[str]) -> list[str]:
model = requests.get(args.url + "/v1/models", timeout=60).json()["data"][0]["id"]
body = {"model": model, "prompt": prompts, "max_tokens": 16, "temperature": 0}
choices = post("/v1/completions", body)["choices"]
return [c["text"] for c in sorted(choices, key=lambda c: c["index"])]
# Weights are {gpu_uuid: {name: tensor}}: one full copy per colocated GPU. The
# UUID is str(torch.cuda.get_device_properties(i).uuid), with no "GPU-" prefix.
Weights = dict[str, dict[str, torch.Tensor]]
def update_weights(
weights: Weights,
names: list[str],
ipc_handles: list[dict[str, tuple]] | dict[str, tuple],
tensor_sizes: list[int] | None = None,
) -> None:
w = next(iter(weights.values()))
update_info = {
"names": names,
"dtype_names": [str(w[n].dtype).removeprefix("torch.") for n in names],
"shapes": [list(w[n].shape) for n in names],
"ipc_handles_pickled": base64.b64encode(pickle.dumps(ipc_handles)).decode(),
}
if tensor_sizes is not None:
update_info["tensor_sizes"] = tensor_sizes
post("/update_weights", {"update_info": update_info})
def send_unpacked(weights: Weights, names: list[str]) -> list[torch.Tensor]:
"""One /update_weights call; one {uuid: args} dict per parameter."""
copies = {u: {n: w[n].contiguous() for n in names} for u, w in weights.items()}
handles = [{u: reduce_tensor(copies[u][n])[1] for u in copies} for n in names]
update_weights(weights, names, handles)
return [t for c in copies.values() for t in c.values()]
def send_packed(weights: Weights, names: list[str], cap: int) -> list[torch.Tensor]:
"""One /update_weights call per chunk; one {uuid: args} dict per chunk."""
buffers = {
u: torch.empty(cap, dtype=torch.uint8, device=w[names[0]].device)
for u, w in weights.items()
}
handle = {u: reduce_tensor(b)[1] for u, b in buffers.items()} # export once
chunk: list[str] = []
sizes: list[int] = []
def flush() -> None:
for b in buffers.values():
torch.accelerator.synchronize(b.device) # copies must land before the read
update_weights(weights, chunk, handle, sizes)
chunk.clear()
sizes.clear()
for n in names:
nbytes = next(iter(weights.values()))[n].nbytes
assert nbytes <= cap, f"{n} ({nbytes} B) exceeds the buffer"
if chunk and sum(sizes) + nbytes > cap: # close *before* overflowing
flush()
offset = sum(sizes)
for u, w in weights.items():
flat = w[n].contiguous().view(-1).view(torch.uint8)
buffers[u][offset : offset + nbytes].copy_(flat)
chunk.append(n)
sizes.append(nbytes)
if chunk:
flush()
return list(buffers.values())
def main() -> None:
weights: Weights = {}
for i in range(args.num_gpus):
uuid = str(torch.cuda.get_device_properties(i).uuid)
weights[uuid] = {}
for path in sorted(glob.glob(f"{args.checkpoint}/*.safetensors")):
weights[uuid].update(load_file(path, device=f"cuda:{i}"))
names = sorted(next(iter(weights.values())))
prompts = ["The capital of France is", "def fibonacci(n):"]
print("before:", generate(prompts))
post("/init_weight_transfer_engine", {"init_info": {"packed": args.packed}})
post("/start_weight_update")
if args.packed:
exported = send_packed(weights, names, args.packed_buffer_size_bytes)
else:
exported = send_unpacked(weights, names)
post("/finish_weight_update", {"weight_version": None})
# The worker reads the exported memory until finish; only then release it.
del exported
torch.cuda.ipc_collect()
print("after: ", generate(prompts))
assert not [m for m in sys.modules if m.split(".")[0] == "vllm"]
if __name__ == "__main__":
main()
+174
View File
@@ -0,0 +1,174 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""NCCL weight sync over HTTP from a trainer that never imports vLLM.
The trainer uses only `torch`, `requests`, `safetensors`, `pybase64` and the
stdlib, and speaks the wire protocol in
`docs/training/weight_transfer/nccl.md` ("Trainer Without vLLM over HTTP"):
`nccl_unique_id_b64` rendezvous, a one-element warm-up `all_reduce`, then
`/start_weight_update` -> `/update_weights` (concurrent with the rank-0
broadcast) -> `/finish_weight_update`.
The NCCL communicator is a plain `torch.distributed.ProcessGroupNCCL` over a
private `TCPStore` that only the trainer joins. Its `ncclUniqueId` is read back
from store key "0", which is a torch implementation detail.
Start a server on GPUs 0-1 (the weights it starts with are discarded):
VLLM_SERVER_DEV_MODE=1 CUDA_VISIBLE_DEVICES=0,1 vllm serve Qwen/Qwen3-0.6B \
--tensor-parallel-size 2 --load-format dummy --enforce-eager \
--weight-transfer-config '{"backend": "nccl"}'
Then run the trainer on GPU 2 (`--packed` for packed broadcasting):
CUDA_VISIBLE_DEVICES=2 python examples/rl/rlhf_http_nccl_no_vllm.py \
--checkpoint /path/to/Qwen3-0.6B
"""
import sys
class _BlockVLLM:
def find_spec(self, name, path=None, target=None):
if name.split(".")[0] == "vllm":
raise ImportError(f"this trainer must not import {name!r}")
return None
sys.meta_path.insert(0, _BlockVLLM())
import argparse # noqa: E402
import datetime # noqa: E402
import glob # noqa: E402
import threading # noqa: E402
import pybase64 as base64 # noqa: E402
import requests # noqa: E402
import torch # noqa: E402
import torch.distributed as dist # noqa: E402
from safetensors.torch import load_file # noqa: E402
parser = argparse.ArgumentParser()
parser.add_argument("--url", default="http://localhost:8000")
parser.add_argument("--checkpoint", required=True, help="dir with *.safetensors")
parser.add_argument("--packed", action="store_true")
parser.add_argument("--packed-buffer-size-bytes", type=int, default=64 << 20)
parser.add_argument("--store-port", type=int, default=29612)
args = parser.parse_args()
def post(path: str, body: dict | None = None) -> dict:
r = requests.post(args.url + path, json=body or {}, timeout=600)
r.raise_for_status()
return r.json()
class PostInBackground(threading.Thread):
"""An HTTP call that only returns once this process joins the collective."""
def __init__(self, path: str, body: dict) -> None:
super().__init__()
self.path, self.body = path, body
self.error: BaseException | None = None
self.start()
def run(self) -> None:
try:
post(self.path, self.body)
except BaseException as e:
self.error = e
def result(self) -> None:
self.join()
if self.error is not None:
raise self.error
def generate(prompts: list[str]) -> list[str]:
model = requests.get(args.url + "/v1/models", timeout=60).json()["data"][0]["id"]
body = {"model": model, "prompt": prompts, "max_tokens": 16, "temperature": 0}
choices = post("/v1/completions", body)["choices"]
return [c["text"] for c in sorted(choices, key=lambda c: c["index"])]
def main() -> None:
device = torch.device("cuda:0")
torch.accelerator.set_device_index(device)
weights: dict[str, torch.Tensor] = {}
for path in sorted(glob.glob(f"{args.checkpoint}/*.safetensors")):
weights.update(load_file(path, device="cuda:0"))
names = sorted(weights)
prompts = ["The capital of France is", "def fibonacci(n):"]
print("before:", generate(prompts))
# Trainer is NCCL rank 0; vLLM worker i is rank 1 + i.
world_size = 1 + requests.get(args.url + "/get_world_size").json()["world_size"]
store = dist.TCPStore(
"127.0.0.1", args.store_port, is_master=True, wait_for_workers=False
)
pg = dist.ProcessGroupNCCL(store, 0, world_size, dist.ProcessGroupNCCL.Options())
connect = threading.Thread(target=pg.eager_connect_single_device, args=(device,))
connect.start() # publishes the unique id, then waits in ncclCommInitRank
reader = dist.TCPStore("127.0.0.1", args.store_port, is_master=False)
reader.wait(["0"], datetime.timedelta(seconds=60))
unique_id = reader.get("0")
assert len(unique_id) == 128
init = PostInBackground(
"/init_weight_transfer_engine",
{
"init_info": {
"nccl_unique_id_b64": base64.b64encode(unique_id).decode(),
"rank_offset": 1,
"world_size": world_size,
"packed": args.packed,
"packed_buffer_size_bytes": args.packed_buffer_size_bytes,
}
},
)
connect.join()
pg.allreduce([torch.zeros(1, device=device)]).wait() # vLLM's warm-up
init.result()
def broadcast(t: torch.Tensor) -> None:
opts = dist.BroadcastOptions()
opts.rootRank = 0
pg.broadcast([t], opts).wait()
post("/start_weight_update")
update_info = {
"names": names,
"dtype_names": [str(weights[n].dtype).removeprefix("torch.") for n in names],
"shapes": [list(weights[n].shape) for n in names],
}
update = PostInBackground("/update_weights", {"update_info": update_info})
if not args.packed:
for n in names:
broadcast(weights[n].contiguous())
else:
# Same chunking as the worker: a chunk closes right after the tensor
# that pushes it past the buffer size.
chunk: list[torch.Tensor] = []
size = 0
for n in names:
chunk.append(weights[n].contiguous().view(-1).view(torch.uint8))
size += chunk[-1].numel()
if size > args.packed_buffer_size_bytes:
broadcast(torch.cat(chunk))
chunk, size = [], 0
if chunk:
broadcast(torch.cat(chunk))
torch.accelerator.synchronize()
update.result()
post("/finish_weight_update", {"weight_version": None})
print("after: ", generate(prompts))
post("/shutdown_weight_transfer_engine")
pg.shutdown()
assert not [m for m in sys.modules if m.split(".")[0] == "vllm"]
if __name__ == "__main__":
main()
+25 -10
View File
@@ -17,9 +17,8 @@ Every FSDP rank builds an ``M2NTrainerWeightTransferEngine`` and calls
``send_weights()``; all ranks run every reshard, and only rank 0 drives the ``send_weights()``; all ranks run every reshard, and only rank 0 drives the
inference side through its ``RayVLLMWeightSyncClient``. inference side through its ``RayVLLMWeightSyncClient``.
Requires the ``nccl-extensions`` package (NCCL 2.30.5+) and a Requires the ``nccl-extensions`` package (NCCL 2.30.5+). The backend creates
``VLLM_NCCL_SO_PATH`` pointing at the same ``libnccl.so`` that the data-plane communicator with ``nccl4py`` and passes it directly to M2N.
``libnccl_m2n.so`` was linked against.
This example was written for 4xH100. This example was written for 4xH100.
""" """
@@ -28,6 +27,7 @@ from __future__ import annotations
import os import os
import nccl.core as nccl
import pybase64 as base64 import pybase64 as base64
import ray import ray
import torch import torch
@@ -38,7 +38,6 @@ from transformers import AutoModelForCausalLM
from vllm import LLM, SamplingParams from vllm import LLM, SamplingParams
from vllm.config import WeightTransferConfig from vllm.config import WeightTransferConfig
from vllm.distributed.device_communicators.pynccl_wrapper import NCCLLibrary
from vllm.distributed.weight_transfer import ( from vllm.distributed.weight_transfer import (
RayVLLMWeightSyncClient, RayVLLMWeightSyncClient,
WeightTransferTrainerFactory, WeightTransferTrainerFactory,
@@ -94,7 +93,15 @@ class FSDPTrainWorker:
def ready(self): def ready(self):
return True return True
def setup_engine(self, llm_handle, nccl_unique_id_b64, world_size, num_workers): def setup_engine(
self,
llm_handle,
metadata_address,
metadata_port,
nccl_unique_id_b64,
world_size,
num_workers,
):
"""Build the trainer engine on every FSDP rank. """Build the trainer engine on every FSDP rank.
`DTensorModuleSource` reads each parameter's FSDP device mesh and `DTensorModuleSource` reads each parameter's FSDP device mesh and
@@ -103,6 +110,8 @@ class FSDPTrainWorker:
""" """
self.engine = WeightTransferTrainerFactory.trainer_init( self.engine = WeightTransferTrainerFactory.trainer_init(
init_info=M2NTrainerInitInfo( init_info=M2NTrainerInitInfo(
master_address=metadata_address,
master_port=metadata_port,
nccl_unique_id_b64=nccl_unique_id_b64, nccl_unique_id_b64=nccl_unique_id_b64,
world_size=world_size, world_size=world_size,
num_trainer_ranks=FSDP_WORLD_SIZE, num_trainer_ranks=FSDP_WORLD_SIZE,
@@ -166,16 +175,22 @@ def main():
# Trainer ranks [0, FSDP_WORLD_SIZE) and inference ranks after them share # Trainer ranks [0, FSDP_WORLD_SIZE) and inference ranks after them share
# one communicator, so the two meshes are contiguous rank intervals. # one communicator, so the two meshes are contiguous rank intervals.
nccl = NCCLLibrary() nccl_unique_id_b64 = base64.b64encode(nccl.get_unique_id().as_bytes).decode()
nccl_unique_id_b64 = base64.b64encode(
bytes(nccl.ncclGetUniqueId().internal)
).decode()
world_size = FSDP_WORLD_SIZE + num_workers world_size = FSDP_WORLD_SIZE + num_workers
metadata_address = get_ip()
metadata_port = get_open_port()
print("[transfer] Initializing nccl_m2n weight transfer (all FSDP ranks)...") print("[transfer] Initializing nccl_m2n weight transfer (all FSDP ranks)...")
ray.get( ray.get(
[ [
w.setup_engine.remote(llm, nccl_unique_id_b64, world_size, num_workers) w.setup_engine.remote(
llm,
metadata_address,
metadata_port,
nccl_unique_id_b64,
world_size,
num_workers,
)
for w in fsdp_workers for w in fsdp_workers
] ]
) )
@@ -3,6 +3,8 @@
"""Call a token-in/token-out server started with: """Call a token-in/token-out server started with:
vllm serve Qwen/Qwen3-0.6B --enable-scale-out vllm serve Qwen/Qwen3-0.6B --enable-scale-out
Fields and DP rank routing: docs/serving/online_serving/token_in_token_out.md
""" """
import httpx import httpx
+1
View File
@@ -106,6 +106,7 @@ plugins:
examples/features/pause_resume.md: examples/rl/pause_resume.md examples/features/pause_resume.md: examples/rl/pause_resume.md
serving/online_serving/decisions.md: models/structured_decisions_models.md serving/online_serving/decisions.md: models/structured_decisions_models.md
serving/online_serving/structured_decisions.md: models/structured_decisions_models.md serving/online_serving/structured_decisions.md: models/structured_decisions_models.md
training/routed_experts_replay.md: training/routed_experts.md
markdown_extensions: markdown_extensions:
- attr_list - attr_list
+2 -1
View File
@@ -201,7 +201,8 @@ extend-exclude = ["tests/models/fixtures/*", "tests/prompts/*", "tests/tokenizer
"tests/v1/engine/test_fast_incdec_prefix_err.py", ".git/*", "csrc/cpu/sgl-kernels/*", "tests/v1/engine/test_fast_incdec_prefix_err.py", ".git/*", "csrc/cpu/sgl-kernels/*",
"rust/src/chat/src/renderer/deepseek_v32/fixtures/*", "rust/src/parser/**", "rust/src/chat/src/renderer/deepseek_v32/fixtures/*", "rust/src/parser/**",
"rust/src/text/src/output/decoded.rs", "rust/src/text/src/output/decoded.rs",
"rust/src/tokenizer/src/incremental.rs"] "rust/src/tokenizer/src/incremental.rs",
"recipes/models/*"]
ignore-hidden = false ignore-hidden = false
[tool.typos.default] [tool.typos.default]
+40
View File
@@ -0,0 +1,40 @@
# vLLM model recipes
This directory holds the deployment recipes for models served by vLLM, one
YAML per model at `models/<hf_org>/<hf_repo>.yaml`. The path mirrors the
Hugging Face ID, so `models/Qwen/Qwen3.8-27B.yaml` renders at
[recipes.vllm.ai/Qwen/Qwen3.8-27B](https://recipes.vllm.ai/Qwen/Qwen3.8-27B).
The site, the command builder, and the JSON API are built by
[vllm-project/recipes](https://github.com/vllm-project/recipes). That
repository owns the schema and the shared vocabulary (hardware profiles,
serving strategies, KV-offload deployments) and pins this directory to a
commit. Omni (image, video, audio) recipes stay in vllm-project/recipes for
now (moving to vllm-omni), and Ascend NPU content is inline in these recipes
(moving to vllm-ascend overlays).
## Editing a recipe
Start a new recipe from
[`templates/recipe.yaml`](https://github.com/vllm-project/recipes/blob/main/templates/recipe.yaml).
Fields are defined in
[SCHEMA.md](https://github.com/vllm-project/recipes/blob/main/SCHEMA.md) and
checked against [`recipe.schema.json`](https://recipes.vllm.ai/recipe.schema.json)
on every build; an unknown or misspelled key is an error. For completion and
inline errors in YAML-aware editors, start the file with
`# yaml-language-server: $schema=https://recipes.vllm.ai/recipe.schema.json`.
Follow the recipes repository's
[add-recipe skill](https://github.com/vllm-project/recipes/blob/main/.agents/skills/add-recipe/SKILL.md)
for the schema and the validation steps. To validate a change exactly as this
repository's `Recipes` PR check does, run from a checkout of
`vllm-project/recipes`:
```bash
node scripts/validate-source.mjs --source vllm --dir ~/vllm/recipes
```
To preview the site with your changes, add a gitignored `sources.local.yaml`
to that checkout with `overrides: { vllm: ~/vllm/recipes }` and run `pnpm dev`.
This works once vllm-project/recipes reads this directory as its `vllm`
source (vllm-project/recipes#1095).
+4
View File
@@ -0,0 +1,4 @@
# Recipe source manifest, read by the vllm-project/recipes renderer
# (scripts/assemble-content.mjs). Its sources.yaml grants this directory's
# role and scope; this file only declares the schema the recipes target.
schema_version: 1
@@ -0,0 +1,164 @@
meta:
title: "Seed-OSS-36B"
slug: "seed-oss-36b"
provider: "Seed (ByteDance)"
description: "ByteDance Seed-OSS 36B dense model with unique 'thinking budget' control and 512K context support"
date_added: 2025-08-28
date_updated: 2026-04-17
tasks:
- text
related_recipes: []
model:
model_id: "ByteDance-Seed/Seed-OSS-36B-Instruct"
min_vllm_version: "0.11.0"
architecture: dense
parameter_count: "36B"
active_parameters: "36B"
context_length: 524288
base_args: []
base_env: {}
dependencies:
- note: "Pinned transformers commit required for Seed-OSS tokenizer compatibility"
command: "uv pip install git+https://github.com/huggingface/transformers.git@56d68c6706ee052b445e1e476056ed92ac5eb383"
features:
tool_calling:
description: "Seed-OSS tool-call parser with automatic tool choice"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "seed_oss"
opt_in_features: []
variants:
default:
precision: bf16
vram_minimum_gb: 86
description: "Native BF16 weights on 8x GPU (TP=8)"
extra_args:
- "--tensor-parallel-size"
- "8"
compatible_strategies:
- single_node_tp
- multi_node_tp
kv_offload_support:
offloading_cpu: verified
offloading_fs: verified
hardware_overrides:
amd:
extra_args: []
extra_env:
VLLM_ROCM_USE_AITER: "1"
strategy_overrides: {}
guide: |
## Overview
Seed-OSS-36B is a dense language model from ByteDance Seed with a unique **thinking
budget** feature for controlled reasoning, and up to 512K context. Users can choose
tensor-parallel (low latency) or data-parallel (high throughput).
## Prerequisites
- Hardware: 8x GPU recommended (TP=8); also runs on consumer hardware like RTX 3090
- vLLM >= 0.11.0 (support may require main branch)
- Latest transformers for compatibility
### Install vLLM (NVIDIA)
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
uv pip install git+https://github.com/huggingface/transformers.git@56d68c6706ee052b445e1e476056ed92ac5eb383
```
### Install vLLM (AMD ROCm)
```bash
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
```
## Launch commands
NVIDIA:
```bash
vllm serve ByteDance-Seed/Seed-OSS-36B-Instruct \
--host localhost --port 8000 \
--tensor-parallel-size 8 \
--enable-auto-tool-choice \
--tool-call-parser seed_oss
```
AMD:
```bash
export VLLM_ROCM_USE_AITER=1
vllm serve ByteDance-Seed/Seed-OSS-36B-Instruct \
--tensor-parallel-size 8 \
--enable-auto-tool-choice --tool-call-parser seed_oss \
--trust-remote-code
```
Tuning:
- `--max-model-len=65536` works well; max is 512K.
- `--max-num-batched-tokens=32768` for prompt-heavy; reduce to 8K–16K for latency.
- `--gpu-memory-utilization=0.95` to maximize KV cache.
## Thinking Budget
Control the model's chain-of-thought length via `chat_template_kwargs`. Recommended
values are multiples of 512 (512, 1K, 2K, 4K, 8K, 16K). Use `0` for direct answers.
```python
from openai import OpenAI
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
model = client.models.list().data[0].id
response = client.chat.completions.create(
model=model,
messages=[
{"role": "system", "content": "You are a helpful assistant"},
{"role": "user", "content": "Janet's ducks lay 16 eggs per day..."},
],
extra_body={"chat_template_kwargs": {"thinking_budget": 512}},
)
print(response.choices[0].message.content)
```
curl:
```bash
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "ByteDance-Seed/Seed-OSS-36B-Instruct",
"messages": [{"role": "user", "content": "Explain quantum computing"}],
"chat_template_kwargs": {"thinking_budget": 512}
}'
```
The model emits `<seed:think>` blocks with `<seed:cot_budget_reflect>` markers that
report token usage against the budget.
## Benchmarking
```bash
vllm bench serve \
--backend vllm --model ByteDance-Seed/Seed-OSS-36B-Instruct \
--endpoint /v1/completions --host localhost --port 8000 \
--dataset-name random --random-input 800 --random-output 100 \
--request-rate 2 --num-prompt 100
```
## References
- [Seed-OSS-36B-Instruct on Hugging Face](https://huggingface.co/ByteDance-Seed/Seed-OSS-36B-Instruct)
@@ -0,0 +1,358 @@
meta:
title: "DiffusionGemma 26B-A4B IT"
slug: "diffusiongemma-26b-a4b-it"
provider: "Google"
description: "Google's DiffusionGemma — a block-diffusion language model built on Gemma 4's MoE backbone (26B total / 4B active). Generates tokens via iterative denoising over a fixed-length canvas rather than left-to-right autoregressive decoding, enabling higher throughput with parallel block generation."
date_added: 2026-06-11
date_updated: 2026-09-23
tasks:
- multimodal
- text
performance_headline: "Block-diffusion MoE — 26B total / 4B active, canvas-based parallel generation with ~1.9x throughput vs autoregressive baseline"
related_recipes:
- google/gemma-4-26B-A4B-it
- google/gemma-4-31B-it
hardware:
h100: verified
h200: verified
dgx_spark_gb10: verified
xeon6: verified
arc_pro_b60: verified
arc_pro_b70: verified
model:
model_id: "google/diffusiongemma-26B-A4B-it"
min_vllm_version: "0.24.0"
docker_image: "vllm/vllm-openai:gemma"
architecture: moe
parameter_count: "26B"
active_parameters: "4B"
context_length: 262144
base_args:
- "--max-num-seqs"
- "4"
base_env: {}
dependencies: []
features:
tool_calling:
description: "Enable automatic tool choice with Gemma 4 parser and chat template"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "gemma4"
- "--chat-template"
- "examples/tool_chat_template_gemma4.jinja"
reasoning:
description: "Enable structured thinking/reasoning output"
args:
- "--reasoning-parser"
- "gemma4"
opt_in_features: []
variants:
default:
precision: bf16
vram_minimum_gb: 64
description: "Full BF16 — single 80 GB NVIDIA GPU (H100/H200). Requires --max-num-seqs 4 to avoid OOM from diffusion state buffers."
supported_hardware: [h100, h200, dgx_spark_gb10, xeon6]
fp8:
model_id: "RedHatAI/diffusiongemma-26B-A4B-it-FP8-dynamic"
precision: fp8
vram_minimum_gb: 32
description: "FP8 (E4M3) weights with dynamic per-token activation quantization"
supported_hardware: [h100, h200, dgx_spark_gb10]
nvfp4:
model_id: "RedHatAI/diffusiongemma-26B-A4B-it-NVFP4"
precision: nvfp4
label: "NVFP4 - RedHatAI"
vram_minimum_gb: 24
description: "NVFP4 (4-bit) quantized weights"
supported_hardware: [dgx_spark_gb10]
nvidia_nvfp4:
model_id: "nvidia/diffusiongemma-26B-A4B-it-NVFP4"
precision: nvfp4
label: "NVFP4 - NVIDIA"
vram_minimum_gb: 24
description: "NVIDIA NVFP4 (4-bit) quantized weights"
supported_hardware: [dgx_spark_gb10]
hardware_overrides:
dgx_spark_gb10:
extra_args:
- "--gpu-memory-utilization"
- "0.8"
- "--max-num-seqs"
- "8"
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "gemma4"
- "--load-format"
- "fastsafetensors"
- "--attention-backend"
- "TRITON_ATTN"
- "--enable-prefix-caching"
- "--diffusion-config"
- '{"canvas_length":256}'
- "--override-generation-config"
- '{"max_new_tokens": null}'
- "--reasoning-parser"
- "gemma4"
- "--default-chat-template-kwargs"
- '{"enable_thinking": true}'
compatible_strategies:
- single_node_tp
hardware_overrides:
cpu: {}
xpu:
extra_args:
- "--enforce-eager"
- "--attention-backend"
- "TRITON_ATTN"
- "--tensor-parallel-size"
- "4"
strategy_overrides:
single_node_tp:
tp: 1
guide: |
## Overview
[DiffusionGemma 26B-A4B](https://huggingface.co/google/diffusiongemma-26B-A4B-it) is a block-diffusion language model built on Gemma 4's MoE backbone. Instead of generating tokens one at a time (autoregressive), it generates blocks of 256 tokens simultaneously via iterative denoising over a fixed canvas — trading higher time-to-first-token for significantly higher per-request generation throughput.
### Key Architecture Features
- **Block Diffusion**: Generates tokens in 256-token canvas blocks via iterative denoising (up to 48 steps per block).
- **MoE Backbone**: Same Gemma 4 architecture — 128 fine-grained experts with top-8 routing, 26B total / 4B active parameters.
- **Entropy-Bound Sampling**: Uses an entropy-based sampler (`diffusion_sampler: entropy_bound`, `entropy_bound: 0.1`) for the denoising process.
- **Multimodal**: Supports text + images via the Gemma 4 vision encoder.
- **Thinking Mode**: Supports structured reasoning via `<|channel>thought\n...<channel|>` delimiters.
- **Function Calling**: Supports Gemma 4's tool-call protocol (works best in thinking mode).
### Important Deployment Notes
DiffusionGemma requires several specific flags that differ from standard Gemma 4:
| Flag | Why |
|------|-----|
| `--max-num-seqs 4` | The diffusion state buffers (`self_conditioning_probs`) pre-allocate `max_seqs × canvas_length × vocab_size` tensors. With Gemma's 262K vocab and canvas_length=256, higher values cause OOM. |
| `--generation-config vllm` | The checkpoint's `generation_config.json` sets `max_tokens: 256`, which would override per-request limits. This flag ignores it. |
| `--gpu-memory-utilization 0.85` | Leaves headroom for activation memory during denoising. |
| `--hf-overrides '{"diffusion_sampler":"entropy_bound","diffusion_entropy_bound":0.1}'` | Configures the entropy-bound denoising sampler. |
| `--diffusion-config '{"canvas_length": 256}'` | Sets the canvas block size for generation. |
## Prerequisites
DiffusionGemma requires a vLLM build with diffusion model support, available in the
Gemma docker image:
```bash
docker pull vllm/vllm-openai:gemma
```
## Deployment
### Single GPU (H100/H200, BF16)
```bash
docker run -itd --name diffusiongemma \
--ipc=host --network host --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:gemma \
--model google/diffusiongemma-26B-A4B-it \
--max-model-len 262144 \
--max-num-seqs 4 \
--gpu-memory-utilization 0.85 \
--host 0.0.0.0 --port 8000
```
### Intel Xeon 6 (CPU, BF16)
DiffusionGemma runs on Intel Xeon 6 CPUs using vLLM's CPU backend with AMX BF16 acceleration.
The generated example uses TP=1. Adjust `--tensor-parallel-size` based on the
detected NUMA topology and deployment requirements.
```bash
docker run \
--privileged --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai-cpu:latest-x86_64 google/diffusiongemma-26B-A4B-it \
--max-num-seqs 4 \
--tensor-parallel-size 1 \
--enable-auto-tool-choice \
--tool-call-parser gemma4 \
--chat-template examples/tool_chat_template_gemma4.jinja \
--reasoning-parser gemma4
```
### Intel XPU (B60 / B70, BF16)
Validated on 4x Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card)
with the official vLLM XPU image `vllm/vllm-openai-xpu:latest`. `--enforce-eager`
and `--attention-backend TRITON_ATTN` are required on this backend. Select Arc Pro
B60/B70 in the hardware picker above for the exact command.
### Full-Featured Server (text + image + thinking + tools)
```bash
docker run -itd --name diffusiongemma \
--ipc=host --network host --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:gemma \
--model google/diffusiongemma-26B-A4B-it \
--max-model-len 262144 \
--max-num-seqs 4 \
--gpu-memory-utilization 0.85 \
--mm-processor-kwargs '{"max_soft_tokens": 1120}' \
--limit-mm-per-prompt '{"image": 7}' \
--enable-auto-tool-choice \
--tool-call-parser gemma4 \
--reasoning-parser gemma4 \
--host 0.0.0.0 --port 8000
```
### Jev-Style Structured Decisions (Nightly)
vLLM's structured-read support for DiffusionGemma can answer yes/no, multiple-choice,
and scored questions with a confidence value. The example server seeds a canvas with
the response template, leaves only the answer slots noisy, and reads a probability
distribution from every slot after one denoising step. It exposes the Jev-compatible
`/v1/systemone` endpoint; this is an example server rather than a standard vLLM endpoint.
Use the pinned nightly image that contains the merged structured-read support:
```bash
docker run -d --name dgemma \
--gpus '"device=0"' \
--ipc=host \
-p 8000:8000 \
-p 8011:8011 \
-v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
vllm/vllm-openai:nightly-e9757321527ca1ecd514c07c1418dd2c53da3d19 \
google/diffusiongemma-26B-A4B-it \
--served-model-name dgemma \
--diffusion-config '{"canvas_length":64}' \
--max-logprobs 32 \
--enable-prefix-caching
```
Wait for the vLLM server to become healthy, then start the example server in the same
container:
```bash
until curl -fsS localhost:8000/health >/dev/null; do sleep 2; done
docker exec -d dgemma \
python /vllm-workspace/examples/features/structured_diffusion/structured_server.py \
--upstream http://127.0.0.1:8000 \
--tokenizer google/diffusiongemma-26B-A4B-it \
--canvas 64
```
After the example server becomes healthy, send a Jev-style decision request:
```bash
until curl -fsS localhost:8011/health >/dev/null; do sleep 2; done
curl -sS localhost:8011/v1/systemone \
-H 'content-type: application/json' \
-d '{
"model": "jev-latest",
"state": {"ticket": "Everything is down, demo at noon."},
"questions": {
"urgent": {
"type": "noul",
"instructions": "Needs a reply within the hour?"
}
}
}'
```
## Client Usage
### Text Generation
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="google/diffusiongemma-26B-A4B-it",
messages=[{"role": "user", "content": "Write a poem about the ocean."}],
max_tokens=512,
temperature=0.7,
)
print(response.choices[0].message.content)
```
### Offline Inference
```python
from vllm import LLM, SamplingParams
from transformers import AutoTokenizer
model_path = "google/diffusiongemma-26B-A4B-it"
tokenizer = AutoTokenizer.from_pretrained(model_path)
llm = LLM(
model=model_path,
max_num_seqs=4,
gpu_memory_utilization=0.85,
hf_overrides={
"diffusion_sampler": "entropy_bound",
"diffusion_entropy_bound": 0.1,
},
diffusion_config={"canvas_length": 256},
)
messages = [{"role": "user", "content": "Explain quantum computing in simple terms."}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
outputs = llm.generate(prompt, SamplingParams(temperature=0.0, max_tokens=1024))
print(outputs[0].outputs[0].text)
```
### Thinking Mode
Launch the server with `--reasoning-parser gemma4`, then enable per-request:
```python
response = client.chat.completions.create(
model="google/diffusiongemma-26B-A4B-it",
messages=[{"role": "user", "content": "What is the derivative of x^3 * ln(x)?"}],
max_tokens=32768,
extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)
```
## Performance Characteristics
Compared to the autoregressive Gemma 4 26B-A4B baseline (single H100, concurrency=1, SPEED-Bench):
| Metric | Gemma 4 26B-A4B (baseline) | DiffusionGemma 26B-A4B |
|--------|---------------------------|------------------------|
| Output TPS | 199 tok/s | 375 tok/s (**1.9×**) |
| E2E Request Time (mean) | 2.87s | 0.88s (**3.3× faster**) |
| TTFT (mean) | 53ms | 489ms (higher due to canvas denoising setup) |
| Per-request Gen TPS (mean) | 205 tok/s | 1,282 tok/s (**6.2×**) |
The diffusion model trades higher time-to-first-token for significantly higher generation throughput. It generates fewer total tokens per prompt on average (shorter outputs) but completes requests much faster.
## Known Limitations
- **TTFT**: ~10× higher than autoregressive baseline — the model must denoise an entire canvas block before emitting the first token.
- **`--max-num-seqs`**: Must be kept low (≤4) due to the large diffusion state tensors. Higher values cause CUDA OOM.
- **Audio**: Not supported — the diffusion checkpoints do not include an audio encoder.
- **Function Calling**: Works best in thinking mode. Non-thinking mode may answer directly instead of emitting tool calls.
## References
- [Model card](https://huggingface.co/google/diffusiongemma-26B-A4B-it)
- [vLLM structured-read support for DiffusionGemma](https://github.com/vllm-project/vllm/pull/57250)
- [djev-run Cloud Run deployment](https://github.com/taeold/djev-run)
- [Gemma 4 26B-A4B (autoregressive baseline)](https://huggingface.co/google/gemma-4-26B-A4B-it)
- [Gemma docs](https://ai.google.dev/gemma/docs)
+316
View File
@@ -0,0 +1,316 @@
meta:
title: "Gemma 4 12B IT"
slug: "gemma-4-12b-it"
provider: "Google"
description: "Google's encoder-free unified Gemma 4 dense model (12B) with native text, image, and audio, plus thinking mode and tool-use protocol."
date_added: 2026-06-03
date_updated: 2026-09-09
tasks:
- multimodal
- text
performance_headline: "Encoder-free unified multimodal model with audio, structured thinking, and function calling — runs on a single 40 GB+ GPU"
related_recipes:
- google/gemma-4-E4B-it
- google/gemma-4-26B-A4B-it
- google/gemma-4-31B-it
hardware:
h100: verified
arc_pro_b60: verified
arc_pro_b70: verified
model:
model_id: "google/gemma-4-12B-it"
min_vllm_version: "0.23.0"
docker_image: "vllm/vllm-openai:gemma4-unified"
install:
docker: {}
pip: {}
architecture: dense
parameter_count: "12B"
active_parameters: "12B"
context_length: 131072
base_args: []
base_env: {}
dependencies:
- note: "Audio extras — only needed when serving the audio modality"
command: 'uv pip install "vllm[audio]"'
optional: true
features:
tool_calling:
description: "Enable automatic tool choice with Gemma 4 parser and chat template"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "gemma4"
- "--chat-template"
- "examples/tool_chat_template_gemma4.jinja"
reasoning:
description: "Enable structured thinking/reasoning output"
args:
- "--reasoning-parser"
- "gemma4"
spec_decoding:
description: "MTP speculative decoding for accelerated inference"
args:
- "--speculative-config"
- '{"model":"google/gemma-4-12B-it-assistant","num_speculative_tokens":4}'
opt_in_features:
- spec_decoding
variants:
default:
precision: bf16
vram_minimum_gb: 29
description: "Full BF16 — single 40 GB+ NVIDIA GPU"
nvfp4:
model_id: "RedHatAI/gemma-4-12B-it-NVFP4"
precision: nvfp4
vram_minimum_gb: 8
description: "RedHatAI compressed-tensors NVFP4 quantization for Blackwell GPUs"
extra_args:
- "--kv-cache-dtype"
- "fp8"
w4a16:
model_id: "google/gemma-4-12B-it-qat-w4a16-ct"
precision: int4
vram_minimum_gb: 9
description: "W4A16 QAT (4-bit weights, 16-bit activations, group_size=32) via compressed-tensors — runs on any GPU"
fp8:
model_id: "RedHatAI/gemma-4-12B-it-FP8-Dynamic"
precision: fp8
vram_minimum_gb: 15
description: "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization"
extra_args:
- "--kv-cache-dtype"
- "fp8"
compatible_strategies:
- single_node_tp
- multi_node_tp
hardware_overrides:
xpu:
extra_args:
- "--enforce-eager"
- "--attention-backend"
- "flash_attn"
- "--trust-remote-code"
- "--gpu-memory-utilization"
- "0.9"
- "--no-enable-prefix-caching"
- "--max-num-batched-tokens"
- "8192"
- "--max-model-len"
- "5120"
- "--tensor-parallel-size"
- "2"
strategy_overrides:
single_node_tp:
tp: 1
guide: |
## Overview
[Gemma 4](https://ai.google.dev/gemma/docs) is Google's most capable open model family, featuring a unified multimodal architecture that natively processes text, images, and audio. The **12B Unified** model is *encoder-free*: instead of dedicated vision/audio encoders, it projects raw image patches and audio waveforms directly into the decoder's embedding space through lightweight linear layers, so every modality flows into a single decoder-only transformer. Gemma 4 models support structured thinking/reasoning, function calling with a custom tool-use protocol, and dynamic vision resolution — all available through vLLM's OpenAI-compatible API.
### Key Features
- **Encoder-free multimodality**: Text, image, and audio inputs handled by a single decoder-only transformer — no separate encoders to load, reducing multimodal latency.
- **Dual Attention**: Alternating sliding-window (local, 1024 tokens) and global attention; the final layer is always global, with unified Keys/Values and Proportional RoPE on global layers for long-context efficiency.
- **Thinking Mode**: Structured reasoning via `<|channel>thought\n...<channel|>` delimiters.
- **Function Calling**: Custom tool-call protocol with dedicated special tokens.
- **Native System Prompt Support**: First-class `system` role.
### Supported Variants
Dense:
- `google/gemma-4-E2B-it` (effective 2B)
- `google/gemma-4-E4B-it` (effective 4B)
- `google/gemma-4-12B-it` (11.95B, encoder-free)
- `google/gemma-4-31B-it` (31B)
MoE:
- `google/gemma-4-26B-A4B-it` (26B total / 4B active)
> **Context length**: this recipe pins `context_length` to the value in `config.json` (`max_position_embeddings = 131072`, i.e. 128K). The Gemma 4 model card markets the 12B at up to 256K — raise `--max-model-len` only if your vLLM build accepts it.
TPU support is provided through [vLLM TPU](https://github.com/vllm-project/tpu-inference) with recipes for [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) and [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/).
> **Nightly required**: support for the encoder-free 12B Unified model landed in [vllm-project/vllm#44429](https://github.com/vllm-project/vllm/pull/44429) and has not yet shipped in a stable release. Use the pinned Docker image (recommended) or a nightly pip wheel.
## Prerequisites
### Docker (recommended)
```bash
docker pull vllm/vllm-openai:gemma4-unified # NVIDIA (CUDA 13; append -cu129 for CUDA 12.9 hosts)
docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70)
```
TPU images are published separately by [vllm-project/tpu-inference](https://github.com/vllm-project/tpu-inference); see the Trillium / Ironwood tpu-recipes below for the pinned tag.
### pip (nightly, NVIDIA CUDA)
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --pre \
--extra-index-url https://wheels.vllm.ai/nightly/cu129 \
--extra-index-url https://download.pytorch.org/whl/cu129 \
--index-strategy unsafe-best-match
```
## Deployment Configurations
### Quick Start (Single GPU, BF16)
The 12B fits comfortably on a single 40 GB+ GPU.
```bash
vllm serve google/gemma-4-12B-it \
--max-model-len 32768 \
--gpu-memory-utilization 0.90
```
### Full-Featured Server Launch
Enables text, image, audio, thinking, and tool calling:
```bash
vllm serve google/gemma-4-12B-it \
--max-model-len 16384 \
--gpu-memory-utilization 0.90 \
--enable-auto-tool-choice \
--reasoning-parser gemma4 \
--tool-call-parser gemma4 \
--chat-template examples/tool_chat_template_gemma4.jinja \
--limit-mm-per-prompt '{"image": 4, "audio": 1}' \
--async-scheduling \
--host 0.0.0.0 \
--port 8000
```
### Docker (NVIDIA)
```bash
docker run -itd --name gemma4 \
--ipc=host --network host --shm-size 16G --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:gemma4-unified \
--model google/gemma-4-12B-it \
--max-model-len 32768 \
--gpu-memory-utilization 0.90 \
--host 0.0.0.0 --port 8000
```
On CUDA 12.9 hosts, use the `vllm/vllm-openai:gemma4-unified-cu129` tag instead.
### Docker (Intel XPU B60 / B70, TP=2, Full-Featured)
Validated on 2× Intel Arc Pro B60 / B70 with the official vLLM XPU image.
```bash
docker run --device /dev/dri \
-v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
--privileged --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai-xpu:latest \
google/gemma-4-12B-it \
--enforce-eager \
--attention-backend flash_attn \
--host 0.0.0.0 \
--port 8000 \
--trust-remote-code \
--enable-auto-tool-choice \
--reasoning-parser gemma4 \
--tool-call-parser gemma4 \
--chat-template examples/tool_chat_template_gemma4.jinja \
--limit-mm-per-prompt '{"image": 4, "audio": 1}' \
--async-scheduling \
--gpu-memory-utilization 0.9 \
--no-enable-prefix-caching \
--max-num-batched-tokens 8192 \
--max-model-len 5120 \
--tensor-parallel-size 2
```
### Docker (Cloud TPU — Trillium / Ironwood)
TPU uses the separate `vllm/vllm-tpu` image (no pip wheel). Pull the tag specified by the upstream [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) or [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/) recipe, then run:
```bash
docker run -itd --name gemma4-tpu \
--privileged --network host --shm-size 16G \
-v /dev/shm:/dev/shm -e HF_TOKEN=$HF_TOKEN \
vllm/vllm-tpu:latest \
--model google/gemma-4-12B-it \
--max-model-len 16384 \
--disable_chunked_mm_input \
--host 0.0.0.0 --port 8000
```
## Client Usage
### Text Generation
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="google/gemma-4-12B-it",
messages=[{"role": "user", "content": "Write a poem about the ocean."}],
max_tokens=512, temperature=0.7,
)
print(response.choices[0].message.content)
```
### Image Understanding
```python
response = client.chat.completions.create(
model="google/gemma-4-12B-it",
messages=[{"role": "user", "content": [
{"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg"}},
{"type": "text", "text": "Describe this image in detail."},
]}],
max_tokens=1024,
)
```
### Audio
Requires `uv pip install "vllm[audio]"`.
```bash
vllm serve google/gemma-4-12B-it \
--max-model-len 8192 \
--limit-mm-per-prompt '{"image": 4, "audio": 1}'
```
### Thinking Mode
```bash
vllm serve google/gemma-4-12B-it \
--max-model-len 16384 \
--enable-auto-tool-choice \
--reasoning-parser gemma4 \
--tool-call-parser gemma4 \
--chat-template examples/tool_chat_template_gemma4.jinja
```
Enable thinking per-request via `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`, or default-on with `--default-chat-template-kwargs '{"enable_thinking": true}'`.
### Structured Outputs
vLLM guided decoding constrains output to a JSON schema. Include semantic instructions in the system prompt — the model does not see schema descriptions.
## Configuration Tips
- Set `--max-model-len` to match your workload.
- `--gpu-memory-utilization 0.90–0.95` maximizes KV cache.
- Image-only workloads: pass `--limit-mm-per-prompt.audio 0`.
- Text-only workloads: pass `--limit-mm-per-prompt '{"image": 0, "audio": 0}'` to skip MM profiling.
- `--async-scheduling` improves throughput.
- FP8 KV cache (`--kv-cache-dtype fp8`) saves ~50% KV memory.
## Speculative Decoding (MTP)
Enable the **Spec Decoding** feature toggle (above) or add `--speculative-config` manually to use MTP drafting with the [assistant model](https://huggingface.co/google/gemma-4-12B-it-assistant). Recommended `num_speculative_tokens`: 4–8 for this model.
> **Note:** MTP speculative decoding for Gemma 4 requires the nightly build — the pinned `vllm/vllm-openai:gemma4-unified` image or a nightly pip wheel. The standard `:latest` stable tag does not include this feature.
## References
- [Model card](https://huggingface.co/google/gemma-4-12B-it)
- [vLLM support PR (#44429)](https://github.com/vllm-project/vllm/pull/44429)
- [Assistant (MTP draft) model](https://huggingface.co/google/gemma-4-12B-it-assistant)
- [Gemma docs](https://ai.google.dev/gemma/docs)
- [vLLM Gemma 4 tool-call template](https://github.com/vllm-project/vllm/blob/main/examples/tool_chat_template_gemma4.jinja)
- [TPU recipes: Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4)
- [TPU recipes: Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/)
@@ -0,0 +1,602 @@
meta:
title: "Gemma 4 26B-A4B IT"
slug: "gemma-4-26b-a4b-it"
provider: "Google"
description: "Google's Gemma 4 MoE multimodal model (26B total / 4B active) with 128 fine-grained experts, top-8 routing, thinking mode, and tool-use protocol."
date_added: 2026-04-02
date_updated: 2026-09-27
tasks:
- multimodal
- text
performance_headline: "MoE multimodal model — 26B total / 4B active, 128 experts with top-8 routing"
related_recipes:
- google/gemma-4-E2B-it
- google/gemma-4-E4B-it
- google/gemma-4-31B-it
platforms:
- id: modal
blurb: "Serverless deploy via the Gemma 4 Modal script bundled with this recipe."
script: https://github.com/vllm-project/recipes/blob/main/Google/gemma4-modal.py
install: |
curl -O https://raw.githubusercontent.com/vllm-project/recipes/main/Google/gemma4-modal.py
pip install modal
modal setup
modal deploy gemma4-modal.py
modal run gemma4-modal.py
hardware:
h100: verified
h200: verified
mi300x: verified
mi325x: verified
mi355x: verified
trillium: verified
ironwood: verified
xeon6: verified
dgx_spark_gb10: verified
dgx_station_gb300: verified
rtx_pro_6000: verified
rtx_5090: verified
arc_pro_b60: verified
arc_pro_b70: verified
model:
model_id: "google/gemma-4-26B-A4B-it"
min_vllm_version: "0.25.0"
architecture: moe
parameter_count: "26B"
active_parameters: "4B"
context_length: 131072
base_args: []
base_env: {}
dependencies: []
features:
tool_calling:
description: "Enable automatic tool choice with Gemma 4 parser and chat template"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "gemma4"
- "--chat-template"
- "examples/tool_chat_template_gemma4.jinja"
hardware_overrides:
dgx_spark_gb10:
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "gemma4"
reasoning:
description: "Enable structured thinking/reasoning output"
args:
- "--reasoning-parser"
- "gemma4"
text_only:
description: "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
args:
- "--language-model-only"
spec_decoding:
description: "MTP speculative decoding for accelerated inference"
default_mode: mtp
modes:
mtp:
label: "MTP"
description: "Built-in Multi-Token Prediction, 4 draft tokens."
args:
- "--speculative-config"
- '{"model":"google/gemma-4-26B-A4B-it-assistant","num_speculative_tokens":4}'
# The single-GPU Blackwell desktops run the draft MoE on the Triton
# backend — the fused CUTLASS path the drafter picks by default is not
# built for GB10/GB300/RTX PRO in the validated images.
hardware_overrides:
dgx_spark_gb10: &gemma4_mtp_triton
args:
- "--speculative-config"
- '{"method":"mtp","model":"google/gemma-4-26B-A4B-it-assistant","num_speculative_tokens":4,"moe_backend":"triton"}'
dgx_station_gb300: *gemma4_mtp_triton
rtx_pro_6000: *gemma4_mtp_triton
eagle3:
label: "Eagle3"
description: "Eagle3 speculative decoding with externally trained draft model; the served checkpoint is unchanged."
variants:
- default
args:
- "--speculative-config"
- '{"model": "RedHatAI/gemma-4-26B-A4B-it-speculator.eagle3", "num_speculative_tokens": 3, "method": "eagle3"}'
opt_in_features:
- text_only
- spec_decoding
variants:
default:
precision: bf16
vram_minimum_gb: 64
description: "Full BF16 — single 80 GB NVIDIA GPU or 1x MI300X/MI325X/MI350X/MI355X or 2x Xeon6/Xeon5 NUMA nodes"
supported_hardware: [h100, h200, mi300x, mi325x, mi355x, trillium, ironwood, xeon6, dgx_spark_gb10, dgx_station_gb300, rtx_pro_6000, rtx_5090]
hardware_overrides:
dgx_spark_gb10:
extra_env:
VLLM_USE_V2_MODEL_RUNNER: "1"
dgx_station_gb300:
extra_env:
VLLM_USE_V2_MODEL_RUNNER: "1"
rtx_pro_6000:
extra_env:
VLLM_USE_V2_MODEL_RUNNER: "1"
extra_args:
- "--gpu-memory-utilization"
- "0.9"
fp8:
model_id: "RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic"
precision: fp8
vram_minimum_gb: 32
description: "FP8 (E4M3) weights with dynamic per-token activation quantization — Hopper or Blackwell"
supported_hardware: [h100, h200, dgx_spark_gb10, dgx_station_gb300, rtx_pro_6000, rtx_5090]
extra_args:
- "--kv-cache-dtype"
- "fp8"
hardware_overrides:
dgx_spark_gb10:
extra_env:
VLLM_USE_V2_MODEL_RUNNER: "1"
dgx_station_gb300:
extra_env:
VLLM_USE_V2_MODEL_RUNNER: "1"
rtx_pro_6000:
extra_env:
VLLM_USE_V2_MODEL_RUNNER: "1"
extra_args:
- "--gpu-memory-utilization"
- "0.9"
rtx_5090:
extra_env:
VLLM_USE_V2_MODEL_RUNNER: "1"
extra_args:
- "--gpu-memory-utilization"
- "0.92"
- "--max-model-len"
- "131072"
- "--max-num-seqs"
- "1"
nvfp4:
model_id: "RedHatAI/gemma-4-26B-A4B-it-NVFP4"
precision: nvfp4
label: "NVFP4 - RedHatAI"
vram_minimum_gb: 16
description: "NVFP4 (4-bit) quantized weights — requires Blackwell (B200/B300)"
supported_hardware: [dgx_spark_gb10, dgx_station_gb300, rtx_pro_6000, rtx_5090]
extra_args:
- "--kv-cache-dtype"
- "fp8"
hardware_overrides:
dgx_spark_gb10:
extra_env:
VLLM_USE_V2_MODEL_RUNNER: "1"
dgx_station_gb300:
extra_env:
VLLM_USE_V2_MODEL_RUNNER: "1"
rtx_pro_6000:
extra_env:
VLLM_USE_V2_MODEL_RUNNER: "1"
extra_args:
- "--gpu-memory-utilization"
- "0.9"
rtx_5090:
extra_env:
VLLM_USE_V2_MODEL_RUNNER: "1"
extra_args:
- "--gpu-memory-utilization"
- "0.9"
- "--max-num-batched-tokens"
- "4096"
- "--max-num-seqs"
- "4"
nvidia_nvfp4:
model_id: "nvidia/Gemma-4-26B-A4B-NVFP4"
precision: nvfp4
label: "NVFP4 - NVIDIA"
vram_minimum_gb: 16
description: "NVIDIA ModelOpt NVFP4 (4-bit) quantized weights — validated on the single-GPU Blackwell desktops (DGX Spark GB10, DGX Station GB300, RTX PRO 6000)"
supported_hardware: [dgx_spark_gb10, dgx_station_gb300, rtx_pro_6000]
hardware_overrides:
dgx_spark_gb10:
# vLLM 0.24 and the generic nightly both fail on GB10 with this
# checkpoint; the dated ARM64 community build is the tested runtime, so
# the wheel tab is hidden here rather than offering a command that
# won't start.
docker_image: "eugr/spark-vllm:nightly-20260704"
# This image's entrypoint is nvidia_entrypoint.sh, not `vllm serve`.
docker_serve_command: "vllm serve"
install:
pip: false
extra_env:
VLLM_USE_V2_MODEL_RUNNER: "1"
extra_args:
# GB10 shares its 128 GB with the CPU, so leave headroom for the host.
- "--gpu-memory-utilization"
- "0.8"
- "--max-model-len"
- "262144"
- "--max-num-seqs"
- "8"
- "--max-num-batched-tokens"
- "8192"
- "--enable-prefix-caching"
- "--load-format"
- "fastsafetensors"
dgx_station_gb300:
extra_env:
VLLM_USE_V2_MODEL_RUNNER: "1"
extra_args:
# 252 GB on one GB300 — full context at high utilization and a 64-way
# batch.
- "--gpu-memory-utilization"
- "0.92"
- "--max-model-len"
- "262144"
- "--max-num-seqs"
- "64"
- "--max-num-batched-tokens"
- "8192"
- "--enable-chunked-prefill"
- "--enable-prefix-caching"
- "--load-format"
- "fastsafetensors"
rtx_pro_6000:
extra_env:
VLLM_USE_V2_MODEL_RUNNER: "1"
extra_args:
- "--gpu-memory-utilization"
- "0.92"
- "--max-model-len"
- "262144"
- "--max-num-seqs"
- "8"
- "--max-num-batched-tokens"
- "8192"
- "--enable-prefix-caching"
- "--load-format"
- "fastsafetensors"
rtx_5090:
extra_env:
VLLM_USE_V2_MODEL_RUNNER: "1"
extra_args:
- "--kv-cache-dtype"
- "fp8"
- "--gpu-memory-utilization"
- "0.9"
- "--max-num-batched-tokens"
- "4096"
- "--enable-prefix-caching"
- "--enable-chunked-prefill"
- "--async-scheduling"
- "--max-num-seqs"
- "4"
compatible_strategies:
- single_node_tp
- single_node_tep
- single_node_dep
- multi_node_tp
- multi_node_tep
- multi_node_dep
hardware_overrides:
xpu:
extra_args:
- "--enforce-eager"
- "--attention-backend"
- "flash_attn"
- "--trust-remote-code"
- "--gpu-memory-utilization"
- "0.9"
- "--no-enable-prefix-caching"
- "--max-num-batched-tokens"
- "8192"
- "--max-model-len"
- "5120"
- "--tensor-parallel-size"
- "4"
strategy_overrides:
single_node_tp:
tp: 1
guide: |
## Overview
[Gemma 4 26B-A4B](https://huggingface.co/google/gemma-4-26B-A4B-it) is the Mixture-of-Experts member of Google's Gemma 4 family — 26B total parameters with only 4B active per token via 128 fine-grained experts and top-8 routing. It supports text + images natively, structured thinking, function calling, and dynamic vision resolution.
Intel Xeon 6 CPU support remains verified. Benchmark-specific TP/DP, CPU binding,
sequence limits, and repro-only timeouts remain deployment choices.
### Key Features
- **MoE**: 128 fine-grained experts with top-8 routing and custom GELU-activated FFN.
- **Multimodal**: Text + images natively (video via custom frame-extraction pipeline). Audio is only supported on the smaller E2B/E4B variants.
- **Dual Attention**: Alternating sliding-window (local) and global attention with different head dimensions.
- **Thinking Mode**: Structured reasoning via `<|channel>thought\n...<channel|>` delimiters.
- **Function Calling**: Custom tool-call protocol with dedicated special tokens.
- **Dynamic Vision Resolution**: Per-request configurable vision token budget (70, 140, 280, 560, 1120 tokens).
TPU support is provided through [vLLM TPU](https://github.com/vllm-project/tpu-inference) with recipes for [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) and [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/).
## Prerequisites
- Hardware: Intel Xeon 6/Xeon 5 CPUs
- vLLM >= 0.25.0
### pip (NVIDIA CUDA)
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --pre \
--extra-index-url https://wheels.vllm.ai/nightly/cu129 \
--extra-index-url https://download.pytorch.org/whl/cu129 \
--index-strategy unsafe-best-match
```
### pip (AMD ROCm: MI300X, MI325X, MI350X, MI355X)
Requires Python 3.12, ROCm 7.2.1, glibc >= 2.35 (Ubuntu 22.04+).
```bash
uv venv --python 3.12
source .venv/bin/activate
uv pip install vllm --pre \
--extra-index-url https://wheels.vllm.ai/rocm/nightly/rocm721 --upgrade
```
### pip (Intel Xeon 6 CPUs)
For Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions.
### Docker
```bash
docker pull vllm/vllm-openai:gemma4-0505-cu129 # NVIDIA Hopper (H100/H200, CUDA 12.9)
docker pull vllm/vllm-openai:gemma4-0505-cu130 # NVIDIA Blackwell (B200/B300, CUDA 13.0)
docker pull vllm/vllm-openai-rocm:latest # AMD
docker pull vllm/vllm-openai-cpu:latest-x86_64 # For Intel Xeon 6
docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70)
```
TPU images are published separately by [vllm-project/tpu-inference](https://github.com/vllm-project/tpu-inference); see the Trillium / Ironwood tpu-recipes below for the pinned tag.
## Deployment Configurations
### 26B MoE on 1x A100/H100 (BF16)
```bash
vllm serve google/gemma-4-26B-A4B-it \
--max-model-len 32768 \
--gpu-memory-utilization 0.90
```
### Full-Featured Server Launch
Enables text, image, thinking, and tool calling:
```bash
vllm serve google/gemma-4-26B-A4B-it \
--max-model-len 16384 \
--gpu-memory-utilization 0.90 \
--enable-auto-tool-choice \
--reasoning-parser gemma4 \
--tool-call-parser gemma4 \
--chat-template examples/tool_chat_template_gemma4.jinja \
--limit-mm-per-prompt.image 4 \
--async-scheduling \
--host 0.0.0.0 \
--port 8000
```
### Docker (NVIDIA)
```bash
docker run -itd --name gemma4-moe \
--ipc=host --network host --shm-size 16G --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:gemma4-0505-cu129 \
--model google/gemma-4-26B-A4B-it \
--max-model-len 32768 \
--gpu-memory-utilization 0.90 \
--host 0.0.0.0 --port 8000
```
Swap `vllm/vllm-openai:gemma4-0505-cu129` for `vllm/vllm-openai:gemma4-0505-cu130` on Blackwell (B200/B300).
### Docker (AMD MI300X/MI325X/MI350X/MI355X)
```bash
docker run -itd --name gemma4-rocm \
--ipc=host --network=host --privileged \
--cap-add=CAP_SYS_ADMIN --device=/dev/kfd --device=/dev/dri \
--group-add=video --cap-add=SYS_PTRACE \
--security-opt=seccomp=unconfined --shm-size 16G \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai-rocm:latest \
--model google/gemma-4-26B-A4B-it \
--host 0.0.0.0 --port 8000
```
### Docker (Cloud TPU — Trillium / Ironwood)
TPU uses the separate `vllm/vllm-tpu` image (no pip wheel). Pull the tag specified by the upstream [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) or [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/) recipe, then run:
```bash
docker run -itd --name gemma4-tpu \
--privileged --network host --shm-size 16G \
-v /dev/shm:/dev/shm -e HF_TOKEN=$HF_TOKEN \
vllm/vllm-tpu:latest \
--model google/gemma-4-26B-A4B-it \
--tensor-parallel-size 8 \
--max-model-len 16384 \
--disable_chunked_mm_input \
--host 0.0.0.0 --port 8000
```
Trillium requires a 4-chip slice minimum; Ironwood runs on a single chip.
### Intel Xeon 6 Deployment via Docker
Launch the x86 CPU vLLM Docker container for `google/gemma-4-26B-A4B-it`:
```bash
docker run \
--privileged --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai-cpu:latest-x86_64 google/gemma-4-26B-A4B-it \
--tensor-parallel-size 1 \
--enable-auto-tool-choice \
--tool-call-parser gemma4 \
--chat-template examples/tool_chat_template_gemma4.jinja \
--reasoning-parser gemma4
```
For additional Intel Xeon 6 deployment details, see the Intel Software Catalog entries for [Gemma 4 26B-A4B IT](https://aiswcatalog.intel.com/models/google-gemma-4-26b-a4b-it).
The following settings are intentionally not prescribed as portable CPU model
defaults because they depend on the workload, hardware, or runtime environment:
- `--max-num-batched-tokens`: scheduler/throughput tuning.
- `--max-num-seqs`: concurrency and scheduler-capacity tuning.
- `--gpu-memory-utilization`: platform memory-budget tuning.
- `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning.
- `VLLM_ENGINE_ITERATION_TIMEOUT_S`: operational runtime timeout.
These settings may still appear in validated hardware-specific overrides. Tune
them at deployment time based on platform resources, workload shape, and
latency/throughput goals.
### Docker (Intel XPU B60 / B70, TP=4, Full-Featured)
Validated on 4x Intel Arc Pro B60 / B70 with the official vLLM XPU image.
```bash
docker run --device /dev/dri \
-v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
--privileged --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai-xpu:latest \
google/gemma-4-26B-A4B-it \
--enforce-eager \
--attention-backend flash_attn \
--host 0.0.0.0 \
--port 8000 \
--trust-remote-code \
--enable-auto-tool-choice \
--reasoning-parser gemma4 \
--tool-call-parser gemma4 \
--chat-template examples/tool_chat_template_gemma4.jinja \
--limit-mm-per-prompt.image 4 \
--async-scheduling \
--gpu-memory-utilization 0.9 \
--no-enable-prefix-caching \
--max-num-batched-tokens 8192 \
--max-model-len 5120 \
--tensor-parallel-size 4
```
## Client Usage
### Text Generation
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="google/gemma-4-26B-A4B-it",
messages=[{"role": "user", "content": "Write a poem about the ocean."}],
max_tokens=512, temperature=0.7,
)
print(response.choices[0].message.content)
```
### Image Understanding
```python
response = client.chat.completions.create(
model="google/gemma-4-26B-A4B-it",
messages=[{"role": "user", "content": [
{"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg"}},
{"type": "text", "text": "Describe this image in detail."},
]}],
max_tokens=1024,
)
```
### Thinking Mode
```bash
vllm serve google/gemma-4-26B-A4B-it \
--max-model-len 16384 \
--reasoning-parser gemma4 \
--tool-call-parser gemma4 \
--enable-auto-tool-choice \
--chat-template examples/tool_chat_template_gemma4.jinja
```
Enable per-request via `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`.
### Dynamic Vision Resolution
Supported values: 70, 140, 280 (default), 560, 1120 tokens/image.
```bash
vllm serve google/gemma-4-26B-A4B-it \
--mm-processor-kwargs '{"max_soft_tokens": 560}'
```
## Configuration Tips
- Set `--max-model-len` to match your workload.
- `--gpu-memory-utilization 0.90-0.95` maximizes KV cache.
- Text-only workloads: `--limit-mm-per-prompt '{"image": 0, "audio": 0}'`.
- `--async-scheduling` improves throughput.
- FP8 KV cache (`--kv-cache-dtype fp8`) saves ~50% KV memory.
- For MoE, TEP (tensor-expert parallelism) and DEP (data-expert parallelism) strategies scale better than pure TP at large node counts.
## Quantized Variants
Three pre-quantized checkpoints are available:
- [`RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic`](https://huggingface.co/RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic) — FP8 (E4M3) weights with dynamic per-token activation quantization; runs on Hopper and Blackwell.
- [`RedHatAI/gemma-4-26B-A4B-it-NVFP4`](https://huggingface.co/RedHatAI/gemma-4-26B-A4B-it-NVFP4) — NVFP4 (4-bit) weights; requires Blackwell (B200/B300).
- [`nvidia/Gemma-4-26B-A4B-NVFP4`](https://huggingface.co/nvidia/Gemma-4-26B-A4B-NVFP4) — NVIDIA ModelOpt NVFP4 (4-bit) weights; this is the checkpoint validated on the single-GPU Blackwell desktops below.
Pick them from the **Variant** dropdown above, or pass the repo id directly to `vllm serve`.
### NVFP4 on the Blackwell desktops
[`nvidia/Gemma-4-26B-A4B-NVFP4`](https://huggingface.co/nvidia/Gemma-4-26B-A4B-NVFP4)
has been run end to end on DGX Spark (GB10), DGX Station (GB300) and RTX PRO
6000, in each case on one GPU at TP1 with MTP drafting against
[`google/gemma-4-26B-A4B-it-assistant`](https://huggingface.co/google/gemma-4-26B-A4B-it-assistant).
Select the **NVIDIA NVFP4** variant, enable **Spec Decoding**, and switch the
hardware pill — memory utilization, batch size, prefill scheduling and the
drafter's MoE backend all differ per box and are encoded per hardware, so the
generated command is the tested one. Highlights:
- **DGX Spark (GB10)** — 128 GB shared with the CPU, so `--gpu-memory-utilization 0.8`
and an 8-way batch. Serve from `eugr/spark-vllm:nightly-20260704`: vLLM 0.24
and the generic nightly both fail on this checkpoint, so the pip tab is
hidden on this hardware.
- **DGX Station (GB300)** — 252 GB on one GPU, so `--gpu-memory-utilization 0.92`
with chunked prefill and a 64-way batch.
- **RTX PRO 6000** — 96 GB, `--gpu-memory-utilization 0.92` and an 8-way batch.
All three run the full 262,144-token context, above the 131,072 in
`config.json`, and pass `"moe_backend":"triton"` in `--speculative-config`
because the drafter's default fused MoE path is not built in these images.
The GB300 and RTX PRO 6000 runs used an internal NVIDIA nightly vLLM build.
As noted under Speculative Decoding below, Gemma 4 MTP has not landed in a
stable release, so the default `vllm/vllm-openai:latest` tag will not serve
these commands with Spec Decoding enabled — use a nightly wheel or a nightly
image on those two boxes.
## Throughput vs Latency
| Goal | TP | `--max-num-seqs` | Notes |
|------|----|------------------|-------|
| Max throughput | 1-2 | 256-512 | Best tok/s per GPU |
| Min latency | 4-8 | 8-16 | Best TTFT/TPOT |
| Balanced | 2 | 128 | Mixed workloads |
## Speculative Decoding (MTP)
Enable the **Spec Decoding** feature toggle (above) or add `--speculative-config` manually to use MTP drafting with the [assistant model](https://huggingface.co/google/gemma-4-26B-A4B-it-assistant). Recommended `num_speculative_tokens`: 4 for this model. See the [Gemma 4 usage guide](../../Google/Gemma4) for details and benchmarks.
> **Note:** MTP speculative decoding for Gemma 4 is only available on the vLLM nightly build — it has not yet landed in a stable release. Install via the nightly wheel (`uv pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly/cu129 …`) or use the `vllm/vllm-openai:gemma4-0505-cu129` / `vllm/vllm-openai:gemma4-0505-cu130` images above; the standard `:latest` stable tag does not include this feature.
## References
- [Model card](https://huggingface.co/google/gemma-4-26B-A4B-it)
- [FP8 variant](https://huggingface.co/RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic)
- [NVFP4 variant](https://huggingface.co/RedHatAI/gemma-4-26B-A4B-it-NVFP4)
- [NVIDIA ModelOpt NVFP4 variant](https://huggingface.co/nvidia/Gemma-4-26B-A4B-NVFP4)
- [Gemma docs](https://ai.google.dev/gemma/docs)
- [vLLM Gemma 4 tool-call template](https://github.com/vllm-project/vllm/blob/main/examples/tool_chat_template_gemma4.jinja)
- [TPU recipes: Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4)
- [TPU recipes: Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/)
- [Xeon 6 recipe: Gemma 4 26B-A4B IT](https://aiswcatalog.intel.com/models/google-gemma-4-26b-a4b-it)
+425
View File
@@ -0,0 +1,425 @@
meta:
title: "Gemma 4 31B IT"
slug: "gemma-4-31b-it"
provider: "Google"
description: "Google's unified multimodal Gemma 4 dense model (31B) with native text, image, and audio, plus thinking mode and tool-use protocol."
date_added: 2026-04-02
date_updated: 2026-09-09
tasks:
- multimodal
- text
performance_headline: "Unified multimodal model with structured thinking, function calling, dynamic vision resolution"
related_recipes: []
hardware:
h100: verified
mi300x: verified
mi325x: verified
mi355x: verified
trillium: verified
ironwood: verified
arc_pro_b60: verified
arc_pro_b70: verified
model:
model_id: "google/gemma-4-31B-it"
min_vllm_version: "0.19.1"
architecture: dense
parameter_count: "31B"
active_parameters: "31B"
context_length: 262144
base_args: []
base_env: {}
dependencies:
- note: "Audio extras — only needed when serving the audio modality"
command: 'uv pip install "vllm[audio]"'
optional: true
features:
tool_calling:
description: "Enable automatic tool choice with Gemma 4 parser and chat template"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "gemma4"
- "--chat-template"
- "examples/tool_chat_template_gemma4.jinja"
reasoning:
description: "Enable structured thinking/reasoning output"
args:
- "--reasoning-parser"
- "gemma4"
text_only:
description: "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
args:
- "--language-model-only"
spec_decoding:
description: "MTP speculative decoding for accelerated inference"
default_mode: mtp
modes:
mtp:
label: "MTP"
description: "Built-in Multi-Token Prediction, 4 draft tokens."
args:
- "--speculative-config"
- '{"model":"google/gemma-4-31B-it-assistant","num_speculative_tokens":4}'
dflash:
label: "DFlash"
description: "DFlash speculative decoding with an externally trained draft model; the served checkpoint is unchanged."
variants:
- default
args:
- "--speculative-config"
- '{"model": "RedHatAI/gemma-4-31B-it-speculator.dflash", "num_speculative_tokens": 7, "method": "dflash"}'
dspark:
label: "DSpark"
description: "DSpark drafting with an externally trained draft model; the served checkpoint is unchanged."
variants:
- default
args:
- "--speculative-config"
- '{"model": "RedHatAI/gemma-4-31B-it-speculator.dspark", "num_speculative_tokens": 7, "method": "dspark"}'
eagle3:
label: "Eagle3"
description: "Eagle3 speculative decoding with externally trained draft model; the served checkpoint is unchanged."
variants:
- default
args:
- "--speculative-config"
- '{"model": "RedHatAI/gemma-4-31B-it-speculator.eagle3", "num_speculative_tokens": 3, "method": "eagle3"}'
opt_in_features:
- text_only
- spec_decoding
variants:
default:
precision: bf16
vram_minimum_gb: 75
description: "Full BF16 — single 80 GB NVIDIA GPU or 1x MI300X/MI325X/MI350X/MI355X"
fp8:
model_id: "RedHatAI/gemma-4-31B-it-FP8-dynamic"
precision: fp8
vram_minimum_gb: 38
description: "FP8 (E4M3) linear weights with dynamic per-token activation quantization (vision tower stays BF16) — Hopper or Blackwell"
extra_args:
- "--kv-cache-dtype"
- "fp8"
nvfp4:
model_id: "nvidia/gemma-4-31B-it-NVFP4"
precision: nvfp4
vram_minimum_gb: 19
description: "NVIDIA NVFP4 quantized weights for Blackwell GPUs"
extra_args:
- "--kv-cache-dtype"
- "fp8"
redhat_nvfp4:
model_id: "RedHatAI/gemma-4-31B-it-NVFP4"
precision: nvfp4
label: "NVFP4 (RedHat)"
vram_minimum_gb: 19
description: "RedHatAI compressed-tensors NVFP4 quantization for Blackwell GPUs"
extra_args:
- "--kv-cache-dtype"
- "fp8"
w4a16:
model_id: "google/gemma-4-31B-it-qat-w4a16-ct"
precision: int4
vram_minimum_gb: 20
description: "W4A16 QAT (4-bit weights, 16-bit activations, group_size=32) via compressed-tensors — runs on any GPU"
redhat_fp8:
model_id: "RedHatAI/gemma-4-31B-it-FP8-block"
precision: fp8
label: "FP8 Block (RedHat)"
vram_minimum_gb: 38
description: "RedHatAI FP8 block quantization"
extra_args:
- "--kv-cache-dtype"
- "fp8"
compatible_strategies:
- single_node_tp
- multi_node_tp
hardware_overrides:
xpu:
extra_args:
- "--enforce-eager"
- "--attention-backend"
- "flash_attn"
- "--trust-remote-code"
- "--gpu-memory-utilization"
- "0.9"
- "--no-enable-prefix-caching"
- "--max-num-batched-tokens"
- "8192"
- "--max-model-len"
- "5120"
- "--tensor-parallel-size"
- "4"
strategy_overrides:
single_node_tp:
tp: 1
guide: |
## Overview
[Gemma 4](https://ai.google.dev/gemma/docs) is Google's most capable open model family, featuring a unified multimodal architecture that natively processes text, images, and audio. Gemma 4 models support structured thinking/reasoning, function calling with a custom tool-use protocol, and dynamic vision resolution — all available through vLLM's OpenAI-compatible API.
### Key Features
- **Multimodal**: Text + images natively (video via custom frame-extraction pipeline). The smaller E2B and E4B models also support audio.
- **MoE variant**: 128 fine-grained experts with top-8 routing and custom GELU-activated FFN (Gemma 4 26B-A4B).
- **Dual Attention**: Alternating sliding-window (local) and global attention with different head dimensions.
- **Thinking Mode**: Structured reasoning via `<|channel>thought\n...<channel|>` delimiters.
- **Function Calling**: Custom tool-call protocol with dedicated special tokens.
- **Dynamic Vision Resolution**: Per-request configurable vision token budget (70, 140, 280, 560, 1120 tokens).
### Supported Variants
Dense:
- `google/gemma-4-E2B-it` (effective 2B)
- `google/gemma-4-E4B-it` (effective 4B)
- `google/gemma-4-31B-it` (31B)
MoE:
- `google/gemma-4-26B-A4B-it` (26B total / 4B active)
TPU support is provided through [vLLM TPU](https://github.com/vllm-project/tpu-inference) with recipes for [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) and [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/).
## Prerequisites
### pip (NVIDIA CUDA)
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --pre \
--extra-index-url https://wheels.vllm.ai/nightly/cu129 \
--extra-index-url https://download.pytorch.org/whl/cu129 \
--index-strategy unsafe-best-match
```
### pip (AMD ROCm: MI300X, MI325X, MI350X, MI355X)
Requires Python 3.12, ROCm 7.2.1, glibc >= 2.35 (Ubuntu 22.04+).
```bash
uv venv --python 3.12
source .venv/bin/activate
uv pip install vllm --pre \
--extra-index-url https://wheels.vllm.ai/rocm/nightly/rocm721 --upgrade
```
### Docker
```bash
docker pull vllm/vllm-openai:gemma4-0505-cu129 # NVIDIA Hopper (H100/H200, CUDA 12.9)
docker pull vllm/vllm-openai:gemma4-0505-cu130 # NVIDIA Blackwell (B200/B300, CUDA 13.0)
docker pull vllm/vllm-openai-rocm:latest # AMD
docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70)
```
TPU images are published separately by [vllm-project/tpu-inference](https://github.com/vllm-project/tpu-inference); see the Trillium / Ironwood tpu-recipes below for the pinned tag.
## Deployment Configurations
### Quick Start (Single GPU)
```bash
vllm serve google/gemma-4-E4B-it \
--max-model-len <n_of_tokens> # up to 131072
```
### 31B Dense on 2xA100/H100 (TP=2, BF16)
```bash
vllm serve google/gemma-4-31B-it \
--tensor-parallel-size 2 \
--max-model-len 32768 \
--gpu-memory-utilization 0.90
```
### 26B MoE on 1xA100/H100 (BF16)
```bash
vllm serve google/gemma-4-26B-A4B-it \
--max-model-len 32768 \
--gpu-memory-utilization 0.90
```
### Full-Featured Server Launch
Enables text, image, audio, thinking, and tool calling:
```bash
vllm serve google/gemma-4-31B-it \
--tensor-parallel-size 2 \
--max-model-len 16384 \
--gpu-memory-utilization 0.90 \
--enable-auto-tool-choice \
--reasoning-parser gemma4 \
--tool-call-parser gemma4 \
--chat-template examples/tool_chat_template_gemma4.jinja \
--limit-mm-per-prompt '{"image": 4, "audio": 1}' \
--async-scheduling \
--host 0.0.0.0 \
--port 8000
```
### Docker (NVIDIA)
```bash
docker run -itd --name gemma4 \
--ipc=host --network host --shm-size 16G --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:gemma4-0505-cu129 \
--model google/gemma-4-31B-it \
--tensor-parallel-size 2 \
--max-model-len 32768 \
--gpu-memory-utilization 0.90 \
--host 0.0.0.0 --port 8000
```
Swap `vllm/vllm-openai:gemma4-0505-cu129` for `vllm/vllm-openai:gemma4-0505-cu130` on Blackwell (B200/B300).
### Docker (AMD MI300X/MI325X/MI350X/MI355X)
```bash
docker run -itd --name gemma4-rocm \
--ipc=host --network=host --privileged \
--cap-add=CAP_SYS_ADMIN --device=/dev/kfd --device=/dev/dri \
--group-add=video --cap-add=SYS_PTRACE \
--security-opt=seccomp=unconfined --shm-size 16G \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai-rocm:latest \
--model google/gemma-4-31B-it \
--host 0.0.0.0 --port 8000
```
### Docker (Intel XPU B60 / B70, TP=4, Full-Featured)
Validated on 4× Intel Arc Pro B60 / B70 with the official vLLM XPU image.
```bash
docker run --device /dev/dri \
-v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
--privileged --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai-xpu:latest \
google/gemma-4-31B-it \
--enforce-eager \
--attention-backend flash_attn \
--host 0.0.0.0 \
--port 8000 \
--trust-remote-code \
--enable-auto-tool-choice \
--reasoning-parser gemma4 \
--tool-call-parser gemma4 \
--chat-template examples/tool_chat_template_gemma4.jinja \
--limit-mm-per-prompt '{"image": 4, "audio": 1}' \
--async-scheduling \
--gpu-memory-utilization 0.9 \
--no-enable-prefix-caching \
--max-num-batched-tokens 8192 \
--max-model-len 5120 \
--tensor-parallel-size 4
```
### Docker (Cloud TPU — Trillium / Ironwood)
TPU uses the separate `vllm/vllm-tpu` image (no pip wheel). Pull the tag specified by the upstream [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) or [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/) recipe, then run:
```bash
docker run -itd --name gemma4-tpu \
--privileged --network host --shm-size 16G \
-v /dev/shm:/dev/shm -e HF_TOKEN=$HF_TOKEN \
vllm/vllm-tpu:latest \
--model google/gemma-4-31B-it \
--tensor-parallel-size 8 \
--max-model-len 16384 \
--disable_chunked_mm_input \
--host 0.0.0.0 --port 8000
```
## Client Usage
### Text Generation
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="google/gemma-4-31B-it",
messages=[{"role": "user", "content": "Write a poem about the ocean."}],
max_tokens=512, temperature=0.7,
)
print(response.choices[0].message.content)
```
### Image Understanding
```python
response = client.chat.completions.create(
model="google/gemma-4-31B-it",
messages=[{"role": "user", "content": [
{"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg"}},
{"type": "text", "text": "Describe this image in detail."},
]}],
max_tokens=1024,
)
```
### Dynamic Vision Resolution
Supported values: 70, 140, 280 (default), 560, 1120 tokens/image.
```bash
vllm serve google/gemma-4-31B-it \
--mm-processor-kwargs '{"max_soft_tokens": 560}'
```
### Audio (E2B / E4B)
Requires `uv pip install "vllm[audio]"`.
```bash
vllm serve google/gemma-4-E2B-it \
--max-model-len 8192 \
--limit-mm-per-prompt '{"image": 4, "audio": 1}'
```
### Thinking Mode
```bash
vllm serve google/gemma-4-31B-it \
--max-model-len 16384 \
--enable-auto-tool-choice \
--reasoning-parser gemma4 \
--tool-call-parser gemma4 \
--chat-template examples/tool_chat_template_gemma4.jinja
```
Enable thinking per-request via `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`, or default-on with `--default-chat-template-kwargs '{"enable_thinking": true}'`.
### Structured Outputs
vLLM guided decoding constrains output to a JSON schema. Include semantic instructions in the system prompt — the model does not see schema descriptions.
## Configuration Tips
- Set `--max-model-len` to match your workload.
- `--gpu-memory-utilization 0.90–0.95` maximizes KV cache.
- Image-only workloads: pass `--limit-mm-per-prompt.audio 0`.
- Text-only workloads: pass `--limit-mm-per-prompt '{"image": 0, "audio": 0}'` to skip MM profiling.
- `--async-scheduling` improves throughput.
- FP8 KV cache (`--kv-cache-dtype fp8`) saves ~50% KV memory.
## Quantized Variants
Two pre-quantized checkpoints are available:
- [`RedHatAI/gemma-4-31B-it-FP8-dynamic`](https://huggingface.co/RedHatAI/gemma-4-31B-it-FP8-dynamic) — FP8 (E4M3) linear weights with dynamic per-token activation quantization (vision tower stays BF16); runs on Hopper and Blackwell.
- [`nvidia/gemma-4-31B-it-NVFP4`](https://huggingface.co/nvidia/gemma-4-31B-it-NVFP4) — NVFP4 (4-bit) weights; requires Blackwell (B200/B300).
Pick them from the **Variant** dropdown above, or pass the repo id directly to `vllm serve`.
## Throughput vs Latency
| Goal | TP | `--max-num-seqs` | Notes |
|------|----|------------------|-------|
| Max throughput | 1-2 | 256-512 | Best tok/s per GPU |
| Min latency | 4-8 | 8-16 | Best TTFT/TPOT |
| Balanced | 2 | 128 | Mixed workloads |
## Speculative Decoding (MTP)
Enable the **Spec Decoding** feature toggle (above) or add `--speculative-config` manually to use MTP drafting with the [assistant model](https://huggingface.co/google/gemma-4-31B-it-assistant). Recommended `num_speculative_tokens`: 4–8 for this model. See the [Gemma 4 usage guide](../../Google/Gemma4) for details and benchmarks.
> **Note:** MTP speculative decoding for Gemma 4 is only available on the vLLM nightly build — it has not yet landed in a stable release. Install via the nightly wheel (`uv pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly/cu129 …`) or use the `vllm/vllm-openai:gemma4-0505-cu129` / `vllm/vllm-openai:gemma4-0505-cu130` images above; the standard `:latest` stable tag does not include this feature.
## References
- [Model card](https://huggingface.co/google/gemma-4-31B-it)
- [FP8 variant](https://huggingface.co/RedHatAI/gemma-4-31B-it-FP8-dynamic)
- [NVFP4 variant](https://huggingface.co/nvidia/gemma-4-31B-it-NVFP4)
- [Gemma docs](https://ai.google.dev/gemma/docs)
- [vLLM Gemma 4 tool-call template](https://github.com/vllm-project/vllm/blob/main/examples/tool_chat_template_gemma4.jinja)
- [TPU recipes: Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4)
- [TPU recipes: Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/)
+322
View File
@@ -0,0 +1,322 @@
meta:
title: "Gemma 4 E2B IT"
slug: "gemma-4-e2b-it"
provider: "Google"
description: "Google's compact Gemma 4 multimodal model (effective 2B) with native text, image, and audio, plus thinking mode and tool-use protocol."
date_added: 2026-04-02
date_updated: 2026-09-12
tasks:
- multimodal
- text
performance_headline: "Compact unified multimodal model with audio, thinking, and function calling — runs on a single 24 GB+ GPU"
related_recipes:
- google/gemma-4-E4B-it
- google/gemma-4-31B-it
- google/gemma-4-26B-A4B-it
hardware:
h100: verified
mi300x: verified
mi325x: verified
mi355x: verified
trillium: verified
ironwood: verified
xeon6: verified
model:
model_id: "google/gemma-4-E2B-it"
min_vllm_version: "0.19.1"
architecture: dense
parameter_count: "5B"
active_parameters: "5B"
context_length: 131072
base_args: []
base_env: {}
dependencies:
- note: "Audio extras — needed to serve the audio modality"
command: 'uv pip install "vllm[audio]"'
optional: true
features:
tool_calling:
description: "Enable automatic tool choice with Gemma 4 parser and chat template"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "gemma4"
- "--chat-template"
- "examples/tool_chat_template_gemma4.jinja"
reasoning:
description: "Enable structured thinking/reasoning output"
args:
- "--reasoning-parser"
- "gemma4"
text_only:
description: "Skip loading the vision and audio encoders for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
args:
- "--language-model-only"
spec_decoding:
description: "MTP speculative decoding with centroids masking for accelerated inference"
args:
- "--speculative-config"
- '{"model":"google/gemma-4-E2B-it-assistant","num_speculative_tokens":2}'
opt_in_features:
- text_only
- spec_decoding
variants:
default:
precision: bf16
vram_minimum_gb: 13
description: "Full BF16 — single 24 GB+ NVIDIA GPU or 1x MI300X/MI325X/MI350X/MI355X or 1x Xeon6/Xeon5 NUMA node"
supported_hardware: [h100, mi300x, mi325x, mi355x, trillium, ironwood, xeon6]
fp8:
model_id: "RedHatAI/gemma-4-E2B-it-FP8-dynamic"
precision: fp8
vram_minimum_gb: 6
description: "FP8 (E4M3) linear weights with dynamic per-token activation quantization (vision/audio encoders stay BF16) — Hopper or Blackwell"
supported_hardware: [h100]
extra_args:
- "--kv-cache-dtype"
- "fp8"
w4a16:
model_id: "google/gemma-4-E2B-it-qat-w4a16-ct"
precision: int4
vram_minimum_gb: 8
description: "W4A16 QAT (4-bit weights, 16-bit activations, group_size=32) via compressed-tensors — runs on any GPU"
supported_hardware: [h100, mi300x, mi325x, mi355x]
mobile:
model_id: "google/gemma-4-E2B-it-qat-mobile-ct"
precision: int2/4/8
vram_minimum_gb: 4
description: "Mobile QAT — mixed int2/4/8 compressed-tensors (int2 embeddings + lm_head + deep MLP, int4 attention + early MLP, int8 vision + gates). 3.6× weight compression vs BF16, up to 1.25× faster decode at low batch"
supported_hardware: [h100, mi300x, mi325x, mi355x]
compatible_strategies:
- single_node_tp
- multi_node_tp
strategy_overrides:
single_node_tp:
tp: 1
guide: |
## Overview
[Gemma 4 E2B](https://huggingface.co/google/gemma-4-E2B-it) is the smallest member of Google's Gemma 4 family — an effective-2B unified multimodal model that natively processes text, images, and audio, with structured thinking/reasoning, function calling, and dynamic vision resolution. It runs comfortably on a single 24 GB+ GPU.
Intel Xeon 6 CPU support remains verified. No additional benchmark-specific CPU
tuning is prescribed for general deployment.
### Key Features
- **Multimodal**: Text + images + audio natively (video via custom frame-extraction pipeline).
- **Dual Attention**: Alternating sliding-window (local) and global attention with different head dimensions.
- **Thinking Mode**: Structured reasoning via `<|channel>thought\n...<channel|>` delimiters.
- **Function Calling**: Custom tool-call protocol with dedicated special tokens.
- **Dynamic Vision Resolution**: Per-request configurable vision token budget (70, 140, 280, 560, 1120 tokens).
TPU support is provided through [vLLM TPU](https://github.com/vllm-project/tpu-inference) with recipes for [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) and [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/).
## Prerequisites
- Hardware: Intel Xeon 6/Xeon 5 CPUs
- vLLM >= 0.19.1
### pip (NVIDIA CUDA)
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --pre \
--extra-index-url https://wheels.vllm.ai/nightly/cu129 \
--extra-index-url https://download.pytorch.org/whl/cu129 \
--index-strategy unsafe-best-match
```
### pip (AMD ROCm: MI300X, MI325X, MI350X, MI355X)
Requires Python 3.12, ROCm 7.2.1, glibc >= 2.35 (Ubuntu 22.04+).
```bash
uv venv --python 3.12
source .venv/bin/activate
uv pip install vllm --pre \
--extra-index-url https://wheels.vllm.ai/rocm/nightly/rocm721 --upgrade
```
### pip (Intel Xeon 6 CPUs)
For Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions.
### Docker
```bash
docker pull vllm/vllm-openai:gemma4-0505-cu129 # NVIDIA Hopper (H100/H200, CUDA 12.9)
docker pull vllm/vllm-openai:gemma4-0505-cu130 # NVIDIA Blackwell (B200/B300, CUDA 13.0)
docker pull vllm/vllm-openai-rocm:latest # AMD
docker pull vllm/vllm-openai-cpu:latest-x86_64 # For Intel Xeon 6
```
## Deployment Configurations
### Quick Start (Single GPU)
```bash
vllm serve google/gemma-4-E2B-it \
--max-model-len 32768
```
### With Audio Support
```bash
vllm serve google/gemma-4-E2B-it \
--max-model-len 8192 \
--limit-mm-per-prompt '{"image": 4, "audio": 1}'
```
### Full-Featured Server Launch
Enables text, image, audio, thinking, and tool calling:
```bash
vllm serve google/gemma-4-E2B-it \
--max-model-len 16384 \
--gpu-memory-utilization 0.90 \
--enable-auto-tool-choice \
--reasoning-parser gemma4 \
--tool-call-parser gemma4 \
--chat-template examples/tool_chat_template_gemma4.jinja \
--limit-mm-per-prompt '{"image": 4, "audio": 1}' \
--async-scheduling \
--host 0.0.0.0 \
--port 8000
```
### Docker (NVIDIA)
```bash
docker run -itd --name gemma4-e2b \
--ipc=host --network host --shm-size 16G --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:gemma4-0505-cu129 \
--model google/gemma-4-E2B-it \
--max-model-len 32768 \
--host 0.0.0.0 --port 8000
```
Swap `vllm/vllm-openai:gemma4-0505-cu129` for `vllm/vllm-openai:gemma4-0505-cu130` on Blackwell (B200/B300).
### Docker (AMD MI300X/MI325X/MI350X/MI355X)
```bash
docker run -itd --name gemma4-rocm \
--ipc=host --network=host --privileged \
--cap-add=CAP_SYS_ADMIN --device=/dev/kfd --device=/dev/dri \
--group-add=video --cap-add=SYS_PTRACE \
--security-opt=seccomp=unconfined --shm-size 16G \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai-rocm:latest \
--model google/gemma-4-E2B-it \
--host 0.0.0.0 --port 8000
```
### Docker (Cloud TPU — Trillium / Ironwood)
TPU uses the separate `vllm/vllm-tpu` image (no pip wheel). Pull the tag specified by the upstream [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) or [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/) recipe, then run:
```bash
docker run -itd --name gemma4-tpu \
--privileged --network host --shm-size 16G \
-v /dev/shm:/dev/shm -e HF_TOKEN=$HF_TOKEN \
vllm/vllm-tpu:latest \
--model google/gemma-4-E2B-it \
--max-model-len 16384 \
--disable_chunked_mm_input \
--host 0.0.0.0 --port 8000
```
### Intel Xeon 6 Deployment via Docker
Launch the x86 CPU vLLM Docker container for `google/gemma-4-E2B-it`:
```bash
docker run \
--privileged --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai-cpu:latest-x86_64 google/gemma-4-E2B-it \
--tensor-parallel-size 1 \
--enable-auto-tool-choice \
--tool-call-parser gemma4 \
--chat-template examples/tool_chat_template_gemma4.jinja \
--reasoning-parser gemma4
```
The following settings are intentionally not prescribed as portable CPU model
defaults because they depend on the workload, hardware, or runtime environment:
- `--max-num-batched-tokens`: scheduler/throughput tuning.
- `--max-num-seqs`: concurrency and scheduler-capacity tuning.
- `--gpu-memory-utilization`: platform memory-budget tuning.
- `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning.
- `VLLM_ENGINE_ITERATION_TIMEOUT_S`: operational runtime timeout.
These settings may still appear in validated hardware-specific overrides. Tune
them at deployment time based on platform resources, workload shape, and
latency/throughput goals.
## Client Usage
### Audio Transcription
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="google/gemma-4-E2B-it",
messages=[{"role": "user", "content": [
{"type": "audio_url", "audio_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/2/22/Beatbox_by_Wikipedia_user_Wikipedia_Brown.ogg"}},
{"type": "text", "text": "Provide a verbatim, word-for-word transcription of the audio."},
]}],
max_tokens=512,
)
print(response.choices[0].message.content)
```
### Image Understanding
```python
response = client.chat.completions.create(
model="google/gemma-4-E2B-it",
messages=[{"role": "user", "content": [
{"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg"}},
{"type": "text", "text": "Describe this image in detail."},
]}],
max_tokens=1024,
)
```
### Thinking Mode
Launch with reasoning parser, then enable per-request:
```bash
vllm serve google/gemma-4-E2B-it \
--max-model-len 16384 \
--reasoning-parser gemma4 \
--tool-call-parser gemma4 \
--enable-auto-tool-choice \
--chat-template examples/tool_chat_template_gemma4.jinja
```
Enable per-request via `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`.
## Configuration Tips
- Set `--max-model-len` to match your workload (max 131072).
- Image-only workloads: `--limit-mm-per-prompt.audio 0`.
- Text-only workloads: `--limit-mm-per-prompt '{"image": 0, "audio": 0}'` to skip MM profiling.
- `--async-scheduling` improves throughput.
- FP8 KV cache (`--kv-cache-dtype fp8`) saves ~50% KV memory.
## Quantized Variant
[`RedHatAI/gemma-4-E2B-it-FP8-dynamic`](https://huggingface.co/RedHatAI/gemma-4-E2B-it-FP8-dynamic) is a pre-quantized FP8 (E4M3) checkpoint — linear weights with dynamic per-token activation quantization, vision/audio encoders kept in BF16. Runs on Hopper and Blackwell. Pick the **fp8** variant above, or pass the repo id directly to `vllm serve`.
## Speculative Decoding (MTP)
Enable the **Spec Decoding** feature toggle (above) or add `--speculative-config` manually to use MTP drafting with the [assistant model](https://huggingface.co/google/gemma-4-E2B-it-assistant). Recommended `num_speculative_tokens`: 2 for this model. The E2B assistant uses centroids masking for efficient sparse logit computation. See the [Gemma 4 usage guide](../../Google/Gemma4) for details and benchmarks.
> **Note:** MTP speculative decoding for Gemma 4 is only available on the vLLM nightly build — it has not yet landed in a stable release. Install via the nightly wheel (`uv pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly/cu129 …`) or use the `vllm/vllm-openai:gemma4-0505-cu129` / `vllm/vllm-openai:gemma4-0505-cu130` images above; the standard `:latest` stable tag does not include this feature.
## References
- [Model card](https://huggingface.co/google/gemma-4-E2B-it)
- [FP8 variant](https://huggingface.co/RedHatAI/gemma-4-E2B-it-FP8-dynamic)
- [Gemma docs](https://ai.google.dev/gemma/docs)
- [vLLM Gemma 4 tool-call template](https://github.com/vllm-project/vllm/blob/main/examples/tool_chat_template_gemma4.jinja)
- [TPU recipes: Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4)
- [TPU recipes: Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/)
+322
View File
@@ -0,0 +1,322 @@
meta:
title: "Gemma 4 E4B IT"
slug: "gemma-4-e4b-it"
provider: "Google"
description: "Google's compact Gemma 4 multimodal model (effective 4B) with native text, image, and audio, plus thinking mode and tool-use protocol."
date_added: 2026-04-02
date_updated: 2026-09-12
tasks:
- multimodal
- text
performance_headline: "Effective-4B unified multimodal model with audio, thinking, and function calling — runs on a single 24 GB+ GPU"
related_recipes:
- google/gemma-4-E2B-it
- google/gemma-4-31B-it
- google/gemma-4-26B-A4B-it
hardware:
h100: verified
mi300x: verified
mi325x: verified
mi355x: verified
trillium: verified
ironwood: verified
xeon6: verified
model:
model_id: "google/gemma-4-E4B-it"
min_vllm_version: "0.19.1"
architecture: dense
parameter_count: "8B"
active_parameters: "8B"
context_length: 131072
base_args: []
base_env: {}
dependencies:
- note: "Audio extras — needed to serve the audio modality"
command: 'uv pip install "vllm[audio]"'
optional: true
features:
tool_calling:
description: "Enable automatic tool choice with Gemma 4 parser and chat template"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "gemma4"
- "--chat-template"
- "examples/tool_chat_template_gemma4.jinja"
reasoning:
description: "Enable structured thinking/reasoning output"
args:
- "--reasoning-parser"
- "gemma4"
text_only:
description: "Skip loading the vision and audio encoders for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
args:
- "--language-model-only"
spec_decoding:
description: "MTP speculative decoding with centroids masking for accelerated inference"
args:
- "--speculative-config"
- '{"model":"google/gemma-4-E4B-it-assistant","num_speculative_tokens":4}'
opt_in_features:
- text_only
- spec_decoding
variants:
default:
precision: bf16
vram_minimum_gb: 20
description: "Full BF16 — single 24 GB+ NVIDIA GPU or 1x MI300X/MI325X/MI350X/MI355Xi or 2x Xeon6/Xeon5 NUMA nodes"
supported_hardware: [h100, mi300x, mi325x, mi355x, trillium, ironwood, xeon6]
fp8:
model_id: "RedHatAI/gemma-4-E4B-it-FP8-dynamic"
precision: fp8
vram_minimum_gb: 10
description: "FP8 (E4M3) linear weights with dynamic per-token activation quantization (vision/audio encoders stay BF16) — Hopper or Blackwell"
supported_hardware: [h100]
extra_args:
- "--kv-cache-dtype"
- "fp8"
w4a16:
model_id: "google/gemma-4-E4B-it-qat-w4a16-ct"
precision: int4
vram_minimum_gb: 10
description: "W4A16 QAT (4-bit weights, 16-bit activations, group_size=32) via compressed-tensors — runs on any GPU"
supported_hardware: [h100, mi300x, mi325x, mi355x]
mobile:
model_id: "google/gemma-4-E4B-it-qat-mobile-ct"
precision: int2/4/8
vram_minimum_gb: 5
description: "Mobile QAT — mixed int2/4/8 compressed-tensors (int2 embeddings + lm_head + audio, int4 LLM attention + MLP, int8 vision + gates). 4.1× weight compression vs BF16, up to 1.5× faster decode at low batch"
supported_hardware: [h100, mi300x, mi325x, mi355x]
compatible_strategies:
- single_node_tp
- multi_node_tp
strategy_overrides:
single_node_tp:
tp: 1
guide: |
## Overview
[Gemma 4 E4B](https://huggingface.co/google/gemma-4-E4B-it) is Google's effective-4B unified multimodal model — text + images + audio in a single model, with structured thinking/reasoning, function calling, and dynamic vision resolution. It fits on a single 24 GB+ GPU.
Intel Xeon 6 CPU support remains verified. No additional benchmark-specific CPU
tuning is prescribed for general deployment.
### Key Features
- **Multimodal**: Text + images + audio natively (video via custom frame-extraction pipeline).
- **Dual Attention**: Alternating sliding-window (local) and global attention with different head dimensions.
- **Thinking Mode**: Structured reasoning via `<|channel>thought\n...<channel|>` delimiters.
- **Function Calling**: Custom tool-call protocol with dedicated special tokens.
- **Dynamic Vision Resolution**: Per-request configurable vision token budget (70, 140, 280, 560, 1120 tokens).
TPU support is provided through [vLLM TPU](https://github.com/vllm-project/tpu-inference) with recipes for [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) and [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/).
## Prerequisites
- Hardware: Intel Xeon 6/Xeon 5 CPUs
- vLLM >= 0.19.1
### pip (NVIDIA CUDA)
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --pre \
--extra-index-url https://wheels.vllm.ai/nightly/cu129 \
--extra-index-url https://download.pytorch.org/whl/cu129 \
--index-strategy unsafe-best-match
```
### pip (AMD ROCm: MI300X, MI325X, MI350X, MI355X)
Requires Python 3.12, ROCm 7.2.1, glibc >= 2.35 (Ubuntu 22.04+).
```bash
uv venv --python 3.12
source .venv/bin/activate
uv pip install vllm --pre \
--extra-index-url https://wheels.vllm.ai/rocm/nightly/rocm721 --upgrade
```
### pip (Intel Xeon 6 CPUs)
For Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions.
### Docker
```bash
docker pull vllm/vllm-openai:gemma4-0505-cu129 # NVIDIA Hopper (H100/H200, CUDA 12.9)
docker pull vllm/vllm-openai:gemma4-0505-cu130 # NVIDIA Blackwell (B200/B300, CUDA 13.0)
docker pull vllm/vllm-openai-rocm:latest # AMD
docker pull vllm/vllm-openai-cpu:latest-x86_64 # For Intel Xeon 6
```
## Deployment Configurations
### Quick Start (Single GPU)
```bash
vllm serve google/gemma-4-E4B-it \
--max-model-len 32768
```
### With Audio Support
```bash
vllm serve google/gemma-4-E4B-it \
--max-model-len 8192 \
--limit-mm-per-prompt '{"image": 4, "audio": 1}'
```
### Full-Featured Server Launch
Enables text, image, audio, thinking, and tool calling:
```bash
vllm serve google/gemma-4-E4B-it \
--max-model-len 16384 \
--gpu-memory-utilization 0.90 \
--enable-auto-tool-choice \
--reasoning-parser gemma4 \
--tool-call-parser gemma4 \
--chat-template examples/tool_chat_template_gemma4.jinja \
--limit-mm-per-prompt '{"image": 4, "audio": 1}' \
--async-scheduling \
--host 0.0.0.0 \
--port 8000
```
### Docker (NVIDIA)
```bash
docker run -itd --name gemma4-e4b \
--ipc=host --network host --shm-size 16G --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:gemma4-0505-cu129 \
--model google/gemma-4-E4B-it \
--max-model-len 32768 \
--host 0.0.0.0 --port 8000
```
Swap `vllm/vllm-openai:gemma4-0505-cu129` for `vllm/vllm-openai:gemma4-0505-cu130` on Blackwell (B200/B300).
### Docker (AMD MI300X/MI325X/MI350X/MI355X)
```bash
docker run -itd --name gemma4-rocm \
--ipc=host --network=host --privileged \
--cap-add=CAP_SYS_ADMIN --device=/dev/kfd --device=/dev/dri \
--group-add=video --cap-add=SYS_PTRACE \
--security-opt=seccomp=unconfined --shm-size 16G \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai-rocm:latest \
--model google/gemma-4-E4B-it \
--host 0.0.0.0 --port 8000
```
### Docker (Cloud TPU — Trillium / Ironwood)
TPU uses the separate `vllm/vllm-tpu` image (no pip wheel). Pull the tag specified by the upstream [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) or [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/) recipe, then run:
```bash
docker run -itd --name gemma4-tpu \
--privileged --network host --shm-size 16G \
-v /dev/shm:/dev/shm -e HF_TOKEN=$HF_TOKEN \
vllm/vllm-tpu:latest \
--model google/gemma-4-E4B-it \
--max-model-len 16384 \
--disable_chunked_mm_input \
--host 0.0.0.0 --port 8000
```
### Intel Xeon 6 Deployment via Docker
Launch the x86 CPU vLLM Docker container for `google/gemma-4-E4B-it`:
```bash
docker run \
--privileged --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai-cpu:latest-x86_64 google/gemma-4-E4B-it \
--tensor-parallel-size 1 \
--enable-auto-tool-choice \
--tool-call-parser gemma4 \
--chat-template examples/tool_chat_template_gemma4.jinja \
--reasoning-parser gemma4
```
For additional Intel Xeon 6 deployment details, see the Intel Software Catalog entries for [Gemma 4 E4B IT](https://aiswcatalog.intel.com/models/google-gemma-4-e4b-it).
The following settings are intentionally not prescribed as portable CPU model
defaults because they depend on the workload, hardware, or runtime environment:
- `--max-num-batched-tokens`: scheduler/throughput tuning.
- `--max-num-seqs`: concurrency and scheduler-capacity tuning.
- `--gpu-memory-utilization`: platform memory-budget tuning.
- `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning.
- `VLLM_ENGINE_ITERATION_TIMEOUT_S`: operational runtime timeout.
These settings may still appear in validated hardware-specific overrides. Tune
them at deployment time based on platform resources, workload shape, and
latency/throughput goals.
## Client Usage
### Audio Transcription
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="google/gemma-4-E4B-it",
messages=[{"role": "user", "content": [
{"type": "audio_url", "audio_url": {"url": "https://example.com/audio.wav"}},
{"type": "text", "text": "Transcribe this audio."},
]}],
max_tokens=512,
)
print(response.choices[0].message.content)
```
### Image Understanding
```python
response = client.chat.completions.create(
model="google/gemma-4-E4B-it",
messages=[{"role": "user", "content": [
{"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg"}},
{"type": "text", "text": "Describe this image in detail."},
]}],
max_tokens=1024,
)
```
### Thinking Mode
```bash
vllm serve google/gemma-4-E4B-it \
--max-model-len 16384 \
--reasoning-parser gemma4 \
--tool-call-parser gemma4 \
--enable-auto-tool-choice \
--chat-template examples/tool_chat_template_gemma4.jinja
```
Enable per-request via `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`.
## Configuration Tips
- Set `--max-model-len` to match your workload (max 131072).
- Image-only workloads: `--limit-mm-per-prompt.audio 0`.
- Text-only workloads: `--limit-mm-per-prompt '{"image": 0, "audio": 0}'` to skip MM profiling.
- `--async-scheduling` improves throughput.
- FP8 KV cache (`--kv-cache-dtype fp8`) saves ~50% KV memory.
## Quantized Variant
[`RedHatAI/gemma-4-E4B-it-FP8-dynamic`](https://huggingface.co/RedHatAI/gemma-4-E4B-it-FP8-dynamic) is a pre-quantized FP8 (E4M3) checkpoint — linear weights with dynamic per-token activation quantization, vision/audio encoders kept in BF16. Runs on Hopper and Blackwell. Pick the **fp8** variant above, or pass the repo id directly to `vllm serve`.
## Speculative Decoding (MTP)
Enable the **Spec Decoding** feature toggle (above) or add `--speculative-config` manually to use MTP drafting with the [assistant model](https://huggingface.co/google/gemma-4-E4B-it-assistant). Recommended `num_speculative_tokens`: 4 for this model. The E4B assistant uses centroids masking for efficient sparse logit computation. See the [Gemma 4 usage guide](../../Google/Gemma4) for details and benchmarks.
> **Note:** MTP speculative decoding for Gemma 4 is only available on the vLLM nightly build — it has not yet landed in a stable release. Install via the nightly wheel (`uv pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly/cu129 …`) or use the `vllm/vllm-openai:gemma4-0505-cu129` / `vllm/vllm-openai:gemma4-0505-cu130` images above; the standard `:latest` stable tag does not include this feature.
## References
- [Model card](https://huggingface.co/google/gemma-4-E4B-it)
- [FP8 variant](https://huggingface.co/RedHatAI/gemma-4-E4B-it-FP8-dynamic)
- [Gemma docs](https://ai.google.dev/gemma/docs)
- [vLLM Gemma 4 tool-call template](https://github.com/vllm-project/vllm/blob/main/examples/tool_chat_template_gemma4.jinja)
- [TPU recipes: Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4)
- [TPU recipes: Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/)
- [Xeon 6 recipe: Gemma 4 E4B IT](https://aiswcatalog.intel.com/models/google-gemma-4-e4b-it)
@@ -0,0 +1,122 @@
meta:
title: "TranslateGemma 27B IT"
slug: "translategemma-27b-it"
provider: "Google"
description: "Lightweight open translation model from Google (based on Gemma 3) supporting 55 languages. Served via the vLLM-optimized Infomaniak-AI checkpoint."
date_added: 2026-03-06
date_updated: 2026-04-17
tasks:
- text
performance_headline: "Deployable on laptops/desktops and cloud GPUs; vLLM-optimized checkpoint removes custom JSON inputs"
related_recipes: []
model:
model_id: "google/translategemma-27b-it"
min_vllm_version: "0.14.1"
architecture: dense
parameter_count: "27B"
active_parameters: "27B"
context_length: 131072
base_args: []
base_env: {}
features: {}
opt_in_features: []
variants:
default:
precision: bf16
vram_minimum_gb: 65
description: "Original Google checkpoint in BF16 (has vLLM compatibility issues — prefer vllm-optimized variant)"
vllm_optimized:
model_id: "Infomaniak-AI/vllm-translategemma-27b-it"
precision: bf16
vram_minimum_gb: 65
description: "Infomaniak-AI vLLM-optimized checkpoint — recommended. Fixes RoPE config, EOS token, and replaces custom JSON inputs with string delimiters."
small_4b:
model_id: "google/translategemma-4b-it"
precision: bf16
vram_minimum_gb: 10
description: "4B variant for lower-resource deployments (prefer Infomaniak-AI/vllm-translategemma-4b-it)"
compatible_strategies:
- single_node_tp
- multi_node_tp
hardware_overrides: {}
strategy_overrides: {}
guide: |
## Overview
[TranslateGemma](https://huggingface.co/collections/google/translategemma) is a family of lightweight, state-of-the-art open translation models from Google, based on the Gemma 3 family. TranslateGemma models handle translation across 55 languages and are small enough to deploy on laptops, desktops, and modest cloud GPU environments.
### Original Models
- [google/translategemma-27b-it](https://huggingface.co/google/translategemma-27b-it)
- [google/translategemma-4b-it](https://huggingface.co/google/translategemma-4b-it)
### Optimized vLLM Models
- [Infomaniak-AI/vllm-translategemma-27b-it](https://huggingface.co/Infomaniak-AI/vllm-translategemma-27b-it)
- [Infomaniak-AI/vllm-translategemma-4b-it](https://huggingface.co/Infomaniak-AI/vllm-translategemma-4b-it)
### Why use the vLLM-optimized models?
The original Google models have compatibility issues with standard inference engines like vLLM. The optimized versions from Infomaniak-AI ([detailed changes](https://huggingface.co/Infomaniak-AI/vllm-translategemma-27b-it#changes-from-original-model)):
- **vLLM Compatibility**: Originals require custom JSON parameters (`source_lang_code`/`target_lang_code`). The optimized version uses string delimiters.
- **RoPE Simplification**: Originals use a complex RoPE configuration for sliding attention. Optimized uses a standard linear RoPE format (`factor: 8.0`).
- **EOS Token Fix**: Corrects the EOS token from `<end_of_turn>` to `<eos>`.
## Prerequisites
### Docker
```bash
docker pull vllm/vllm-openai:latest
```
## Deployment Configurations
Verified for both 4B and 27B:
```bash
docker run -itd --name google-translategemma-27b-it \
--ipc=host \
--network host \
--shm-size 16G \
--gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:latest \
Infomaniak-AI/vllm-translategemma-27b-it \
--served-model-name translategemma-27b-it \
--gpu-memory-utilization 0.8 \
--host 0.0.0.0 \
--port 8000
```
## Client Usage
Tips:
- **Prompt Delimiters**: Encode language metadata directly in the content string:
`<<<source>>>{src_lang}<<<target>>>{tgt_lang}<<<text>>>{text}`
- **Language Codes**: ISO 639-1 Alpha-2 (e.g. `en`, `zh`) and regional variants (e.g. `en_US`, `zh_CN`).
- **Context Limit**: ~2K tokens.
### cURL
```bash
curl -X POST http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "translategemma-27b-it",
"messages": [{
"role": "user",
"content": "<<<source>>>en<<<target>>>zh<<<text>>>We distribute two models for language identification, which can recognize 176 languages."
}]
}'
```
## References
- [TranslateGemma collection](https://huggingface.co/collections/google/translategemma)
- [Infomaniak-AI optimized 27B](https://huggingface.co/Infomaniak-AI/vllm-translategemma-27b-it)
- [Infomaniak-AI optimized 4B](https://huggingface.co/Infomaniak-AI/vllm-translategemma-4b-it)
+109
View File
@@ -0,0 +1,109 @@
meta:
title: "Jais-3-0.9B"
slug: "jais-3-0.9b"
provider: "IFM"
description: "Compact dense model for mathematics, code, instruction following, and STEM tasks"
date_added: 2026-10-05
date_updated: 2026-10-05
tasks:
- text
performance_headline: "1.08B stored parameters including embeddings, with 131K context and native K2-Horizon parsing"
related_recipes:
- "IFM/Jais-3-3.7B"
hardware:
h200: verified
model:
model_id: "IFM/Jais-3-0.9B"
architecture: dense
parameter_count: "1.08B"
active_parameters: "1.08B"
context_length: 131072
base_args:
- "--trust-remote-code"
- "--chat-template-content-format"
- "string"
- "--dtype"
- "bfloat16"
- "--max-model-len"
- "131072"
- "--gpu-memory-utilization"
- "0.85"
base_env: {}
features:
reasoning:
description: "Extract Jais-3 reasoning into message.reasoning"
args: ["--reasoning-parser", "k2_horizon"]
tool_calling:
description: "Enable automatic tool choice with the K2-Horizon parser"
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
variants:
default:
precision: bf16
vram_minimum_gb: 3
description: "Released BF16 checkpoint; the command starts at an 8K runtime context"
compatible_strategies:
- single_node_tp
- multi_node_tp
- multi_node_tp_pp
hardware_overrides: {}
guide: |
## Overview
Jais-3-0.9B is a compact dense model in the IFM Jais-3
family, with 1.08B stored parameters including embeddings and a 131,072-token
context window.
Jais-3 is an open-weight IFM model family built for transparent
foundation-model research, staged checkpoint analysis, and practical
deployment. The series spans compact dense models such as Jais-3-0.9B
for local experimentation, dense mid-size and large models for high-quality
research workloads, and mixture-of-expert (MoE) models for higher-capacity
serving.
## Launch Commands
Jais-3-0.9B's `config.json` already enables YaRN, so the full 131,072-token context needs no RoPE override.
```bash
vllm serve IFM/Jais-3-0.9B \
--trust-remote-code \
--chat-template-content-format string \
--dtype bfloat16 \
--max-model-len 131072 \
--gpu-memory-utilization 0.85 \
--reasoning-parser k2_horizon \
--enable-auto-tool-choice \
--tool-call-parser k2_horizon
```
Lower `--max-model-len` (e.g. 8192) to reduce KV-cache memory.
## Client usage
```python
from openai import OpenAI
client = OpenAI(
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
)
resp = client.chat.completions.create(
model="IFM/Jais-3-0.9B",
messages=[{"role": "user", "content": "لدينا عددان مجموعهما 50 وحاصل ضربهما 600. ما هما العددان؟ اشرح خطوات الحل."}],
temperature=1.0, top_p=0.95, max_tokens=2048,
)
print(resp.choices[0].message.content)
```
### Thinking modes
The model supports selectable thinking through `chat_template_kwargs`, per
request or server-wide via `--default-chat-template-kwargs`:
- `{"reasoning_effort": "high"}` — full thinking (default).
- `{"reasoning_effort": "low"}` — no thinking.
## References
- [IFM/Jais-3-0.9B on Hugging Face](https://huggingface.co/IFM/Jais-3-0.9B)
- [Jais 3 Collection](https://huggingface.co/collections/IFM/jais-3)
+88
View File
@@ -0,0 +1,88 @@
meta:
title: "Jais-3-3.7B"
slug: "jais-3-3.7b"
provider: "IFM"
description: "Dense model for long-context research, serving, and downstream adaptation"
date_added: 2026-10-05
date_updated: 2026-10-05
tasks: [text]
performance_headline: "5.06B stored parameters including embeddings, with up to 512K context"
related_recipes: ["IFM/Jais-3-0.9B", "IFM/Jais-3-7B"]
hardware:
h200: verified
model:
model_id: "IFM/Jais-3-3.7B"
architecture: dense
parameter_count: "5.06B"
active_parameters: "5.06B"
context_length: 524288
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16"]
base_env: {}
features:
reasoning:
description: "Extract Jais-3 reasoning into message.reasoning"
args: ["--reasoning-parser", "k2_horizon"]
tool_calling:
description: "Enable automatic tool choice with the K2-Horizon parser"
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
variants:
default:
precision: bf16
vram_minimum_gb: 13
description: "BF16 serving runtime with a 131K context baseline"
compatible_strategies: [single_node_tp, multi_node_tp, multi_node_tp_pp]
hardware_overrides: {}
guide: |
## Overview
Jais-3-3.7B is a small dense model in the IFM Jais-3 family,
with 5.06B stored parameters including embeddings and support for up to
524,288 tokens of context.
Jais-3 is an open-weight IFM model family built for transparent
foundation-model research, staged checkpoint analysis, and practical
deployment. The series spans compact dense models for local experimentation,
dense mid-size and large models for high-quality research workloads, and
mixture-of-expert (MoE) models for higher-capacity serving.
## Launch Commands
```bash
vllm serve IFM/Jais-3-3.7B \
--trust-remote-code \
--chat-template-content-format string \
--dtype bfloat16 \
--reasoning-parser k2_horizon \
--enable-auto-tool-choice \
--tool-call-parser k2_horizon
```
## Client usage
```python
from openai import OpenAI
client = OpenAI(
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
)
resp = client.chat.completions.create(
model="IFM/Jais-3-3.7B",
messages=[{"role": "user", "content": "لدينا عددان مجموعهما 50 وحاصل ضربهما 600. ما هما العددان؟ اشرح خطوات الحل."}],
temperature=1.0, top_p=0.95, max_tokens=2048,
)
print(resp.choices[0].message.content)
```
### Thinking modes
The model supports selectable thinking through `chat_template_kwargs`, per
request or server-wide via `--default-chat-template-kwargs`:
- `{"reasoning_effort": "high"}` — full thinking (default).
- `{"reasoning_effort": "low"}` — no thinking.
## References
- [IFM/Jais-3-3.7B on Hugging Face](https://huggingface.co/IFM/Jais-3-3.7B)
- [Jais 3 Collection](https://huggingface.co/collections/IFM/jais-3)
+93
View File
@@ -0,0 +1,93 @@
meta:
title: "Jais-3-32B"
slug: "jais-3-32b"
provider: "IFM"
description: "Dense long-context model for reasoning research and high-capacity serving"
date_added: 2026-10-05
date_updated: 2026-10-05
tasks: [text]
performance_headline: "34.78B stored parameters including embeddings, with native vLLM execution and up to 512K context"
related_recipes: ["IFM/Jais-3-7B", "IFM/Jais-3-MoVA-36B-A4B"]
hardware:
h200: verified
model:
model_id: "IFM/Jais-3-32B"
architecture: dense
parameter_count: "34.78B"
active_parameters: "34.78B"
context_length: 524288
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16"]
base_env: {}
features:
reasoning:
description: "Extract Jais-3 reasoning into message.reasoning"
args: ["--reasoning-parser", "k2_horizon"]
tool_calling:
description: "Enable automatic tool choice with the K2-Horizon parser"
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
variants:
default:
precision: bf16
vram_minimum_gb: 84
description: "BF16 serving runtime with native vLLM execution"
compatible_strategies: [single_node_tp, multi_node_tp, multi_node_tp_pp]
hardware_overrides: {}
strategy_overrides:
single_node_tp:
tp: 2
guide: |
## Overview
Jais-3-32B is a large dense model in the IFM Jais-3 family,
with 34.78B stored parameters including embeddings and support for up to
524,288 tokens of context.
Jais-3 is an open-weight IFM model family built for transparent
foundation-model research, staged checkpoint analysis, and practical
deployment. The series spans compact dense models for local experimentation,
dense mid-size and large models such as Jais-3-32B for high-quality
research workloads, and mixture-of-expert (MoE) models for higher-capacity
serving.
## Launch Commands
```bash
vllm serve IFM/Jais-3-32B \
--tensor-parallel-size 2 \
--trust-remote-code \
--chat-template-content-format string \
--dtype bfloat16 \
--reasoning-parser k2_horizon \
--enable-auto-tool-choice \
--tool-call-parser k2_horizon
```
## Client usage
```python
from openai import OpenAI
client = OpenAI(
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
)
resp = client.chat.completions.create(
model="IFM/Jais-3-32B",
messages=[{"role": "user", "content": "لدينا عددان مجموعهما 50 وحاصل ضربهما 600. ما هما العددان؟ اشرح خطوات الحل."}],
temperature=1.0, top_p=0.95, max_tokens=2048,
)
print(resp.choices[0].message.content)
```
### Thinking modes
The model supports selectable thinking through `chat_template_kwargs`, per
request or server-wide via `--default-chat-template-kwargs`:
- `{"reasoning_effort": "high"}` — full thinking (default).
- `{"reasoning_effort": "low"}` — no thinking.
## References
- [IFM/Jais-3-32B on Hugging Face](https://huggingface.co/IFM/Jais-3-32B)
- [Jais 3 Collection](https://huggingface.co/collections/IFM/jais-3)
+148
View File
@@ -0,0 +1,148 @@
meta:
title: "Jais-3-375B-A23B"
slug: "jais-3-375b-a23b"
provider: "IFM"
description: "Frontier-scale sparse MoE model for long-context research and high-capacity serving"
date_added: 2026-10-05
date_updated: 2026-10-05
tasks: [text]
performance_headline: "379.17B stored parameters including embeddings / 26.67B active parameters per token"
related_recipes: ["IFM/Jais-3-MoVA-36B-A4B"]
hardware:
h200: verified
mi355x: verified
model:
model_id: "IFM/Jais-3-375B-A23B"
architecture: moe
parameter_count: "379.17B"
active_parameters: "26.67B"
context_length: 524288
# --enable-expert-parallel is not here: in base_args it would reach the AMD
# path too, which deliberately runs pure TP (see strategy_hardware). The TEP
# strategies emit it themselves; multi_node_tp_pp does not, so it carries its
# own copy in strategy_overrides below.
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16"]
base_env: {}
features:
reasoning:
description: "Extract Jais-3 reasoning into message.reasoning"
args: ["--reasoning-parser", "k2_horizon"]
tool_calling:
description: "Enable automatic tool choice with the K2-Horizon parser"
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
variants:
default:
precision: bf16
vram_minimum_gb: 911
description: "BF16 serving runtime with an eight-GPU tensor-parallel baseline"
compatible_strategies: [single_node_tep, single_node_tp, multi_node_tep, multi_node_tp_pp]
default_strategy: single_node_tep
# On AMD the expert-parallel dispatch is slower than replicating the experts
# across a pure tensor-parallel shard, so the TEP strategies are opted out and
# single-node TP is the offered path. multi_node_tp_pp is left available: EP
# works there, it is just not the fast default at single-node scale.
strategy_hardware:
single_node_tep:
amd: unsupported
multi_node_tep:
amd: unsupported
hardware_overrides:
amd:
extra_args:
- "--attention-backend"
- "ROCM_AITER_UNIFIED_ATTN"
- "--gpu-memory-utilization"
- "0.9"
extra_env:
VLLM_ROCM_USE_AITER: "1"
NCCL_MIN_NCHANNELS: "112"
strategy_overrides:
single_node_tep:
tp: 8
# multi_node_tp_pp emits no EP flag of its own, and this recipe has always
# run it with expert parallel on. Restored here rather than in base_args so
# it stays off the AMD single-node TP path.
multi_node_tp_pp:
extra_args: ["--enable-expert-parallel"]
guide: |
## Overview
Jais-3-375B-A23B is a frontier-scale mixture-of-expert (MoE) model in the IFM
Jais-3 family, with 379.17B stored parameters including embeddings,
26.67B active parameters per token, and support for up to 524,288 tokens of
context.
Jais-3 is an open-weight IFM model family built for transparent
foundation-model research, staged checkpoint analysis, and practical
deployment. The series spans compact dense models for local experimentation,
dense mid-size and large models for high-quality research workloads, and
mixture-of-expert (MoE) models such as Jais-3-375B-A23B for frontier-scale
research and higher-capacity serving.
## Launch Commands
```bash
vllm serve IFM/Jais-3-375B-A23B \
--tensor-parallel-size 8 \
--trust-remote-code \
--chat-template-content-format string \
--dtype bfloat16 \
--enable-expert-parallel \
--reasoning-parser k2_horizon \
--enable-auto-tool-choice \
--tool-call-parser k2_horizon
```
### AMD MI355X
Verified on an 8-GPU MI355X (gfx950) node. AITER supplies the fused MoE and
the unified attention backend:
```bash
export VLLM_ROCM_USE_AITER=1
export NCCL_MIN_NCHANNELS=112
vllm serve IFM/Jais-3-375B-A23B \
--tensor-parallel-size 8 \
--trust-remote-code \
--chat-template-content-format string \
--dtype bfloat16 \
--max-model-len 131072 \
--gpu-memory-utilization 0.9 \
--attention-backend ROCM_AITER_UNIFIED_ATTN \
--generation-config auto \
--reasoning-parser k2_horizon \
--enable-auto-tool-choice \
--tool-call-parser k2_horizon
```
## Client usage
```python
from openai import OpenAI
client = OpenAI(
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
)
resp = client.chat.completions.create(
model="IFM/Jais-3-375B-A23B",
messages=[{"role": "user", "content": "لدينا عددان مجموعهما 50 وحاصل ضربهما 600. ما هما العددان؟ اشرح خطوات الحل."}],
temperature=1.0, top_p=0.95, max_tokens=2048,
)
print(resp.choices[0].message.content)
```
### Thinking modes
The model supports selectable thinking through `chat_template_kwargs`, per
request or server-wide via `--default-chat-template-kwargs`:
- `{"reasoning_effort": "high"}` — full thinking (default).
- `{"reasoning_effort": "low"}` — no thinking.
## References
- [IFM/Jais-3-375B-A23B on Hugging Face](https://huggingface.co/IFM/Jais-3-375B-A23B)
- [Jais 3 Collection](https://huggingface.co/collections/IFM/jais-3)
+89
View File
@@ -0,0 +1,89 @@
meta:
title: "Jais-3-7B"
slug: "jais-3-7b"
provider: "IFM"
description: "Dense model for long-context research, fine-tuning, and cost-conscious deployment"
date_added: 2026-10-05
date_updated: 2026-10-05
tasks: [text]
performance_headline: "9B stored parameters including embeddings, with up to 512K context"
related_recipes: ["IFM/Jais-3-3.7B", "IFM/Jais-3-32B"]
hardware:
h200: verified
model:
model_id: "IFM/Jais-3-7B"
architecture: dense
parameter_count: "9B"
active_parameters: "9B"
context_length: 524288
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16"]
base_env: {}
features:
reasoning:
description: "Extract Jais-3 reasoning into message.reasoning"
args: ["--reasoning-parser", "k2_horizon"]
tool_calling:
description: "Enable automatic tool choice with the K2-Horizon parser"
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
variants:
default:
precision: bf16
vram_minimum_gb: 22
description: "BF16 serving runtime with a 131K context baseline"
compatible_strategies: [single_node_tp, multi_node_tp, multi_node_tp_pp]
hardware_overrides: {}
guide: |
## Overview
Jais-3-7B is a compact dense model in the IFM Jais-3 family,
with 9B stored parameters including embeddings and support for up to 524,288
tokens of context.
Jais-3 is an open-weight IFM model family built for transparent
foundation-model research, staged checkpoint analysis, and practical
deployment. The series spans compact dense models such as Jais-3-7B for
local experimentation, dense mid-size and large models for high-quality
research workloads, and mixture-of-expert (MoE) models for higher-capacity
serving.
## Launch Commands
```bash
vllm serve IFM/Jais-3-7B \
--trust-remote-code \
--chat-template-content-format string \
--dtype bfloat16 \
--reasoning-parser k2_horizon \
--enable-auto-tool-choice \
--tool-call-parser k2_horizon
```
## Client usage
```python
from openai import OpenAI
client = OpenAI(
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
)
resp = client.chat.completions.create(
model="IFM/Jais-3-7B",
messages=[{"role": "user", "content": "لدينا عددان مجموعهما 50 وحاصل ضربهما 600. ما هما العددان؟ اشرح خطوات الحل."}],
temperature=1.0, top_p=0.95, max_tokens=2048,
)
print(resp.choices[0].message.content)
```
### Thinking modes
The model supports selectable thinking through `chat_template_kwargs`, per
request or server-wide via `--default-chat-template-kwargs`:
- `{"reasoning_effort": "high"}` — full thinking (default).
- `{"reasoning_effort": "low"}` — no thinking.
## References
- [IFM/Jais-3-7B on Hugging Face](https://huggingface.co/IFM/Jais-3-7B)
- [Jais 3 Collection](https://huggingface.co/collections/IFM/jais-3)
@@ -0,0 +1,96 @@
meta:
title: "Jais-3-MoVA-36B-A4B"
slug: "jais-3-mova-36b-a4b"
provider: "IFM"
description: "Sparse MoVA and MoE long-context model for research and production-style serving experiments"
date_added: 2026-10-05
date_updated: 2026-10-05
tasks: [text]
performance_headline: "37.44B stored parameters including embeddings / 5.95B active parameters per token"
related_recipes: ["IFM/Jais-3-32B", "IFM/Jais-3-375B-A23B"]
hardware:
h200: verified
model:
model_id: "IFM/Jais-3-MoVA-36B-A4B"
architecture: moe
parameter_count: "37.44B"
active_parameters: "5.95B"
context_length: 524288
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16", "--enable-expert-parallel"]
base_env: {}
features:
reasoning:
description: "Extract Jais-3 reasoning into message.reasoning"
args: ["--reasoning-parser", "k2_horizon"]
tool_calling:
description: "Enable automatic tool choice with the K2-Horizon parser"
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
variants:
default:
precision: bf16
vram_minimum_gb: 90
description: "BF16 serving runtime with native vLLM execution"
compatible_strategies: [single_node_tep, multi_node_tep, multi_node_tp_pp]
default_strategy: single_node_tep
hardware_overrides: {}
strategy_overrides:
single_node_tep:
tp: 2
guide: |
## Overview
Jais-3-MoVA-36B-A4B is a mixture-of-expert (MoE) model in the IFM Jais-3
family. It combines MoVA attention and MoE layers, with 37.44B stored parameters
including embeddings, 5.95B active parameters per token, and support for up
to 524,288 tokens of context.
Jais-3 is an open-weight IFM model family built for transparent
foundation-model research, staged checkpoint analysis, and practical
deployment. The series spans compact dense models for local experimentation,
dense mid-size and large models for high-quality research workloads, and
mixture-of-expert (MoE) models such as Jais-3-MoVA-36B-A4B for higher-capacity
serving.
## Launch Commands
```bash
vllm serve IFM/Jais-3-MoVA-36B-A4B \
--tensor-parallel-size 2 \
--trust-remote-code \
--chat-template-content-format string \
--dtype bfloat16 \
--enable-expert-parallel \
--reasoning-parser k2_horizon \
--enable-auto-tool-choice \
--tool-call-parser k2_horizon
```
## Client usage
```python
from openai import OpenAI
client = OpenAI(
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
)
resp = client.chat.completions.create(
model="IFM/Jais-3-MoVA-36B-A4B",
messages=[{"role": "user", "content": "لدينا عددان مجموعهما 50 وحاصل ضربهما 600. ما هما العددان؟ اشرح خطوات الحل."}],
temperature=1.0, top_p=0.95, max_tokens=2048,
)
print(resp.choices[0].message.content)
```
### Thinking modes
The model supports selectable thinking through `chat_template_kwargs`, per
request or server-wide via `--default-chat-template-kwargs`:
- `{"reasoning_effort": "high"}` — full thinking (default).
- `{"reasoning_effort": "low"}` — no thinking.
## References
- [IFM/Jais-3-MoVA-36B-A4B on Hugging Face](https://huggingface.co/IFM/Jais-3-MoVA-36B-A4B)
- [Jais 3 Collection](https://huggingface.co/collections/IFM/jais-3)
+110
View File
@@ -0,0 +1,110 @@
meta:
title: "K2-Horizon-0.9B"
slug: "k2-horizon-0.9b"
provider: "IFM"
description: "Compact dense model for mathematics, code, instruction following, and STEM tasks"
date_added: 2026-08-31
date_updated: 2026-08-31
tasks:
- text
performance_headline: "1.08B stored parameters including embeddings, with 131K context and native K2-Horizon parsing"
related_recipes:
- "IFM/K2-Horizon-3.7B"
hardware:
h200: verified
model:
model_id: "IFM/K2-Horizon-0.9B"
architecture: dense
parameter_count: "1.08B"
active_parameters: "1.08B"
context_length: 131072
base_args:
- "--trust-remote-code"
- "--chat-template-content-format"
- "string"
- "--dtype"
- "bfloat16"
- "--max-model-len"
- "131072"
- "--gpu-memory-utilization"
- "0.85"
base_env: {}
features:
reasoning:
description: "Extract K2-Horizon reasoning into message.reasoning"
args: ["--reasoning-parser", "k2_horizon"]
tool_calling:
description: "Enable automatic tool choice with the K2-Horizon parser"
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
variants:
default:
precision: bf16
vram_minimum_gb: 3
description: "Released BF16 checkpoint; the command starts at an 8K runtime context"
compatible_strategies:
- single_node_tp
- multi_node_tp
- multi_node_tp_pp
hardware_overrides: {}
guide: |
## Overview
K2-Horizon-0.9B is a compact dense model in the IFM K2-Horizon
family, with 1.08B stored parameters including embeddings and a 131,072-token
context window.
K2-Horizon is an open-weight IFM model family built for transparent
foundation-model research, staged checkpoint analysis, and practical
deployment. The series spans compact dense models such as K2-Horizon-0.9B
for local experimentation, dense mid-size and large models for high-quality
research workloads, and mixture-of-expert (MoE) models for higher-capacity
serving.
## Launch Commands
K2-Horizon-0.9B's `config.json` already enables YaRN, so the full 131,072-token context needs no RoPE override.
```bash
vllm serve IFM/K2-Horizon-0.9B \
--trust-remote-code \
--chat-template-content-format string \
--dtype bfloat16 \
--max-model-len 131072 \
--gpu-memory-utilization 0.85 \
--reasoning-parser k2_horizon \
--enable-auto-tool-choice \
--tool-call-parser k2_horizon
```
Lower `--max-model-len` (e.g. 8192) to reduce KV-cache memory.
## Client usage
```python
from openai import OpenAI
client = OpenAI(
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
)
resp = client.chat.completions.create(
model="IFM/K2-Horizon-0.9B",
messages=[{"role": "user", "content": "Give me three primes above 100."}],
temperature=1.0, top_p=0.95, max_tokens=2048,
)
print(resp.choices[0].message.content)
```
### Thinking modes
The model supports selectable thinking through `chat_template_kwargs`, per
request or server-wide via `--default-chat-template-kwargs`:
- `{"reasoning_effort": "high"}` — full thinking (default).
- `{"reasoning_effort": "medium"}` — faster thinking.
- `{"reasoning_effort": "low"}` — fastest thinking.
## References
- [IFM/K2-Horizon-0.9B on Hugging Face](https://huggingface.co/IFM/K2-Horizon-0.9B)
- [K2 Horizon Collection](https://huggingface.co/collections/IFM/k2-horizon)
+89
View File
@@ -0,0 +1,89 @@
meta:
title: "K2-Horizon-3.7B"
slug: "k2-horizon-3.7b"
provider: "IFM"
description: "Dense model for long-context research, serving, and downstream adaptation"
date_added: 2026-08-31
date_updated: 2026-09-02
tasks: [text]
performance_headline: "5.06B stored parameters including embeddings, with up to 512K context"
related_recipes: ["IFM/K2-Horizon-0.9B", "IFM/K2-Horizon-7B"]
hardware:
h200: verified
model:
model_id: "IFM/K2-Horizon-3.7B"
architecture: dense
parameter_count: "5.06B"
active_parameters: "5.06B"
context_length: 524288
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16"]
base_env: {}
features:
reasoning:
description: "Extract K2-Horizon reasoning into message.reasoning"
args: ["--reasoning-parser", "k2_horizon"]
tool_calling:
description: "Enable automatic tool choice with the K2-Horizon parser"
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
variants:
default:
precision: bf16
vram_minimum_gb: 13
description: "BF16 serving runtime with a 131K context baseline"
compatible_strategies: [single_node_tp, multi_node_tp, multi_node_tp_pp]
hardware_overrides: {}
guide: |
## Overview
K2-Horizon-3.7B is a small dense model in the IFM K2-Horizon family,
with 5.06B stored parameters including embeddings and support for up to
524,288 tokens of context.
K2-Horizon is an open-weight IFM model family built for transparent
foundation-model research, staged checkpoint analysis, and practical
deployment. The series spans compact dense models for local experimentation,
dense mid-size and large models for high-quality research workloads, and
mixture-of-expert (MoE) models for higher-capacity serving.
## Launch Commands
```bash
vllm serve IFM/K2-Horizon-3.7B \
--trust-remote-code \
--chat-template-content-format string \
--dtype bfloat16 \
--reasoning-parser k2_horizon \
--enable-auto-tool-choice \
--tool-call-parser k2_horizon
```
## Client usage
```python
from openai import OpenAI
client = OpenAI(
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
)
resp = client.chat.completions.create(
model="IFM/K2-Horizon-3.7B",
messages=[{"role": "user", "content": "Give me three primes above 100."}],
temperature=1.0, top_p=0.95, max_tokens=2048,
)
print(resp.choices[0].message.content)
```
### Thinking modes
The model supports selectable thinking through `chat_template_kwargs`, per
request or server-wide via `--default-chat-template-kwargs`:
- `{"reasoning_effort": "high"}` — full thinking (default).
- `{"reasoning_effort": "medium"}` — faster thinking.
- `{"reasoning_effort": "low"}` — fastest thinking.
## References
- [IFM/K2-Horizon-3.7B on Hugging Face](https://huggingface.co/IFM/K2-Horizon-3.7B)
- [K2 Horizon Collection](https://huggingface.co/collections/IFM/k2-horizon)
+94
View File
@@ -0,0 +1,94 @@
meta:
title: "K2-Horizon-32B"
slug: "k2-horizon-32b"
provider: "IFM"
description: "Dense long-context model for reasoning research and high-capacity serving"
date_added: 2026-08-31
date_updated: 2026-09-02
tasks: [text]
performance_headline: "34.78B stored parameters including embeddings, with native vLLM execution and up to 512K context"
related_recipes: ["IFM/K2-Horizon-7B", "IFM/K2-Horizon-MoVA-36B-A4B"]
hardware:
h200: verified
model:
model_id: "IFM/K2-Horizon-32B"
architecture: dense
parameter_count: "34.78B"
active_parameters: "34.78B"
context_length: 524288
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16"]
base_env: {}
features:
reasoning:
description: "Extract K2-Horizon reasoning into message.reasoning"
args: ["--reasoning-parser", "k2_horizon"]
tool_calling:
description: "Enable automatic tool choice with the K2-Horizon parser"
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
variants:
default:
precision: bf16
vram_minimum_gb: 84
description: "BF16 serving runtime with native vLLM execution"
compatible_strategies: [single_node_tp, multi_node_tp, multi_node_tp_pp]
hardware_overrides: {}
strategy_overrides:
single_node_tp:
tp: 2
guide: |
## Overview
K2-Horizon-32B is a large dense model in the IFM K2-Horizon family,
with 34.78B stored parameters including embeddings and support for up to
524,288 tokens of context.
K2-Horizon is an open-weight IFM model family built for transparent
foundation-model research, staged checkpoint analysis, and practical
deployment. The series spans compact dense models for local experimentation,
dense mid-size and large models such as K2-Horizon-32B for high-quality
research workloads, and mixture-of-expert (MoE) models for higher-capacity
serving.
## Launch Commands
```bash
vllm serve IFM/K2-Horizon-32B \
--tensor-parallel-size 2 \
--trust-remote-code \
--chat-template-content-format string \
--dtype bfloat16 \
--reasoning-parser k2_horizon \
--enable-auto-tool-choice \
--tool-call-parser k2_horizon
```
## Client usage
```python
from openai import OpenAI
client = OpenAI(
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
)
resp = client.chat.completions.create(
model="IFM/K2-Horizon-32B",
messages=[{"role": "user", "content": "Give me three primes above 100."}],
temperature=1.0, top_p=0.95, max_tokens=2048,
)
print(resp.choices[0].message.content)
```
### Thinking modes
The model supports selectable thinking through `chat_template_kwargs`, per
request or server-wide via `--default-chat-template-kwargs`:
- `{"reasoning_effort": "high"}` — full thinking (default).
- `{"reasoning_effort": "medium"}` — faster thinking.
- `{"reasoning_effort": "low"}` — fastest thinking.
## References
- [IFM/K2-Horizon-32B on Hugging Face](https://huggingface.co/IFM/K2-Horizon-32B)
- [K2 Horizon Collection](https://huggingface.co/collections/IFM/k2-horizon)
@@ -0,0 +1,149 @@
meta:
title: "K2-Horizon-375B-A23B"
slug: "k2-horizon-375b-a23b"
provider: "IFM"
description: "Frontier-scale sparse MoE model for long-context research and high-capacity serving"
date_added: 2026-08-31
date_updated: 2026-09-08
tasks: [text]
performance_headline: "379.17B stored parameters including embeddings / 26.67B active parameters per token"
related_recipes: ["IFM/K2-Horizon-MoVA-36B-A4B"]
hardware:
h200: verified
mi355x: verified
model:
model_id: "IFM/K2-Horizon-375B-A23B"
architecture: moe
parameter_count: "379.17B"
active_parameters: "26.67B"
context_length: 524288
# --enable-expert-parallel is not here: in base_args it would reach the AMD
# path too, which deliberately runs pure TP (see strategy_hardware). The TEP
# strategies emit it themselves; multi_node_tp_pp does not, so it carries its
# own copy in strategy_overrides below.
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16"]
base_env: {}
features:
reasoning:
description: "Extract K2-Horizon reasoning into message.reasoning"
args: ["--reasoning-parser", "k2_horizon"]
tool_calling:
description: "Enable automatic tool choice with the K2-Horizon parser"
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
variants:
default:
precision: bf16
vram_minimum_gb: 911
description: "BF16 serving runtime with an eight-GPU tensor-parallel baseline"
compatible_strategies: [single_node_tep, single_node_tp, multi_node_tep, multi_node_tp_pp]
default_strategy: single_node_tep
# On AMD the expert-parallel dispatch is slower than replicating the experts
# across a pure tensor-parallel shard, so the TEP strategies are opted out and
# single-node TP is the offered path. multi_node_tp_pp is left available: EP
# works there, it is just not the fast default at single-node scale.
strategy_hardware:
single_node_tep:
amd: unsupported
multi_node_tep:
amd: unsupported
hardware_overrides:
amd:
extra_args:
- "--attention-backend"
- "ROCM_AITER_UNIFIED_ATTN"
- "--gpu-memory-utilization"
- "0.9"
extra_env:
VLLM_ROCM_USE_AITER: "1"
NCCL_MIN_NCHANNELS: "112"
strategy_overrides:
single_node_tep:
tp: 8
# multi_node_tp_pp emits no EP flag of its own, and this recipe has always
# run it with expert parallel on. Restored here rather than in base_args so
# it stays off the AMD single-node TP path.
multi_node_tp_pp:
extra_args: ["--enable-expert-parallel"]
guide: |
## Overview
K2-Horizon-375B-A23B is a frontier-scale mixture-of-expert (MoE) model in the IFM
K2-Horizon family, with 379.17B stored parameters including embeddings,
26.67B active parameters per token, and support for up to 524,288 tokens of
context.
K2-Horizon is an open-weight IFM model family built for transparent
foundation-model research, staged checkpoint analysis, and practical
deployment. The series spans compact dense models for local experimentation,
dense mid-size and large models for high-quality research workloads, and
mixture-of-expert (MoE) models such as K2-Horizon-375B-A23B for frontier-scale
research and higher-capacity serving.
## Launch Commands
```bash
vllm serve IFM/K2-Horizon-375B-A23B \
--tensor-parallel-size 8 \
--trust-remote-code \
--chat-template-content-format string \
--dtype bfloat16 \
--enable-expert-parallel \
--reasoning-parser k2_horizon \
--enable-auto-tool-choice \
--tool-call-parser k2_horizon
```
### AMD MI355X
Verified on an 8-GPU MI355X (gfx950) node. AITER supplies the fused MoE and
the unified attention backend:
```bash
export VLLM_ROCM_USE_AITER=1
export NCCL_MIN_NCHANNELS=112
vllm serve IFM/K2-Horizon-375B-A23B \
--tensor-parallel-size 8 \
--trust-remote-code \
--chat-template-content-format string \
--dtype bfloat16 \
--max-model-len 131072 \
--gpu-memory-utilization 0.9 \
--attention-backend ROCM_AITER_UNIFIED_ATTN \
--generation-config auto \
--reasoning-parser k2_horizon \
--enable-auto-tool-choice \
--tool-call-parser k2_horizon
```
## Client usage
```python
from openai import OpenAI
client = OpenAI(
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
)
resp = client.chat.completions.create(
model="IFM/K2-Horizon-375B-A23B",
messages=[{"role": "user", "content": "Give me three primes above 100."}],
temperature=1.0, top_p=0.95, max_tokens=2048,
)
print(resp.choices[0].message.content)
```
### Thinking modes
The model supports selectable thinking through `chat_template_kwargs`, per
request or server-wide via `--default-chat-template-kwargs`:
- `{"reasoning_effort": "high"}` — full thinking (default).
- `{"reasoning_effort": "medium"}` — faster thinking.
- `{"reasoning_effort": "low"}` — fastest thinking.
## References
- [IFM/K2-Horizon-375B-A23B on Hugging Face](https://huggingface.co/IFM/K2-Horizon-375B-A23B)
- [K2 Horizon Collection](https://huggingface.co/collections/IFM/k2-horizon)
+90
View File
@@ -0,0 +1,90 @@
meta:
title: "K2-Horizon-7B"
slug: "k2-horizon-7b"
provider: "IFM"
description: "Dense model for long-context research, fine-tuning, and cost-conscious deployment"
date_added: 2026-08-31
date_updated: 2026-09-02
tasks: [text]
performance_headline: "9B stored parameters including embeddings, with up to 512K context"
related_recipes: ["IFM/K2-Horizon-3.7B", "IFM/K2-Horizon-32B"]
hardware:
h200: verified
model:
model_id: "IFM/K2-Horizon-7B"
architecture: dense
parameter_count: "9B"
active_parameters: "9B"
context_length: 524288
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16"]
base_env: {}
features:
reasoning:
description: "Extract K2-Horizon reasoning into message.reasoning"
args: ["--reasoning-parser", "k2_horizon"]
tool_calling:
description: "Enable automatic tool choice with the K2-Horizon parser"
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
variants:
default:
precision: bf16
vram_minimum_gb: 22
description: "BF16 serving runtime with a 131K context baseline"
compatible_strategies: [single_node_tp, multi_node_tp, multi_node_tp_pp]
hardware_overrides: {}
guide: |
## Overview
K2-Horizon-7B is a compact dense model in the IFM K2-Horizon family,
with 9B stored parameters including embeddings and support for up to 524,288
tokens of context.
K2-Horizon is an open-weight IFM model family built for transparent
foundation-model research, staged checkpoint analysis, and practical
deployment. The series spans compact dense models such as K2-Horizon-7B for
local experimentation, dense mid-size and large models for high-quality
research workloads, and mixture-of-expert (MoE) models for higher-capacity
serving.
## Launch Commands
```bash
vllm serve IFM/K2-Horizon-7B \
--trust-remote-code \
--chat-template-content-format string \
--dtype bfloat16 \
--reasoning-parser k2_horizon \
--enable-auto-tool-choice \
--tool-call-parser k2_horizon
```
## Client usage
```python
from openai import OpenAI
client = OpenAI(
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
)
resp = client.chat.completions.create(
model="IFM/K2-Horizon-7B",
messages=[{"role": "user", "content": "Give me three primes above 100."}],
temperature=1.0, top_p=0.95, max_tokens=2048,
)
print(resp.choices[0].message.content)
```
### Thinking modes
The model supports selectable thinking through `chat_template_kwargs`, per
request or server-wide via `--default-chat-template-kwargs`:
- `{"reasoning_effort": "high"}` — full thinking (default).
- `{"reasoning_effort": "medium"}` — faster thinking.
- `{"reasoning_effort": "low"}` — fastest thinking.
## References
- [IFM/K2-Horizon-7B on Hugging Face](https://huggingface.co/IFM/K2-Horizon-7B)
- [K2 Horizon Collection](https://huggingface.co/collections/IFM/k2-horizon)
@@ -0,0 +1,97 @@
meta:
title: "K2-Horizon-MoVA-36B-A4B"
slug: "k2-horizon-mova-36b-a4b"
provider: "IFM"
description: "Sparse MoVA and MoE long-context model for research and production-style serving experiments"
date_added: 2026-08-31
date_updated: 2026-09-03
tasks: [text]
performance_headline: "37.44B stored parameters including embeddings / 5.95B active parameters per token"
related_recipes: ["IFM/K2-Horizon-32B", "IFM/K2-Horizon-375B-A23B"]
hardware:
h200: verified
model:
model_id: "IFM/K2-Horizon-MoVA-36B-A4B"
architecture: moe
parameter_count: "37.44B"
active_parameters: "5.95B"
context_length: 524288
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16", "--enable-expert-parallel"]
base_env: {}
features:
reasoning:
description: "Extract K2-Horizon reasoning into message.reasoning"
args: ["--reasoning-parser", "k2_horizon"]
tool_calling:
description: "Enable automatic tool choice with the K2-Horizon parser"
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
variants:
default:
precision: bf16
vram_minimum_gb: 90
description: "BF16 serving runtime with native vLLM execution"
compatible_strategies: [single_node_tep, multi_node_tep, multi_node_tp_pp]
default_strategy: single_node_tep
hardware_overrides: {}
strategy_overrides:
single_node_tep:
tp: 2
guide: |
## Overview
K2-Horizon-MoVA-36B-A4B is a mixture-of-expert (MoE) model in the IFM K2-Horizon
family. It combines MoVA attention and MoE layers, with 37.44B stored parameters
including embeddings, 5.95B active parameters per token, and support for up
to 524,288 tokens of context.
K2-Horizon is an open-weight IFM model family built for transparent
foundation-model research, staged checkpoint analysis, and practical
deployment. The series spans compact dense models for local experimentation,
dense mid-size and large models for high-quality research workloads, and
mixture-of-expert (MoE) models such as K2-Horizon-MoVA-36B-A4B for higher-capacity
serving.
## Launch Commands
```bash
vllm serve IFM/K2-Horizon-MoVA-36B-A4B \
--tensor-parallel-size 2 \
--trust-remote-code \
--chat-template-content-format string \
--dtype bfloat16 \
--enable-expert-parallel \
--reasoning-parser k2_horizon \
--enable-auto-tool-choice \
--tool-call-parser k2_horizon
```
## Client usage
```python
from openai import OpenAI
client = OpenAI(
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
)
resp = client.chat.completions.create(
model="IFM/K2-Horizon-MoVA-36B-A4B",
messages=[{"role": "user", "content": "Give me three primes above 100."}],
temperature=1.0, top_p=0.95, max_tokens=2048,
)
print(resp.choices[0].message.content)
```
### Thinking modes
The model supports selectable thinking through `chat_template_kwargs`, per
request or server-wide via `--default-chat-template-kwargs`:
- `{"reasoning_effort": "high"}` — full thinking (default).
- `{"reasoning_effort": "medium"}` — faster thinking.
- `{"reasoning_effort": "low"}` — fastest thinking.
## References
- [IFM/K2-Horizon-MoVA-36B-A4B on Hugging Face](https://huggingface.co/IFM/K2-Horizon-MoVA-36B-A4B)
- [K2 Horizon Collection](https://huggingface.co/collections/IFM/k2-horizon)
@@ -0,0 +1,153 @@
meta:
title: "Mellum2-12B-A2.5B-Instruct"
slug: "mellum2-12b-a2.5b-instruct"
provider: "JetBrains"
description: "JetBrains' instruction-tuned code MoE (12B total / 2.5B active) that answers directly without an externalized chain of thought — low-latency coding and tool use"
date_added: 2026-06-02
date_updated: 2026-06-02
tasks:
- text
performance_headline: "78.4 EvalPlus, 67.1 MultiPL-E — direct answers, fits on a single GPU"
related_recipes:
- "JetBrains/Mellum2-12B-A2.5B-Thinking"
hardware:
h200: verified
mi300x: verified
mi325x: verified
mi355x: verified
model:
model_id: "JetBrains/Mellum2-12B-A2.5B-Instruct"
min_vllm_version: "0.23.0"
architecture: moe
parameter_count: "12B"
active_parameters: "2.5B"
context_length: 131072
base_args:
- "--max-model-len"
- "131072"
- "--trust-remote-code"
base_env: {}
features:
tool_calling:
description: "Hermes tool-call parser for function calling"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "hermes"
opt_in_features: []
variants:
default:
precision: bf16
vram_minimum_gb: 29
description: "Native bfloat16 weights; fits comfortably on a single H200/H100/A100"
compatible_strategies:
- single_node_tp
- single_node_tep
- single_node_dep
hardware_overrides: {}
strategy_overrides:
single_node_tp:
tp: 1
guide: |
## Overview
[Mellum2-12B-A2.5B-Instruct](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Instruct)
is JetBrains' instruction-tuned code assistant. It shares the same Mixture-of-Experts
backbone as the rest of the Mellum2 family — 64 experts (8 activated per token), 12B total
/ 2.5B active parameters, sliding-window + full-attention layers, 131,072-token context —
but is post-trained (SFT + RLVR on math, coding, tool use, instruction following, reasoning,
and knowledge) to **answer directly, without an externalized chain of thought**. For complex
debugging, multi-step planning, or math/reasoning-heavy tasks where you want explicit
reasoning traces, use the
[Thinking](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Thinking) variant instead.
## Prerequisites
- Hardware: a single H200, H100, or A100 (~29 GB at bf16) is plenty. On AMD,
use MI300X / MI325X / MI355X with TP=1.
- vLLM **nightly** — `MellumForCausalLM` support landed after v0.22.0 and is not yet in a
stable release. Install the nightly wheels until the next tagged release ships.
### Install vLLM (nightly)
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --extra-index-url https://wheels.vllm.ai/nightly
```
## Launch command
Unlike the Thinking checkpoint, Instruct does not emit `<think>` blocks, so no
`--reasoning-parser` is needed.
### Single GPU
```bash
vllm serve JetBrains/Mellum2-12B-A2.5B-Instruct \
--max-model-len 131072 \
--trust-remote-code \
--tensor-parallel-size 1
# Optional: add tool calling.
vllm serve JetBrains/Mellum2-12B-A2.5B-Instruct \
--max-model-len 131072 \
--trust-remote-code \
--tensor-parallel-size 1 \
--enable-auto-tool-choice \
--tool-call-parser hermes
```
### Docker (AMD MI300X / MI325X / MI355X)
MI300X / MI325X / MI355X GPUs have larger per-GPU HBM, so you can use
`--max-model-len auto`.
```bash
# Plain serving
vllm serve JetBrains/Mellum2-12B-A2.5B-Instruct \
--max-model-len auto \
--trust-remote-code \
--tensor-parallel-size 1
# Add tool calling.
vllm serve JetBrains/Mellum2-12B-A2.5B-Instruct \
--max-model-len auto \
--trust-remote-code \
--tensor-parallel-size 1 \
--enable-auto-tool-choice \
--tool-call-parser hermes
```
## Client usage
JetBrains recommends sampling at `temperature=0.6`, `top_p=0.95`, `top_k=20`.
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
model="JetBrains/Mellum2-12B-A2.5B-Instruct",
messages=[{"role": "user", "content": "Write a Python function to reverse a string."}],
max_tokens=81920,
temperature=0.6,
top_p=0.95,
extra_body={"top_k": 20},
)
print(resp.choices[0].message.content)
```
## References
- [Model card](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Instruct)
- [vLLM support PR #43992](https://github.com/vllm-project/vllm/pull/43992)
- [Mellum2 Technical Report](https://arxiv.org/abs/2605.31268)
@@ -0,0 +1,146 @@
meta:
title: "Mellum2-12B-A2.5B-Thinking"
slug: "mellum2-12b-a2.5b-thinking"
provider: "JetBrains"
description: "JetBrains' reasoning-augmented code MoE (12B total / 2.5B active) that emits explicit <think> chains for debugging, planning, and agentic coding"
date_added: 2026-06-02
date_updated: 2026-06-02
tasks:
- text
performance_headline: "69.9 LiveCodeBench v6, 58.4 AIME — fits on a single GPU"
related_recipes:
- "JetBrains/Mellum2-12B-A2.5B-Instruct"
hardware:
h200: verified
model:
model_id: "JetBrains/Mellum2-12B-A2.5B-Thinking"
min_vllm_version: "0.23.0"
architecture: moe
parameter_count: "12B"
active_parameters: "2.5B"
context_length: 131072
base_args:
- "--max-model-len"
- "131072"
base_env: {}
features:
reasoning:
description: "Parse the model's <think>...</think> reasoning blocks (Qwen3-style parser)"
args:
- "--reasoning-parser"
- "qwen3"
tool_calling:
description: "Hermes tool-call parser for function calling"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "hermes"
spec_decoding:
description: "DFlash speculative decoding with an externally trained draft model; the served checkpoint is unchanged."
args:
- "--speculative-config"
- '{"model": "RedHatAI/Mellum2-12B-A2.5B-Thinking-speculator.dflash", "num_speculative_tokens": 7, "method": "dflash"}'
opt_in_features:
- spec_decoding
variants:
default:
precision: bf16
vram_minimum_gb: 29
description: "Native bfloat16 weights; fits comfortably on a single H200/H100/A100"
fp8:
model_id: "RedHatAI/Mellum2-12B-A2.5B-Thinking-FP8-Dynamic"
precision: fp8
vram_minimum_gb: 15
description: "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization"
extra_args:
- "--kv-cache-dtype"
- "fp8"
nvfp4:
model_id: "RedHatAI/Mellum2-12B-A2.5B-Thinking-NVFP4"
precision: nvfp4
vram_minimum_gb: 8
description: "RedHatAI NVFP4 quantization"
compatible_strategies:
- single_node_tp
- single_node_tep
- single_node_dep
strategy_overrides:
single_node_tp:
tp: 1
guide: |
## Overview
[Mellum2-12B-A2.5B-Thinking](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Thinking)
is JetBrains' reasoning-augmented code assistant. It uses a Mixture-of-Experts architecture
with 64 experts (8 activated per token) — 12B total parameters, 2.5B active — combining
sliding-window and full-attention layers for a 131,072-token context. The model emits its
chain-of-thought inside `<think>...</think>` blocks before the final answer, making it
suited to complex debugging, multi-step planning, and agentic workflows. For direct,
low-latency answers without reasoning traces, use the
[Instruct](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Instruct) variant instead.
## Prerequisites
- Hardware: a single H200, H100, or A100 (~29 GB at bf16) is plenty
- vLLM **nightly** — `MellumForCausalLM` support landed after v0.22.0 and is not yet in a
stable release. Install the nightly wheels until the next tagged release ships.
### Install vLLM (nightly)
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --extra-index-url https://wheels.vllm.ai/nightly
```
## Launch command
```bash
# With reasoning (recommended for the Thinking checkpoint)
vllm serve JetBrains/Mellum2-12B-A2.5B-Thinking \
--max-model-len 131072 \
--reasoning-parser qwen3
# Add tool calling
vllm serve JetBrains/Mellum2-12B-A2.5B-Thinking \
--max-model-len 131072 \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser hermes
```
## Client usage
JetBrains recommends sampling at `temperature=0.6`, `top_p=0.95`, `top_k=20` for the
Thinking checkpoint.
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
model="JetBrains/Mellum2-12B-A2.5B-Thinking",
messages=[{"role": "user", "content": "Is 1024 a power of 2? Explain your reasoning."}],
max_tokens=81920,
temperature=0.6,
top_p=0.95,
extra_body={"top_k": 20},
)
print(resp.choices[0].message.content)
```
## References
- [Model card](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Thinking)
- [vLLM support PR #43992](https://github.com/vllm-project/vllm/pull/43992)
- [Mellum2 Technical Report](https://arxiv.org/abs/2605.31268)
@@ -0,0 +1,137 @@
meta:
title: "LFM2.5 1.2B Base"
slug: "lfm2.5-1.2b-base"
provider: "LiquidAI"
description: "Liquid AI's 1.2B pretrained base model on the LFM2 hybrid conv+attention backbone — a text-completion and fine-tuning foundation (no chat template)."
date_added: 2026-06-25
date_updated: 2026-06-24
tasks:
- text
performance_headline: "1.2B pretrained base (no chat template) — completions endpoint, ideal as a fine-tuning base"
related_recipes:
- LiquidAI/LFM2.5-1.2B-Instruct
- LiquidAI/LFM2.5-350M
hardware:
h100: verified
h200: verified
b200: verified
model:
model_id: "LiquidAI/LFM2.5-1.2B-Base"
min_vllm_version: "0.23.0"
architecture: dense
parameter_count: "1.2B"
active_parameters: "1.2B"
context_length: 32768
base_args: []
base_env: {}
variants:
default:
precision: bf16
vram_minimum_gb: 3
description: "Full BF16 — single 8 GB+ NVIDIA GPU with room for KV cache"
compatible_strategies:
- single_node_tp
hardware_overrides:
# Blackwell (B200) only: route RMSNorm to the bundled oink kernels (measured ~+2.6% on B200,
# output identical). Inert (falls back to native) on non-Blackwell GPUs.
blackwell:
extra_env:
VLLM_USE_OINK_OPS: "1"
strategy_overrides:
single_node_tp:
tp: 1
guide: |
## Overview
[LFM2.5-1.2B-Base](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Base) is the pretrained base
checkpoint behind [Liquid AI's](https://www.liquid.ai/) LFM2.5-1.2B instruct models, built on
the **LFM2 hybrid backbone** (short-range gated convolution blocks interleaved with grouped-query
attention). It is **not** instruction-tuned and has no chat template — use the
`/v1/completions` endpoint (raw text continuation), or fine-tune it for your own task.
### Key Features
- **Pretrained base**: No chat template, no instruction tuning — a clean foundation for fine-tuning or raw completion.
- **Hybrid backbone**: Gated short convolutions interleaved with grouped-query attention — a smaller KV cache and lower decode latency than a same-size full-attention transformer.
- **32K context**: 32,768-token context window.
- **Native vLLM support**: Served via the `Lfm2ForCausalLM` architecture.
### Supported Variants
Dense:
- `LiquidAI/LFM2.5-230M` (230M)
- `LiquidAI/LFM2.5-350M` (350M)
- `LiquidAI/LFM2.5-1.2B-Instruct` (1.2B)
- `LiquidAI/LFM2.5-1.2B-Thinking` (1.2B, reasoning)
- `LiquidAI/LFM2.5-1.2B-JP` / `LiquidAI/LFM2.5-1.2B-JP-202606` (Japanese)
- `LiquidAI/LFM2.5-1.2B-Base` (pretrained base)
MoE:
- `LiquidAI/LFM2.5-8B-A1B` (8B total / ~1B active)
Vision-Language:
- `LiquidAI/LFM2.5-VL-450M`, `LiquidAI/LFM2.5-VL-1.6B`
See the [LFM2.5 usage guide](../../LiquidAI/LFM2.5) for the full family.
## Prerequisites
- **Hardware**: 1× GPU with ≥8 GB VRAM.
- **vLLM**: ≥ 0.23.0.
### pip (NVIDIA CUDA)
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```
## Deployment Configurations
### Quick Start (Single GPU, BF16)
```bash
vllm serve LiquidAI/LFM2.5-1.2B-Base
```
### Docker (NVIDIA)
```bash
docker run -itd --name lfm2.5-base \
--ipc=host --network host --shm-size 16G --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:latest \
--model LiquidAI/LFM2.5-1.2B-Base \
--host 0.0.0.0 --port 8000
```
## Client Usage
### Text Completion
This is a base model — use the **completions** endpoint, not chat. Recommended sampling:
temperature `0.3`, `min_p 0.15`, `repetition_penalty 1.05`.
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.completions.create(
model="LiquidAI/LFM2.5-1.2B-Base",
prompt="The three laws of thermodynamics are:",
temperature=0.3,
extra_body={"min_p": 0.15, "repetition_penalty": 1.05},
)
print(response.choices[0].text)
```
## Configuration Tips
- Use `/v1/completions` (raw continuation) — there is no chat template on a base model.
- For fine-tuning, this checkpoint is the recommended starting point for custom LFM2.5-1.2B tasks.
- Set `--max-model-len` to match your workload (up to 32K).
- Sampling presets are per-request client defaults — don't bake them into `vllm serve`.
## References
- [Model card](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Base)
- [LFM2.5 usage guide](../../LiquidAI/LFM2.5)
- [Liquid AI](https://www.liquid.ai/)
- [vLLM supported models](https://docs.vllm.ai/en/latest/models/supported_models.html)

Some files were not shown because too many files have changed in this diff Show More