Mark McLoughlin and AI Assistant
45d56c0baf
[Core] Model console logging as CLI configuration ( #57205 )
...
Add `--logging-config` CLI argument which can be supplied as
JSON or using dotted arguments. The `--log-level` argument
is provided for convenience, and `--log-config-file` is deprecated
in favor of `--logging-config.pylogging_config_file`.
Signed-off-by: Mark McLoughlin <markmc@redhat.com >
Co-authored-by: AI Assistant <noreply@openai.com >
2026-09-25 11:36:59 +01:00
Andreas Karatzas and Codex
153f2ba1b9
[Compile][CI] Honor Triton cache overrides and add AMD timeout headroom ( #58474 )
...
Signed-off-by: Andreas Karatzas <andreas.karatzas@protonmail.com >
Co-authored-by: Codex <noreply@openai.com >
2026-09-24 01:27:07 -05:00
b715699df4
[ROCm][Perf] Extend QK-norm/RoPE/KV-cache fusion to MRoPE ( #50212 )
...
Signed-off-by: Vorapol Assavasangthong <Vorapol.Assavasangthong@amd.com >
Co-authored-by: Santosh Hiremath <Santosh.Hiremath@amd.com >
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com >
2026-09-23 11:11:57 -05:00
Andreas Karatzas and OpenAI Codex
26879f3260
[ROCm][Compile] Fuse AITER static FP8 attention output ( #58099 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-09-23 03:16:52 -05:00
Andreas Karatzas
0eb42dbbfc
[ROCm][Compile] Support BF16 AsyncTP fusion ( #58098 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
2026-09-22 20:36:38 -05:00
Mikko Tukiainen and Cursor
68363d64ec
[ROCm][Perf] Avoid extra reshape kernel in Qwen GDN output norm ( #47842 )
...
Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com >
Co-authored-by: Cursor <cursoragent@cursor.com >
2026-09-22 14:38:16 +08:00
Taneem Ibrahim and Harry Mellor
e960ead0aa
[Mypy] Fix mypy typing for Transformers models ( #54320 )
...
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com >
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-09-17 12:11:31 -04:00
Harry Mellor and Claude Opus 5
651a88c09a
[CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings ( #57212 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 10:57:24 -07:00
Harry Mellor and Claude Opus 5
8c1557a79c
Add pydocstyle to the ruff rules ( #52136 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Signed-off-by: Harry Mellor <hej.mellor@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-16 16:44:29 +00:00
d43bb2f37f
[3/N] HiSparse: host-resident sparse-MLA decode hot-buffering ( #53781 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com >
Signed-off-by: Matthew Bonanni <matthew@neuralmagic.com >
Signed-off-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Lucas Wilkinson <lwilkinson@neuralmagic.com >
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
2026-09-12 00:36:47 -04:00
Nick Hill and Andreas Karatzas
cc5dd0a857
[MRV1] Scope breakable cudagraphs to the piecewise path only ( #56312 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-09-11 19:55:34 +00:00
e77daef89e
[Model] Support DeepSeek-V4.1-Flash ( #56214 )
...
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai >
Signed-off-by: andyluo7 <andy.luo@amd.com >
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai >
Co-authored-by: andyluo7 <andy.luo@amd.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-09-11 02:11:19 -07:00
Qidong Su and OpenAI Codex
9e137d8d23
[Kernel] Enable optimized FlashInfer add-RMSNorm NVFP4 fusion ( #51925 )
...
Signed-off-by: Qidong Su <qidongs@nvidia.com >
Signed-off-by: Qidong Su <soodoshll@gmail.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-09-08 17:20:32 -07:00
Wentao Ye
6b15bea080
[Refactor] Remove utils dead code ( #53941 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-09-08 11:22:09 -04:00
Jee Jee Li
199cb9b964
[Kernel] Remove unused fake implementation ( #55535 )
...
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai >
2026-09-06 17:54:32 -07:00
Nick Hill
c28feab989
[Core][MRV2] Freeze gc during V2 CG capture; skip per-descriptor cleanup ( #54646 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-08-31 19:37:46 -07:00
Nick Hill
4aab2b0ebe
[Model Runner V2] Use MRV2 for all models by default ( #53183 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-08-27 11:59:34 -07:00
Jee Jee Li
88e1b11313
[Perf] Tune FlashInfer all-reduce thresholds for single-node TP8 on SM103 ( #53606 )
...
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai >
2026-08-26 01:46:56 +00:00
Khushali Desai
23ab0cfdbc
speculative decoding under tensor parallelism (TP>1) , workspace creation select max hidden dim of target and draft model ( #52193 )
...
Signed-off-by: khushali9 <khushali.desai9@gmail.com >
2026-08-24 21:33:36 +00:00
Wentao Ye
811d12ee26
[Mypy Fix] Mypy fix for "vllm/model_executor/models/[eE][fF]" ( #53381 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-08-22 15:41:27 -04:00
Lucas Wilkinson
8bdc70ec7b
[6/N][KV-Cache Layout Refactor] Standardize KV cache layout ( #51718 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
2026-08-21 17:47:49 -07:00
Kevin H. Luu
c0ff33404b
Revert compile-cache device index regression on CPU ( #53304 )
2026-08-21 16:08:13 -04:00
Frederik Gossen
47cd1c8885
[Core] Add dynamo_timed tracing for print_readable ( #40834 )
...
Signed-off-by: Frederik Gossen <frgossen@meta.com >
2026-08-21 09:07:21 -07:00
Neil Schemenauer and Tyler Michael Smith
6dcc5d7cae
[Bugfix] Include device index in compile cache paths ( #38962 )
...
Signed-off-by: Neil Schemenauer <nas@arctrix.com >
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com >
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com >
2026-08-21 09:07:16 -07:00
shanjiaz
e078a2238a
[Bugfix] Widen flashinfer.comm import guard so a failed import doesn't abort engine startup ( #52241 )
...
Signed-off-by: shanjiaz <zsjwpianpian@gmail.com >
2026-08-14 16:27:24 +00:00
Andreas Karatzas
7e724dca89
[ROCm] Fix AITER all-reduce fusion coverage ( #50802 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
2026-08-06 14:52:23 +08:00
Rohan Potdar and Claude Opus 4.8
d0ce3dadb6
[ROCm]: Bump torch 2.12, triton 3.7, torchaudio, torchvision ( #50607 )
...
Signed-off-by: Rohan Potdar <rohan.potdar@amd.com >
Signed-off-by: Rohan138 <rohanpotdar138@gmail.com >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
2026-08-04 20:14:11 -05:00
skysnow2001 and Douglas Lehr
f43e1d26e3
[ROCm] Enable AITER and FP8 inference on GFX120x ( #43615 )
...
Signed-off-by: skysnow2001 <skysnow9285@gmail.com >
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com >
2026-08-03 16:45:37 -05:00
Wentao Ye
0cf49a5d15
[Refactor] Remove multiple dead codes ( #50285 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-08-03 10:59:08 -04:00
Richard Zou and Codex
c27b080d26
[compile] Fix fake kernel return dtype ( #50444 )
...
Signed-off-by: Richard Zou <zou3519@gmail.com >
Co-authored-by: Codex <noreply@openai.com >
2026-07-30 19:50:02 +00:00
BadrBasowid
59e831c09a
[Compilation]Fuse Transformers Residual Add + RMSNorm ( #48757 )
...
Signed-off-by: BadrBasowid <badr.basowid@gmail.com >
2026-07-30 13:46:36 +00:00
5fa0154448
[Rubin] Enable NVLink all-reduce paths on SM107 ( #49647 )
...
Signed-off-by: Zachary Aristei <zaristei@nvidia.com >
Co-authored-by: Sahithi Chigurupati <chigurupati.sahithi@gmail.com >
Co-authored-by: hlu1 <14827759+hlu1@users.noreply.github.com >
2026-07-29 20:53:25 +00:00
afriedri
f472ab0a4c
Remove triton per group quant [ROCm] [Bugfix] ( #49621 )
...
Signed-off-by: Andy Friedrich <afriedri@amd.com >
2026-07-28 05:53:46 +00:00
Chaojun Zhang and Kunshang Ji
bf4f633b4c
[XPU] Enable QK Norm + RoPE fusion pass on XPU ( #49394 )
...
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com >
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com >
2026-07-27 03:22:29 +00:00
Schwinn Saereesitthipitak
b5b61c622c
[Core][Distributed] Add process-checkpoint lifecycle hooks for communicators (starting with Flashinfer) ( #46877 )
...
Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com >
2026-07-26 14:47:50 -04:00
Woosuk Kwon
213f681f81
Revert "[Perf][GLM-5.2] Blackwell decode optimizations" ( #49768 )
2026-07-24 16:04:59 -07:00
dd72658e7d
[Perf][GLM-5.2] Blackwell decode optimizations ( #48597 )
...
Signed-off-by: Peiyuan Zhou <peiyuanzhou1994@gmail.com >
Signed-off-by: zhou <zhou@zhoudeMacBook-Pro.local >
Signed-off-by: Stefan Koncarevic <stefan.koncarevic@amd.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Giancarlo Delfin <32987265+TheEpicDolphin@users.noreply.github.com >
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: zhou <zhou@zhoudeMacBook-Pro.local >
Co-authored-by: stefankoncarevic <Stefan.Koncarevic@amd.com >
2026-07-23 21:36:27 -07:00
stefankoncarevic and Andreas Karatzas
387189c429
[ROCm] Remove redundant AITER fused_qk_rmsnorm probe (avoids config-time HIP init) ( #47992 )
...
Signed-off-by: Stefan Koncarevic <stefan.koncarevic@amd.com >
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
2026-07-21 22:58:14 -05:00
Jack Hu and Douglas Lehr
2cab53ddee
[Model][Hardware][AMD]: Part 1/2 -> Enable e2e QK Norm + RoPE + KV Cache runtime fusion for Qwen3-30B-A3B on ROCM_AITER_FA, and ROCM_AITER_UNIFIED_ATTN ( #42749 )
...
Signed-off-by: Jack Hu <Jack.Hu@amd.com >
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com >
2026-07-16 17:39:04 -05:00
Wentao Ye
626c90b2d5
[Refactor] Move fla to third party ( #48500 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-07-16 19:22:36 +01:00
nemanjaudovic
81e13a0591
[Compilation] Skip x.size(dim) in _decompose_size_nodes ( #42543 )
...
Signed-off-by: nemanjaudovic <nudovic@amd.com >
2026-07-15 17:59:14 -07:00
aoshen02 and OpenAI Codex
3b39fd284a
[Bugfix][Spec Decode] Support heterogeneous QK fusion geometry ( #48671 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-07-14 22:37:10 -07:00
emricksini-h and Simon Mo
fec64fea75
[BugFix] Correct OTEL span start time for Dynamo compilation ( #40698 )
...
Signed-off-by: emricksini-h <emrick.birivoutin@hcompany.ai >
Co-authored-by: Simon Mo <simon.mo@hey.com >
2026-07-13 16:25:03 -07:00
Mohammad Miadh Angkad
75fe92a316
[Distributed][Perf] Enable FlashInfer MNNVL allreduce RMS quant fusion ( #48064 )
...
Signed-off-by: Mohammad Miadh Angkad <176301910+mmangkad@users.noreply.github.com >
2026-07-13 15:02:59 +08:00
Hugo Centeno
5f8e73cb8b
[Bugfix] Guard mixed-dtype allreduce RMSNorm quant fusions ( #48330 )
...
Signed-off-by: hcenteno <hugo.centeno@estudiantat.upc.edu >
2026-07-12 09:39:27 +00:00
429f405748
[Bugfix] Guard CUDA-only rms_norm_per_block_quant in FUSED_OPS for non-CUDA builds ( #47296 )
...
Signed-off-by: Tsvika Shapira <tsvika@moonmath.ai >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-09 10:10:53 -04:00
jesco and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
598d51153a
[Bugfix][Distributed] Delegate MNNVL allreduce one-shot selection ( #47589 )
...
Signed-off-by: jesco-absolut <team@srswti.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-06 07:47:06 -07:00
BadrBasowid
740f379fae
[ROCm][AITER] Directly Implement AITER Custom All-reduce in CudaCommunicator ( #46065 )
...
Signed-off-by: BadrBasowid <badr.basowid@gmail.com >
2026-07-06 12:16:32 +00:00
xaguilar-amd
09663abde0
[ROCm][MLA] Fuse MLA q/kv RMSNorm + FP8 per-token quant in the FP8 attention path ( #44977 )
...
Signed-off-by: Xavier Aguilar <xavier.aguilarfruto@amd.com >
Signed-off-by: Xavier Aguilar <Xavier.AguilarFruto@amd.com >
2026-07-02 13:00:41 +08:00
Yejing Lai and Kunshang Ji
f651a8a9a4
[XPU][UT]Enable ut qk_norm_rope_fusion ( #42486 )
...
Signed-off-by: Lai, Yejing <yejing.lai@intel.com >
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com >
2026-07-01 07:38:03 +00:00
