iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
23,095 Commits 3 Branches 0 Tags
bfdbfa1632ef638db7d201f366c6f46cc26ca2b6
Commit Graph
461 Commits
Author SHA1 Message Date
TaimysTaimysHarry Mellor
 		c068db0e1d [Bugfix][Compile] Replace Transformers AOT caching with graph caching (#58918) 
Signed-off-by: fangyao fan <2281693949@qq.com>
Signed-off-by: Taimys <ffy2281693949@gmail.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Taimys <ffy2281693949@gmail.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-10-13 19:17:29 +00:00
Mikko Tukiainen abad32a718 [ROCm][Perf] Fix Gemma AR+RMS prefill fallback (#58547) 
Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com>
Signed-off-by: Mikko Tukiainen <mikko.tukiainen@amd.com>
 2026-10-12 19:53:08 +00:00
Laith Sakkaandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 12e6a585ae [Fix][Compile] Declare group_shape in the static FP8 quant fusion … (#61108) 
Signed-off-by: Laith Sakka <lsakka@meta.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-10-13 01:09:57 +08:00
Nick Hill 7d1d8660d6 [MRV2] Miscellaneous code structure updates (#60754) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-10-09 17:58:09 -07:00
Woosuk Kwon 194bd72be7 [Compile] Remove torch.compile-based sequence parallelism and AsyncTP (#60517) 2026-10-08 08:28:23 -07:00
PatchyTISPatchouliTaisaNick HillXuanan ChenGiancarlo Delfin
 		819852df6b [ModelRunner V2] Speculative Decoding NGram GPU Implementations (#40704) 
Signed-off-by: PatchouliTaisa <patchychen@tencent.com>
Signed-off-by: PatchouliTaisa <pyramkar@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Signed-off-by: Xuanan Chen <xuananchenc@nvidia.com>
Co-authored-by: PatchouliTaisa <patchychen@tencent.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Xuanan Chen <xuananchenc@nvidia.com>
Co-authored-by: Giancarlo Delfin <32987265+TheEpicDolphin@users.noreply.github.com>
 2026-10-06 13:32:59 -07:00
eky-amdandClaude Fable 5 29f955cce0 [ROCm] Fuse MLA dual RMSNorm + FP8 group quant for DeepSeek-R1 (#54857) 
Signed-off-by: Ethan Ky <ethan.ky@amd.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-10-06 11:26:38 -05:00
aoshen02andClaude Opus 5.5 5e56e9af2a [Feature] Release the CUDA graph pool on sleep (#59160) 
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-04 01:13:10 +00:00
Francesco FuscoClaudeLuka Govedič
 		d5ee309314 [UX] Tag torch.compile log lines with the component being compiled (#48133) 
Signed-off-by: Francesco Fusco <ffu@zurich.ibm.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
 2026-09-30 10:34:31 +00:00
Jung JiyuandMisha Goin cb4f016c5d [Bugfix] Fix standalone torch.compile cache loading after relocation (#52142) 
Signed-off-by: jungjiyu <libraryofjiyu@gmail.com>
Co-authored-by: Misha Goin <mgoin64@gmail.com>
 2026-09-29 12:53:20 -04:00
Cheese CakeOpenAI CodexNick HillWentao Ye
 		4e59d53ab5 [MRV2] Support stock torch.compile mode (#59079) 
Signed-off-by: cheesecake <farzanaman99@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
 2026-09-28 20:17:48 -07:00
Laith Sakka b39f760cc7 [torch.compile] Canonicalize functionalized split slices for fusion pa… (#57299) 
Signed-off-by: Laith Sakka <lsakka@meta.com>
 2026-09-28 20:21:59 -04:00
Mark McLoughlinandAI Assistant 45d56c0baf [Core] Model console logging as CLI configuration (#57205) 
Add `--logging-config` CLI argument which can be supplied as
JSON or using dotted arguments. The `--log-level` argument
is provided for convenience, and `--log-config-file` is deprecated
in favor of `--logging-config.pylogging_config_file`.

Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: AI Assistant <noreply@openai.com>
 2026-09-25 11:36:59 +01:00
Andreas KaratzasandCodex 153f2ba1b9 [Compile][CI] Honor Triton cache overrides and add AMD timeout headroom (#58474) 
Signed-off-by: Andreas Karatzas <andreas.karatzas@protonmail.com>
Co-authored-by: Codex <noreply@openai.com>
 2026-09-24 01:27:07 -05:00
vorapolsiloaiSantosh HiremathDouglas Lehr
 		b715699df4 [ROCm][Perf] Extend QK-norm/RoPE/KV-cache fusion to MRoPE (#50212) 
Signed-off-by: Vorapol Assavasangthong <Vorapol.Assavasangthong@amd.com>
Co-authored-by: Santosh Hiremath <Santosh.Hiremath@amd.com>
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com>
 2026-09-23 11:11:57 -05:00
Andreas KaratzasandOpenAI Codex 26879f3260 [ROCm][Compile] Fuse AITER static FP8 attention output (#58099) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
 2026-09-23 03:16:52 -05:00
Andreas Karatzas 0eb42dbbfc [ROCm][Compile] Support BF16 AsyncTP fusion (#58098) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
 2026-09-22 20:36:38 -05:00
Mikko TukiainenandCursor 68363d64ec [ROCm][Perf] Avoid extra reshape kernel in Qwen GDN output norm (#47842) 
Signed-off-by: Mikko Tukiainen <Mikko.Tukiainen@amd.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-09-22 14:38:16 +08:00
Taneem IbrahimandHarry Mellor e960ead0aa [Mypy] Fix mypy typing for Transformers models (#54320) 
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-09-17 12:11:31 -04:00
Harry MellorandClaude Opus 5 651a88c09a [CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings (#57212) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-16 10:57:24 -07:00
Harry MellorandClaude Opus 5 8c1557a79c Add pydocstyle to the ruff rules (#52136) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <hej.mellor@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-16 16:44:29 +00:00
Matthew BonanniOpenAI CodexLucas WilkinsonLucas WilkinsonNicolò LucchesiOpenAI CodexClaude Opus 5Lucas Wilkinson
 		d43bb2f37f [3/N] HiSparse: host-resident sparse-MLA decode hot-buffering (#53781) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Signed-off-by: Matthew Bonanni <matthew@neuralmagic.com>
Signed-off-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
 2026-09-12 00:36:47 -04:00
Nick HillandAndreas Karatzas cc5dd0a857 [MRV1] Scope breakable cudagraphs to the piecewise path only (#56312) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
 2026-09-11 19:55:34 +00:00
Yongye ZhuJee Jee Liandyluo7OpenAI Codexzjy0516Claude Opus 5
 		e77daef89e [Model] Support DeepSeek-V4.1-Flash (#56214) 
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
Signed-off-by: andyluo7 <andy.luo@amd.com>
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai>
Co-authored-by: andyluo7 <andy.luo@amd.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-09-11 02:11:19 -07:00
Qidong SuandOpenAI Codex 9e137d8d23 [Kernel] Enable optimized FlashInfer add-RMSNorm NVFP4 fusion (#51925) 
Signed-off-by: Qidong Su <qidongs@nvidia.com>
Signed-off-by: Qidong Su <soodoshll@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-09-08 17:20:32 -07:00
Wentao Ye 6b15bea080 [Refactor] Remove utils dead code (#53941) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-09-08 11:22:09 -04:00
Jee Jee Li 199cb9b964 [Kernel] Remove unused fake implementation (#55535) 
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
 2026-09-06 17:54:32 -07:00
Nick Hill c28feab989 [Core][MRV2] Freeze gc during V2 CG capture; skip per-descriptor cleanup (#54646) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-08-31 19:37:46 -07:00
Nick Hill 4aab2b0ebe [Model Runner V2] Use MRV2 for all models by default (#53183) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-08-27 11:59:34 -07:00
Jee Jee Li 88e1b11313 [Perf] Tune FlashInfer all-reduce thresholds for single-node TP8 on SM103 (#53606) 
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
 2026-08-26 01:46:56 +00:00
Khushali Desai 23ab0cfdbc speculative decoding under tensor parallelism (TP>1) , workspace creation select max hidden dim of target and draft model (#52193) 
Signed-off-by: khushali9 <khushali.desai9@gmail.com>
 2026-08-24 21:33:36 +00:00
Wentao Ye 811d12ee26 [Mypy Fix] Mypy fix for "vllm/model_executor/models/[eE][fF]" (#53381) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-08-22 15:41:27 -04:00
Lucas Wilkinson 8bdc70ec7b [6/N][KV-Cache Layout Refactor] Standardize KV cache layout (#51718) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
 2026-08-21 17:47:49 -07:00
Kevin H. Luu c0ff33404b Revert compile-cache device index regression on CPU (#53304) 2026-08-21 16:08:13 -04:00
Frederik Gossen 47cd1c8885 [Core] Add dynamo_timed tracing for print_readable (#40834) 
Signed-off-by: Frederik Gossen <frgossen@meta.com>
 2026-08-21 09:07:21 -07:00
Neil SchemenauerandTyler Michael Smith 6dcc5d7cae [Bugfix] Include device index in compile cache paths (#38962) 
Signed-off-by: Neil Schemenauer <nas@arctrix.com>
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
 2026-08-21 09:07:16 -07:00
shanjiaz e078a2238a [Bugfix] Widen flashinfer.comm import guard so a failed import doesn't abort engine startup (#52241) 
Signed-off-by: shanjiaz <zsjwpianpian@gmail.com>
 2026-08-14 16:27:24 +00:00
Andreas Karatzas 7e724dca89 [ROCm] Fix AITER all-reduce fusion coverage (#50802) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
 2026-08-06 14:52:23 +08:00
Rohan PotdarandClaude Opus 4.8 d0ce3dadb6 [ROCm]: Bump torch 2.12, triton 3.7, torchaudio, torchvision (#50607) 
Signed-off-by: Rohan Potdar <rohan.potdar@amd.com>
Signed-off-by: Rohan138 <rohanpotdar138@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-08-04 20:14:11 -05:00
skysnow2001andDouglas Lehr f43e1d26e3 [ROCm] Enable AITER and FP8 inference on GFX120x (#43615) 
Signed-off-by: skysnow2001 <skysnow9285@gmail.com>
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com>
 2026-08-03 16:45:37 -05:00
Wentao Ye 0cf49a5d15 [Refactor] Remove multiple dead codes (#50285) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-08-03 10:59:08 -04:00
Richard ZouandCodex c27b080d26 [compile] Fix fake kernel return dtype (#50444) 
Signed-off-by: Richard Zou <zou3519@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
 2026-07-30 19:50:02 +00:00
BadrBasowid 59e831c09a [Compilation]Fuse Transformers Residual Add + RMSNorm (#48757) 
Signed-off-by: BadrBasowid <badr.basowid@gmail.com>
 2026-07-30 13:46:36 +00:00
zaristeiSahithi Chigurupatihlu1
 		5fa0154448 [Rubin] Enable NVLink all-reduce paths on SM107 (#49647) 
Signed-off-by: Zachary Aristei <zaristei@nvidia.com>
Co-authored-by: Sahithi Chigurupati <chigurupati.sahithi@gmail.com>
Co-authored-by: hlu1 <14827759+hlu1@users.noreply.github.com>
 2026-07-29 20:53:25 +00:00
afriedri f472ab0a4c Remove triton per group quant [ROCm] [Bugfix] (#49621) 
Signed-off-by: Andy Friedrich <afriedri@amd.com>
 2026-07-28 05:53:46 +00:00
Chaojun ZhangandKunshang Ji bf4f633b4c [XPU] Enable QK Norm + RoPE fusion pass on XPU (#49394) 
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
 2026-07-27 03:22:29 +00:00
Schwinn Saereesitthipitak b5b61c622c [Core][Distributed] Add process-checkpoint lifecycle hooks for communicators (starting with Flashinfer) (#46877) 
Signed-off-by: Schwinn Saereesitthipitak <schwinns@nvidia.com>
 2026-07-26 14:47:50 -04:00
Woosuk Kwon 213f681f81 Revert "[Perf][GLM-5.2] Blackwell decode optimizations" (#49768) 2026-07-24 16:04:59 -07:00
xiaozhoupyClaudeGiancarlo DelfinThien TranOpenAI Codexzhoustefankoncarevic
 		dd72658e7d [Perf][GLM-5.2] Blackwell decode optimizations (#48597) 
Signed-off-by: Peiyuan Zhou <peiyuanzhou1994@gmail.com>
Signed-off-by: zhou <zhou@zhoudeMacBook-Pro.local>
Signed-off-by: Stefan Koncarevic <stefan.koncarevic@amd.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Giancarlo Delfin <32987265+TheEpicDolphin@users.noreply.github.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: zhou <zhou@zhoudeMacBook-Pro.local>
Co-authored-by: stefankoncarevic <Stefan.Koncarevic@amd.com>
 2026-07-23 21:36:27 -07:00
stefankoncarevicandAndreas Karatzas 387189c429 [ROCm] Remove redundant AITER fused_qk_rmsnorm probe (avoids config-time HIP init) (#47992) 
Signed-off-by: Stefan Koncarevic <stefan.koncarevic@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
 2026-07-21 22:58:14 -05:00