iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
18,130 Commits 3 Branches 0 Tags
68ee8300a047db78fb52bac477daaaac7be11216
Commit Graph
1115 Commits
Author SHA1 Message Date
Brandon PelfreyandRoger Wang af16446bf3 Vram semaphore infra (#44465) 
Signed-off-by: Brandon Pelfrey <bpelfrey@nvidia.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2026-06-26 17:32:51 -07:00
Ting SUNandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> f1a6703edd [Bugfix][Config] Keep pydantic validation for fields with a TYPE_CHECKING Literal alias (#46220) 
Signed-off-by: Ting Sun <suntcrick@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-24 12:25:50 +00:00
Roberto L. Castro d86c66c981 [Feat] Add runtime monitor for post-warmup CuTeDSL compilation (#46167) 2026-06-23 23:33:17 +00:00
Tyler Michael SmithClaudeCodexmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>kourosh hakhamaneshi
 		ebfbcfe46a Stop setting CUDA_VISIBLE_DEVICES internally in vLLM, add device_ids arg (#45026) 
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: kourosh hakhamaneshi <kouroshHakha@users.noreply.github.com>
 2026-06-20 13:38:10 -07:00
Ting SUN 9d4b87f4f0 [Bugfix][Model] Validate DefaultModelLoader / LoadConfig and fail with clear errors (#45196) 
Signed-off-by: Ting Sun <suntcrick@gmail.com>
 2026-06-17 21:46:33 +00:00
Nick Hill d8d95998dc [Core] Add prefill step cadence for better non-PD DP balancing (#44558) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-06-16 13:17:18 -07:00
Thien Tran b2cfae777d Add Triton recompile detection (#45631) 
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
 2026-06-16 18:25:28 +08:00
Isotr0py 6635279d8a [Migration] Migrate GGUF quantization support to plugin (#39612) 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2026-06-12 12:02:21 -07:00
Lucas WilkinsonMartin KuklaMatthew BonanniDipika SikkaNickLucchejiahancAlec KohlhoffPorras HuangClaudescoootscooob
 		eb28452b10 [Model] Add DiffusionGemma Support (#45163) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Martin Kukla <martin.kukla@cantab.net>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Dipika Sikka <dsikka@redhat.com>
Co-authored-by: NickLucche <nlucches@redhat.com>
Co-authored-by: jiahanc <173873397+jiahanc@users.noreply.github.com>
Co-authored-by: Alec Kohlhoff <134344302+aleckohlhoff@users.noreply.github.com>
Co-authored-by: Porras Huang <20535584+porrashuang@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: scoootscooob <167050519+scoootscooob@users.noreply.github.com>
 2026-06-11 22:17:35 -07:00
Nick HillandClaude Opus 4.8 4085ff7cb4 [Core] Add kvcache watermark to reduce preemptions (#44594) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-11 08:27:31 -07:00
Injae Ryou 165460941f [BugFix] HFValidationError with cloud storage URIs when HF_HUB_OFFLINE=1 (#39155) 
Signed-off-by: Injae Ryou <injaeryou@gmail.com>
 2026-05-27 10:53:32 -05:00
Thien Tran d56612c621 [GDN] GDN Prefill kernel for SM100 (#43273) 
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
 2026-05-26 14:02:11 +08:00
Ashwin Giridharanandabinggo 84e351555a [Bugfix] Auto-raise max_num_batched_tokens for prefix-LM multimodal models (#43051) 
Signed-off-by: Ashwin Giridharan <girida@amazon.com>
Co-authored-by: abinggo <107740309+abinggo@users.noreply.github.com>
 2026-05-22 21:23:50 -07:00
Wentao Yerobertgshaw2-redhatRobert ShawNick Hill
 		0ddd7dd656 [Frontend] DP Supervisor (#40841) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Robert Shaw <robertgshaw2@gmail.com>
Signed-off-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: robertgshaw2-redhat <robertgshaw2@gmail.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-05-21 20:33:16 -07:00
fangyuchuzWaNg3mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		e26e1f0928 [Feature] Add --cpu-distributed-timeout-seconds CLI Option for CPU Process Group Timeout (#42968) 
Signed-off-by: fangyuchu <fangyuchu@qq.com>
Signed-off-by: zWaNg3 <389750525@qq.com>
Co-authored-by: zWaNg3 <389750525@qq.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-21 15:42:07 -07:00
Kebe 19cf334207 [Feature] Support manually enabling the cumem allocator (#33648) 
Signed-off-by: Kebe <mail@kebe7jun.com>
 2026-05-20 08:58:30 -04:00
Michael GoinandClaude 239b5ff30c [Frontend] Add --spec-method/--spec-model/--spec-tokens CLI aliases (#42476) 
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-05-18 17:22:27 -07:00
Jee Jee Ligemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>Roger Wang
 		7d5b033782 [LoRA] Support 2D and 3D MoE LoRA adapter at the same time (#42242) 
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2026-05-18 15:22:26 +08:00
Michael Goin b2a27b82d9 [Kernel][UX] Add --linear-backend arg for linear kernel selection (#39538) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2026-05-15 17:07:39 -07:00
PatchyTISandPatchouliTaisa 0162596603 [Model Runner V2] FP32 gumbel sampling. (#41775) 
Signed-off-by: PatchouliTaisa <patchychen@tencent.com>
Co-authored-by: PatchouliTaisa <patchychen@tencent.com>
 2026-05-15 09:20:08 -07:00
Michael Goin 8efd508204 [Quantization] Rework quantization_config to use QuantKey and allow for activation override (#41566) 2026-05-13 16:58:32 -04:00
CynicDora 256dbcaabf [Feature] Support custom callable proposer backend for speculative decoding (#39487) 
Signed-off-by: 524031910363 <hyzhyzsh@sjtu.edu.cn>
Signed-off-by: CynicDora <hyzhyzsh@sjtu.edu.cn>
 2026-05-13 16:53:01 +00:00
Mohammad Miadh Angkad 21943d4c25 [Performance] Make safetensors checkpoint prefetch settings configurable (#41499) 
Signed-off-by: Mohammad Miadh Angkad <MAngkad.BSDSBA2027@aim.edu>
 2026-05-10 15:55:15 +00:00
Dao007forever 3f5bd482f5 [Bugfix][KV Transfer][NIXL] Notify P node on pre-admission rejection to free stranded KV blocks (#41269) 2026-05-09 22:52:09 -07:00
Christian VanandChristian Van 0d382ecde8 Handle optional bool-or-string CLI args in get_kwargs (#40951) 
Signed-off-by: Christian Van <cvan20191@gmail.com>
Co-authored-by: Christian Van <cvan20191@gmail.com>
 2026-05-09 19:47:21 -07:00
Sumanth R Hegde e3b65a5ba0 [feat] Add explicit /start_weight_update and /finish_weight_update APIs for weight transfer (#39212) 2026-05-08 18:03:33 -07:00
JartXJim SmithSandermageClaude
 		4f2af1a7c0 [Feature] TurboQuant: support hybrid models and uniform quantization (#39931) 
Signed-off-by: JartX <sagformas@epdcenter.es>
Signed-off-by: Jim Smith <jhsmith0@me.com>
Co-authored-by: Jim Smith <jhsmith0@me.com>
Co-authored-by: Sandermage <sandermage@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-05-04 20:14:01 -04:00
Wentao Ye 577b9623e6 [Bug] Fix status update address for non-MOE model within external dp mode (#40839) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-05-04 16:37:16 -07:00
Bugen Zhaoandwang.yuqi a07642667d [Bugfix] Pass reasoning parser kwargs to structured output (#41199) 
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: wang.yuqi <yuqi.wang@daocloud.io>
 2026-04-30 23:38:02 -07:00
Zhanda ZhuandYubo Gao 5d5c776444 [Perf] FP8 FlashInfer Attn for ViT (#38065) 
Signed-off-by: Zhanda Zhu <zhandazhu@gmail.com>
Co-authored-by: Yubo Gao <ybgao-nvidia@users.noreply.github.com>
 2026-04-27 13:44:15 +08:00
Nicolò Lucchesi 9c2492e501 [Misc] Support Human-readable (k/K/m/M..) json cli arg (#40473) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2026-04-23 09:42:23 +02:00
Wentao Yeandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> 8f87eb4622 [Refactor] Clean up log once scope="local" (#40540) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-04-22 16:42:43 -04:00
Vibhav AgarwalXinyu ChenMichael Goin
 		f4b42df048 [Attention Backend] TurboQuant: 2-bit KV cache compression with 4x capacity (#38479) 
Signed-off-by: vibhavagarwal5 <vibhavagarwal5@gmail.com>
Signed-off-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Xinyu Chen <xinyu1.chen@intel.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
 2026-04-14 19:57:13 -07:00
roikoren755 ecf5ff7ce3 [Mamba] Flashinfer selective_state_update (#36162) 
Signed-off-by: Roi Koren <roik@nvidia.com>
 2026-04-14 15:10:58 -04:00
bnellnmandRobert Shaw e1e318af01 [MoE Refactor] Remove MoE DP chunking (#39107) 
Signed-off-by: Bill Nell <bnell@redhat.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
 2026-04-14 09:48:05 -04:00
Chauncey ecbfbb8d61 [Feature] Add auto-detection for reasoning_config when only reasoning_parser is set (#38214) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
 2026-04-10 01:36:26 +00:00
wang.yuqi 66c079ae83 [Frontend][4/n] Improve pooling entrypoints | pooling. (#39153) 
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
 2026-04-09 10:09:45 +00:00
Shengqi ChenJason LiRoger Wang
 		75e01a39a1 [Feature] NUMA binding support for GPU workers (#38635) 
Signed-off-by: Shengqi Chen <harry-chen@outlook.com>
Co-authored-by: Jason Li <jasonlizhengjian@gmail.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2026-04-08 09:55:24 -07:00
Vasiliy Kuznetsov 7b1a7423be [Frontend] new online quantization frontend (#38138) 
Signed-off-by: Vasiliy Kuznetsov <vasiliy@meta.com>
 2026-04-03 11:58:39 -04:00
Luka GovedičXinyu ChenChaojun ZhangLuka Govedič
 		40bb175027 [vLLM IR] 1/N Implement IR skeleton and rms_norm op (#33825) 
Signed-off-by: Luka Govedič <lgovedic@redhat.com>
Signed-off-by: Xinyu Chen <xinyu1.chen@intel.com>
Signed-off-by: chzhang <chaojun.zhang@intel.com>
Signed-off-by: Luka Govedic <luka.govedic@gmail.com>
Co-authored-by: Xinyu Chen <xinyu1.chen@intel.com>
Co-authored-by: Chaojun Zhang <chaojun.zhang@intel.com>
Co-authored-by: Luka Govedič <ProExpertProg@h100-01.nemg-001.lab.rdu2.dc.redhat.com>
 2026-03-31 22:15:05 -04:00
roikoren755 8e6293e838 [Mamba] Add stochastic rounding support (#35753) 
Signed-off-by: Roi Koren <roik@nvidia.com>
 2026-03-30 12:33:49 -04:00
Michael GoinandClaude 7ba425e916 Add short flag -sc for --speculative-config argument (#38380) 
Co-authored-by: Claude <noreply@anthropic.com>
 2026-03-27 12:04:22 -07:00
Jonas M. Kübler 98e7f223b9 enable skipping of SW attention layers when using FP8 KV cache (#33695) 
Signed-off-by: Jonas Kuebler <kuebj@amazon.com>
 2026-03-27 07:25:02 -06:00
BviciiandCyrus Leung 999dfc1622 [Bugfix] Offload blocking tokenizer ops to shared thread pool to unblock event loop (#34789) 
Signed-off-by: Bvicii <yizhanhuang2002@gmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2026-03-26 22:17:00 -07:00
Cyrus Leung ba2f0acc2d [Misc] Reorganize inputs (#35182) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-03-25 10:22:54 -07:00
Harry Mellor d215d1efca [Mypy] Better fixes for the mypy issues in vllm/config (#37902) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-03-25 06:14:43 -07:00
Artem Perevedentsev a93a53f8a1 [Performance] Auto-enable prefetch on NFS with RAM guard (#37673) 
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
 2026-03-24 17:31:14 -07:00
Javier De Jesus 54b0578ada [Bugfix] Pass hf_token through config loading paths for gated model support (#37920) 
Signed-off-by: javierdejesusda <javier.dejesusj9@gmail.com>
 2026-03-24 15:22:05 -04:00
Dan BlanaruandClaude dc78c2c933 [Core] add option to schedule requests based on full ISL (#37307) 
Signed-off-by: Dan Blanaru <48605845+DanBlanaru@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-03-24 13:01:12 -04:00
Sungjae LeeChaunceyHarry Mellor
 		4731884796 [Feature] limit thinking tokens (hard limit) (#20859) 
Signed-off-by: Sungjae Lee <33976427+llsj14@users.noreply.github.com>
Signed-off-by: Sungjae Lee <sung-jae.lee@navercorp.com>
Signed-off-by: Chauncey <chaunceyjiang@gmail.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-03-24 09:53:07 -07:00