iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity
16,919 Commits 3 Branches 0 Tags
1806d1adfc9b598bc6eb94de38a330aaad04c291
Commit Graph
750 Commits
Author SHA1 Message Date
Benjamin Chislettandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> 4e2eba28be [Perf] Optimize hidden state extraction logic (#37374) 
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com>
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-05-22 18:23:08 -04:00
Nick Hill 47d4407d7c [Model Runner V2] Support sharing kv cache layers (#35045) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-05-22 22:18:23 +00:00
Ilya MarkovandMarkov Ilya d3a563501b [EPLB] Change default EPLB communicator (#43110) 
Signed-off-by: Markov Ilya <markovilya19@gmail.com>
Co-authored-by: Markov Ilya <markovilya19@gmail.com>
 2026-05-22 09:43:27 -04:00
wangxiyuan 7e1b45a092 [Attention] Mamba attention module refactor (#41126) 
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
 2026-05-22 17:13:12 +08:00
fangyuchuzWaNg3mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		e26e1f0928 [Feature] Add --cpu-distributed-timeout-seconds CLI Option for CPU Process Group Timeout (#42968) 
Signed-off-by: fangyuchu <fangyuchu@qq.com>
Signed-off-by: zWaNg3 <389750525@qq.com>
Co-authored-by: zWaNg3 <389750525@qq.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-21 15:42:07 -07:00
Nick Hill 0f66623b0d [Frontend] Rework fastokens integration (#43168) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-05-21 15:36:58 -07:00
Zheng LuoOpenAI CodexRobert Shaw
 		17b69828a0 [Core] Add native ModelExpress load format (#43105) 
Signed-off-by: Zheng Luo <zheluo@nvidia.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
 2026-05-21 16:05:01 -04:00
anishanishClaude Opus 4.6Wentao Ye
 		1c78f76c29 [Bugfix] Add early validation to reject incompatible runner types for embedding models (#43079) 
Signed-off-by: anish <anishesg@users.noreply.github.com>
Signed-off-by: Your Name <ak8686@princeton.edu>
Signed-off-by: anish <145943060+anishesg@users.noreply.github.com>
Co-authored-by: anish <anishesg@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
 2026-05-21 11:07:46 -04:00
Daoyuan Li a6682d1d25 [Bugfix] Warn when renderer_num_workers has no effect on offline LLM (#42905) 
Signed-off-by: Daoyuan Li <94409450+DaoyuanLi2816@users.noreply.github.com>
 2026-05-20 21:35:08 -07:00
meena-at-workandClaude Sonnet 4.6 363fc84407 Integrate flashinfer b12x MoE and FP4 GEMM kernels for SM120/121 (#40082) 
Signed-off-by: Meenakshi Venkataraman <meenakshiv@nvidia.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-05-20 17:21:11 +00:00
Kebe 19cf334207 [Feature] Support manually enabling the cumem allocator (#33648) 
Signed-off-by: Kebe <mail@kebe7jun.com>
 2026-05-20 08:58:30 -04:00
Aaron Hao 73dd2f33b7 [bug] fix WeightTransferConfig.backend to allow for all strings (#43121) 
Signed-off-by: ahao-anyscale <ahao@anyscale.com>
 2026-05-19 21:01:29 -04:00
Nicolò LucchesiandNick Hill fba010dd74 [Bugfix][MRV2] Fix KVCache tensor explicit kernel_block_size dim (#42766) 
Signed-off-by: NickLucche <nlucches@redhat.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-05-18 20:25:41 -07:00
shanjiaz a171e6b52d Add parallel drafting to v2 model runner unsupported features (#43010) 
Signed-off-by: shanjiaz <zsjwpianpian@gmail.com>
 2026-05-18 16:39:09 -07:00
Wei Zhao 8c296de63b [Perf] Re-enable flashinfer autotune by default and cleanup (#42857) 
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
 2026-05-18 09:12:27 -07:00
Harry Mellor b12745e4f3 Fix --convert passed without --runner on causal models (#42935) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-05-18 15:56:09 +00:00
Nicolò Lucchesi 69c91d010a [MRv2] Default to MRv1 when a connector is present (#42955) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2026-05-18 20:34:16 +08:00
Jee Jee Ligemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>Roger Wang
 		7d5b033782 [LoRA] Support 2D and 3D MoE LoRA adapter at the same time (#42242) 
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2026-05-18 15:22:26 +08:00
Qi Zhou 504a26ce2b Support bf16 for mamba ssm cache (#41680) 
Signed-off-by: Qi Zhou <qizzzh@google.com>
 2026-05-16 17:54:58 -07:00
Jiangyun Zhu 8a56da3845 [Experimental] Breakable CUDA graph (#42304) 
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
 2026-05-16 22:04:12 +08:00
Michael Goin b2a27b82d9 [Kernel][UX] Add --linear-backend arg for linear kernel selection (#39538) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2026-05-15 17:07:39 -07:00
PatchyTISandPatchouliTaisa 0162596603 [Model Runner V2] FP32 gumbel sampling. (#41775) 
Signed-off-by: PatchouliTaisa <patchychen@tencent.com>
Co-authored-by: PatchouliTaisa <patchychen@tencent.com>
 2026-05-15 09:20:08 -07:00
BadrBasowid fb5bd03f51 [Perf] Set IR Op Priority Once at Worker Init (#42631) 
Signed-off-by: BadrBasowid <badr.basowid@gmail.com>
 2026-05-15 15:56:13 +00:00
Isotr0pyandzxy faa4b76afa [Model] Support InternS2 Preview (#42705) 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Co-authored-by: zxy <46674730+CUHKSZzxy@users.noreply.github.com>
 2026-05-14 21:30:26 -07:00
Chengyi NieChengyi NieCursor
 		fa2a33b893 [Quant] Consolidate GPTQ: rename gptq_marlin.py to auto_gptq.py (#38288) 
Signed-off-by: Chengyi Nie <cnie@roblox.com>
Co-authored-by: Chengyi Nie <cnie@roblox.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-05-15 08:25:52 +08:00
Matthew Bonanni 9898f94abe [Attention] Remove deprecated MLA prefill arguments (#42555) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
 2026-05-14 10:34:06 -07:00
Wentao YeandNick Hill ae4f59f0ec [Model Runner v2] Oracle for model runner v2 - qwen3 dense model by default [1/N] (#39337) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-05-14 10:02:33 -07:00
Baorun (Lauren) Mu a7737cb4f3 [Fix] Misc Fixes in ViT CUDA Graph (#38040) 
Signed-off-by: Baorun Mu <bmu@nvidia.com>
 2026-05-14 23:49:06 +08:00
Hongxin XuClaude Opus 4.6arlenxuJunjie Zhang
 		c7560af424 [RFC] Replace shared-memory routed experts with ModelRunnerOutput transfer and HTTP support (#39568) 
Signed-off-by: xhx1022 <1737006628@qq.com>
Signed-off-by: arlenxu <arlenxu@tencent.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: arlenxu <arlenxu@tencent.com>
Co-authored-by: Junjie Zhang <junj.jay.zhang@gmail.com>
 2026-05-14 14:12:30 +00:00
aoshen02 8c79ad6580 Revert "[Core] Replace routing replay with device cache and async D2H pipeline" (#39917) (#42434) 
Signed-off-by: aoshen02 <aoshen@inferact.ai>
 2026-05-13 23:49:01 -07:00
Siddharth Bedekar f51f6844f9 [Bugfix][Spec Decode] Wire draft_probs into probabilistic draft_model rejection (#40269) 2026-05-13 21:04:03 -04:00
liangel-02 6b5c389ee3 expose flex block size for batch invariant mode (#41252) 
Signed-off-by: Angel Li <liangel@meta.com>
 2026-05-13 14:11:57 -07:00
Michael Goin 8efd508204 [Quantization] Rework quantization_config to use QuantKey and allow for activation override (#41566) 2026-05-13 16:58:32 -04:00
Fynn Schmitt-Ulms ab1ad0d7a9 Remove verifier model type check in speculative config (#42536) 
Signed-off-by: Fynn Schmitt-Ulms <fschmitt@redhat.com>
 2026-05-13 18:14:39 +00:00
Michael GoinandClaude Opus 4.6 2f821faeae [Spec Decode] Support hybrid attention models in extract_hidden_states (#39949) 
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-13 10:45:53 -07:00
CynicDora 256dbcaabf [Feature] Support custom callable proposer backend for speculative decoding (#39487) 
Signed-off-by: 524031910363 <hyzhyzsh@sjtu.edu.cn>
Signed-off-by: CynicDora <hyzhyzsh@sjtu.edu.cn>
 2026-05-13 16:53:01 +00:00
Viktor PusCursorWentao Ye
 		0ce6613b9c platforms: add uses_cpu_device() hook to Platform for DeviceConfig (#42313) 
Signed-off-by: Viktor Pus <viktorpus@tenstorrent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
 2026-05-12 12:39:17 -07:00
Thomas ParnellandClaude Opus 4.6 8f89381fc6 [Hybrid] Warmup Mamba2 SSD kernel (#39822) 
Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-05-12 12:46:22 +00:00
Rohan PotdarandElizaWszola a51376b3f0 [Performance][DSR1]: Fused RoPE+KVCache+q_concat for MLA (#40392) 
Signed-off-by: Rohan138 <rohanpotdar138@gmail.com>
Signed-off-by: Rohan Potdar <66227218+Rohan138@users.noreply.github.com>
Co-authored-by: ElizaWszola <ewszola@redhat.com>
 2026-05-11 14:10:50 +00:00
Mohammad Miadh Angkad 21943d4c25 [Performance] Make safetensors checkpoint prefetch settings configurable (#41499) 
Signed-off-by: Mohammad Miadh Angkad <MAngkad.BSDSBA2027@aim.edu>
 2026-05-10 15:55:15 +00:00
Lucas Wilkinson b1728c1e66 [Attention][Cleanup] Remove tree attention (#42121) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
 2026-05-08 18:36:19 -07:00
Roy Wang 60851b1d22 [Bugfix][KV Transfer] Reject NixlConnector + expandable_segments:True (#41237) 2026-05-08 16:47:33 -07:00
MeganEFlynnandClaude Opus 4.6 969fbfb4a9 Laguna xs dflash support (#41880) 
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-05-07 13:31:16 -07:00
TomerBN-Nvidia 8189a15914 [Core] Replace routing replay with device cache and async D2H pipeline (#39917) 
Signed-off-by: Tomer Barnatan <tbarnatan@nvidia.com>
 2026-05-07 11:24:56 -07:00
zhrrr 7a08b34fbf [Model Runner V2] support qwen35 / mamba hybrid model (#35520) 
Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com>
 2026-05-07 09:31:05 -07:00
AlonKejzmanandRoger Wang 2a16ece2d3 tokenizer: Add fastokens support (#41741) 
Signed-off-by: AlonKejzman <alonkeizman@gmail.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2026-05-07 22:49:42 +08:00
Luciano MartinsandLuciano Martins 27e0057aed [Spec Decode] Add Gemma4 MTP speculative decoding support (#41745) 
Signed-off-by: Luciano Martins <lucianommartins@users.noreply.github.com>
Co-authored-by: Luciano Martins <lucianommartins@users.noreply.github.com>
 2026-05-06 22:39:29 +08:00
Lanze Liu 79246b5ea6 [Spec Decode] Fix max_model_len logging in speculative config for draft model (#41571) 
Signed-off-by: Lanze Liu <lanzetech@gmail.com>
 2026-05-05 21:56:06 +00:00
Hexiang Wangganyitjtanaatjtanaavllm
 		628c436301 [New Model][ROCm] Add AMD support for DeepSeek V4 (#40871) 
Signed-off-by: ganyi <ygan@amd.com>
Signed-off-by: whx-sjtu <xiaowang990929@gmail.com>
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
Signed-off-by: tjtanaavllm <tunjian.tan@amd.com>
Co-authored-by: ganyi <ygan@amd.com>
Co-authored-by: tjtanaa <tunjian.tan@embeddedllm.com>
Co-authored-by: tjtanaavllm <tunjian.tan@amd.com>
 2026-05-05 08:55:37 -07:00
Wentao Ye 577b9623e6 [Bug] Fix status update address for non-MOE model within external dp mode (#40839) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-05-04 16:37:16 -07:00