iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
19,362 Commits 3 Branches 0 Tags
f1899b2ffdf87598350c416a074c8c291c845902
Commit Graph
108 Commits
Author SHA1 Message Date
Giancarlo Delfin dec13a33b7 [Model Runner V2][Spec Decode] Add multi-layer MTP speculator (#48892) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-07-30 15:54:15 -07:00
+13
Jiangyun ZhuIsotr0pyThien TranBugen ZhaoYongye ZhuZiming HuangRoger WangIsotr0pyaoshen02Woosuk KwonJee Jee Liaoshen02Summer YangKevin H. LuuBowen WanggnovackNick HillYifan QiaoxiaozhoupyRoy WangJeff MaCodexWei Zhao
 		aeeb36b1f1 [New model] Kimi K3 (#50000) 
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Ziming Huang <zelda.huanghuang@gmail.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Isotr0py <mozf@inferact.ai>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Summer Yang <girasoleyang@gmail.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
Co-authored-by: Bowen Wang <abmfy@icloud.com>
Co-authored-by: gnovack <novackgm@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: xiaozhoupy <peiyuanzhou1994@gmail.com>
Co-authored-by: Roy Wang <yasong.wang@inferact.ai>
Co-authored-by: Jeff (Junze) Ma <93145857+majunze2001@users.noreply.github.com>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com>
 2026-07-30 03:49:58 -07:00
Giancarlo Delfin 272abd5f48 [Tests][Spec Decode] Add gemma4 MTP acceptance rates test (#47920) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-07-27 16:15:54 -07:00
Nick HillandWoosuk Kwon 8112b6c997 [MRV2] Always build attn metadata at capture time (#49364) (#49995) 
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
 2026-07-27 17:25:17 +00:00
Woosuk Kwon 213f681f81 Revert "[Perf][GLM-5.2] Blackwell decode optimizations" (#49768) 2026-07-24 16:04:59 -07:00
xiaozhoupyClaudeGiancarlo DelfinThien TranOpenAI Codexzhoustefankoncarevic
 		dd72658e7d [Perf][GLM-5.2] Blackwell decode optimizations (#48597) 
Signed-off-by: Peiyuan Zhou <peiyuanzhou1994@gmail.com>
Signed-off-by: zhou <zhou@zhoudeMacBook-Pro.local>
Signed-off-by: Stefan Koncarevic <stefan.koncarevic@amd.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Giancarlo Delfin <32987265+TheEpicDolphin@users.noreply.github.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: zhou <zhou@zhoudeMacBook-Pro.local>
Co-authored-by: stefankoncarevic <Stefan.Koncarevic@amd.com>
 2026-07-23 21:36:27 -07:00
vllm-agentandvllm-agent CI bot 494845e79f Revert "[MRV2] Always build attn metadata at capture time" (#49364) (#49451) 
Co-authored-by: vllm-agent CI bot <ci-bot@vllm-agent.local>
 2026-07-23 09:51:33 -07:00
Michael GoinandNick Hill ac36a7a1e7 [MRV2][Spec Decode] Avoid rejection sampler OOM by chunking (#48630) 
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-07-23 18:13:13 +08:00
Woosuk Kwon 6049424b7e [MRV2] Always build attn metadata at capture time (#49364) 
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
 2026-07-21 19:27:16 -07:00
Michael GoinandCodex a7d00ec051 [Bugfix] DFlash fc sized wrong when num_target_layers != num_hidden_layers (#48524) 
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
 2026-07-21 14:42:52 -07:00
Fynn Schmitt-UlmsandMichael Goin 642076d26c Support loading sample_from_anchor flag from speculators config (#48639) 
Signed-off-by: Fynn Schmitt-Ulms <fschmitt@redhat.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
 2026-07-20 14:52:43 -07:00
Oxana KorzhCursorokorzh-amdMatthew Bonanni
 		4ec199b66a [Bugfix][Spec-Decode] Populate draft seq_lens_cpu_upper_bound for spec-decode attention metadata (#44492) 
Signed-off-by: Oxana Korzh <okorzh@amd.com>
Signed-off-by: okorzh-amd <okorzh-amd@users.noreply.github.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: okorzh-amd <okorzh-amd@users.noreply.github.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
 2026-07-20 20:50:58 +00:00
aoshen02Wojciech WaisFederico Kamelhar
 		41ea2dd44a [Bugfix][V1/V2] Fix prompt_logprobs to respect logprobs_mode (#47680) 
Signed-off-by: Wojciech Wais <wojciech.wais@gmail.com>
Signed-off-by: Federico Kamelhar <209537060+fede-kamel@users.noreply.github.com>
Signed-off-by: Allen Shen <aoshen@inferact.ai>
Co-authored-by: Wojciech Wais <wojciech.wais@gmail.com>
Co-authored-by: Federico Kamelhar <209537060+fede-kamel@users.noreply.github.com>
 2026-07-17 21:58:59 +01:00
Woosuk KwonBugen ZhaoGiancarlo DelfinIsotr0pyIsotr0pyJee Jee LiRoger WangYifan QiaoClaude Fable 5OpenAI Codex
 		251f7e478e [Model] Add PW CUDA graph support for Inkling [2/N] (#48822) 
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Giancarlo Delfin <32987265+TheEpicDolphin@users.noreply.github.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Isotr0py <mozf@inferact.ai>
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-07-16 10:28:56 -07:00
Michael GoinandClaude Opus 4.8 8bfd683901 [Spec Decode] Add kv_cache_dtype to speculative_config to control separately from target (#48787) 
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-07-15 23:49:23 -06:00
Michael GoinandClaude Fable 5 ecf4aa5ce2 [Bugfix] Fix FlashInfer non-causal draft attention (DFlash/DSpark) on Blackwell (#48167) 
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-15 12:44:01 -07:00
Nick Hill 26587f9519 [BugFix][ModelRunner V2] Fix stale attn metadata in speculator prefill cudagraph capture (#48261) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-07-13 09:39:15 -07:00
Michael GoinandClaude Opus 4.8 0d12618e98 [Spec Decode] Support hybrid (SWA + full attention) DFlash drafters (#47914) 
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-07-08 11:12:45 -04:00
Benjamin Chislett 7bd154375d [Bugfix] Fix mamba+dflash for MRV2 (#47698) 2026-07-07 15:59:13 -04:00
Summer Yang d3e69fd671 [Perf] Use blocking CUDA events to avoid busy polling cuda driver lock (#47081) 
Signed-off-by: Jingyi Yang <girasoleyang@gmail.com>
 2026-07-07 09:36:10 +00:00
Kunshang JiandHarry Mellor 07f9baf756 Revert "[Platform] Replace torch.cuda.Event with torch.Event (#47140)" (#47668) 
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-07-06 14:18:33 +01:00
Woosuk KwonandClaude Fable 5 a47f38f825 [Bugfix][Model Runner V2][Spec Decode] Fix int32 offset overflow in block verification kernels (#47383) 
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-02 07:32:38 -07:00
Michael Goin 2b753ad200 [Spec Decode] DSpark speculators checkpoint support (#47093) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2026-07-01 17:32:27 -07:00
Benjamin ChislettGiancarlo Delfinmgoin
 		f5a8d73377 [Spec Decode] DSpark (#46995) 
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com>
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: mgoin <mgoin64@gmail.com>
 2026-07-01 08:30:24 -07:00
Giancarlo Delfin 3cecee40f3 [Model Runner V2][Spec Decode] Fix stale values in idx_mapping from CG num reqs padding (#47066) 2026-06-30 11:25:32 -07:00
Kunshang JiandHarry Mellor e840f0d3f5 [Platform] Replace torch.cuda.Event with torch.Event (#47140) 
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-06-30 08:39:59 -07:00
Giancarlo Delfin db808b3961 [Model Runner V2][Spec Decode] Implement block verification for rejection sampling (#46781) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-06-30 08:07:24 -07:00
Kunshang Ji fb42e5219e [Platform] Replace torch.cuda.mem_get_info with torch.accelerator.get_memory_info (#44825) 
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Signed-off-by: Kunshang Ji <jikunshang95@gmail.com>
 2026-06-30 14:39:52 +08:00
Ilya MarkovandMarkov Ilya a2abce646f [EPLB] Mask padding in EPLB load recording (#38128) 
Signed-off-by: ilmarkov <markovilya197@gmail.com>
Signed-off-by: Markov Ilya <markovilya19@gmail.com>
Co-authored-by: Markov Ilya <markovilya19@gmail.com>
 2026-06-28 19:43:58 -07:00
Woosuk Kwon 0472436541 [Spec Decode] Avoid redundant hidden-states gather in draft prefill (#46968) 2026-06-28 17:04:01 -07:00
Chauncey 11a12305c0 [Model Runner V2][Spec Decode] Handle tuple hidden states from MTP draft models (#46786) 2026-06-27 18:38:07 -07:00
Giancarlo Delfin b6caeb5a09 [Model Runner V2][Spec Decode] Use fp32 uniform threshold for acceptance (#46878) 2026-06-27 14:09:25 -07:00
yihengandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 652d962bc9 [Model Runner V2][Spec Decode] Reduce TP communication for draft token generation (#46448) 
Signed-off-by: EanWang211123 <wangyiheng@sangfor.com.cn>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-25 19:30:07 -07:00
Giancarlo Delfin 5314665bad [Model Runner V2][DFlash] Enable dflash attention backend selection (#46770) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-06-25 19:29:25 -07:00
Giancarlo Delfin c53994e134 [Model Runner V2][Spec Decode] Use log1p to compute residual during rejection sampling (#46665) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-06-25 23:46:10 +00:00
JessieWeiClaude Opus 4.8Robert Shaw
 		3c43237233 [Bugfix][Model Runner V2][Spec Decode] Fix int32 offset overflow in sampler kernels (#46560) 
Signed-off-by: xiaojun.wei <jessiewei747@gmail.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
 2026-06-24 11:00:57 -07:00
Nick Hill 7ee4d22009 [Spec Decode] Reject placeholder (-1) draft tokens in rejection sampler (#46533) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-06-24 03:32:32 +00:00
Giancarlo Delfin ef361de916 [Model Runer V2][DFlash] Fix lm head sharing for dflash (#46435) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-06-23 19:09:06 +00:00
Jee Jee Li ebbb2d55ac [CI/Build][Bugfix] Fix SD LoRA (#45941) 
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
 2026-06-18 00:34:15 +00:00
Lucas WilkinsonMartin KuklaMatthew BonanniDipika SikkaNickLucchejiahancAlec KohlhoffPorras HuangClaudescoootscooob
 		eb28452b10 [Model] Add DiffusionGemma Support (#45163) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Martin Kukla <martin.kukla@cantab.net>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Dipika Sikka <dsikka@redhat.com>
Co-authored-by: NickLucche <nlucches@redhat.com>
Co-authored-by: jiahanc <173873397+jiahanc@users.noreply.github.com>
Co-authored-by: Alec Kohlhoff <134344302+aleckohlhoff@users.noreply.github.com>
Co-authored-by: Porras Huang <20535584+porrashuang@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: scoootscooob <167050519+scoootscooob@users.noreply.github.com>
 2026-06-11 22:17:35 -07:00
Benjamin ChislettandGiancarlo Delfin 0bae1d3848 [MRV2][Spec Decode] DFlash (#44586) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com>
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com>
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-06-10 08:47:46 -07:00
Jared Wenandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 32f34d3935 [feature] add index share feature for DSA MTP (#44420) 
Signed-off-by: JaredforReal <w13431838023@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-06 22:04:14 -07:00
Giancarlo Delfin ceb0111a90 [Model Runner V2][Spec Decode] Add Gemma4 MTP support (#43241) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-06-04 00:51:06 +00:00
Giancarlo DelfinandWoosuk Kwon 91945b6e4a [Bug Fix][Model Runner V2][Spec Decode] Warmup & capture with different attention states for speculator prefill (#44253) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
 2026-06-03 13:32:40 -07:00
zhrrr 1edfd09ffd [Model Runner V2] Use actual batch max_seq_len for attn metadata (#43991) 
Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com>
 2026-06-02 06:07:56 +00:00
Lucas Wilkinson 4721bb3aa4 [MRV2] Remove Eagle's dedicated CUDA graph pool (#44078) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
 2026-05-31 22:00:33 -07:00
Woosuk Kwon 27fa5aa3b9 [MRV2] Support breakable CUDA graph (#44050) 
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
 2026-05-30 09:40:52 -07:00
Mengqing CaoandNick Hill 1223732dda [ModelRunnerV2][Hybrid model] Support kernel block size in hybrid model (#38831) 
Signed-off-by: MengqingCao <cmq0113@163.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Signed-off-by: Mengqing Cao <cmq0113@163.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-05-28 00:55:55 +00:00
Nicolò LucchesiandNick Hill fba010dd74 [Bugfix][MRV2] Fix KVCache tensor explicit kernel_block_size dim (#42766) 
Signed-off-by: NickLucche <nlucches@redhat.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-05-18 20:25:41 -07:00
PatchyTISandPatchouliTaisa 0162596603 [Model Runner V2] FP32 gumbel sampling. (#41775) 
Signed-off-by: PatchouliTaisa <patchychen@tencent.com>
Co-authored-by: PatchouliTaisa <patchychen@tencent.com>
 2026-05-15 09:20:08 -07:00