iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
19,362 Commits 3 Branches 0 Tags
f1899b2ffdf87598350c416a074c8c291c845902
Commit Graph
164 Commits
Author SHA1 Message Date
Nick HillandJeff Ma 6fbbcf2151 [BugFix] Stop dummy runs from writing mamba state through stale block-table rows (#49757) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Signed-off-by: Jeff Ma <jeffjma@umich.edu>
Co-authored-by: Jeff Ma <jeffjma@umich.edu>
 2026-07-29 01:09:41 +00:00
Kevin Glynn 56f31af62a [Bugfix] Fix /wake_up crash on hybrid models (Mamba/DeltaNet) (#41602) 
Signed-off-by: Kevin Glynn <kevglynn@gmail.com>
 2026-07-29 00:17:35 +00:00
Nick Hill 0d0504b54c [Core] Warm up runner-owned Triton kernels before the first request (#49903) 2026-07-28 07:58:02 -07:00
liuzhenwei afc94523c9 [XPU][CI] Use platform device in InputBatch V2 test (#49939) 
Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com>
 2026-07-27 15:28:02 +08:00
Nick Hill 50aa830482 [BugFix][MRV2] Don't create dummy requests longer than max_model_len (#49751) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-07-27 02:04:20 +00:00
Elvir CrnčevićandClaude Opus 4.6 972848f276 [Bugfix] Support non-uniform page sizes in KVBlockZeroer (#49704) 
Signed-off-by: Elvir Crncevic <elvircrn@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-07-24 13:38:59 -07:00
Taneem Ibrahimandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 7b40fb9645 [UX] Reject incompatible nested runtime overrides (#49247) 
Signed-off-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-07-24 07:16:22 -07:00
hotTeahanxi-java韩熙
 		833483f357 Encoder cache extension hooks (#48218) 
Signed-off-by: hotTea <958436561@qq.com>
Signed-off-by: hanxi-java <634498162@qq.com>
Co-authored-by: hanxi-java <634498162@qq.com>
Co-authored-by: 韩熙 <63780107+hanxi-java@users.noreply.github.com>
 2026-07-24 02:52:00 -07:00
Michael GoinandNick Hill ac36a7a1e7 [MRV2][Spec Decode] Avoid rejection sampler OOM by chunking (#48630) 
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-07-23 18:13:13 +08:00
Nick Hillandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 521aa80f71 [Core] Simplify KVBlockZeroer index tensor handling (#48399) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-07-23 18:12:46 +08:00
Isotr0py 1750e443f2 [Misc] Move PyNvVideoCodec stuff out of gpu worker (#49322) 
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
 2026-07-22 10:11:56 +08:00
aoshen02 9459fc6471 [Bugfix][RL] Set vLLM config during weight reload (#45989) 
Signed-off-by: aoshen02 <aoshen@inferact.ai>
 2026-07-20 15:02:56 +08:00
devalshahamdandDeval Shah fcd2255d16 [Hardware][GPU] Profiler config additional to increase it scope and annotation details (#37524) 
Signed-off-by: devalshahamd <deval.shah@amd.com>
Signed-off-by: Deval Shah <devashah@amd.com>
Signed-off-by: Deval Shah <deval.shah@amd.com>
Co-authored-by: Deval Shah <devashah@amd.com>
 2026-07-17 13:38:59 -07:00
wendadawenandTianyu Guo 69d4f5ef63 [Bugfix][Multimodal] Fix Qwen3-Omni use_audio_in_video with mixed image/video inputs (#46213) 
Signed-off-by: wendadawen <wendadawen@qq.com>
Signed-off-by: Tianyu Guo <guoty9@mail2.sysu.edu.cn>
Co-authored-by: Tianyu Guo <guoty9@mail2.sysu.edu.cn>
 2026-07-17 08:31:16 +00:00
vx120crp0128aoshen02Claude Opus 4.8
 		fc1c548093 Runtime Draft Weight Update for Speculative Decoding (#46725) 
Signed-off-by: vx120 <893600387@qq.com>
Signed-off-by: vx120 <57470515+vx120@users.noreply.github.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: crp0128 <191679376@qq.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
 2026-07-11 22:51:53 -07:00
Lucas WilkinsonClaudeOpenAI CodexMatthew BonanniNicolò Lucchesi
 		51878e5b6e [2/N][KV-Cache Layout Refactor] Pack K/V into the content dim across attention backends (#44455) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Signed-off-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
 2026-07-11 11:11:16 -04:00
Yan XuandJingyi Yang 95ed0feaa5 DCP supports hybrid attention (#40996) 
Signed-off-by: YanXu <yancey.yx@alibaba-inc.com>
Signed-off-by: Jingyi Yang <girasoleyang@gmail.com>
Co-authored-by: Jingyi Yang <girasoleyang@gmail.com>
 2026-07-09 21:34:45 -07:00
Woosuk KwonClaude Fable 5Nick Hill
 		85b3a7264b [Bugfix][Model Runner V2] Order uniform decodes first so spec decodes aren't misclassified as prefills (#47381) 
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-07-09 14:26:27 +01:00
Benjamin ChislettandWentao Ye 1cd75b3dd4 [Bugfix] Fix race condition in KVBlockZeroer (#48085) 
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com>
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
 2026-07-09 09:18:19 +00:00
Nick Hill a5d19cbb95 [Core] Move MRV1 late_interaction_runner.py out of MRV2 subtree (#48014) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-07-08 18:30:11 +00:00
Walter Beller-Morales dd127d82ed [Core][Engine] only materialize tokens when thinking budget is in req (#47053) 
Signed-off-by: walterbm <walter.beller.morales@gmail.com>
 2026-07-07 21:02:38 -06:00
Nils MattesonNils Mattesonmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		e97c3cb303 [Core] Persist and reuse the memory-profiling result across boots (opt-in) (#47388) 
Signed-off-by: Nils Matteson <nils@thaw.sh>
Signed-off-by: Nils Matteson <nilsmatteson@icloud.com>
Co-authored-by: Nils Matteson <nils@thaw.sh>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-07-08 00:53:02 +00:00
Łukasz Ślusarczykmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>Kunshang Ji
 		78a04c208d [XPU] Fix CUDA API shims breaking Torch Dynamo during AOT compile (#43092) 
Signed-off-by: Łukasz Ślusarczyk <lukasz.slusarczyk@intel.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
 2026-07-06 08:29:20 +08:00
Woosuk KwonandClaude Fable 5 a47f38f825 [Bugfix][Model Runner V2][Spec Decode] Fix int32 offset overflow in block verification kernels (#47383) 
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
 2026-07-02 07:32:38 -07:00
Yiliu DongandNick Hill 00eb7cefa3 [Bugfix] Prevent padding placeholders from reaching embeddings (#47029) 
Signed-off-by: qianlihuang <91178480+qianlihuang@users.noreply.github.com>
Signed-off-by: Yiliu Dong <91178480+qianlihuang@users.noreply.github.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-07-01 09:26:03 -07:00
Aaron HaoandClaude Sonnet 4.6 77a9c5ae28 Weight sync refactor + move sparse nccl engine (#44353) 
Signed-off-by: hao-aaron <ahao@anyscale.com>
Signed-off-by: haoaaron <ahao@anyscale.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
 2026-07-01 01:25:19 -07:00
Nils Matteson a461070d1c [Core] Make sleep-mode backend capability flags communicator-agnostic (#47243) 2026-07-01 07:17:44 +00:00
Nils Matteson 93d8f834dd [Core] Pluggable sleep-mode backend abstraction (RFC #34303) (#44074) 2026-06-30 22:00:53 -07:00
Nick Hillandziminghuang fcaa84efa7 [BugFix] Gate MRV2 mixed sparse-MLA warmup on max_num_seqs > 1 (#47050) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: ziminghuang <ziminghuang@inferact.ai>
 2026-06-30 16:31:27 +01:00
Chauncey 11a12305c0 [Model Runner V2][Spec Decode] Handle tuple hidden states from MTP draft models (#46786) 2026-06-27 18:38:07 -07:00
Nick Hill c6dd32a810 [ModelRunner V2] Support realtime embeddings (#46762) 2026-06-26 19:42:27 -07:00
Brandon PelfreyandRoger Wang af16446bf3 Vram semaphore infra (#44465) 
Signed-off-by: Brandon Pelfrey <bpelfrey@nvidia.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2026-06-26 17:32:51 -07:00
Nick HillClaude Opus 4.8Roger Wang
 		3e6529cc0e [Bugfix][Spec Decode] Fix EAGLE drafter multimodal encoder cache misses (#46315) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2026-06-22 18:14:02 +00:00
Alex SteinerClaude Opus 4.8Giancarlo DelfinYifan Qiao
 		2cac89f9da [Spec Decode] Support mixed KV page sizes for DFlash (#45181) 
Signed-off-by: Alex Steiner <asteiner@nvidia.com>
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
 2026-06-21 22:45:14 +08:00
Nick Hill 7df3d7dada [Core] Ensure memory is pinned prior to async h2d copy (#45424) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-06-20 20:02:24 -07:00
Woosuk Kwon 16908e132e [MRV2] Make FP32 Gumbel sampling more accurate (#45996) 
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
 2026-06-18 19:42:09 +00:00
sungsoo haandThomas Parnell 4bf699d310 [Kernel] Support DS Mamba tail copy for MTP align mode (#45473) 
Signed-off-by: Sungsoo Ha <sungsooh@nvidia.com>
Co-authored-by: Thomas Parnell <tom.parnell@gmail.com>
 2026-06-16 22:50:30 +00:00
Song ZhixinandNick Hill 8e27a9c215 [PERF] Fuse multi-group block table staged writes (#44944) 
Signed-off-by: jesse <szxfml@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-06-16 10:53:27 -07:00
Juan Pérez de Algaba 470229c37e [Security] Fix DoS via prompt_embeds on M-RoPE models (#45252) 
Signed-off-by: jperezde <jperezde@redhat.com>
 2026-06-13 10:17:38 +00:00
Lucas WilkinsonMartin KuklaMatthew BonanniDipika SikkaNickLucchejiahancAlec KohlhoffPorras HuangClaudescoootscooob
 		eb28452b10 [Model] Add DiffusionGemma Support (#45163) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Martin Kukla <martin.kukla@cantab.net>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Dipika Sikka <dsikka@redhat.com>
Co-authored-by: NickLucche <nlucches@redhat.com>
Co-authored-by: jiahanc <173873397+jiahanc@users.noreply.github.com>
Co-authored-by: Alec Kohlhoff <134344302+aleckohlhoff@users.noreply.github.com>
Co-authored-by: Porras Huang <20535584+porrashuang@users.noreply.github.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: scoootscooob <167050519+scoootscooob@users.noreply.github.com>
 2026-06-11 22:17:35 -07:00
Wentao Ye 2c27c294c0 [Model Runner V2] Fix mrv2 mm lora issue (#44450) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-06-08 14:30:09 -04:00
Yan Maandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 0414d75410 [XPU] skip unapplied UT in test_gpu_model_runner.py (#44289) 
Signed-off-by: Yan Ma <yan.ma@intel.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-04 08:48:17 +08:00
Nick Hill e15f20258b [ModelRunnerV2] Avoid pipeline parallel bubbles (#42187) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-06-02 14:02:01 -07:00
Siddharth BedekarandOpenAI Codex 266b9d9c64 [Frontend][Core] Add sparse NCCL weight transfer support for in-place updates (#40096) 
Signed-off-by: Siddharth Bedekar <bedeksid@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-06-01 15:37:30 -04:00
Nick Hillandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 7e53283b1c [Core] Cleanup KVConnector handling with PP + fix MRV2 (#43732) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-28 13:12:03 -07:00
Nick HillandWentao Ye 8c94938cfb [MRV2][BugFix] Fix KV connector handling in spec decode case (#43719) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
 2026-05-27 06:37:56 +00:00
Lucas WilkinsonMatthew BonanniNicolò Lucchesi
 		7e33081cee [Attention] Make FlexAttention and FlashAttention use num-blocks first layouts (#42095) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
 2026-05-26 19:55:56 -07:00
Francesco Fuscoandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> b730c46352 [Perf] [Hybrid] Fused Triton kernel for GPU-side Mamba state postprocessing (#40172) 
Signed-off-by: Francesco Fusco <ffu@zurich.ibm.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-21 04:50:54 -07:00
Siddharth Bedekar f51f6844f9 [Bugfix][Spec Decode] Wire draft_probs into probabilistic draft_model rejection (#40269) 2026-05-13 21:04:03 -04:00
Jing WangCopilotWentao Yemergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		27702f6d08 [Bugfix] Fix token loss in PP mode which causes degraded accuracy (#41133) 
Signed-off-by: Jing Wang <jingwang96@qq.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-06 14:07:32 -04:00