iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
9,510 Commits 3 Branches 0 Tags
d593cf28fa020dbae53dae19122aac8aeeeae0bc
Commit Graph
476 Commits
Author SHA1 Message Date
Sage MooreLucas WilkinsonLucas Wilkinsonyewentao256Lucas WilkinsonRobert Shaw
 		567939953b [Core/DBO][1/N] Add Dual-Batch Overlap mechanism to VLLM (#23693) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Sage Moore <sage@neuralmagic.com>
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Co-authored-by: yewentao256 <zhyanwentao@126.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
 2025-09-16 12:21:48 -04:00
Chen Brucelemon412Harry Mellor
 		7ea5c73ad7 [Feat][EPLB] A novel static EPLB placement strategy for MoE models. (#23745) 
Signed-off-by: bruceszchen <bruceszchen@tencent.com>
Signed-off-by: Chen Bruce <bruceszchen@tencent.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Chen Bruce <cszwwdz@vip.qq.com>
Co-authored-by: lemon412 <lemon412@foxmail.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-09-16 10:55:16 +00:00
Harry Mellorandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> c4afdb69cc Move MultiModalConfig from config/__init__.py to config/multimodal.py (#24659) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-09-15 17:43:16 +00:00
Nick HillRonald1995Robert Shaw
 		4fdd6f5cbf [Core] Support async scheduling with uniproc executor (#24219) 
Signed-off-by: Nick Hill <nhill@redhat.com>
Signed-off-by: Ronald1995 <ronaldautomobile@163.com>
Co-authored-by: Ronald1995 <ronaldautomobile@163.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
 2025-09-12 16:34:28 -07:00
dongluw a5b84f1cbf [Core] Shared memory based object store for Multimodal data caching and IPC (#20452) 
Signed-off-by: donglu <donglu@cohere.com>
 2025-09-12 07:54:17 -07:00
RichardoMuMu HuaiYe ZhangBenjamin Bartelssimon-mo瑜琮Aaron Pham22quinn
 		40b6c9122b [V1] feat:add engine v1 tracing (#20372) 
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
Signed-off-by: Ye Zhang <zhysishu@gmail.com>
Signed-off-by: RichardoMu <44485717+RichardoMrMu@users.noreply.github.com>
Signed-off-by: simon-mo <simon.mo@hey.com>
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
Co-authored-by: Mu Huai <tianbowen.tbw@antgroup.com>
Co-authored-by: Ye Zhang <zhysishu@gmail.com>
Co-authored-by: Benjamin Bartels <benjamin@bartels.dev>
Co-authored-by: simon-mo <simon.mo@hey.com>
Co-authored-by: 瑜琮 <ly186375@antfin.com>
Co-authored-by: Aaron Pham <contact@aarnphm.xyz>
Co-authored-by: 22quinn <33176974+22quinn@users.noreply.github.com>
 2025-09-11 17:10:39 -07:00
Boyuan FengandHarry Mellor 94e6b2d55f Allow users to specify kv cache memory size (#21489) 
Signed-off-by: Boyuan Feng <boyuan@meta.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-09-11 13:41:07 +00:00
Harry Mellor d6249d0699 Fix typing for safetensors_load_strategy (#24641) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-09-11 10:41:39 +00:00
shengshiqi-googleandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> 41329a0ff9 [Core] feat: Add --safetensors-load-strategy flag for faster safetensors loading from Lustre (#24469) 
Signed-off-by: Shiqi Sheng <shengshiqi@google.com>
Signed-off-by: shengshiqi-google <160179165+shengshiqi-google@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-09-10 23:10:01 -07:00
Hanjie QiuandMichael Goin dcb28a332b [Kernel] Flashinfer MLA (trtllm-gen) decode kernel integration (#21078) 
Signed-off-by: hjjq <hanjieq@nvidia.com>
Signed-off-by: Michael Goin <mgoin64@gmail.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
 2025-09-10 15:31:10 -07:00
Xingyu Liuandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> 9fb74c27a7 [Core] Support configuration parsing plugin (#24277) 
Signed-off-by: Xingyu Liu <charlotteliu12x@gmail.com>
Signed-off-by: Xingyu Liu <38244988+charlotte12l@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-09-10 11:32:43 -07:00
pwschuurmanOmer DayanCyrus Leung
 		4377b1ae3b [Bugfix] Update Run:AI Model Streamer Loading Integration (#23845) 
Signed-off-by: Omer Dayan (SW-GPU) <omer@run.ai>
Signed-off-by: Peter Schuurman <psch@google.com>
Co-authored-by: Omer Dayan (SW-GPU) <omer@run.ai>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2025-09-09 21:37:17 -07:00
Zebing Lin 82dfb12e52 [Core] Use sha256 bytes instead of BlockHash to reduce GC overhead (#23673) 
Signed-off-by: linzebing <linzebing1995@gmail.com>
 2025-09-08 21:34:37 -07:00
Didier Durand f4962a6d55 [Doc]: fix typos in Python comments (#24417) 
Signed-off-by: Didier Durand <durand.didier@gmail.com>
 2025-09-08 00:22:16 -07:00
Woosuk Kwon 4172235ab7 [V0 deprecation] Deprecate V0 Neuron backend (#21159) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-09-06 16:15:18 -07:00
yzdshongchaoyoukaichao
 		ac201a0eaf [Feature] Support Decode Context Parallel (DCP) for MLA (#23734) 
Signed-off-by: hongchao <hongchao@msh.team>
Signed-off-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: hongchao <hongchao@msh.team>
Co-authored-by: youkaichao <youkaichao@gmail.com>
 2025-09-06 13:24:05 +08:00
Lucas WilkinsonMatthew BonanniMatthew Bonanni
 		402759d472 [Attention] FlashAttn MLA (#14258) 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Matthew Bonanni <mbonanni001@gmail.com>
Co-authored-by: Matthew Bonanni <mbonanni001@gmail.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
 2025-09-04 02:47:59 -07:00
Isotr0py d7fbc6ddac [Misc] Enable V1 FP16 inference on pre-Ampere GPUs (#24022) 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2025-09-01 08:12:22 +00:00
Christian PintoandMax de Bayser 1cb39dbcdd [Misc] IO Processor plugins for pooling models (#22820) 
Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
Co-authored-by: Max de Bayser <mbayser@br.ibm.com>
 2025-08-31 23:07:12 -07:00
Maximilien de BayserandWoosuk Kwon 2554b27baa [V0 Deprecation] Remove pooling model support in V0 (#23434) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-08-29 00:04:02 -07:00
Didier Durand d3da2eea54 [Doc]: fix typos in Python scripts (#23828) 
Signed-off-by: Didier Durand <durand.didier@gmail.com>
 2025-08-28 05:37:38 -07:00
Asaf Joseph Gardin 853c371fc3 [V1][Mamba] - Enable V1 by default for Mamba Models (#23650) 
Signed-off-by: asafg <39553475+Josephasafg@users.noreply.github.com>
 2025-08-27 20:53:30 +00:00
Harry Mellor 513c1fe255 Only run get_attr_docs if generating help text (#23723) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-08-27 13:55:12 +00:00
Cyrus Leung 69244e67e6 [Core] Use key-only cache for BaseMultiModalProcessor (#23018) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-08-27 14:19:13 +08:00
Cyrus Leung 50fede6634 [V1] Enable V1 for compute capability < 8.0 + FP32 (#23614) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-08-26 03:00:18 -07:00
rongfu.leng 1b9b16649c [Misc] update dict parse to EPLBConfig from json dumps to dict unpacking (#23305) 
Signed-off-by: rongfu.leng <rongfu.leng@daocloud.io>
 2025-08-24 08:06:34 +00:00
Didier Durand 22cf679aad [Doc]: fix various typos in multiple files (#23179) 
Signed-off-by: Didier Durand <durand.didier@gmail.com>
 2025-08-22 10:38:46 -07:00
Matthew Bonanni 19fe1a0510 [Kernel] Add FP8 support with FlashMLA backend (#22668) 
Signed-off-by: Matthew Bonanni <mbonanni001@gmail.com>
 2025-08-22 02:26:32 +00:00
22quinn 480bdf5a7b [Core] Support custom executor qualname (#23314) 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
 2025-08-22 09:40:54 +08:00
Robert ShawandRobert Shaw c8e33c72c6 [V1] Remove unnecessary check for main thread (#23298) 
Signed-off-by: Robert Shaw <robshaw@redhat.com>
Co-authored-by: Robert Shaw <robshaw@redhat.com>
 2025-08-21 14:08:35 +00:00
22quinnNick HillWoosuk Kwon
 		f571ff8eb6 [Sampler] Support returning final logprobs (#22387) 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-08-20 21:28:32 -07:00
Michael Goin bbea1cefdd [CI Bugfix] Fix CI by fully removing --enable-prompt-adapter (#23284) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-08-20 17:18:12 -07:00
rongfu.lengandHarry Mellor 4fbda0b20c [Feature] use --eplb_config to set eplb param (#20562) 
Signed-off-by: rongfu.leng <rongfu.leng@daocloud.io>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: rongfu.leng <lenronfu@gmail.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-08-20 14:07:28 -07:00
Cyrus Leung 5efd6905bc [CLI][Doc] Formalize --mm-encoder-tp-mode (#23190) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-08-20 23:42:28 +08:00
rongfu.leng 38217877aa [Fix] fix offline env use local mode path (#22526) 
Signed-off-by: rongfu.leng <rongfu.leng@daocloud.io>
 2025-08-20 13:34:49 +00:00
Nikhil Suryawanshi 78dba404ad [Hardware][IBM Z]Enable v1 for s390x and s390x dockerfile fixes (#22725) 
Signed-off-by: Nikhil Suryawanshi <suryawanshin74@gmail.com>
 2025-08-19 04:40:37 +00:00
afeldman-nmNick HillAndrew FeldmanHarry Mellor
 		bf7f470b22 [V1] Logits processors extensibility (#19912) 
Signed-off-by: Andrew Feldman <afeldman@redhat.com>
Signed-off-by: Andrew Feldman <afeld2012@gmail.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Andrew Feldman <afeld2012@gmail.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-08-16 12:59:17 -07:00
Maximilien de Bayser 52ce1420e9 Fix handling of max_num_batched_tokens for pooling tasks (#23004) 
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
 2025-08-16 17:36:30 +00:00
Thomas ParnellDaniel AfrimiBurkhard RingleinChen Zhang
 		75531a6c13 [V1] [Hybrid] Support using float32 for state in Hybrid Models (Mamba2, Mamba1, Minimax) (#22928) 
Signed-off-by: Daniel Afrimi <danielafrimi8@gmail.com>
Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
Co-authored-by: Daniel Afrimi <danielafrimi8@gmail.com>
Co-authored-by: Burkhard Ringlein <ngl@zurich.ibm.com>
Co-authored-by: Chen Zhang <zhangch99@outlook.com>
 2025-08-15 12:57:06 +00:00
Roger Wangandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> 49252cf59e [MM] Allow skipping memory profiling for multimodal models. (#22950) 
Signed-off-by: Roger Wang <hey@rogerw.me>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-08-15 11:41:38 +00:00
Cyrus Leung dbe298046c [Bugfix] Fix parsing of --disable-mm-preprocessor-cache (#22909) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-08-14 08:09:44 -07:00
Woosuk Kwon 71683ca6f6 [V0 Deprecation] Remove multi-step scheduling (#22138) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Signed-off-by: Woosuk Kwon <woosuk@thinkingmachines.ai>
 2025-08-12 20:18:39 -07:00
Harry Mellor 4fbd8bb597 Fix passing SpeculativeConfig from the CLI (#22652) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-08-11 22:13:32 -07:00
wang.yuqi 84cf78acee [Model] Pooling models default to using chunked prefill & prefix caching if supported. (#20930) 
Signed-off-by: wang.yuqi <noooop@126.com>
 2025-08-11 09:41:37 -07:00
Harry Mellor c49848396d Refactor sliding window configuration to Transformers best practice (#21927) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-08-09 20:50:48 -07:00
Harry Mellor 56186474f6 [Docs] Reduce noise in docs and --help from the JSON tip (#22567) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-08-09 08:31:32 -07:00
Harry Mellor e3edc0a7a8 Extract CompilationConfig from config.py (#22524) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-08-08 16:34:25 -07:00
Harry Mellor 7e3a8dc906 Remove from_dict from SpeculativeConfig (#22451) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-08-07 10:13:04 -07:00
Cyrus Leung 139d155781 [Frontend] Use engine argument to control MM cache size (#22441) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-08-07 09:47:10 -07:00
Cyrus Leung 766bc8162c [Core] Store only the keys for multi-modal data in P0 (#22198) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-08-07 01:45:04 -07:00