iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
14,386 Commits 3 Branches 0 Tags
fa6a6be51978bd4b49ba0da17039e60f96dc5b13
Commit Graph
1000 Commits
Author SHA1 Message Date
JartX 68af77e51c [FIXBUG] Correctly Apply Grammar Bitmask in Mixed Batches (#22896) 
Signed-off-by: JartX <sagformas@epdcenter.es>
 2025-08-15 17:42:49 +00:00
fhl2000Luka GovedičLucas WilkinsonLucas Wilkinson
 		74f441f4b5 [Core] Allow full cudagraph with separate attention routines and orthogonal to compilation, add support for FA2 and FlashInfer (#20059) 
Signed-off-by: fhl <2410591650@qq.com>
Signed-off-by: fhl2000 <63384265+fhl2000@users.noreply.github.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
 2025-08-15 10:01:39 -04:00
Thomas ParnellDaniel AfrimiBurkhard RingleinChen Zhang
 		75531a6c13 [V1] [Hybrid] Support using float32 for state in Hybrid Models (Mamba2, Mamba1, Minimax) (#22928) 
Signed-off-by: Daniel Afrimi <danielafrimi8@gmail.com>
Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
Co-authored-by: Daniel Afrimi <danielafrimi8@gmail.com>
Co-authored-by: Burkhard Ringlein <ngl@zurich.ibm.com>
Co-authored-by: Chen Zhang <zhangch99@outlook.com>
 2025-08-15 12:57:06 +00:00
Roger Wangandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> 49252cf59e [MM] Allow skipping memory profiling for multimodal models. (#22950) 
Signed-off-by: Roger Wang <hey@rogerw.me>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-08-15 11:41:38 +00:00
nvjullinandmgoin 279a5f31b3 [Kernel] Add nvfp4 gemm flashinfer backends (#22346) 
Signed-off-by: Julien Lin <jullin@nvidia.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
 2025-08-14 16:03:55 -04:00
Lucas Wilkinson 829b9a62d0 [Perf] Dont create unnecessary pooling params (#22876) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
 2025-08-14 05:28:09 -07:00
Cyrus Leung 19b927e52d [Core] Use individual MM items in P0/P1 cache and model runner (#22570) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-08-13 07:18:07 -07:00
Rahul Tuli 5a4b4b3729 Add: SupportsEagle3 interface for explicit EAGLE3 support (#22642) 
Signed-off-by: Rahul Tuli <rtuli@redhat.com>
 2025-08-12 09:24:52 -07:00
wang.yuqiandMax de Bayser 6d729c43fb [Bugfix] Fix ModernBert load & Enable sliding window attention for bidirectional attention. (#22637) 
Signed-off-by: wang.yuqi <noooop@126.com>
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
Co-authored-by: Max de Bayser <mbayser@br.ibm.com>
 2025-08-12 00:23:17 -07:00
wang.yuqi 84cf78acee [Model] Pooling models default to using chunked prefill & prefix caching if supported. (#20930) 
Signed-off-by: wang.yuqi <noooop@126.com>
 2025-08-11 09:41:37 -07:00
Maximilien de Bayser 39052dbca8 Support token_type_ids in V1 with less code changes (#21985) 
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
 2025-08-10 22:54:59 -07:00
Nick Hill 5898b135ab [BugFix] Fix KVConnectorOutput TPU breakage (#22598) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-08-10 19:33:48 -07:00
Chengji YaoandChengji Yao 2a84fb422f [TPU] kv cache update kernel doesn't need to be padded slices to multiple of num_slices_per_block (#22394) 
Signed-off-by: Chengji Yao <chengjiyao@gmail.com>
Co-authored-by: Chengji Yao <chengjiyao@gmail.com>
 2025-08-09 20:49:04 -07:00
+10
Roger WangNing XieTJianAndrew SansomZhiyuShu WangXIn LiJunhao LiChaunceyYuxuan ZhangZiTian ZhaoCyrus LeungPo-Han HuangiAmir97iAmir97gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>Cyrus LeungHong Hanhyoukaichaolkchen
 		08b751ba74 Implicit language-model-only mode via limit-mm-per-prompt (#22299) 
Signed-off-by: Roger Wang <hey@rogerw.me>
Signed-off-by: Andy Xie <andy.xning@gmail.com>
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
Signed-off-by: Andrew Sansom <andrew@protopia.ai>
Signed-off-by: Zhiyu Cheng <zhiyuc@nvidia.com>
Signed-off-by: Shu Wang <shuw@nvidia.com>
Signed-off-by: Po-Han Huang <pohanh@nvidia.com>
Signed-off-by: Shu Wang. <shuw@nvidia.com>
Signed-off-by: XIn Li <xinli@nvidia.com>
Signed-off-by: Junhao Li <junhao@ubicloud.com>
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
Signed-off-by: zRzRzRzRzRzRzR <2448370773@qq.com>
Signed-off-by: zitian.zhao <zitian.zhao@tencentmusic.com>
Signed-off-by: zitian zhao <zitian.zhao@tencentmusic.com>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: iAmir97 <Amir.balwel@embeddedllm.com>
Signed-off-by: iAmir97 <71513472+iAmir97@users.noreply.github.com>
Signed-off-by: Linkun <github@lkchen.net>
Co-authored-by: Ning Xie <andy.xning@gmail.com>
Co-authored-by: TJian <tunjian.tan@embeddedllm.com>
Co-authored-by: Andrew Sansom <andrew@protopia.ai>
Co-authored-by: Zhiyu <zhiyuc@nvidia.com>
Co-authored-by: Shu Wang <shuw@nvidia.com>
Co-authored-by: XIn Li <xinli@nvidia.com>
Co-authored-by: Junhao Li <streaver91@gmail.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
Co-authored-by: Yuxuan Zhang <2448370773@qq.com>
Co-authored-by: ZiTian Zhao <zitian.zhao@tencentmusic.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
Co-authored-by: iAmir97 <71513472+iAmir97@users.noreply.github.com>
Co-authored-by: iAmir97 <Amir.balwel@embeddedllm.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Co-authored-by: Hong Hanh <hanh.usth@gmail.com>
Co-authored-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: lkchen <github@lkchen.net>
 2025-08-08 22:21:40 -07:00
Kunshang Ji 81c57f60a2 [XPU] upgrade torch 2.8 on for XPU (#22300) 
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
 2025-08-08 17:03:45 -07:00
Varun Sundar RabindranathandVarun Sundar Rabindranath f703b923f3 [Misc] DeepGEMM : Avoid JIT generation in the hot-path (#22215) 
Signed-off-by: Varun Sundar Rabindranath <vsundarr@redhat.com>
Co-authored-by: Varun Sundar Rabindranath <vsundarr@redhat.com>
 2025-08-08 16:09:59 -07:00
Lucas Wilkinson 1dc8a70b6d [Attention] Support multiple attention metadata builders per kv_cache_spec + proper local attention no hybrid kv cache fix (#21588) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
 2025-08-06 18:40:52 -07:00
Benjamin Chislett 7e6544c797 [Perf] Parallelize fill_bitmask to accelerate high-throughput guided decoding (#21862) 
Signed-off-by: Benjamin Chislett <benjamin.chislett@centml.ai>
 2025-08-05 19:57:49 -07:00
Cyrus Leung 811ac13d03 [Core] Factor out common logic for MM budget calculation (#22228) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-08-04 23:54:55 -07:00
lkchen f4f4e7ef27 [V0 deprecation][P/D] Deprecate v0 KVConnectorBase code (1/2) (#21785) 
Signed-off-by: Linkun Chen <github@lkchen.net>
 2025-08-04 19:11:33 -07:00
22quinn 54de71d0df [Sampler] Support returning all logprobs or logits (#21792) 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
 2025-08-04 03:04:12 -07:00
David Ben-DavidandDavid Ben-David aefeea0fde [V1] [P/D] Refactor KV Connector Path (#21980) 
Signed-off-by: David Ben-David <davidb@pliops.com>
Co-authored-by: David Ben-David <davidb@pliops.com>
 2025-08-03 04:03:40 -07:00
Thomas Parnell 4abfd8796f [V1] [Hybrid] Validate compatibility of attention backend batch reordering at init time (#21557) 
Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>
 2025-08-02 05:29:40 -07:00
fhl2000 23322431c8 [V1][CUDA] Full cudagraph support for FlashInfer (#21367) 2025-08-01 21:49:34 -04:00
zhiweizandRoger Wang 9e0726e5bf [Meta] Official Eagle mm support, first enablement on llama4 (#20788) 
Signed-off-by: morgendave <morgendave@gmail.com>
Co-authored-by: Roger Wang <hey@rogerw.me>
 2025-07-31 10:35:07 -07:00
Yong Hoon Shin ad510309ee Override attention metadata for fast prefill in some KV sharing setups (#21590) 
Signed-off-by: Yong Hoon Shin <yhshin@meta.com>
 2025-07-30 08:54:15 -07:00
Csrayz b917da442b Expose PyTorch profiler configuration to environment variables (#21803) 
Signed-off-by: Csrayz <33659823+Csrayz@users.noreply.github.com>
 2025-07-29 19:46:31 -07:00
Chen Zhang 555e7225bc [v1][attention] Support Hybrid Allocator + FlashInfer (#21412) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-07-30 01:45:29 +00:00
Richard Zou 04e38500ee [Bugfix] VLLM_V1 supports passing other compilation levels (#19340) 
Signed-off-by: Richard Zou <zou3519@gmail.com>
 2025-07-29 09:35:58 -04:00
Kuntai Du b18b417fbf Revert "[V1] Exception Handling when Loading KV Cache from Remote Store" (#21778) 
Signed-off-by: KuntaiDu <kuntai@uchicago.edu>
 2025-07-28 20:15:18 +00:00
Asaf Joseph Gardinandasafg a6c050286a [v1][mamba] Added mamba_type into MambaSpec (#21715) 
Signed-off-by: asafg <asafg@ai21.com>
Co-authored-by: asafg <asafg@ai21.com>
 2025-07-28 08:15:55 +00:00
Lucas Wilkinson 139a7f07bd [BugFix] Fix ChunkedLocalAttention when the hybrid kv-cache is disabled (#21707) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
 2025-07-28 07:18:47 +00:00
Adelineandliuyumoye 15a72ac478 [V1] Exception Handling when Loading KV Cache from Remote Store (#21534) 
Signed-off-by: liuyumoye <adeline_ly2023@outlook.com>
Co-authored-by: liuyumoye <adeline_ly2023@outlook.com>
 2025-07-27 20:34:17 -07:00
Cyrus LeungandHarry Mellor 86ae693f20 [Deprecation][2/N] Replace --task with --runner and --convert (#21470) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-07-27 19:42:40 -07:00
Ye (Charlotte) Qi a40a8506df [Misc] Improve memory profiling debug message (#21429) 
Signed-off-by: Ye (Charlotte) Qi <yeq@meta.com>
 2025-07-26 07:07:21 -07:00
Maximilien de BayserandRussell Bryant 1cd6eaba54 Support encoder-only models without KV-Cache (#21270) 
Signed-off-by: Max de Bayser <maxdebayser@gmail.com>
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
Co-authored-by: Russell Bryant <rbryant@redhat.com>
 2025-07-26 21:09:52 +08:00
Rui Qiao cea96a0156 [Bugfix] Fix sync_and_slice_intermediate_tensors (#21537) 
Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
 2025-07-25 17:07:58 -07:00
Cyrus Leung 46d81d6951 [V1] Get supported tasks from model runner instead of model config (#21585) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-07-25 05:36:45 -07:00
Juncheng GuRichard LiuRichard Liu
 		6066284914 [P/D] Support CPU Transfer in NixlConnector (#18293) 
Signed-off-by: Juncheng Gu <juncgu@gmail.com>
Signed-off-by: Richard Liu <ricliu@google.com>
Co-authored-by: Richard Liu <39319471+richardsliu@users.noreply.github.com>
Co-authored-by: Richard Liu <ricliu@google.com>
 2025-07-24 17:58:42 +01:00
Nick Hill eec6942014 [BugFix] Fix KVConnector TP worker aggregation (#21473) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-07-23 20:56:49 -07:00
Michael Goin f3137cdd81 [Core] Freeze gc during cuda graph capture to speed up init (#21146) 
Signed-off-by: Codex <codex@openai.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-07-23 17:20:14 -07:00
Michael Goin 82ec66f514 [V0 Deprecation] Remove Prompt Adapters (#20588) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-07-23 16:36:48 -07:00
22quinn 5c9b807b34 [Core] Add reload_weights RPC method (#20096) 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
 2025-07-23 14:24:52 -07:00
Christian Pinto 8560a5b258 [Core][Model] PrithviMAE Enablement on vLLM v1 engine (#20577) 
Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
 2025-07-23 11:00:23 -07:00
Cyrus Leung f59ec35b7f [V1] Check all pooling tasks during profiling (#21299) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-07-23 05:53:26 -07:00
Lu Fang accac82928 [Sampler] Introduce logprobs mode for logging (#21398) 
Signed-off-by: Lu Fang <lufang@fb.com>
 2025-07-23 01:39:25 -07:00
Chaojun Zhang 25d585ab7b [XPU] Enable external_launcher to serve as an executor via torchrun (#21021) 
Signed-off-by: chzhang <chaojun.zhang@intel.com>
 2025-07-21 21:47:35 -07:00
Lucas WilkinsonandMichael Goin 304dce7ec0 [Attention] Clean up iRoPE in V1 (#21188) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
 2025-07-21 09:10:30 -07:00
Cyrus Leung 042af0c8d3 [Model][1/N] Support multiple poolers at model level (#21227) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-07-21 02:22:21 -07:00
Chengji Yao 3a1d8940ae [TPU] support fp8 kv cache quantization (#19292) 
Signed-off-by: Chengji Yao <chengjiyao@google.com>
 2025-07-20 03:01:00 +00:00