iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
14,814 Commits 3 Branches 0 Tags
8b346309a5efbe80ee64f7d3633d2d7dedcc202b
Commit Graph
1056 Commits
Author SHA1 Message Date
Benjamin Chislett 8b346309a5 [Refactor] Consolidate SupportsEagle (#36063) 
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com>
 2026-03-13 23:22:40 +00:00
Ekagra Ranjan d0b402974f [Bugfix][Spec Decode] Avoid double call of Ngram CPU (#36952) 
Signed-off-by: Ekagra Ranjan <3116519+ekagra-ranjan@users.noreply.github.com>
 2026-03-13 20:33:19 +00:00
Nick Hill cd32d6f586 [Model Runner V2] Some code simplification (#36929) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-03-13 00:59:23 +00:00
Kunshang Ji 53ec16a705 [Hardware] Replace torch.cuda.device_count/current_device/set_device API (#36145) 
Signed-off-by: Kunshang Ji <jikunshang95@gmail.com>
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
 2026-03-12 07:57:47 -07:00
Woosuk Kwon 2f8b4ce0c0 [Model Runner V2] Do not initialize sampler for non-last PP ranks (#36824) 
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
 2026-03-12 03:55:28 +00:00
Wentao Ye c34ba6b961 [Perf] Optimize compute maxsim using batched version, 3.2% E2E throughput improvement (#36710) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-03-12 08:37:01 +08:00
Aaron Hao d6b61e5166 [BUG] Fix async rlhf tests (#35811) 
Signed-off-by: ahao-anyscale <ahao@anyscale.com>
 2026-03-11 18:06:10 -04:00
Woosuk Kwon 55eed6b7a5 [Model Runner V2] Add WhisperModelState [6/N] (#35790) 
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
 2026-03-11 14:20:38 -07:00
Giancarlo Delfin c77181e534 [Model Runner V2] Add probabilistic rejection sampling for spec decoding (#35461) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-03-11 14:04:32 -07:00
Or OzeriandNicolò Lucchesi a1a3523a56 [KVConnector] Support worker -> scheduler metadata (#31964) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
 2026-03-11 17:36:37 +00:00
Hongxin Xuandarlenxu bea02cdf93 Fix routed experts capture for hybrid models (Mamba + Attention) (#35744) 
Signed-off-by: arlenxu <arlenxu@tencent.com>
Signed-off-by: xhx1022 <1737006628@qq.com>
Co-authored-by: arlenxu <arlenxu@tencent.com>
 2026-03-11 08:53:10 -07:00
Woosuk Kwon 8ccbcda5c0 [Model Runner V2] Remove unused warmup_for_prefill method (#36762) 
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
 2026-03-11 08:02:44 -07:00
YiSheng5 c910eeb125 [XPU]Bug fix for some unexpected error when use AgRs backend on XPU device. (#36593) 
Signed-off-by: yisheng <yi.sheng@intel.com>
 2026-03-11 09:17:46 +00:00
Nicolò Lucchesi 098d844731 [NIXL][1/N] Refactor kernel_block_size detection (#35752) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2026-03-11 01:11:23 -07:00
Hongbin Guo 4bf533623b [Doc] Fix duplicate words in comments (#36713) 
Signed-off-by: Hongbin10 <jdmjdm1998@163.com>
 2026-03-10 21:28:31 -07:00
Matthew Bonanni 8ab3d7427c [Bugfix] Fix DeepSeek V3.2 OOM during CG memory profiling (#36691) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
 2026-03-11 03:01:07 +00:00
Nick Hill 65b2f405dc [Core] Simplify core kv-cache blocks initialization logic (#36521) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-03-10 20:20:02 +00:00
Woosuk Kwon f088a831dd [Model Runner V2] Use unpadded num_tokens for PW CUDA graph attn metadata (#36626) 
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
 2026-03-10 09:30:56 -07:00
Vadim Gimpelson 4ff8c3c8f9 [BUGFIX][Mamba][Qwen3.5] Zero freed SSM cache blocks on GPU (#35219) 
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>
 2026-03-10 03:32:20 -07:00
Nick Hill ddbb0d230a [Model Runner V2] Fix mm input embeddings lookup (#36588) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-03-10 00:24:58 -07:00
Nick Hill 9efc3bdcd6 [Model Runner V2] Fix _compute_slot_mappings_kernel for chunked prefill (#36580) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-03-10 00:23:42 -07:00
Wentao Ye 7279374f91 [Perf] Compute maxsim in worker side, reducing redundant copies, 2.7% E2E throughput improvement (#36159) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-03-09 20:55:58 -07:00
Woosuk Kwon 2a194ddd72 [Model Runner V2] Add model_state inputs to CUDA graph capture (#36544) 
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
 2026-03-09 15:14:51 -07:00
Lucas Wilkinson 483463f735 [MRV2] Extensible CG dispatch rework (#35959) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
 2026-03-09 13:58:45 -07:00
Woosuk Kwon 10a5f4d53d [Model Runner V2] Use NamedTuple for execute_model_state (#35930) 
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
 2026-03-09 11:17:34 -07:00
Woosuk Kwon 6e956d9eca [Model Runner V2] Add dummy profile_cudagraph_memory API (#36520) 
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
 2026-03-09 10:20:13 -07:00
Matthew Bonanni 00c4cb5606 [Bugfix] Clear stale CG keys after memory profiling (#36416) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
 2026-03-09 11:56:00 -04:00
Matthew Bonanni 77a73458e3 Reapply [Attention] Refactor check_and_update_config (#35122) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
 2026-03-09 07:17:14 -07:00
Li, Jiang 217f27598d [Bugfix] Avoid to replace non-tensor members in cpu model runner (#36430) 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
 2026-03-09 13:06:28 +08:00
Weiguang LiClaude Opus 4.6Li, Jiang <jiang1.li@intel.com>
 		43aa389231 [Bugfix] Fix CPU OMP autobind assertion to use local_world_size (#35815) 
Signed-off-by: liweiguang <codingpunk@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Li, Jiang <jiang1.li@intel.com>
 2026-03-08 20:07:29 -07:00
PatchyTIS a6be75dbd2 [Core] NGram GPU Implementation compatible with Async Scheduler (#29184) 2026-03-07 13:51:37 -08:00
Matthew Bonanni ebb9cc5f2b [UX][Startup] Account for CUDA graphs during memory profiling (#30515) 2026-03-07 13:49:23 -08:00
Nick Hill b354686524 [Model Runner V2] Fix warmup for pipeline parallel (#36280) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-03-06 16:58:51 -08:00
zhanqiuhuandNicolò Lucchesi 90f3c01fa4 [Spec Decode][KV Connector] Fix KV transfer in PD + speculative decoding (#35158) 
Signed-off-by: Claude <noreply@anthropic.com>
Signed-off-by: Zhanqiu Hu <zh338@cornell.edu>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
 2026-03-06 08:50:44 +01:00
Shiyan DengandLu Fang 03a49bb8f0 [Feature] Add --distributed-timeout-seconds CLI option (#36047) 
Signed-off-by: Shiyan Deng <dsy842974287@meta.com>
Co-authored-by: Lu Fang <30275821+houseroad@users.noreply.github.com>
 2026-03-05 20:57:51 -08:00
Cyrus Leung 6dd302653f [Misc] Rename group_mm_kwargs_by_modality -> group_and_batch_mm_kwargs (#36158) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-03-06 12:32:48 +08:00
Nick Hill a73af584fe [Model Runner V2] Fix warmup for very small kvcache and/or blocksizes (#36176) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-03-05 14:48:10 -08:00
Xinyu Chen d8839ef7d9 [XPU] Enable ModelRunnerV2 on XPU (#36078) 
Signed-off-by: Xinyu Chen <xinyu1.chen@intel.com>
 2026-03-05 17:19:18 +00:00
Jiayi YanandHarry Mellor 6a895197fa [Bugfix][CI] fix typos (#34934) 
Signed-off-by: 1195343015 <1195343015@qq.com>
Signed-off-by: Jiayi Yan <66017932+1195343015@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-03-05 17:05:46 +00:00
Kunshang Ji 66a2209645 [Hardware] Replace torch.cuda.synchronize() api with torch.accelerator.synchronize (#36085) 
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
 2026-03-05 10:36:39 +00:00
Nick Hill 16c472abe7 [Core] Move ray-specific WorkerWrapperBase methods to RayWorkerWrapper (#35328) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-03-05 12:11:59 +08:00
Nick Hill a3299c3d1d [Model Runner V2] Misc code simplification (#35941) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-03-04 15:26:35 -08:00
Thomas ParnellandClaude Opus 4.6 be0a3f7570 [Bugfix] Fix race in non-blocking num_accepted_tokens GPU->CPU copy (#36013) 
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-03-04 13:52:44 -08:00
Nick Hill 417fd28fb1 [Model Runner V2] Fix pooling (#36019) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-03-04 10:53:17 -08:00
Qi Wang 6aa6ad8992 [BugFix] Fix implicit and incorrect assumption on ECConnector is_producer (#34783) 
Signed-off-by: Qi Wang <qiwa@nvidia.com>
 2026-03-04 15:01:30 +01:00
haosdentandzjy0516 d6e04f4c43 [Bugfix] Cap FULL decode cudagraph sizes for Mamba/hybrid models (#34094) (#34571) 
Signed-off-by: haosdent <haosdent@gmail.com>
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com>
 2026-03-04 11:56:22 +01:00
Kunshang JiandHarry Mellor 16d2ad1d38 [Hardware] Replace torch.cuda.empty_cache with torch.accelerator.empty_cache (#30681) 
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Signed-off-by: Kunshang Ji <jikunshang95@gmail.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-03-04 09:49:47 +00:00
Nick Hill d15c3b90fc [Core] Move save_tensorized_model logic to Worker (#35825) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-03-03 15:31:59 -08:00
zhrrrandGiancarlo Delfin 97286a20ed [Model Runner V2] support dp & ep for spec decoding (#35294) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com>
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-03-03 15:19:45 -08:00
Woosuk Kwon 467886a0c4 [Model Runner V2] Fix inputs_embeds=None bug for MM models (#35917) 
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
 2026-03-03 13:47:45 -08:00