iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
15,858 Commits 3 Branches 0 Tags
bf45e6d0a558da2b8d7b60efb07b4aa394f3b60b
Commit Graph
324 Commits
Author SHA1 Message Date
Vibhav AgarwalXinyu ChenMichael Goin
 		f4b42df048 [Attention Backend] TurboQuant: 2-bit KV cache compression with 4x capacity (#38479) 
Signed-off-by: vibhavagarwal5 <vibhavagarwal5@gmail.com>
Signed-off-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Xinyu Chen <xinyu1.chen@intel.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
 2026-04-14 19:57:13 -07:00
haosdentandDarkLight1337 995e9a209e [Bugfix] Use is_integrated to detect UMA GPUs for memory reporting (#35356) 
Signed-off-by: haosdent <haosdent@gmail.com>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-04-13 11:07:40 -07:00
z1ying a3e750c0a5 [Misc] Update deprecation warning for --model flag (#39518) 
Signed-off-by: Ziying Tao <tzzying@outlook.com>
 2026-04-10 23:25:20 -07:00
Richard Zou f44afef6d6 [compile] Allow strings in custom ops without regressing compilation times (#38123) 
Signed-off-by: Richard Zou <zou3519@gmail.com>
 2026-04-10 07:26:37 +00:00
Yan Ma ec68d53b2b Add platform manual_seed_all API (#38468) 
Signed-off-by: Yan Ma <yan.ma@intel.com>
 2026-04-10 13:43:50 +08:00
Ganesh R 445a2a4d1a feat(cpu): add CPU support for draft model speculative decoding (#32662) 
Signed-off-by: R <Ganesh.R@amd.com>
 2026-04-10 11:49:52 +08:00
lalit10andDarkLight1337 91eea72330 [Tests] Add Qwen3-VL multimodal memory leak check (#39268) 
Signed-off-by: Lalit Laxminarayan Bangad <lalitbangad@gmail.com>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-04-09 04:54:46 -07:00
Michael GoinandClaude eb4205fee5 [UX] Integrate DeepGEMM into vLLM wheel via CMake (#37980) 
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-04-08 18:56:32 -07:00
Maral 2e9034c998 [W8A8 Block Linear Refactor][2/N] Remove W8A8Fp8BlockLinearOp and adopt Fp8 block linear kernel selections. (#33892) 
Signed-off-by: maral <maralbahari.98@gmail.com>
Signed-off-by: Maral <maralbahari.98@gmail.com>
 2026-04-09 08:50:39 +08:00
Richard Zou ba4a78eb5d [torch.compile] Allow usage of Opaque Objects in PyTorch 2.11 (#39286) 
Signed-off-by: Richard Zou <zou3519@gmail.com>
 2026-04-08 23:21:10 +00:00
Shengqi ChenJason LiRoger Wang
 		75e01a39a1 [Feature] NUMA binding support for GPU workers (#38635) 
Signed-off-by: Shengqi Chen <harry-chen@outlook.com>
Co-authored-by: Jason Li <jasonlizhengjian@gmail.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2026-04-08 09:55:24 -07:00
Andrey Talman 2111997f96 [release 2.11] Update to torch 2.11 (#34644) 2026-04-07 18:55:48 -07:00
fxmarty-amdandKyle Sayers 00d7b497b3 [NVFP4] Support NVFP4 dense models from modelopt and compressed-tensors on AMD Instinct MI300, MI355X and Hopper through emulation (#35733) 
Signed-off-by: Felix Marty <Felix.Marty@amd.com>
Signed-off-by: fxmarty-amd <felmarty@amd.com>
Co-authored-by: Kyle Sayers <kylesayrs@gmail.com>
 2026-04-06 16:18:27 -06:00
Yongye ZhuMichael GoinRoger Wang
 		e8ebbdde83 [Quantization] Add FlashInfer CuteDSL batched experts backend for NVFP4 MoE (#38251) 
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2026-04-06 11:57:53 -07:00
Ziming Qi 2a36d8fb72 [Bugfix][CPU] Fix macOS compatibility broken by #36487 (#38970) 
Signed-off-by: Ziming (2imi9) <148090931+2imi9@users.noreply.github.com>
 2026-04-04 14:05:58 +00:00
Anton Ivanov abebd9323d [CPU] Replace OMP initialization (#36487) 
Signed-off-by: Anton Ivanov <anton.ivanov@cambridgegreys.com>
 2026-04-03 18:42:43 +08:00
JartXyangyang4991Wentao YeIsotr0py
 		2ce3d0ce36 [Feature] KV cache per-token-head INT8/FP8 quantization (#38378) 
Signed-off-by: JartX <sagformas@epdcenter.es>
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: yangyang4991 <yangyang4991@gmail.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: Isotr0py <2037008807@qq.com>
 2026-04-02 08:13:26 -04:00
Stefano Castagnetta 6183cae1bd [Bugfix] Restrict TRTLLM attention to SM100, fixing GB300 (SM103) hang (#38730) 
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com>
 2026-04-01 12:08:40 -07:00
Matthew Bonanni 116f4be405 [1/N][Cleanup] Standardize on use of is_quantized_kv_cache (#38659) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
 2026-04-01 04:08:01 +00:00
yzong-rhRobert ShawRobert Shaw
 		d9b90a07ac [MoE Refactor] Migrate Unquantized to Full Oracle Flow (#36286) 
Signed-off-by: Yifan Zong <yzong@redhat.com>
Signed-off-by: Robert Shaw <robshaw@redhat.com>
Signed-off-by: yzong-rh <yzong@redhat.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Robert Shaw <robshaw@redhat.com>
 2026-03-31 15:43:33 -04:00
wliao2Harry MellorKunshang Ji
 		4dfad17ed1 replace cuda_device_count_stateless() to current_platform.device_count() (#37841) 
Signed-off-by: Liao, Wei <wei.liao@intel.com>
Signed-off-by: wliao2 <wei.liao@intel.com>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
 2026-03-31 22:32:54 +08:00
Harry Mellorandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> 21d2b53f88 Remove need for explicit \n in docstring lists for --help formatting (#38350) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-03-27 08:38:00 -07:00
BviciiandCyrus Leung 999dfc1622 [Bugfix] Offload blocking tokenizer ops to shared thread pool to unblock event loop (#34789) 
Signed-off-by: Bvicii <yizhanhuang2002@gmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2026-03-26 22:17:00 -07:00
Zhewen LiandZhewen Li be1a85b7a2 Revert "[MoE Kernel] Flashinfer nvfp4 cutedsl moe kernel integration" (#38050) (#38169) 
Co-authored-by: Zhewen Li <zhewenli@inferact.ai>
 2026-03-26 07:59:09 -07:00
Wentao Ye bf5eec638d [Refactor] Remove unused utils (#38153) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-03-26 17:08:19 +08:00
Vadim Gimpelson 52069012fe [Bugfix] Fix DeepGemm E8M0 accuracy degradation for Qwen3.5 FP8 on Blackwell (#38083) 
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>
 2026-03-26 01:21:47 -07:00
Andreas Karatzas 36f6aede23 [Misc] Optimized check to encapsulate both CUDA and ROCm platforms (#34549) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
 2026-03-26 09:43:07 +08:00
Yongye Zhu 678b3c99e8 [MoE Kernel] Flashinfer nvfp4 cutedsl moe kernel integration (#38050) 2026-03-25 10:16:40 -07:00
Chauncey 09c3dc9186 [Revert] Remove CUDA torch fallbacks for fp8_mqa_logits/fp8_paged_mqa_logits_torch function (#37968) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
 2026-03-25 06:19:37 +00:00
Li, Jiang 352b90c4a4 [Bugfix] Add replacement of _compute_slot_mapping_kernel on CPU (#37987) 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
 2026-03-24 07:00:20 -07:00
RanranandWentao Ye dc6908ac6a [Bugfix] Register VLLM_BATCH_INVARIANT in envs.py to fix spurious unknown env var warning (#35007) 
Signed-off-by: Ranran <1012869439@qq.com>
Signed-off-by: Ranran <hzz5361@psu.edu>
Signed-off-by: ran <hzz5361@psu.edu>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
 2026-03-23 18:31:14 -04:00
yzong-rh e85f8f0932 [Bug][MoE] Strengthen _supports_current_device() checks in the TRTLLM FP8, NVFP4, and FlashInfer CuteDSL MoE experts (#36728) 
Signed-off-by: Yifan Zong <yzong@redhat.com>
 2026-03-23 17:02:57 -04:00
SherryC41andsherryC41 6ec5e9fd37 refactor: abstract deepgemm support into platform (#37519) 
Co-authored-by: sherryC41 <sherry.c.c41@gmail.com>
 2026-03-20 17:54:08 +00:00
Itay Alroy ca1ac1a4b4 Fix DP coordinator ZMQ port TOCTOU (#37452) 
Signed-off-by: Itay Alroy <ialroy@nvidia.com>
 2026-03-20 00:58:31 +00:00
Xin Yang f1740006e4 [Perf] Enable dual stream execution of input projection for Qwen3 (#36795) 
Signed-off-by: Xin Yang <xyangx@amazon.com>
 2026-03-18 11:13:27 +08:00
Kunshang Ji 747b068136 [Hardware] Replace memory related torch.cuda APIs (#37031) 
Signed-off-by: Kunshang Ji <jikunshang95@gmail.com>
 2026-03-16 10:24:48 +00:00
leo-cf-tianwzhao18Stefano Castagnettaroot
 		2754231ba3 [Kernel] Add FlashInfer MoE A2A Kernel (#36022) 
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
Signed-off-by: Leo Tian <lctian@nvidia.com>
Co-authored-by: wzhao18 <wzhao18.sz@gmail.com>
Co-authored-by: Stefano Castagnetta <scastagnetta@nvidia.com>
Co-authored-by: root <root@lyris0267.lyris.clusters.nvidia.com>
 2026-03-15 23:45:32 -07:00
Itay AlroyYongji WuRon Tourgeman
 		d5af196c18 [2/N] Elastic EP Milestone 2: Integrating NIXL-EP (#35627) 
Signed-off-by: Itay Alroy <ialroy@nvidia.com>
Co-authored-by: Yongji Wu <wuyongji317@gmail.com>
Co-authored-by: Ron Tourgeman <rtourgeman@nvidia.com>
 2026-03-13 09:25:33 -04:00
Kunshang Ji 53ec16a705 [Hardware] Replace torch.cuda.device_count/current_device/set_device API (#36145) 
Signed-off-by: Kunshang Ji <jikunshang95@gmail.com>
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
 2026-03-12 07:57:47 -07:00
Nick Hill 36735fd772 [BugFix] Fix multiple/duplicate stdout prefixes (#36822) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-03-12 12:23:21 +08:00
Vadim Gimpelson 4ff8c3c8f9 [BUGFIX][Mamba][Qwen3.5] Zero freed SSM cache blocks on GPU (#35219) 
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>
 2026-03-10 03:32:20 -07:00
wang.yuqigemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>Harry Mellor
 		dcf8862fd4 [Examples][1/n] Resettle basic examples. (#35579) 
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
Signed-off-by: wang.yuqi <noooop@126.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-03-08 20:22:53 -07:00
Walter Beller-Morales 43e77e59ab [BugFix] avoid infinite loop with VLLM_PORT and get_open_ports_list (#36191) 
Signed-off-by: walterbm <walter.beller.morales@gmail.com>
 2026-03-05 22:15:29 -08:00
Kunshang JiandHarry Mellor 16d2ad1d38 [Hardware] Replace torch.cuda.empty_cache with torch.accelerator.empty_cache (#30681) 
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Signed-off-by: Kunshang Ji <jikunshang95@gmail.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-03-04 09:49:47 +00:00
Robert ShawRobert ShawRobert Shaw
 		97995f6376 [MoE Refactor] Create MK for TRTLLM Kernels (#32564) 
Signed-off-by: Robert Shaw <robshaw@redhat.com>
Signed-off-by: Robert Shaw <rshaw@neuralmagic.com>
Signed-off-by: Robert Shaw <robertgshaw2@gmail.com>
Co-authored-by: Robert Shaw <robshaw@redhat.com>
Co-authored-by: Robert Shaw <rshaw@neuralmagic.com>
 2026-03-03 10:39:50 -08:00
Richard Zou e82fbeec7b [torch.compile] Undo the fast_moe_cold_start hack in torch>=2.11 (#35475) 
Signed-off-by: Richard Zou <zou3519@gmail.com>
 2026-03-01 21:44:22 +00:00
Chauncey 7e08c22b8c [Feat] Add CUDA torch fallbacks for fp8_mqa_logits/fp8_paged_mqa_logits_torch function (#35271) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
 2026-02-28 10:12:00 +00:00
Andreas Karatzas 94029ffaf0 [ROCm] Derive device capability from GCN arch string without CUDA init (#35069) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
 2026-02-28 13:55:28 +08:00
Michael Goin 4fec53cfcb [CI] Actually run tests/kernels/quantization/test_block_fp8.py in CI (#34274) 2026-02-26 17:58:03 -07:00
Tyler Michael SmithandClaude Opus 4.6 eb19955c37 [WideEP] Remove pplx all2all backend (#33724) 
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com>
 2026-02-26 14:30:10 -08:00