baonudesifeizhai
c3868bbbe4
[compile] Add FlashInfer FP8 async TP fusion and preserve allreduce fusion ordering #27893 ( #39505 )
...
Signed-off-by: baonudesifeizhai <baonudesifeizhai@gmail.com >
Signed-off-by: baonudesifeizhai <85092850+baonudesifeizhai@users.noreply.github.com >
Signed-off-by: roG0d <baonudesifeizhai@gmail.com >
2026-05-01 05:08:34 +00:00
Yongye Zhu
5aa371dc8e
[DSV4] Enable Multi-stream for Pre-Attn GEMM ( #41061 )
...
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
2026-04-28 09:08:55 -07:00
Zhanda Zhu and Yubo Gao
5d5c776444
[Perf] FP8 FlashInfer Attn for ViT ( #38065 )
...
Signed-off-by: Zhanda Zhu <zhandazhu@gmail.com >
Co-authored-by: Yubo Gao <ybgao-nvidia@users.noreply.github.com >
2026-04-27 13:44:15 +08:00
+7
4d51588e23
[Feat] DeepSeek V4 Rebased ( #40860 )
...
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Signed-off-by: qizixi <zixi@inferact.ai >
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com >
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Yongye Zhu <yongye@inferact.ai >
Co-authored-by: Simon Mo <simon@inferact.ai >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai >
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: Roy Wang <yasong.wang@inferact.ai >
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: youkaichao <youkaichao@gmail.com >
Co-authored-by: Zhewen Li <jerven.vllm@gmail.com >
Co-authored-by: Zijing Liu <liuzijing2014@gmail.com >
Co-authored-by: khluu <khluu000@gmail.com >
Co-authored-by: qizixi <zixi@inferact.ai >
Co-authored-by: Zhewen Li <zhewenli@inferact.ai >
2026-04-26 18:31:08 -07:00
Andreas Karatzas
95995bbef8
[ROCm][Engine] Fix GPU memory leaks in engine shutdown and test workaround for async KV prefix cache reset ( #38503 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
2026-04-25 05:25:20 +00:00
lyd1992
100c7b65e7
[Platform] Fix RISC-V platform detection (lscpu parsing + non-NUMA meminfo) ( #40427 )
...
Signed-off-by: liuyudong <liuyudong@iscas.ac.cn >
2026-04-24 04:33:05 +00:00
Neil Schemenauer
56bdf85e10
[Feature] Avoid eager import of the "mistral_common" package. ( #40043 )
...
Signed-off-by: Neil Schemenauer <nas@arctrix.com >
2026-04-24 02:49:16 +00:00
Nicolò Lucchesi
9c2492e501
[Misc] Support Human-readable (k/K/m/M..) json cli arg ( #40473 )
...
Signed-off-by: NickLucche <nlucches@redhat.com >
2026-04-23 09:42:23 +02:00
Wentao Ye and gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
8f87eb4622
[Refactor] Clean up log once scope="local" ( #40540 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com >
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-04-22 16:42:43 -04:00
bnellnm
5e584ce9ec
[MoE Refactor] Remove SharedFusedMoE class ( #35782 )
...
Signed-off-by: Bill Nell <bnell@redhat.com >
2026-04-21 18:12:12 -04:00
Vadim Gimpelson
6d85b36a9f
Revert #38730 and #38791 ( #40032 )
...
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com >
Signed-off-by: Vadim Gimpelson <156319763+vadiklyutiy@users.noreply.github.com >
2026-04-21 11:44:11 -04:00
SeongJun Lee
989cc12d88
[Fix] Add missing space in IP fallback warning ( #40359 )
...
Signed-off-by: lesj0610 <lesj0610@gmail.com >
2026-04-20 20:26:06 -07:00
Wentao Ye
301024aa9c
[Deprecation] Deprecate cprofile and cprofile_context ( #39100 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-04-21 11:25:22 +08:00
bai
191e3fdaa1
Update flashinfer to 0.6.8 ( #39959 )
...
Signed-off-by: bai <v@gor.io >
2026-04-20 10:37:23 -07:00
Galigator and kse
a023edfa5b
[bugfix] Use only onlines CPUs in lscpu ( #40161 )
...
Signed-off-by: kse <kevin.sejourne@cloud-temple.com >
Co-authored-by: kse <kevin.sejourne@cloud-temple.com >
2026-04-20 13:19:57 +00:00
Nithin Chalapathi and Cyrus Leung
80b18230e0
[Frontend] Add multimodal support to /inference/v1/generate endpoint ( #38405 )
...
Signed-off-by: Nithin Chalapathi <nithin.ch10@gmail.com >
Signed-off-by: Nithin Chalapathi <nithinc@berkeley.edu >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-04-17 20:31:56 -07:00
sychen52
6b2b7bd0eb
Add nvfp4 support to reshape_and_cache_flash ( #37332 )
...
Signed-off-by: Shiyang Chen <shiychen@nvidia.com >
2026-04-17 07:28:00 -07:00
Li, Jiang
d02421a7db
[CPU] Refactor CPU affinity and memory management ( #39781 )
...
Signed-off-by: jiang1.li <jiang1.li@intel.com >
2026-04-17 21:01:08 +08:00
f4b42df048
[Attention Backend] TurboQuant: 2-bit KV cache compression with 4x capacity ( #38479 )
...
Signed-off-by: vibhavagarwal5 <vibhavagarwal5@gmail.com >
Signed-off-by: Michael Goin <mgoin64@gmail.com >
Co-authored-by: Xinyu Chen <xinyu1.chen@intel.com >
Co-authored-by: Michael Goin <mgoin64@gmail.com >
2026-04-14 19:57:13 -07:00
haosdent and DarkLight1337
995e9a209e
[Bugfix] Use is_integrated to detect UMA GPUs for memory reporting ( #35356 )
...
Signed-off-by: haosdent <haosdent@gmail.com >
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk >
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk >
2026-04-13 11:07:40 -07:00
z1ying
a3e750c0a5
[Misc] Update deprecation warning for --model flag ( #39518 )
...
Signed-off-by: Ziying Tao <tzzying@outlook.com >
2026-04-10 23:25:20 -07:00
Richard Zou
f44afef6d6
[compile] Allow strings in custom ops without regressing compilation times ( #38123 )
...
Signed-off-by: Richard Zou <zou3519@gmail.com >
2026-04-10 07:26:37 +00:00
Yan Ma
ec68d53b2b
Add platform manual_seed_all API ( #38468 )
...
Signed-off-by: Yan Ma <yan.ma@intel.com >
2026-04-10 13:43:50 +08:00
Ganesh R
445a2a4d1a
feat(cpu): add CPU support for draft model speculative decoding ( #32662 )
...
Signed-off-by: R <Ganesh.R@amd.com >
2026-04-10 11:49:52 +08:00
lalit10 and DarkLight1337
91eea72330
[Tests] Add Qwen3-VL multimodal memory leak check ( #39268 )
...
Signed-off-by: Lalit Laxminarayan Bangad <lalitbangad@gmail.com >
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk >
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk >
2026-04-09 04:54:46 -07:00
Michael Goin and Claude
eb4205fee5
[UX] Integrate DeepGEMM into vLLM wheel via CMake ( #37980 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Signed-off-by: Michael Goin <mgoin64@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-04-08 18:56:32 -07:00
Maral
2e9034c998
[W8A8 Block Linear Refactor][2/N] Remove W8A8Fp8BlockLinearOp and adopt Fp8 block linear kernel selections. ( #33892 )
...
Signed-off-by: maral <maralbahari.98@gmail.com >
Signed-off-by: Maral <maralbahari.98@gmail.com >
2026-04-09 08:50:39 +08:00
Richard Zou
ba4a78eb5d
[torch.compile] Allow usage of Opaque Objects in PyTorch 2.11 ( #39286 )
...
Signed-off-by: Richard Zou <zou3519@gmail.com >
2026-04-08 23:21:10 +00:00
75e01a39a1
[Feature] NUMA binding support for GPU workers ( #38635 )
...
Signed-off-by: Shengqi Chen <harry-chen@outlook.com >
Co-authored-by: Jason Li <jasonlizhengjian@gmail.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
2026-04-08 09:55:24 -07:00
Andrey Talman
2111997f96
[release 2.11] Update to torch 2.11 ( #34644 )
2026-04-07 18:55:48 -07:00
fxmarty-amd and Kyle Sayers
00d7b497b3
[NVFP4] Support NVFP4 dense models from modelopt and compressed-tensors on AMD Instinct MI300, MI355X and Hopper through emulation ( #35733 )
...
Signed-off-by: Felix Marty <Felix.Marty@amd.com >
Signed-off-by: fxmarty-amd <felmarty@amd.com >
Co-authored-by: Kyle Sayers <kylesayrs@gmail.com >
2026-04-06 16:18:27 -06:00
e8ebbdde83
[Quantization] Add FlashInfer CuteDSL batched experts backend for NVFP4 MoE ( #38251 )
...
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Michael Goin <mgoin64@gmail.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
2026-04-06 11:57:53 -07:00
Ziming Qi
2a36d8fb72
[Bugfix][CPU] Fix macOS compatibility broken by #36487 ( #38970 )
...
Signed-off-by: Ziming (2imi9) <148090931+2imi9@users.noreply.github.com >
2026-04-04 14:05:58 +00:00
Anton Ivanov
abebd9323d
[CPU] Replace OMP initialization ( #36487 )
...
Signed-off-by: Anton Ivanov <anton.ivanov@cambridgegreys.com >
2026-04-03 18:42:43 +08:00
2ce3d0ce36
[Feature] KV cache per-token-head INT8/FP8 quantization ( #38378 )
...
Signed-off-by: JartX <sagformas@epdcenter.es >
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com >
Co-authored-by: yangyang4991 <yangyang4991@gmail.com >
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com >
Co-authored-by: Isotr0py <2037008807@qq.com >
2026-04-02 08:13:26 -04:00
Stefano Castagnetta
6183cae1bd
[Bugfix] Restrict TRTLLM attention to SM100, fixing GB300 (SM103) hang ( #38730 )
...
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com >
2026-04-01 12:08:40 -07:00
Matthew Bonanni
116f4be405
[1/N][Cleanup] Standardize on use of is_quantized_kv_cache ( #38659 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
2026-04-01 04:08:01 +00:00
d9b90a07ac
[MoE Refactor] Migrate Unquantized to Full Oracle Flow ( #36286 )
...
Signed-off-by: Yifan Zong <yzong@redhat.com >
Signed-off-by: Robert Shaw <robshaw@redhat.com >
Signed-off-by: yzong-rh <yzong@redhat.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
Co-authored-by: Robert Shaw <robshaw@redhat.com >
2026-03-31 15:43:33 -04:00
4dfad17ed1
replace cuda_device_count_stateless() to current_platform.device_count() ( #37841 )
...
Signed-off-by: Liao, Wei <wei.liao@intel.com >
Signed-off-by: wliao2 <wei.liao@intel.com >
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com >
2026-03-31 22:32:54 +08:00
Harry Mellor and gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
21d2b53f88
Remove need for explicit \n in docstring lists for --help formatting ( #38350 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-03-27 08:38:00 -07:00
Bvicii and Cyrus Leung
999dfc1622
[Bugfix] Offload blocking tokenizer ops to shared thread pool to unblock event loop ( #34789 )
...
Signed-off-by: Bvicii <yizhanhuang2002@gmail.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-03-26 22:17:00 -07:00
Zhewen Li and Zhewen Li
be1a85b7a2
Revert "[MoE Kernel] Flashinfer nvfp4 cutedsl moe kernel integration" ( #38050 ) ( #38169 )
...
Co-authored-by: Zhewen Li <zhewenli@inferact.ai >
2026-03-26 07:59:09 -07:00
Wentao Ye
bf5eec638d
[Refactor] Remove unused utils ( #38153 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-03-26 17:08:19 +08:00
Vadim Gimpelson
52069012fe
[Bugfix] Fix DeepGemm E8M0 accuracy degradation for Qwen3.5 FP8 on Blackwell ( #38083 )
...
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com >
2026-03-26 01:21:47 -07:00
Andreas Karatzas
36f6aede23
[Misc] Optimized check to encapsulate both CUDA and ROCm platforms ( #34549 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
2026-03-26 09:43:07 +08:00
Yongye Zhu
678b3c99e8
[MoE Kernel] Flashinfer nvfp4 cutedsl moe kernel integration ( #38050 )
2026-03-25 10:16:40 -07:00
Chauncey
09c3dc9186
[Revert] Remove CUDA torch fallbacks for fp8_mqa_logits/fp8_paged_mqa_logits_torch function ( #37968 )
...
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com >
2026-03-25 06:19:37 +00:00
Li, Jiang
352b90c4a4
[Bugfix] Add replacement of _compute_slot_mapping_kernel on CPU ( #37987 )
...
Signed-off-by: jiang1.li <jiang1.li@intel.com >
2026-03-24 07:00:20 -07:00
Ranran and Wentao Ye
dc6908ac6a
[Bugfix] Register VLLM_BATCH_INVARIANT in envs.py to fix spurious unknown env var warning ( #35007 )
...
Signed-off-by: Ranran <1012869439@qq.com >
Signed-off-by: Ranran <hzz5361@psu.edu >
Signed-off-by: ran <hzz5361@psu.edu >
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com >
2026-03-23 18:31:14 -04:00
yzong-rh
e85f8f0932
[Bug][MoE] Strengthen _supports_current_device() checks in the TRTLLM FP8, NVFP4, and FlashInfer CuteDSL MoE experts ( #36728 )
...
Signed-off-by: Yifan Zong <yzong@redhat.com >
2026-03-23 17:02:57 -04:00
baonudesifeizhai