Or Ozeri
2fa1f8ec00
[kv_offload+HMA][13/N]: Enable HMA support ( #41445 )
...
This is the final PR in a series to enables HMA support for the
offloading connector. The connector advertises `SupportsHMA`
and is validated with unit tests and e2e tests.
Signed-off-by: Or Ozeri <oro@il.ibm.com >
2026-05-01 12:30:03 +01:00
raviguptaamd
7075df79b3
[ROCm] Enable DBO (Dynamic Batch Optimization) on ROCm ( #34726 )
...
Signed-off-by: raviguptaamd <ravi.gupta@amd.com >
2026-05-01 09:18:30 +00:00
Yuyi Ao
0dbaf9daad
Refractor longcat loading to use AutoWeightsLoader ( #41448 )
...
Signed-off-by: George-ao <yuyiao772@gmail.com >
2026-05-01 09:07:23 +00:00
a3ec4a35f5
[Bugfix][Metrics] Fix RayPrometheusMetric.labels() returning shared labeled child ( #40840 )
...
When vLLM runs with Ray Prometheus `vllm:request_success{finished_reason=...}`
only ever increments the repetition bucket regardless of the request's actual finish
reason; stop, length, abort, and error stay at zero. Root cause was `labels()` mutated
the wrapped Ray metric's default tags in place and returned self, so every `.labels(...)`
call on a given wrapper returned the same object.
Co-authored-by: Marwan Sarieddine <sarieddine.marwan@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com >
Signed-off-by: Seiji Eicher <seiji@anyscale.com >
2026-05-01 08:43:39 +01:00
Bugen Zhao and wang.yuqi
a07642667d
[Bugfix] Pass reasoning parser kwargs to structured output ( #41199 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-04-30 23:38:02 -07:00
baonudesifeizhai
c3868bbbe4
[compile] Add FlashInfer FP8 async TP fusion and preserve allreduce fusion ordering #27893 ( #39505 )
...
Signed-off-by: baonudesifeizhai <baonudesifeizhai@gmail.com >
Signed-off-by: baonudesifeizhai <85092850+baonudesifeizhai@users.noreply.github.com >
Signed-off-by: roG0d <baonudesifeizhai@gmail.com >
2026-05-01 05:08:34 +00:00
sychen52
947138b6c2
Add nvfp4 kv cache support ( #40177 )
...
Signed-off-by: Shiyang Chen <shiychen@nvidia.com >
2026-05-01 04:55:16 +00:00
Or Ozeri
941fb50835
[kv_offload+HMA][12/N]: Scheduler-side support for sliding window groups ( #41228 )
...
Signed-off-by: Or Ozeri <oro@il.ibm.com >
2026-05-01 06:59:17 +03:00
Juhi Mittal and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
6b6ac6c3c7
[Kernel][MoE] Support GELU on TRT-LLM NvFP4 fused MoE for Gemma4 ( #41050 )
...
Signed-off-by: Juhi Mittal <juhim@nvidia.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-01 03:37:43 +00:00
Stefano Castagnetta
b542bdf7fb
[Bugfix] Disable FlashInfer CUTLASS MoE on SM110 (Jetson Thor AGX) ( #40808 )
...
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com >
2026-04-30 20:08:49 -07:00
Ronen Schaffer
415a879899
[KV Offload] Use Collection instead of Sequence/Iterable for OffloadingManager key parameters ( #41361 )
...
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com >
2026-05-01 05:18:38 +03:00
Dong W
7198940b39
[Model] Add Moondream3 model support(only query and caption skills) ( #32325 )
...
Signed-off-by: Dong Wang <dongw2019@gmail.com >
2026-05-01 10:06:48 +08:00
Luis 🚀 and Andrew Sansom
14043dfecd
feat: Enable prompt_embeds Content Part Support in vLLM Chat Completions API ( #40720 )
...
Signed-off-by: Luis Robaina <luis@protopia.ai >
Signed-off-by: Luis Robaina 🚀 <luisfabian1545@gmail.com >
Signed-off-by: LuisRobaina <luis@protopia.ai >
Co-authored-by: Andrew Sansom <qthequartermasterman@gmail.com >
2026-05-01 10:05:55 +08:00
Soyaazz and Matthew Bonanni
4d5c89295b
(bugfix): block_size check for flex attn ( #41363 )
...
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com >
2026-04-30 18:59:26 -07:00
Nick Hill
dd5506a157
[Core] Simplify handling of scheduler_reserve_full_isl option ( #41064 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-04-30 18:10:00 -07:00
Woosuk Kwon
9c61864bf8
[DeepSeek] Use torch.mm for bf16xbf16->fp32 gemm ( #41300 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
2026-04-30 16:28:57 -07:00
Tran Le
71725f6730
[Bugfix] Fix RoutedExpertsCapturer for Gemma 4 MoE (top_k_experts) ( #41401 )
...
Signed-off-by: Tran Le <tranle@fireworks.ai >
2026-04-30 16:19:59 -07:00
b4806c8ee1
[DSV4] Add BF16 and MXFP8 A2A support for flashinfer a2a one sided ( #40960 )
...
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Signed-off-by: Zijing Liu <liuzijing2014@gmail.com >
Co-authored-by: Zijing Liu <liuzijing2014@users.noreply.github.com >
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-04-30 15:33:12 -07:00
Wentao Ye
526927be94
[Model Runner v2] Fix v2 compile counter num_gpu_runner_capture_triggers and num_cudagraph_captured ( #41285 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-04-30 15:20:11 -07:00
Michael Goin
75a4c166f2
Fix typo in log message for indexer cache ( #41419 )
...
Signed-off-by: Michael Goin <mgoin64@gmail.com >
2026-04-30 15:02:14 -07:00
fxmarty-amd and Claude
2917d6363a
[NVFP4][Hopper/AMD Instinct] Add Triton kernels for NVFP4 dequantization and QDQ emulation ( #40033 )
...
Signed-off-by: Felix Marty <Felix.Marty@amd.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-04-30 17:35:48 -04:00
Jee Jee Li
307b17ce33
[DSV4] Avoid redundant dtype conversion. ( #41374 )
...
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com >
2026-04-30 09:57:27 -07:00
121dbe7a22
[ROCm] ROCm DeepEP API updated to latest ( #39721 )
...
Signed-off-by: Tej Kiran <vpolamre@amd.com >
Signed-off-by: tej <37236721+itej89@users.noreply.github.com >
Co-authored-by: TJian <tunjian.tan@embeddedllm.com >
Co-authored-by: HAIAI <39548240+HAIAI@users.noreply.github.com >
2026-04-30 07:46:59 -07:00
Matthew Bonanni
f03d82efdd
[UX][Bugfix] Fix OOM by setting PyTorch max_split_size_mb during model loading ( #41268 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
2026-04-30 07:46:54 -07:00
a7fb008510
[EPLB] Optimize memory overhead in Nixl communicator ( #40013 )
...
Signed-off-by: ilmarkov <markovilya197@gmail.com >
Signed-off-by: Markov Ilya <markovilya19@gmail.com >
Co-authored-by: Markov Ilya <markovilya19@gmail.com >
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com >
2026-04-30 07:46:49 -07:00
b55b26520c
[MoE] Make MoERunnerInterface a PluggableLayer for OOT support ( #35178 )
...
Signed-off-by: wxsIcey <1790571317@qq.com >
Signed-off-by: Icey <1790571317@qq.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-04-30 03:31:08 -07:00
snadampal
3179e53135
[P/D] Prefill compute optimizations with bi-directional KV cache transfers between P and D nodes ( #32553 )
...
Signed-off-by: Sunita Nadampalli <nadampal@amazon.com >
2026-04-30 10:14:20 +00:00
Nicolò Lucchesi
efdc95674d
[KVConnector] MultiConnector SupportsHMA ( #39571 )
...
Signed-off-by: NickLucche <nlucches@redhat.com >
2026-04-30 02:10:50 -07:00
Chenxi Qian and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
54146a9bf9
[Bugfix] correct h matrix layout in chunk_kda output kernel ( #40956 )
...
Signed-off-by: ChenxiQian <chenxi.qian.cq@outlook.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-04-30 16:22:41 +08:00
ca97f7b9bb
Fix Gemma4 MoE expert weight remapping ( #41206 )
...
Signed-off-by: sunghoon.baek <sunghoon.baek@connectfy.cloud >
Co-authored-by: sunghoon.baek <sunghoon.baek@connectfy.cloud >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-04-30 00:12:42 -07:00
Ekagra Ranjan
a04e0cf3b8
Fix Cohere ASR after HF upgrade ( #40582 )
...
Signed-off-by: Ekagra Ranjan <3116519+ekagra-ranjan@users.noreply.github.com >
2026-04-29 23:39:04 -07:00
cb1b02d0e8
[Frontend] Add VLLM_SKIP_MODEL_NAME_VALIDATION environment variable ( #34676 )
...
Signed-off-by: Dhruv Singal <dhruvsingalabc@gmail.com >
Signed-off-by: Dhruv Singal <dsingal@Dhruvs-MacBook-Pro.local >
Signed-off-by: Your Name <you@example.com >
Signed-off-by: vLLM Assistant <assistant@vllm.ai >
Signed-off-by: Simon Mo <simon.mo@hey.com >
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com >
Co-authored-by: Dhruv Singal <dsingal@Dhruvs-MacBook-Pro.local >
Co-authored-by: Your Name <you@example.com >
Co-authored-by: OpenCode <noreply@openai.com >
Co-authored-by: Simon Mo <simon.mo@hey.com >
2026-04-29 23:19:09 -07:00
Martin Hickey and Or Ozeri
c42981d034
[Refactor][kv_offload] KV Offloading maintainability improvements ( #40538 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
Co-authored-by: Or Ozeri <oro@il.ibm.com >
2026-04-30 05:55:31 +03:00
Wei Zhao
0ff1bf9bb1
[Bugfix] Fix failure to allocate KV blocks error ( #41282 )
...
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com >
2026-04-29 18:44:07 -07:00
Rohan Potdar
3795d7acf4
[ROCm][Bugfix][GPTOSS]: fix input_ids and expert_map args for quark w4a8 gptoss ( #41165 )
...
Signed-off-by: Rohan138 <rohanpotdar138@gmail.com >
2026-04-29 16:39:01 -07:00
Thien Tran
296741d025
[DSv4] Use cvt PTX for FP32->FP4 conversion ( #41015 )
...
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg >
2026-04-29 16:16:40 -07:00
a966aaed30
[Bugfix][MLA] Size arange_buffer to max_num_batched_tokens to prevent CUDA IMA ( #39277 )
...
Signed-off-by: UranusSeven <109661872+UranusSeven@users.noreply.github.com >
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com >
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai >
2026-04-29 16:14:50 -07:00
Hemanth Acharya
6841f5dc77
[ROCm] Add env flags to disable dynamic MXFP4 quant and enable AITER tuned GEMMs for Attention Projection Layers ( #39987 )
...
Signed-off-by: Hemanth Acharya <heachary@amd.com >
2026-04-29 16:07:46 -07:00
roikoren755
c2fb013312
[Bugfix][Compile] Fix gc.collect/empty_cache patch arity in CUDAGraphWrapper ( #41235 )
...
Signed-off-by: Roi Koren <roik@nvidia.com >
2026-04-29 21:59:18 +00:00
0335316a9b
[BUG] Two phase pause to prevent deadlock ( #39366 )
...
Signed-off-by: ahao-anyscale <ahao@anyscale.com >
Signed-off-by: Aaron Hao <ahao@anyscale.com >
Co-authored-by: Junjie Zhang <junj.jay.zhang@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-04-29 17:51:03 -04:00
Rohan Potdar
944e138bcf
[ROCm][Bugfix]: W4A4 MOE using emulation instead of AITER on MXFP4-supported hardware ( #41175 )
...
Signed-off-by: Rohan138 <rohanpotdar138@gmail.com >
2026-04-29 16:39:03 -05:00
Luochao Wang and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
b58669cb42
[Perf][Spec Decode] Avoid per-step numpy allocation in prepare_next_t… ( #41043 )
...
Signed-off-by: wangluochao902 <wangluochao902@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-04-29 14:20:13 -07:00
Isotr0py
1628239eb2
[Multimodal][Render] Skip mm processor initialization and warmup for text-only mode ( #41246 )
...
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn >
2026-04-29 14:16:19 -07:00
Andrew Barnes
169988a3c0
[ROCm] Use quant_dtype in per_token_quant instead of hardcoded FP8 ( #39121 )
...
Signed-off-by: Bortlesboat <bortstheboat@gmail.com >
2026-04-29 20:46:01 +00:00
Chauncey and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
faab189554
[Feature]: IndexCache support for DSA models ( #37735 )
...
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-04-29 15:15:35 -04:00
Laith Sakka
6f20f81cbf
Replace shape_invariants with simpler apprach in dynamic_arg_dims utilizing shape_id property. ( #36194 )
...
Signed-off-by: Laith Sakka <lsakka@meta.com >
2026-04-29 18:32:15 +00:00
danisereb
d1a75e303d
Fix timeout when using LoRA adapters with Nemotron Super ( #40916 )
...
Signed-off-by: Daniel Serebrenik <daserebrenik@nvidia.com >
2026-04-30 01:39:49 +08:00
Avshalom Manevich
a80d6f150c
better logging for large uncachable items ( #41145 )
...
Signed-off-by: h-avsha <avshalom.manevich@hcompany.ai >
2026-04-29 09:48:47 -07:00
Terrence Zhao
91a2d39014
[Models] Cohere MoE ( #40817 )
...
Signed-off-by: Terrencezzj <terrence@cohere.ai >
2026-04-29 15:54:54 +00:00
Frederik Gossen
a05848e255
[Bugfix] Report compile time for in-memory cache hit path ( #41023 )
...
Signed-off-by: Frederik Gossen <frgossen@meta.com >
2026-04-29 15:32:03 +00:00
Or Ozeri