Commit Graph
1242 Commits
Author SHA1 Message Date
Wentao Ye 0d453e2336 [Perf] Batch invariance with Cutlass fp8 support, 28.9% E2E latency improvement (#40408)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
2026-05-11 12:20:58 -04:00
Martin Hickeyandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> 4955990f1b [kv_offload] Move FilterReusedOffloadingManager logic to CPUOffloadingManager (#41727)
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-05-11 18:09:29 +03:00
Nicolò Lucchesi 5672d100ed [KV Connector][NIXL][Bugfix] Fix NIXL handshake failures not honoring kv_load_failure_policy (#40364)
When NIXL handshake fails (e.g., due to compatibility hash mismatch
between prefill and decode instances), requests fail with "engine dead"
error instead of gracefully falling back to local recomputation as configured
by kv_load_failure_policy='recompute'.

Signed-off-by: NickLucche <nlucches@redhat.com>
2026-05-11 09:37:21 +00:00
Nicolò Lucchesi 770e9bd6b3 [Nixl][PD] Lease renewal TTL KV blocks on P (#41383)
Signed-off-by: NickLucche <nlucches@redhat.com>
2026-05-11 09:27:30 +00:00
shaharmor98andmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> f9f770ca0b fix nixl side-channel host selection (#41806)
Signed-off-by: Shahar Mor <smor@nvidia.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-11 07:40:37 +00:00
wangxiyuan 5536fc0c01 [Misc] Replace mamba_type string literals with MambaAttentionBackendEnum (#41188)
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
2026-05-11 03:59:36 +00:00
Ronen Schaffer e175192d33 [KV Offload] Pass ReqContext to touch(), complete_load(), and complete_store() (#41366)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
2026-05-10 15:09:25 +03:00
Ethan Feng a2c9d548d7 [Docs] Fix broken local links (#42160)
Signed-off-by: Ethan Feng <ethan.fengch@gmail.com>
2026-05-10 01:15:38 -07:00
Andreas Karatzas f2840120f6 [ROCm][CI] Fix NIXL spec-decode acceptance startup and diagnostics (#41313)
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
2026-05-10 14:50:16 +08:00
Dao007forever 3f5bd482f5 [Bugfix][KV Transfer][NIXL] Notify P node on pre-admission rejection to free stranded KV blocks (#41269) 2026-05-09 22:52:09 -07:00
Wentao Ye ea0e501bb1 [KV Connector] Remove compat support for pre-v0.12.0 constructor signatures without KVCacheConfig (#39832)
The v0.12.0 release contained initial support for HMA in KV Connectors. As part
of these changes, a KVCacheConfig argument was added to KV connector
constructors. Backwards compatibility support for out-of-tree connectors was
included in this change, with a very prominent warning. See #25712 and #27887.

Since the warning has been around for over 5 months, we can safely remove
the support of it.

Signed-off-by: yewentao256 <zhyanwentao@126.com>
2026-05-09 23:39:46 +00:00
Nicolò LucchesiandZhanqiuHu 171d59ae8d [Bugfix][PD] Fix DSv4 Disaggregated (#41957)
Signed-off-by: NickLucche <nlucches@redhat.com>
Co-authored-by: ZhanqiuHu <zhu@redhat.com>
2026-05-09 16:48:24 +00:00
David Zhenggemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>Andreas KaratzasKunshang Ji
df2636a9d8 [Bugfix] Fix LOGITPROC_SOURCE_ENTRYPOINT test to use spawn-compatible dist-info registration for XPU/ROCm (#42040)
Signed-off-by: dqzhengAP <dqzheng1996@gmail.com>
Signed-off-by: David Zheng <153074367+dzhengAP@users.noreply.github.com>
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
2026-05-09 12:32:04 +08:00
Lucas Wilkinson b1728c1e66 [Attention][Cleanup] Remove tree attention (#42121)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
2026-05-08 18:36:19 -07:00
Roy Wang 60851b1d22 [Bugfix][KV Transfer] Reject NixlConnector + expandable_segments:True (#41237) 2026-05-08 16:47:33 -07:00
David Zheng 1acd67a795 [Bugfix] Fix XPU/ROCm compatibility in spawn_new_process_for_each_test (#41895)
Signed-off-by: dqzhengAP <dqzheng1996@gmail.com>
2026-05-08 00:47:22 -04:00
Hongxia YangandTJian 20cac26b19 [ROCm] Enable SimpleCPUOffloadConnector on ROCm (#40549)
Signed-off-by: Hongxia Yang <hongxia.yang@amd.com>
Co-authored-by: TJian <tunjian.tan@embeddedllm.com>
2026-05-06 20:52:02 -07:00
Jing WangCopilotWentao Yemergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
27702f6d08 [Bugfix] Fix token loss in PP mode which causes degraded accuracy (#41133)
Signed-off-by: Jing Wang <jingwang96@qq.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-06 14:07:32 -04:00
David Zheng ee38750a75 [Bugfix] Fix spawn_new_process_for_each_test silently swallowing test failures (#41423)
Signed-off-by: dqzhengAP <dqzheng1996@gmail.com>
2026-05-06 11:17:15 -04:00
Luciano MartinsandLuciano Martins 27e0057aed [Spec Decode] Add Gemma4 MTP speculative decoding support (#41745)
Signed-off-by: Luciano Martins <lucianommartins@users.noreply.github.com>
Co-authored-by: Luciano Martins <lucianommartins@users.noreply.github.com>
2026-05-06 22:39:29 +08:00
Ronen Schaffer f39bcf1e30 [KV Offload] Return None from lookup() for in-flight blocks (#41795)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
2026-05-06 17:31:21 +03:00
zhanqiuhuandNickLucche df8e63f4ed nixl refactor: new transfer design (#40731)
Signed-off-by: ZhanqiuHu <zhu@redhat.com>
Signed-off-by: NickLucche <nlucches@redhat.com>
Co-authored-by: NickLucche <nlucches@redhat.com>
2026-05-06 06:16:25 -07:00
wang.yuqiandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> 51c1ee9b7c [Examples] Resettle Disaggregated examples. (#40759)
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
Signed-off-by: wang.yuqi <noooop@126.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-05-06 01:20:38 -07:00
wi-adam b53c507bc9 [Bugfix] Skip PP sampled-token receive on last rank during async scheduling (#40749)
Signed-off-by: Adam Winstanley <adam@winstanley.industries>
2026-05-06 05:31:14 +00:00
Matthew Bonanni 01b9b5af67 [Attention] Minor refactor: layer takes ownership of the MLA prefill backend (#41744)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
2026-05-05 23:22:41 +00:00
Lanze Liu 79246b5ea6 [Spec Decode] Fix max_model_len logging in speculative config for draft model (#41571)
Signed-off-by: Lanze Liu <lanzetech@gmail.com>
2026-05-05 21:56:06 +00:00
Zhewen LiandZhewen Li bee126165f [P/D][Mooncake] Add KVConnectorStats for transfer observability (#40414)
Signed-off-by: Zhewen Li <zhewenli@inferact.ai>
Co-authored-by: Zhewen Li <zhewenli@inferact.ai>
2026-05-05 02:17:38 -07:00
Wentao Ye 577b9623e6 [Bug] Fix status update address for non-MOE model within external dp mode (#40839)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
2026-05-04 16:37:16 -07:00
Andreas Karatzas 1cb0838721 [ROCm][CI] Fix MLA prefill scale for DeepSeek GSM8K (#41569)
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
2026-05-04 16:32:55 -07:00
Joachim StudniaandClaude Opus 4.6 422dd02598 [bugfix] Fix prompt logprobs on request eviction during chunked prefill (#41411)
Signed-off-by: Joachim Studnia <joachim@mistral.ai>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
2026-05-04 11:46:00 -07:00
Andreas Karatzas 01d4d1ad37 [ROCm][CI] Align spec decode logprob test prefill settings (#41335)
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
2026-05-04 04:33:29 +00:00
66dfee7121 [Bugfix] Fix degenerate KV cache stride causing TMA cudaErrorIllegalInstruction (#40737)
Signed-off-by: David Oy <david@baseten.co>
Signed-off-by: David Oy <58150256+the-david-oy@users.noreply.github.com>
Signed-off-by: David Oy <david.oy@baseten.co>
Co-authored-by: David Oy <david@baseten.co>
Co-authored-by: Claude <claude@anthropic.com>
Co-authored-by: Vadim Gimpelson <156319763+vadiklyutiy@users.noreply.github.com>
2026-05-03 23:52:18 +00:00
Luka Govedič d58c42e19c [vLLM IR] 2/N fused_add_rms_norm and maybe_inplace overload (#36823)
Signed-off-by: Luka Govedič <lgovedic@redhat.com>
Signed-off-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
2026-05-01 23:41:15 -04:00
Matthew BonanniMichael GoinLucas Wilkinsonmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
f3fef12350 [Attention] Abstract the MLA prefill backends and eliminate cuDNN (#32623)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-01 13:36:20 -04:00
Or Ozeri 2fa1f8ec00 [kv_offload+HMA][13/N]: Enable HMA support (#41445)
This is the final PR in a series to enables HMA support for the
offloading connector. The connector advertises `SupportsHMA`
and is validated with unit tests and e2e tests.

Signed-off-by: Or Ozeri <oro@il.ibm.com>
2026-05-01 12:30:03 +01:00
a3ec4a35f5 [Bugfix][Metrics] Fix RayPrometheusMetric.labels() returning shared labeled child (#40840)
When vLLM runs with Ray Prometheus `vllm:request_success{finished_reason=...}`
only ever increments the repetition bucket regardless of the request's actual finish
reason; stop, length, abort, and error stay at zero. Root cause was `labels()` mutated
the wrapped Ray metric's default tags in place and returned self, so every `.labels(...)`
call on a given wrapper returned the same object. 

Co-authored-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com>
Signed-off-by: Seiji Eicher <seiji@anyscale.com>
2026-05-01 08:43:39 +01:00
Bugen Zhaoandwang.yuqi a07642667d [Bugfix] Pass reasoning parser kwargs to structured output (#41199)
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: wang.yuqi <yuqi.wang@daocloud.io>
2026-04-30 23:38:02 -07:00
sychen52 947138b6c2 Add nvfp4 kv cache support (#40177)
Signed-off-by: Shiyang Chen <shiychen@nvidia.com>
2026-05-01 04:55:16 +00:00
Luis 🚀andAndrew Sansom 14043dfecd feat: Enable prompt_embeds Content Part Support in vLLM Chat Completions API (#40720)
Signed-off-by: Luis Robaina <luis@protopia.ai>
Signed-off-by: Luis Robaina 🚀 <luisfabian1545@gmail.com>
Signed-off-by: LuisRobaina <luis@protopia.ai>
Co-authored-by: Andrew Sansom <qthequartermasterman@gmail.com>
2026-05-01 10:05:55 +08:00
Nick Hill dd5506a157 [Core] Simplify handling of scheduler_reserve_full_isl option (#41064)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
2026-04-30 18:10:00 -07:00
Stefano Castagnettaandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 92a7c121b6 [CI] Add MTP coverage: Qwen3.5 correctness + no-sync spec decode (#40472)
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-04-30 12:24:09 -07:00
snadampal 3179e53135 [P/D] Prefill compute optimizations with bi-directional KV cache transfers between P and D nodes (#32553)
Signed-off-by: Sunita Nadampalli <nadampal@amazon.com>
2026-04-30 10:14:20 +00:00
Nicolò Lucchesi efdc95674d [KVConnector] MultiConnector SupportsHMA (#39571)
Signed-off-by: NickLucche <nlucches@redhat.com>
2026-04-30 02:10:50 -07:00
Martin HickeyandOr Ozeri c42981d034 [Refactor][kv_offload] KV Offloading maintainability improvements (#40538)
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-04-30 05:55:31 +03:00
Wei Zhao 0ff1bf9bb1 [Bugfix] Fix failure to allocate KV blocks error (#41282)
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
2026-04-29 18:44:07 -07:00
Nick Hill 18599bfdf2 [Ci][BugFix] Fix slow DP tests due to bad teardown logic (#41166)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
2026-04-29 19:31:00 -04:00
0335316a9b [BUG] Two phase pause to prevent deadlock (#39366)
Signed-off-by: ahao-anyscale <ahao@anyscale.com>
Signed-off-by: Aaron Hao <ahao@anyscale.com>
Co-authored-by: Junjie Zhang <junj.jay.zhang@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-04-29 17:51:03 -04:00
Artem Perevedentsev b92ef9ec5a [Perf] Enable FlashInfer top-k/top-p sampler by default (#40376)
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
2026-04-29 19:10:34 +04:00
68dd7db810 [Reasoning] Support for speculative decoding with thinking budget (#34668)
Signed-off-by: rishitdholakia13 <rishit+github@cohere.com>
Signed-off-by: rishitdholakia13 <123388671+rishitdholakia13@users.noreply.github.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
2026-04-29 06:14:52 +00:00
8a8c9b564e [KV Offload] Per-job store completion for CPU offloading connector (#39186)
Signed-off-by: Itay Etelis <itay.etelis@ibm.com>
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Co-authored-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: Or Ozeri <or@ozery.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-04-29 08:52:55 +03:00