Wentao Ye
0d453e2336
[Perf] Batch invariance with Cutlass fp8 support, 28.9% E2E latency improvement ( #40408 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com >
2026-05-11 12:20:58 -04:00
Martin Hickey and gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
4955990f1b
[kv_offload] Move FilterReusedOffloadingManager logic to CPUOffloadingManager ( #41727 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-05-11 18:09:29 +03:00
Nicolò Lucchesi
5672d100ed
[KV Connector][NIXL][Bugfix] Fix NIXL handshake failures not honoring kv_load_failure_policy ( #40364 )
...
When NIXL handshake fails (e.g., due to compatibility hash mismatch
between prefill and decode instances), requests fail with "engine dead"
error instead of gracefully falling back to local recomputation as configured
by kv_load_failure_policy='recompute'.
Signed-off-by: NickLucche <nlucches@redhat.com >
2026-05-11 09:37:21 +00:00
Nicolò Lucchesi
770e9bd6b3
[Nixl][PD] Lease renewal TTL KV blocks on P ( #41383 )
...
Signed-off-by: NickLucche <nlucches@redhat.com >
2026-05-11 09:27:30 +00:00
shaharmor98 and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
f9f770ca0b
fix nixl side-channel host selection ( #41806 )
...
Signed-off-by: Shahar Mor <smor@nvidia.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-11 07:40:37 +00:00
wangxiyuan
5536fc0c01
[Misc] Replace mamba_type string literals with MambaAttentionBackendEnum ( #41188 )
...
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com >
2026-05-11 03:59:36 +00:00
Ronen Schaffer
e175192d33
[KV Offload] Pass ReqContext to touch(), complete_load(), and complete_store() ( #41366 )
...
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com >
2026-05-10 15:09:25 +03:00
Ethan Feng
a2c9d548d7
[Docs] Fix broken local links ( #42160 )
...
Signed-off-by: Ethan Feng <ethan.fengch@gmail.com >
2026-05-10 01:15:38 -07:00
Andreas Karatzas
f2840120f6
[ROCm][CI] Fix NIXL spec-decode acceptance startup and diagnostics ( #41313 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
2026-05-10 14:50:16 +08:00
Dao007forever
3f5bd482f5
[Bugfix][KV Transfer][NIXL] Notify P node on pre-admission rejection to free stranded KV blocks ( #41269 )
2026-05-09 22:52:09 -07:00
Wentao Ye
ea0e501bb1
[KV Connector] Remove compat support for pre-v0.12.0 constructor signatures without KVCacheConfig ( #39832 )
...
The v0.12.0 release contained initial support for HMA in KV Connectors. As part
of these changes, a KVCacheConfig argument was added to KV connector
constructors. Backwards compatibility support for out-of-tree connectors was
included in this change, with a very prominent warning. See #25712 and #27887 .
Since the warning has been around for over 5 months, we can safely remove
the support of it.
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-05-09 23:39:46 +00:00
Nicolò Lucchesi and ZhanqiuHu
171d59ae8d
[Bugfix][PD] Fix DSv4 Disaggregated ( #41957 )
...
Signed-off-by: NickLucche <nlucches@redhat.com >
Co-authored-by: ZhanqiuHu <zhu@redhat.com >
2026-05-09 16:48:24 +00:00
df2636a9d8
[Bugfix] Fix LOGITPROC_SOURCE_ENTRYPOINT test to use spawn-compatible dist-info registration for XPU/ROCm ( #42040 )
...
Signed-off-by: dqzhengAP <dqzheng1996@gmail.com >
Signed-off-by: David Zheng <153074367+dzhengAP@users.noreply.github.com >
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com >
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com >
2026-05-09 12:32:04 +08:00
Lucas Wilkinson
b1728c1e66
[Attention][Cleanup] Remove tree attention ( #42121 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
2026-05-08 18:36:19 -07:00
Roy Wang
60851b1d22
[Bugfix][KV Transfer] Reject NixlConnector + expandable_segments:True ( #41237 )
2026-05-08 16:47:33 -07:00
David Zheng
1acd67a795
[Bugfix] Fix XPU/ROCm compatibility in spawn_new_process_for_each_test ( #41895 )
...
Signed-off-by: dqzhengAP <dqzheng1996@gmail.com >
2026-05-08 00:47:22 -04:00
Hongxia Yang and TJian
20cac26b19
[ROCm] Enable SimpleCPUOffloadConnector on ROCm ( #40549 )
...
Signed-off-by: Hongxia Yang <hongxia.yang@amd.com >
Co-authored-by: TJian <tunjian.tan@embeddedllm.com >
2026-05-06 20:52:02 -07:00
27702f6d08
[Bugfix] Fix token loss in PP mode which causes degraded accuracy ( #41133 )
...
Signed-off-by: Jing Wang <jingwang96@qq.com >
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-06 14:07:32 -04:00
David Zheng
ee38750a75
[Bugfix] Fix spawn_new_process_for_each_test silently swallowing test failures ( #41423 )
...
Signed-off-by: dqzhengAP <dqzheng1996@gmail.com >
2026-05-06 11:17:15 -04:00
Luciano Martins and Luciano Martins
27e0057aed
[Spec Decode] Add Gemma4 MTP speculative decoding support ( #41745 )
...
Signed-off-by: Luciano Martins <lucianommartins@users.noreply.github.com >
Co-authored-by: Luciano Martins <lucianommartins@users.noreply.github.com >
2026-05-06 22:39:29 +08:00
Ronen Schaffer
f39bcf1e30
[KV Offload] Return None from lookup() for in-flight blocks ( #41795 )
...
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com >
2026-05-06 17:31:21 +03:00
zhanqiuhu and NickLucche
df8e63f4ed
nixl refactor: new transfer design ( #40731 )
...
Signed-off-by: ZhanqiuHu <zhu@redhat.com >
Signed-off-by: NickLucche <nlucches@redhat.com >
Co-authored-by: NickLucche <nlucches@redhat.com >
2026-05-06 06:16:25 -07:00
wang.yuqi and gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
51c1ee9b7c
[Examples] Resettle Disaggregated examples. ( #40759 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
Signed-off-by: wang.yuqi <noooop@126.com >
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-05-06 01:20:38 -07:00
wi-adam
b53c507bc9
[Bugfix] Skip PP sampled-token receive on last rank during async scheduling ( #40749 )
...
Signed-off-by: Adam Winstanley <adam@winstanley.industries >
2026-05-06 05:31:14 +00:00
Matthew Bonanni
01b9b5af67
[Attention] Minor refactor: layer takes ownership of the MLA prefill backend ( #41744 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
2026-05-05 23:22:41 +00:00
Lanze Liu
79246b5ea6
[Spec Decode] Fix max_model_len logging in speculative config for draft model ( #41571 )
...
Signed-off-by: Lanze Liu <lanzetech@gmail.com >
2026-05-05 21:56:06 +00:00
Zhewen Li and Zhewen Li
bee126165f
[P/D][Mooncake] Add KVConnectorStats for transfer observability ( #40414 )
...
Signed-off-by: Zhewen Li <zhewenli@inferact.ai >
Co-authored-by: Zhewen Li <zhewenli@inferact.ai >
2026-05-05 02:17:38 -07:00
Wentao Ye
577b9623e6
[Bug] Fix status update address for non-MOE model within external dp mode ( #40839 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-05-04 16:37:16 -07:00
Andreas Karatzas
1cb0838721
[ROCm][CI] Fix MLA prefill scale for DeepSeek GSM8K ( #41569 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
2026-05-04 16:32:55 -07:00
Joachim Studnia and Claude Opus 4.6
422dd02598
[bugfix] Fix prompt logprobs on request eviction during chunked prefill ( #41411 )
...
Signed-off-by: Joachim Studnia <joachim@mistral.ai >
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com >
2026-05-04 11:46:00 -07:00
Andreas Karatzas
01d4d1ad37
[ROCm][CI] Align spec decode logprob test prefill settings ( #41335 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
2026-05-04 04:33:29 +00:00
66dfee7121
[Bugfix] Fix degenerate KV cache stride causing TMA cudaErrorIllegalInstruction ( #40737 )
...
Signed-off-by: David Oy <david@baseten.co >
Signed-off-by: David Oy <58150256+the-david-oy@users.noreply.github.com >
Signed-off-by: David Oy <david.oy@baseten.co >
Co-authored-by: David Oy <david@baseten.co >
Co-authored-by: Claude <claude@anthropic.com >
Co-authored-by: Vadim Gimpelson <156319763+vadiklyutiy@users.noreply.github.com >
2026-05-03 23:52:18 +00:00
Luka Govedič
d58c42e19c
[vLLM IR] 2/N fused_add_rms_norm and maybe_inplace overload ( #36823 )
...
Signed-off-by: Luka Govedič <lgovedic@redhat.com >
Signed-off-by: Luka Govedič <ProExpertProg@users.noreply.github.com >
2026-05-01 23:41:15 -04:00
f3fef12350
[Attention] Abstract the MLA prefill backends and eliminate cuDNN ( #32623 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Michael Goin <mgoin64@gmail.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-01 13:36:20 -04:00
Or Ozeri
2fa1f8ec00
[kv_offload+HMA][13/N]: Enable HMA support ( #41445 )
...
This is the final PR in a series to enables HMA support for the
offloading connector. The connector advertises `SupportsHMA`
and is validated with unit tests and e2e tests.
Signed-off-by: Or Ozeri <oro@il.ibm.com >
2026-05-01 12:30:03 +01:00
a3ec4a35f5
[Bugfix][Metrics] Fix RayPrometheusMetric.labels() returning shared labeled child ( #40840 )
...
When vLLM runs with Ray Prometheus `vllm:request_success{finished_reason=...}`
only ever increments the repetition bucket regardless of the request's actual finish
reason; stop, length, abort, and error stay at zero. Root cause was `labels()` mutated
the wrapped Ray metric's default tags in place and returned self, so every `.labels(...)`
call on a given wrapper returned the same object.
Co-authored-by: Marwan Sarieddine <sarieddine.marwan@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
Signed-off-by: Marwan Sarieddine <sarieddine.marwan@gmail.com >
Signed-off-by: Seiji Eicher <seiji@anyscale.com >
2026-05-01 08:43:39 +01:00
Bugen Zhao and wang.yuqi
a07642667d
[Bugfix] Pass reasoning parser kwargs to structured output ( #41199 )
...
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: wang.yuqi <yuqi.wang@daocloud.io >
2026-04-30 23:38:02 -07:00
sychen52
947138b6c2
Add nvfp4 kv cache support ( #40177 )
...
Signed-off-by: Shiyang Chen <shiychen@nvidia.com >
2026-05-01 04:55:16 +00:00
Luis 🚀 and Andrew Sansom
14043dfecd
feat: Enable prompt_embeds Content Part Support in vLLM Chat Completions API ( #40720 )
...
Signed-off-by: Luis Robaina <luis@protopia.ai >
Signed-off-by: Luis Robaina 🚀 <luisfabian1545@gmail.com >
Signed-off-by: LuisRobaina <luis@protopia.ai >
Co-authored-by: Andrew Sansom <qthequartermasterman@gmail.com >
2026-05-01 10:05:55 +08:00
Nick Hill
dd5506a157
[Core] Simplify handling of scheduler_reserve_full_isl option ( #41064 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-04-30 18:10:00 -07:00
Stefano Castagnetta and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
92a7c121b6
[CI] Add MTP coverage: Qwen3.5 correctness + no-sync spec decode ( #40472 )
...
Signed-off-by: Stefano Castagnetta <scastagnetta@nvidia.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-04-30 12:24:09 -07:00
snadampal
3179e53135
[P/D] Prefill compute optimizations with bi-directional KV cache transfers between P and D nodes ( #32553 )
...
Signed-off-by: Sunita Nadampalli <nadampal@amazon.com >
2026-04-30 10:14:20 +00:00
Nicolò Lucchesi
efdc95674d
[KVConnector] MultiConnector SupportsHMA ( #39571 )
...
Signed-off-by: NickLucche <nlucches@redhat.com >
2026-04-30 02:10:50 -07:00
Martin Hickey and Or Ozeri
c42981d034
[Refactor][kv_offload] KV Offloading maintainability improvements ( #40538 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
Co-authored-by: Or Ozeri <oro@il.ibm.com >
2026-04-30 05:55:31 +03:00
Wei Zhao
0ff1bf9bb1
[Bugfix] Fix failure to allocate KV blocks error ( #41282 )
...
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com >
2026-04-29 18:44:07 -07:00
Nick Hill
18599bfdf2
[Ci][BugFix] Fix slow DP tests due to bad teardown logic ( #41166 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-04-29 19:31:00 -04:00
0335316a9b
[BUG] Two phase pause to prevent deadlock ( #39366 )
...
Signed-off-by: ahao-anyscale <ahao@anyscale.com >
Signed-off-by: Aaron Hao <ahao@anyscale.com >
Co-authored-by: Junjie Zhang <junj.jay.zhang@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-04-29 17:51:03 -04:00
Artem Perevedentsev
b92ef9ec5a
[Perf] Enable FlashInfer top-k/top-p sampler by default ( #40376 )
...
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com >
2026-04-29 19:10:34 +04:00
68dd7db810
[Reasoning] Support for speculative decoding with thinking budget ( #34668 )
...
Signed-off-by: rishitdholakia13 <rishit+github@cohere.com >
Signed-off-by: rishitdholakia13 <123388671+rishitdholakia13@users.noreply.github.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk >
2026-04-29 06:14:52 +00:00
8a8c9b564e
[KV Offload] Per-job store completion for CPU offloading connector ( #39186 )
...
Signed-off-by: Itay Etelis <itay.etelis@ibm.com >
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com >
Co-authored-by: Itay Etelis <itay.etelis@ibm.com >
Co-authored-by: Or Ozeri <or@ozery.com >
Co-authored-by: Or Ozeri <oro@il.ibm.com >
2026-04-29 08:52:55 +03:00