+13
aeeb36b1f1
[New model] Kimi K3 ( #50000 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Ziming Huang <zelda.huanghuang@gmail.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: Isotr0py <mozf@inferact.ai >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Summer Yang <girasoleyang@gmail.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
Co-authored-by: Bowen Wang <abmfy@icloud.com >
Co-authored-by: gnovack <novackgm@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: xiaozhoupy <peiyuanzhou1994@gmail.com >
Co-authored-by: Roy Wang <yasong.wang@inferact.ai >
Co-authored-by: Jeff (Junze) Ma <93145857+majunze2001@users.noreply.github.com >
Co-authored-by: Codex <codex@openai.com >
Co-authored-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com >
2026-07-30 03:49:58 -07:00
Nick Hill and Claude Opus 5
437e0b7f8e
[BugFix] Fix P/D preemption race condition ( #50297 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-07-30 01:11:18 +00:00
a0c092ee72
[BugFix] Fix num_output_placeholders preemption underflow ( #48245 )
...
Signed-off-by: Chris Eastwood <chris.eastwood@pwn4g3.dev >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Chris Eastwood <chris.eastwood@pwn4g3.dev >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
2026-07-29 06:41:54 -07:00
Dao007forever and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
d742856610
[3/N][Core][KV Connector] Support reliable partial-tail KV offload for sub-block prompts ( #49502 )
...
Signed-off-by: Dao Le <Dao007forever@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-26 23:22:17 -07:00
liuzhenwei and Kunshang Ji
6a1acac3fe
[BUGFIX] Fix log capture in KV test ( #49655 )
...
Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com >
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com >
2026-07-25 00:36:53 +00:00
Michael Goin and OpenAI Codex
275556c35c
[Bugfix] Detect mixed precision in packed KV cache specs ( #49623 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Signed-off-by: Michael Goin <mgoin64@gmail.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-07-23 20:07:06 -07:00
Michael Goin and OpenAI Codex
e18f0037a5
[Bugfix][KV cache] Support sparse-MLA targets with SWA drafts ( #48776 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-07-23 11:30:50 -07:00
zhrrr and Nick Hill
12213c6795
[Bugfix] handle grammar compilation failures to avoid engine crash ( #47312 )
...
Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-07-23 18:13:49 +08:00
Nick Hill
229e01e9e1
[BugFix] Handle per-group prefix-hit divergence for hybrid models with KV connector ( #48425 )
2026-07-22 17:19:11 -07:00
Summer Yang
f3a920a076
[Core][DSV4] Compact MXFP4 indexer KV cache and packed group overlays ( #48993 )
2026-07-22 16:58:44 -07:00
8950394e0a
[Bugfix] Prefix-cache metrics double-counted when a KV connector defers requests ( #48860 )
...
Signed-off-by: Seiji Eicher <58963096+eicherseiji@users.noreply.github.com >
Signed-off-by: Seiji Eicher <seiji@anyscale.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Lixi Zhang <muxixibbb@gmail.com >
Co-authored-by: harjothkhara <harjoth.khara@gmail.com >
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-07-21 15:02:03 +01:00
2e2e626b40
[Bugfix] Count per-group blocks in get_max_concurrency_for_kv_cache_config ( #48317 )
...
Signed-off-by: David Orman <ormandj@corenode.com >
Co-authored-by: Luke Alonso <lalonso@gmail.com >
Co-authored-by: Martin Vit <martin@voipmonitor.org >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
2026-07-21 00:29:10 +00:00
yzong-rh
a287eb163f
[Front-end] [Messages] Populate num_cache_creation_tokens ( #48535 )
...
Signed-off-by: Yifan Zong <yzong@redhat.com >
2026-07-18 13:04:35 -04:00
5a65ba5f17
[Refactor] Move iteration logging to the frontend ( #46647 )
...
Signed-off-by: maxyanghu <hyoung2991@gmail.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: Shang Wang <shangw@nvidia.com >
2026-07-15 17:59:05 -07:00
32aef44388
[Bugfix] Include inline per-token-head scales in offloaded page transfer width ( #48411 )
...
Signed-off-by: Itay Etelis <itay.etelis@ibm.com >
Signed-off-by: Itay Etelis <Itay.etelis@gmail.com >
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com >
Co-authored-by: Itay Etelis <itay.etelis@ibm.com >
Co-authored-by: Itay Etelis <Itay.etelis@gmail.com >
2026-07-14 16:07:26 +03:00
Nick Hill
8ac8375270
[Core] Preserve Marconi caching with selective hybrid cache retention ( #47782 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-07-13 21:24:20 +01:00
Nicolò Lucchesi and Claude
56a357ed33
[Bugfix][KV Cache] Don't route uniform-page-size MLA+SWA models into DeepseekV4 packing ( #48256 )
...
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai >
Co-authored-by: Claude <noreply@anthropic.com >
2026-07-13 08:16:24 +00:00
omerpaz95 and Or Ozeri
8df14cfc8c
[EC Connector] Add EC Transfer Params ( #42433 )
...
Signed-off-by: omerpaz95 <omerpaz95@gmail.com >
Co-authored-by: Or Ozeri <oro@il.ibm.com >
2026-07-12 14:35:33 +03:00
Jiangyun Zhu and Yifan Qiao
481e481be7
[2/N][Core] support partial prefix cache hit for hybrid model ( #46384 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
2026-07-12 05:37:51 +00:00
e5588e49bc
[Core][KV events] Report prefix-cache-reused blocks in full report mode ( #45261 )
...
Signed-off-by: Lei Gong <gonglei25@huawei.com >
Co-authored-by: Lei Gong <gonglei25@huawei.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-07-09 22:46:54 -07:00
cd0de48d08
[Bugfix][V1] Free out-of-window blocks on the processed-token basis under async scheduling ( #47728 )
...
Signed-off-by: Saddss <28726669061@qq.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Saddss <28726669061@qq.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-07-08 13:34:19 +01:00
Sahil Kadadekar and Claude Fable 5
55da232db6
[Bugfix] Pad Mamba page size instead of scaling block_size in unify_kv_cache_spec_page_size ( #45207 )
...
Signed-off-by: Sahil170595 <147995121+Sahil170595@users.noreply.github.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-07-07 22:01:34 +00:00
93e2ab7111
Disable dynamic speculative decoding when DP is enabled ( #45963 )
...
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com >
2026-07-07 12:52:41 +00:00
Kyung Sub Lee (Daniel) and Claude Fable 5
c5b66233b2
[Bugfix][Spec Decode] Skip uniform spec-decode padding for diffusion models ( #47464 )
...
Signed-off-by: kl527 <kl527@cornell.edu >
Signed-off-by: Kyung Sub Lee (Daniel) <66861800+kl527@users.noreply.github.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-07-07 09:25:12 +00:00
Ting SUN and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
373eb314af
[Bugfix][Core] Fix num_output_placeholders underflow with async scheduling + spec decode ( #46066 )
...
Signed-off-by: Ting Sun <suntcrick@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-06 13:50:38 +00:00
yue.yu and Benjamin Chislett
e7c9df9449
[Bugfix][Structured Output][Spec Decode] Constrain bitmask and trim grammar advance at the reasoning boundary ( #44297 )
...
Signed-off-by: Allen.Yu <yuyue0225sc@163.com >
Signed-off-by: yue.yu <yuyue0225sc@163.com >
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com >
2026-07-04 09:08:45 +00:00
Nick Hill
e392bf7a68
[BugFix][MRV2] Ensure all req slots are accounted for when scheduling ( #46974 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-07-02 10:19:24 -07:00
c6741b2ad4
[Model] Support Unlimited OCR ( #46564 )
...
Signed-off-by: Tianyu Guo <guoty9@mail2.sysu.edu.cn >
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn >
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn >
Co-authored-by: Roger Wang <hey@rogerw.io >
2026-06-27 23:09:18 -07:00
Nick Hill
658b54efe4
[ModelRunner V2] Update scheduler tests to cover MRV2 paths ( #46771 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-06-26 09:36:31 -07:00
d490b98162
[Core] Avoid mixed length specdec batches via padding ( #45237 )
...
Signed-off-by: Yiliu Dong <91178480+qianlihuang@users.noreply.github.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Jade Zheng <zheng.shoujian@outlook.com >
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai >
Co-authored-by: Zijing Liu <liuzijing2014@gmail.com >
2026-06-25 08:34:44 -07:00
Lucas Wilkinson
e7df232288
[KV Offload] Gate packed HMA KV cache on cross-layer config ( #46252 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
2026-06-24 11:55:30 -04:00
Nick Hill and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
6af0559ddb
[Core][DP] Throttle prefills based on local prefill work ( #46532 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-06-24 02:27:12 +00:00
Dao007forever and Claude Opus 4.8
430a95ae3a
[v1][kvcache] Honor prefix-cache retention interval for Mamba/linear attention ( #45845 )
...
Signed-off-by: Dao Le <daole@inferact.ai >
Signed-off-by: Dao Le <Dao007forever@gmail.com >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
2026-06-22 19:51:11 -07:00
3e6529cc0e
[Bugfix][Spec Decode] Fix EAGLE drafter multimodal encoder cache misses ( #46315 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
2026-06-22 18:14:02 +00:00
Jiangyun Zhu and Yifan Qiao
6bc6f2d86d
[1/N][Core] add partial prefix cache primitives ( #45939 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
2026-06-21 23:43:10 -07:00
2cac89f9da
[Spec Decode] Support mixed KV page sizes for DFlash ( #45181 )
...
Signed-off-by: Alex Steiner <asteiner@nvidia.com >
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com >
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
2026-06-21 22:45:14 +08:00
6e919960af
[Perf] Skip/shrink all_token_ids copy in scheduler for non-async and V2 runner ( #45840 )
...
Signed-off-by: amanchugh89 <amanchugh.89@gmail.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-06-20 22:36:57 +00:00
cc22621b51
[KV Offload] Support packed HMA KV cache layout ( #46205 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com >
2026-06-20 21:19:40 +00:00
01192139bf
[DSv4] Pack KV caches into contiguous per-block allocations for DeepSeek V4 ( #44577 )
...
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com >
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com >
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-06-19 12:55:42 -04:00
Stan Wozniak
520828789c
Apply LRU policy only to proper cache entries ( #42656 )
...
Signed-off-by: Stanislaw Wozniak <stw@zurich.ibm.com >
2026-06-16 21:49:15 +00:00
Nick Hill
d8d95998dc
[Core] Add prefill step cadence for better non-PD DP balancing ( #44558 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-06-16 13:17:18 -07:00
d467a2a7f2
[Bugfix] Defer block freeing until in-flight steps finish under async scheduling + PD KV consumer ( #45357 )
...
Signed-off-by: llx-08 <2596671364@qq.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com >
2026-06-15 21:36:09 +00:00
Saddss
588db18362
[Bugfix] Two-phase KV allocation for cross-group prefix cache hits (supersedes #33775 ) ( #44409 )
...
Signed-off-by: Saddss <2872669061@qq.com >
2026-06-15 22:39:59 +08:00
Nick Hill
1a369783e9
[BugFix] Avoid prematurely freeing cached mm encoder outputs ( #45347 )
...
Signed-off-by: Roger Wang <hey@rogerw.io >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-06-12 15:39:40 -07:00
Srinivas Krovvidi and srinivas_oo7
9ff278b1d2
[Core][KV Connector] fix scheduler KV connector stats aggregation ( #43877 )
...
Fixes scheduler-side KV connector stats collection so that:
1. update_connector_output() runs before scheduler-side stats are collected.
2. worker-side and scheduler-side KV connector stats are aggregated when both are present.
3. scheduler-only KV connector stats are still emitted when no worker-side stats exist.
Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com >
Co-authored-by: srinivas_oo7 <sklinkedin0120@gmail.com >
2026-06-12 14:51:55 +00:00
Ethan Feng
b7f9b6ab27
[Metrics] Add group-aware KV cache capacity to vllm:cache_config_info ( #42206 )
...
The startup log already reports the correct group-aware KV cache capacity for
hybrid models, but Prometheus did not expose matching info in 'vllm:cache_config_info`.
This PR adds kv_cache_size_tokens and kv_cache_max_concurrency.
Signed-off-by: Ethan Feng <ethan.fengch@gmail.com >
2026-06-12 11:49:44 +00:00
Nick Hill and Claude Opus 4.8
4085ff7cb4
[Core] Add kvcache watermark to reduce preemptions ( #44594 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
2026-06-11 08:27:31 -07:00
Stan Wozniak
dc66e01a70
[Hybrid] Marconi-style admission policy for hybrid cache ( #37898 )
...
Signed-off-by: Stanislaw Wozniak <stw@zurich.ibm.com >
2026-06-10 10:03:13 -07:00
303916e93d
[Bugfix]: Fix assertion in MambaManager.allocate_slots() ( #39562 )
...
Signed-off-by: Holworth <kangqihan17@mails.ucas.ac.cn >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-06-08 00:34:37 -04:00
Nick Hill
3b3d5287fa
[BugFix] Resolve multiple async kv load deadlock ( #44560 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-06-06 23:05:47 +00:00
