76fb6d210a
[Bugfix][Core] Reserve the KV null block when validating max_model_len ( #47272 )
...
Signed-off-by: hyungjun oh <92hyungjun@gmail.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-08-20 02:49:23 +00:00
Xiaoan (Sean) Liu and Nick Hill
db92053e97
[Core] Skip broadcasting mm tensor data to workers for prefix-cache-covered items ( #52041 )
...
Signed-off-by: Xiaoan (Sean) Liu <shawnliu0327@gmail.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-08-19 07:16:39 -07:00
Russell Bryant and Claude Opus 4.8
ef47a897e2
[Core] Make prefix-cache NONE_HASH deterministic by default ( #51875 )
...
Signed-off-by: Russell Bryant <rbryant@redhat.com >
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com >
2026-08-18 23:14:39 +00:00
0db502c8d8
[Kimi-K3] support DCP partial prefix cache hit ( #50493 )
...
Signed-off-by: Summer Yang <girasoleyang@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
2026-08-17 17:05:51 -07:00
Tyler Michael Smith and OpenAI Codex
017e9f4448
Promote prefix_cache_retention_interval to an argument and change the default to 0 ( #52216 )
...
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-17 13:20:50 +00:00
Tianyu Guo
eee538d5da
[Bugfix][V1][Multimodal] Ignore stale same-step encoder cache evictions ( #52482 )
...
Signed-off-by: Tianyu Guo <guoty@inferact.ai >
2026-08-16 22:21:55 +00:00
mispa-ms
6593754e61
[Bugfix][Spec Decode] Keep EAGLE cache registration on the partial-hash-hit path ( #52419 )
2026-08-15 21:03:53 -07:00
Lucas Wilkinson and Claude
57bd0ed441
[5/N][KV-Cache Layout Refactor] Backend-published KV packing via customize_spec ( #51704 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-08-14 09:47:55 -04:00
Giancarlo Delfin and Claude Fable 5
f80b66f548
[Model Runner V2][Spec Decode] Add KV cache support for multi-layer MTP ( #50062 )
...
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-08-13 16:22:29 -07:00
50ba4bc6b2
[Feature] Mask Replay ( #49577 )
...
Signed-off-by: vx120 <893600387@qq.com >
Signed-off-by: vx120 <57470515+vx120@users.noreply.github.com >
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-13 08:44:31 +00:00
Michael Goin
8e958902ee
[Bugfix] Disable fine-grained prefix-cache hits for incompatible hybrid KV layouts ( #51843 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
2026-08-12 04:08:40 -07:00
Michael Goin and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
e3fe212eaf
[Bugfix] Generalize KV block zeroing to AttentionSpec ( #51749 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-11 12:13:09 -04:00
Dao007forever and OpenAI Codex
ce07118669
[Bugfix][Core] Preserve Mamba running CoW after external hits ( #51766 )
...
Signed-off-by: Dao Le <Dao007forever@gmail.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-11 18:15:24 +08:00
Wentao Ye
0914ed2e81
[Perf] Adaptive budget for spec scheduled input tokens, ~60% better Kimi K3 DSpark TTFT ( #51725 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-08-10 20:52:21 -07:00
243c63baf5
[V1][Scheduler] Apply Mamba alignment before encoder caps ( #51603 )
...
Signed-off-by: Jiangyun Zhu <riverclouds.zhu@qq.com >
Co-authored-by: Akshat Anand <40275336+cipheraxat@users.noreply.github.com >
Co-authored-by: Codex <codex@openai.com >
2026-08-10 16:37:17 +08:00
61c1dd0966
[EC Connector] Call to EC Connector update_connector_output from scheduler ( #49579 )
...
Signed-off-by: omerpaz95 <omerpaz95@gmail.com >
Signed-off-by: Nicolò Lucchesi <nicolo.lucchesi@gmail.com >
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-08-10 14:25:57 +08:00
Yifan Qiao and OpenAI Codex
d6941300fc
[BugFix] Scope divergent hybrid cache hits to capable connectors ( #50344 )
...
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-09 15:38:13 -07:00
58d3918e3e
[BugFix] Preserve divergent FA hits with external Mamba state ( #51468 )
...
Signed-off-by: Roger Wang <hey@rogerw.io >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Signed-off-by: Jeff Ma <jeffjma@umich.edu >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-08-08 20:20:16 +08:00
Tianyu Guo
dd856e48bb
[Bugfix][EPD][Model Runner V2] Skip gather mm embeddings for encoder only instance ( #51222 )
...
Signed-off-by: Tianyu Guo <guoty@inferact.ai >
2026-08-06 20:40:47 -07:00
Tejas
4d341ca829
fix: resolve silent request skipping in PRIORITY scheduling ( #49206 )
...
Signed-off-by: Tejas-Raj01 <rajtejas.xyz@gmail.com >
2026-08-06 22:21:21 +00:00
c56f169d9a
[Bugfix] Keep mamba align prefill chunks block-aligned past last_cache_position ( #51113 )
...
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Hernan <kodek@eat1337.com >
Co-authored-by: yanghui1-arch <3053034939@qq.com >
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
2026-08-06 10:21:00 -07:00
Canlin Guo
5370461649
[BugFix][KV Cache] Fix hybrid prefix caching with hidden-state extraction ( #51108 )
...
Signed-off-by: Canlin Guo <canlinguosdu@gmail.com >
2026-08-06 06:42:37 +00:00
Tianyu Guo
50c51682a1
[Bugfix][EC Connector] Don't stop an encoder-instance request before its images are encoded ( #50275 )
...
Signed-off-by: Tianyu Guo <guoty@inferact.ai >
2026-08-05 05:53:58 +00:00
f0de1a604c
[Bugfix] Shard UniformTypeKVCacheSpecs block table width under DCP ( #50823 )
...
Signed-off-by: Mikhail Kostryukov <mike@triptrack.net >
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com >
2026-08-03 17:34:29 +00:00
+13
aeeb36b1f1
[New model] Kimi K3 ( #50000 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Signed-off-by: Bugen Zhao <i@bugenzhao.com >
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com >
Co-authored-by: Isotr0py <Isotr0py@outlook.com >
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Ziming Huang <zelda.huanghuang@gmail.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: Isotr0py <mozf@inferact.ai >
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com >
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Summer Yang <girasoleyang@gmail.com >
Co-authored-by: Kevin H. Luu <khluu000@gmail.com >
Co-authored-by: Bowen Wang <abmfy@icloud.com >
Co-authored-by: gnovack <novackgm@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: xiaozhoupy <peiyuanzhou1994@gmail.com >
Co-authored-by: Roy Wang <yasong.wang@inferact.ai >
Co-authored-by: Jeff (Junze) Ma <93145857+majunze2001@users.noreply.github.com >
Co-authored-by: Codex <codex@openai.com >
Co-authored-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com >
2026-07-30 03:49:58 -07:00
Nick Hill and Claude Opus 5
437e0b7f8e
[BugFix] Fix P/D preemption race condition ( #50297 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
2026-07-30 01:11:18 +00:00
a0c092ee72
[BugFix] Fix num_output_placeholders preemption underflow ( #48245 )
...
Signed-off-by: Chris Eastwood <chris.eastwood@pwn4g3.dev >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Chris Eastwood <chris.eastwood@pwn4g3.dev >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
2026-07-29 06:41:54 -07:00
Dao007forever and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
d742856610
[3/N][Core][KV Connector] Support reliable partial-tail KV offload for sub-block prompts ( #49502 )
...
Signed-off-by: Dao Le <Dao007forever@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-26 23:22:17 -07:00
liuzhenwei and Kunshang Ji
6a1acac3fe
[BUGFIX] Fix log capture in KV test ( #49655 )
...
Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com >
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com >
2026-07-25 00:36:53 +00:00
Michael Goin and OpenAI Codex
275556c35c
[Bugfix] Detect mixed precision in packed KV cache specs ( #49623 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Signed-off-by: Michael Goin <mgoin64@gmail.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-07-23 20:07:06 -07:00
Michael Goin and OpenAI Codex
e18f0037a5
[Bugfix][KV cache] Support sparse-MLA targets with SWA drafts ( #48776 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-07-23 11:30:50 -07:00
zhrrr and Nick Hill
12213c6795
[Bugfix] handle grammar compilation failures to avoid engine crash ( #47312 )
...
Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-07-23 18:13:49 +08:00
Nick Hill
229e01e9e1
[BugFix] Handle per-group prefix-hit divergence for hybrid models with KV connector ( #48425 )
2026-07-22 17:19:11 -07:00
Summer Yang
f3a920a076
[Core][DSV4] Compact MXFP4 indexer KV cache and packed group overlays ( #48993 )
2026-07-22 16:58:44 -07:00
8950394e0a
[Bugfix] Prefix-cache metrics double-counted when a KV connector defers requests ( #48860 )
...
Signed-off-by: Seiji Eicher <58963096+eicherseiji@users.noreply.github.com >
Signed-off-by: Seiji Eicher <seiji@anyscale.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Lixi Zhang <muxixibbb@gmail.com >
Co-authored-by: harjothkhara <harjoth.khara@gmail.com >
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-07-21 15:02:03 +01:00
2e2e626b40
[Bugfix] Count per-group blocks in get_max_concurrency_for_kv_cache_config ( #48317 )
...
Signed-off-by: David Orman <ormandj@corenode.com >
Co-authored-by: Luke Alonso <lalonso@gmail.com >
Co-authored-by: Martin Vit <martin@voipmonitor.org >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
2026-07-21 00:29:10 +00:00
yzong-rh
a287eb163f
[Front-end] [Messages] Populate num_cache_creation_tokens ( #48535 )
...
Signed-off-by: Yifan Zong <yzong@redhat.com >
2026-07-18 13:04:35 -04:00
5a65ba5f17
[Refactor] Move iteration logging to the frontend ( #46647 )
...
Signed-off-by: maxyanghu <hyoung2991@gmail.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: Shang Wang <shangw@nvidia.com >
2026-07-15 17:59:05 -07:00
32aef44388
[Bugfix] Include inline per-token-head scales in offloaded page transfer width ( #48411 )
...
Signed-off-by: Itay Etelis <itay.etelis@ibm.com >
Signed-off-by: Itay Etelis <Itay.etelis@gmail.com >
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com >
Co-authored-by: Itay Etelis <itay.etelis@ibm.com >
Co-authored-by: Itay Etelis <Itay.etelis@gmail.com >
2026-07-14 16:07:26 +03:00
Nick Hill
8ac8375270
[Core] Preserve Marconi caching with selective hybrid cache retention ( #47782 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-07-13 21:24:20 +01:00
Nicolò Lucchesi and Claude
56a357ed33
[Bugfix][KV Cache] Don't route uniform-page-size MLA+SWA models into DeepseekV4 packing ( #48256 )
...
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai >
Co-authored-by: Claude <noreply@anthropic.com >
2026-07-13 08:16:24 +00:00
omerpaz95 and Or Ozeri
8df14cfc8c
[EC Connector] Add EC Transfer Params ( #42433 )
...
Signed-off-by: omerpaz95 <omerpaz95@gmail.com >
Co-authored-by: Or Ozeri <oro@il.ibm.com >
2026-07-12 14:35:33 +03:00
Jiangyun Zhu and Yifan Qiao
481e481be7
[2/N][Core] support partial prefix cache hit for hybrid model ( #46384 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
2026-07-12 05:37:51 +00:00
e5588e49bc
[Core][KV events] Report prefix-cache-reused blocks in full report mode ( #45261 )
...
Signed-off-by: Lei Gong <gonglei25@huawei.com >
Co-authored-by: Lei Gong <gonglei25@huawei.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-07-09 22:46:54 -07:00
cd0de48d08
[Bugfix][V1] Free out-of-window blocks on the processed-token basis under async scheduling ( #47728 )
...
Signed-off-by: Saddss <28726669061@qq.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Saddss <28726669061@qq.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-07-08 13:34:19 +01:00
Sahil Kadadekar and Claude Fable 5
55da232db6
[Bugfix] Pad Mamba page size instead of scaling block_size in unify_kv_cache_spec_page_size ( #45207 )
...
Signed-off-by: Sahil170595 <147995121+Sahil170595@users.noreply.github.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-07-07 22:01:34 +00:00
93e2ab7111
Disable dynamic speculative decoding when DP is enabled ( #45963 )
...
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com >
2026-07-07 12:52:41 +00:00
Kyung Sub Lee (Daniel) and Claude Fable 5
c5b66233b2
[Bugfix][Spec Decode] Skip uniform spec-decode padding for diffusion models ( #47464 )
...
Signed-off-by: kl527 <kl527@cornell.edu >
Signed-off-by: Kyung Sub Lee (Daniel) <66861800+kl527@users.noreply.github.com >
Co-authored-by: Claude Fable 5 <noreply@anthropic.com >
2026-07-07 09:25:12 +00:00
Ting SUN and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
373eb314af
[Bugfix][Core] Fix num_output_placeholders underflow with async scheduling + spec decode ( #46066 )
...
Signed-off-by: Ting Sun <suntcrick@gmail.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-07-06 13:50:38 +00:00
yue.yu and Benjamin Chislett
e7c9df9449
[Bugfix][Structured Output][Spec Decode] Constrain bitmask and trim grammar advance at the reasoning boundary ( #44297 )
...
Signed-off-by: Allen.Yu <yuyue0225sc@163.com >
Signed-off-by: yue.yu <yuyue0225sc@163.com >
Co-authored-by: Benjamin Chislett <chislett.ben@gmail.com >
2026-07-04 09:08:45 +00:00
