Taksh Kothari
12b005eefa
[Bugfix] Don't split a multimodal item that a prefill chunk starts exactly on ( #52412 )
...
Signed-off-by: Taksh <takshkothari09@gmail.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-10-13 15:32:38 +00:00
2cc2e68d1c
[Core][Mamba] Remove leftovers of mamba_cache_mode="all" ( #60043 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Signed-off-by: Jiangyun Zhu <zhujiangyun@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-13 00:58:59 -07:00
df6c7ad031
[Core][Refactor] Move KV cache planning logic to kv_cache_planning.py ( #61183 )
...
Signed-off-by: Dakai An <dakaian108@gmail.com >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-13 03:05:39 +00:00
ztysth and OpenAI Codex
1bdd3fa2d5
[Bugfix][Core] Check multimodal chunk boundaries before encoder cache reuse ( #59721 )
...
Signed-off-by: ztysth <rnk.liulw@gmail.com >
Signed-off-by: ztysth <166926442+ztysth@users.noreply.github.com >
Co-authored-by: OpenAI Codex <noreply@openai.com >
2026-10-12 19:25:35 +00:00
Woosuk Kwon and Claude Opus 5.5
e1a5086256
[Attention] Remove cascade attention ( #61126 )
...
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-12 10:41:07 -07:00
Nick Hill
c059bee812
[MRV2] Switch CI tests still running on MRV1 unnecessarily ( #61014 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-10-11 09:57:07 -07:00
Matthew Bonanni and Claude Opus 5.5
d5f356dae0
[HiSparse] Own GPU residency instead of editing core block-table rows ( #60647 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-09 21:25:44 -07:00
02e55d2af3
[KV Cache] GLM-5.3-Flash: use the generic packed KV layout; CircularBufferSpec tail with speculative ring slots ( #57169 )
...
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Signed-off-by: Dakai An <dakaian108@gmail.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: OpenAI Codex <noreply@openai.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
2026-10-09 15:48:37 -04:00
d8fa463e2b
[Bugfix][Prefix caching] Fix inconsistent hybrid Mamba prefix cache boundaries ( #60533 )
...
Signed-off-by: Wei Zhao <weizha@aws-cmh-slurm-1-login-01.cm.cluster >
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Wei Zhao <weizha@aws-cmh-slurm-1-login-01.cm.cluster >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-10-08 18:33:48 +00:00
Nicolò Lucchesi
e161a0798c
[Perf][Core] Reduce free KV cache queue overhead ( #60269 )
...
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai >
2026-10-08 10:17:11 +00:00
7816afdb25
[Bugfix][Core] Drain async KV loads in pause(mode="wait"), and pause the drain test only once a request is in flight ( #60572 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-08 09:33:41 +00:00
ef63c23d35
[Bugfix][DSv4.1] Skip SWA bounded replay for KV loads that carry the window ( #59197 )
...
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-07 15:28:42 -07:00
e6fc81bc78
[Bugfix][Scheduler] Let pooling chunked prefill use full context ( #48039 )
...
Signed-off-by: Tommy <tran.tommy@hotmail.com >
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-10-07 07:32:00 +00:00
e37e51dd24
[Metrics] Expose cached prompt tokens by cache tier ( #56318 )
...
Signed-off-by: Cam Quilici <cjquilici@gmail.com >
Signed-off-by: Cam Quilici <cameron@semianalysis.com >
Co-authored-by: Cam Quilici <cameron@semianalysis.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai >
2026-10-06 14:52:05 -04:00
6fa67afed9
[Bugfix][Core] Retire stale Mamba spec blocks in prefill checkpoint steps ( #59759 )
...
Signed-off-by: Simon Veitner <sveitner@redhat.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
2026-10-06 11:01:19 +08:00
2e3154aa18
[HiSparse] Log steady-state max concurrency and expose host-tier utilization gauges ( #58949 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com >
2026-10-05 14:20:32 -07:00
bzsuni and cjackal
528772a4bf
[Bugfix][Core] Retain encoder cache references for repeated multimodal inputs ( #59942 )
...
Signed-off-by: bzsuni <bingzhe.sun@daocloud.io >
Signed-off-by: cjackal <44624812+cjackal@users.noreply.github.com >
Co-authored-by: cjackal <44624812+cjackal@users.noreply.github.com >
2026-10-05 15:52:46 +00:00
Nicolò Lucchesi
30d4032363
[Model][GLM-5.3-Flash] Support DCP for the kpool sparse indexer ( #59211 )
...
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai >
2026-10-05 16:44:14 +02:00
ca0df1f122
[Bugfix][Core] Exempt exactly the blocks an async KV load writes from zeroing ( #59504 )
...
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Codex <noreply@openai.com >
2026-10-03 06:26:12 +00:00
f7999d2e44
[Bugfix][HiSparse] Size the KV cache from the groups HiSparse allocates ( #59450 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com >
2026-10-02 04:11:55 +00:00
ca65eb67d9
[Perf][HiSparse] Avoid repeated prefix scans and residency updates ( #57930 )
...
Signed-off-by: S1ro1 <matej.sirovatka@gmail.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com >
2026-10-01 21:48:18 +00:00
Matthew Bonanni and Claude Opus 5.5
3a69636645
[Bugfix][HiSparse] Fix a chunked-prefill preemption livelock ( #59494 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-01 13:02:31 -04:00
cb6458c5ce
[Core] Remove deprecated mamba_cache_mode "all" ( #58997 )
...
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: Thomas Parnell <tpa@zurich.ibm.com >
2026-10-01 02:59:37 -07:00
70be6aae19
[Feature][Spec Decode] Support sampling mask replay for MRV2 MTP ( #59359 )
...
Signed-off-by: Cuiping Cheng <chengcuipingswu@163.com >
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Cuiping Cheng <chengcuipingswu@163.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-01 13:51:39 +08:00
Nick Hill and Claude Opus 5.5
c055c1e075
[Core] Include the LoRA path in prefix-cache block hashes ( #59335 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-10-01 08:37:09 +08:00
kliuae and Nick Hill
7e583e615c
[Bugfix][Core] Fix mamba prefill checkpoint block reservation and prompt-end eviction in align mode ( #59175 )
...
Signed-off-by: kliuae <kuanfu.liu@embeddedllm.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-09-30 21:26:54 +00:00
3eb6cec22a
[Bugfix][HiSparse] Adopt GPU prefix copies after the hit's allocation ( #59282 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-30 15:43:09 -04:00
Yueh-Ting (eop) Chen and Matthew Bonanni
ff1b87cca2
[Bugfix][HiSparse] Preserve host prefix publication after request completion ( #59007 )
...
Signed-off-by: Yueh-Ting Chen <yueh.ting.chen@gmail.com >
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com >
2026-09-30 16:14:51 +00:00
Giulio De Pasquale and Isotr0py
2e52a558f0
[Bugfix][Core] Schedule encoder-only prompts larger than one step ( #59029 )
...
Signed-off-by: Giulio De Pasquale <git@depasquale.giugl.io >
Co-authored-by: Isotr0py <2037008807@qq.com >
2026-09-30 22:04:28 +08:00
Varun Sundar Rabindranath
5463fe4962
[KV-Offloading][TP] : Expand replicated_layout detection to multi-group MLA ( #57652 )
2026-09-30 09:11:03 -04:00
90e13fc757
[Bugfix][HiSparse] Never allocate GPU pages without host backing ( #59036 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-30 04:13:29 +00:00
Mia Ojeda
e006d761a5
[Bugfix][CPU][DiffusionGemma] Support narrower canvas w/ sync scheduling ( #59107 )
...
Signed-off-by: ojeda-e <estefania.b.ojeda@gmail.com >
2026-09-29 23:28:50 -04:00
765872e7ed
[Core][BugFix] Tag prefix-cache extra keys by source ( #51899 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Lucas Bourtoule <35483370+dhalf@users.noreply.github.com >
Co-authored-by: Tai An <antai12232931@outlook.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com >
2026-09-29 18:24:59 -07:00
d882bddbea
[Bugfix][Mamba] Keep the prompt-end prefill checkpoint under sparse retention ( #59146 )
...
Signed-off-by: Jared Wen <jaredwen@inferact.ai >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com >
2026-09-29 21:00:16 +00:00
aoshen02 and OpenAI Codex
4861833ae2
[Bugfix][Core] Allow AuxOutput reset after abort without running requests ( #59060 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-09-29 09:23:32 +00:00
Nick Hill and Robert Shaw
42dbe4578c
[Core] Rework scheduler skipped_waiting queue ( #58947 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
2026-09-28 20:03:16 +00:00
yzong-rh
2f4a975901
[Bugfix] Fix resumable request + async scheduling handoff race ( #58259 )
...
Signed-off-by: Yifan Zong <yzong@redhat.com >
2026-09-28 15:26:57 -04:00
927c87b347
[PCP][DCP] Support DCP target model with non-DCP Dspark ( #56723 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com >
Co-authored-by: Codex <noreply@openai.com >
2026-09-26 18:12:35 -04:00
29468dde8b
[Bugfix][KV Cache][MLA] Align packed block strides for V3.2 sparse MLA ( #55528 )
...
Signed-off-by: lz <145014769+200lz@users.noreply.github.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-25 05:33:36 +00:00
d5051abaf1
[Bugfix][Mamba] Restore prompt-tail prefix-cache hits with MTP ( #58368 )
...
Signed-off-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com >
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com >
Co-authored-by: Benjamin Chislett <bchislett@nvidia.com >
2026-09-24 23:46:42 +00:00
f6aa2919bc
[Bugfix][Core] Keep every multimodal feature in the partial-block KV event ( #58288 )
...
Signed-off-by: haosenwang1018 <haosenwang1018@users.noreply.github.com >
Co-authored-by: haosenwang1018 <haosenwang1018@users.noreply.github.com >
Co-authored-by: Claude Opus 5 <noreply@anthropic.com >
2026-09-24 23:25:01 +00:00
Jellow
e6c07ea576
[Bugfix][KV Cache] Fix incremental multimodal block hashing ( #51694 )
...
Signed-off-by: Jellow <49915976+CZT0@users.noreply.github.com >
Signed-off-by: Jellow <dvdx@foxmail.com >
2026-09-24 15:38:03 -07:00
Wentao Ye
3c42385d7d
[Refactor] Remove dead or duplicate tests ( #58446 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-09-24 10:29:56 -04:00
jackLei and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
bbc4ddeca9
[Bugfix][V1] Reject encoder-cache hits with mismatched embedding counts ( #57696 )
...
Signed-off-by: jackLei0901 <42642542+jackLei0901@users.noreply.github.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-24 08:40:47 +00:00
Robert Shaw
6697a7dd30
[Scheduler] Tune --long-prefill-token-threshold adaptiveness ( #58459 )
...
Signed-off-by: Robert Shaw <robertgshaw2@gmail.com >
2026-09-23 22:23:18 -04:00
Andreas Karatzas and OpenAI Codex
0f2a15c927
[Tests] Select V2 for diffusion scheduler unit tests ( #58272 )
...
Signed-off-by: Andreas Karatzas <Andreas.Karatzas@amd.com >
Co-authored-by: OpenAI Codex <codex@openai.com >
2026-09-23 02:04:22 -05:00
1b3b88ec2b
[Core] structured generation mode for DiffusionGemma model (Jev-like) ( #57250 )
...
Signed-off-by: Matt Mastracci <matthew@mastracci.com >
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: Razorback16 <razorback16@protonmail.com >
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com >
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com >
Co-authored-by: Codex <noreply@openai.com >
Co-authored-by: Razorback16 <razorback16@protonmail.com >
2026-09-22 20:51:17 +00:00
Navjot Singh and Codex
0bce411a07
[Bugfix] Annotate MTP draft KV cache groups positionally on the hybrid grouping path ( #55390 )
...
Signed-off-by: Navjot Singh <navjot.singh@shopify.com >
Co-authored-by: Codex <noreply@openai.com >
2026-09-22 00:31:26 +00:00
Robert Shaw and Robert Shaw
c64b15cde5
[Scheduler] Soften Long Prefill Tokens Threshhold ( #57951 )
...
Signed-off-by: Robert Shaw <robertgshaw2@gmail.com >
Signed-off-by: Robert Shaw <robshaw@redhat.com >
Co-authored-by: Robert Shaw <robshaw@redhat.com >
2026-09-21 17:19:42 -07:00
0b7f11a1ee
[AuxOutput] Add block-keyed storage for routed-expert outputs ( #45635 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: aoshen02 <aoshen@inferact.ai >
Co-authored-by: OpenAI Codex <codex@openai.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Codex <noreply@openai.com >
2026-09-21 10:25:05 +00:00