 Ziming HuangandOpenAI Codex
|
a26b71d868
|
[Bugfix][PD] Pad resumed speculative decode requests (#55126)
Signed-off-by: Ziming Huang <48115868+ZeldaHuang@users.noreply.github.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
|
2026-09-04 11:05:33 +08:00 |
|
|
bnellnmandTyler Michael Smith
|
25268f0c9c
|
[KVConnector] Add retention interval to OffloadingConnector (#51886)
Signed-off-by: Bill Nell <bnell@redhat.com>
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
|
2026-09-04 02:42:20 +00:00 |
|
|
|
d9e2b5238d
|
[Core][KV Events] Echo session_id on GPU BlockStored events (#51381)
Signed-off-by: liuguangli <liuguangli35@gmail.com>
Signed-off-by: jinx-machinepulse <2679622408@qq.com>
Co-authored-by: liuguangli <liuguangli35@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
2026-09-04 02:22:50 +00:00 |
|
|
|
560ef78bfe
|
[Perf][Model Runner V2] Compact sampling masks on GPU instead of unpacking the full-vocab bitmask on CPU (#54901)
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: vx120 <893600387@qq.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: vx120 <893600387@qq.com>
|
2026-09-03 19:03:36 -07:00 |
|
|
Jiangyun Zhu
|
98ed0856f3
|
[Model] add GLM-5.3-Flash support (#53906)
|
2026-09-03 09:40:35 -07:00 |
|
|
waizuichougouandNick Hill
|
3f1af35e01
|
[Bugfix][Multimodal] Handle prefix-covered items in SHM worker cache (#54994)
Signed-off-by: waizuichougou <2082431897@qq.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
|
2026-09-03 06:39:34 +00:00 |
|
|
|
7cb9a88f50
|
[Bugfix][Mooncake] Offload producer partial tails on request finish (#52832)
Signed-off-by: Dao Le <Dao007forever@gmail.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: OpenAI Codex <noreply@openai.com>
|
2026-09-01 13:23:35 -07:00 |
|
|
Ziming HuangandOpenAI Codex
|
481839ad9e
|
[Feature][Spec] Support disabling trailing prefix-cache block dropping (#53388)
Signed-off-by: Ziming Huang <zelda.huanghuang@gmail.com>
Signed-off-by: Ziming Huang <48115868+ZeldaHuang@users.noreply.github.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
|
2026-09-01 19:12:36 +08:00 |
|
|
YukioZzzandandyluo7
|
2ba984a5d0
|
[ROCm][DSpark][DCP] Serve prefix cache hits under DCP for Kimi-K3 (#53598)
Signed-off-by: Yichao Zhu <Yichao.Zhu@amd.com>
Co-authored-by: andyluo7 <andy.luo@amd.com>
|
2026-08-31 11:18:10 -07:00 |
|
|
Artem Perevedentsev
|
39e276eaeb
|
[Structured Output] Let terminal grammars stop under min_tokens (II) (#54218)
Signed-off-by: Artem Perevedentsev <aperevedents@nvidia.com>
|
2026-08-31 20:39:27 +04:00 |
|
|
+5
|
e126687a9a
|
[Model] Support Qwen3.8-Flash-Next (#53896)
Signed-off-by: huanghaoyan.hhy <huanghaoyan.hhy@alibaba-inc.com>
Signed-off-by: andyluo7 <andy.luo@amd.com>
Signed-off-by: lishunyang12 <lishunyang03@gmail.com>
Signed-off-by: Harry Huang <huanghaoyan.hhy@alibaba-inc.com>
Signed-off-by: Thien Tran <gau.nernst@yahoo.com.sg>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: KungYork <firelikest@163.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: andyluo7 <43718156+andyluo7@users.noreply.github.com>
Co-authored-by: Isotr0py <mozf@inferact.ai>
Co-authored-by: andyluo7 <andy.luo@amd.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: lishunyang12 <lishunyang03@gmail.com>
Co-authored-by: Vadim Gimpelson <156319763+vadiklyutiy@users.noreply.github.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
|
2026-08-31 05:57:56 +00:00 |
|
|
Hotragn Pettugani
|
3958a420f0
|
[Bugfix][V1] Keep an encoder cache entry until its last occurrence is freed (#54284)
Signed-off-by: hotragn <hotragn.pettugani_2024@woxsen.edu.in>
|
2026-08-29 05:08:01 +00:00 |
|
|
|
93ab92be0c
|
[Bugfix][AMD] Annotate draft KV cache groups on the hybrid grouping path (#52047)
Signed-off-by: okorzh-amd <okorzh-amd@users.noreply.github.com>
Co-authored-by: okorzh-amd <okorzh-amd@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com>
|
2026-08-28 21:53:46 -05:00 |
|
|
|
6b110badbb
|
[Bugfix][Mooncake] Save exact Mamba boundary states (#51358)
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Dao Le <daole@inferact.ai>
|
2026-08-29 02:40:53 +00:00 |
|
|
Wei ZhaoandYifan Qiao
|
7fd9cc036e
|
[KV Connector] Support MooncakeStore with hybrid DCP prefix caching (#53324)
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
Signed-off-by: wzhao18 <weizha@nvidia.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
|
2026-08-29 02:10:23 +00:00 |
|
|
Summer YangandNick Hill
|
2aac565cae
|
[Core][KV Connector] Start async KV loads after the forward launch when no sync loads are scheduled (#53333)
Signed-off-by: Summer Yang <girasoleyang@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
|
2026-08-28 21:58:00 +00:00 |
|
|
haic0andCursor
|
53bf990502
|
[Bugfix][KV Cache] Prevent negative external block allocation (#52707)
Signed-off-by: haic0 <149741444+haic0@users.noreply.github.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
|
2026-08-27 20:10:45 -07:00 |
|
|
Nick Hill
|
76c0c6530e
|
[Bugfix][Scheduler] Don't pad spec decode up to max_model_len (#53962)
|
2026-08-27 08:30:22 -07:00 |
|
|
Matthew Bonanni
|
d85708f7a4
|
[1/N][KV Connector] Identify externally transferable KV cache groups (#53779)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
|
2026-08-27 11:21:24 -04:00 |
|
|
Tianyu Guo
|
12c84b98dd
|
[Config][EC] Normalize producer-only encoder config (#53656)
Signed-off-by: Tianyu Guo <guoty@inferact.ai>
|
2026-08-25 07:53:04 +00:00 |
|
|
drakoshaandClaude
|
9bf0d4717e
|
[Bugfix] Keep grid dims for XD-RoPE models on a prefix-cache hit (#53456)
Signed-off-by: Mikhail Kostryukov <mike@triptrack.net>
Co-authored-by: Claude <noreply@anthropic.com>
|
2026-08-25 01:14:32 +00:00 |
|
|
|
9eb9d9d395
|
[Perf] Support internal prefill checkpoints for Mamba prefix caching, 9%~25% TTFT improvement (#52789)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Ziming Huang <zelda.huanghuang@gmail.com>
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: Ziming Huang <zelda.huanghuang@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
|
2026-08-22 10:32:40 -07:00 |
|
|
ℍ𝕠𝕝𝕝𝕠𝕨 𝕄𝕒𝕟
|
b2db227a7c
|
[Bugfix][R3] Unwrap UniformTypeKVCacheSpecs when selecting the routed-experts KV group (#53240)
Signed-off-by: Hollow Man <hollowman@opensuse.org>
|
2026-08-21 19:13:53 -07:00 |
|
|
Lucas Wilkinson
|
8bdc70ec7b
|
[6/N][KV-Cache Layout Refactor] Standardize KV cache layout (#51718)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
|
2026-08-21 17:47:49 -07:00 |
|
|
|
7cfb97e337
|
[Frontend][Core][Spec Decode] Per-request acceptance stats in OpenAI API responses (#48915)
Signed-off-by: Matthew Kotila <7692737+matthewkotila@users.noreply.github.com>
Co-authored-by: Reed Meyerson <31574681+reed-meyerson@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
2026-08-20 15:42:49 -07:00 |
|
|
|
76fb6d210a
|
[Bugfix][Core] Reserve the KV null block when validating max_model_len (#47272)
Signed-off-by: hyungjun oh <92hyungjun@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
|
2026-08-20 02:49:23 +00:00 |
|
|
Xiaoan (Sean) LiuandNick Hill
|
db92053e97
|
[Core] Skip broadcasting mm tensor data to workers for prefix-cache-covered items (#52041)
Signed-off-by: Xiaoan (Sean) Liu <shawnliu0327@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
|
2026-08-19 07:16:39 -07:00 |
|
|
Russell BryantandClaude Opus 4.8
|
ef47a897e2
|
[Core] Make prefix-cache NONE_HASH deterministic by default (#51875)
Signed-off-by: Russell Bryant <rbryant@redhat.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
|
2026-08-18 23:14:39 +00:00 |
|
|
|
0db502c8d8
|
[Kimi-K3] support DCP partial prefix cache hit (#50493)
Signed-off-by: Summer Yang <girasoleyang@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
|
2026-08-17 17:05:51 -07:00 |
|
|
Tyler Michael SmithandOpenAI Codex
|
017e9f4448
|
Promote prefix_cache_retention_interval to an argument and change the default to 0 (#52216)
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
|
2026-08-17 13:20:50 +00:00 |
|
|
Tianyu Guo
|
eee538d5da
|
[Bugfix][V1][Multimodal] Ignore stale same-step encoder cache evictions (#52482)
Signed-off-by: Tianyu Guo <guoty@inferact.ai>
|
2026-08-16 22:21:55 +00:00 |
|
|
mispa-ms
|
6593754e61
|
[Bugfix][Spec Decode] Keep EAGLE cache registration on the partial-hash-hit path (#52419)
|
2026-08-15 21:03:53 -07:00 |
|
|
Lucas WilkinsonandClaude
|
57bd0ed441
|
[5/N][KV-Cache Layout Refactor] Backend-published KV packing via customize_spec (#51704)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Claude <noreply@anthropic.com>
|
2026-08-14 09:47:55 -04:00 |
|
|
Giancarlo DelfinandClaude Fable 5
|
f80b66f548
|
[Model Runner V2][Spec Decode] Add KV cache support for multi-layer MTP (#50062)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
|
2026-08-13 16:22:29 -07:00 |
|
|
|
50ba4bc6b2
|
[Feature] Mask Replay (#49577)
Signed-off-by: vx120 <893600387@qq.com>
Signed-off-by: vx120 <57470515+vx120@users.noreply.github.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
|
2026-08-13 08:44:31 +00:00 |
|
|
Michael Goin
|
8e958902ee
|
[Bugfix] Disable fine-grained prefix-cache hits for incompatible hybrid KV layouts (#51843)
Signed-off-by: mgoin <mgoin64@gmail.com>
|
2026-08-12 04:08:40 -07:00 |
|
|
Michael Goinandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
e3fe212eaf
|
[Bugfix] Generalize KV block zeroing to AttentionSpec (#51749)
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
2026-08-11 12:13:09 -04:00 |
|
|
Dao007foreverandOpenAI Codex
|
ce07118669
|
[Bugfix][Core] Preserve Mamba running CoW after external hits (#51766)
Signed-off-by: Dao Le <Dao007forever@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
|
2026-08-11 18:15:24 +08:00 |
|
|
Wentao Ye
|
0914ed2e81
|
[Perf] Adaptive budget for spec scheduled input tokens, ~60% better Kimi K3 DSpark TTFT (#51725)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
|
2026-08-10 20:52:21 -07:00 |
|
|
|
243c63baf5
|
[V1][Scheduler] Apply Mamba alignment before encoder caps (#51603)
Signed-off-by: Jiangyun Zhu <riverclouds.zhu@qq.com>
Co-authored-by: Akshat Anand <40275336+cipheraxat@users.noreply.github.com>
Co-authored-by: Codex <codex@openai.com>
|
2026-08-10 16:37:17 +08:00 |
|
|
|
61c1dd0966
|
[EC Connector] Call to EC Connector update_connector_output from scheduler (#49579)
Signed-off-by: omerpaz95 <omerpaz95@gmail.com>
Signed-off-by: Nicolò Lucchesi <nicolo.lucchesi@gmail.com>
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
2026-08-10 14:25:57 +08:00 |
|
|
Yifan QiaoandOpenAI Codex
|
d6941300fc
|
[BugFix] Scope divergent hybrid cache hits to capable connectors (#50344)
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
|
2026-08-09 15:38:13 -07:00 |
|
|
|
58d3918e3e
|
[BugFix] Preserve divergent FA hits with external Mamba state (#51468)
Signed-off-by: Roger Wang <hey@rogerw.io>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Signed-off-by: Jeff Ma <jeffjma@umich.edu>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
|
2026-08-08 20:20:16 +08:00 |
|
|
Tianyu Guo
|
dd856e48bb
|
[Bugfix][EPD][Model Runner V2] Skip gather mm embeddings for encoder only instance (#51222)
Signed-off-by: Tianyu Guo <guoty@inferact.ai>
|
2026-08-06 20:40:47 -07:00 |
|
|
Tejas
|
4d341ca829
|
fix: resolve silent request skipping in PRIORITY scheduling (#49206)
Signed-off-by: Tejas-Raj01 <rajtejas.xyz@gmail.com>
|
2026-08-06 22:21:21 +00:00 |
|
|
|
c56f169d9a
|
[Bugfix] Keep mamba align prefill chunks block-aligned past last_cache_position (#51113)
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Hernan <kodek@eat1337.com>
Co-authored-by: yanghui1-arch <3053034939@qq.com>
Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
|
2026-08-06 10:21:00 -07:00 |
|
|
Canlin Guo
|
5370461649
|
[BugFix][KV Cache] Fix hybrid prefix caching with hidden-state extraction (#51108)
Signed-off-by: Canlin Guo <canlinguosdu@gmail.com>
|
2026-08-06 06:42:37 +00:00 |
|
|
Tianyu Guo
|
50c51682a1
|
[Bugfix][EC Connector] Don't stop an encoder-instance request before its images are encoded (#50275)
Signed-off-by: Tianyu Guo <guoty@inferact.ai>
|
2026-08-05 05:53:58 +00:00 |
|
|
|
f0de1a604c
|
[Bugfix] Shard UniformTypeKVCacheSpecs block table width under DCP (#50823)
Signed-off-by: Mikhail Kostryukov <mike@triptrack.net>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
|
2026-08-03 17:34:29 +00:00 |
|
|
+13
|
aeeb36b1f1
|
[New model] Kimi K3 (#50000)
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Signed-off-by: Jee Jee Li <jeejeelee@inferact.ai>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
Co-authored-by: Thien Tran <gau.nernst@yahoo.com.sg>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Ziming Huang <zelda.huanghuang@gmail.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Isotr0py <mozf@inferact.ai>
Co-authored-by: aoshen02 <aoshen02@users.noreply.github.com>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Jee Jee Li <jeejeelee@inferact.ai>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Summer Yang <girasoleyang@gmail.com>
Co-authored-by: Kevin H. Luu <khluu000@gmail.com>
Co-authored-by: Bowen Wang <abmfy@icloud.com>
Co-authored-by: gnovack <novackgm@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: xiaozhoupy <peiyuanzhou1994@gmail.com>
Co-authored-by: Roy Wang <yasong.wang@inferact.ai>
Co-authored-by: Jeff (Junze) Ma <93145857+majunze2001@users.noreply.github.com>
Co-authored-by: Codex <codex@openai.com>
Co-authored-by: Wei Zhao <51183510+wzhao18@users.noreply.github.com>
|
2026-07-30 03:49:58 -07:00 |
|