Commit Graph
646 Commits
Author SHA1 Message Date
Varun Sundar Rabindranath 5463fe4962 [KV-Offloading][TP] : Expand replicated_layout detection to multi-group MLA (#57652) 2026-09-30 09:11:03 -04:00
765872e7ed [Core][BugFix] Tag prefix-cache extra keys by source (#51899)
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Lucas Bourtoule <35483370+dhalf@users.noreply.github.com>
Co-authored-by: Tai An <antai12232931@outlook.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
2026-09-29 18:24:59 -07:00
faacc13565 [Perf][KV Connector][Mooncake] Pack hybrid/MLA KV into coalesced transfer regions (#57952)
Signed-off-by: staryxchen <staryxchen@tencent.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
2026-09-29 13:29:04 -07:00
YukioZzzandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 30ae8248b6 [KVConnector][MoRIIO] Support K3 DSpark hybrid READ (#57700)
Signed-off-by: Yichao Zhu <Yichao.Zhu@amd.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-29 19:14:24 +00:00
Andreas KaratzasandOpenAI Codex 3acddf01e7 [Bugfix][ROCm] Keep Mooncake bootstrap ports bound during startup (#58967)
Signed-off-by: Andreas Karatzas <andreas.karatzas@protonmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-09-29 04:14:26 -05:00
qizixi 6d3ea3c2c9 [KVConnector][NIXL] Support packed MLA KV layouts in pipeline-parallel push prefill (#50499)
Signed-off-by: zixi-qi <zixi@inferact.ai>
2026-09-28 18:50:46 +02:00
Chao Lei 386ac2573e [Bugfix][KV Connector] Retry Mooncake bootstrap registration on timeout (reopens #55763) (#58919) 2026-09-27 19:47:36 +00:00
YannikHintereggerandxijiade.aihemaiti fff03267da [Bugfix][NIXL] Release a dead peer's NIXL state without waiting for TTL (#50047)
Signed-off-by: Yannik Hinteregger <37209495+YannikHinteregger@users.noreply.github.com>
Co-authored-by: xijiade.aihemaiti <3146335281@qq.com>
2026-09-26 20:30:09 -07:00
eb0f2ca37f [mooncake] support CUSTOM_MEM_POOL in vllm (#49300)
Signed-off-by: bruce.xu <bruce.xb@alibaba-inc.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: bruce.xu <bruce.xb@alibaba-inc.com>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-09-27 00:26:05 +00:00
mevince cd1daf58a9 [Metrics][KV Offload] Add Prometheus metrics for SimpleCPUOffloadConnector (#57251)
Signed-off-by: Vincent <vincexxchan@gmail.com>
2026-09-26 17:17:11 -07:00
927c87b347 [PCP][DCP] Support DCP target model with non-DCP Dspark (#56723)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Co-authored-by: Codex <noreply@openai.com>
2026-09-26 18:12:35 -04:00
GokayAIandGokayAI c89b5b4176 [Bugfix][KV Connector] Reap expired NIXL leases behind a heartbeated head (#58292)
Signed-off-by: GokayAI <60583610+gokay-ai@users.noreply.github.com>
Co-authored-by: GokayAI <gokay-ai@users.noreply.github.com>
2026-09-25 17:23:44 +00:00
b3a87b2d04 [Bugfix][KVConnector] Finalize saves on steps without a forward (#57775)
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Kimi Code <noreply@moonshot.ai>
2026-09-25 00:06:36 +00:00
Zijing LiuYifan QiaoCodexmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
7a9406690e [KV Connector] Fix DecodeBench fp8 fill values and add a startup fill mode (#58472)
Signed-off-by: Zijing Liu <liuzijing2014@gmail.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-24 23:55:47 +00:00
stefankoncarevicandAndreas Karatzas 9ac3b7c69f [ROCm][CI] Mirror the DSv4-Flash disaggregated DP EP group on MI355 (#58558)
Signed-off-by: Stefan Koncarevic <Stefan.Koncarevic@amd.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-09-24 18:38:25 -05:00
snadampalandNicolò Lucchesi 6a86bf626e [PD][PushConnector] Record last activity of remotes on the D side (#52245)
Signed-off-by: Sunita Nadampalli <nadampal@amazon.com>
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@mistral.ai>
2026-09-24 23:15:32 +00:00
92e9c6bf13 [ROCm] Fix misrouting race-condition in multi-decode P/D disagg with mori-io (#51681)
Signed-off-by: Vincent Cave <vincent.cave@amd.com>
Signed-off-by: Shiksha Patel <shikpate@amd.com>
Co-authored-by: Shiksha Patel <shikpate@amd.com>
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
2026-09-24 19:52:43 +00:00
Kapil AryaandOr Ozeri 70fc359d25 [Bugfix][KV Offload] Retain offload event metadata through batch translation (#57453)
Signed-off-by: Kapil Arya <kapila@nvidia.com>
Signed-off-by: Kapil Arya <kapil.arya.17@gmail.com>
Co-authored-by: Or Ozeri <or@ozery.com>
2026-09-24 07:09:12 +03:00
Dao007foreverandCodex c4f6ce4874 [Bugfix][NIXL] Restore successful push completion reporting (#58188)
Signed-off-by: Dao Le <Dao007forever@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
2026-09-23 09:39:04 -07:00
Nicolò LucchesiandYifan Qiao 9488318c6a [Mooncake] Address review nits from #56855 (#57174)
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
Co-authored-by: Yifan Qiao <17067717+ivanium@users.noreply.github.com>
2026-09-23 10:15:45 +02:00
Karen Chung 986e217158 [Feature] Add first-class KV hints request envelope for programmatic KV management (#53423)
Signed-off-by: Karen Chung <karenc@nvidia.com>
2026-09-21 09:56:11 -07:00
Lucas WilkinsonandOpenAI Codex 8c0825090d [Bugfix][NIXL] Fix DCP pulls across MLA cache regions (#57389)
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-09-21 11:57:49 -04:00
YukioZzz 9679173788 [KVConnector][MoRIIO] Transfer hybrid mamba/KDA recurrent state in READ mode (#51052)
Signed-off-by: Yichao Zhu <Yichao.Zhu@amd.com>
2026-09-20 17:30:34 -04:00
ys2025-AI 6aab78f477 [Bugfix][KVConnector] Make ExampleHiddenStatesConnector abort-safe (#56841)
Signed-off-by: ys2025-AI <yuanshuaien@163.com>
2026-09-18 21:53:02 +00:00
Nicolò LucchesiandOpenAI Codex 49e0427ca5 [Bugfix][NIXL] Avoid receive reports for notification-only requests (#57570)
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-09-18 13:33:02 -07:00
Nicolò Lucchesi 164dca8d1e [Nixl] Separate transport-failure metrics from KV expiry (#55854)
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
2026-09-18 18:08:59 +02:00
67e5b0acc9 [Bugfix][HiSparse][NIXL] Import full blocks without tail prefill on D (#57049)
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
2026-09-17 15:26:10 -04:00
Itay EtelisandOr Ozeri b6e7c1f1f0 [kv_offload] Skip scratch groups (#57145)
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-09-17 13:22:49 +03:00
2743dc4f92 [Feature][Multimodal] Add cross-encoder caching to Mooncake P2P (#56242)
Signed-off-by: Zhou ziheng <jiaranran2@gmail.com>
Co-authored-by: Tianyu Guo <guoty@inferact.ai>
Co-authored-by: jiangkuaixue123 <jiangxiaozhou111@163.com>
Co-authored-by: Codex <noreply@openai.com>
2026-09-17 13:16:41 +08:00
Itay EtelisandOr Ozeri 9854b580df [Bugfix][KV Offload] Compact canonical MLA rows (#56799)
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-09-17 06:24:06 +03:00
Nicolò Lucchesi fc8132a5e5 [Bugfix] Fix np.float64 leaking into the KV transfer metrics log (#57068)
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
2026-09-16 18:09:05 +00:00
Harry MellorandClaude Opus 5 651a88c09a [CI] Ignore ruff D209, rejoin the docstrings it split, and silence incompatible-rule warnings (#57212)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 10:57:24 -07:00
Harry MellorandClaude Opus 5 8c1557a79c Add pydocstyle to the ruff rules (#52136)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Signed-off-by: Harry Mellor <hej.mellor@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-09-16 16:44:29 +00:00
Nicolò Lucchesi 804e5377aa [Bugfix][HiSparse][PD] Align region-mapped pulls across logical block sizes (#57077)
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
2026-09-16 18:39:07 +02:00
Nicolò Lucchesi 8b1d188046 [Bugfix][Mooncake] Report request-level KV load failures under HMA (#56855)
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
2026-09-16 14:55:40 +02:00
MINJUN GILandOr Ozeri f12fe10c5c [Bugfix][KV Offload] Track cache recency once per request (#51787)
Signed-off-by: mindungil <alswnsrlf12@naver.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-09-16 10:47:26 +03:00
961c5b6b7e [Bugfix][KV Offload] Restore MTP-retained sliding-window history (#56709)
Signed-off-by: jacklin78911-collab <jacklin78911@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-09-16 08:14:09 +03:00
alberto 75dc588269 [KV Offload] Add per-request max_load_tokens control (#55885)
Signed-off-by: Alberto Perdomo <aperdomo@redhat.com>
2026-09-16 06:57:38 +03:00
Giancarlo Delfin 85c1f58d50 [BugFix][Model Runner V2][Spec Decode] Fix decode instance's multi-layer MTP kv caches during P/D (#55055)
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
2026-09-15 17:31:25 -07:00
0fefffc934 [KVConnector][NIXL] Support attention-HMA layouts in pipeline-parallel push prefill (#50494)
Signed-off-by: zixi-qi <zixi@inferact.ai>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Yiliu Dong <91178480+qianlihuang@users.noreply.github.com>
Co-authored-by: Codex <noreply@openai.com>
2026-09-15 15:43:56 -07:00
Nicolò LucchesiandNick Hill 45779ef99f [Bugfix][NIXL] Fix multiple handles xfer race (#56104)
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-09-15 04:27:10 +00:00
Yizheng JiaoandNick Hill 381c61e008 [Bugfix][NIXL] Report a full prefix cache hit as finished (#56640)
Signed-off-by: Yizheng Jiao <jyizheng@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
2026-09-14 20:15:19 -07:00
Liqian Lin e45bb43984 [Bugfix][KV Offload] Ignore pending chunks in invalid sliding windows (#56486)
Signed-off-by: Liqian Lin <jacklin78911@gmail.com>
2026-09-15 05:48:38 +03:00
Simon Danielsson 1d0d1081c4 [ROCm][Connector] SWA+HMA-support in MoRI-IO connector (Gemma4) (#53721)
Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>
2026-09-14 20:52:56 +08:00
Nicolò Lucchesi 0ca7aef4e3 [Bugfix][NixlPush] Guard _remote_agents read in _do_send_reg_notif (X1) (#56317)
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
2026-09-14 18:37:12 +08:00
BruceLoveDecimalliuqihaomergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2c7ee87223 [Bugfix][Mooncake] Report failed remote KV loads to the scheduler (#50984)
Signed-off-by: liuqihao <liuqihao@myhexin.com>
Co-authored-by: liuqihao <liuqihao@myhexin.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-09-14 09:55:39 +00:00
58d45fd767 [Bugfix][KV Connector] MooncakeStore: exclude non-prefix-cacheable (QSA ring) groups; fix align-mode check (#55027)
Signed-off-by: Zhewen Li <zhewenli@inferact.ai>
Co-authored-by: Zhewen Li <zhewenli@inferact.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
2026-09-13 19:18:16 -07:00
Lucas WilkinsonandOpenAI Codex 7fe8fc803d [NIXL][PCP][DCP] Expose PCP producer KV shards as transfer ranks (#56645)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-09-13 20:23:15 +00:00
e19a3e172e [5/N] Share HiSparse host cache across TP ranks (reopens #52760) (#56629)
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Signed-off-by: Chao Lei <leichao139636@163.com>
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Chao Lei <leichao139636@163.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
2026-09-12 19:48:25 -04:00
29332cf936 [4/N] Expose HiSparse cache metrics via KV connector stats (#56061)
Signed-off-by: NickLucche <nicolo.lucchesi@mistral.ai>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
2026-09-12 10:47:55 -07:00