Itay Etelis and Itay Etelis
d63108fb18
[kv_offload] Skip decode-phase blocks in CPU offload ( #43797 )
...
Signed-off-by: Itay Etelis <itay.etelis@ibm.com >
Co-authored-by: Itay Etelis <itay.etelis@ibm.com >
2026-05-29 06:39:43 +03:00
a3ed5ab10c
[KV Offload] Add per-request offloading policy via on_new_request lifecycle hook ( #43205 )
...
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com >
Co-authored-by: Or Ozeri <or@ozery.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-28 20:45:18 +00:00
Simon Danielsson
552eb81918
[Bugfix][ROCm] Resolve MoRI connector hangs at high concurrency ( #40344 )
...
Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com >
2026-05-28 14:30:21 +00:00
Angelo Ruocco and Or Ozeri
864990e8d9
Add token-offset based selective offload in OffloadConnector ( #39983 )
...
Signed-off-by: Angelo Ruocco <ang@zurich.ibm.com >
Co-authored-by: Or Ozeri <or@ozery.com >
2026-05-28 14:11:02 +00:00
1fc2cee50a
[KVConnector][Mooncake] Wire reset_cache cascade end-to-end ( #42694 )
...
Signed-off-by: aoshen524 <aoshen524@gmail.com >
Signed-off-by: Ao Shen <aoshen@inferact.ai >
Co-authored-by: aoshen524 <aoshen524@gmail.com >
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-05-26 20:52:35 -07:00
7e33081cee
[Attention] Make FlexAttention and FlashAttention use num-blocks first layouts ( #42095 )
...
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com >
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com >
2026-05-26 19:55:56 -07:00
d98cbf472b
[KV Connector] MooncakeStore: drop dead discard_partial_chunks parameter ( #43627 )
...
Signed-off-by: Zhewen Li <zhewen@inferact.ai >
Co-authored-by: Zhewen Li <zhewen@inferact.ai >
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-05-26 13:40:21 -07:00
Simon Danielsson and TJian
681d7dd38b
[Misc][Refactor][ROCm] Convert MoRI-related envvars to extra config args ( #43303 )
...
Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com >
Co-authored-by: TJian <tunjian.tan@embeddedllm.com >
2026-05-26 03:33:35 -07:00
Ethan Feng
755043cf3c
[KV Transfer] Enable HMA by default for connectors that support it ( #41847 )
...
Signed-off-by: Ethan Feng <ethan.fengch@gmail.com >
2026-05-26 12:28:51 +02:00
Dao007forever
c2a4005c70
[KV Connector] Propagate MooncakeStore load failures ( #42788 )
...
Signed-off-by: Dao Le <Dao007forever@gmail.com >
2026-05-25 22:12:15 -07:00
Dao007forever and Claude
7966fc7233
[KV Connector][Bugfix] MooncakeStore: don't double-apply Eagle prune in load_mask ( #43516 )
...
Signed-off-by: Dao Le <daole@inferact.ai >
Signed-off-by: Dao Le <Dao007forever@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-05-25 22:11:57 -07:00
Zhewen Li and Zhewen Li
873758c13a
[KV Connector] Handle Mooncake finish after preemption ( #43281 )
...
Signed-off-by: Zhewen Li <zhewenli@inferact.ai >
Co-authored-by: Zhewen Li <zhewenli@inferact.ai >
2026-05-25 01:58:38 -07:00
Yifan Qiao
81252d4e24
[Feat][KVConnector] Support DSV4 in SimpleCPUOffloadBackend ( #42296 )
...
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
2026-05-25 14:04:30 +08:00
Or Ozeri
357fddf614
[kv_offload]: Add DSv4 support ( #43142 )
...
Signed-off-by: Or Ozeri <oro@il.ibm.com >
2026-05-24 11:10:12 +03:00
Dao007forever and Claude
0902d8e62f
[KV Connector] Keep MooncakeStore full hits block-aligned ( #43494 )
...
Signed-off-by: Dao Le <daole@inferact.ai >
Signed-off-by: Dao Le <Dao007forever@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-05-23 23:15:03 -07:00
Dao007forever
819c610f9b
[Mooncake] Add metrics for MooncakeStoreConnector operations ( #43392 )
2026-05-23 13:34:40 -07:00
Benjamin Chislett and gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
4e2eba28be
[Perf] Optimize hidden state extraction logic ( #37374 )
...
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com >
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com >
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-05-22 18:23:08 -04:00
Dao007forever and Claude Opus 4.7
b21f3d56d4
[KV Connector] MooncakeStore: don't co-queue save with load to avoid double delayed-free ( #43371 )
...
Signed-off-by: Dao Le <Dao007forever@gmail.com >
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-05-22 16:14:11 +00:00
Simon Danielsson
86ccef7d44
[ROCm] Add XGMI backend for MoRI Connector ( #41753 )
...
Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com >
2026-05-22 03:06:40 +00:00
Yifan Qiao
056bc2e166
[KVConnector][DSV4] HMA support for Mooncake store connector ( #42828 )
...
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
2026-05-19 01:07:46 -07:00
Revital Sur and gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
afd7b1dce9
[Bugfix] Use platform-agnostic device in example_connector load ( #42926 )
...
Signed-off-by: Revital Sur <eres@il.ibm.com >
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-05-19 03:12:04 +00:00
e5417657e5
[KV Connector][Offloading] Flush all pending jobs on last step ( #42611 )
...
Signed-off-by: Liran Schour <lirans@il.ibm.com >
Signed-off-by: liranschour <liranschour@users.noreply.github.com >
Co-authored-by: Or Ozeri <or@ozery.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-18 12:59:42 +00:00
Kfir Toledo
e414e1f1c0
[Bugfix][KV Offload] count appended GPU blocks in store group_sizes ( #42945 )
...
Signed-off-by: Kfir Toledo <kfir.toledo@ibm.com >
2026-05-18 11:36:02 +00:00
36e74c9ea4
[KV Connector] Support disk offloading in MooncakeStoreConnector ( #42689 )
...
Signed-off-by: Zhewen Li <zhewenli@inferact.ai >
Co-authored-by: Zhewen Li <zhewenli@inferact.ai >
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-05-16 13:34:15 -07:00
chunxiaozheng
f45c210885
[LMCacheMPConnector] Prioritize importing the lmcache_mp_connector from lmcache ( #42596 )
...
Signed-off-by: idellzheng <idellzheng@tencent.com >
2026-05-15 17:46:31 +00:00
zhanqiuhu and Nicolò Lucchesi
24337fb860
PD disagg with NIXL Connector: GDN support (Qwen3.5) ( #41869 )
...
Signed-off-by: Zhanqiu Hu <zhu@redhat.com >
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com >
2026-05-14 16:33:01 +02:00
5bd8c71e79
[kv_offload] Implement reset_cache() for the offloading connector ( #41956 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Or Ozeri <or@ozery.com >
2026-05-14 16:00:10 +03:00
Michael Goin and Claude Opus 4.6
2f821faeae
[Spec Decode] Support hybrid attention models in extract_hidden_states ( #39949 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com >
2026-05-13 10:45:53 -07:00
Ronen Schaffer
79fd1bc7ed
[kv_offload] Add req_id to ReqContext for per-request tracking ( #42507 )
...
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com >
2026-05-13 11:11:10 +00:00
Yifan Qiao and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
13bf242100
[Feat][KVConnector] Add bind_gpu_block_pool() to KVConnectorBase_V1 ( #39654 )
...
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-13 02:10:29 -07:00
Nicolò Lucchesi
71bcd02ef3
[Bugfix][PD] Fix multi-node TP (TP>8) ( #39907 )
...
Signed-off-by: NickLucche <nlucches@redhat.com >
2026-05-12 22:20:57 -07:00
+3
ebeb09d822
[KV Transfer] Add MooncakeStoreConnector for KV cache offloading via Mooncake distributed store ( #40900 )
...
Signed-off-by: leichao.lc <leichao.lc@antgroup.com >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: leichao.lc <leichao.lc@antgroup.com >
Co-authored-by: ivanium <yifanqiao@inferact.ai >
Co-authored-by: aoshen524 <aoshen@inferact.ai >
Co-authored-by: Dao007forever <daole@inferact.ai >
Co-authored-by: Teng Ma <sima.mt@alibaba-inc.com >
Co-authored-by: Pz1116 <zpbzpb123123@gmail.com >
Co-authored-by: foraxe <1055696449@qq.com >
Co-authored-by: Skywalker-EP <173423846@qq.com >
Co-authored-by: fems14 <1804143737@qq.com >
Co-authored-by: jianzs <zheng.shoujian@outlook.com >
Co-authored-by: baxingpiaochong <771405853@qq.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-12 16:09:10 -07:00
Martin Hickey
418ba8ef14
[kv_offload][BugFix] Fix store deferral ( #41945 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
2026-05-12 18:04:44 +03:00
zhanqiuhu and NickLucche
593d5a4033
[Bugfix] Fix mismatched kernel-per-logical blocks in NIXL HMA transfer ( #42097 )
...
Signed-off-by: ZhanqiuHu <zhu@redhat.com >
Signed-off-by: Zhanqiu Hu <zhu@redhat.com >
Signed-off-by: NickLucche <nlucches@redhat.com >
Co-authored-by: NickLucche <nlucches@redhat.com >
2026-05-12 15:53:30 +02:00
Martin Hickey
8415bf2cdb
[kv_offload] Set offloading connector to prefer HND layout ( #41928 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
2026-05-11 15:05:41 +03:00
Nicolò Lucchesi
5672d100ed
[KV Connector][NIXL][Bugfix] Fix NIXL handshake failures not honoring kv_load_failure_policy ( #40364 )
...
When NIXL handshake fails (e.g., due to compatibility hash mismatch
between prefill and decode instances), requests fail with "engine dead"
error instead of gracefully falling back to local recomputation as configured
by kv_load_failure_policy='recompute'.
Signed-off-by: NickLucche <nlucches@redhat.com >
2026-05-11 09:37:21 +00:00
Nicolò Lucchesi
770e9bd6b3
[Nixl][PD] Lease renewal TTL KV blocks on P ( #41383 )
...
Signed-off-by: NickLucche <nlucches@redhat.com >
2026-05-11 09:27:30 +00:00
shaharmor98 and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
f9f770ca0b
fix nixl side-channel host selection ( #41806 )
...
Signed-off-by: Shahar Mor <smor@nvidia.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-11 07:40:37 +00:00
wangxiyuan
5536fc0c01
[Misc] Replace mamba_type string literals with MambaAttentionBackendEnum ( #41188 )
...
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com >
2026-05-11 03:59:36 +00:00
Ronen Schaffer
e175192d33
[KV Offload] Pass ReqContext to touch(), complete_load(), and complete_store() ( #41366 )
...
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com >
2026-05-10 15:09:25 +03:00
Andreas Karatzas
f2840120f6
[ROCm][CI] Fix NIXL spec-decode acceptance startup and diagnostics ( #41313 )
...
Signed-off-by: Andreas Karatzas <akaratza@amd.com >
2026-05-10 14:50:16 +08:00
Dao007forever
3f5bd482f5
[Bugfix][KV Transfer][NIXL] Notify P node on pre-admission rejection to free stranded KV blocks ( #41269 )
2026-05-09 22:52:09 -07:00
Wentao Ye
ea0e501bb1
[KV Connector] Remove compat support for pre-v0.12.0 constructor signatures without KVCacheConfig ( #39832 )
...
The v0.12.0 release contained initial support for HMA in KV Connectors. As part
of these changes, a KVCacheConfig argument was added to KV connector
constructors. Backwards compatibility support for out-of-tree connectors was
included in this change, with a very prominent warning. See #25712 and #27887 .
Since the warning has been around for over 5 months, we can safely remove
the support of it.
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-05-09 23:39:46 +00:00
Wentao Ye
f80aa53c9d
[Refactor] Nixl util using lazy init ( #41392 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-05-09 17:46:52 -04:00
Nicolò Lucchesi and ZhanqiuHu
171d59ae8d
[Bugfix][PD] Fix DSv4 Disaggregated ( #41957 )
...
Signed-off-by: NickLucche <nlucches@redhat.com >
Co-authored-by: ZhanqiuHu <zhu@redhat.com >
2026-05-09 16:48:24 +00:00
Zijing Liu and Claude Opus 4.7
09a7cc5ba9
[KV Connector] Opt DecodeBenchConnector into SupportsHMA ( #41770 )
...
Signed-off-by: Zijing Liu <liuzijing2014@gmail.com >
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com >
2026-05-07 16:10:00 -07:00
zhanqiuhu and NickLucche
df8e63f4ed
nixl refactor: new transfer design ( #40731 )
...
Signed-off-by: ZhanqiuHu <zhu@redhat.com >
Signed-off-by: NickLucche <nlucches@redhat.com >
Co-authored-by: NickLucche <nlucches@redhat.com >
2026-05-06 06:16:25 -07:00
wang.yuqi and gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
51c1ee9b7c
[Examples] Resettle Disaggregated examples. ( #40759 )
...
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io >
Signed-off-by: wang.yuqi <noooop@126.com >
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-05-06 01:20:38 -07:00
Zhewen Li and Zhewen Li
bee126165f
[P/D][Mooncake] Add KVConnectorStats for transfer observability ( #40414 )
...
Signed-off-by: Zhewen Li <zhewenli@inferact.ai >
Co-authored-by: Zhewen Li <zhewenli@inferact.ai >
2026-05-05 02:17:38 -07:00
Or Ozeri
2fa1f8ec00
[kv_offload+HMA][13/N]: Enable HMA support ( #41445 )
...
This is the final PR in a series to enables HMA support for the
offloading connector. The connector advertises `SupportsHMA`
and is validated with unit tests and e2e tests.
Signed-off-by: Or Ozeri <oro@il.ibm.com >
2026-05-01 12:30:03 +01:00