|
9dfc313bdc
|
Feature/offloading manager stats (#35669)
Signed-off-by: Sriusa4414@gmail.com
Signed-off-by: srinivas_oo7 <Sriusa4414@gmail.com>
Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Signed-off-by: Srinivasoo7 <158864704+Srinivasoo7@users.noreply.github.com>
Signed-off-by: Or Ozeri <oro@il.ibm.com>
Co-authored-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Co-authored-by: Srinivasoo7 <158864704+Srinivasoo7@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
|
2026-06-10 12:44:55 +00:00 |
|
|
Dao007foreverandClaude Opus 4.7
|
c1d754d681
|
[Mooncake] Use all HCAs on multi-NIC hosts instead of GPU-indexed RNIC selection (#43799)
Signed-off-by: Dao Le <Dao007forever@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
2026-06-09 11:05:36 -07:00 |
|
|
Nicolò Lucchesi
|
6690a0c4de
|
[PD][Bugfix] Fix KV Cache sharing with HMA (#44629)
Signed-off-by: NickLucche <nlucches@redhat.com>
|
2026-06-09 06:10:06 -07:00 |
|
|
Nicolò Lucchesi
|
5add018beb
|
[Connector] Remove P2pNcclConnector (#44854)
Signed-off-by: NickLucche <nlucches@redhat.com>
|
2026-06-08 18:58:29 +08:00 |
|
|
Yifan Qiao
|
51ef688831
|
[Bugfix][Mooncake] Fix per-group block_size/block_hash and group_idx in MooncakeStoreConnector KV events (#44103)
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
|
2026-06-07 07:12:43 +00:00 |
|
|
Harry Mellor
|
a80af24356
|
Speed up docs build (#44635)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
|
2026-06-05 14:51:44 +00:00 |
|
|
Nicolò Lucchesi
|
d98b8f371c
|
[NixlConnector] Initiate deprecation cycle for kv_both role (#43874)
Signed-off-by: NickLucche <nlucches@redhat.com>
|
2026-06-05 11:08:17 +02:00 |
|
|
qizixi
|
96229fa99e
|
[KVConnector][1/N] PP-aware handshake aggregation and intermediate-PP output plumbing (#43720)
Signed-off-by: zixi-qi <zixi@inferact.ai>
|
2026-06-04 22:04:19 -07:00 |
|
|
Nicolò Lucchesi
|
68f5e565c9
|
[PD][Nixl] Mamba prefix caching mode support (#42554)
Signed-off-by: NickLucche <nlucches@redhat.com>
|
2026-06-04 06:41:46 -07:00 |
|
|
Wentao Yeandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
e6018c644a
|
[Refactor] Remove dead code in tests and parallel_state (#41471)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
2026-06-03 19:32:39 -07:00 |
|
|
|
0c6631f02a
|
[KVCache] Support Pluggable KVCacheSpec (#37505)
Signed-off-by: MengqingCao <cmq0113@163.com>
Signed-off-by: Mengqing Cao <cmq0113@163.com>
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
2026-06-03 09:05:16 -07:00 |
|
|
Yifan Qiao
|
e9e08c49b9
|
[Bugfix] Cache the EAGLE/MTP lookahead block in the SWA prefix-cache mask (#44082)
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
|
2026-06-02 12:21:07 -07:00 |
|
|
Ronen Schaffer
|
2a2b5ca791
|
[KV Offload] Add on_schedule_end() hook to separate step lifecycle from event draining (#44206)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
|
2026-06-02 13:42:52 +03:00 |
|
|
Or Ozeri
|
480fadab1b
|
[BugFix][kv_offload]: Prevent offloading stale sliding window blocks (#42959)
Signed-off-by: Or Ozeri <oro@il.ibm.com>
|
2026-06-02 05:59:48 +03:00 |
|
|
Dao007foreverandClaude
|
d68f0b220e
|
[Bugfix][Mooncake] Release GPU pin on failed store in MooncakeStoreConnector (#43742)
Signed-off-by: Dao Le <Dao007forever@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
|
2026-06-01 18:29:18 -07:00 |
|
|
Itay EtelisandItay Etelis
|
d63108fb18
|
[kv_offload] Skip decode-phase blocks in CPU offload (#43797)
Signed-off-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: Itay Etelis <itay.etelis@ibm.com>
|
2026-05-29 06:39:43 +03:00 |
|
|
|
a3ed5ab10c
|
[KV Offload] Add per-request offloading policy via on_new_request lifecycle hook (#43205)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
Co-authored-by: Or Ozeri <or@ozery.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
2026-05-28 20:45:18 +00:00 |
|
|
Simon Danielsson
|
552eb81918
|
[Bugfix][ROCm] Resolve MoRI connector hangs at high concurrency (#40344)
Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>
|
2026-05-28 14:30:21 +00:00 |
|
|
Angelo RuoccoandOr Ozeri
|
864990e8d9
|
Add token-offset based selective offload in OffloadConnector (#39983)
Signed-off-by: Angelo Ruocco <ang@zurich.ibm.com>
Co-authored-by: Or Ozeri <or@ozery.com>
|
2026-05-28 14:11:02 +00:00 |
|
|
|
1fc2cee50a
|
[KVConnector][Mooncake] Wire reset_cache cascade end-to-end (#42694)
Signed-off-by: aoshen524 <aoshen524@gmail.com>
Signed-off-by: Ao Shen <aoshen@inferact.ai>
Co-authored-by: aoshen524 <aoshen524@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
2026-05-26 20:52:35 -07:00 |
|
|
|
7e33081cee
|
[Attention] Make FlexAttention and FlashAttention use num-blocks first layouts (#42095)
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
|
2026-05-26 19:55:56 -07:00 |
|
|
|
d98cbf472b
|
[KV Connector] MooncakeStore: drop dead discard_partial_chunks parameter (#43627)
Signed-off-by: Zhewen Li <zhewen@inferact.ai>
Co-authored-by: Zhewen Li <zhewen@inferact.ai>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
2026-05-26 13:40:21 -07:00 |
|
|
Simon DanielssonandTJian
|
681d7dd38b
|
[Misc][Refactor][ROCm] Convert MoRI-related envvars to extra config args (#43303)
Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>
Co-authored-by: TJian <tunjian.tan@embeddedllm.com>
|
2026-05-26 03:33:35 -07:00 |
|
|
Ethan Feng
|
755043cf3c
|
[KV Transfer] Enable HMA by default for connectors that support it (#41847)
Signed-off-by: Ethan Feng <ethan.fengch@gmail.com>
|
2026-05-26 12:28:51 +02:00 |
|
|
Dao007forever
|
c2a4005c70
|
[KV Connector] Propagate MooncakeStore load failures (#42788)
Signed-off-by: Dao Le <Dao007forever@gmail.com>
|
2026-05-25 22:12:15 -07:00 |
|
|
Dao007foreverandClaude
|
7966fc7233
|
[KV Connector][Bugfix] MooncakeStore: don't double-apply Eagle prune in load_mask (#43516)
Signed-off-by: Dao Le <daole@inferact.ai>
Signed-off-by: Dao Le <Dao007forever@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
|
2026-05-25 22:11:57 -07:00 |
|
|
Zhewen LiandZhewen Li
|
873758c13a
|
[KV Connector] Handle Mooncake finish after preemption (#43281)
Signed-off-by: Zhewen Li <zhewenli@inferact.ai>
Co-authored-by: Zhewen Li <zhewenli@inferact.ai>
|
2026-05-25 01:58:38 -07:00 |
|
|
Yifan Qiao
|
81252d4e24
|
[Feat][KVConnector] Support DSV4 in SimpleCPUOffloadBackend (#42296)
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
|
2026-05-25 14:04:30 +08:00 |
|
|
Or Ozeri
|
357fddf614
|
[kv_offload]: Add DSv4 support (#43142)
Signed-off-by: Or Ozeri <oro@il.ibm.com>
|
2026-05-24 11:10:12 +03:00 |
|
|
Dao007foreverandClaude
|
0902d8e62f
|
[KV Connector] Keep MooncakeStore full hits block-aligned (#43494)
Signed-off-by: Dao Le <daole@inferact.ai>
Signed-off-by: Dao Le <Dao007forever@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
|
2026-05-23 23:15:03 -07:00 |
|
|
Dao007forever
|
819c610f9b
|
[Mooncake] Add metrics for MooncakeStoreConnector operations (#43392)
|
2026-05-23 13:34:40 -07:00 |
|
|
Benjamin Chislettandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
|
4e2eba28be
|
[Perf] Optimize hidden state extraction logic (#37374)
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com>
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
|
2026-05-22 18:23:08 -04:00 |
|
|
Dao007foreverandClaude Opus 4.7
|
b21f3d56d4
|
[KV Connector] MooncakeStore: don't co-queue save with load to avoid double delayed-free (#43371)
Signed-off-by: Dao Le <Dao007forever@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
2026-05-22 16:14:11 +00:00 |
|
|
Simon Danielsson
|
86ccef7d44
|
[ROCm] Add XGMI backend for MoRI Connector (#41753)
Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>
|
2026-05-22 03:06:40 +00:00 |
|
|
Yifan Qiao
|
056bc2e166
|
[KVConnector][DSV4] HMA support for Mooncake store connector (#42828)
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
|
2026-05-19 01:07:46 -07:00 |
|
|
Revital Surandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
|
afd7b1dce9
|
[Bugfix] Use platform-agnostic device in example_connector load (#42926)
Signed-off-by: Revital Sur <eres@il.ibm.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
|
2026-05-19 03:12:04 +00:00 |
|
|
|
e5417657e5
|
[KV Connector][Offloading] Flush all pending jobs on last step (#42611)
Signed-off-by: Liran Schour <lirans@il.ibm.com>
Signed-off-by: liranschour <liranschour@users.noreply.github.com>
Co-authored-by: Or Ozeri <or@ozery.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
2026-05-18 12:59:42 +00:00 |
|
|
Kfir Toledo
|
e414e1f1c0
|
[Bugfix][KV Offload] count appended GPU blocks in store group_sizes (#42945)
Signed-off-by: Kfir Toledo <kfir.toledo@ibm.com>
|
2026-05-18 11:36:02 +00:00 |
|
|
|
36e74c9ea4
|
[KV Connector] Support disk offloading in MooncakeStoreConnector (#42689)
Signed-off-by: Zhewen Li <zhewenli@inferact.ai>
Co-authored-by: Zhewen Li <zhewenli@inferact.ai>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
2026-05-16 13:34:15 -07:00 |
|
|
chunxiaozheng
|
f45c210885
|
[LMCacheMPConnector] Prioritize importing the lmcache_mp_connector from lmcache (#42596)
Signed-off-by: idellzheng <idellzheng@tencent.com>
|
2026-05-15 17:46:31 +00:00 |
|
|
zhanqiuhuandNicolò Lucchesi
|
24337fb860
|
PD disagg with NIXL Connector: GDN support (Qwen3.5) (#41869)
Signed-off-by: Zhanqiu Hu <zhu@redhat.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
|
2026-05-14 16:33:01 +02:00 |
|
|
|
5bd8c71e79
|
[kv_offload] Implement reset_cache() for the offloading connector (#41956)
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Or Ozeri <or@ozery.com>
|
2026-05-14 16:00:10 +03:00 |
|
|
Michael GoinandClaude Opus 4.6
|
2f821faeae
|
[Spec Decode] Support hybrid attention models in extract_hidden_states (#39949)
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
|
2026-05-13 10:45:53 -07:00 |
|
|
Ronen Schaffer
|
79fd1bc7ed
|
[kv_offload] Add req_id to ReqContext for per-request tracking (#42507)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
|
2026-05-13 11:11:10 +00:00 |
|
|
Yifan Qiaoandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
13bf242100
|
[Feat][KVConnector] Add bind_gpu_block_pool() to KVConnectorBase_V1 (#39654)
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
2026-05-13 02:10:29 -07:00 |
|
|
Nicolò Lucchesi
|
71bcd02ef3
|
[Bugfix][PD] Fix multi-node TP (TP>8) (#39907)
Signed-off-by: NickLucche <nlucches@redhat.com>
|
2026-05-12 22:20:57 -07:00 |
|
|
+3
|
ebeb09d822
|
[KV Transfer] Add MooncakeStoreConnector for KV cache offloading via Mooncake distributed store (#40900)
Signed-off-by: leichao.lc <leichao.lc@antgroup.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: leichao.lc <leichao.lc@antgroup.com>
Co-authored-by: ivanium <yifanqiao@inferact.ai>
Co-authored-by: aoshen524 <aoshen@inferact.ai>
Co-authored-by: Dao007forever <daole@inferact.ai>
Co-authored-by: Teng Ma <sima.mt@alibaba-inc.com>
Co-authored-by: Pz1116 <zpbzpb123123@gmail.com>
Co-authored-by: foraxe <1055696449@qq.com>
Co-authored-by: Skywalker-EP <173423846@qq.com>
Co-authored-by: fems14 <1804143737@qq.com>
Co-authored-by: jianzs <zheng.shoujian@outlook.com>
Co-authored-by: baxingpiaochong <771405853@qq.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
|
2026-05-12 16:09:10 -07:00 |
|
|
Martin Hickey
|
418ba8ef14
|
[kv_offload][BugFix] Fix store deferral (#41945)
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
|
2026-05-12 18:04:44 +03:00 |
|
|
zhanqiuhuandNickLucche
|
593d5a4033
|
[Bugfix] Fix mismatched kernel-per-logical blocks in NIXL HMA transfer (#42097)
Signed-off-by: ZhanqiuHu <zhu@redhat.com>
Signed-off-by: Zhanqiu Hu <zhu@redhat.com>
Signed-off-by: NickLucche <nlucches@redhat.com>
Co-authored-by: NickLucche <nlucches@redhat.com>
|
2026-05-12 15:53:30 +02:00 |
|
|
Martin Hickey
|
8415bf2cdb
|
[kv_offload] Set offloading connector to prefer HND layout (#41928)
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
|
2026-05-11 15:05:41 +03:00 |
|