Commit Graph
326 Commits
Author SHA1 Message Date
Cao Qian 8b1f3bebca [LMCache MP Connector] Add num_lmcache_extra_cached_token in KVTransferParams (#39843)
Signed-off-by: aeon-x <talexcao@gmail.com>
2026-04-20 20:42:49 +00:00
zhanqiuhu cc3993b05d nixl refactor [2/N]: unify TpKVTopology + HeteroTPTransferConfig into TransferTopology (#39529)
Signed-off-by: Zhanqiu Hu <zhu@redhat.com>
2026-04-20 12:39:08 +02:00
Ilya Markov 50dd4cb427 [EPLB] Add nixl-based eplb communicator (#36276)
Signed-off-by: ilmarkov <markovilya197@gmail.com>
Signed-off-by: Markov Ilya <markovilya19@gmail.com>
2026-04-20 10:24:23 +00:00
Or OzeriandNicolò Lucchesi f774ba028a [kv_offload+HMA][4/N]: Support sliding window lookup (#36645)
Signed-off-by: Or Ozeri <oro@il.ibm.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
2026-04-20 12:53:51 +03:00
omerpaz95 4353c9cb4a [KV Offload] Pass request context (#39185)
Signed-off-by: omerpaz95 <omerpaz95@gmail.com>
2026-04-19 08:54:59 +03:00
omerpaz95andOr Ozeri 4b7f5ea1a0 [KV Connector] Allow metrics of multiple connectors of same types in multi connector. (#40010)
Signed-off-by: omerpaz95 <omerpaz95@gmail.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-04-19 07:49:10 +03:00
zhanqiuhu 41488f2acd [Bugfix][NIXL] Fix _logical_to_kernel_block_ids conversion for non-mamba models (#39724)
Signed-off-by: Zhanqiu Hu <zhu@redhat.com>
2026-04-15 20:08:58 +00:00
Zhewen LiZhewen LiOpenAI Codexmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
55e1a8e103 [Mooncake] Fix mixed MLA+Eagle block-size validation (#39596)
Signed-off-by: Zhewen Li <zhewenli@inferact.ai>
Co-authored-by: Zhewen Li <zhewenli@inferact.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-04-15 11:36:47 -07:00
Roy Huang ed33310552 [KVConnector][LMCache] Propagate cache_salt through MP connector for per-user cache isolation (#39837)
Signed-off-by: royyhuang <royyhuang@gmail.com>
Signed-off-by: royyhuang <roy.y.huang@gmail.com>
2026-04-15 09:10:49 -07:00
Or Ozeri 235e1f930a [kv_offload+HMA][3/N]: Remove block_size from KVEvents (#36644)
Signed-off-by: Or Ozeri <oro@il.ibm.com>
2026-04-15 11:53:19 +03:00
Zhewen LiandZhewen Li 1696c864b9 [Bugfix][Mooncake] Fix thread-local CUDA context for NVLink transfers in _send_blocks (#39548)
Signed-off-by: Zhewen Li <zhewenli@inferact.ai>
Co-authored-by: Zhewen Li <zhewenli@inferact.ai>
2026-04-14 14:13:58 -07:00
maobaolong b2f749dc97 fix(lmcache): correct store for cached requests while enable prefix cache (#39719)
Signed-off-by: baoloongmao <baoloongmao@tencent.com>
2026-04-14 20:51:27 +00:00
chunxiaozhengandYihua Cheng c687bf226a [LMCache][MP] optimize save when mla enabled (#38810)
Signed-off-by: idellzheng <idellzheng@tencent.com>
Co-authored-by: Yihua Cheng <yihua98@uchicago.edu>
2026-04-13 17:56:43 -07:00
maobaolong 2a3c32ce67 fix(lmcache): correct store for cached requests and num_scheduled_tokens in lmcache_mp_connector.py (#39655)
Signed-off-by: baoloongmao <baoloongmao@tencent.com>
2026-04-13 03:29:19 +00:00
Nicolò Lucchesi 3244a2ebf2 [KVConnector][NIXL] Organize NIXL connector into its own directory (#39354)
The number of features supported by the connector has grown substantially
and the `nixl_connector.py` file has accumulated a lot of code. Creates a separate
directory and isolates connector/scheduler code in the hope of improving clarity
and maintainability.

Further refactor of components aimed at improving clarity and simplifying code
will follow soon.

Signed-off-by: NickLucche <nlucches@redhat.com>
2026-04-12 13:10:50 +00:00
Ronen Schaffer a5b17fba8f [KV Offload] Implement shutdown() in OffloadingConnector and related classes (#39182)
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
2026-04-10 08:06:22 +03:00
Yuwei Anandyuwei 467d3247c3 [LMCache] vLLM Block Allocation Event (#38856)
Signed-off-by: yuwei <yuwei@dev.local>
Co-authored-by: yuwei <yuwei@dev.local>
2026-04-09 17:30:29 -07:00
Chendi.Xue ef5a226819 [PD][HeteroArch]Fix accuracy issue with CPU_ATTN as Decoder and Flash_ATTN as prefiller (#38935)
Signed-off-by: Chendi Xue <chendi.xue@intel.com>
2026-04-09 11:19:07 +08:00
Or Ozeri 512c5eb455 [kv_offload+HMA][5/N]: Track group block hashes and block IDs (#37109)
Signed-off-by: Or Ozeri <oro@il.ibm.com>
2026-04-08 19:50:28 +03:00
ibifrostandSimon Mo 96b5004b71 [KVConnector] Support 3FS KVConnector (#37636)
Signed-off-by: wuchenxin <wuchenxin.wcx@alibaba-inc.com>
Signed-off-by: ibifrost <47308427+ibifrost@users.noreply.github.com>
Co-authored-by: Simon Mo <simon.mo@hey.com>
2026-04-07 15:46:00 +00:00
maobaolongandChauncey 729eb59f60 [KVConnector]: prioritize external connector over internal registry (#38301)
Signed-off-by: baoloongmao <baoloongmao@tencent.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
2026-04-07 15:03:11 +00:00
zhanqiuhu bfdc0a3a99 [NIXL][Mamba][3/N] Heterogeneous TP: 3-read conv state transfer (#37635) 2026-04-06 19:07:02 +02:00
yzong-rhandNicolò Lucchesi dc0428ebb8 [NIXL][BUG] Fix Triton heterogeneous TP (#37940)
Signed-off-by: Yifan <yzong@redhat.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
2026-04-01 17:23:15 +02:00
yjzandNicolò Lucchesi 992368522f [KVTransfer] Fix TpKVTopology.is_kv_replicated equality case (#38179)
Signed-off-by: JianDan0212 <zhangyj0212@gmail.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
2026-04-01 12:41:49 +02:00
Yifan Qiao 91e4521f9f [Feat][v1] Simple yet General CPU KV Cache Offloading (#37160)
Signed-off-by: Yifan Qiao <yifanqiao@berkeley.edu>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
2026-03-31 17:58:37 -07:00
Kfir ToledoandOr Ozeri 6cc7abdc66 [kv_offload+HMA] Fix num_blocks with different per-layer page sizes and improve assert message (#38554)
Signed-off-by: Kfir Toledo <kfir.toledo@ibm.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-03-31 06:00:40 +00:00
dtc 6287e7fa20 [P/D] Mooncake: Add unit tests and minor fixes for mooncake connector (#36946)
Signed-off-by: Tianchen Ding <dtcccc@linux.alibaba.com>
2026-03-27 09:26:40 +01:00
Or Ozeri 7cc302dd87 [kv_offload+HMA][7/N]: Support register_kv_caches for hybrid models (#37853)
Signed-off-by: Or Ozeri <oro@il.ibm.com>
2026-03-27 08:38:33 +03:00
yjz b745e8b5d3 [KVTransfer][Mooncake] Add heterogeneous TP support for disaggregated P/D in MooncakeConnector (#36869)
Signed-off-by: JianDan0212 <zhangyj0212@gmail.com>
2026-03-25 14:24:07 +01:00
Wentao Ye c59a132f96 [V0 Deprecation] Refactor kv cache from list to element (#37487)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
2026-03-23 20:10:11 -07:00
Wentao Ye b3d0b37908 [Refactor] Remove unused dead code (#36171)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
2026-03-20 16:12:51 -06:00
Martin Hickey 880be2b1b8 [Metrics] Some small refactoring for better maintainability (#33898)
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
2026-03-20 16:11:34 +00:00
zhanqiuhu d49f273144 [SSM/Mamba] Follow-up: N-1 prefill for P/D disaggregation (#37310) 2026-03-19 08:22:00 +01:00
Wentao Ye 0d81a1fe61 [V0 Deprecation] Deprecate virtual engine (#37195)
Signed-off-by: yewentao256 <zhyanwentao@126.com>
2026-03-18 14:30:14 -07:00
Or Ozeri 5dd8df0701 [kv_offload+HMA][2/N]: Support multiple KV groups in GPULoadStoreSpec (#36642)
Signed-off-by: Or Ozeri <oro@il.ibm.com>
2026-03-18 19:26:40 +02:00
Or Ozeri 525f2eeb0b [kv_offload+HMA][6/N]: Split offloading_connector.py (#37405)
Signed-off-by: Or Ozeri <oro@il.ibm.com>
2026-03-18 14:42:46 +01:00
Andy Lo 98b09ddc27 [NIXL][Bugfix] metrics & testing minor bug (#36051)
Signed-off-by: Andy Lo <andy@mistral.ai>
2026-03-18 14:39:14 +01:00
Or OzeriandNicolò Lucchesi fcf0687b27 [kv_offload+HMA][0/N]: Support block-level preemption handling (#34805)
Signed-off-by: Or Ozeri <oro@il.ibm.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
2026-03-18 08:49:53 +02:00
youkaichao 0fb142a454 [perf][connector] optimize build_connector_meta when host buffer transfer is not used (#37165)
Signed-off-by: youkaichao <youkaichao@gmail.com>
2026-03-17 11:59:35 +00:00
zhanqiuhu 2dccb38f73 [Bugfix][MultiConnector] Fix MultiConnector for SupportsHMA sub-connectors (#36549) 2026-03-16 20:51:04 +00:00
Nicolò Lucchesi f5c081d432 [PD][Nixl] Add support for hybrid SSM-FA models (#36687) 2026-03-16 19:58:06 +01:00
haosdentandOr Ozeri ca1954d58c [Bugfix] Disable cross-layer KV cache for MLA attention backends (#37090)
Signed-off-by: haosdent <haosdent@gmail.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
2026-03-16 19:03:10 +02:00
Fynn Schmitt-Ulms 04bf5a35fa [Spec Decode] Update extract_hidden_states to use deferred kv_connector clear (#37013) 2026-03-16 14:53:45 +01:00
rasmith 0024f39a32 [ROCm][P/D][MORI][BugFix] Add transfer_id for moriio_connector so moriio_connector to restore P/D functionality (#34907)
Signed-off-by: Randall Smith <Randall.Smith@amd.com>
2026-03-16 10:36:51 +08:00
Harry Mellor 5a3f1eb62f [Misc] Set default kv_buffer_device in a better way (#36862)
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
2026-03-13 19:07:33 +00:00
Or Ozeri cfaf4668f7 [kv_offload+HMA][1/N]: Support multiple KV groups in OffloadingSpec (#36610)
Signed-off-by: Or Ozeri <oro@il.ibm.com>
2026-03-13 08:04:21 +00:00
Kunshang Ji 53ec16a705 [Hardware] Replace torch.cuda.device_count/current_device/set_device API (#36145)
Signed-off-by: Kunshang Ji <jikunshang95@gmail.com>
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
2026-03-12 07:57:47 -07:00
sfeiqiangandphaedonsun 8cb24d3aed [KV Connector] Support using FlexKV as KV Cache Offloading option. (#34328)
Signed-off-by: phaedonsun <phaedonsun@tencent.com>
Co-authored-by: phaedonsun <phaedonsun@tencent.com>
2026-03-12 00:46:20 -07:00
Yan Ma 894843eb25 replace with torch.cuda.device with with torch.accelerator.device_index (#36144)
Signed-off-by: Yan Ma <yan.ma@intel.com>
2026-03-11 23:12:57 -07:00
Yuwei An 2ef69456f5 [LMCache] Fault Tolerance Mechanism (#36586)
Signed-off-by: Oasis-Git <ayw.sirius19@gmail.com>
2026-03-12 03:54:39 +00:00