iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
16,085 Commits 3 Branches 0 Tags
333529deae59cd4100df540f225470c9bc539bee
Commit Graph
331 Commits
Author SHA1 Message Date
Or Ozeri 51adca74e6 [kv_offload+HMA][9/N]: Support lookup with multiple KV groups (#39401) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-04-24 15:32:29 +03:00
Or Ozeri 5ef33ab250 [kv_offload+HMA][10/N]: Support load with multiple KV groups (#39402) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-04-23 20:00:45 +03:00
Simon DanielssonandMatvei Pashkovskii ac58e2a170 [Fix][MoRI] Align MoRI-IO message format with P2pNcclConnector and vllm-router (#39565) 
Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>
Co-authored-by: Matvei Pashkovskii <mpashkov@amd.com>
 2026-04-23 08:06:31 +09:00
Or Ozeriandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> aad88f8486 [kv_offload+HMA][8/N]: Support multi-group worker transfer (#38453) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-04-22 08:44:00 +03:00
rasmith cefa5281a7 [ROCm][P/D][MORI][BugFix] Ensure correct api is used when making requests to prefill / decode nodes (#39835) 
Signed-off-by: Randall Smith <Randall.Smith@amd.com>
 2026-04-22 09:48:25 +09:00
Cao Qian 8b1f3bebca [LMCache MP Connector] Add num_lmcache_extra_cached_token in KVTransferParams (#39843) 
Signed-off-by: aeon-x <talexcao@gmail.com>
 2026-04-20 20:42:49 +00:00
zhanqiuhu cc3993b05d nixl refactor [2/N]: unify TpKVTopology + HeteroTPTransferConfig into TransferTopology (#39529) 
Signed-off-by: Zhanqiu Hu <zhu@redhat.com>
 2026-04-20 12:39:08 +02:00
Ilya Markov 50dd4cb427 [EPLB] Add nixl-based eplb communicator (#36276) 
Signed-off-by: ilmarkov <markovilya197@gmail.com>
Signed-off-by: Markov Ilya <markovilya19@gmail.com>
 2026-04-20 10:24:23 +00:00
Or OzeriandNicolò Lucchesi f774ba028a [kv_offload+HMA][4/N]: Support sliding window lookup (#36645) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
 2026-04-20 12:53:51 +03:00
omerpaz95 4353c9cb4a [KV Offload] Pass request context (#39185) 
Signed-off-by: omerpaz95 <omerpaz95@gmail.com>
 2026-04-19 08:54:59 +03:00
omerpaz95andOr Ozeri 4b7f5ea1a0 [KV Connector] Allow metrics of multiple connectors of same types in multi connector. (#40010) 
Signed-off-by: omerpaz95 <omerpaz95@gmail.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-04-19 07:49:10 +03:00
zhanqiuhu 41488f2acd [Bugfix][NIXL] Fix _logical_to_kernel_block_ids conversion for non-mamba models (#39724) 
Signed-off-by: Zhanqiu Hu <zhu@redhat.com>
 2026-04-15 20:08:58 +00:00
Zhewen LiZhewen LiOpenAI Codexmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		55e1a8e103 [Mooncake] Fix mixed MLA+Eagle block-size validation (#39596) 
Signed-off-by: Zhewen Li <zhewenli@inferact.ai>
Co-authored-by: Zhewen Li <zhewenli@inferact.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-04-15 11:36:47 -07:00
Roy Huang ed33310552 [KVConnector][LMCache] Propagate cache_salt through MP connector for per-user cache isolation (#39837) 
Signed-off-by: royyhuang <royyhuang@gmail.com>
Signed-off-by: royyhuang <roy.y.huang@gmail.com>
 2026-04-15 09:10:49 -07:00
Or Ozeri 235e1f930a [kv_offload+HMA][3/N]: Remove block_size from KVEvents (#36644) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-04-15 11:53:19 +03:00
Zhewen LiandZhewen Li 1696c864b9 [Bugfix][Mooncake] Fix thread-local CUDA context for NVLink transfers in _send_blocks (#39548) 
Signed-off-by: Zhewen Li <zhewenli@inferact.ai>
Co-authored-by: Zhewen Li <zhewenli@inferact.ai>
 2026-04-14 14:13:58 -07:00
maobaolong b2f749dc97 fix(lmcache): correct store for cached requests while enable prefix cache (#39719) 
Signed-off-by: baoloongmao <baoloongmao@tencent.com>
 2026-04-14 20:51:27 +00:00
chunxiaozhengandYihua Cheng c687bf226a [LMCache][MP] optimize save when mla enabled (#38810) 
Signed-off-by: idellzheng <idellzheng@tencent.com>
Co-authored-by: Yihua Cheng <yihua98@uchicago.edu>
 2026-04-13 17:56:43 -07:00
maobaolong 2a3c32ce67 fix(lmcache): correct store for cached requests and num_scheduled_tokens in lmcache_mp_connector.py (#39655) 
Signed-off-by: baoloongmao <baoloongmao@tencent.com>
 2026-04-13 03:29:19 +00:00
Nicolò Lucchesi 3244a2ebf2 [KVConnector][NIXL] Organize NIXL connector into its own directory (#39354) 
The number of features supported by the connector has grown substantially
and the `nixl_connector.py` file has accumulated a lot of code. Creates a separate
directory and isolates connector/scheduler code in the hope of improving clarity
and maintainability.

Further refactor of components aimed at improving clarity and simplifying code
will follow soon.

Signed-off-by: NickLucche <nlucches@redhat.com>
 2026-04-12 13:10:50 +00:00
Ronen Schaffer a5b17fba8f [KV Offload] Implement shutdown() in OffloadingConnector and related classes (#39182) 
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
 2026-04-10 08:06:22 +03:00
Yuwei Anandyuwei 467d3247c3 [LMCache] vLLM Block Allocation Event (#38856) 
Signed-off-by: yuwei <yuwei@dev.local>
Co-authored-by: yuwei <yuwei@dev.local>
 2026-04-09 17:30:29 -07:00
Chendi.Xue ef5a226819 [PD][HeteroArch]Fix accuracy issue with CPU_ATTN as Decoder and Flash_ATTN as prefiller (#38935) 
Signed-off-by: Chendi Xue <chendi.xue@intel.com>
 2026-04-09 11:19:07 +08:00
Or Ozeri 512c5eb455 [kv_offload+HMA][5/N]: Track group block hashes and block IDs (#37109) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-04-08 19:50:28 +03:00
ibifrostandSimon Mo 96b5004b71 [KVConnector] Support 3FS KVConnector (#37636) 
Signed-off-by: wuchenxin <wuchenxin.wcx@alibaba-inc.com>
Signed-off-by: ibifrost <47308427+ibifrost@users.noreply.github.com>
Co-authored-by: Simon Mo <simon.mo@hey.com>
 2026-04-07 15:46:00 +00:00
maobaolongandChauncey 729eb59f60 [KVConnector]: prioritize external connector over internal registry (#38301) 
Signed-off-by: baoloongmao <baoloongmao@tencent.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
 2026-04-07 15:03:11 +00:00
zhanqiuhu bfdc0a3a99 [NIXL][Mamba][3/N] Heterogeneous TP: 3-read conv state transfer (#37635) 2026-04-06 19:07:02 +02:00
yzong-rhandNicolò Lucchesi dc0428ebb8 [NIXL][BUG] Fix Triton heterogeneous TP (#37940) 
Signed-off-by: Yifan <yzong@redhat.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
 2026-04-01 17:23:15 +02:00
yjzandNicolò Lucchesi 992368522f [KVTransfer] Fix TpKVTopology.is_kv_replicated equality case (#38179) 
Signed-off-by: JianDan0212 <zhangyj0212@gmail.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
 2026-04-01 12:41:49 +02:00
Yifan Qiao 91e4521f9f [Feat][v1] Simple yet General CPU KV Cache Offloading (#37160) 
Signed-off-by: Yifan Qiao <yifanqiao@berkeley.edu>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
 2026-03-31 17:58:37 -07:00
Kfir ToledoandOr Ozeri 6cc7abdc66 [kv_offload+HMA] Fix num_blocks with different per-layer page sizes and improve assert message (#38554) 
Signed-off-by: Kfir Toledo <kfir.toledo@ibm.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-03-31 06:00:40 +00:00
dtc 6287e7fa20 [P/D] Mooncake: Add unit tests and minor fixes for mooncake connector (#36946) 
Signed-off-by: Tianchen Ding <dtcccc@linux.alibaba.com>
 2026-03-27 09:26:40 +01:00
Or Ozeri 7cc302dd87 [kv_offload+HMA][7/N]: Support register_kv_caches for hybrid models (#37853) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-03-27 08:38:33 +03:00
yjz b745e8b5d3 [KVTransfer][Mooncake] Add heterogeneous TP support for disaggregated P/D in MooncakeConnector (#36869) 
Signed-off-by: JianDan0212 <zhangyj0212@gmail.com>
 2026-03-25 14:24:07 +01:00
Wentao Ye c59a132f96 [V0 Deprecation] Refactor kv cache from list to element (#37487) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-03-23 20:10:11 -07:00
Wentao Ye b3d0b37908 [Refactor] Remove unused dead code (#36171) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-03-20 16:12:51 -06:00
Martin Hickey 880be2b1b8 [Metrics] Some small refactoring for better maintainability (#33898) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
 2026-03-20 16:11:34 +00:00
zhanqiuhu d49f273144 [SSM/Mamba] Follow-up: N-1 prefill for P/D disaggregation (#37310) 2026-03-19 08:22:00 +01:00
Wentao Ye 0d81a1fe61 [V0 Deprecation] Deprecate virtual engine (#37195) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-03-18 14:30:14 -07:00
Or Ozeri 5dd8df0701 [kv_offload+HMA][2/N]: Support multiple KV groups in GPULoadStoreSpec (#36642) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-03-18 19:26:40 +02:00
Or Ozeri 525f2eeb0b [kv_offload+HMA][6/N]: Split offloading_connector.py (#37405) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-03-18 14:42:46 +01:00
Andy Lo 98b09ddc27 [NIXL][Bugfix] metrics & testing minor bug (#36051) 
Signed-off-by: Andy Lo <andy@mistral.ai>
 2026-03-18 14:39:14 +01:00
Or OzeriandNicolò Lucchesi fcf0687b27 [kv_offload+HMA][0/N]: Support block-level preemption handling (#34805) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
 2026-03-18 08:49:53 +02:00
youkaichao 0fb142a454 [perf][connector] optimize build_connector_meta when host buffer transfer is not used (#37165) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2026-03-17 11:59:35 +00:00
zhanqiuhu 2dccb38f73 [Bugfix][MultiConnector] Fix MultiConnector for SupportsHMA sub-connectors (#36549) 2026-03-16 20:51:04 +00:00
Nicolò Lucchesi f5c081d432 [PD][Nixl] Add support for hybrid SSM-FA models (#36687) 2026-03-16 19:58:06 +01:00
haosdentandOr Ozeri ca1954d58c [Bugfix] Disable cross-layer KV cache for MLA attention backends (#37090) 
Signed-off-by: haosdent <haosdent@gmail.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-03-16 19:03:10 +02:00
Fynn Schmitt-Ulms 04bf5a35fa [Spec Decode] Update extract_hidden_states to use deferred kv_connector clear (#37013) 2026-03-16 14:53:45 +01:00
rasmith 0024f39a32 [ROCm][P/D][MORI][BugFix] Add transfer_id for moriio_connector so moriio_connector to restore P/D functionality (#34907) 
Signed-off-by: Randall Smith <Randall.Smith@amd.com>
 2026-03-16 10:36:51 +08:00
Harry Mellor 5a3f1eb62f [Misc] Set default kv_buffer_device in a better way (#36862) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2026-03-13 19:07:33 +00:00