iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
16,247 Commits 3 Branches 0 Tags
2fa1f8ec00cf85b15422cb4c0e8eb3632ee13ea8
Commit Graph
341 Commits
Author SHA1 Message Date
Or Ozeri 2fa1f8ec00 [kv_offload+HMA][13/N]: Enable HMA support (#41445) 
This is the final PR in a series to enables HMA support for the
offloading connector. The connector advertises `SupportsHMA`
and is validated with unit tests and e2e tests.

Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-05-01 12:30:03 +01:00
Or Ozeri 941fb50835 [kv_offload+HMA][12/N]: Scheduler-side support for sliding window groups (#41228) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-05-01 06:59:17 +03:00
snadampal 3179e53135 [P/D] Prefill compute optimizations with bi-directional KV cache transfers between P and D nodes (#32553) 
Signed-off-by: Sunita Nadampalli <nadampal@amazon.com>
 2026-04-30 10:14:20 +00:00
Nicolò Lucchesi efdc95674d [KVConnector] MultiConnector SupportsHMA (#39571) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2026-04-30 02:10:50 -07:00
Martin HickeyandOr Ozeri c42981d034 [Refactor][kv_offload] KV Offloading maintainability improvements (#40538) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-04-30 05:55:31 +03:00
Itay EtelisItay EtelisOr OzeriOr Ozeri
 		8a8c9b564e [KV Offload] Per-job store completion for CPU offloading connector (#39186) 
Signed-off-by: Itay Etelis <itay.etelis@ibm.com>
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Co-authored-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: Or Ozeri <or@ozery.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-04-29 08:52:55 +03:00
yangrzandRoger Wang de3fe8dc62 [Bugfix] release KV blocks for skipped P-ranks to prevent invalid KV errors and timeouts when P_tp > D_tp and MLA (#40449) 
Signed-off-by: yangruize <yangruize7@163.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2026-04-28 11:38:43 -07:00
+7
Yifan QiaoYongye ZhuYongye ZhuSimon MoBugen ZhaoGiancarlo DelfinJee Jee LiNick HillRoger WangRoy WangWoosuk KwonyoukaichaoZhewen LiZijing LiukhluuqizixiZhewen Li
 		4d51588e23 [Feat] DeepSeek V4 Rebased (#40860) 
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Signed-off-by: qizixi <zixi@inferact.ai>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Yongye Zhu <yongye@inferact.ai>
Co-authored-by: Simon Mo <simon@inferact.ai>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Roy Wang <yasong.wang@inferact.ai>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: Zhewen Li <jerven.vllm@gmail.com>
Co-authored-by: Zijing Liu <liuzijing2014@gmail.com>
Co-authored-by: khluu <khluu000@gmail.com>
Co-authored-by: qizixi <zixi@inferact.ai>
Co-authored-by: Zhewen Li <zhewenli@inferact.ai>
 2026-04-26 18:31:08 -07:00
omerpaz95andOr Ozeri b39c266dae [KV Offload] Offload all KV blocks when doing prefill in P/D (#40346) 
Signed-off-by: omerpaz95 <omerpaz95@gmail.com>
Signed-off-by: omerpaz95 <73347585+omerpaz95@users.noreply.github.com>
Co-authored-by: Or Ozeri <or@ozery.com>
 2026-04-26 15:06:01 +03:00
Or Ozeri 60cd878a3b [kv_offload+HMA][11/N]: Support store with multiple KV groups (#39403) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-04-25 20:00:46 +03:00
Or Ozeri 51adca74e6 [kv_offload+HMA][9/N]: Support lookup with multiple KV groups (#39401) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-04-24 15:32:29 +03:00
Or Ozeri 5ef33ab250 [kv_offload+HMA][10/N]: Support load with multiple KV groups (#39402) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-04-23 20:00:45 +03:00
Simon DanielssonandMatvei Pashkovskii ac58e2a170 [Fix][MoRI] Align MoRI-IO message format with P2pNcclConnector and vllm-router (#39565) 
Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>
Co-authored-by: Matvei Pashkovskii <mpashkov@amd.com>
 2026-04-23 08:06:31 +09:00
Or Ozeriandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> aad88f8486 [kv_offload+HMA][8/N]: Support multi-group worker transfer (#38453) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-04-22 08:44:00 +03:00
rasmith cefa5281a7 [ROCm][P/D][MORI][BugFix] Ensure correct api is used when making requests to prefill / decode nodes (#39835) 
Signed-off-by: Randall Smith <Randall.Smith@amd.com>
 2026-04-22 09:48:25 +09:00
Cao Qian 8b1f3bebca [LMCache MP Connector] Add num_lmcache_extra_cached_token in KVTransferParams (#39843) 
Signed-off-by: aeon-x <talexcao@gmail.com>
 2026-04-20 20:42:49 +00:00
zhanqiuhu cc3993b05d nixl refactor [2/N]: unify TpKVTopology + HeteroTPTransferConfig into TransferTopology (#39529) 
Signed-off-by: Zhanqiu Hu <zhu@redhat.com>
 2026-04-20 12:39:08 +02:00
Ilya Markov 50dd4cb427 [EPLB] Add nixl-based eplb communicator (#36276) 
Signed-off-by: ilmarkov <markovilya197@gmail.com>
Signed-off-by: Markov Ilya <markovilya19@gmail.com>
 2026-04-20 10:24:23 +00:00
Or OzeriandNicolò Lucchesi f774ba028a [kv_offload+HMA][4/N]: Support sliding window lookup (#36645) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
 2026-04-20 12:53:51 +03:00
omerpaz95 4353c9cb4a [KV Offload] Pass request context (#39185) 
Signed-off-by: omerpaz95 <omerpaz95@gmail.com>
 2026-04-19 08:54:59 +03:00
omerpaz95andOr Ozeri 4b7f5ea1a0 [KV Connector] Allow metrics of multiple connectors of same types in multi connector. (#40010) 
Signed-off-by: omerpaz95 <omerpaz95@gmail.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-04-19 07:49:10 +03:00
zhanqiuhu 41488f2acd [Bugfix][NIXL] Fix _logical_to_kernel_block_ids conversion for non-mamba models (#39724) 
Signed-off-by: Zhanqiu Hu <zhu@redhat.com>
 2026-04-15 20:08:58 +00:00
Zhewen LiZhewen LiOpenAI Codexmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		55e1a8e103 [Mooncake] Fix mixed MLA+Eagle block-size validation (#39596) 
Signed-off-by: Zhewen Li <zhewenli@inferact.ai>
Co-authored-by: Zhewen Li <zhewenli@inferact.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-04-15 11:36:47 -07:00
Roy Huang ed33310552 [KVConnector][LMCache] Propagate cache_salt through MP connector for per-user cache isolation (#39837) 
Signed-off-by: royyhuang <royyhuang@gmail.com>
Signed-off-by: royyhuang <roy.y.huang@gmail.com>
 2026-04-15 09:10:49 -07:00
Or Ozeri 235e1f930a [kv_offload+HMA][3/N]: Remove block_size from KVEvents (#36644) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-04-15 11:53:19 +03:00
Zhewen LiandZhewen Li 1696c864b9 [Bugfix][Mooncake] Fix thread-local CUDA context for NVLink transfers in _send_blocks (#39548) 
Signed-off-by: Zhewen Li <zhewenli@inferact.ai>
Co-authored-by: Zhewen Li <zhewenli@inferact.ai>
 2026-04-14 14:13:58 -07:00
maobaolong b2f749dc97 fix(lmcache): correct store for cached requests while enable prefix cache (#39719) 
Signed-off-by: baoloongmao <baoloongmao@tencent.com>
 2026-04-14 20:51:27 +00:00
chunxiaozhengandYihua Cheng c687bf226a [LMCache][MP] optimize save when mla enabled (#38810) 
Signed-off-by: idellzheng <idellzheng@tencent.com>
Co-authored-by: Yihua Cheng <yihua98@uchicago.edu>
 2026-04-13 17:56:43 -07:00
maobaolong 2a3c32ce67 fix(lmcache): correct store for cached requests and num_scheduled_tokens in lmcache_mp_connector.py (#39655) 
Signed-off-by: baoloongmao <baoloongmao@tencent.com>
 2026-04-13 03:29:19 +00:00
Nicolò Lucchesi 3244a2ebf2 [KVConnector][NIXL] Organize NIXL connector into its own directory (#39354) 
The number of features supported by the connector has grown substantially
and the `nixl_connector.py` file has accumulated a lot of code. Creates a separate
directory and isolates connector/scheduler code in the hope of improving clarity
and maintainability.

Further refactor of components aimed at improving clarity and simplifying code
will follow soon.

Signed-off-by: NickLucche <nlucches@redhat.com>
 2026-04-12 13:10:50 +00:00
Ronen Schaffer a5b17fba8f [KV Offload] Implement shutdown() in OffloadingConnector and related classes (#39182) 
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
 2026-04-10 08:06:22 +03:00
Yuwei Anandyuwei 467d3247c3 [LMCache] vLLM Block Allocation Event (#38856) 
Signed-off-by: yuwei <yuwei@dev.local>
Co-authored-by: yuwei <yuwei@dev.local>
 2026-04-09 17:30:29 -07:00
Chendi.Xue ef5a226819 [PD][HeteroArch]Fix accuracy issue with CPU_ATTN as Decoder and Flash_ATTN as prefiller (#38935) 
Signed-off-by: Chendi Xue <chendi.xue@intel.com>
 2026-04-09 11:19:07 +08:00
Or Ozeri 512c5eb455 [kv_offload+HMA][5/N]: Track group block hashes and block IDs (#37109) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-04-08 19:50:28 +03:00
ibifrostandSimon Mo 96b5004b71 [KVConnector] Support 3FS KVConnector (#37636) 
Signed-off-by: wuchenxin <wuchenxin.wcx@alibaba-inc.com>
Signed-off-by: ibifrost <47308427+ibifrost@users.noreply.github.com>
Co-authored-by: Simon Mo <simon.mo@hey.com>
 2026-04-07 15:46:00 +00:00
maobaolongandChauncey 729eb59f60 [KVConnector]: prioritize external connector over internal registry (#38301) 
Signed-off-by: baoloongmao <baoloongmao@tencent.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
 2026-04-07 15:03:11 +00:00
zhanqiuhu bfdc0a3a99 [NIXL][Mamba][3/N] Heterogeneous TP: 3-read conv state transfer (#37635) 2026-04-06 19:07:02 +02:00
yzong-rhandNicolò Lucchesi dc0428ebb8 [NIXL][BUG] Fix Triton heterogeneous TP (#37940) 
Signed-off-by: Yifan <yzong@redhat.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
 2026-04-01 17:23:15 +02:00
yjzandNicolò Lucchesi 992368522f [KVTransfer] Fix TpKVTopology.is_kv_replicated equality case (#38179) 
Signed-off-by: JianDan0212 <zhangyj0212@gmail.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
 2026-04-01 12:41:49 +02:00
Yifan Qiao 91e4521f9f [Feat][v1] Simple yet General CPU KV Cache Offloading (#37160) 
Signed-off-by: Yifan Qiao <yifanqiao@berkeley.edu>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
 2026-03-31 17:58:37 -07:00
Kfir ToledoandOr Ozeri 6cc7abdc66 [kv_offload+HMA] Fix num_blocks with different per-layer page sizes and improve assert message (#38554) 
Signed-off-by: Kfir Toledo <kfir.toledo@ibm.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-03-31 06:00:40 +00:00
dtc 6287e7fa20 [P/D] Mooncake: Add unit tests and minor fixes for mooncake connector (#36946) 
Signed-off-by: Tianchen Ding <dtcccc@linux.alibaba.com>
 2026-03-27 09:26:40 +01:00
Or Ozeri 7cc302dd87 [kv_offload+HMA][7/N]: Support register_kv_caches for hybrid models (#37853) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-03-27 08:38:33 +03:00
yjz b745e8b5d3 [KVTransfer][Mooncake] Add heterogeneous TP support for disaggregated P/D in MooncakeConnector (#36869) 
Signed-off-by: JianDan0212 <zhangyj0212@gmail.com>
 2026-03-25 14:24:07 +01:00
Wentao Ye c59a132f96 [V0 Deprecation] Refactor kv cache from list to element (#37487) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-03-23 20:10:11 -07:00
Wentao Ye b3d0b37908 [Refactor] Remove unused dead code (#36171) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-03-20 16:12:51 -06:00
Martin Hickey 880be2b1b8 [Metrics] Some small refactoring for better maintainability (#33898) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
 2026-03-20 16:11:34 +00:00
zhanqiuhu d49f273144 [SSM/Mamba] Follow-up: N-1 prefill for P/D disaggregation (#37310) 2026-03-19 08:22:00 +01:00
Wentao Ye 0d81a1fe61 [V0 Deprecation] Deprecate virtual engine (#37195) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-03-18 14:30:14 -07:00
Or Ozeri 5dd8df0701 [kv_offload+HMA][2/N]: Support multiple KV groups in GPULoadStoreSpec (#36642) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-03-18 19:26:40 +02:00