iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
16,247 Commits 3 Branches 0 Tags
2fa1f8ec00cf85b15422cb4c0e8eb3632ee13ea8
Commit Graph
721 Commits
Author SHA1 Message Date
Or Ozeri 2fa1f8ec00 [kv_offload+HMA][13/N]: Enable HMA support (#41445) 
This is the final PR in a series to enables HMA support for the
offloading connector. The connector advertises `SupportsHMA`
and is validated with unit tests and e2e tests.

Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-05-01 12:30:03 +01:00
Or Ozeri 941fb50835 [kv_offload+HMA][12/N]: Scheduler-side support for sliding window groups (#41228) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-05-01 06:59:17 +03:00
Yongye ZhuZijing LiuClaude Opus 4.7
 		b4806c8ee1 [DSV4] Add BF16 and MXFP8 A2A support for flashinfer a2a one sided (#40960) 
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Signed-off-by: Zijing Liu <liuzijing2014@gmail.com>
Co-authored-by: Zijing Liu <liuzijing2014@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-30 15:33:12 -07:00
tejTJianHAIAI
 		121dbe7a22 [ROCm] ROCm DeepEP API updated to latest (#39721) 
Signed-off-by: Tej Kiran <vpolamre@amd.com>
Signed-off-by: tej <37236721+itej89@users.noreply.github.com>
Co-authored-by: TJian <tunjian.tan@embeddedllm.com>
Co-authored-by: HAIAI <39548240+HAIAI@users.noreply.github.com>
 2026-04-30 07:46:59 -07:00
Ilya MarkovMarkov IlyaNicolò Lucchesi
 		a7fb008510 [EPLB] Optimize memory overhead in Nixl communicator (#40013) 
Signed-off-by: ilmarkov <markovilya197@gmail.com>
Signed-off-by: Markov Ilya <markovilya19@gmail.com>
Co-authored-by: Markov Ilya <markovilya19@gmail.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
 2026-04-30 07:46:49 -07:00
snadampal 3179e53135 [P/D] Prefill compute optimizations with bi-directional KV cache transfers between P and D nodes (#32553) 
Signed-off-by: Sunita Nadampalli <nadampal@amazon.com>
 2026-04-30 10:14:20 +00:00
Nicolò Lucchesi efdc95674d [KVConnector] MultiConnector SupportsHMA (#39571) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2026-04-30 02:10:50 -07:00
Martin HickeyandOr Ozeri c42981d034 [Refactor][kv_offload] KV Offloading maintainability improvements (#40538) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-04-30 05:55:31 +03:00
Itay EtelisItay EtelisOr OzeriOr Ozeri
 		8a8c9b564e [KV Offload] Per-job store completion for CPU offloading connector (#39186) 
Signed-off-by: Itay Etelis <itay.etelis@ibm.com>
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Co-authored-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: Or Ozeri <or@ozery.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-04-29 08:52:55 +03:00
yangrzandRoger Wang de3fe8dc62 [Bugfix] release KV blocks for skipped P-ranks to prevent invalid KV errors and timeouts when P_tp > D_tp and MLA (#40449) 
Signed-off-by: yangruize <yangruize7@163.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2026-04-28 11:38:43 -07:00
+7
Yifan QiaoYongye ZhuYongye ZhuSimon MoBugen ZhaoGiancarlo DelfinJee Jee LiNick HillRoger WangRoy WangWoosuk KwonyoukaichaoZhewen LiZijing LiukhluuqizixiZhewen Li
 		4d51588e23 [Feat] DeepSeek V4 Rebased (#40860) 
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Signed-off-by: qizixi <zixi@inferact.ai>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Yongye Zhu <yongye@inferact.ai>
Co-authored-by: Simon Mo <simon@inferact.ai>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Roy Wang <yasong.wang@inferact.ai>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: Zhewen Li <jerven.vllm@gmail.com>
Co-authored-by: Zijing Liu <liuzijing2014@gmail.com>
Co-authored-by: khluu <khluu000@gmail.com>
Co-authored-by: qizixi <zixi@inferact.ai>
Co-authored-by: Zhewen Li <zhewenli@inferact.ai>
 2026-04-26 18:31:08 -07:00
omerpaz95andOr Ozeri b39c266dae [KV Offload] Offload all KV blocks when doing prefill in P/D (#40346) 
Signed-off-by: omerpaz95 <omerpaz95@gmail.com>
Signed-off-by: omerpaz95 <73347585+omerpaz95@users.noreply.github.com>
Co-authored-by: Or Ozeri <or@ozery.com>
 2026-04-26 15:06:01 +03:00
Dao007forever 9558f43903 [Bugfix] Size FlashInfer NVLink MNNVL workspace to EP group (#40893) 
Signed-off-by: Dao Le <Dao007forever@gmail.com>
 2026-04-26 01:26:34 -07:00
Or Ozeri 60cd878a3b [kv_offload+HMA][11/N]: Support store with multiple KV groups (#39403) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-04-25 20:00:46 +03:00
Or Ozeri 51adca74e6 [kv_offload+HMA][9/N]: Support lookup with multiple KV groups (#39401) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-04-24 15:32:29 +03:00
Sage Moore 62b1bbe470 [EPLB] Remove asyncio infrastructure from Async EPLB (#40730) 
Signed-off-by: Sage Moore <sage@neuralmagic.com>
 2026-04-24 00:21:15 +00:00
Or Ozeri 5ef33ab250 [kv_offload+HMA][10/N]: Support load with multiple KV groups (#39402) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-04-23 20:00:45 +03:00
Simon DanielssonandMatvei Pashkovskii ac58e2a170 [Fix][MoRI] Align MoRI-IO message format with P2pNcclConnector and vllm-router (#39565) 
Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>
Co-authored-by: Matvei Pashkovskii <mpashkov@amd.com>
 2026-04-23 08:06:31 +09:00
Wentao Yeandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> 8f87eb4622 [Refactor] Clean up log once scope="local" (#40540) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-04-22 16:42:43 -04:00
Or Ozeriandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> aad88f8486 [kv_offload+HMA][8/N]: Support multi-group worker transfer (#38453) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-04-22 08:44:00 +03:00
Soila Kavulya f90aa44662 [NIXL][XPU]Fix nixl import on XPU (#40430) 
Signed-off-by: Soila Kavulya <soila.p.kavulya.intel.com>
 2026-04-22 09:26:33 +08:00
rasmith cefa5281a7 [ROCm][P/D][MORI][BugFix] Ensure correct api is used when making requests to prefill / decode nodes (#39835) 
Signed-off-by: Randall Smith <Randall.Smith@amd.com>
 2026-04-22 09:48:25 +09:00
bnellnm 5e584ce9ec [MoE Refactor] Remove SharedFusedMoE class (#35782) 
Signed-off-by: Bill Nell <bnell@redhat.com>
 2026-04-21 18:12:12 -04:00
Sage Moore 3173441b0f [EPLB] Consolidate is_unchanged/is_received_locally into TransferMetadata (#37341) 
Signed-off-by: Sage Moore <sage@neuralmagic.com>
 2026-04-20 21:12:42 +00:00
Cao Qian 8b1f3bebca [LMCache MP Connector] Add num_lmcache_extra_cached_token in KVTransferParams (#39843) 
Signed-off-by: aeon-x <talexcao@gmail.com>
 2026-04-20 20:42:49 +00:00
Tyler Michael Smith 81d954f454 [WideEP] Remove naive all2all. Use allgather_reducescatter instead (#33728) 
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
 2026-04-20 17:53:55 +00:00
Sage MooreandTyler Michael Smith 3461c8b027 [EPLB] Refactor Async EPLB synchronization logic (#37601) 
Signed-off-by: Sage Moore <sage@neuralmagic.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
 2026-04-20 17:05:41 +00:00
zhanqiuhu cc3993b05d nixl refactor [2/N]: unify TpKVTopology + HeteroTPTransferConfig into TransferTopology (#39529) 
Signed-off-by: Zhanqiu Hu <zhu@redhat.com>
 2026-04-20 12:39:08 +02:00
Ilya Markov 50dd4cb427 [EPLB] Add nixl-based eplb communicator (#36276) 
Signed-off-by: ilmarkov <markovilya197@gmail.com>
Signed-off-by: Markov Ilya <markovilya19@gmail.com>
 2026-04-20 10:24:23 +00:00
Or OzeriandNicolò Lucchesi f774ba028a [kv_offload+HMA][4/N]: Support sliding window lookup (#36645) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
 2026-04-20 12:53:51 +03:00
Fadi Arafeh 2aab9acf48 [CPU][BugFix] Fix inter-node pipeline parallel (#40150) 
Signed-off-by: Fadi Arafeh <fadi.arafeh@arm.com>
 2026-04-20 17:21:12 +08:00
omerpaz95 4353c9cb4a [KV Offload] Pass request context (#39185) 
Signed-off-by: omerpaz95 <omerpaz95@gmail.com>
 2026-04-19 08:54:59 +03:00
omerpaz95andOr Ozeri 4b7f5ea1a0 [KV Connector] Allow metrics of multiple connectors of same types in multi connector. (#40010) 
Signed-off-by: omerpaz95 <omerpaz95@gmail.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-04-19 07:49:10 +03:00
Chaojun Zhangandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 993859ceb0 [XPU] fix all_reduce all-zero accuracy issue under torch.compile (#39844) 
Signed-off-by: Chaojun Zhang <chaojun.zhang@intel.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-04-18 02:33:07 +00:00
zhanqiuhu 41488f2acd [Bugfix][NIXL] Fix _logical_to_kernel_block_ids conversion for non-mamba models (#39724) 
Signed-off-by: Zhanqiu Hu <zhu@redhat.com>
 2026-04-15 20:08:58 +00:00
Zhewen LiZhewen LiOpenAI Codexmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		55e1a8e103 [Mooncake] Fix mixed MLA+Eagle block-size validation (#39596) 
Signed-off-by: Zhewen Li <zhewenli@inferact.ai>
Co-authored-by: Zhewen Li <zhewenli@inferact.ai>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-04-15 11:36:47 -07:00
Roy Huang ed33310552 [KVConnector][LMCache] Propagate cache_salt through MP connector for per-user cache isolation (#39837) 
Signed-off-by: royyhuang <royyhuang@gmail.com>
Signed-off-by: royyhuang <roy.y.huang@gmail.com>
 2026-04-15 09:10:49 -07:00
Or Ozeri 235e1f930a [kv_offload+HMA][3/N]: Remove block_size from KVEvents (#36644) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-04-15 11:53:19 +03:00
Zhewen LiandZhewen Li 1696c864b9 [Bugfix][Mooncake] Fix thread-local CUDA context for NVLink transfers in _send_blocks (#39548) 
Signed-off-by: Zhewen Li <zhewenli@inferact.ai>
Co-authored-by: Zhewen Li <zhewenli@inferact.ai>
 2026-04-14 14:13:58 -07:00
maobaolong b2f749dc97 fix(lmcache): correct store for cached requests while enable prefix cache (#39719) 
Signed-off-by: baoloongmao <baoloongmao@tencent.com>
 2026-04-14 20:51:27 +00:00
chunxiaozhengandYihua Cheng c687bf226a [LMCache][MP] optimize save when mla enabled (#38810) 
Signed-off-by: idellzheng <idellzheng@tencent.com>
Co-authored-by: Yihua Cheng <yihua98@uchicago.edu>
 2026-04-13 17:56:43 -07:00
maobaolong 2a3c32ce67 fix(lmcache): correct store for cached requests and num_scheduled_tokens in lmcache_mp_connector.py (#39655) 
Signed-off-by: baoloongmao <baoloongmao@tencent.com>
 2026-04-13 03:29:19 +00:00
Nicolò Lucchesi 3244a2ebf2 [KVConnector][NIXL] Organize NIXL connector into its own directory (#39354) 
The number of features supported by the connector has grown substantially
and the `nixl_connector.py` file has accumulated a lot of code. Creates a separate
directory and isolates connector/scheduler code in the hope of improving clarity
and maintainability.

Further refactor of components aimed at improving clarity and simplifying code
will follow soon.

Signed-off-by: NickLucche <nlucches@redhat.com>
 2026-04-12 13:10:50 +00:00
Martin HickeyandOr Ozeri cc07dad789 [HMA] [KVEvent] Enable GPU-side KV events for HMA (#37688) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
Co-authored-by: Or Ozeri <or@ozery.com>
 2026-04-12 10:01:02 +03:00
Ronen Schaffer a5b17fba8f [KV Offload] Implement shutdown() in OffloadingConnector and related classes (#39182) 
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
 2026-04-10 08:06:22 +03:00
Yuwei Anandyuwei 467d3247c3 [LMCache] vLLM Block Allocation Event (#38856) 
Signed-off-by: yuwei <yuwei@dev.local>
Co-authored-by: yuwei <yuwei@dev.local>
 2026-04-09 17:30:29 -07:00
Chendi.Xue ef5a226819 [PD][HeteroArch]Fix accuracy issue with CPU_ATTN as Decoder and Flash_ATTN as prefiller (#38935) 
Signed-off-by: Chendi Xue <chendi.xue@intel.com>
 2026-04-09 11:19:07 +08:00
Or Ozeri 512c5eb455 [kv_offload+HMA][5/N]: Track group block hashes and block IDs (#37109) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-04-08 19:50:28 +03:00
Andrey Talman 2111997f96 [release 2.11] Update to torch 2.11 (#34644) 2026-04-07 18:55:48 -07:00
ibifrostandSimon Mo 96b5004b71 [KVConnector] Support 3FS KVConnector (#37636) 
Signed-off-by: wuchenxin <wuchenxin.wcx@alibaba-inc.com>
Signed-off-by: ibifrost <47308427+ibifrost@users.noreply.github.com>
Co-authored-by: Simon Mo <simon.mo@hey.com>
 2026-04-07 15:46:00 +00:00