iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity
16,919 Commits 3 Branches 0 Tags
1806d1adfc9b598bc6eb94de38a330aaad04c291
Commit Graph
377 Commits
Author SHA1 Message Date
Or Ozeri 357fddf614 [kv_offload]: Add DSv4 support (#43142) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-05-24 11:10:12 +03:00
Dao007foreverandClaude 0902d8e62f [KV Connector] Keep MooncakeStore full hits block-aligned (#43494) 
Signed-off-by: Dao Le <daole@inferact.ai>
Signed-off-by: Dao Le <Dao007forever@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-05-23 23:15:03 -07:00
Dao007forever 819c610f9b [Mooncake] Add metrics for MooncakeStoreConnector operations (#43392) 2026-05-23 13:34:40 -07:00
Benjamin Chislettandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> 4e2eba28be [Perf] Optimize hidden state extraction logic (#37374) 
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com>
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-05-22 18:23:08 -04:00
Dao007foreverandClaude Opus 4.7 b21f3d56d4 [KV Connector] MooncakeStore: don't co-queue save with load to avoid double delayed-free (#43371) 
Signed-off-by: Dao Le <Dao007forever@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-22 16:14:11 +00:00
Simon Danielsson 86ccef7d44 [ROCm] Add XGMI backend for MoRI Connector (#41753) 
Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>
 2026-05-22 03:06:40 +00:00
Yifan Qiao 056bc2e166 [KVConnector][DSV4] HMA support for Mooncake store connector (#42828) 
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
 2026-05-19 01:07:46 -07:00
Revital Surandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> afd7b1dce9 [Bugfix] Use platform-agnostic device in example_connector load (#42926) 
Signed-off-by: Revital Sur <eres@il.ibm.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-05-19 03:12:04 +00:00
liranschourOr Ozerimergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		e5417657e5 [KV Connector][Offloading] Flush all pending jobs on last step (#42611) 
Signed-off-by: Liran Schour <lirans@il.ibm.com>
Signed-off-by: liranschour <liranschour@users.noreply.github.com>
Co-authored-by: Or Ozeri <or@ozery.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-18 12:59:42 +00:00
Kfir Toledo e414e1f1c0 [Bugfix][KV Offload] count appended GPU blocks in store group_sizes (#42945) 
Signed-off-by: Kfir Toledo <kfir.toledo@ibm.com>
 2026-05-18 11:36:02 +00:00
Zhewen LiZhewen LiClaude Opus 4.7
 		36e74c9ea4 [KV Connector] Support disk offloading in MooncakeStoreConnector (#42689) 
Signed-off-by: Zhewen Li <zhewenli@inferact.ai>
Co-authored-by: Zhewen Li <zhewenli@inferact.ai>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-16 13:34:15 -07:00
chunxiaozheng f45c210885 [LMCacheMPConnector] Prioritize importing the lmcache_mp_connector from lmcache (#42596) 
Signed-off-by: idellzheng <idellzheng@tencent.com>
 2026-05-15 17:46:31 +00:00
zhanqiuhuandNicolò Lucchesi 24337fb860 PD disagg with NIXL Connector: GDN support (Qwen3.5) (#41869) 
Signed-off-by: Zhanqiu Hu <zhu@redhat.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
 2026-05-14 16:33:01 +02:00
Martin Hickeygemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>Or Ozeri
 		5bd8c71e79 [kv_offload] Implement reset_cache() for the offloading connector (#41956) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Or Ozeri <or@ozery.com>
 2026-05-14 16:00:10 +03:00
Michael GoinandClaude Opus 4.6 2f821faeae [Spec Decode] Support hybrid attention models in extract_hidden_states (#39949) 
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-13 10:45:53 -07:00
Ronen Schaffer 79fd1bc7ed [kv_offload] Add req_id to ReqContext for per-request tracking (#42507) 
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
 2026-05-13 11:11:10 +00:00
Yifan Qiaoandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 13bf242100 [Feat][KVConnector] Add bind_gpu_block_pool() to KVConnectorBase_V1 (#39654) 
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-13 02:10:29 -07:00
Nicolò Lucchesi 71bcd02ef3 [Bugfix][PD] Fix multi-node TP (TP>8) (#39907) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2026-05-12 22:20:57 -07:00
+3
Chao Leileichao.lcivaniumaoshen524Dao007foreverTeng MaPz1116foraxeSkywalker-EPfems14jianzsbaxingpiaochongmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		ebeb09d822 [KV Transfer] Add MooncakeStoreConnector for KV cache offloading via Mooncake distributed store (#40900) 
Signed-off-by: leichao.lc <leichao.lc@antgroup.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: leichao.lc <leichao.lc@antgroup.com>
Co-authored-by: ivanium <yifanqiao@inferact.ai>
Co-authored-by: aoshen524 <aoshen@inferact.ai>
Co-authored-by: Dao007forever <daole@inferact.ai>
Co-authored-by: Teng Ma <sima.mt@alibaba-inc.com>
Co-authored-by: Pz1116 <zpbzpb123123@gmail.com>
Co-authored-by: foraxe <1055696449@qq.com>
Co-authored-by: Skywalker-EP <173423846@qq.com>
Co-authored-by: fems14 <1804143737@qq.com>
Co-authored-by: jianzs <zheng.shoujian@outlook.com>
Co-authored-by: baxingpiaochong <771405853@qq.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-12 16:09:10 -07:00
Martin Hickey 418ba8ef14 [kv_offload][BugFix] Fix store deferral (#41945) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
 2026-05-12 18:04:44 +03:00
zhanqiuhuandNickLucche 593d5a4033 [Bugfix] Fix mismatched kernel-per-logical blocks in NIXL HMA transfer (#42097) 
Signed-off-by: ZhanqiuHu <zhu@redhat.com>
Signed-off-by: Zhanqiu Hu <zhu@redhat.com>
Signed-off-by: NickLucche <nlucches@redhat.com>
Co-authored-by: NickLucche <nlucches@redhat.com>
 2026-05-12 15:53:30 +02:00
Martin Hickey 8415bf2cdb [kv_offload] Set offloading connector to prefer HND layout (#41928) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
 2026-05-11 15:05:41 +03:00
Nicolò Lucchesi 5672d100ed [KV Connector][NIXL][Bugfix] Fix NIXL handshake failures not honoring kv_load_failure_policy (#40364) 
When NIXL handshake fails (e.g., due to compatibility hash mismatch
between prefill and decode instances), requests fail with "engine dead"
error instead of gracefully falling back to local recomputation as configured
by kv_load_failure_policy='recompute'.

Signed-off-by: NickLucche <nlucches@redhat.com>
 2026-05-11 09:37:21 +00:00
Nicolò Lucchesi 770e9bd6b3 [Nixl][PD] Lease renewal TTL KV blocks on P (#41383) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2026-05-11 09:27:30 +00:00
shaharmor98andmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> f9f770ca0b fix nixl side-channel host selection (#41806) 
Signed-off-by: Shahar Mor <smor@nvidia.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-11 07:40:37 +00:00
wangxiyuan 5536fc0c01 [Misc] Replace mamba_type string literals with MambaAttentionBackendEnum (#41188) 
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
 2026-05-11 03:59:36 +00:00
Ronen Schaffer e175192d33 [KV Offload] Pass ReqContext to touch(), complete_load(), and complete_store() (#41366) 
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
 2026-05-10 15:09:25 +03:00
Andreas Karatzas f2840120f6 [ROCm][CI] Fix NIXL spec-decode acceptance startup and diagnostics (#41313) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
 2026-05-10 14:50:16 +08:00
Dao007forever 3f5bd482f5 [Bugfix][KV Transfer][NIXL] Notify P node on pre-admission rejection to free stranded KV blocks (#41269) 2026-05-09 22:52:09 -07:00
Wentao Ye ea0e501bb1 [KV Connector] Remove compat support for pre-v0.12.0 constructor signatures without KVCacheConfig (#39832) 
The v0.12.0 release contained initial support for HMA in KV Connectors. As part
of these changes, a KVCacheConfig argument was added to KV connector
constructors. Backwards compatibility support for out-of-tree connectors was
included in this change, with a very prominent warning. See #25712 and #27887.

Since the warning has been around for over 5 months, we can safely remove
the support of it.

Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-05-09 23:39:46 +00:00
Wentao Ye f80aa53c9d [Refactor] Nixl util using lazy init (#41392) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-05-09 17:46:52 -04:00
Nicolò LucchesiandZhanqiuHu 171d59ae8d [Bugfix][PD] Fix DSv4 Disaggregated (#41957) 
Signed-off-by: NickLucche <nlucches@redhat.com>
Co-authored-by: ZhanqiuHu <zhu@redhat.com>
 2026-05-09 16:48:24 +00:00
Zijing LiuandClaude Opus 4.7 09a7cc5ba9 [KV Connector] Opt DecodeBenchConnector into SupportsHMA (#41770) 
Signed-off-by: Zijing Liu <liuzijing2014@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-07 16:10:00 -07:00
zhanqiuhuandNickLucche df8e63f4ed nixl refactor: new transfer design (#40731) 
Signed-off-by: ZhanqiuHu <zhu@redhat.com>
Signed-off-by: NickLucche <nlucches@redhat.com>
Co-authored-by: NickLucche <nlucches@redhat.com>
 2026-05-06 06:16:25 -07:00
wang.yuqiandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> 51c1ee9b7c [Examples] Resettle Disaggregated examples. (#40759) 
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
Signed-off-by: wang.yuqi <noooop@126.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-05-06 01:20:38 -07:00
Zhewen LiandZhewen Li bee126165f [P/D][Mooncake] Add KVConnectorStats for transfer observability (#40414) 
Signed-off-by: Zhewen Li <zhewenli@inferact.ai>
Co-authored-by: Zhewen Li <zhewenli@inferact.ai>
 2026-05-05 02:17:38 -07:00
Or Ozeri 2fa1f8ec00 [kv_offload+HMA][13/N]: Enable HMA support (#41445) 
This is the final PR in a series to enables HMA support for the
offloading connector. The connector advertises `SupportsHMA`
and is validated with unit tests and e2e tests.

Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-05-01 12:30:03 +01:00
Or Ozeri 941fb50835 [kv_offload+HMA][12/N]: Scheduler-side support for sliding window groups (#41228) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-05-01 06:59:17 +03:00
snadampal 3179e53135 [P/D] Prefill compute optimizations with bi-directional KV cache transfers between P and D nodes (#32553) 
Signed-off-by: Sunita Nadampalli <nadampal@amazon.com>
 2026-04-30 10:14:20 +00:00
Nicolò Lucchesi efdc95674d [KVConnector] MultiConnector SupportsHMA (#39571) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2026-04-30 02:10:50 -07:00
Martin HickeyandOr Ozeri c42981d034 [Refactor][kv_offload] KV Offloading maintainability improvements (#40538) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-04-30 05:55:31 +03:00
Itay EtelisItay EtelisOr OzeriOr Ozeri
 		8a8c9b564e [KV Offload] Per-job store completion for CPU offloading connector (#39186) 
Signed-off-by: Itay Etelis <itay.etelis@ibm.com>
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Co-authored-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: Or Ozeri <or@ozery.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-04-29 08:52:55 +03:00
yangrzandRoger Wang de3fe8dc62 [Bugfix] release KV blocks for skipped P-ranks to prevent invalid KV errors and timeouts when P_tp > D_tp and MLA (#40449) 
Signed-off-by: yangruize <yangruize7@163.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2026-04-28 11:38:43 -07:00
+7
Yifan QiaoYongye ZhuYongye ZhuSimon MoBugen ZhaoGiancarlo DelfinJee Jee LiNick HillRoger WangRoy WangWoosuk KwonyoukaichaoZhewen LiZijing LiukhluuqizixiZhewen Li
 		4d51588e23 [Feat] DeepSeek V4 Rebased (#40860) 
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Signed-off-by: qizixi <zixi@inferact.ai>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Yongye Zhu <yongye@inferact.ai>
Co-authored-by: Simon Mo <simon@inferact.ai>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Roy Wang <yasong.wang@inferact.ai>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: Zhewen Li <jerven.vllm@gmail.com>
Co-authored-by: Zijing Liu <liuzijing2014@gmail.com>
Co-authored-by: khluu <khluu000@gmail.com>
Co-authored-by: qizixi <zixi@inferact.ai>
Co-authored-by: Zhewen Li <zhewenli@inferact.ai>
 2026-04-26 18:31:08 -07:00
omerpaz95andOr Ozeri b39c266dae [KV Offload] Offload all KV blocks when doing prefill in P/D (#40346) 
Signed-off-by: omerpaz95 <omerpaz95@gmail.com>
Signed-off-by: omerpaz95 <73347585+omerpaz95@users.noreply.github.com>
Co-authored-by: Or Ozeri <or@ozery.com>
 2026-04-26 15:06:01 +03:00
Or Ozeri 60cd878a3b [kv_offload+HMA][11/N]: Support store with multiple KV groups (#39403) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-04-25 20:00:46 +03:00
Or Ozeri 51adca74e6 [kv_offload+HMA][9/N]: Support lookup with multiple KV groups (#39401) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-04-24 15:32:29 +03:00
Or Ozeri 5ef33ab250 [kv_offload+HMA][10/N]: Support load with multiple KV groups (#39402) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-04-23 20:00:45 +03:00
Simon DanielssonandMatvei Pashkovskii ac58e2a170 [Fix][MoRI] Align MoRI-IO message format with P2pNcclConnector and vllm-router (#39565) 
Signed-off-by: simondanielsson <simon.danielsson99@hotmail.com>
Co-authored-by: Matvei Pashkovskii <mpashkov@amd.com>
 2026-04-23 08:06:31 +09:00
Or Ozeriandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> aad88f8486 [kv_offload+HMA][8/N]: Support multi-group worker transfer (#38453) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-04-22 08:44:00 +03:00