iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
16,743 Commits 3 Branches 0 Tags
afd7b1dce94fed484351fafd5bf5ea6601ac621e
Commit Graph
761 Commits
Author SHA1 Message Date
Revital Surandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> afd7b1dce9 [Bugfix] Use platform-agnostic device in example_connector load (#42926) 
Signed-off-by: Revital Sur <eres@il.ibm.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-05-19 03:12:04 +00:00
liranschourOr Ozerimergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		e5417657e5 [KV Connector][Offloading] Flush all pending jobs on last step (#42611) 
Signed-off-by: Liran Schour <lirans@il.ibm.com>
Signed-off-by: liranschour <liranschour@users.noreply.github.com>
Co-authored-by: Or Ozeri <or@ozery.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-18 12:59:42 +00:00
Kfir Toledo e414e1f1c0 [Bugfix][KV Offload] count appended GPU blocks in store group_sizes (#42945) 
Signed-off-by: Kfir Toledo <kfir.toledo@ibm.com>
 2026-05-18 11:36:02 +00:00
Zhewen LiZhewen LiClaude Opus 4.7
 		36e74c9ea4 [KV Connector] Support disk offloading in MooncakeStoreConnector (#42689) 
Signed-off-by: Zhewen Li <zhewenli@inferact.ai>
Co-authored-by: Zhewen Li <zhewenli@inferact.ai>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-16 13:34:15 -07:00
chunxiaozheng f45c210885 [LMCacheMPConnector] Prioritize importing the lmcache_mp_connector from lmcache (#42596) 
Signed-off-by: idellzheng <idellzheng@tencent.com>
 2026-05-15 17:46:31 +00:00
Aaron Hao e0a45f1455 [Feat][RL] IPC weight sync optimizations: multigpu support and chunked packed tensors (#37476) 
Signed-off-by: ahao-anyscale <ahao@anyscale.com>
Signed-off-by: hao-aaron <ahao@anyscale.com>
 2026-05-15 22:53:06 +08:00
zhanqiuhuandNicolò Lucchesi 24337fb860 PD disagg with NIXL Connector: GDN support (Qwen3.5) (#41869) 
Signed-off-by: Zhanqiu Hu <zhu@redhat.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
 2026-05-14 16:33:01 +02:00
Martin Hickeygemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>Or Ozeri
 		5bd8c71e79 [kv_offload] Implement reset_cache() for the offloading connector (#41956) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Or Ozeri <or@ozery.com>
 2026-05-14 16:00:10 +03:00
Michael GoinandClaude Opus 4.6 2f821faeae [Spec Decode] Support hybrid attention models in extract_hidden_states (#39949) 
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-13 10:45:53 -07:00
Ronen Schaffer 79fd1bc7ed [kv_offload] Add req_id to ReqContext for per-request tracking (#42507) 
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
 2026-05-13 11:11:10 +00:00
Yifan Qiaoandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 13bf242100 [Feat][KVConnector] Add bind_gpu_block_pool() to KVConnectorBase_V1 (#39654) 
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-13 02:10:29 -07:00
Nicolò Lucchesi 71bcd02ef3 [Bugfix][PD] Fix multi-node TP (TP>8) (#39907) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2026-05-12 22:20:57 -07:00
bnellnm 18f6bf5a21 [MoE Refactor] Add sequence parallel tests to test_moe_layer.py (#41299) 
Signed-off-by: Bill Nell <bnell@redhat.com>
 2026-05-12 21:52:19 -04:00
+3
Chao Leileichao.lcivaniumaoshen524Dao007foreverTeng MaPz1116foraxeSkywalker-EPfems14jianzsbaxingpiaochongmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		ebeb09d822 [KV Transfer] Add MooncakeStoreConnector for KV cache offloading via Mooncake distributed store (#40900) 
Signed-off-by: leichao.lc <leichao.lc@antgroup.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: leichao.lc <leichao.lc@antgroup.com>
Co-authored-by: ivanium <yifanqiao@inferact.ai>
Co-authored-by: aoshen524 <aoshen@inferact.ai>
Co-authored-by: Dao007forever <daole@inferact.ai>
Co-authored-by: Teng Ma <sima.mt@alibaba-inc.com>
Co-authored-by: Pz1116 <zpbzpb123123@gmail.com>
Co-authored-by: foraxe <1055696449@qq.com>
Co-authored-by: Skywalker-EP <173423846@qq.com>
Co-authored-by: fems14 <1804143737@qq.com>
Co-authored-by: jianzs <zheng.shoujian@outlook.com>
Co-authored-by: baxingpiaochong <771405853@qq.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-12 16:09:10 -07:00
bnellnmandRobert Shaw d9b4990783 [MoE Refactor] EPLB refactoring for FusedMoE (#41055) 
Signed-off-by: Bill Nell <bnell@redhat.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
 2026-05-12 14:16:31 -04:00
Yan Ru Pei bcb9c133ba feat(kv-events): emit KV cache metadata (#40984) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-05-12 15:58:48 +00:00
Martin Hickey 418ba8ef14 [kv_offload][BugFix] Fix store deferral (#41945) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
 2026-05-12 18:04:44 +03:00
zhanqiuhuandNickLucche 593d5a4033 [Bugfix] Fix mismatched kernel-per-logical blocks in NIXL HMA transfer (#42097) 
Signed-off-by: ZhanqiuHu <zhu@redhat.com>
Signed-off-by: Zhanqiu Hu <zhu@redhat.com>
Signed-off-by: NickLucche <nlucches@redhat.com>
Co-authored-by: NickLucche <nlucches@redhat.com>
 2026-05-12 15:53:30 +02:00
pschlan-amd 39dff5ff39 Add VLLM_USE_SPINLOOP_EXT to use more efficient busy polling (#36517) 
Signed-off-by: Patrick Schlangen <pschlan@amd.com>
 2026-05-11 16:11:49 -07:00
Jeffrey WangandTyler Michael Smith 56e5810ff1 [BugFix] Prevent orphaned process on NCCL destroy (#39846) 
Signed-off-by: Jeffrey Wang <jeffreywang@anyscale.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
 2026-05-11 15:25:26 -04:00
Martin Hickey 8415bf2cdb [kv_offload] Set offloading connector to prefer HND layout (#41928) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
 2026-05-11 15:05:41 +03:00
Nicolò Lucchesi 5672d100ed [KV Connector][NIXL][Bugfix] Fix NIXL handshake failures not honoring kv_load_failure_policy (#40364) 
When NIXL handshake fails (e.g., due to compatibility hash mismatch
between prefill and decode instances), requests fail with "engine dead"
error instead of gracefully falling back to local recomputation as configured
by kv_load_failure_policy='recompute'.

Signed-off-by: NickLucche <nlucches@redhat.com>
 2026-05-11 09:37:21 +00:00
Nicolò Lucchesi 770e9bd6b3 [Nixl][PD] Lease renewal TTL KV blocks on P (#41383) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2026-05-11 09:27:30 +00:00
shaharmor98andmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> f9f770ca0b fix nixl side-channel host selection (#41806) 
Signed-off-by: Shahar Mor <smor@nvidia.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-11 07:40:37 +00:00
wangxiyuan 5536fc0c01 [Misc] Replace mamba_type string literals with MambaAttentionBackendEnum (#41188) 
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
 2026-05-11 03:59:36 +00:00
Ronen Schaffer e175192d33 [KV Offload] Pass ReqContext to touch(), complete_load(), and complete_store() (#41366) 
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
 2026-05-10 15:09:25 +03:00
Andreas Karatzas f2840120f6 [ROCm][CI] Fix NIXL spec-decode acceptance startup and diagnostics (#41313) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
 2026-05-10 14:50:16 +08:00
Dao007forever 3f5bd482f5 [Bugfix][KV Transfer][NIXL] Notify P node on pre-admission rejection to free stranded KV blocks (#41269) 2026-05-09 22:52:09 -07:00
Wentao Ye ea0e501bb1 [KV Connector] Remove compat support for pre-v0.12.0 constructor signatures without KVCacheConfig (#39832) 
The v0.12.0 release contained initial support for HMA in KV Connectors. As part
of these changes, a KVCacheConfig argument was added to KV connector
constructors. Backwards compatibility support for out-of-tree connectors was
included in this change, with a very prominent warning. See #25712 and #27887.

Since the warning has been around for over 5 months, we can safely remove
the support of it.

Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-05-09 23:39:46 +00:00
Wentao Ye f80aa53c9d [Refactor] Nixl util using lazy init (#41392) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-05-09 17:46:52 -04:00
Nicolò LucchesiandZhanqiuHu 171d59ae8d [Bugfix][PD] Fix DSv4 Disaggregated (#41957) 
Signed-off-by: NickLucche <nlucches@redhat.com>
Co-authored-by: ZhanqiuHu <zhu@redhat.com>
 2026-05-09 16:48:24 +00:00
Sumanth R Hegde e3b65a5ba0 [feat] Add explicit /start_weight_update and /finish_weight_update APIs for weight transfer (#39212) 2026-05-08 18:03:33 -07:00
haosdentandNicolò Lucchesi 44e6b44a21 [CI][Elastic EP] Fix Elastic EP Scaling Test Failure (#41792) 
Signed-off-by: haosdent <haosdent@gmail.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
 2026-05-08 15:17:44 -04:00
Zijing LiuandClaude Opus 4.7 09a7cc5ba9 [KV Connector] Opt DecodeBenchConnector into SupportsHMA (#41770) 
Signed-off-by: Zijing Liu <liuzijing2014@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-07 16:10:00 -07:00
Nicolò Lucchesi 9d6500b89d [Misc] Delay EPLB Nixl import until needed (#41805) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2026-05-07 09:43:07 -07:00
Chaojun Zhangandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 2a84da3b17 [XPU] Implement out-of-place all-reduce functionality (#41808) 
Signed-off-by: Chaojun,Zhang <chaojun.zhang@intel.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-07 05:58:05 -07:00
zhanqiuhuandNickLucche df8e63f4ed nixl refactor: new transfer design (#40731) 
Signed-off-by: ZhanqiuHu <zhu@redhat.com>
Signed-off-by: NickLucche <nlucches@redhat.com>
Co-authored-by: NickLucche <nlucches@redhat.com>
 2026-05-06 06:16:25 -07:00
wang.yuqiandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> 51c1ee9b7c [Examples] Resettle Disaggregated examples. (#40759) 
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
Signed-off-by: wang.yuqi <noooop@126.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-05-06 01:20:38 -07:00
Zhewen LiandZhewen Li bee126165f [P/D][Mooncake] Add KVConnectorStats for transfer observability (#40414) 
Signed-off-by: Zhewen Li <zhewenli@inferact.ai>
Co-authored-by: Zhewen Li <zhewenli@inferact.ai>
 2026-05-05 02:17:38 -07:00
vllmellmRita Brugarolasjunkang1991Luka GovedičTJian
 		529c671e80 [ROCm][FEAT] AITER Fused Allreduce + RMSNorm (#37646) 
Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com>
Signed-off-by: Rita Brugarolas Brufau <rita.brugarolasbrufau@amd.com>
Signed-off-by: junkang1991 <junkangchow@gmail.com>
Co-authored-by: Rita Brugarolas <Rita.BrugarolasBrufau@amd.com>
Co-authored-by: junkang1991 <junkangchow@gmail.com>
Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
Co-authored-by: TJian <tunjian.tan@embeddedllm.com>
 2026-05-01 23:07:18 +08:00
Or Ozeri 2fa1f8ec00 [kv_offload+HMA][13/N]: Enable HMA support (#41445) 
This is the final PR in a series to enables HMA support for the
offloading connector. The connector advertises `SupportsHMA`
and is validated with unit tests and e2e tests.

Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-05-01 12:30:03 +01:00
Or Ozeri 941fb50835 [kv_offload+HMA][12/N]: Scheduler-side support for sliding window groups (#41228) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-05-01 06:59:17 +03:00
Yongye ZhuZijing LiuClaude Opus 4.7
 		b4806c8ee1 [DSV4] Add BF16 and MXFP8 A2A support for flashinfer a2a one sided (#40960) 
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Signed-off-by: Zijing Liu <liuzijing2014@gmail.com>
Co-authored-by: Zijing Liu <liuzijing2014@users.noreply.github.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-30 15:33:12 -07:00
tejTJianHAIAI
 		121dbe7a22 [ROCm] ROCm DeepEP API updated to latest (#39721) 
Signed-off-by: Tej Kiran <vpolamre@amd.com>
Signed-off-by: tej <37236721+itej89@users.noreply.github.com>
Co-authored-by: TJian <tunjian.tan@embeddedllm.com>
Co-authored-by: HAIAI <39548240+HAIAI@users.noreply.github.com>
 2026-04-30 07:46:59 -07:00
Ilya MarkovMarkov IlyaNicolò Lucchesi
 		a7fb008510 [EPLB] Optimize memory overhead in Nixl communicator (#40013) 
Signed-off-by: ilmarkov <markovilya197@gmail.com>
Signed-off-by: Markov Ilya <markovilya19@gmail.com>
Co-authored-by: Markov Ilya <markovilya19@gmail.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
 2026-04-30 07:46:49 -07:00
snadampal 3179e53135 [P/D] Prefill compute optimizations with bi-directional KV cache transfers between P and D nodes (#32553) 
Signed-off-by: Sunita Nadampalli <nadampal@amazon.com>
 2026-04-30 10:14:20 +00:00
Nicolò Lucchesi efdc95674d [KVConnector] MultiConnector SupportsHMA (#39571) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2026-04-30 02:10:50 -07:00
Martin HickeyandOr Ozeri c42981d034 [Refactor][kv_offload] KV Offloading maintainability improvements (#40538) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-04-30 05:55:31 +03:00
Itay EtelisItay EtelisOr OzeriOr Ozeri
 		8a8c9b564e [KV Offload] Per-job store completion for CPU offloading connector (#39186) 
Signed-off-by: Itay Etelis <itay.etelis@ibm.com>
Signed-off-by: Itay Etelis <92247226+Etelis@users.noreply.github.com>
Co-authored-by: Itay Etelis <itay.etelis@ibm.com>
Co-authored-by: Or Ozeri <or@ozery.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-04-29 08:52:55 +03:00
yangrzandRoger Wang de3fe8dc62 [Bugfix] release KV blocks for skipped P-ranks to prevent invalid KV errors and timeouts when P_tp > D_tp and MLA (#40449) 
Signed-off-by: yangruize <yangruize7@163.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2026-04-28 11:38:43 -07:00