iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
16,603 Commits 3 Branches 0 Tags
ca7e4546da4cb716df671dfcf26e459dd3dfd9f0
Commit Graph
1260 Commits
Author SHA1 Message Date
liangel-02 6b5c389ee3 expose flex block size for batch invariant mode (#41252) 
Signed-off-by: Angel Li <liangel@meta.com>
 2026-05-13 14:11:57 -07:00
Michael GoinandClaude Opus 4.6 2f821faeae [Spec Decode] Support hybrid attention models in extract_hidden_states (#39949) 
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-13 10:45:53 -07:00
Wentao Ye e35c0d4c63 [Feature] Support compile mode for batch invariance on SM80 (#42456) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-05-13 11:02:39 -04:00
Ronen Schaffer 11f6b545d4 [kv_offload] Add multi-tier KV cache offloading framework (#40020) 
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
 2026-05-13 17:21:43 +03:00
Ronen Schaffer 79fd1bc7ed [kv_offload] Add req_id to ReqContext for per-request tracking (#42507) 
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
 2026-05-13 11:11:10 +00:00
Yifan Qiaoandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 13bf242100 [Feat][KVConnector] Add bind_gpu_block_pool() to KVConnectorBase_V1 (#39654) 
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-13 02:10:29 -07:00
Yifan Qiaoandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 9ce74042d3 [Bugfix][SimpleCPUOffloadBackend] Dedup in-flight CPU offload stores across scheduler steps (#41289) 
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-13 01:53:32 -07:00
Nicolò Lucchesi 71bcd02ef3 [Bugfix][PD] Fix multi-node TP (TP>8) (#39907) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2026-05-12 22:20:57 -07:00
+3
Chao Leileichao.lcivaniumaoshen524Dao007foreverTeng MaPz1116foraxeSkywalker-EPfems14jianzsbaxingpiaochongmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		ebeb09d822 [KV Transfer] Add MooncakeStoreConnector for KV cache offloading via Mooncake distributed store (#40900) 
Signed-off-by: leichao.lc <leichao.lc@antgroup.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: leichao.lc <leichao.lc@antgroup.com>
Co-authored-by: ivanium <yifanqiao@inferact.ai>
Co-authored-by: aoshen524 <aoshen@inferact.ai>
Co-authored-by: Dao007forever <daole@inferact.ai>
Co-authored-by: Teng Ma <sima.mt@alibaba-inc.com>
Co-authored-by: Pz1116 <zpbzpb123123@gmail.com>
Co-authored-by: foraxe <1055696449@qq.com>
Co-authored-by: Skywalker-EP <173423846@qq.com>
Co-authored-by: fems14 <1804143737@qq.com>
Co-authored-by: jianzs <zheng.shoujian@outlook.com>
Co-authored-by: baxingpiaochong <771405853@qq.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-12 16:09:10 -07:00
Nick Hill fe8b42e80c [CI] Fix test_async_scheduling.py flakiness (#42455) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-05-12 21:38:32 +00:00
Giancarlo Delfin fe5b4e0fe7 [Model Runner V2] Apply synthetic mode to probabilistic rejection sampler (#41035) 2026-05-12 13:37:03 -07:00
Yan Ru Pei bcb9c133ba feat(kv-events): emit KV cache metadata (#40984) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-05-12 15:58:48 +00:00
Martin Hickey 418ba8ef14 [kv_offload][BugFix] Fix store deferral (#41945) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
 2026-05-12 18:04:44 +03:00
shanjiaz 6ccb10d794 Added peagle speculators support (#41826) 
Signed-off-by: shanjiaz <zsjwpianpian@gmail.com>
 2026-05-12 07:55:57 -07:00
zhanqiuhuandNickLucche 593d5a4033 [Bugfix] Fix mismatched kernel-per-logical blocks in NIXL HMA transfer (#42097) 
Signed-off-by: ZhanqiuHu <zhu@redhat.com>
Signed-off-by: Zhanqiu Hu <zhu@redhat.com>
Signed-off-by: NickLucche <nlucches@redhat.com>
Co-authored-by: NickLucche <nlucches@redhat.com>
 2026-05-12 15:53:30 +02:00
liuzhenwei 07a40ede19 [UT][XPU] fix test_parallel_sampling due to global random state (#42388) 
Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com>
 2026-05-12 18:03:23 +08:00
Kevin H. Luu 1ff9d33535 [CI] Migrate remaining B200 jobs to b200-k8s with test fixes (#42387) 
Signed-off-by: khluu <khluu000@gmail.com>
 2026-05-12 02:00:37 -07:00
Wentao YeNick Hillmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		d7af6b34d8 [Model Runner V2] Bug fix: logprob dtype int64/int32 issue (#41761) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-11 21:55:43 +00:00
Wentao Ye 0d453e2336 [Perf] Batch invariance with Cutlass fp8 support, 28.9% E2E latency improvement (#40408) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
 2026-05-11 12:20:58 -04:00
Martin Hickeyandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> 4955990f1b [kv_offload] Move FilterReusedOffloadingManager logic to CPUOffloadingManager (#41727) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-05-11 18:09:29 +03:00
Nicolò Lucchesi 5672d100ed [KV Connector][NIXL][Bugfix] Fix NIXL handshake failures not honoring kv_load_failure_policy (#40364) 
When NIXL handshake fails (e.g., due to compatibility hash mismatch
between prefill and decode instances), requests fail with "engine dead"
error instead of gracefully falling back to local recomputation as configured
by kv_load_failure_policy='recompute'.

Signed-off-by: NickLucche <nlucches@redhat.com>
 2026-05-11 09:37:21 +00:00
Nicolò Lucchesi 770e9bd6b3 [Nixl][PD] Lease renewal TTL KV blocks on P (#41383) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2026-05-11 09:27:30 +00:00
shaharmor98andmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> f9f770ca0b fix nixl side-channel host selection (#41806) 
Signed-off-by: Shahar Mor <smor@nvidia.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-11 07:40:37 +00:00
wangxiyuan 5536fc0c01 [Misc] Replace mamba_type string literals with MambaAttentionBackendEnum (#41188) 
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
 2026-05-11 03:59:36 +00:00
Ronen Schaffer e175192d33 [KV Offload] Pass ReqContext to touch(), complete_load(), and complete_store() (#41366) 
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
 2026-05-10 15:09:25 +03:00
Ethan Feng a2c9d548d7 [Docs] Fix broken local links (#42160) 
Signed-off-by: Ethan Feng <ethan.fengch@gmail.com>
 2026-05-10 01:15:38 -07:00
Andreas Karatzas f2840120f6 [ROCm][CI] Fix NIXL spec-decode acceptance startup and diagnostics (#41313) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
 2026-05-10 14:50:16 +08:00
Dao007forever 3f5bd482f5 [Bugfix][KV Transfer][NIXL] Notify P node on pre-admission rejection to free stranded KV blocks (#41269) 2026-05-09 22:52:09 -07:00
Wentao Ye ea0e501bb1 [KV Connector] Remove compat support for pre-v0.12.0 constructor signatures without KVCacheConfig (#39832) 
The v0.12.0 release contained initial support for HMA in KV Connectors. As part
of these changes, a KVCacheConfig argument was added to KV connector
constructors. Backwards compatibility support for out-of-tree connectors was
included in this change, with a very prominent warning. See #25712 and #27887.

Since the warning has been around for over 5 months, we can safely remove
the support of it.

Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-05-09 23:39:46 +00:00
Nicolò LucchesiandZhanqiuHu 171d59ae8d [Bugfix][PD] Fix DSv4 Disaggregated (#41957) 
Signed-off-by: NickLucche <nlucches@redhat.com>
Co-authored-by: ZhanqiuHu <zhu@redhat.com>
 2026-05-09 16:48:24 +00:00
David Zhenggemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>Andreas KaratzasKunshang Ji
 		df2636a9d8 [Bugfix] Fix LOGITPROC_SOURCE_ENTRYPOINT test to use spawn-compatible dist-info registration for XPU/ROCm (#42040) 
Signed-off-by: dqzhengAP <dqzheng1996@gmail.com>
Signed-off-by: David Zheng <153074367+dzhengAP@users.noreply.github.com>
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Andreas Karatzas <akaratza@amd.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
 2026-05-09 12:32:04 +08:00
Lucas Wilkinson b1728c1e66 [Attention][Cleanup] Remove tree attention (#42121) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
 2026-05-08 18:36:19 -07:00
Roy Wang 60851b1d22 [Bugfix][KV Transfer] Reject NixlConnector + expandable_segments:True (#41237) 2026-05-08 16:47:33 -07:00
David Zheng 1acd67a795 [Bugfix] Fix XPU/ROCm compatibility in spawn_new_process_for_each_test (#41895) 
Signed-off-by: dqzhengAP <dqzheng1996@gmail.com>
 2026-05-08 00:47:22 -04:00
Hongxia YangandTJian 20cac26b19 [ROCm] Enable SimpleCPUOffloadConnector on ROCm (#40549) 
Signed-off-by: Hongxia Yang <hongxia.yang@amd.com>
Co-authored-by: TJian <tunjian.tan@embeddedllm.com>
 2026-05-06 20:52:02 -07:00
Jing WangCopilotWentao Yemergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		27702f6d08 [Bugfix] Fix token loss in PP mode which causes degraded accuracy (#41133) 
Signed-off-by: Jing Wang <jingwang96@qq.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-06 14:07:32 -04:00
David Zheng ee38750a75 [Bugfix] Fix spawn_new_process_for_each_test silently swallowing test failures (#41423) 
Signed-off-by: dqzhengAP <dqzheng1996@gmail.com>
 2026-05-06 11:17:15 -04:00
Luciano MartinsandLuciano Martins 27e0057aed [Spec Decode] Add Gemma4 MTP speculative decoding support (#41745) 
Signed-off-by: Luciano Martins <lucianommartins@users.noreply.github.com>
Co-authored-by: Luciano Martins <lucianommartins@users.noreply.github.com>
 2026-05-06 22:39:29 +08:00
Ronen Schaffer f39bcf1e30 [KV Offload] Return None from lookup() for in-flight blocks (#41795) 
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
 2026-05-06 17:31:21 +03:00
zhanqiuhuandNickLucche df8e63f4ed nixl refactor: new transfer design (#40731) 
Signed-off-by: ZhanqiuHu <zhu@redhat.com>
Signed-off-by: NickLucche <nlucches@redhat.com>
Co-authored-by: NickLucche <nlucches@redhat.com>
 2026-05-06 06:16:25 -07:00
wang.yuqiandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> 51c1ee9b7c [Examples] Resettle Disaggregated examples. (#40759) 
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
Signed-off-by: wang.yuqi <noooop@126.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-05-06 01:20:38 -07:00
wi-adam b53c507bc9 [Bugfix] Skip PP sampled-token receive on last rank during async scheduling (#40749) 
Signed-off-by: Adam Winstanley <adam@winstanley.industries>
 2026-05-06 05:31:14 +00:00
Matthew Bonanni 01b9b5af67 [Attention] Minor refactor: layer takes ownership of the MLA prefill backend (#41744) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
 2026-05-05 23:22:41 +00:00
Lanze Liu 79246b5ea6 [Spec Decode] Fix max_model_len logging in speculative config for draft model (#41571) 
Signed-off-by: Lanze Liu <lanzetech@gmail.com>
 2026-05-05 21:56:06 +00:00
Zhewen LiandZhewen Li bee126165f [P/D][Mooncake] Add KVConnectorStats for transfer observability (#40414) 
Signed-off-by: Zhewen Li <zhewenli@inferact.ai>
Co-authored-by: Zhewen Li <zhewenli@inferact.ai>
 2026-05-05 02:17:38 -07:00
Wentao Ye 577b9623e6 [Bug] Fix status update address for non-MOE model within external dp mode (#40839) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-05-04 16:37:16 -07:00
Andreas Karatzas 1cb0838721 [ROCm][CI] Fix MLA prefill scale for DeepSeek GSM8K (#41569) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
 2026-05-04 16:32:55 -07:00
Joachim StudniaandClaude Opus 4.6 422dd02598 [bugfix] Fix prompt logprobs on request eviction during chunked prefill (#41411) 
Signed-off-by: Joachim Studnia <joachim@mistral.ai>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-04 11:46:00 -07:00
Andreas Karatzas 01d4d1ad37 [ROCm][CI] Align spec decode logprob test prefill settings (#41335) 
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
 2026-05-04 04:33:29 +00:00
David OyDavid OyClaudeVadim Gimpelson
 		66dfee7121 [Bugfix] Fix degenerate KV cache stride causing TMA cudaErrorIllegalInstruction (#40737) 
Signed-off-by: David Oy <david@baseten.co>
Signed-off-by: David Oy <58150256+the-david-oy@users.noreply.github.com>
Signed-off-by: David Oy <david.oy@baseten.co>
Co-authored-by: David Oy <david@baseten.co>
Co-authored-by: Claude <claude@anthropic.com>
Co-authored-by: Vadim Gimpelson <156319763+vadiklyutiy@users.noreply.github.com>
 2026-05-03 23:52:18 +00:00