iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity
16,857 Commits 3 Branches 0 Tags
5ea76fa89aa2e307f0d9a2e7fc19d13aed65a82f
Commit Graph
1288 Commits
Author SHA1 Message Date
haosdentandWentao Ye 5ea76fa89a [CI] Fix test_lora_with_spec_decode on V2 model runner (#43314) 
Signed-off-by: haosdent <haosdent@gmail.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
 2026-05-22 14:24:18 +08:00
Lanze Liu 39d5fa96a7 [Bugfix] Zero stale is_prefilling in padded CUDA graph rows for Mamba (#41873) 
Signed-off-by: Lanze Liu <lanzetech@gmail.com>
 2026-05-21 15:42:42 -07:00
Francesco Fuscoandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> b730c46352 [Perf] [Hybrid] Fused Triton kernel for GPU-side Mamba state postprocessing (#40172) 
Signed-off-by: Francesco Fusco <ffu@zurich.ibm.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-21 04:50:54 -07:00
haosdentandNickLucche f2d5e3d3ae [CI] Lower granite-4.0-h-tiny gsm8k threshold for Hybrid SSM NixlConnector PD accuracy tests (4 GPUs) (#43186) 
Signed-off-by: haosdent <haosdent@gmail.com>
Signed-off-by: NickLucche <nlucches@redhat.com>
Co-authored-by: NickLucche <nlucches@redhat.com>
 2026-05-20 17:00:24 +00:00
rishitdholakia13andCursor ded871201a [Bug][Structured Outputs] Fix bug that leads to unconstrained generations with structural tags (#42452) 
Signed-off-by: rishitdholakia13 <rishit+github@cohere.com>
Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-05-20 07:08:58 -07:00
Kebe 19cf334207 [Feature] Support manually enabling the cumem allocator (#33648) 
Signed-off-by: Kebe <mail@kebe7jun.com>
 2026-05-20 08:58:30 -04:00
Ronen Schaffer 4f940896a3 [KV Offload] Pass OffloadingSpec instead of VllmConfig to secondary tiers (#43076) 
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
 2026-05-20 03:32:08 +00:00
Benjamin Chislett c628a93a64 [Perf][Bugfix] Update dflash aux layer indexing (#40727) 
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com>
 2026-05-19 20:15:57 -07:00
Nick Hill b82e908b4c [Perf][4/n] Eliminate various GPU<->CPU syncs (#42347) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-05-19 10:35:54 -04:00
zhanqiuhuandNicolò Lucchesi 129019f334 [CI] Add MTP + PD disagg test for Qwen3.5 (#42677) 
Signed-off-by: ZhanqiuHu <zhu@redhat.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
 2026-05-19 11:44:33 +02:00
Yifan Qiao 056bc2e166 [KVConnector][DSV4] HMA support for Mooncake store connector (#42828) 
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
 2026-05-19 01:07:46 -07:00
Qiuyang YueClaude Opus 4.7gemini-code-assist
 		fab07e4d0f [Bugfix][KV Connector] Fix SimpleCPUOffloadScheduler TOCTOU between Phase A and Phase B (#42289) 
Signed-off-by: Qiuyang Yue <yueqiuyang1389@gmail.com>
Co-authored-by: Claude Opus 4.7 <noreply@anthropic.com>
Co-authored-by: gemini-code-assist <noreply@google.com>
 2026-05-18 21:22:33 -07:00
Ronen Schaffer 84747489de Tier offload followup (#42529) 
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
 2026-05-18 19:41:58 +00:00
Netanel Haber 47829b1159 [Bugfix] mamba: run single-token extends as decodes (#42430) 
Signed-off-by: Netanel Haber <58652339+netanel-haber@users.noreply.github.com>
 2026-05-18 15:26:00 +00:00
liranschourOr Ozerimergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		e5417657e5 [KV Connector][Offloading] Flush all pending jobs on last step (#42611) 
Signed-off-by: Liran Schour <lirans@il.ibm.com>
Signed-off-by: liranschour <liranschour@users.noreply.github.com>
Co-authored-by: Or Ozeri <or@ozery.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-18 12:59:42 +00:00
roikoren755 737bfa3a43 [Bugfix][Hybrid][NemotronH] Fix mamba_cache_mode=all + speculative decoding crash (#41233) 
Signed-off-by: Roi Koren <roik@nvidia.com>
 2026-05-18 14:54:00 +03:00
AlecandOpenAI Codex 107210442d [CI] Add NIXL EP import canary (#42567) 
Signed-off-by: Alec Flowers <aflowers@nvidia.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-05-17 19:11:46 -07:00
Zhewen LiZhewen LiClaude Opus 4.7
 		36e74c9ea4 [KV Connector] Support disk offloading in MooncakeStoreConnector (#42689) 
Signed-off-by: Zhewen Li <zhewenli@inferact.ai>
Co-authored-by: Zhewen Li <zhewenli@inferact.ai>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-16 13:34:15 -07:00
Jiangyun Zhu 8a56da3845 [Experimental] Breakable CUDA graph (#42304) 
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
 2026-05-16 22:04:12 +08:00
Yifan Qiao 4b364f810e [Core][DSV4] Skip caching SWA blocks that can never serve a prefix-cache hit (#42258) 
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
 2026-05-15 15:59:18 +08:00
Keyi LiKeyi Limergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		bf610c2f56 [Bugfix] Fix inverted condition causing thinking_token_budget to be silently ignored (#41674) 
Signed-off-by: Keyi Li <likey6688@gmail.com>
Co-authored-by: Keyi Li <likey6688@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-15 12:48:49 +08:00
Matthew Bonanni 9898f94abe [Attention] Remove deprecated MLA prefill arguments (#42555) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
 2026-05-14 10:34:06 -07:00
Baorun (Lauren) Mu a7737cb4f3 [Fix] Misc Fixes in ViT CUDA Graph (#38040) 
Signed-off-by: Baorun Mu <bmu@nvidia.com>
 2026-05-14 23:49:06 +08:00
zhanqiuhuandNicolò Lucchesi 24337fb860 PD disagg with NIXL Connector: GDN support (Qwen3.5) (#41869) 
Signed-off-by: Zhanqiu Hu <zhu@redhat.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
 2026-05-14 16:33:01 +02:00
Hongxin XuClaude Opus 4.6arlenxuJunjie Zhang
 		c7560af424 [RFC] Replace shared-memory routed experts with ModelRunnerOutput transfer and HTTP support (#39568) 
Signed-off-by: xhx1022 <1737006628@qq.com>
Signed-off-by: arlenxu <arlenxu@tencent.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: arlenxu <arlenxu@tencent.com>
Co-authored-by: Junjie Zhang <junj.jay.zhang@gmail.com>
 2026-05-14 14:12:30 +00:00
Martin Hickeygemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>Or Ozeri
 		5bd8c71e79 [kv_offload] Implement reset_cache() for the offloading connector (#41956) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Or Ozeri <or@ozery.com>
 2026-05-14 16:00:10 +03:00
Yongye ZhuandRoger Wang 0d2732dd91 [MLA Attention Backend] Add TOKENSPEED_MLA backend for DSR1/Kimi K25 prefill + decode on Blackwell (#41778) 
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Signed-off-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2026-05-13 23:48:02 -07:00
Siddharth Bedekar f51f6844f9 [Bugfix][Spec Decode] Wire draft_probs into probabilistic draft_model rejection (#40269) 2026-05-13 21:04:03 -04:00
liangel-02 6b5c389ee3 expose flex block size for batch invariant mode (#41252) 
Signed-off-by: Angel Li <liangel@meta.com>
 2026-05-13 14:11:57 -07:00
Michael GoinandClaude Opus 4.6 2f821faeae [Spec Decode] Support hybrid attention models in extract_hidden_states (#39949) 
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
 2026-05-13 10:45:53 -07:00
Wentao Ye e35c0d4c63 [Feature] Support compile mode for batch invariance on SM80 (#42456) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-05-13 11:02:39 -04:00
Ronen Schaffer 11f6b545d4 [kv_offload] Add multi-tier KV cache offloading framework (#40020) 
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
 2026-05-13 17:21:43 +03:00
Ronen Schaffer 79fd1bc7ed [kv_offload] Add req_id to ReqContext for per-request tracking (#42507) 
Signed-off-by: Ronen Schaffer <ronen.schaffer@ibm.com>
 2026-05-13 11:11:10 +00:00
Yifan Qiaoandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 13bf242100 [Feat][KVConnector] Add bind_gpu_block_pool() to KVConnectorBase_V1 (#39654) 
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-13 02:10:29 -07:00
Yifan Qiaoandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 9ce74042d3 [Bugfix][SimpleCPUOffloadBackend] Dedup in-flight CPU offload stores across scheduler steps (#41289) 
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-13 01:53:32 -07:00
Nicolò Lucchesi 71bcd02ef3 [Bugfix][PD] Fix multi-node TP (TP>8) (#39907) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2026-05-12 22:20:57 -07:00
+3
Chao Leileichao.lcivaniumaoshen524Dao007foreverTeng MaPz1116foraxeSkywalker-EPfems14jianzsbaxingpiaochongmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		ebeb09d822 [KV Transfer] Add MooncakeStoreConnector for KV cache offloading via Mooncake distributed store (#40900) 
Signed-off-by: leichao.lc <leichao.lc@antgroup.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: leichao.lc <leichao.lc@antgroup.com>
Co-authored-by: ivanium <yifanqiao@inferact.ai>
Co-authored-by: aoshen524 <aoshen@inferact.ai>
Co-authored-by: Dao007forever <daole@inferact.ai>
Co-authored-by: Teng Ma <sima.mt@alibaba-inc.com>
Co-authored-by: Pz1116 <zpbzpb123123@gmail.com>
Co-authored-by: foraxe <1055696449@qq.com>
Co-authored-by: Skywalker-EP <173423846@qq.com>
Co-authored-by: fems14 <1804143737@qq.com>
Co-authored-by: jianzs <zheng.shoujian@outlook.com>
Co-authored-by: baxingpiaochong <771405853@qq.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-12 16:09:10 -07:00
Nick Hill fe8b42e80c [CI] Fix test_async_scheduling.py flakiness (#42455) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-05-12 21:38:32 +00:00
Giancarlo Delfin fe5b4e0fe7 [Model Runner V2] Apply synthetic mode to probabilistic rejection sampler (#41035) 2026-05-12 13:37:03 -07:00
Yan Ru Pei bcb9c133ba feat(kv-events): emit KV cache metadata (#40984) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-05-12 15:58:48 +00:00
Martin Hickey 418ba8ef14 [kv_offload][BugFix] Fix store deferral (#41945) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
 2026-05-12 18:04:44 +03:00
shanjiaz 6ccb10d794 Added peagle speculators support (#41826) 
Signed-off-by: shanjiaz <zsjwpianpian@gmail.com>
 2026-05-12 07:55:57 -07:00
zhanqiuhuandNickLucche 593d5a4033 [Bugfix] Fix mismatched kernel-per-logical blocks in NIXL HMA transfer (#42097) 
Signed-off-by: ZhanqiuHu <zhu@redhat.com>
Signed-off-by: Zhanqiu Hu <zhu@redhat.com>
Signed-off-by: NickLucche <nlucches@redhat.com>
Co-authored-by: NickLucche <nlucches@redhat.com>
 2026-05-12 15:53:30 +02:00
liuzhenwei 07a40ede19 [UT][XPU] fix test_parallel_sampling due to global random state (#42388) 
Signed-off-by: zhenwei-intel <zhenwei.liu@intel.com>
 2026-05-12 18:03:23 +08:00
Kevin H. Luu 1ff9d33535 [CI] Migrate remaining B200 jobs to b200-k8s with test fixes (#42387) 
Signed-off-by: khluu <khluu000@gmail.com>
 2026-05-12 02:00:37 -07:00
Wentao YeNick Hillmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		d7af6b34d8 [Model Runner V2] Bug fix: logprob dtype int64/int32 issue (#41761) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-11 21:55:43 +00:00
Wentao Ye 0d453e2336 [Perf] Batch invariance with Cutlass fp8 support, 28.9% E2E latency improvement (#40408) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
 2026-05-11 12:20:58 -04:00
Martin Hickeyandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> 4955990f1b [kv_offload] Move FilterReusedOffloadingManager logic to CPUOffloadingManager (#41727) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2026-05-11 18:09:29 +03:00
Nicolò Lucchesi 5672d100ed [KV Connector][NIXL][Bugfix] Fix NIXL handshake failures not honoring kv_load_failure_policy (#40364) 
When NIXL handshake fails (e.g., due to compatibility hash mismatch
between prefill and decode instances), requests fail with "engine dead"
error instead of gracefully falling back to local recomputation as configured
by kv_load_failure_policy='recompute'.

Signed-off-by: NickLucche <nlucches@redhat.com>
 2026-05-11 09:37:21 +00:00
Nicolò Lucchesi 770e9bd6b3 [Nixl][PD] Lease renewal TTL KV blocks on P (#41383) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2026-05-11 09:27:30 +00:00