iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
22,991 Commits 3 Branches 0 Tags
c19c263ac8ea3fd6e1d2d689f0e4e725855d89cf
Commit Graph
357 Commits
Author SHA1 Message Date
Dakai AnClaude Opus 5.5aoshen02
 		de8291e24f [Core] Add lifecycle hooks for worker extensions (#59463) 
Signed-off-by: Dakai An <dakaian108@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
 2026-10-13 21:51:05 +08:00
Dakai AnYifan QiaoClaude Opus 5.5
 		df6c7ad031 [Core][Refactor] Move KV cache planning logic to kv_cache_planning.py (#61183) 
Signed-off-by: Dakai An <dakaian108@gmail.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-13 03:05:39 +00:00
Wentao Ye d0948f754d [Bug] Honor spec decode draft model config (#61240) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-10-12 15:13:23 -04:00
Woosuk KwonandClaude Opus 5.5 e1a5086256 [Attention] Remove cascade attention (#61126) 
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-12 10:41:07 -07:00
Ilya Markov a281ad32f0 [EPLB] Add EPLB support to dspark draft models (DeepSeekV4) (#57549) 
Signed-off-by: Markov Ilya <markovilya197@gmail.com>
 2026-10-12 10:03:09 -04:00
Giancarlo DelfinandRishi Puri 36f3972812 [Spec Decode] Skip metadata build for dflash/dspark when all model backends support it (#60692) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: Rishi Puri <20074092+puririshi98@users.noreply.github.com>
 2026-10-12 00:09:10 -07:00
dingdangmaoandCodex 124b89d5b7 [Bugfix][RecoverSSM] Correct aligned state indices at exact block boundaries (#59962) 
Signed-off-by: apex-mochen <2756823972@qq.com>
Co-authored-by: Codex <noreply@openai.com>
 2026-10-12 02:08:14 +00:00
Nick Hill c059bee812 [MRV2] Switch CI tests still running on MRV1 unnecessarily (#61014) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-10-11 09:57:07 -07:00
aoshen02andCodex 239a3fa3a8 [Sleep] Retain draft-private weights with existing parameter selector (#61046) 
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Codex <noreply@openai.com>
 2026-10-11 23:30:56 +08:00
aoshen02 bd5bbce6e4 [RL] Release weight-transfer communicators without stopping inference (#61060) 
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Signed-off-by: fh3 <a@b>
 2026-10-11 21:15:14 +08:00
Nick Hill f8471164bf [MRV2] Detatch weight-reload logic from MRV1 (#60937) 2026-10-10 21:15:05 -07:00
Matthew BonanniandClaude Opus 5.5 d5f356dae0 [HiSparse] Own GPU residency instead of editing core block-table rows (#60647) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 21:25:44 -07:00
DimensionSTPandOpenAI Codex 10cc2f6ae2 [Bugfix] Bound GDN speculative decode query widths (#55504) 
Signed-off-by: DimensionSTP <65501090+DimensionSTP@users.noreply.github.com>
Co-authored-by: OpenAI Codex <noreply@openai.com>
 2026-10-10 11:00:16 +08:00
Nick Hill 7d1d8660d6 [MRV2] Miscellaneous code structure updates (#60754) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-10-09 17:58:09 -07:00
Matthew BonanniandClaude Opus 5.5 4c43254d52 [HiSparse] Report host-pool block residency as HiSparse metrics (#59485) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-09 16:45:34 -06:00
Nick HillvllmellmClaude Opus 5.5
 		ed7ae705a1 [Bugfix][MRV2] Use int64 index mappings to fix row-offset overflow (#60736) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: vllmellm <vllm.ellm@embeddedllm.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-10-09 14:38:11 -07:00
Dakai AnYifan QiaoOpenAI CodexOpenAI CodexLucas WilkinsonClaude Opus 5.5Lucas Wilkinson
 		02e55d2af3 [KV Cache] GLM-5.3-Flash: use the generic packed KV layout; CircularBufferSpec tail with speculative ring slots (#57169) 
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Signed-off-by: Dakai An <dakaian108@gmail.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: OpenAI Codex <noreply@openai.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
 2026-10-09 15:48:37 -04:00
zzaebokandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 702ce313f4 [Perf][Multimodal] Sort multimodal encoder inputs by modality (#60196) 
Signed-off-by: Jaebok Lee <jaebok9541@naver.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-10-09 10:55:36 +08:00
Mia OjedaandLucas Wilkinson 4611c2c3d2 [BugFix][DiffusionGemma][CPU] Copy async output snapshots instead of aliasing reused buffers (#59829) 
Signed-off-by: ojeda-e <eojeda@league.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
 2026-10-08 19:16:51 -07:00
Nishant RevurandClaude 581692cc0f [Bugfix][Spec Decode] Fix multimodal embedding merge in Llama 4 and Mistral Large 3 EAGLE drafters (#57633) 
Signed-off-by: Nishant Revur <kumrfni@amazon.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-10-09 01:44:50 +00:00
Woosuk Kwon 194bd72be7 [Compile] Remove torch.compile-based sequence parallelism and AsyncTP (#60517) 2026-10-08 08:28:23 -07:00
Matthew BonanniandClaude Opus 5.5 4ed4c4433b [Bugfix][HiSparse] Drop the unaccounted, write-only prefill mirror staging buffer (#60429) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-08 14:00:49 +00:00
Giancarlo Delfin 6ee9be6b3c [Spec Decode] Rename AR speculators to StandaloneAR / TargetDependentAR (#60335) 
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
 2026-10-07 14:01:59 -04:00
pondzikkClaude Fable 5.1zjy0516ptorsten
 		3d6853b32c [Bugfix] Seed hybrid mamba state index with mamba_block_size on prefix-cache hits (#55601) 
Signed-off-by: pondzikk <pondzik@gmail.com>
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: ptorsten <1652174+ptorsten@users.noreply.github.com>
 2026-10-07 17:09:50 +00:00
viktorkouClaude Fable 5.1Isotr0py
 		c83935b300 [Bugfix][Model Loader] Support reload_weights with runai_streamer load format (#58124) 
Signed-off-by: Viktor Koukouliev <vkoukouliev@nvidia.com>
Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com>
Co-authored-by: Isotr0py <Isotr0py@outlook.com>
 2026-10-07 14:04:35 +00:00
Jiangyun ZhuKarl0007Yongqin Wang
 		8a26869bca [Bugfix] Index Mamba align state copies by request slot in MRV2 (#60210) 
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: Karl0007 <44745209+Karl0007@users.noreply.github.com>
Co-authored-by: Yongqin Wang <yongqinwang@roblox.com>
 2026-10-07 02:32:46 +00:00
Miłosz Grunwald 703c29179c [TEST][XPU][CI] Generalize model runner prefill tail classification tests to all platforms (#60201) 
Signed-off-by: microslaw <milosz.grunwald@intel.com>
 2026-10-07 08:59:20 +08:00
PatchyTISPatchouliTaisaNick HillXuanan ChenGiancarlo Delfin
 		819852df6b [ModelRunner V2] Speculative Decoding NGram GPU Implementations (#40704) 
Signed-off-by: PatchouliTaisa <patchychen@tencent.com>
Signed-off-by: PatchouliTaisa <pyramkar@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Signed-off-by: Xuanan Chen <xuananchenc@nvidia.com>
Co-authored-by: PatchouliTaisa <patchychen@tencent.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Xuanan Chen <xuananchenc@nvidia.com>
Co-authored-by: Giancarlo Delfin <32987265+TheEpicDolphin@users.noreply.github.com>
 2026-10-06 13:32:59 -07:00
Cameron QuiliciCam QuiliciNick HillYifan Qiao
 		e37e51dd24 [Metrics] Expose cached prompt tokens by cache tier (#56318) 
Signed-off-by: Cam Quilici <cjquilici@gmail.com>
Signed-off-by: Cam Quilici <cameron@semianalysis.com>
Co-authored-by: Cam Quilici <cameron@semianalysis.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
 2026-10-06 14:52:05 -04:00
czhu-cohereandaoshen02 6fddc9b21e [Profiler] Add per-session profiling controls to Python and Rust frontends (#57875) 
Signed-off-by: czhu <conway.zhu@cohere.com>
Signed-off-by: Conway Zhu <conway.zhu@cohere.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
 2026-10-06 05:21:19 +00:00
Juntian LiuClaude Opus 5.5mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		eb2901b735 [Perf][DSv4.1] Run decoder replay layers in CUDA graphs and trim in PIECEWISE graphs (#59532) 
Signed-off-by: Juntian Liu <Juntianl777@gmail.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-10-05 22:11:45 -07:00
Tianyu Guo 30ab428235 [Bugfix][Attention] Restrict FlashInfer sparse MLA FULL graphs to decode (#59751) 
Signed-off-by: Tianyu Guo <guoty@inferact.ai>
 2026-10-06 04:13:23 +00:00
Vadim GimpelsonCursor Agentmergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		0eac152707 [Bugfix] Route every speculation-capable row through the speculative path so recurrent state stays consistent (#56531) 
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-10-06 03:03:44 +04:00
Kevin H. LuuandClaude Opus 5.5 89a00995c9 [CI] Read each profiler round's trace as it stops in test_gpu_profiler (#59769) 
Signed-off-by: khluu <khluu000@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-05 15:25:11 -07:00
Robert ShawClaude Opus 5.5Matthew Bonanni
 		2e3154aa18 [HiSparse] Log steady-state max concurrency and expose host-tier utilization gauges (#58949) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
 2026-10-05 14:20:32 -07:00
Matthew BonanniandClaude Opus 5.5 8542c3eff5 [CI] Fix test_mixed_warmup_gate after #57053 (#60116) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-05 21:17:00 +00:00
Wentao Ye 74c5cbcd7b [Bugfix][MRV2][Spec Decode] Honor dynamic K in autoregressive speculators, 1.2~1.3x kernel perf improvement (#57053) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-10-05 14:54:11 -04:00
Raphaël RiallandandSimon Veitner c32495d58f [watermark] compatibility validation (#56801) 
Signed-off-by: Raphael Rialland <raphael.rialland@mistral.ai>
Signed-off-by: Simon Veitner <sveitner@redhat.com>
Co-authored-by: Simon Veitner <sveitner@redhat.com>
 2026-10-05 14:37:13 -04:00
Evgeny SavinovCodexTomas Ruiz
 		877ddcc456 [Bugfix][Spec Decode] Reserve the bonus KV slot for fill-in DSpark (#59105) 
Signed-off-by: Evgeny Savinov <notime.sea@gmail.com>
Co-authored-by: Codex <noreply@openai.com>
Co-authored-by: Tomas Ruiz <tomas.ruiz.te@gmail.com>
 2026-10-05 17:35:51 +00:00
Thiago Mota MartinsandFangzhou Ai 92044241a0 [Bugfix][ROCm] Preserve config during GPU memory profiling (#58014) 
Co-authored-by: Fangzhou Ai <31551580+Fangzhou-Ai@users.noreply.github.com>
 2026-10-04 22:28:33 -07:00
DivyClaude Opus 5.5Wentao Ye
 		bd42276e1e [Misc][Platform] Check aligned KV block sizes against every attention backend (#58457) 
Signed-off-by: Divy <divy@coralbricks.ai>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
 2026-10-04 11:13:20 -04:00
Wentao Ye 8bdd8d847d [GLM 5.3 Perf] Enable fused multi-step decode, 13.3% E2E throughput improvement for concurrency 1 (#57443) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-10-02 20:05:27 -04:00
Chaoyinglinchaoying6OpenAI Codex
 		592c6f3fbc [Bugfix][Pooling] Handle multimodal cache misses without crashing (#58975) 
Signed-off-by: linchaoying6 <linchaoying6@jd.com>
Co-authored-by: linchaoying6 <linchaoying6@jd.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-10-02 10:14:33 -04:00
Lucas WilkinsonClaude Opus 5.5Matthew Bonanni
 		4056c8ac1f [Bugfix][HiSparse] Fix MTP acceptance collapse under FULL graphs with a saturated GPU pool (#59309) 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
 2026-10-01 23:04:21 -07:00
Matthew BonanniClaude Opus 5.5Lucas Wilkinson
 		f7999d2e44 [Bugfix][HiSparse] Size the KV cache from the groups HiSparse allocates (#59450) 
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
 2026-10-02 04:11:55 +00:00
Qiu ChunshuoandLucas Wilkinson 8486f1ebfe [Perf][PCP] Shard decode requests across PCP ranks (#52162) 
Signed-off-by: QiuChunshuo <qiuchunshuo@huawei.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
 2026-10-01 23:50:13 -04:00
roiponyroi.ponyTaneem Ibrahim
 		e9be5323f0 [Perf] Integrate flash-maxsim Triton kernels for late-interaction scoring (#40337) 
Signed-off-by: roi.pony <roi.pony@ibm.com>
Co-authored-by: roi.pony <roi.pony@ibm.com>
Co-authored-by: Taneem Ibrahim <taneem.ibrahim@gmail.com>
 2026-10-01 14:06:43 -04:00
Jiangyun ZhuClaude Opus 5.5Thomas Parnell
 		cb6458c5ce [Core] Remove deprecated mamba_cache_mode "all" (#58997) 
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Co-authored-by: Thomas Parnell <tpa@zurich.ibm.com>
 2026-10-01 02:59:37 -07:00
vx120Cuiping Chengaoshen02Claude Opus 5.5
 		70be6aae19 [Feature][Spec Decode] Support sampling mask replay for MRV2 MTP (#59359) 
Signed-off-by: Cuiping Cheng <chengcuipingswu@163.com>
Signed-off-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Cuiping Cheng <chengcuipingswu@163.com>
Co-authored-by: aoshen02 <aoshen@inferact.ai>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-01 13:51:39 +08:00
LostFox11LostFox11Nick Hill
 		7314360c9e [Perf][PP] Skip sampled-token broadcasts whose requests leave the engine (#58542) 
Signed-off-by: LostFox11 <wangziyue17@huawei.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: LostFox11 <wangziyue17@huawei.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-09-30 19:18:55 -07:00