iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
18,130 Commits 3 Branches 0 Tags
68ee8300a047db78fb52bac477daaaac7be11216
Commit Graph
256 Commits
Author SHA1 Message Date
Nick Hill 658b54efe4 [ModelRunner V2] Update scheduler tests to cover MRV2 paths (#46771) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-06-26 09:36:31 -07:00
Yiliu DongNick HillJade ZhengGiancarlo DelfinZijing Liu
 		d490b98162 [Core] Avoid mixed length specdec batches via padding (#45237) 
Signed-off-by: Yiliu Dong <91178480+qianlihuang@users.noreply.github.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Jade Zheng <zheng.shoujian@outlook.com>
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: Zijing Liu <liuzijing2014@gmail.com>
 2026-06-25 08:34:44 -07:00
Lucas Wilkinson e7df232288 [KV Offload] Gate packed HMA KV cache on cross-layer config (#46252) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
 2026-06-24 11:55:30 -04:00
Nick Hillandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> 6af0559ddb [Core][DP] Throttle prefills based on local prefill work (#46532) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-24 02:27:12 +00:00
Dao007foreverandClaude Opus 4.8 430a95ae3a [v1][kvcache] Honor prefix-cache retention interval for Mamba/linear attention (#45845) 
Signed-off-by: Dao Le <daole@inferact.ai>
Signed-off-by: Dao Le <Dao007forever@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-22 19:51:11 -07:00
Nick HillClaude Opus 4.8Roger Wang
 		3e6529cc0e [Bugfix][Spec Decode] Fix EAGLE drafter multimodal encoder cache misses (#46315) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
 2026-06-22 18:14:02 +00:00
Jiangyun ZhuandYifan Qiao 6bc6f2d86d [1/N][Core] add partial prefix cache primitives (#45939) 
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
 2026-06-21 23:43:10 -07:00
Alex SteinerClaude Opus 4.8Giancarlo DelfinYifan Qiao
 		2cac89f9da [Spec Decode] Support mixed KV page sizes for DFlash (#45181) 
Signed-off-by: Alex Steiner <asteiner@nvidia.com>
Signed-off-by: Giancarlo Delfin <gdelfin@inferact.ai>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
 2026-06-21 22:45:14 +08:00
amanClaudeNick Hill
 		6e919960af [Perf] Skip/shrink all_token_ids copy in scheduler for non-async and V2 runner (#45840) 
Signed-off-by: amanchugh89 <amanchugh.89@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-06-20 22:36:57 +00:00
Lucas WilkinsonOpenAI CodexTyler Michael Smith
 		cc22621b51 [KV Offload] Support packed HMA KV cache layout (#46205) 
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
 2026-06-20 21:19:40 +00:00
Tyler Michael SmithClaude Opus 4.6Matthew BonanniLucas WilkinsonLucas WilkinsonOpenAI Codex
 		01192139bf [DSv4] Pack KV caches into contiguous per-block allocations for DeepSeek V4 (#44577) 
Signed-off-by: Tyler Michael Smith <tlrmchlsmth@gmail.com>
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Matthew Bonanni <mbonanni@redhat.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
 2026-06-19 12:55:42 -04:00
Stan Wozniak 520828789c Apply LRU policy only to proper cache entries (#42656) 
Signed-off-by: Stanislaw Wozniak <stw@zurich.ibm.com>
 2026-06-16 21:49:15 +00:00
Nick Hill d8d95998dc [Core] Add prefill step cadence for better non-PD DP balancing (#44558) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-06-16 13:17:18 -07:00
llxNick HillJiangyun Zhu
 		d467a2a7f2 [Bugfix] Defer block freeing until in-flight steps finish under async scheduling + PD KV consumer (#45357) 
Signed-off-by: llx-08 <2596671364@qq.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com>
 2026-06-15 21:36:09 +00:00
Saddss 588db18362 [Bugfix] Two-phase KV allocation for cross-group prefix cache hits (supersedes #33775) (#44409) 
Signed-off-by: Saddss <2872669061@qq.com>
 2026-06-15 22:39:59 +08:00
Nick Hill 1a369783e9 [BugFix] Avoid prematurely freeing cached mm encoder outputs (#45347) 
Signed-off-by: Roger Wang <hey@rogerw.io>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-06-12 15:39:40 -07:00
Srinivas Krovvidiandsrinivas_oo7 9ff278b1d2 [Core][KV Connector] fix scheduler KV connector stats aggregation (#43877) 
Fixes scheduler-side KV connector stats collection so that:

1. update_connector_output() runs before scheduler-side stats are collected.
2. worker-side and scheduler-side KV connector stats are aggregated when both are present.
3. scheduler-only KV connector stats are still emitted when no worker-side stats exist.

Signed-off-by: srinivas_oo7 <sklinkedin0120@gmail.com>
Co-authored-by: srinivas_oo7 <sklinkedin0120@gmail.com>
 2026-06-12 14:51:55 +00:00
Ethan Feng b7f9b6ab27 [Metrics] Add group-aware KV cache capacity to vllm:cache_config_info (#42206) 
The startup log already reports the correct group-aware KV cache capacity for
hybrid models, but Prometheus did not expose matching info in 'vllm:cache_config_info`.

This PR adds kv_cache_size_tokens and kv_cache_max_concurrency.

Signed-off-by: Ethan Feng <ethan.fengch@gmail.com>
 2026-06-12 11:49:44 +00:00
Nick HillandClaude Opus 4.8 4085ff7cb4 [Core] Add kvcache watermark to reduce preemptions (#44594) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-06-11 08:27:31 -07:00
Stan Wozniak dc66e01a70 [Hybrid] Marconi-style admission policy for hybrid cache (#37898) 
Signed-off-by: Stanislaw Wozniak <stw@zurich.ibm.com>
 2026-06-10 10:03:13 -07:00
KANG QihanOpenAI CodexNick Hill
 		303916e93d [Bugfix]: Fix assertion in MambaManager.allocate_slots() (#39562) 
Signed-off-by: Holworth <kangqihan17@mails.ucas.ac.cn>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-06-08 00:34:37 -04:00
Nick Hill 3b3d5287fa [BugFix] Resolve multiple async kv load deadlock (#44560) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-06-06 23:05:47 +00:00
Wei ZhaoandYifan Qiao a6183563b6 [Prefix Caching] DeepSeekv4 - Support selective prefix-cache retention for sliding-window KV cache (#43447) 
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
 2026-06-04 00:48:31 -07:00
Mengqing Caozjy0516mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 		0c6631f02a [KVCache] Support Pluggable KVCacheSpec (#37505) 
Signed-off-by: MengqingCao <cmq0113@163.com>
Signed-off-by: Mengqing Cao <cmq0113@163.com>
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-03 09:05:16 -07:00
Andy Lo 95b1615ec9 [Perf] Improve multimodal item handling from O(n) to O(log n) per step (#44212) 
Signed-off-by: Andy Lo <andy@mistral.ai>
 2026-06-03 11:00:26 +00:00
Yifan Qiao e9e08c49b9 [Bugfix] Cache the EAGLE/MTP lookahead block in the SWA prefix-cache mask (#44082) 
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
 2026-06-02 12:21:07 -07:00
omerpaz95andmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> d247a9dc13 [EC Connector] Non blocking EC Connector lookup (#41627) 
Signed-off-by: omerpaz95 <omerpaz95@gmail.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-06-02 08:48:25 +00:00
Yifan Qiao 7c37096620 [Core][Refactor]: thread scheduler_block_size into KVCacheManager and KVCacheCoordinator (#44165) 
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
 2026-06-02 01:14:44 -07:00
aoshen02aoshen524Claude Opus 4.7
 		1fc2cee50a [KVConnector][Mooncake] Wire reset_cache cascade end-to-end (#42694) 
Signed-off-by: aoshen524 <aoshen524@gmail.com>
Signed-off-by: Ao Shen <aoshen@inferact.ai>
Co-authored-by: aoshen524 <aoshen524@gmail.com>
Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-26 20:52:35 -07:00
Gabriel Wu 82536acc54 Keep scheduler alive for delayed KV connector frees (#43433) 
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com>
 2026-05-23 06:23:32 +00:00
Yifan Qiao 4b364f810e [Core][DSV4] Skip caching SWA blocks that can never serve a prefix-cache hit (#42258) 
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
 2026-05-15 15:59:18 +08:00
Hongxin XuClaude Opus 4.6arlenxuJunjie Zhang
 		c7560af424 [RFC] Replace shared-memory routed experts with ModelRunnerOutput transfer and HTTP support (#39568) 
Signed-off-by: xhx1022 <1737006628@qq.com>
Signed-off-by: arlenxu <arlenxu@tencent.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: arlenxu <arlenxu@tencent.com>
Co-authored-by: Junjie Zhang <junj.jay.zhang@gmail.com>
 2026-05-14 14:12:30 +00:00
Yan Ru Pei bcb9c133ba feat(kv-events): emit KV cache metadata (#40984) 
Signed-off-by: PeaBrane <yanrpei@gmail.com>
 2026-05-12 15:58:48 +00:00
Nick Hill dd5506a157 [Core] Simplify handling of scheduler_reserve_full_isl option (#41064) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-04-30 18:10:00 -07:00
Wei Zhao 0ff1bf9bb1 [Bugfix] Fix failure to allocate KV blocks error (#41282) 
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com>
 2026-04-29 18:44:07 -07:00
Nick Hill e68fa1b90a [Core] Account for num_gpu_blocks_override in max_model_len checks (#41069) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-04-28 15:44:09 -07:00
Dao007foreverClaudeNick Hill
 		7b1bc0a3eb [Bugfix] Cap SWA/chunked-local runtime admission to startup pool-sizing bound (#40946) 
Signed-off-by: Dao Le <Dao007forever@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-04-27 04:33:13 +00:00
+7
Yifan QiaoYongye ZhuYongye ZhuSimon MoBugen ZhaoGiancarlo DelfinJee Jee LiNick HillRoger WangRoy WangWoosuk KwonyoukaichaoZhewen LiZijing LiukhluuqizixiZhewen Li
 		4d51588e23 [Feat] DeepSeek V4 Rebased (#40860) 
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai>
Signed-off-by: qizixi <zixi@inferact.ai>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com>
Co-authored-by: Yongye Zhu <yongye@inferact.ai>
Co-authored-by: Simon Mo <simon@inferact.ai>
Co-authored-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Roy Wang <yasong.wang@inferact.ai>
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai>
Co-authored-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: Zhewen Li <jerven.vllm@gmail.com>
Co-authored-by: Zijing Liu <liuzijing2014@gmail.com>
Co-authored-by: khluu <khluu000@gmail.com>
Co-authored-by: qizixi <zixi@inferact.ai>
Co-authored-by: Zhewen Li <zhewenli@inferact.ai>
 2026-04-26 18:31:08 -07:00
Mark McLoughlinandOr Ozeri d3af8c1831 [Core][Metrics][BugFix] Replace num_cached_tokens/num_external_computed_tokens with PrefillStats (#37460) 
Related to `Counters can only be incremented by non-negative amounts`
error with the `vllm:prompt_tokens_by_source_total` metric.

Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: Or Ozeri <or@ozery.com>
 2026-04-14 09:00:45 +01:00
mukesh-haiClaude Sonnet 4.6Mark McLoughlin
 		5c18b961d6 [Core][Metrics] expose waiting request breakdown via labeled metric (capacity/deferred) (#38435) 
Signed-off-by: Mukesh Baphna <mukesh@hippocraticai.com>
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: Mark McLoughlin <markmc@redhat.com>
 2026-04-13 15:30:55 -04:00
Martin HickeyandOr Ozeri cc07dad789 [HMA] [KVEvent] Enable GPU-side KV events for HMA (#37688) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
Co-authored-by: Or Ozeri <or@ozery.com>
 2026-04-12 10:01:02 +03:00
Walter Beller-Morales e69a265135 [Feat][Core] safely abort requests when FSM fails to advance (#38663) 
Signed-off-by: walterbm <walter.beller.morales@gmail.com>
 2026-04-06 08:00:16 -07:00
HarshRathvaandOr Ozeri 17b72fd1c8 Fix priority preemption regression test in scheduler (#37051) 
Signed-off-by: HarshRathva <harshrathvaai@gmail.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-04-01 06:36:12 +03:00
Wentao Ye 1bf2ddd0ee [Refactor] Rename WAITING_FOR_FSM to WAITING_FOR_STRUCTURED_OUTPUT_GRAMMAR (#38048) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-03-25 11:41:44 -04:00
tianshu-Michael-yu 269bf46d99 fix: disambiguate multimodal prefix cache keys (#36708) 
Signed-off-by: tianshu.yu <tianshuyu.formal@gmail.com>
 2026-03-20 10:33:20 +08:00
Yong Hoon Shin de35c06c66 Make KV connector metadata build overridable via plugin (#37336) 
Signed-off-by: Yong Hoon Shin <yhshin@meta.com>
 2026-03-17 21:29:06 +00:00
Harry Huang 45f526d652 [BugFix] Correct max memory usage for multiple KV-cache groups (#36030) 
Signed-off-by: huanghaoyan.hhy <huanghaoyan.hhy@alibaba-inc.com>
 2026-03-17 00:38:52 +00:00
Wentao YeandOr Ozeri a8ff2cca92 [Perf] Optimize scheduler overhead for PD disaggregation, around 5% E2E perf improvement (#35781) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-03-10 21:25:30 -07:00
lifandmcelrath 00b814ba5a [V0 Deprecation] Remove unused swap_space parameter (#36216) 
Signed-off-by: majiayu000 <1835304752@qq.com>
Co-authored-by: mcelrath
 2026-03-07 22:09:55 +08:00
cong-or 57c84ff129 perf: add __slots__ to KVCacheBlock (#36164) 
Signed-off-by: cong-or <conchubhar.gannon@gmail.com>
 2026-03-05 22:04:09 -08:00