Gabriel Wu
82536acc54
Keep scheduler alive for delayed KV connector frees ( #43433 )
...
Signed-off-by: Zihua Wu <13583761+lucifer1004@users.noreply.github.com >
2026-05-23 06:23:32 +00:00
Aaron Hao and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
f34623bf3c
[bug] AsyncScheduler drops first post-resume token after pause_generation + clear_cache ( #42117 )
...
Signed-off-by: hao-aaron <ahao@anyscale.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-19 01:06:21 -07:00
Yifan Qiao
4b364f810e
[Core][DSV4] Skip caching SWA blocks that can never serve a prefix-cache hit ( #42258 )
...
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
2026-05-15 15:59:18 +08:00
Daniel W and Li, Jiang <jiang1.li@intel.com>
27b85d2084
[Bugfix] Clarify CPU backend memory error messages reference shared flag ( #42479 )
...
Signed-off-by: daniel-devlab <282598346+daniel-devlab@users.noreply.github.com >
Co-authored-by: Li, Jiang <jiang1.li@intel.com >
2026-05-15 06:35:05 +00:00
Wentao Ye and Nick Hill
ae4f59f0ec
[Model Runner v2] Oracle for model runner v2 - qwen3 dense model by default [1/N] ( #39337 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-05-14 10:02:33 -07:00
c7560af424
[RFC] Replace shared-memory routed experts with ModelRunnerOutput transfer and HTTP support ( #39568 )
...
Signed-off-by: xhx1022 <1737006628@qq.com >
Signed-off-by: arlenxu <arlenxu@tencent.com >
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com >
Co-authored-by: arlenxu <arlenxu@tencent.com >
Co-authored-by: Junjie Zhang <junj.jay.zhang@gmail.com >
2026-05-14 14:12:30 +00:00
aoshen02
8c79ad6580
Revert "[Core] Replace routing replay with device cache and async D2H pipeline" ( #39917 ) ( #42434 )
...
Signed-off-by: aoshen02 <aoshen@inferact.ai >
2026-05-13 23:49:01 -07:00
Michael Goin and Claude Opus 4.6
2f821faeae
[Spec Decode] Support hybrid attention models in extract_hidden_states ( #39949 )
...
Signed-off-by: mgoin <mgoin64@gmail.com >
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com >
2026-05-13 10:45:53 -07:00
Yifan Qiao and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
13bf242100
[Feat][KVConnector] Add bind_gpu_block_pool() to KVConnectorBase_V1 ( #39654 )
...
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-05-13 02:10:29 -07:00
Yan Ru Pei
bcb9c133ba
feat(kv-events): emit KV cache metadata ( #40984 )
...
Signed-off-by: PeaBrane <yanrpei@gmail.com >
2026-05-12 15:58:48 +00:00
Nicolò Lucchesi
770e9bd6b3
[Nixl][PD] Lease renewal TTL KV blocks on P ( #41383 )
...
Signed-off-by: NickLucche <nlucches@redhat.com >
2026-05-11 09:27:30 +00:00
TomerBN-Nvidia
8189a15914
[Core] Replace routing replay with device cache and async D2H pipeline ( #39917 )
...
Signed-off-by: Tomer Barnatan <tbarnatan@nvidia.com >
2026-05-07 11:24:56 -07:00
Wentao Ye and Nick Hill
51295793a2
[Model Runner V2] Add logprob_token_ids support ( #40559 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-05-01 10:02:03 -07:00
Luis 🚀 and Andrew Sansom
14043dfecd
feat: Enable prompt_embeds Content Part Support in vLLM Chat Completions API ( #40720 )
...
Signed-off-by: Luis Robaina <luis@protopia.ai >
Signed-off-by: Luis Robaina 🚀 <luisfabian1545@gmail.com >
Signed-off-by: LuisRobaina <luis@protopia.ai >
Co-authored-by: Andrew Sansom <qthequartermasterman@gmail.com >
2026-05-01 10:05:55 +08:00
Nick Hill
dd5506a157
[Core] Simplify handling of scheduler_reserve_full_isl option ( #41064 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-04-30 18:10:00 -07:00
Wei Zhao
0ff1bf9bb1
[Bugfix] Fix failure to allocate KV blocks error ( #41282 )
...
Signed-off-by: wzhao18 <wzhao18.sz@gmail.com >
2026-04-29 18:44:07 -07:00
Nick Hill
e68fa1b90a
[Core] Account for num_gpu_blocks_override in max_model_len checks ( #41069 )
...
Signed-off-by: Nick Hill <nickhill123@gmail.com >
2026-04-28 15:44:09 -07:00
c245d35ff4
[Model] Add MiMo-V2.5 support ( #40967 )
...
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com >
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn >
Signed-off-by: Isotr0py <Isotr0py@outlook.com >
Signed-off-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com >
Co-authored-by: zjy0516 <riverclouds.zhu@qq.com >
Co-authored-by: zjy0516 <zhujiangyun@inferact.ai >
Co-authored-by: yasong <yasong.wang@inferact.ai >
Co-authored-by: Claude Opus 4.6 <noreply@anthropic.com >
Co-authored-by: Copilot <copilot@github.com >
2026-04-27 13:26:51 +00:00
7b1bc0a3eb
[Bugfix] Cap SWA/chunked-local runtime admission to startup pool-sizing bound ( #40946 )
...
Signed-off-by: Dao Le <Dao007forever@gmail.com >
Signed-off-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Claude <noreply@anthropic.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
2026-04-27 04:33:13 +00:00
+7
4d51588e23
[Feat] DeepSeek V4 Rebased ( #40860 )
...
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
Signed-off-by: Woosuk Kwon <woosuk@inferact.ai >
Signed-off-by: qizixi <zixi@inferact.ai >
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com >
Signed-off-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Yongye Zhu <zyy1102000@gmail.com >
Co-authored-by: Yongye Zhu <yongye@inferact.ai >
Co-authored-by: Simon Mo <simon@inferact.ai >
Co-authored-by: Bugen Zhao <i@bugenzhao.com >
Co-authored-by: Giancarlo Delfin <gdelfin@inferact.ai >
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com >
Co-authored-by: Nick Hill <nickhill123@gmail.com >
Co-authored-by: Roger Wang <hey@rogerw.io >
Co-authored-by: Roy Wang <yasong.wang@inferact.ai >
Co-authored-by: Woosuk Kwon <woosuk@inferact.ai >
Co-authored-by: youkaichao <youkaichao@gmail.com >
Co-authored-by: Zhewen Li <jerven.vllm@gmail.com >
Co-authored-by: Zijing Liu <liuzijing2014@gmail.com >
Co-authored-by: khluu <khluu000@gmail.com >
Co-authored-by: qizixi <zixi@inferact.ai >
Co-authored-by: Zhewen Li <zhewenli@inferact.ai >
2026-04-26 18:31:08 -07:00
Wentao Ye and gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
8f87eb4622
[Refactor] Clean up log once scope="local" ( #40540 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com >
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
2026-04-22 16:42:43 -04:00
Matthew Bonanni and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
c075702eae
[Misc][UX] Suppress confusing num_gpu_blocks log lines ( #40402 )
...
Signed-off-by: Matthew Bonanni <mbonanni@redhat.com >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-04-20 22:32:46 +00:00
f4b42df048
[Attention Backend] TurboQuant: 2-bit KV cache compression with 4x capacity ( #38479 )
...
Signed-off-by: vibhavagarwal5 <vibhavagarwal5@gmail.com >
Signed-off-by: Michael Goin <mgoin64@gmail.com >
Co-authored-by: Xinyu Chen <xinyu1.chen@intel.com >
Co-authored-by: Michael Goin <mgoin64@gmail.com >
2026-04-14 19:57:13 -07:00
Mark McLoughlin
906a8c15d0
[Core][Metrics] Remove unused SchedulerStats.encoder_cache_usage ( #39693 )
...
Signed-off-by: Mark McLoughlin <markmc@redhat.com >
2026-04-14 12:53:57 -04:00
Mark McLoughlin and Or Ozeri
d3af8c1831
[Core][Metrics][BugFix] Replace num_cached_tokens/num_external_computed_tokens with PrefillStats ( #37460 )
...
Related to `Counters can only be incremented by non-negative amounts`
error with the `vllm:prompt_tokens_by_source_total` metric.
Signed-off-by: Mark McLoughlin <markmc@redhat.com >
Co-authored-by: Or Ozeri <or@ozery.com >
2026-04-14 09:00:45 +01:00
5c18b961d6
[Core][Metrics] expose waiting request breakdown via labeled metric (capacity/deferred) ( #38435 )
...
Signed-off-by: Mukesh Baphna <mukesh@hippocraticai.com >
Signed-off-by: Mark McLoughlin <markmc@redhat.com >
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com >
Co-authored-by: Mark McLoughlin <markmc@redhat.com >
2026-04-13 15:30:55 -04:00
Martin Hickey and Or Ozeri
cc07dad789
[HMA] [KVEvent] Enable GPU-side KV events for HMA ( #37688 )
...
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com >
Co-authored-by: Or Ozeri <or@ozery.com >
2026-04-12 10:01:02 +03:00
Tianyu Guo and mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
8d0aabdde9
Fix the order of _free_encoder_inputs ( #38907 )
...
Signed-off-by: Tianyu Guo <guoty9@mail2.sysu.edu.cn >
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2026-04-10 22:47:48 -07:00
e7cfd7c5b9
Add Gemma4 Eagle3 support ( #39450 )
...
Signed-off-by: Rahul-Tuli <rtuli@redhat.com >
Signed-off-by: Fynn Schmitt-Ulms <fschmitt@redhat.com >
Co-authored-by: Rahul-Tuli <rtuli@redhat.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-04-10 12:35:35 -07:00
Walter Beller-Morales
e69a265135
[Feat][Core] safely abort requests when FSM fails to advance ( #38663 )
...
Signed-off-by: walterbm <walter.beller.morales@gmail.com >
2026-04-06 08:00:16 -07:00
Yifan Qiao
91e4521f9f
[Feat][v1] Simple yet General CPU KV Cache Offloading ( #37160 )
...
Signed-off-by: Yifan Qiao <yifanqiao@berkeley.edu >
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai >
2026-03-31 17:58:37 -07:00
Mark McLoughlin
e38817fadb
[Core][KV Connector] Remove use of num_cached_tokens in error handling ( #38096 )
...
Signed-off-by: Mark McLoughlin <markmc@redhat.com >
2026-03-25 18:20:48 +00:00
Wentao Ye
1bf2ddd0ee
[Refactor] Rename WAITING_FOR_FSM to WAITING_FOR_STRUCTURED_OUTPUT_GRAMMAR ( #38048 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-03-25 11:41:44 -04:00
Dan Blanaru and Claude
dc78c2c933
[Core] add option to schedule requests based on full ISL ( #37307 )
...
Signed-off-by: Dan Blanaru <48605845+DanBlanaru@users.noreply.github.com >
Co-authored-by: Claude <noreply@anthropic.com >
2026-03-24 13:01:12 -04:00
tianshu-Michael-yu
269bf46d99
fix: disambiguate multimodal prefix cache keys ( #36708 )
...
Signed-off-by: tianshu.yu <tianshuyu.formal@gmail.com >
2026-03-20 10:33:20 +08:00
Yong Hoon Shin
de35c06c66
Make KV connector metadata build overridable via plugin ( #37336 )
...
Signed-off-by: Yong Hoon Shin <yhshin@meta.com >
2026-03-17 21:29:06 +00:00
Harry Huang
45f526d652
[BugFix] Correct max memory usage for multiple KV-cache groups ( #36030 )
...
Signed-off-by: huanghaoyan.hhy <huanghaoyan.hhy@alibaba-inc.com >
2026-03-17 00:38:52 +00:00
jaime campos salas
891c60dcd5
fix(kv-cache): increase hybrid attention grouping threshold from 1.25 to 1.5 ( #36684 )
...
Signed-off-by: Jaime Campos Salas <jaime.campos.salas@gmail.com >
2026-03-12 23:28:27 -04:00
Wentao Ye
35bdca5431
[Refactor] Remove dead code in KV connector ( #36424 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
2026-03-11 19:40:17 +00:00
Hongxin Xu and arlenxu
bea02cdf93
Fix routed experts capture for hybrid models (Mamba + Attention) ( #35744 )
...
Signed-off-by: arlenxu <arlenxu@tencent.com >
Signed-off-by: xhx1022 <1737006628@qq.com >
Co-authored-by: arlenxu <arlenxu@tencent.com >
2026-03-11 08:53:10 -07:00
Wentao Ye and Or Ozeri
a8ff2cca92
[Perf] Optimize scheduler overhead for PD disaggregation, around 5% E2E perf improvement ( #35781 )
...
Signed-off-by: yewentao256 <zhyanwentao@126.com >
Signed-off-by: Wentao Ye <44945378+yewentao256@users.noreply.github.com >
Co-authored-by: Or Ozeri <oro@il.ibm.com >
2026-03-10 21:25:30 -07:00
Vadim Gimpelson
4ff8c3c8f9
[BUGFIX][Mamba][Qwen3.5] Zero freed SSM cache blocks on GPU ( #35219 )
...
Signed-off-by: Vadim Gimpelson <vadim.gimpelson@gmail.com >
2026-03-10 03:32:20 -07:00
Nicolò Lucchesi
5b3ba94ab4
[Core][KVConnector] Support HMA+NixlConnector ( #35758 )
...
Signed-off-by: NickLucche <nlucches@redhat.com >
2026-03-06 08:51:21 +01:00
cong-or
57c84ff129
perf: add __slots__ to KVCacheBlock ( #36164 )
...
Signed-off-by: cong-or <conchubhar.gannon@gmail.com >
2026-03-05 22:04:09 -08:00
Jiayi Yan and Harry Mellor
6a895197fa
[Bugfix][CI] fix typos ( #34934 )
...
Signed-off-by: 1195343015 <1195343015@qq.com >
Signed-off-by: Jiayi Yan <66017932+1195343015@users.noreply.github.com >
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-03-05 17:05:46 +00:00
Or Ozeri
612e7729c2
[KVConnector] Scheduler: Fix num_computed_tokens after async KV load ( #34616 )
...
Signed-off-by: Or Ozeri <oro@il.ibm.com >
2026-03-05 14:25:15 +00:00
Qi Wang
6aa6ad8992
[BugFix] Fix implicit and incorrect assumption on ECConnector is_producer ( #34783 )
...
Signed-off-by: Qi Wang <qiwa@nvidia.com >
2026-03-04 15:01:30 +01:00
aykoppol
25e02647c2
[Core] Add optional flags to check for repetitive token patterns in engine output ( #35451 )
...
Signed-off-by: aykoppol <aykoppol+git@gmail.com >
2026-03-03 12:23:25 +08:00
Harry Mellor
01914445b0
Remove bc-lint ( #35274 )
...
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com >
2026-02-26 03:01:01 -08:00
Benjamin Chislett
682566b18e
[Bug] Refactor max_num_batched_tokens to account for drafting ( #34898 )
...
Signed-off-by: Benjamin Chislett <bchislett@nvidia.com >
2026-02-22 11:18:46 -05:00
Gabriel Wu