iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
20,845 Commits 3 Branches 0 Tags
78300cdabffbaaffe7625ed34412ec2aec75535c
Commit Graph
89 Commits
Author SHA1 Message Date
Matthew KotilaReed MeyersonClaude Opus 4.8
 		7cfb97e337 [Frontend][Core][Spec Decode] Per-request acceptance stats in OpenAI API responses (#48915) 
Signed-off-by: Matthew Kotila <7692737+matthewkotila@users.noreply.github.com>
Co-authored-by: Reed Meyerson <31574681+reed-meyerson@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
 2026-08-20 15:42:49 -07:00
Canlin GuoandNick Hill 455edc022b [ModelRunnerV2] Support prompt embeds (#42963) 
Signed-off-by: gcanlin <canlinguosdu@gmail.com>
Signed-off-by: Canlin Guo <961750412@qq.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-08-17 21:58:50 +00:00
Karen Chung f57123aa2d feat(frontend): session id plumbing into requests (#48048) 
Signed-off-by: Karen Chung <karenc@nvidia.com>
 2026-08-03 22:18:58 +00:00
Nick HillChris EastwoodYifan Qiao
 		a0c092ee72 [BugFix] Fix num_output_placeholders preemption underflow (#48245) 
Signed-off-by: Chris Eastwood <chris.eastwood@pwn4g3.dev>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Chris Eastwood <chris.eastwood@pwn4g3.dev>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
 2026-07-29 06:41:54 -07:00
Nick Hill 8ac8375270 [Core] Preserve Marconi caching with selective hybrid cache retention (#47782) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-07-13 21:24:20 +01:00
omerpaz95andOr Ozeri 8df14cfc8c [EC Connector] Add EC Transfer Params (#42433) 
Signed-off-by: omerpaz95 <omerpaz95@gmail.com>
Co-authored-by: Or Ozeri <oro@il.ibm.com>
 2026-07-12 14:35:33 +03:00
GongLei-HWLei GongClaude
 		e5588e49bc [Core][KV events] Report prefix-cache-reused blocks in full report mode (#45261) 
Signed-off-by: Lei Gong <gonglei25@huawei.com>
Co-authored-by: Lei Gong <gonglei25@huawei.com>
Co-authored-by: Claude <noreply@anthropic.com>
 2026-07-09 22:46:54 -07:00
SaddssYifan QiaoSaddssNick Hill
 		cd0de48d08 [Bugfix][V1] Free out-of-window blocks on the processed-token basis under async scheduling (#47728) 
Signed-off-by: Saddss <28726669061@qq.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Signed-off-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Yifan Qiao <yifanqiao@inferact.ai>
Co-authored-by: Saddss <28726669061@qq.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-07-08 13:34:19 +01:00
llxNick HillJiangyun Zhu
 		d467a2a7f2 [Bugfix] Defer block freeing until in-flight steps finish under async scheduling + PD KV consumer (#45357) 
Signed-off-by: llx-08 <2596671364@qq.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
Co-authored-by: Jiangyun Zhu <riverclouds.zhu@qq.com>
 2026-06-15 21:36:09 +00:00
Nick Hill e15f20258b [ModelRunnerV2] Avoid pipeline parallel bubbles (#42187) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-06-02 14:02:01 -07:00
Aaron Haoandmergify[bot] <37929162+mergify[bot]@users.noreply.github.com> f34623bf3c [bug] AsyncScheduler drops first post-resume token after pause_generation + clear_cache (#42117) 
Signed-off-by: hao-aaron <ahao@anyscale.com>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
 2026-05-19 01:06:21 -07:00
Dao007forever 3f5bd482f5 [Bugfix][KV Transfer][NIXL] Notify P node on pre-admission rejection to free stranded KV blocks (#41269) 2026-05-09 22:52:09 -07:00
Bugen Zhaoandwang.yuqi a07642667d [Bugfix] Pass reasoning parser kwargs to structured output (#41199) 
Signed-off-by: Bugen Zhao <i@bugenzhao.com>
Co-authored-by: wang.yuqi <yuqi.wang@daocloud.io>
 2026-04-30 23:38:02 -07:00
Luis 🚀andAndrew Sansom 14043dfecd feat: Enable prompt_embeds Content Part Support in vLLM Chat Completions API (#40720) 
Signed-off-by: Luis Robaina <luis@protopia.ai>
Signed-off-by: Luis Robaina 🚀 <luisfabian1545@gmail.com>
Signed-off-by: LuisRobaina <luis@protopia.ai>
Co-authored-by: Andrew Sansom <qthequartermasterman@gmail.com>
 2026-05-01 10:05:55 +08:00
Mark McLoughlinandOr Ozeri d3af8c1831 [Core][Metrics][BugFix] Replace num_cached_tokens/num_external_computed_tokens with PrefillStats (#37460) 
Related to `Counters can only be incremented by non-negative amounts`
error with the `vllm:prompt_tokens_by_source_total` metric.

Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: Or Ozeri <or@ozery.com>
 2026-04-14 09:00:45 +01:00
Wentao Ye 1bf2ddd0ee [Refactor] Rename WAITING_FOR_FSM to WAITING_FOR_STRUCTURED_OUTPUT_GRAMMAR (#38048) 
Signed-off-by: yewentao256 <zhyanwentao@126.com>
 2026-03-25 11:41:44 -04:00
Cyrus Leung f96c3ab08c [Deprecation][1/2] Remove items deprecated in v0.18 (#36470) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-03-09 03:43:23 -07:00
aykoppol 25e02647c2 [Core] Add optional flags to check for repetitive token patterns in engine output (#35451) 
Signed-off-by: aykoppol <aykoppol+git@gmail.com>
 2026-03-03 12:23:25 +08:00
zhongdaor-nvNicolò LucchesiCyrus Leung
 		a0fe7ea2f0 [feat] Add per-block extra_keys to KV events (#33304) 
Signed-off-by: zhongdaor-nv <zhongdaor@nvidia.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2026-02-20 20:11:40 -08:00
Cyrus Leung ea5ff3a1f6 [Refactor] Simplify BOS/EOS token handling (#34435) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-02-12 18:18:24 -08:00
Roger Wang 8a5e0e2b2b [Bugfix][Core] Fix CPU memory leak from Request reference cycle in prefix caching (#34183) 
Signed-off-by: Roger Wang <hey@rogerw.io>
 2026-02-10 13:03:32 +08:00
Cyrus Leung 48312e579a [Misc] Make PlaceholderRange.get_num_embeds a method (#34035) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-02-07 05:30:17 +00:00
Chauncey 6abb0454ad [Perf] Optimize the performance of structured output + reasoning (#33557) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
 2026-02-05 15:45:29 +08:00
Nick Hill fa4e0fb028 [Core] Don't schedule spec tokens with prefill chunks (#33652) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-02-04 23:40:22 +00:00
杨朱 · KikiandClaude Opus 4.5 1a7894dbdf [Misc] Replace Optional[X] with X | None syntax (#33332) 
Signed-off-by: carlory <baofa.fan@daocloud.io>
Co-authored-by: Claude Opus 4.5 <noreply@anthropic.com>
 2026-01-30 01:56:59 -08:00
Joshua DengRoger WangPatrick von PlatenNick Hill
 		91601ff478 [Feature] add session based streaming input support to v1 (#28973) 
Signed-off-by: Joshua Deng <joshuakdeng@gmail.com>
Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com>
Signed-off-by: Nick Hill <nickhill123@gmail.com>
Signed-off-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>
Co-authored-by: Nick Hill <nickhill123@gmail.com>
 2026-01-24 12:06:28 -08:00
Or Ozeri 028599739d [BugFix] scheduler: Fix resuming of preempted requests after async load (#31583) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2026-01-10 12:39:25 -08:00
Nick Hill 3b312fb792 [Minor] Various small code cleanups/simplifications (#31508) 
Signed-off-by: njhill <nickhill123@gmail.com>
 2025-12-29 22:42:06 -08:00
Roger WangandSun Kim f5f51e5931 [Core][MM] Optimize encoder cache manager by operating with embeddings only (#30475) 
Signed-off-by: Roger Wang <hey@rogerw.io>
Co-authored-by: Sun Kim <sunytokki@gmail.com>
 2025-12-16 14:18:17 -08:00
Will EatonMark McLoughlinNick Hillchaunceyjiang
 		a9e4106f28 [P/D] KV Load Failure Recovery/Abort Configuration (#26813) 
Signed-off-by: Will Eaton <weaton@redhat.com>
Signed-off-by: Will Eaton <me@wseaton.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
Co-authored-by: chaunceyjiang <chaunceyjiang@gmail.com>
 2025-12-10 11:00:52 -08:00
ChaunceyandNick Hill 0a9caca9f5 [Bugfix] fix --scheduling-policy=priority & n>1 crashes engine (#29764) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-12-02 22:42:28 +00:00
Zhuohan LiandNick Hill d0cd728907 [Core] Support reseting all running requests' KV while calling reset_prefix_cache (#28827) 
Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-12-02 02:25:05 +00:00
Mark McLoughlinandNicolò Lucchesi c6fa3895e9 [KV Connector] Fix async connector prefix cache metrics (#28585) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: Nicolò Lucchesi <nlucches@redhat.com>
 2025-11-21 17:45:00 -05:00
wang.yuqi a55b64635c [Model] Allow users to control skip reading cache per request. (#28194) 
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
Signed-off-by: wang.yuqi <noooop@126.com>
 2025-11-16 00:04:50 -08:00
Snehlata e15601789b [Feature]: Add corrupted request metric to V1 metrics system. (#27306) 
Signed-off-by: atalhens <sneh.lata@nutanix.com>
 2025-11-05 13:45:29 -08:00
Nick Hill 4aed506b65 [Core] Streamline some structured output related code (#26737) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-10-14 23:27:44 +00:00
Harry Mellor 8fcaaf6a16 Update Optional[x] -> x | None and Union[x, y] to x | y (#26633) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-10-12 09:51:31 -07:00
Harry Mellor d6953beb91 Convert formatting to use ruff instead of yapf + isort (#26247) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-10-05 07:06:22 -07:00
Zhuohan Li 8bf8f45822 [Core] Don't count preempted tokens in prefix cache hit rate (#25787) 
Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>
 2025-09-27 00:16:40 +00:00
Andrew SansomandCyrus Leung 9a4600e4dc [CORE] Prompt Embeddings Support for v1 Engine (#24278) 
Signed-off-by: Andrew Sansom <andrew@protopia.ai>
Signed-off-by: Andrew Sansom <qthequartermasterman@gmail.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2025-09-19 08:03:09 +08:00
Aaron PhamandHarry Mellor 29283e8976 [Chore] Cleanup guided namespace, move to structured outputs config (#22772) 
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-09-18 09:20:27 +00:00
Flora Feng 0377802c20 [Multimodal] Remove legacy multimodal fields in favor of MultiModalFeatureSpec (#24548) 
Signed-off-by: sfeng33 <4florafeng@gmail.com>
 2025-09-12 21:42:23 +08:00
RichardoMuMu HuaiYe ZhangBenjamin Bartelssimon-mo瑜琮Aaron Pham22quinn
 		40b6c9122b [V1] feat:add engine v1 tracing (#20372) 
Signed-off-by: Mu Huai <tianbowen.tbw@antgroup.com>
Signed-off-by: Ye Zhang <zhysishu@gmail.com>
Signed-off-by: RichardoMu <44485717+RichardoMrMu@users.noreply.github.com>
Signed-off-by: simon-mo <simon.mo@hey.com>
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
Co-authored-by: Mu Huai <tianbowen.tbw@antgroup.com>
Co-authored-by: Ye Zhang <zhysishu@gmail.com>
Co-authored-by: Benjamin Bartels <benjamin@bartels.dev>
Co-authored-by: simon-mo <simon.mo@hey.com>
Co-authored-by: 瑜琮 <ly186375@antfin.com>
Co-authored-by: Aaron Pham <contact@aarnphm.xyz>
Co-authored-by: 22quinn <33176974+22quinn@users.noreply.github.com>
 2025-09-11 17:10:39 -07:00
Flora Feng 69f46359dd [Multimodal] Consolidate mm inputs into MultiModalFeatureSpec (#23779) 
Signed-off-by: sfeng33 <4florafeng@gmail.com>
 2025-08-29 18:36:57 +08:00
Woosuk Kwon 6d243efeda [Misc] Convert use_structured_output property into constant (#23060) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-08-17 12:41:38 -07:00
Cyrus Leung 4dff91c93d [Refactor] Allow optional MultiModalKwargsItem in IPC (#23022) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-08-16 11:30:49 +00:00
Or Ozeri c280066f9d [v1] Move block_hashes from KVCacheManager to Request.block_hashes (#19728) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2025-08-15 16:52:52 -07:00
Cyrus Leung 19b927e52d [Core] Use individual MM items in P0/P1 cache and model runner (#22570) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-08-13 07:18:07 -07:00
Woosuk Kwon d4d309409f Implement Async Scheduling (#19970) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-07-14 23:01:46 -07:00
amitandRoger Wang 4a0f7888a3 [Core] feat: Implement Priority Scheduling in V1 Engine (#19057) 
Signed-off-by: amit <amit.man@gmail.com>
Co-authored-by: Roger Wang <Rogerw0108@gmail.com>
 2025-06-22 20:18:08 -07:00