iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
14,386 Commits 3 Branches 0 Tags
fa6a6be51978bd4b49ba0da17039e60f96dc5b13
Commit Graph
539 Commits
Author SHA1 Message Date
Ming Yang 10f535c086 [Bugfix] Fix port conflict by obtaining a list of open ports upfront (#21894) 
Signed-off-by: Ming Yang <minos.future@gmail.com>
 2025-08-21 10:22:18 -07:00
Roger WangandDarkLight1337 79f05e4436 [Multimodal] Always enable hashing mm data (#23308) 
Signed-off-by: Roger Wang <hey@rogerw.io>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-08-21 07:23:28 -07:00
Chenheli Hua e58c5a9768 [Core] Add torch profiler CPU traces for AsyncLLM. (#21794) 
Signed-off-by: Chenheli Hua <huachenheli@outlook.com>
 2025-08-20 02:32:47 +00:00
Woosuk Kwon c9b38be8aa [Spec Decode] Make propose_draft_token_ids non-blocking for lower TTFT (#23041) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-08-18 17:20:38 -07:00
Cyrus Leung 27e8d1ea3e [Refactor] Define MultiModalKwargsItems separate from MultiModalKwargs (#23053) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-08-18 09:52:00 +00:00
Cyrus Leung 4dff91c93d [Refactor] Allow optional MultiModalKwargsItem in IPC (#23022) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-08-16 11:30:49 +00:00
Calvin ChenandNick Hill e4e37ded56 [V1] support min_tokens for detokener (#22014) 
Signed-off-by: calvin chen <wen.chen@dynamia.ai>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-08-16 02:28:10 +00:00
Nick Hill ad0297d113 [Misc] Support passing multiple request ids at once to AsyncLLM.abort() (#22944) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-08-15 17:00:36 -07:00
Or Ozeri c280066f9d [v1] Move block_hashes from KVCacheManager to Request.block_hashes (#19728) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2025-08-15 16:52:52 -07:00
Nick HillandYinghai Lu b9dc9d2607 [BugFix] Handle case where async utility call is cancelled (#22996) 
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Yinghai Lu <yinghai@thinkingmachines.ai>
 2025-08-15 17:38:42 -06:00
Nick Hill 919234fe17 [BugFix] Fix initial DP request load imbalance (#22910) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-08-14 15:20:28 -07:00
Nick Hill ebcce2cd36 [Core] Return final response for aborted requests from AsyncLLM.generate (#22283) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-08-14 14:49:02 -07:00
iAmir97iAmir97gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 		7655dc3e45 [Bugfix] Add reset prefix cache for online serving (#22726) 
Signed-off-by: iAmir97 <Amir.balwel@embeddedllm.com>
Signed-off-by: iAmir97 <71513472+iAmir97@users.noreply.github.com>
Co-authored-by: iAmir97 <Amir.balwel@embeddedllm.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-08-14 04:04:18 -07:00
Nick HillandIsotr0py eb08487b18 [BugFix] Threadsafe close async zmq sockets (#22877) 
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2025-08-14 03:44:29 -07:00
Cyrus Leung 19b927e52d [Core] Use individual MM items in P0/P1 cache and model runner (#22570) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-08-13 07:18:07 -07:00
+10
Roger WangNing XieTJianAndrew SansomZhiyuShu WangXIn LiJunhao LiChaunceyYuxuan ZhangZiTian ZhaoCyrus LeungPo-Han HuangiAmir97iAmir97gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>Cyrus LeungHong Hanhyoukaichaolkchen
 		08b751ba74 Implicit language-model-only mode via limit-mm-per-prompt (#22299) 
Signed-off-by: Roger Wang <hey@rogerw.me>
Signed-off-by: Andy Xie <andy.xning@gmail.com>
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
Signed-off-by: Andrew Sansom <andrew@protopia.ai>
Signed-off-by: Zhiyu Cheng <zhiyuc@nvidia.com>
Signed-off-by: Shu Wang <shuw@nvidia.com>
Signed-off-by: Po-Han Huang <pohanh@nvidia.com>
Signed-off-by: Shu Wang. <shuw@nvidia.com>
Signed-off-by: XIn Li <xinli@nvidia.com>
Signed-off-by: Junhao Li <junhao@ubicloud.com>
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
Signed-off-by: zRzRzRzRzRzRzR <2448370773@qq.com>
Signed-off-by: zitian.zhao <zitian.zhao@tencentmusic.com>
Signed-off-by: zitian zhao <zitian.zhao@tencentmusic.com>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: iAmir97 <Amir.balwel@embeddedllm.com>
Signed-off-by: iAmir97 <71513472+iAmir97@users.noreply.github.com>
Signed-off-by: Linkun <github@lkchen.net>
Co-authored-by: Ning Xie <andy.xning@gmail.com>
Co-authored-by: TJian <tunjian.tan@embeddedllm.com>
Co-authored-by: Andrew Sansom <andrew@protopia.ai>
Co-authored-by: Zhiyu <zhiyuc@nvidia.com>
Co-authored-by: Shu Wang <shuw@nvidia.com>
Co-authored-by: XIn Li <xinli@nvidia.com>
Co-authored-by: Junhao Li <streaver91@gmail.com>
Co-authored-by: Chauncey <chaunceyjiang@gmail.com>
Co-authored-by: Yuxuan Zhang <2448370773@qq.com>
Co-authored-by: ZiTian Zhao <zitian.zhao@tencentmusic.com>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
Co-authored-by: Po-Han Huang (NVIDIA) <53919306+nvpohanh@users.noreply.github.com>
Co-authored-by: iAmir97 <71513472+iAmir97@users.noreply.github.com>
Co-authored-by: iAmir97 <Amir.balwel@embeddedllm.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Co-authored-by: Hong Hanh <hanh.usth@gmail.com>
Co-authored-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: lkchen <github@lkchen.net>
 2025-08-08 22:21:40 -07:00
Nick Hill ccdae737a0 [BugFix] Don't cancel asyncio tasks directly from destructors (#22476) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-08-08 01:13:18 -07:00
Cyrus Leung 1712543df6 [CI/Build] Fix multimodal tests (#22491) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-08-08 00:31:19 -07:00
Cyrus Leung 766bc8162c [Core] Store only the keys for multi-modal data in P0 (#22198) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-08-07 01:45:04 -07:00
Rui Qiao 302962e806 [Bugfix] Skip dead and non-GPU nodes for Ray DP engine allocation (#22275) 
Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
 2025-08-05 20:35:32 -07:00
22quinn 54de71d0df [Sampler] Support returning all logprobs or logits (#21792) 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
 2025-08-04 03:04:12 -07:00
Nick Hill 8d524ce79f [BugFix] Improve internal DP load balancing (#21617) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-08-01 19:45:27 -07:00
Nick Hill 881e1af43a [BugFix] Harden distributed DP startup (#21538) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-08-01 21:40:45 +00:00
rongfu.leng b879ecd6e2 [Bugfix] fix when skip tokenizer init (#21922) 
Signed-off-by: rongfu.leng <rongfu.leng@daocloud.io>
 2025-08-01 10:09:36 -07:00
Harry Mellor 2d7b09b998 Deprecate --disable-log-requests and replace with --enable-log-requests (#21739) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-08-01 17:16:37 +01:00
wuhang e6680f9e25 [Bugfix] Add log prefix in non-dp mode engine core (#21889) 
Signed-off-by: wuhang <wuhang6@huawei.com>
 2025-08-01 09:04:16 +00:00
Zebing Lin ca9e2be3ed [Core] Move EngineCoreRequest to Request conversion out of EngineCore (#21627) 
Signed-off-by: linzebing <linzebing1995@gmail.com>
 2025-07-30 15:00:54 -07:00
Nick Hill 56bd537dde [Misc] Support more collective_rpc return types (#21845) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-07-30 10:20:20 -07:00
633WHUandchiliu 5c765aec65 [Bugfix] Fix TypeError in scheduler when comparing mixed request_id types (#21816) 
Signed-off-by: chiliu <chiliu@paypal.com>
Co-authored-by: chiliu <chiliu@paypal.com>
 2025-07-30 08:54:44 -07:00
Nick Hill 7234fe2685 [Misc] Rework process titles (#21780) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-07-29 05:14:47 +00:00
wuhang bccc43c033 [Bugfix]check health for engine core process exiting unexpectedly (#21728) 
Signed-off-by: wuhang <wuhang6@huawei.com>
 2025-07-28 06:17:31 -07:00
Maximilien de BayserandRussell Bryant 1cd6eaba54 Support encoder-only models without KV-Cache (#21270) 
Signed-off-by: Max de Bayser <maxdebayser@gmail.com>
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
Co-authored-by: Russell Bryant <rbryant@redhat.com>
 2025-07-26 21:09:52 +08:00
Cyrus Leung 46d81d6951 [V1] Get supported tasks from model runner instead of model config (#21585) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-07-25 05:36:45 -07:00
Chauncey 6da0078523 [Feat] Allow custom naming of vLLM processes (#21445) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
 2025-07-24 03:15:23 -07:00
Rui Qiao 73e3949d07 [Misc] Improve comment for DPEngineCoreActor._set_cuda_visible_devices() (#21501) 
Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
 2025-07-24 03:13:40 -07:00
Yinghai LuNick HillRui Qiao
 		11ef7a611e [BugFix] Set CUDA_VISIBLE_DEVICES before spawning the subprocesses (#21211) 
Signed-off-by: Yinghai Lu <yinghai@thinkingmachines.ai>
Signed-off-by: Nick Hill <nhill@redhat.com>
Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Rui Qiao <ruisearch42@gmail.com>
 2025-07-23 21:44:04 -07:00
Woosuk Kwon dc2f159f8a Dump input metadata on crash for async scheduling (#21258) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-07-23 21:10:30 -07:00
Robert ShawRobert ShawNick HillTyler Michael Smith
 		d5b981f8b1 [DP] Internal Load Balancing Per Node [one-pod-per-node] (#21238) 
Signed-off-by: Robert Shaw <robshaw@redhat.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
Co-authored-by: Robert Shaw <robshaw@redhat.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
 2025-07-23 20:57:32 -07:00
Michael Goin 82ec66f514 [V0 Deprecation] Remove Prompt Adapters (#20588) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-07-23 16:36:48 -07:00
Christian Pinto 8560a5b258 [Core][Model] PrithviMAE Enablement on vLLM v1 engine (#20577) 
Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
 2025-07-23 11:00:23 -07:00
Wang Yijun 44554a0068 Add tokenization_kwargs to encode for embedding model truncation (#21033) 2025-07-22 08:24:00 -07:00
Robert ShawandRobert Shaw 29d1ffc5b4 [DP] Fix Prometheus Logging (#21257) 
Signed-off-by: Robert Shaw <robshaw@redhat.com>
Co-authored-by: Robert Shaw <robshaw@redhat.com>
 2025-07-21 09:11:35 -07:00
Rui Qiao 217937221b Elastic Expert Parallel Initial Support (#20775) 
Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
 2025-07-18 17:46:09 -07:00
Cyrus Leung 45badd05d0 [Core] Set pooling params based on task and model (#21128) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-07-18 05:41:17 -07:00
Christian Pinto 4ffd963fa0 [v1][core] Support for attention free models (#20811) 
Signed-off-by: Christian Pinto <christian.pinto@ibm.com>
 2025-07-15 14:20:01 +00:00
Nathan Hoos d6902ce79f [V0][V1][Core] Add outlines integration for V1, and update V0 integration. (#15975) 
Signed-off-by: Nathan Hoos <thwackyy.y@gmail.com>
 2025-07-10 15:30:26 -04:00
Rui Qiao a6d795d593 [DP] Copy environment variables to Ray DPEngineCoreActors (#20344) 
Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
 2025-07-07 10:14:22 -07:00
Aaron PhamandNick Hill 4a98edff1f [Structured Outputs][V1] Skipping with models doesn't contain tokenizers (#20365) 
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-07-04 15:05:49 +08:00
Nick Hill 657f2f301a [DP] Support external DP Load Balancer mode (#19790) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-07-02 10:21:52 -07:00
Michael Goin f719772281 [Bugfix] Properly reject requests with empty list guided_choice (#20195) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-06-27 22:50:52 -07:00