iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
14,386 Commits 3 Branches 0 Tags
fa6a6be51978bd4b49ba0da17039e60f96dc5b13
Commit Graph
395 Commits
Author SHA1 Message Date
Nick Hill 7e8d97dd3f [BugFix] Honor enable_caching in connector-delayed kvcache load case (#19435) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-06-13 09:46:32 +00:00
jmswen c9280e6346 [Bugfix] Respect num-gpu-blocks-override in v1 (#19503) 
Signed-off-by: Jon Swenson <jmswen@gmail.com>
 2025-06-12 11:00:23 +00:00
Nick Hill 646d62f636 [Core] Use tuple for kv cache group block ids (#19175) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-06-10 07:01:17 +02:00
Nicolò LucchesiandNick Hill b6a3a9f76d [Core] Fix abrupt request abort (#18485) 
Signed-off-by: nicklucche <nlucches@redhat.com>
Signed-off-by: Nick Hill <nhill@redhat.com>

Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-06-06 16:27:59 -07:00
Nick Hill aad30bd306 [BugFix] Fix MultiConnector test after HMA changes (#19291) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-06-06 20:16:24 +00:00
Nick Hill 65c69444b1 [Docs] Improve V1 KVConnector interface documentation (#19172) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-06-06 16:22:45 +08:00
Jinghui Zhangandjinghui 90b78ec5f9 [v1][P/D] Fix a edge case in kv cache schedule (#19182) 
Co-authored-by: jinghui <jinghui@fb.com>
 2025-06-05 23:32:55 -07:00
Chen Zhang f8a1a2d108 [v1] Hybrid Memory Allocator (#17996) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-06-05 20:47:09 -07:00
Robert ShawandNick Hill c56ed8bb0e [Bugfix][Nixl] Fix full prefix cache hit bug (#18632) 
Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-06-05 02:07:32 +00:00
Yan Ru Pei b712be98c7 feat: add data parallel rank to KVEventBatch (#18925) 2025-06-03 17:14:20 -07:00
Chen Zhang a8da78eac9 [Bugfix] Max concurrency estimation and check_enough_kv_cache_memory for models with sliding window layers (#19029) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-06-04 00:14:06 +00:00
Chen Zhang b5fd9506c1 [Bugfix] get_num_blocks_to_allocate with null_block (#19031) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-06-03 15:30:55 -07:00
Simon Mo 02f0c7b220 [Misc] Add SPDX-FileCopyrightText (#19100) 
Signed-off-by: simon-mo <simon.mo@hey.com>
 2025-06-03 11:20:17 -07:00
Chen Zhang f32fcd9444 [v1][KVCacheManager] Rename BlockHashType to BlockHash (#19015) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-06-03 08:01:48 +00:00
Yong Hoon Shin 1e123529d7 [Misc] Fix estimated max model len msg (#18966) 
Signed-off-by: Yong Hoon Shin <yhshin@meta.com>
 2025-05-31 16:43:44 +08:00
Nick Hill 2dbe8c0774 [Perf] API-server scaleout with many-to-many server-engine comms (#17546) 2025-05-30 08:17:00 -07:00
Woosuk Kwon 6825d9a998 [BugFix][Spec Decode] Improve Prefix Caching Logic in Speculative Decoding (#18668) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-05-24 17:33:46 -07:00
Robert Shaw 2b10ba7491 [Bugfix][Nixl] Fix Preemption Bug (#18631) 
Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>
 2025-05-23 23:30:16 +00:00
Chen Zhang 6550114c9c [v1] Redo "Support multiple KV cache groups in GPU model runner (#17945)" (#18593) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-05-23 09:39:47 -07:00
Chaunceyandsimon-mo b046cf792d [Feature][V1]: suupports cached_tokens in response usage (#18149) 
Co-authored-by: simon-mo <xmo@berkeley.edu>
 2025-05-23 01:41:03 -07:00
Mark McLoughlin bb0a311213 Revert "[v1] Support multiple KV cache groups in GPU model runner (#17945) (#18459) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2025-05-21 10:25:23 -07:00
Nick HillandNicolò Lucchesi 1b15df2546 [BugFix] Fix handling of num_computed_tokens with connector (#18232) 
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Nicolò Lucchesi <nicolo.lucchesi@gmail.com>
 2025-05-19 09:03:25 -07:00
Chen Zhang e60f550b38 [v1] Support multiple KV cache groups in GPU model runner (#17945) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-05-14 18:54:54 -07:00
Aaron PhamandRussell Bryant 2fc9075b82 [V1] Structured Outputs + Thinking compatibility (#16577) 
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
Co-authored-by: Russell Bryant <rbryant@redhat.com>
 2025-05-14 15:45:24 -07:00
Nick Hill 59dd311cf5 [KVConnector] Keep KVTransferParams as a dict (#18033) 2025-05-14 08:05:57 -07:00
Chen Zhang f2ae883b67 [v1][KVCacheManager] pass num_new_computed_tokens to kv cache manager (#18001) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-05-13 19:09:39 -07:00
Chen ZhangandWoosuk Kwon f0d610a8ae [v1][KVCacheManager] Avoid full cache hit by controlling max_length (#17999) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-05-13 06:50:38 +00:00
Robert Shaw 195adb47c0 [Chore] Remove unused method (#18024) 
Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>
 2025-05-12 13:59:47 -07:00
Chen Zhang 302f3aca7e [v1][KVCacheManager] Change prefix caching metric from counting blocks to counting tokens (#18003) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-05-12 13:46:12 -07:00
Robert ShawTyler Michael SmithApostaCRobert ShawmgoinNick HillTyler Michael SmithBrent Salisbury
 		d19110204c [P/D] NIXL Integration (#17751) 
Signed-off-by: ApostaC <yihua98@uchicago.edu>
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>
Signed-off-by: Robert Shaw <rshaw@neuralmagic.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Signed-off-by: Brent Salisbury <bsalisbu@redhat.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
Co-authored-by: ApostaC <yihua98@uchicago.edu>
Co-authored-by: Robert Shaw <rshaw@neuralmagic.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Tyler Michael Smith <tysmith@redhat.com>
Co-authored-by: Brent Salisbury <bsalisbu@redhat.com>
 2025-05-12 09:46:16 -07:00
Chen Zhang ca66a1674c [v1] Rename specialized_manager.py to single_type_kv_cache_manager.py (#17946) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-05-10 16:14:12 -07:00
Chen Zhang 200da9a517 [v1] Move block management logic from KVCacheManager to SpecializedManager (#17474) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-05-09 15:25:34 +00:00
Ning Xie d310e6de98 [BUGFIX]: return fast when request requires prompt logprobs (#17251) 2025-05-08 21:25:41 -07:00
Wallas Henrique d43f914d42 [Core][Feature] Input metadata dump on crash (#13407) 
Signed-off-by: Wallas Santos <wallashss@ibm.com>
 2025-05-07 22:15:09 +00:00
Chen Zhang aabcd2cae3 [v1] Introduce KVCacheBlocks as interface between Scheduler and KVCacheManager (#17479) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-05-06 08:50:34 -07:00
Harry Mellor d6484ef3c3 Add full API docs and improve the UX of navigating them (#17485) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-05-03 19:42:43 -07:00
Robert Shaw c777df79f7 [BugFix] Fix Memory Leak (#17567) 
Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>
 2025-05-02 01:07:03 -07:00
Chen Zhang 81ecf425f0 [v1][Spec Decode] Make sliding window compatible with eagle prefix caching (#17398) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-04-30 18:25:53 +00:00
AlecandMark McLoughlin 0be6d05b5e [V1][Metrics] add support for kv event publishing (#16750) 
Signed-off-by: alec-flowers <aflowers@nvidia.com>
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
Co-authored-by: Mark McLoughlin <markmc@redhat.com>
 2025-04-30 07:44:45 -07:00
Marko Rosenmueller 77073c77bc [Core] Prevent side-channel attacks via cache salting (#17045) 
Signed-off-by: Marko Rosenmueller <5467316+dr75@users.noreply.github.com>
 2025-04-30 20:27:21 +08:00
Benjamin Chislett 34120f5acd [V1][Feature] Enable Speculative Decoding with Structured Outputs (#14702) 
Signed-off-by: Benjamin Chislett <benjamin.chislett@centml.ai>
Signed-off-by: Benjamin Chislett <chislett.ben@gmail.com>
 2025-04-30 00:02:10 +00:00
Lily Liu 20e489eaa1 [V1][Spec Decode] Make eagle compatible with prefix caching. (#17137) 
Signed-off-by: LiuXiaoxuanPKU <lilyliupku@gmail.com>
 2025-04-27 09:29:43 -07:00
Ning Xie fd11a325b8 [MISC] rename interval to max_recent_requests (#14285) 2025-04-26 16:59:18 +00:00
Ning Xie dc2ceca5c5 [BUGFIX] use random for NONE_HASH only when PYTHONHASHSEED not set (#17088) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-04-26 14:34:24 +00:00
Nick Hill df6f3ce883 [Core] Remove prompt string from engine core data structures (#17214) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-04-25 23:41:05 -07:00
Woosuk Kwon 1cf0719ebd [Minor][Spec Decode] Add use_eagle to SpeculativeConfig (#17213) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-04-25 21:08:15 -07:00
Benjamin ChislettandBryan Lu a0e619e62a [V1][Spec Decode] EAGLE-3 Support (#16937) 
Signed-off-by: Bryan Lu <yuzhelu@amazon.com>
Signed-off-by: Benjamin Chislett <benjamin.chislett@centml.ai>
Co-authored-by: Bryan Lu <yuzhelu@amazon.com>
 2025-04-25 15:43:07 -07:00
Mark McLoughlin 340d7b1b21 [V1][Spec Decoding] Add num_drafts and num_accepted_tokens_per_position metrics (#16665) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2025-04-24 08:57:40 -07:00
Rui Qiao c0dfd97519 [V1][PP] Optimization: continue scheduling prefill chunks (#17080) 
Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
 2025-04-24 05:27:08 -07:00
Woosuk Kwon c4ab9f3e71 [V1] Remove pre-allocation for KV cache (#16941) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-04-22 00:52:18 -07:00