iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
14,386 Commits 3 Branches 0 Tags
fa6a6be51978bd4b49ba0da17039e60f96dc5b13
Commit Graph
47 Commits
Author SHA1 Message Date
Fynn Schmitt-Ulms 9433acb8df [Spec Decode] Add hidden states extraction system (#33736) 
Signed-off-by: Fynn Schmitt-Ulms <fschmitt@redhat.com>
 2026-03-02 14:29:09 -05:00
Nick Hill 876a16f4fb [ModelRunner V2] Fix spec decoding + logprobs (#33391) 
Signed-off-by: Nick Hill <nickhill123@gmail.com>
 2026-01-31 03:33:26 +00:00
zhrrr 97ba96fbe9 [perf][async] support non cpu sync get logprob tensors for spec (#31336) 
Signed-off-by: izhuhaoran <izhuhaoran@qq.com>
Signed-off-by: zhuhaoran <zhuhaoran.zhr@alibaba-inc.com>
 2026-01-09 21:24:51 +00:00
Cyrus Leung c8ed39b9dd [Model] Reorganize pooling layers (#31973) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-01-09 11:02:14 +00:00
Cyrus Leung d1b6fe007f [Chore] Further cleanup pooler (#31951) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-01-08 02:16:21 -08:00
Cyrus Leung b7036c87a1 [Refactor] Clean up pooler modules (#31897) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2026-01-08 00:07:43 +08:00
Nick Hill 6c2cfb62ff [BugFix] Fix async scheduling for pooling models (#31584) 
Signed-off-by: njhill <nickhill123@gmail.com>
 2025-12-31 14:48:51 -08:00
Martin Hickey f4417f8449 [KVConnector] Add KV events to KV Connectors (#28309) 
Signed-off-by: Martin Hickey <martin.hickey@ie.ibm.com>
 2025-12-11 15:30:29 +01:00
wang.yuqiandgemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> 74c4d80c6c [Model][6/N] Improve all pooling task | Support chunked prefill with ALL pooling (#27145) 
Signed-off-by: wang.yuqi <noooop@126.com>
Signed-off-by: wang.yuqi <yuqi.wang@daocloud.io>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
 2025-12-04 13:44:15 +00:00
Yong Hoon Shin 69520bc695 Add logging for cudagraph related info (#29825) 
Signed-off-by: Yong Hoon Shin <yhshin@meta.com>
 2025-12-03 01:01:48 -08:00
Nick Hill 8e7a891602 [BugFix] Fix spec decoding max_tokens scheduling perf issue (#29542) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-11-28 20:52:23 +08:00
Jialin Ouyang 30b9c67743 Revert "[Redo] #26368 (#28771)" (#29121) 
Signed-off-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>
 2025-11-20 21:27:45 -08:00
Cyrus LeungandJialin Ouyang 98b4d389ed [Redo] #26368 (#28771) 
Signed-off-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>
 2025-11-14 22:47:41 -08:00
Nick Hill ac86bff8cb Revert "[Core] Performance: Use list[np.ndarray] instead of list[list… (#28773) 2025-11-14 20:24:00 -08:00
Jialin Ouyang 186352b270 [Core] Performance: Use list[np.ndarray] instead of list[list[int]] for output tokens for GC optimization (#26368) 
Signed-off-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>
 2025-11-14 16:04:04 -08:00
Chenguang Zhengn00909098knlnguyen1802herotai214gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>Khuong LeKhuong Le
 		4ccffe561f [Core] Encoder separation for Encode-Prefill-Decode Disaggregation (#25233) 
Signed-off-by: n00909098 <nguyen.kha.long@huawei.com>
Signed-off-by: knlnguyen1802 <knlnguyen1802@gmail.com>
Signed-off-by: herotai214 <herotai214@gmail.com>
Signed-off-by: Khuong Le <khuong.le.manh@huawei.com>
Signed-off-by: Khuong Le <lemanhkhuong2611@gmail.com>
Co-authored-by: n00909098 <nguyen.kha.long@huawei.com>
Co-authored-by: knlnguyen1802 <knlnguyen1802@gmail.com>
Co-authored-by: herotai214 <herotai214@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Khuong Le <khuong.le.manh@huawei.com>
Co-authored-by: Khuong Le <lemanhkhuong2611@gmail.com>
 2025-11-11 18:58:33 -08:00
Jialin Ouyang 4228be7959 [Perf] Use np.ndarray instead of list[list[int]] to reduce GC overhead (#28245) 
Signed-off-by: Jialin Ouyang <Jialin.Ouyang@gmail.com>
 2025-11-11 10:28:47 -08:00
usberkeley 4a8d6bd168 Fix cu_num_generated_tokens slicing logic in LogprobsLists.slice() method (#28214) 
Signed-off-by: Bradley <bradley.b.pitt@gmail.com>
 2025-11-09 19:11:46 +00:00
Nick Hill 4fe5895361 [AsyncScheduling] Make async overlap work with logprobs (#27615) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-10-28 22:35:54 +00:00
Giancarlo DelfinandNick Hill 6644796bf4 [V1][spec decode] return logprobs for spec decoding (#26060) 
Signed-off-by: Giancarlo Delfin <gdelfin@meta.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-10-22 22:59:59 -07:00
Nicolò Lucchesi 4dfdb821c8 [P/D] Dynamic kv_output_aggregator collect size (#26734) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2025-10-22 18:07:58 +02:00
Harry Mellor 8fcaaf6a16 Update Optional[x] -> x | None and Union[x, y] to x | y (#26633) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-10-12 09:51:31 -07:00
Harry Mellor d6953beb91 Convert formatting to use ruff instead of yapf + isort (#26247) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-10-05 07:06:22 -07:00
David Ben-DavidandDavid Ben-David 9a9f48dff7 [V1] [P/D] Add Support for KV Load Failure Recovery (#19330) 
Signed-off-by: David Ben-David <davidb@pliops.com>
Co-authored-by: David Ben-David <davidb@pliops.com>
 2025-09-30 14:57:08 -07:00
Cyrus Leung 755ed7b05b [Misc] Simplify PoolerOutput and move to v1/outputs (#25629) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-09-25 06:47:03 +00:00
Nicolò Lucchesi a3d087adec [P/D][Nixl] Introduce KVTransferMetrics and aggregation strategy (#22188) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2025-09-19 11:09:14 +00:00
Benjamin Chislett cee182b297 [Perf][V1] Fully overlap model execution (#23569) 
Signed-off-by: Benjamin Chislett <benjamin.chislett@centml.ai>
 2025-09-05 18:20:17 -07:00
Woosuk Kwon c9b38be8aa [Spec Decode] Make propose_draft_token_ids non-blocking for lower TTFT (#23041) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-08-18 17:20:38 -07:00
David Ben-DavidandDavid Ben-David aefeea0fde [V1] [P/D] Refactor KV Connector Path (#21980) 
Signed-off-by: David Ben-David <davidb@pliops.com>
Co-authored-by: David Ben-David <davidb@pliops.com>
 2025-08-03 04:03:40 -07:00
Kuntai Du b18b417fbf Revert "[V1] Exception Handling when Loading KV Cache from Remote Store" (#21778) 
Signed-off-by: KuntaiDu <kuntai@uchicago.edu>
 2025-07-28 20:15:18 +00:00
Adelineandliuyumoye 15a72ac478 [V1] Exception Handling when Loading KV Cache from Remote Store (#21534) 
Signed-off-by: liuyumoye <adeline_ly2023@outlook.com>
Co-authored-by: liuyumoye <adeline_ly2023@outlook.com>
 2025-07-27 20:34:17 -07:00
Vlad Tiberiu Mihailescu 2e3e3c86dc Export NaNs in logits to scheduler_stats if output is corrupted (#18777) 
Signed-off-by: Vlad Mihailescu <vtmihailescu@gmail.com>
 2025-06-20 22:47:16 +08:00
Maximilien de Bayserand22quinn 799397ee4f Support embedding models in V1 (#16188) 
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
Signed-off-by: Max de Bayser <maxdebayser@gmail.com>
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
Co-authored-by: 22quinn <33176974+22quinn@users.noreply.github.com>
 2025-06-18 21:36:33 -07:00
Simon Mo 02f0c7b220 [Misc] Add SPDX-FileCopyrightText (#19100) 
Signed-off-by: simon-mo <simon.mo@hey.com>
 2025-06-03 11:20:17 -07:00
Robert ShawTyler Michael SmithApostaCRobert ShawmgoinNick HillTyler Michael SmithBrent Salisbury
 		d19110204c [P/D] NIXL Integration (#17751) 
Signed-off-by: ApostaC <yihua98@uchicago.edu>
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>
Signed-off-by: Robert Shaw <rshaw@neuralmagic.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Signed-off-by: Brent Salisbury <bsalisbu@redhat.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
Co-authored-by: ApostaC <yihua98@uchicago.edu>
Co-authored-by: Robert Shaw <rshaw@neuralmagic.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Tyler Michael Smith <tysmith@redhat.com>
Co-authored-by: Brent Salisbury <bsalisbu@redhat.com>
 2025-05-12 09:46:16 -07:00
Nick Hill 3aee6573dc [V1] Aggregate chunked prompt logprobs in model runner (#14875) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-03-24 12:27:57 -04:00
Woosuk Kwon 99abb8b650 [V1][Spec Decode] Optimize Rejection Sampler with Triton Kernels (#14930) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-03-18 14:31:54 -07:00
Nick Hill 8ed5421aaa [V1] Eagerly remove finished requests from the batch (#14388) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-03-07 10:56:00 -08:00
Harry Mellor cf069aa8aa Update deprecated Python 3.8 typing (#13971) 2025-03-02 17:34:51 -08:00
iefgnoixandmgoin c3b6559a10 [V1][TPU] Integrate the new ragged paged attention kernel with vLLM v1 on TPU (#13379) 
Signed-off-by: Xiongfei Wei <isaacwxf23@gmail.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
 2025-02-28 11:01:36 -07:00
Woosuk Kwon cd4a72a28d [V1][Spec decode] Move drafter to model runner (#13363) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-02-17 15:40:12 -08:00
Lily Liu 80f63a3966 [V1][Spec Decode] Ngram Spec Decode (#12193) 
Signed-off-by: LiuXiaoxuanPKU <lilyliupku@gmail.com>
 2025-02-15 18:05:11 -08:00
afeldman-nmrshaw@neuralmagic.com <rshaw@neuralmagic.com>Nick Hill
 		0630d4537a [V1] Logprobs and prompt logprobs support (#9880) 
This PR is adding support for sample logprobs & prompt logprobs to vLLM v1.

New behavior:

- During model execution, model runner computes sample logprobs (if user-provided logprobs setting is not None) and prompt logprobs (if user-provided prompt_logprobs setting is not None). For both sample and prompt logprobs, the engine core returns 3 vectors: token ids, token logprob values, token ranks. Ranks reflect tokens' 1-indexed positions in the vocabulary vector after sorting the vocabulary by log probability in descending order.
- In scheduler.update_from_output(), sample and prompt logprobs are incorporated into the EngineCoreOutput data structure which is transferred to the engine client. If multiprocessing is enabled, then sample and prompt logprobs will be (de)serialized when the EngineCoreOutput data structure is (de)serialized.
- During output processing, the LogprobsProcessor transforms the triplet of token ids, token logprobs values, and token ranks into the OpenAI-compatible List[Dict[token id,Logprob]] format (for sample and prompt logprobs respectively.)
- Each Logprob instance (whether sample- or prompt-) consists of a token's log-probability, rank, and detokenized string representation. Note that logprob detokenization is handled by the LogprobsProcessor not the detokenizer.

Signed-off-by: Andrew Feldman <afeldman@neuralmagic.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Signed-off-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>


Co-authored-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-02-07 07:26:20 -08:00
Russell Bryant e489ad7a21 [Misc] Add SPDX-License-Identifier headers to python source files (#12628) 
- **Add SPDX license headers to python source files**
- **Check for SPDX headers using pre-commit**

commit 9d7ef44c3cfb72ca4c32e1c677d99259d10d4745
Author: Russell Bryant <rbryant@redhat.com>
Date:   Fri Jan 31 14:18:24 2025 -0500

    Add SPDX license headers to python source files
    
This commit adds SPDX license headers to python source files as
recommended to
the project by the Linux Foundation. These headers provide a concise way
that is
both human and machine readable for communicating license information
for each
source file. It helps avoid any ambiguity about the license of the code
and can
    also be easily used by tools to help manage license compliance.
    
The Linux Foundation runs license scans against the codebase to help
ensure
    we are in compliance with the licenses of the code we use, including
dependencies. Having these headers in place helps that tool do its job.
    
    More information can be found on the SPDX site:
    
    - https://spdx.dev/learn/handling-license-info/
    
    Signed-off-by: Russell Bryant <rbryant@redhat.com>

commit 5a1cf1cb3b80759131c73f6a9dddebccac039dea
Author: Russell Bryant <rbryant@redhat.com>
Date:   Fri Jan 31 14:36:32 2025 -0500

    Check for SPDX headers using pre-commit
    
    Signed-off-by: Russell Bryant <rbryant@redhat.com>

---------

Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-02-02 11:58:18 -08:00
Keyun Tong fa63e710c7 [V1][Perf] Reduce scheduling overhead in model runner after cuda sync (#12094) 
Signed-off-by: Keyun Tong <tongkeyun@gmail.com>
 2025-01-26 00:42:37 -08:00
Tyler Michael Smith 28b3a1c7e5 [V1] Multiprocessing Tensor Parallel Support for v1 (#9856) 
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
 2024-12-10 06:28:14 +00:00
Woosuk Kwon 6c5af09b39 [V1] Implement vLLM V1 [1/N] (#9289) 2024-10-22 01:24:07 -07:00