iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
6,768 Commits 3 Branches 0 Tags
6881107948c00a8564bc2fa85308f6fc2f065d64
Commit Graph
73 Commits
Author SHA1 Message Date
Cyrus Leung 7d9216495c [Doc] Update references to doc files (#18637) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-05-23 15:49:21 -07:00
Noah Yoshida 13cf6b6236 [BugFix] fix speculative decoding memory leak when speculation is disabled (#15506) 
Signed-off-by: Noah Yoshida <noahcy117@gmail.com>
 2025-04-30 23:28:17 -07:00
Woosuk Kwon b411418ff0 [Chore] Remove Sampler from Model Code (#17084) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-04-24 02:49:33 -07:00
shangmingc 50c9636d87 [V1][Usage] Refactor speculative decoding configuration and tests (#14434) 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
 2025-03-22 19:28:10 -10:00
Bryan Lu 9ed6ee92d6 [Bugfix] EAGLE output norm bug (#14464) 
Signed-off-by: Bryan Lu <yuzhelu@amazon.com>
 2025-03-15 06:50:33 +00:00
Benjamin Chislett 9804145cac [Model][Speculative Decoding] Expand DeepSeek MTP code to support k > n_predict (#13626) 
Signed-off-by: Benjamin Chislett <benjamin.chislett@centml.ai>
 2025-02-27 15:28:08 -08:00
Jee Jee Li 5157338ed9 [Misc] Improve LoRA spelling (#13831) 2025-02-25 23:43:01 -08:00
Simon Mo 8c755c3b6d [bugfix] spec decode worker get tp group only when initialized (#13578) 2025-02-20 04:46:28 +00:00
shangmingc 5ae9f26a5a [Bugfix] Fix device ordinal for multi-node spec decode (#13269) 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
 2025-02-19 22:13:15 +08:00
Lucia FangandLiuXiaoxuanPKU f525c0be8b [Model][Speculative Decoding] DeepSeek MTP spec decode (#12755) 
Signed-off-by: Lu Fang <fanglu@fb.com>
Co-authored-by: LiuXiaoxuanPKU <lilyliupku@gmail.com>
 2025-02-19 17:06:23 +08:00
shangmingc 46cdd59577 [Feature][Spec Decode] Simplify the use of Eagle Spec Decode (#12304) 
Signed-off-by: Shangming Cai <caishangming@linux.alibaba.com>
 2025-02-16 19:32:26 -08:00
Russell Bryant e489ad7a21 [Misc] Add SPDX-License-Identifier headers to python source files (#12628) 
- **Add SPDX license headers to python source files**
- **Check for SPDX headers using pre-commit**

commit 9d7ef44c3cfb72ca4c32e1c677d99259d10d4745
Author: Russell Bryant <rbryant@redhat.com>
Date:   Fri Jan 31 14:18:24 2025 -0500

    Add SPDX license headers to python source files
    
This commit adds SPDX license headers to python source files as
recommended to
the project by the Linux Foundation. These headers provide a concise way
that is
both human and machine readable for communicating license information
for each
source file. It helps avoid any ambiguity about the license of the code
and can
    also be easily used by tools to help manage license compliance.
    
The Linux Foundation runs license scans against the codebase to help
ensure
    we are in compliance with the licenses of the code we use, including
dependencies. Having these headers in place helps that tool do its job.
    
    More information can be found on the SPDX site:
    
    - https://spdx.dev/learn/handling-license-info/
    
    Signed-off-by: Russell Bryant <rbryant@redhat.com>

commit 5a1cf1cb3b80759131c73f6a9dddebccac039dea
Author: Russell Bryant <rbryant@redhat.com>
Date:   Fri Jan 31 14:36:32 2025 -0500

    Check for SPDX headers using pre-commit
    
    Signed-off-by: Russell Bryant <rbryant@redhat.com>

---------

Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-02-02 11:58:18 -08:00
Mengqing Cao dd66fd2b01 [CI] fix pre-commit error (#12494) 
Signed-off-by: Mengqing Cao <cmq0113@163.com>
 2025-01-28 06:11:05 +00:00
Harry Mellor 823ab79633 Update pre-commit hooks (#12475) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-01-27 17:23:08 -07:00
Nicolò Lucchesiandwallashss 6116ca8cd7 [Feature] [Spec decode]: Enable MLPSpeculator/Medusa and prompt_logprobs with ChunkedPrefill (#10132) 
Signed-off-by: NickLucche <nlucches@redhat.com>
Signed-off-by: wallashss <wallashss@ibm.com>
Co-authored-by: wallashss <wallashss@ibm.com>
 2025-01-27 13:38:35 -08:00
Cyrus Leung 59a0192fb9 [Core] Interface for accessing model from VllmRunner (#10353) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-01-20 15:00:59 +08:00
youkaichao ad34c0df0f [core] platform agnostic executor via collective_rpc (#11256) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2025-01-15 13:45:21 +08:00
Cyrus Leung ee77fdb5de [Doc][2/N] Reorganize Models and Usage sections (#11755) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-01-06 21:40:31 +08:00
Rafael Vasquez 32aa2059ad [Docs] Convert rST to MyST (Markdown) (#11145) 
Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>
 2024-12-23 22:35:38 +00:00
Cyrus Leung c889d5888b [Doc] Explicitly state that PP isn't compatible with speculative decoding yet (#10975) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-12-07 17:20:49 +00:00
Cyrus Leung aa39a8e175 [Doc] Create a new "Usage" section (#10827) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-12-05 11:19:35 +08:00
Chendi.Xue 0a71900bc9 Remove hard-dependencies of Speculative decode to CUDA workers (#10587) 
Signed-off-by: Chendi Xue <chendi.xue@intel.com>
 2024-11-26 17:57:11 -08:00
Murali AndoorveeduandSourashis Roy db66e018ea [Bugfix] Fix for Spec model TP + Chunked Prefill (#10232) 
Signed-off-by: andoorve <37849411+andoorve@users.noreply.github.com>
Signed-off-by: Sourashis Roy <sroy@roblox.com>
Co-authored-by: Sourashis Roy <sroy@roblox.com>
 2024-11-26 09:11:16 -08:00
Nicolò Lucchesi 9d43afcc53 [Feature] [Spec decode]: Combine chunked prefill with speculative decoding (#9291) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2024-11-07 08:15:14 -08:00
youkaichao 2094062b4e [4.5/N] bugfix for quant config in speculative decode (#10007) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2024-11-04 15:11:59 -08:00
youkaichaoandNick Hill e893795443 [2/N] executor pass the complete config to worker/modelrunner (#9938) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2024-11-02 07:35:05 -07:00
科英 67a6882da4 [Misc] SpecDecodeWorker supports profiling (#9719) 
Signed-off-by: Abatom <abatom@163.com>
 2024-10-27 04:18:03 +00:00
Thomas Parnell 496e991da8 [Doc] Consistent naming of attention backends (#9498) 
Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>
 2024-10-21 22:29:57 +08:00
Lily Liu 89feb4c84d [SpecDec] Remove Batch Expansion (2/3) (#9298) 2024-10-12 05:13:37 +00:00
Wallas Henrique 8baf85e4e9 [Doc] Compatibility matrix for mutual exclusive features (#8512) 
Signed-off-by: Wallas Santos <wallashss@ibm.com>
 2024-10-11 11:18:50 -07:00
Lily Liu 1570203864 [Spec Decode] (1/2) Remove batch expansion (#8839) 2024-10-01 16:04:42 -07:00
Travis Johnson 01b6f9e1f0 [Core][Bugfix] Support prompt_logprobs returned with speculative decoding (#8047) 
Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
 2024-09-24 17:29:56 -07:00
Lily Liu c6bd70d772 [SpecDec][Misc] Cleanup, remove bonus token logic. (#8701) 2024-09-22 12:34:14 -07:00
Lily Liu e6a26ed037 [SpecDecode][Kernel] Flashinfer Rejection Sampling (#7244) 2024-09-01 21:23:29 -07:00
afeldman-nm 428dd1445e [Core] Logprobs support in Multi-step (#7652) 2024-08-29 19:19:08 -07:00
Nick Hill 1856aff4d6 [Spec Decoding] Streamline batch expansion tensor manipulation (#7851) 2024-08-25 15:45:14 -07:00
Abhinav Goyal a3fce56b88 [Speculative Decoding] EAGLE Implementation with Top-1 proposer (#6830) 2024-08-22 02:42:24 -07:00
Abhinav Goyal 312f761232 [Speculative Decoding] Fixing hidden states handling in batch expansion (#7508) 2024-08-19 17:58:14 -07:00
William Lin 57b7be0e1c [Speculative decoding] [Multi-Step] decouple should_modify_greedy_probs_inplace (#6971) 2024-08-09 05:42:45 +00:00
Cade Daniel 82a1b1a82b [Speculative decoding] Add periodic log with time spent in proposal/scoring/verification (#6963) 2024-08-05 08:46:44 +00:00
Nick Hill 5cf9254a9c [BugFix] Fix use of per-request seed with pipeline parallel (#6698) 2024-07-30 10:40:08 -07:00
Allen.Dou 40468b13fa [Bugfix] Miscalculated latency lead to time_to_first_token_seconds inaccurate. (#6686) 2024-07-24 08:58:42 -07:00
sroy745 14f91fe67c [Spec Decode] Disable Log Prob serialization to CPU for spec decoding for both draft and target models. (#6485) 2024-07-20 23:58:58 -07:00
Woo-Yeon Lee a921e86392 [BUGFIX] Raise an error for no draft token case when draft_tp>1 (#6369) 2024-07-19 06:01:09 -07:00
Thomas ParnellandNick Hill d4201e06d5 [Bugfix] Make spec. decode respect per-request seed. (#6034) 
Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>
Co-authored-by: Nick Hill <nickhill@us.ibm.com>
 2024-07-18 19:22:08 -07:00
shangmingcandcaishangming.csm a19e8d3726 [Misc][Speculative decoding] Typos and typing fixes (#6467) 
Co-authored-by: caishangming.csm <caishangming.csm@alibaba-inc.com>
 2024-07-17 07:17:07 +00:00
sroy745 ae151d73be [Speculative Decoding] Enabling bonus token in speculative decoding for KV cache based models (#5765) 2024-07-10 16:02:47 -07:00
Abhinav Goyal 2416b26e11 [Speculative Decoding] Medusa Implementation with Top-1 proposer (#4978) 2024-07-09 18:34:02 -07:00
Sirej DuaSirej DuaSirej Dua <Sirej Dua>
 		15aba081f3 [Speculative Decoding] MLPSpeculator Tensor Parallel support (1/2) (#6050) 
Co-authored-by: Sirej Dua <sirej.dua@databricks.com>
Co-authored-by: Sirej Dua <Sirej Dua>
 2024-07-02 07:20:29 -07:00
sroy745 80ca1e6a3a [Speculative Decoding 2/2 ] Integrate typical acceptance sampler into Spec Decode Worker (#5348) 2024-07-01 00:33:05 -07:00