iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
6,768 Commits 3 Branches 0 Tags
6881107948c00a8564bc2fa85308f6fc2f065d64
Commit Graph
124 Commits
Author SHA1 Message Date
Andrew Sansom临景Bryce1010Nan2018Cyrus LeungDarkLight1337
 		cc2a77d7f1 [Core] [Bugfix] Add Input Embeddings (#15428) 
Signed-off-by: Andrew Sansom <andrew@protopia.ai>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: 临景 <linjing.yx@alibaba-inc.com>
Co-authored-by: Bryce1010 <bryceyx@gmail.com>
Co-authored-by: Nan2018 <nan@protopia.ai>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-05-02 01:06:39 -07:00
Cyrus Leung aec9674dbe [Core] Remove legacy input mapper/processor from V0 (#15686) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-04-28 15:38:48 +08:00
maobaolong 26dd972adb [FEAT]Support reset prefix cache by specified device (#15003) 2025-03-19 10:54:41 -07:00
courage17340 caac5c2e59 [Bugfix][Core] fix abort_seq_group and memory leak when n>1 (#14326) 
Signed-off-by: courage17340 <courage17340@163.com>
 2025-03-06 23:59:32 +08:00
Joe RundePrashant GuptaCody Yu
 		3bcb8c75da [Core] Reduce TTFT with concurrent partial prefills (#10235) 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
Signed-off-by: Prashant Gupta <prashantgupta@us.ibm.com>
Co-authored-by: Prashant Gupta <prashantgupta@us.ibm.com>
Co-authored-by: Cody Yu <hao.yu.cody@gmail.com>
 2025-02-14 15:36:07 -08:00
Wang Ran (汪然) 40932d7a05 [Misc] Remove redundant statements in scheduler.py (#13229) 2025-02-13 22:07:25 -08:00
Russell Bryant e489ad7a21 [Misc] Add SPDX-License-Identifier headers to python source files (#12628) 
- **Add SPDX license headers to python source files**
- **Check for SPDX headers using pre-commit**

commit 9d7ef44c3cfb72ca4c32e1c677d99259d10d4745
Author: Russell Bryant <rbryant@redhat.com>
Date:   Fri Jan 31 14:18:24 2025 -0500

    Add SPDX license headers to python source files
    
This commit adds SPDX license headers to python source files as
recommended to
the project by the Linux Foundation. These headers provide a concise way
that is
both human and machine readable for communicating license information
for each
source file. It helps avoid any ambiguity about the license of the code
and can
    also be easily used by tools to help manage license compliance.
    
The Linux Foundation runs license scans against the codebase to help
ensure
    we are in compliance with the licenses of the code we use, including
dependencies. Having these headers in place helps that tool do its job.
    
    More information can be found on the SPDX site:
    
    - https://spdx.dev/learn/handling-license-info/
    
    Signed-off-by: Russell Bryant <rbryant@redhat.com>

commit 5a1cf1cb3b80759131c73f6a9dddebccac039dea
Author: Russell Bryant <rbryant@redhat.com>
Date:   Fri Jan 31 14:36:32 2025 -0500

    Check for SPDX headers using pre-commit
    
    Signed-off-by: Russell Bryant <rbryant@redhat.com>

---------

Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-02-02 11:58:18 -08:00
Harry Mellor 823ab79633 Update pre-commit hooks (#12475) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-01-27 17:23:08 -07:00
Cody Yu 7206ce4ce1 [Core] Support reset_prefix_cache (#12284) 2025-01-22 18:52:27 +00:00
Kathy Yu 2f385183f3 [Bugfix] Free cross attention block table for preempted-for-recompute sequence group. (#10013) 
Signed-off-by: Kathy Yu <feiyangyu@google.com>
 2025-01-02 10:28:09 -08:00
Cyrus Leung 8f10d5e393 [Misc] Split up pooling tasks (#10820) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-12-11 01:28:00 -08:00
Aurick Qiao d5c5154fcf [Misc] LoRA + Chunked Prefill (#9057) 2024-12-11 10:09:20 +08:00
Murali AndoorveeduandSourashis Roy db66e018ea [Bugfix] Fix for Spec model TP + Chunked Prefill (#10232) 
Signed-off-by: andoorve <37849411+andoorve@users.noreply.github.com>
Signed-off-by: Sourashis Roy <sroy@roblox.com>
Co-authored-by: Sourashis Roy <sroy@roblox.com>
 2024-11-26 09:11:16 -08:00
Maximilien de BayserandFlavia Beo 214efc2c3c Support Cross encoder models (#10400) 
Signed-off-by: Max de Bayser <maxdebayser@gmail.com>
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
Signed-off-by: Flavia Beo <flavia.beo@ibm.com>
Co-authored-by: Flavia Beo <flavia.beo@ibm.com>
 2024-11-24 18:56:20 -08:00
Ricky Xu 4634a89d18 Prefix Cache Aware Scheduling [1/n] (#10128) 
Signed-off-by: rickyx <rickyx@anyscale.com>
 2024-11-22 21:15:55 -08:00
Nicolò Lucchesi 9d43afcc53 [Feature] [Spec decode]: Combine chunked prefill with speculative decoding (#9291) 
Signed-off-by: NickLucche <nlucches@redhat.com>
 2024-11-07 08:15:14 -08:00
Peter Salas 6c0b7f548d [Core][VLM] Add precise multi-modal placeholder tracking (#8346) 
Signed-off-by: Peter Salas <peter@fixie.ai>
 2024-11-01 16:21:10 -07:00
André Jonasson 4581d2cc02 [Core] Refactor: Clean up unused argument in Scheduler._preempt (#9696) 
Signed-off-by: André Jonasson <andre.jonasson@gmail.com>
 2024-11-01 11:41:38 -07:00
youkaichaoandZhuohan Li 4fdc581f9e [core] simplify seq group code (#9569) 
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
 2024-10-24 00:16:44 -07:00
Cyrus Leung 051eaf6db3 [Model] Add user-configurable task for models that support both generation and embedding (#9424) 2024-10-18 11:31:58 -07:00
Kuntai Du 81ede99ca4 [Core] Deprecating block manager v1 and make block manager v2 default (#8704) 
Removing the block manager v1. This is the initial piece of prefix-caching-centric design. In order to achieve prefix-caching-centric design, we need to simplify the code path so that we only use v2 block manager (which has much higher performance on prefix caching).
 2024-10-17 11:38:15 -05:00
Russell Bryant 776dbd74f1 [CI/Build] mypy: Resolve some errors from checking vllm/engine (#9267) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2024-10-16 22:55:59 +00:00
Tyler Michael Smith 7342a7d7f8 [Model] Support Mamba (#6484) 2024-10-11 15:40:06 +00:00
youkaichaoandBrendan Wong cbc2ef5529 [misc] hide best_of from engine (#9261) 
Co-authored-by: Brendan Wong <bjwpokemon@gmail.com>
 2024-10-10 21:30:44 -07:00
Alex Brooks a3691b6b5e [Core][Frontend] Add Support for Inference Time mm_processor_kwargs (#9131) 
Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
 2024-10-08 14:12:56 +00:00
youkaichao fa45513a51 [misc] fix comment and variable name (#9139) 2024-10-07 16:07:05 -07:00
youkaichao 18b296fdb2 [core] remove beam search from the core (#9105) 2024-10-07 05:47:04 +00:00
afeldman-nmVarun Sundar RabindranathAndrew Feldman
 		563649aafe [Core] Combined support for multi-step scheduling, chunked prefill & prefix caching (#8804) 
Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com>
Co-authored-by: Andrew Feldman <afeld2012@gmail.com>
 2024-10-02 07:52:20 +00:00
Varun Sundar RabindranathandVarun Sundar Rabindranath c2ec430ab5 [Core] Multi-Step + Single Step Prefills via Chunked Prefill code path (#8378) 
Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com>
 2024-09-27 13:32:07 -07:00
Woo-Yeon Lee 8fae5ed7f6 [Misc] Fix minor typo in scheduler (#8765) 2024-09-25 00:53:03 -07:00
Archit Patke 6da1ab6b41 [Core] Adding Priority Scheduling (#5958) 2024-09-24 19:50:50 -07:00
Alexander Matveev 4ef41b8476 [Bugfix] Fix async postprocessor in case of preemption (#8267) 2024-09-07 21:01:51 -07:00
wang.yuqi 6e36f4fa6c improve chunked prefill performance 
[Bugfix] Fix #7592 vllm 0.5.4 enable_chunked_prefill throughput is slightly lower than 0.5.3~0.5.0. (#7874)
 2024-09-02 14:20:12 -07:00
Alexander Matveev 3f60f2244e [Core] Combine async postprocessor and multi-step (#7921) 2024-08-29 11:18:26 -07:00
Cody Yu e3580537a4 [Performance] Enable chunked prefill and prefix caching together (#7753) 2024-08-28 00:36:31 -07:00
Alexander Matveev f508e03e7f [Core] Async_output_proc: Add virtual engine support (towards pipeline parallel) (#7911) 2024-08-28 00:02:30 -07:00
youkaichao bc6e42a9b1 [hardware][rocm] allow rocm to override default env var (#7926) 2024-08-27 19:50:06 -07:00
Jonathan Berkhahn 9c71c97ae2 [mypy] Enable mypy type checking for vllm/core (#7229) 2024-08-28 07:11:14 +08:00
Megha AgarwalandAlexander Matveev 2eedede875 [Core] Asynchronous Output Processor (#7049) 
Co-authored-by: Alexander Matveev <alexm@neuralmagic.com>
 2024-08-26 20:53:20 -07:00
Cody Yu 3ac50b47d0 [MISC] Add prefix cache hit rate to metrics (#7606) 2024-08-19 11:52:07 -07:00
SangBin Cho ff7ec82c4d [Core] Optimize SPMD architecture with delta + serialization optimization (#7109) 2024-08-18 17:57:20 -07:00
Mahesh Keralapura 93478b63d2 [Core] Fix tracking of model forward time in case of PP>1 (#7440) 
[Core] Fix tracking of model forward time to the span traces in case of PP>1 (#7440)
 2024-08-16 13:46:01 -07:00
William Lin 2ecf7b1757 [core] [3/N] multi-step args and sequence.py (#7452) 2024-08-14 12:32:45 -07:00
Mahesh Keralapura 933790c209 [Core] Add span metrics for model_forward, scheduler and sampler time (#7089) 2024-08-09 13:55:13 -07:00
Alexander Matveev e02ac55617 [Performance] Optimize e2e overheads: Reduce python allocations (#7162) 2024-08-08 21:34:28 -07:00
Rui Qiao 746709642c [Misc] Fix typos in scheduler.py (#7285) 
Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
 2024-08-07 17:06:01 -07:00
afeldman-nmAndrew FeldmanNick Hill
 		fd95e026e0 [Core] Subclass ModelRunner to support cross-attention & encoder sequences (towards eventual encoder/decoder model support) (#4942) 
Co-authored-by: Andrew Feldman <afeld2012@gmail.com>
Co-authored-by: Nick Hill <nickhill@us.ibm.com>
 2024-08-06 16:51:47 -04:00
youkaichao c8a7e93273 [core][scheduler] simplify and improve scheduler (#6867) 2024-07-31 23:51:09 -07:00
youkaichaoandWoosuk Kwon 6ca8031e71 [core][misc] improve free_finished_seq_groups (#6865) 
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2024-07-30 14:32:12 -07:00
Nick Hill 5cf9254a9c [BugFix] Fix use of per-request seed with pipeline parallel (#6698) 2024-07-30 10:40:08 -07:00