iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
6,768 Commits 3 Branches 0 Tags
6881107948c00a8564bc2fa85308f6fc2f065d64
Commit Graph
768 Commits
Author SHA1 Message Date
Roy 9140561059 [Minor] Fix typo and remove unused code (#2305) 2024-01-02 19:23:15 -08:00
Antoni Baum bd29cf3d3a Remove Sampler copy stream (#2209) 2023-12-20 00:04:33 -08:00
Antoni Baum a7347d9a6d Make sampler less blocking (#1889) 2023-12-17 23:03:49 +08:00
Woosuk KwonChen ShenAntoni Baum
 		37ca558103 Optimize model execution with CUDA graph (#1926) 
Co-authored-by: Chen Shen <scv119@gmail.com>
Co-authored-by: Antoni Baum <antoni.baum@protonmail.com>
 2023-12-16 21:12:08 -08:00
CHU Tianxiang 0fbfc4b81b Add GPTQ support (#916) 2023-12-15 03:04:22 -08:00
Megha AgarwalandWoosuk Kwon 6428f1d051 Support MPT with GQA (#1938) 
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2023-12-12 10:16:05 -08:00
wbnwangguoyaYang Zhao
 		dacaf5a400 Replace head_mapping params with num_kv_heads to attention kernel. (#1997) 
Co-authored-by: wangguoya <wangguoya@baidu.com>
Co-authored-by: Yang Zhao <zhaoyangstar@foxmail.com>
 2023-12-10 10:12:53 -08:00
TJianPhilipp MoritzAmir Balwelroottjtanaakuanfumiloice
 		6ccc0bfffb Merge EmbeddedLLM/vllm-rocm into vLLM main (#1836) 
Co-authored-by: Philipp Moritz <pcmoritz@gmail.com>
Co-authored-by: Amir Balwel <amoooori04@gmail.com>
Co-authored-by: root <kuanfu.liu@akirakan.com>
Co-authored-by: tjtanaa <tunjian.tan@embeddedllm.com>
Co-authored-by: kuanfu <kuanfu.liu@embeddedllm.com>
Co-authored-by: miloice <17350011+kliuae@users.noreply.github.com>
 2023-12-07 23:16:52 -08:00
Woosuk Kwon 9b294976a2 Add PyTorch-native implementation of custom layers (#1898) 2023-12-02 21:18:40 -08:00
Roy d27f4bae39 Fix rope cache key error (#1867) 2023-11-30 08:29:28 -08:00
Jee Li 63b2206ad0 Avoid multiple instantiations of the RoPE class (#1828) 2023-11-29 23:06:27 -08:00
Woosuk Kwon 27feead2f8 Refactor Worker & InputMetadata (#1843) 2023-11-29 22:16:37 -08:00
Woosuk Kwon a9e4574261 Refactor Attention (#1840) 2023-11-29 15:37:31 -08:00
Zhuohan Li 708e6c18b0 [FIX] Fix class naming (#1803) 2023-11-28 14:08:01 -08:00
Yanming W e0c6f556e8 [Build] Avoid building too many extensions (#1624) 2023-11-23 16:31:19 -08:00
ljss de23687d16 Fix repetition penalty aligned with huggingface (#1577) 2023-11-22 14:41:44 -08:00
Zhuohan Li 7d761fe3c1 [FIX] Fix the case when input_is_parallel=False for ScaledActivation (#1737) 2023-11-20 23:56:48 -08:00
Woosuk Kwon cf35d8f3d7 [BugFix] Fix TP support for AWQ (#1731) 2023-11-20 21:42:45 -08:00
ljss 819b18e7ba Rewrite torch.repeat_interleave to remove cpu synchronization (#1599) 2023-11-20 17:46:32 -08:00
Simon Mo 5ffc0d13a2 Migrate linter from pylint to ruff (#1665) 2023-11-20 11:58:01 -08:00
ljss e1054247ba [Optimization] Implement fused add rmsnorm (#1667) 2023-11-18 18:18:02 -08:00
Woosuk Kwon 8d17774f92 Add AWQ support for all models (#1714) 2023-11-18 17:56:47 -08:00
Roy e87557b069 Support Min P Sampler (#1642) 2023-11-17 16:20:49 -08:00
Zhuohan Li 7076fa1c9f TP/quantization/weight loading refactor part 2 - Refactor quantized linear logic and extend quantization support to all models (#1622) 
Refactor the tensor parallelism, quantization, and weight-loading codes.

Summary of the new features enabled by this PR:
- **All models** are able to be quantized with AWQ and SqueezeLLM, and [soon GPTQ](https://github.com/vllm-project/vllm/pull/1580).
- Model loading code became much simpler.
- Support model parallelism for all MQA/GQA models when the number of key/value heads is smaller than the tensor parallel size.
 2023-11-15 22:50:41 -08:00
Woosuk Kwon 054072bee5 [Minor] Move RoPE selection logic to get_rope (#1633) 2023-11-12 16:04:50 -08:00
Antoni BaumViktor FerencziWoosuk Kwon
 		9f669a9a7c Support YaRN models (#1264) 
Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>
Co-authored-by: Viktor Ferenczi <viktor@ferenczi.eu>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2023-11-03 14:12:48 -07:00
Noam Gat 555bdcc5a3 Added logits processor API to sampling params (#1469) 2023-11-03 14:12:15 -07:00
Antoni Baum 9738b84a08 Force paged attention v2 for long contexts (#1510) 2023-11-01 16:24:32 -07:00
Antoni Baum 15f5632365 Delay GPU->CPU sync in sampling (#1337) 2023-10-30 09:01:34 -07:00
ljss 69be658bba Support repetition_penalty (#1424) 2023-10-29 10:02:41 -07:00
chooper1squeeze-ai-labWoosuk Kwon
 		1f24755bf8 Support SqueezeLLM (#1326) 
Co-authored-by: squeeze-ai-lab <squeezeailab.bair@gmail.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2023-10-21 23:14:59 -07:00
Woosuk Kwon c1376e0f82 Change scheduler & input tensor shape (#1381) 2023-10-16 17:48:42 -07:00
Zhuohan LiandYunmo Chen 9d9072a069 Implement prompt logprobs & Batched topk for computing logprobs (#1328) 
Co-authored-by: Yunmo Chen <16273544+wanmok@users.noreply.github.com>
 2023-10-16 10:56:50 -07:00
Woosuk Kwon 928de46888 Implement PagedAttention V2 (#1348) 2023-10-16 00:59:57 -07:00
yhlskt23 91fce82c6f change the timing of sorting logits (#1309) 2023-10-10 19:37:42 -07:00
Zhuohan Li ba0bfd40e2 TP/quantization/weight loading refactor part 1 - Simplify parallel linear logic (#1181) 2023-10-02 15:36:09 -07:00
Woosuk Kwon 84e4e37d14 [Minor] Fix type annotations (#1238) 2023-10-02 15:28:31 -07:00
Chris Bamfordandtimlacroix bb1ba58f06 [Mistral] Mistral-7B-v0.1 support (#1196) 
Co-authored-by: timlacroix <t@mistral.ai>
 2023-09-28 10:41:03 -07:00
Lily LiuWing LianWoosuk Kwon
 		21877b0d75 Support Longchat and RoPE scaling (#555) 
Co-authored-by: Wing Lian <wing.lian@gmail.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2023-09-27 03:36:02 -07:00
Woosuk Kwon 03ffd0a022 Add comments on RoPE initialization (#1176) 2023-09-26 10:48:33 -07:00
Zhuohan Li f187877945 [FIX] Simplify sampler logic (#1156) 2023-09-23 17:21:56 -07:00
Zhuohan LiandAntoni Baum 947b794146 [Sampler] Vectorized sampling (simplified) (#1048) 
Co-authored-by: Antoni Baum <antoni.baum@protonmail.com>
 2023-09-22 17:48:04 -07:00
Woosuk KwonRobert IrvinerootCasperjulian-q
 		e3e79e9e8a Implement AWQ quantization support for LLaMA (#1032) 
Co-authored-by: Robert Irvine <robert@seamlessml.com>
Co-authored-by: root <rirv938@gmail.com>
Co-authored-by: Casper <casperbh.96@gmail.com>
Co-authored-by: julian-q <julianhquevedo@gmail.com>
 2023-09-16 00:03:37 -07:00
Zhuohan Li f04908cae7 [FIX] Minor bug fixes (#1035) 
* [FIX] Minor bug fixes

* Address review comments
 2023-09-13 16:38:12 -07:00
Woosuk KwonandOne e67b4f2c2a Use FP32 in RoPE initialization (#1004) 
Co-authored-by: One <imone@tuta.io>
 2023-09-11 00:26:35 -07:00
Antoni Baum a62de9ecfd Fix wrong dtype in PagedAttentionWithALiBi bias (#996) 
---------

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>
 2023-09-09 14:58:35 -07:00
Robert IrvineandRobert Irvine 4b5bcf8906 faster startup of vLLM (#982) 
* update

---------

Co-authored-by: Robert Irvine <robert@seamlessml.com>
 2023-09-08 14:48:54 +09:00
Woosuk Kwon 320a622ec4 [BugFix] Implement RoPE for GPT-J (#941) 2023-09-06 11:54:33 +09:00
Zhuohan Li 002800f081 Align vLLM's beam search implementation with HF generate (#857) 2023-09-04 17:29:42 -07:00
Dong-Yong Leeanddongyong-lee e11222333f fix: bug fix when penalties are negative (#913) 
Co-authored-by: dongyong-lee <dongyong.lee@navercorp.com>
 2023-09-01 00:37:17 +09:00