iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
517 Commits 3 Branches 0 Tags
d06980dfa7e8dcc1738656beb46d3735c86faa21
Commit Graph
59 Commits
Author SHA1 Message Date
Roy d27f4bae39 Fix rope cache key error (#1867) 2023-11-30 08:29:28 -08:00
Jee Li 63b2206ad0 Avoid multiple instantiations of the RoPE class (#1828) 2023-11-29 23:06:27 -08:00
Woosuk Kwon 27feead2f8 Refactor Worker & InputMetadata (#1843) 2023-11-29 22:16:37 -08:00
Woosuk Kwon a9e4574261 Refactor Attention (#1840) 2023-11-29 15:37:31 -08:00
Zhuohan Li 708e6c18b0 [FIX] Fix class naming (#1803) 2023-11-28 14:08:01 -08:00
Yanming W e0c6f556e8 [Build] Avoid building too many extensions (#1624) 2023-11-23 16:31:19 -08:00
ljss de23687d16 Fix repetition penalty aligned with huggingface (#1577) 2023-11-22 14:41:44 -08:00
Zhuohan Li 7d761fe3c1 [FIX] Fix the case when input_is_parallel=False for ScaledActivation (#1737) 2023-11-20 23:56:48 -08:00
Woosuk Kwon cf35d8f3d7 [BugFix] Fix TP support for AWQ (#1731) 2023-11-20 21:42:45 -08:00
ljss 819b18e7ba Rewrite torch.repeat_interleave to remove cpu synchronization (#1599) 2023-11-20 17:46:32 -08:00
Simon Mo 5ffc0d13a2 Migrate linter from pylint to ruff (#1665) 2023-11-20 11:58:01 -08:00
ljss e1054247ba [Optimization] Implement fused add rmsnorm (#1667) 2023-11-18 18:18:02 -08:00
Woosuk Kwon 8d17774f92 Add AWQ support for all models (#1714) 2023-11-18 17:56:47 -08:00
Roy e87557b069 Support Min P Sampler (#1642) 2023-11-17 16:20:49 -08:00
Zhuohan Li 7076fa1c9f TP/quantization/weight loading refactor part 2 - Refactor quantized linear logic and extend quantization support to all models (#1622) 
Refactor the tensor parallelism, quantization, and weight-loading codes.

Summary of the new features enabled by this PR:
- **All models** are able to be quantized with AWQ and SqueezeLLM, and [soon GPTQ](https://github.com/vllm-project/vllm/pull/1580).
- Model loading code became much simpler.
- Support model parallelism for all MQA/GQA models when the number of key/value heads is smaller than the tensor parallel size.
 2023-11-15 22:50:41 -08:00
Woosuk Kwon 054072bee5 [Minor] Move RoPE selection logic to get_rope (#1633) 2023-11-12 16:04:50 -08:00
Antoni BaumViktor FerencziWoosuk Kwon
 		9f669a9a7c Support YaRN models (#1264) 
Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>
Co-authored-by: Viktor Ferenczi <viktor@ferenczi.eu>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2023-11-03 14:12:48 -07:00
Noam Gat 555bdcc5a3 Added logits processor API to sampling params (#1469) 2023-11-03 14:12:15 -07:00
Antoni Baum 9738b84a08 Force paged attention v2 for long contexts (#1510) 2023-11-01 16:24:32 -07:00
Antoni Baum 15f5632365 Delay GPU->CPU sync in sampling (#1337) 2023-10-30 09:01:34 -07:00
ljss 69be658bba Support repetition_penalty (#1424) 2023-10-29 10:02:41 -07:00
chooper1squeeze-ai-labWoosuk Kwon
 		1f24755bf8 Support SqueezeLLM (#1326) 
Co-authored-by: squeeze-ai-lab <squeezeailab.bair@gmail.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2023-10-21 23:14:59 -07:00
Woosuk Kwon c1376e0f82 Change scheduler & input tensor shape (#1381) 2023-10-16 17:48:42 -07:00
Zhuohan LiandYunmo Chen 9d9072a069 Implement prompt logprobs & Batched topk for computing logprobs (#1328) 
Co-authored-by: Yunmo Chen <16273544+wanmok@users.noreply.github.com>
 2023-10-16 10:56:50 -07:00
Woosuk Kwon 928de46888 Implement PagedAttention V2 (#1348) 2023-10-16 00:59:57 -07:00
yhlskt23 91fce82c6f change the timing of sorting logits (#1309) 2023-10-10 19:37:42 -07:00
Zhuohan Li ba0bfd40e2 TP/quantization/weight loading refactor part 1 - Simplify parallel linear logic (#1181) 2023-10-02 15:36:09 -07:00
Woosuk Kwon 84e4e37d14 [Minor] Fix type annotations (#1238) 2023-10-02 15:28:31 -07:00
Chris Bamfordandtimlacroix bb1ba58f06 [Mistral] Mistral-7B-v0.1 support (#1196) 
Co-authored-by: timlacroix <t@mistral.ai>
 2023-09-28 10:41:03 -07:00
Lily LiuWing LianWoosuk Kwon
 		21877b0d75 Support Longchat and RoPE scaling (#555) 
Co-authored-by: Wing Lian <wing.lian@gmail.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2023-09-27 03:36:02 -07:00
Woosuk Kwon 03ffd0a022 Add comments on RoPE initialization (#1176) 2023-09-26 10:48:33 -07:00
Zhuohan Li f187877945 [FIX] Simplify sampler logic (#1156) 2023-09-23 17:21:56 -07:00
Zhuohan LiandAntoni Baum 947b794146 [Sampler] Vectorized sampling (simplified) (#1048) 
Co-authored-by: Antoni Baum <antoni.baum@protonmail.com>
 2023-09-22 17:48:04 -07:00
Woosuk KwonRobert IrvinerootCasperjulian-q
 		e3e79e9e8a Implement AWQ quantization support for LLaMA (#1032) 
Co-authored-by: Robert Irvine <robert@seamlessml.com>
Co-authored-by: root <rirv938@gmail.com>
Co-authored-by: Casper <casperbh.96@gmail.com>
Co-authored-by: julian-q <julianhquevedo@gmail.com>
 2023-09-16 00:03:37 -07:00
Zhuohan Li f04908cae7 [FIX] Minor bug fixes (#1035) 
* [FIX] Minor bug fixes

* Address review comments
 2023-09-13 16:38:12 -07:00
Woosuk KwonandOne e67b4f2c2a Use FP32 in RoPE initialization (#1004) 
Co-authored-by: One <imone@tuta.io>
 2023-09-11 00:26:35 -07:00
Antoni Baum a62de9ecfd Fix wrong dtype in PagedAttentionWithALiBi bias (#996) 
---------

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>
 2023-09-09 14:58:35 -07:00
Robert IrvineandRobert Irvine 4b5bcf8906 faster startup of vLLM (#982) 
* update

---------

Co-authored-by: Robert Irvine <robert@seamlessml.com>
 2023-09-08 14:48:54 +09:00
Woosuk Kwon 320a622ec4 [BugFix] Implement RoPE for GPT-J (#941) 2023-09-06 11:54:33 +09:00
Zhuohan Li 002800f081 Align vLLM's beam search implementation with HF generate (#857) 2023-09-04 17:29:42 -07:00
Dong-Yong Leeanddongyong-lee e11222333f fix: bug fix when penalties are negative (#913) 
Co-authored-by: dongyong-lee <dongyong.lee@navercorp.com>
 2023-09-01 00:37:17 +09:00
Aman Gupta Karmani 28873a2799 Improve _prune_hidden_states micro-benchmark (#707) 2023-08-31 13:28:43 +09:00
Aman Gupta Karmani 75471386de use flash-attn via xformers (#877) 2023-08-29 21:52:13 -07:00
Woosuk Kwon 94d2f59895 Set replacement=True in torch.multinomial (#858) 2023-08-25 12:22:01 +09:00
Woosuk Kwon 2a4ec90854 Fix for breaking changes in xformers 0.0.21 (#834) 2023-08-23 17:44:21 +09:00
Woosuk Kwon d64bf1646c Implement approximate GELU kernels (#828) 2023-08-23 07:43:21 +09:00
Abraham-Xu d1744376ae Align with huggingface Top K sampling (#753) 2023-08-15 16:44:33 -07:00
Woosuk Kwon 55fe8a81ec Refactor scheduler (#658) 2023-08-02 16:42:01 -07:00
Zhuohan Li 1b0bd0fe8a Add Falcon support (new) (#592) 2023-08-02 14:04:39 -07:00
Zhuohan Li 6fc2a38b11 Add support for LLaMA-2 (#505) 2023-07-20 11:38:27 -07:00