iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
477 Commits 3 Branches 0 Tags
06e9ebebd51c3db779dedec5556251c8ecc3a00a
Commit Graph
48 Commits
Author SHA1 Message Date
ljss e1054247ba [Optimization] Implement fused add rmsnorm (#1667) 2023-11-18 18:18:02 -08:00
Woosuk Kwon 8d17774f92 Add AWQ support for all models (#1714) 2023-11-18 17:56:47 -08:00
Roy e87557b069 Support Min P Sampler (#1642) 2023-11-17 16:20:49 -08:00
Zhuohan Li 7076fa1c9f TP/quantization/weight loading refactor part 2 - Refactor quantized linear logic and extend quantization support to all models (#1622) 
Refactor the tensor parallelism, quantization, and weight-loading codes.

Summary of the new features enabled by this PR:
- **All models** are able to be quantized with AWQ and SqueezeLLM, and [soon GPTQ](https://github.com/vllm-project/vllm/pull/1580).
- Model loading code became much simpler.
- Support model parallelism for all MQA/GQA models when the number of key/value heads is smaller than the tensor parallel size.
 2023-11-15 22:50:41 -08:00
Woosuk Kwon 054072bee5 [Minor] Move RoPE selection logic to get_rope (#1633) 2023-11-12 16:04:50 -08:00
Antoni BaumViktor FerencziWoosuk Kwon
 		9f669a9a7c Support YaRN models (#1264) 
Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>
Co-authored-by: Viktor Ferenczi <viktor@ferenczi.eu>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2023-11-03 14:12:48 -07:00
Noam Gat 555bdcc5a3 Added logits processor API to sampling params (#1469) 2023-11-03 14:12:15 -07:00
Antoni Baum 9738b84a08 Force paged attention v2 for long contexts (#1510) 2023-11-01 16:24:32 -07:00
Antoni Baum 15f5632365 Delay GPU->CPU sync in sampling (#1337) 2023-10-30 09:01:34 -07:00
ljss 69be658bba Support repetition_penalty (#1424) 2023-10-29 10:02:41 -07:00
chooper1squeeze-ai-labWoosuk Kwon
 		1f24755bf8 Support SqueezeLLM (#1326) 
Co-authored-by: squeeze-ai-lab <squeezeailab.bair@gmail.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2023-10-21 23:14:59 -07:00
Woosuk Kwon c1376e0f82 Change scheduler & input tensor shape (#1381) 2023-10-16 17:48:42 -07:00
Zhuohan LiandYunmo Chen 9d9072a069 Implement prompt logprobs & Batched topk for computing logprobs (#1328) 
Co-authored-by: Yunmo Chen <16273544+wanmok@users.noreply.github.com>
 2023-10-16 10:56:50 -07:00
Woosuk Kwon 928de46888 Implement PagedAttention V2 (#1348) 2023-10-16 00:59:57 -07:00
yhlskt23 91fce82c6f change the timing of sorting logits (#1309) 2023-10-10 19:37:42 -07:00
Zhuohan Li ba0bfd40e2 TP/quantization/weight loading refactor part 1 - Simplify parallel linear logic (#1181) 2023-10-02 15:36:09 -07:00
Woosuk Kwon 84e4e37d14 [Minor] Fix type annotations (#1238) 2023-10-02 15:28:31 -07:00
Chris Bamfordandtimlacroix bb1ba58f06 [Mistral] Mistral-7B-v0.1 support (#1196) 
Co-authored-by: timlacroix <t@mistral.ai>
 2023-09-28 10:41:03 -07:00
Lily LiuWing LianWoosuk Kwon
 		21877b0d75 Support Longchat and RoPE scaling (#555) 
Co-authored-by: Wing Lian <wing.lian@gmail.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2023-09-27 03:36:02 -07:00
Woosuk Kwon 03ffd0a022 Add comments on RoPE initialization (#1176) 2023-09-26 10:48:33 -07:00
Zhuohan Li f187877945 [FIX] Simplify sampler logic (#1156) 2023-09-23 17:21:56 -07:00
Zhuohan LiandAntoni Baum 947b794146 [Sampler] Vectorized sampling (simplified) (#1048) 
Co-authored-by: Antoni Baum <antoni.baum@protonmail.com>
 2023-09-22 17:48:04 -07:00
Woosuk KwonRobert IrvinerootCasperjulian-q
 		e3e79e9e8a Implement AWQ quantization support for LLaMA (#1032) 
Co-authored-by: Robert Irvine <robert@seamlessml.com>
Co-authored-by: root <rirv938@gmail.com>
Co-authored-by: Casper <casperbh.96@gmail.com>
Co-authored-by: julian-q <julianhquevedo@gmail.com>
 2023-09-16 00:03:37 -07:00
Zhuohan Li f04908cae7 [FIX] Minor bug fixes (#1035) 
* [FIX] Minor bug fixes

* Address review comments
 2023-09-13 16:38:12 -07:00
Woosuk KwonandOne e67b4f2c2a Use FP32 in RoPE initialization (#1004) 
Co-authored-by: One <imone@tuta.io>
 2023-09-11 00:26:35 -07:00
Antoni Baum a62de9ecfd Fix wrong dtype in PagedAttentionWithALiBi bias (#996) 
---------

Signed-off-by: Antoni Baum <antoni.baum@protonmail.com>
 2023-09-09 14:58:35 -07:00
Robert IrvineandRobert Irvine 4b5bcf8906 faster startup of vLLM (#982) 
* update

---------

Co-authored-by: Robert Irvine <robert@seamlessml.com>
 2023-09-08 14:48:54 +09:00
Woosuk Kwon 320a622ec4 [BugFix] Implement RoPE for GPT-J (#941) 2023-09-06 11:54:33 +09:00
Zhuohan Li 002800f081 Align vLLM's beam search implementation with HF generate (#857) 2023-09-04 17:29:42 -07:00
Dong-Yong Leeanddongyong-lee e11222333f fix: bug fix when penalties are negative (#913) 
Co-authored-by: dongyong-lee <dongyong.lee@navercorp.com>
 2023-09-01 00:37:17 +09:00
Aman Gupta Karmani 28873a2799 Improve _prune_hidden_states micro-benchmark (#707) 2023-08-31 13:28:43 +09:00
Aman Gupta Karmani 75471386de use flash-attn via xformers (#877) 2023-08-29 21:52:13 -07:00
Woosuk Kwon 94d2f59895 Set replacement=True in torch.multinomial (#858) 2023-08-25 12:22:01 +09:00
Woosuk Kwon 2a4ec90854 Fix for breaking changes in xformers 0.0.21 (#834) 2023-08-23 17:44:21 +09:00
Woosuk Kwon d64bf1646c Implement approximate GELU kernels (#828) 2023-08-23 07:43:21 +09:00
Abraham-Xu d1744376ae Align with huggingface Top K sampling (#753) 2023-08-15 16:44:33 -07:00
Woosuk Kwon 55fe8a81ec Refactor scheduler (#658) 2023-08-02 16:42:01 -07:00
Zhuohan Li 1b0bd0fe8a Add Falcon support (new) (#592) 2023-08-02 14:04:39 -07:00
Zhuohan Li 6fc2a38b11 Add support for LLaMA-2 (#505) 2023-07-20 11:38:27 -07:00
Songandoliveryuan bda41c70dd hotfix attn alibi wo head mapping (#496) 
Co-authored-by: oliveryuan <oliveryuan@basemind.com>
 2023-07-18 11:31:48 -07:00
Zhuohan Li 96853af5a8 Optimize MQA Kernel (#452) 2023-07-14 20:06:40 -04:00
Andre SlavescuandwoWoosuk Kwon c894836108 [Model] Add support for GPT-J (#226) 
Co-authored-by: woWoosuk Kwon <woosuk.kwon@berkeley.edu>
 2023-07-08 17:55:16 -07:00
Woosuk Kwon 404422f42e [Model] Add support for MPT (#334) 2023-07-03 16:47:53 -07:00
Woosuk Kwon e41f06702c Add support for BLOOM (#331) 2023-07-03 13:12:35 -07:00
Zhuohan Li d6fa1be3a8 [Quality] Add code formatter and linter (#326) 2023-07-03 11:31:55 -07:00
Lily Liu 425040d4c1 remove floats == 0 comparison (#285) 2023-06-28 14:11:51 -07:00
Michael Feil 298695b766 GPTBigCode (StarCoder, SantaCoder Support) (#209) 2023-06-23 01:49:27 +08:00
Woosuk Kwon 0b98ba15c7 Change the name to vLLM (#150) 2023-06-17 03:07:40 -07:00