iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity
761 Commits 3 Branches 0 Tags
264017a2bf030f060ebad91eb9be9b4e0033edb9
Commit Graph
198 Commits
Author SHA1 Message Date
Isotr0py ab3a5a8259 Support OLMo models. (#2832) 2024-02-18 21:05:15 -08:00
shiyi.c_98 64da65b322 Prefix Caching- fix t4 triton error (#2517) 2024-02-16 14:17:55 -08:00
Philipp Moritz 4f2ad11135 Fix DeciLM (#2883) 2024-02-14 22:29:57 -08:00
Philipp MoritzandPernekhan Utemuratov 31348dff03 Align LoRA code between Mistral and Mixtral (fixes #2875) (#2880) 
* Fix AttributeError: MixtralModel object has no attribute org_vocab_size.

* Make LoRA logic for Mistral and Mixtral the same

---------

Co-authored-by: Pernekhan Utemuratov <pernekhan@deepinfra.com>
 2024-02-15 01:00:43 +01:00
Woosuk Kwon 25e86b6a61 Don't use cupy NCCL for AMD backends (#2855) 2024-02-14 12:30:44 -08:00
Roy 4efbac6d35 Migrate AquilaForCausalLM to LlamaForCausalLM (#2867) 2024-02-14 12:30:24 -08:00
Philipp Moritz 0c48b37c31 Fix internlm after https://github.com/vllm-project/vllm/pull/2860 (#2861) 2024-02-13 18:01:15 -08:00
Philipp MoritzandRoy 7eacffd951 Migrate InternLMForCausalLM to LlamaForCausalLM (#2860) 
Co-authored-by: Roy <jasonailu87@gmail.com>
 2024-02-13 17:12:05 -08:00
Terry 2a543d6efe Add LoRA support for Mixtral (#2831) 
* add mixtral lora support

* formatting

* fix incorrectly ported logic

* polish tests

* minor fixes and refactoring

* minor fixes

* formatting

* rename and remove redundant logic

* refactoring

* refactoring

* minor fix

* minor refactoring

* fix code smell
 2024-02-14 00:55:45 +01:00
Philipp MoritzandRoy 317b29de0f Remove Yi model definition, please use LlamaForCausalLM instead (#2854) 
Co-authored-by: Roy <jasonailu87@gmail.com>
 2024-02-13 14:22:22 -08:00
Woosuk Kwon a463c333dd Use CuPy for CUDA graphs (#2811) 2024-02-13 11:32:06 -08:00
Philipp Moritz ea356004d4 Revert "Refactor llama family models (#2637)" (#2851) 
This reverts commit 5c976a7e1a.
 2024-02-13 09:24:59 -08:00
Roy 5c976a7e1a Refactor llama family models (#2637) 2024-02-13 00:09:23 -08:00
RexandChunan Zeng 563836496a Refactor 2 awq gemm kernels into m16nXk32 (#2723) 
Co-authored-by: Chunan Zeng <chunanzeng@Chunans-Air.attlocal.net>
 2024-02-12 11:02:17 -08:00
Hongxia Yang 0580aab02f [ROCm] support Radeon™ 7900 series (gfx1100) without using flash-attention (#2768) 2024-02-10 23:14:37 -08:00
Woosuk Kwon f0d4e14557 Add fused top-K softmax kernel for MoE (#2769) 2024-02-05 17:38:02 -08:00
Kunshang JiandJiang Li 96b6f475dd Remove hardcoded device="cuda" to support more devices (#2503) 
Co-authored-by: Jiang Li <jiang1.li@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
 2024-02-01 15:46:39 -08:00
Pernekhan UtemuratovandPernekhan Utemuratov c410f5d020 Use revision when downloading the quantization config file (#2697) 
Co-authored-by: Pernekhan Utemuratov <pernekhan@deepinfra.com>
 2024-02-01 15:41:58 -08:00
Fengzhe Zhou cd9e60c76c Add Internlm2 (#2666) 2024-02-01 09:27:40 -08:00
Philipp Moritz d0d93b92b1 Add unit test for Mixtral MoE layer (#2677) 2024-01-31 14:34:17 -08:00
Woosuk Kwon 3dad944485 Add quantized mixtral support (#2673) 2024-01-30 16:34:10 -08:00
Woosuk Kwon 105a40f53a [Minor] Fix false warning when TP=1 (#2674) 2024-01-30 14:39:40 -08:00
Philipp Moritz bbe9bd9684 [Minor] Fix a small typo (#2672) 2024-01-30 13:40:37 -08:00
Philipp Moritzandchen shen ab40644669 Fused MOE for Mixtral (#2542) 
Co-authored-by: chen shen <scv119@gmail.com>
 2024-01-29 22:43:37 -08:00
wangding zengandroy 5d60def02c DeepseekMoE support with Fused MoE kernel (#2453) 
Co-authored-by: roy <jasonailu87@gmail.com>
 2024-01-29 21:19:48 -08:00
zhaoyang-starzhaoyangZhuohan Li
 		9090bf02e7 Support FP8-E5M2 KV Cache (#2279) 
Co-authored-by: zhaoyang <zhao.yang16@zte.com.cn>
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
 2024-01-28 16:43:54 -08:00
Hanzhi Zhou 380170038e Implement custom all reduce kernels (#2192) 2024-01-27 12:46:35 -08:00
Casper beb89f68b4 AWQ: Up to 2.66x higher throughput (#2566) 2024-01-26 23:53:17 -08:00
dakotamahan-stabilityandZhuohan Li 3a0e1fc070 Support for Stable LM 2 (#2598) 
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
 2024-01-26 12:45:19 -08:00
Junyang Lin 2832e7b9f9 fix names and license for Qwen2 (#2589) 2024-01-24 22:37:51 -08:00
Antoni BaumChen ShenShreyas KrishnaswamyAvnish Narayan
 		9b945daaf1 [Experimental] Add multi-LoRA support (#1804) 
Co-authored-by: Chen Shen <scv119@gmail.com>
Co-authored-by: Shreyas Krishnaswamy <shrekris@anyscale.com>
Co-authored-by: Avnish Narayan <avnish@anyscale.com>
 2024-01-23 15:26:37 -08:00
Junyang Lin 94b5edeb53 Add qwen2 (#2495) 2024-01-22 14:34:21 -08:00
Cade Daniel 18bfcdd05c [Speculative decoding 2/9] Multi-step worker for draft model (#2424) 2024-01-21 16:31:47 -08:00
Junda Chen 5b23c3f26f Add group as an argument in broadcast ops (#2522) 2024-01-20 16:00:26 -08:00
Roy 91a61da9b1 [Bugfix] fix load local safetensors model (#2512) 2024-01-19 16:26:16 -08:00
Zhuohan Li ef9b636e2d Simplify broadcast logic for control messages (#2501) 2024-01-19 11:23:30 -08:00
Simon Mo dd7e8f5f64 refactor complemention api for readability (#2499) 2024-01-18 16:45:14 -08:00
Nikola Borisov 7e1081139d Don't download both safetensor and bin files. (#2480) 2024-01-18 11:05:53 -08:00
YingchaoX 8a25d3a71a fix stablelm.py tensor-parallel-size bug (#2482) 2024-01-18 09:39:46 -08:00
shiyi.c_98DouHappyZhuohan Li
 		d10f8e1d43 [Experimental] Prefix Caching Support (#1669) 
Co-authored-by: DouHappy <2278958187@qq.com>
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
 2024-01-17 16:32:10 -08:00
Hyunsung Lee e1957c6ebd Add StableLM3B model (#2372) 2024-01-16 20:32:40 -08:00
Chenhui Zhang f780504d12 fix weigit loading for GQA with TP (#2379) 2024-01-15 15:43:59 -08:00
陈序 218dc2ccda Aligning top_p and top_k Sampling (#1885) 
* Align top_p and top_k with huggingface

* remove _get_prompt_and_output_tokens

* rename _apply_top_p_top_k

* compare top_p top_k with hf

* fix test errors
 2024-01-12 22:51:03 +01:00
Gary Hui 7878958c0d Address Phi modeling update 2 (#2428) 2024-01-12 12:16:49 -08:00
Woosuk Kwon 50376faa7b Rename phi_1_5 -> phi (#2385) 2024-01-11 16:23:43 -08:00
Cade Daniel 79d64c4954 [Speculative decoding 1/9] Optimized rejection sampler (#2336) 2024-01-09 15:38:41 -08:00
Woosuk Kwon 28c3f12104 [Minor] Remove unused code in attention (#2384) 2024-01-08 13:13:08 -08:00
Zhuohan Li fd4ea8ef5c Use NCCL instead of ray for control-plane communication to remove serialization overhead (#2221) 2024-01-03 11:30:22 -08:00
Roy 9140561059 [Minor] Fix typo and remove unused code (#2305) 2024-01-02 19:23:15 -08:00
Jong-hun Shin 4934d49274 Support GPT-NeoX Models without attention biases (#2301) 2023-12-30 11:42:04 -05:00