iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
6,768 Commits 3 Branches 0 Tags
6881107948c00a8564bc2fa85308f6fc2f065d64
Commit Graph
4568 Commits
Author SHA1 Message Date
Isotr0py ab3a5a8259 Support OLMo models. (#2832) 2024-02-18 21:05:15 -08:00
Zhuohan Li 537c9755a7 [Minor] Small fix to make distributed init logic in worker looks cleaner (#2905) 2024-02-18 14:39:00 -08:00
Mark Mozolewski 786b7f18a5 Add code-revision config argument for Hugging Face Hub (#2892) 2024-02-17 22:36:53 -08:00
jvmncs 8f36444c4f multi-LoRA as extra models in OpenAI server (#2775) 
how to serve the loras (mimicking the [multilora inference example](https://github.com/vllm-project/vllm/blob/main/examples/multilora_inference.py)):
```terminal
$ export LORA_PATH=~/.cache/huggingface/hub/models--yard1--llama-2-7b-sql-lora-test/
$ python -m vllm.entrypoints.api_server \
 --model meta-llama/Llama-2-7b-hf \
 --enable-lora \
 --lora-modules sql-lora=$LORA_PATH sql-lora2=$LORA_PATH
```
the above server will list 3 separate values if the user queries `/models`: one for the base served model, and one each for the specified lora modules. in this case sql-lora and sql-lora2 point to the same underlying lora, but this need not be the case. lora config values take the same values they do in EngineArgs

no work has been done here to scope client permissions to specific models
 2024-02-17 12:00:48 -08:00
Nick Hill 185b2c29e2 Defensively copy sampling_params (#2881) 
If the SamplingParams object passed to LLMEngine.add_request() is mutated after it returns, it could affect the async sampling process for that request.

Suggested by @Yard1 https://github.com/vllm-project/vllm/pull/2514#discussion_r1490106059
 2024-02-17 11:18:04 -08:00
Woosuk Kwon 5f08050d8d Bump up to v0.3.1 (#2887) 2024-02-16 15:05:18 -08:00
shiyi.c_98 64da65b322 Prefix Caching- fix t4 triton error (#2517) 2024-02-16 14:17:55 -08:00
Philipp Moritz 4f2ad11135 Fix DeciLM (#2883) 2024-02-14 22:29:57 -08:00
Woosuk Kwon d7afab6d3a [BugFix] Fix GC bug for LLM class (#2882) 2024-02-14 22:17:44 -08:00
Philipp MoritzandPernekhan Utemuratov 31348dff03 Align LoRA code between Mistral and Mixtral (fixes #2875) (#2880) 
* Fix AttributeError: MixtralModel object has no attribute org_vocab_size.

* Make LoRA logic for Mistral and Mixtral the same

---------

Co-authored-by: Pernekhan Utemuratov <pernekhan@deepinfra.com>
 2024-02-15 01:00:43 +01:00
Woosuk Kwon 25e86b6a61 Don't use cupy NCCL for AMD backends (#2855) 2024-02-14 12:30:44 -08:00
Roy 4efbac6d35 Migrate AquilaForCausalLM to LlamaForCausalLM (#2867) 2024-02-14 12:30:24 -08:00
Woosuk Kwon 7e45107f51 [Fix] Fix memory profiling when GPU is used by multiple processes (#2863) 2024-02-13 19:52:34 -08:00
Philipp Moritz 0c48b37c31 Fix internlm after https://github.com/vllm-project/vllm/pull/2860 (#2861) 2024-02-13 18:01:15 -08:00
Philipp MoritzandRoy 7eacffd951 Migrate InternLMForCausalLM to LlamaForCausalLM (#2860) 
Co-authored-by: Roy <jasonailu87@gmail.com>
 2024-02-13 17:12:05 -08:00
Terry 2a543d6efe Add LoRA support for Mixtral (#2831) 
* add mixtral lora support

* formatting

* fix incorrectly ported logic

* polish tests

* minor fixes and refactoring

* minor fixes

* formatting

* rename and remove redundant logic

* refactoring

* refactoring

* minor fix

* minor refactoring

* fix code smell
 2024-02-14 00:55:45 +01:00
Philipp MoritzandRoy 317b29de0f Remove Yi model definition, please use LlamaForCausalLM instead (#2854) 
Co-authored-by: Roy <jasonailu87@gmail.com>
 2024-02-13 14:22:22 -08:00
Woosuk Kwon a463c333dd Use CuPy for CUDA graphs (#2811) 2024-02-13 11:32:06 -08:00
Philipp Moritz ea356004d4 Revert "Refactor llama family models (#2637)" (#2851) 
This reverts commit 5c976a7e1a.
 2024-02-13 09:24:59 -08:00
Roy 5c976a7e1a Refactor llama family models (#2637) 2024-02-13 00:09:23 -08:00
RexandChunan Zeng 563836496a Refactor 2 awq gemm kernels into m16nXk32 (#2723) 
Co-authored-by: Chunan Zeng <chunanzeng@Chunans-Air.attlocal.net>
 2024-02-12 11:02:17 -08:00
Hongxia Yang 0580aab02f [ROCm] support Radeon™ 7900 series (gfx1100) without using flash-attention (#2768) 2024-02-10 23:14:37 -08:00
Woosuk Kwon 3711811b1d Disable custom all reduce by default (#2808) 2024-02-08 09:58:03 -08:00
SangBin Cho 65b89d16ee [Ray] Integration compiled DAG off by default (#2471) 2024-02-08 09:57:25 -08:00
Lily Liu fe6d09ae61 [Minor] More fix of test_cache.py CI test failure (#2750) 2024-02-06 11:38:38 -08:00
liuyhwangyh ed70c70ea3 modelscope: fix issue when model parameter is not a model id but path of the model. (#2489) 2024-02-06 09:57:15 -08:00
Woosuk Kwon f0d4e14557 Add fused top-K softmax kernel for MoE (#2769) 2024-02-05 17:38:02 -08:00
Lukas b92adec8e8 Set local logging level via env variable (#2774) 2024-02-05 14:26:50 -08:00
Rex 5a6c81b051 Remove eos tokens from output by default (#2611) 2024-02-04 14:32:42 -08:00
dancingpipiandshujunhua1 51cd22ce56 set&get llm internal tokenizer instead of the TokenizerGroup (#2741) 
Co-authored-by: shujunhua1 <shujunhua1@jd.com>
 2024-02-04 14:25:36 -08:00
zspo 0e163fce18 Fix default length_penalty to 1.0 (#2667) 2024-02-01 15:59:39 -08:00
Kunshang JiandJiang Li 96b6f475dd Remove hardcoded device="cuda" to support more devices (#2503) 
Co-authored-by: Jiang Li <jiang1.li@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
 2024-02-01 15:46:39 -08:00
Pernekhan UtemuratovandPernekhan Utemuratov c410f5d020 Use revision when downloading the quantization config file (#2697) 
Co-authored-by: Pernekhan Utemuratov <pernekhan@deepinfra.com>
 2024-02-01 15:41:58 -08:00
Simon Mo b9e96b17de fix python 3.8 syntax (#2716) 2024-02-01 14:00:58 -08:00
Fengzhe Zhou cd9e60c76c Add Internlm2 (#2666) 2024-02-01 09:27:40 -08:00
Robert Shaw 93b38bea5d Refactor Prometheus and Add Request Level Metrics (#2316) 2024-01-31 14:58:07 -08:00
Philipp Moritz d0d93b92b1 Add unit test for Mixtral MoE layer (#2677) 2024-01-31 14:34:17 -08:00
zspo c664b0e683 fix some bugs (#2689) 2024-01-31 10:09:23 -08:00
Tao He d69ff0cbbb Fixes assertion failure in prefix caching: the lora index mapping should respect prefix_len (#2688) 
Signed-off-by: Tao He <sighingnow@gmail.com>
 2024-01-31 18:00:13 +01:00
Zhuohan Li 1af090b57d Bump up version to v0.3.0 (#2656) 2024-01-31 00:07:07 -08:00
Woosuk Kwon 3dad944485 Add quantized mixtral support (#2673) 2024-01-30 16:34:10 -08:00
Woosuk Kwon 105a40f53a [Minor] Fix false warning when TP=1 (#2674) 2024-01-30 14:39:40 -08:00
Philipp Moritz bbe9bd9684 [Minor] Fix a small typo (#2672) 2024-01-30 13:40:37 -08:00
Wen Sun d79ced3292 Fix 'Actor methods cannot be called directly' when using --engine-use-ray (#2664) 
* fix: engine-useray complain

* fix: typo
 2024-01-30 17:17:05 +01:00
Philipp Moritzandchen shen ab40644669 Fused MOE for Mixtral (#2542) 
Co-authored-by: chen shen <scv119@gmail.com>
 2024-01-29 22:43:37 -08:00
wangding zengandroy 5d60def02c DeepseekMoE support with Fused MoE kernel (#2453) 
Co-authored-by: roy <jasonailu87@gmail.com>
 2024-01-29 21:19:48 -08:00
zhaoyang-starandzhaoyang-star b72af8f1ed Fix error when tp > 1 (#2644) 
Co-authored-by: zhaoyang-star <zhao.yang16@zte.com.cn>
 2024-01-28 22:47:39 -08:00
zhaoyang-starzhaoyangZhuohan Li
 		9090bf02e7 Support FP8-E5M2 KV Cache (#2279) 
Co-authored-by: zhaoyang <zhao.yang16@zte.com.cn>
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
 2024-01-28 16:43:54 -08:00
Murali Andoorveedu 89be30fa7d Small async_llm_engine refactor (#2618) 2024-01-27 23:28:37 -08:00
Woosuk Kwon 5f036d2bcc [Minor] Fix warning on Ray dependencies (#2630) 2024-01-27 15:43:40 -08:00