Zhuohan Li
326fcc8b9f
[Doc] Deprecate Discord ( #12668 )
2025-02-02 19:19:56 -08:00
Zhuohan Li
49628fe13e
[Doc] Update README.md with Ray Summit talk links ( #10610 )
2024-11-24 16:45:09 -08:00
Zhuohan Li
a95354a36e
[Doc] Update README.md with Ray summit slides ( #9088 )
2024-10-05 02:54:45 +00:00
Zhuohan Li
bd0e7802e0
[Bugfix] Add warmup for prefix caching example ( #5235 )
2024-06-03 19:36:41 -07:00
Zhuohan Li
8279078e21
[Bugfix] Remove deprecated @abstractproperty ( #5174 )
2024-06-01 22:40:25 +00:00
1102bef219
[Bugfix / Core] Prefix Caching Guards (merged with main) ( #4846 )
...
Co-authored-by: rsnm2 <rshaw@neuralmagic.com >
Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com >
2024-05-27 15:18:17 -07:00
Zhuohan Li
361c461a12
[Doc] Highlight the fourth meetup in the README ( #4842 )
2024-05-15 11:38:49 -07:00
Zhuohan Li
c579b750a0
[Doc] Add meetups to the doc ( #4798 )
2024-05-13 18:48:00 -07:00
Zhuohan Li
ac1fbf7fd2
[Doc] Shorten README by removing supported model list ( #4796 )
2024-05-13 16:23:54 -07:00
Zhuohan Li
e11e200736
[Bugfix] Fix filelock version requirement ( #4075 )
2024-04-14 21:50:08 -07:00
Zhuohan Li
d4ec9ffb95
[Misc] Fix typo in scheduler.py ( #4022 )
2024-04-12 13:56:04 -07:00
Zhuohan Li
e90fc21f2e
[Hardware][Neuron] Refactor neuron support ( #3471 )
2024-03-22 01:22:17 +00:00
Zhuohan Li
523e30ea0c
[BugFix] Hot fix in setup.py for neuron build ( #3537 )
2024-03-20 17:59:52 -07:00
Zhuohan Li
b30880a762
[Misc] Update README for the Third vLLM Meetup ( #3479 )
2024-03-18 15:58:38 -07:00
Zhuohan Li
c0c17d4896
[Misc] Fix PR Template ( #3478 )
2024-03-18 15:00:31 -07:00
Zhuohan Li
eeab52a4ff
[FIX] Simpler fix for async engine running on ray ( #3371 )
2024-03-13 14:18:40 -07:00
Zhuohan Li
4c922709b6
Add distributed model executor abstraction ( #3191 )
2024-03-11 11:03:45 -07:00
Zhuohan Li
2f8844ba08
Re-enable the 80 char line width limit ( #3305 )
2024-03-10 19:49:14 -07:00
Zhuohan Li
f48c6791b7
[FIX] Fix prefix test error on main ( #3286 )
2024-03-08 17:16:14 -08:00
Zhuohan Li
996d095c54
[FIX] Fix styles in automatic prefix caching & add a automatic prefix caching benchmark ( #3158 )
2024-03-03 14:37:18 -08:00
Zhuohan Li
8fbd84bf78
Bump up version to v0.3.2 ( #2968 )
...
This version is for more model support. Add support for Gemma models (#2964 ) and OLMo models (#2832 ).
2024-02-21 11:47:25 -08:00
Zhuohan Li
a9c8212895
[FIX] Add Gemma model to the doc ( #2966 )
2024-02-21 09:46:15 -08:00
Zhuohan Li
63e2a6419d
[FIX] Fix beam search test ( #2930 )
2024-02-20 14:37:39 -08:00
Zhuohan Li
a61f0521b8
[Test] Add basic correctness test ( #2908 )
2024-02-18 16:44:50 -08:00
Zhuohan Li
537c9755a7
[Minor] Small fix to make distributed init logic in worker looks cleaner ( #2905 )
2024-02-18 14:39:00 -08:00
Zhuohan Li
1af090b57d
Bump up version to v0.3.0 ( #2656 )
2024-01-31 00:07:07 -08:00
Zhuohan Li
ef9b636e2d
Simplify broadcast logic for control messages ( #2501 )
2024-01-19 11:23:30 -08:00
Zhuohan Li
f745847ef7
[Minor] Fix the format in quick start guide related to Model Scope ( #2425 )
2024-01-11 19:44:01 -08:00
Zhuohan Li
fd4ea8ef5c
Use NCCL instead of ray for control-plane communication to remove serialization overhead ( #2221 )
2024-01-03 11:30:22 -08:00
Zhuohan Li
358c328d69
[BUGFIX] Fix communication test ( #2285 )
2023-12-27 17:18:11 -05:00
Zhuohan Li
4aaafdd289
[BUGFIX] Fix the path of test prompts ( #2273 )
2023-12-26 10:37:21 -08:00
Zhuohan Li
66b108d142
[BUGFIX] Fix API server test ( #2270 )
2023-12-26 10:37:06 -08:00
Zhuohan Li
e0ff920001
[BUGFIX] Do not return ignored sentences twice in async llm engine ( #2258 )
2023-12-26 13:41:09 +08:00
Zhuohan Li
e19a64c7ef
[FIX] Fix formatting error in main branch ( #1822 )
2023-11-28 16:56:43 -08:00
Zhuohan Li
1cb4ad8de9
[FIX] Fix formatting error
2023-11-29 00:40:19 +00:00
Zhuohan Li
708e6c18b0
[FIX] Fix class naming ( #1803 )
2023-11-28 14:08:01 -08:00
Zhuohan Li
7d761fe3c1
[FIX] Fix the case when input_is_parallel=False for ScaledActivation ( #1737 )
2023-11-20 23:56:48 -08:00
Zhuohan Li
32c927b53f
[FIX] Update the doc link in README.md ( #1730 )
2023-11-20 12:46:24 -08:00
Zhuohan Li
0fc280b06c
Update the adding-model doc according to the new refactor ( #1692 )
2023-11-16 18:46:26 -08:00
Zhuohan Li
20d0699d49
[Fix] Fix comm test ( #1691 )
2023-11-16 16:28:39 -08:00
Zhuohan Li
415d109527
[Fix] Update Supported Models List ( #1690 )
2023-11-16 14:47:26 -08:00
Zhuohan Li
7076fa1c9f
TP/quantization/weight loading refactor part 2 - Refactor quantized linear logic and extend quantization support to all models ( #1622 )
...
Refactor the tensor parallelism, quantization, and weight-loading codes.
Summary of the new features enabled by this PR:
- **All models** are able to be quantized with AWQ and SqueezeLLM, and [soon GPTQ](https://github.com/vllm-project/vllm/pull/1580 ).
- Model loading code became much simpler.
- Support model parallelism for all MQA/GQA models when the number of key/value heads is smaller than the tensor parallel size.
2023-11-15 22:50:41 -08:00
Zhuohan Li and Woosuk Kwon
06458a0b42
Upgrade to CUDA 12 ( #1527 )
...
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu >
2023-11-08 14:17:49 -08:00
Zhuohan Li
7b895c5976
[Fix] Fix duplicated logging messages ( #1524 )
2023-10-31 09:04:47 -07:00
Zhuohan Li
651c614aa4
Bump up the version to v0.2.1 ( #1355 )
2023-10-16 12:58:57 -07:00
Zhuohan Li and Yunmo Chen
9d9072a069
Implement prompt logprobs & Batched topk for computing logprobs ( #1328 )
...
Co-authored-by: Yunmo Chen <16273544+wanmok@users.noreply.github.com >
2023-10-16 10:56:50 -07:00
Zhuohan Li
b95ee898fe
[Minor] Fix comment in mistral.py ( #1303 )
2023-10-09 19:44:37 -07:00
Zhuohan Li
9eed4d1f3e
Update README.md ( #1292 )
2023-10-08 23:15:50 -07:00
Zhuohan Li
6b5296aa3a
[FIX] Explain why the finished_reason of ignored sequences are length ( #1289 )
2023-10-08 15:22:38 -07:00
Zhuohan Li
ba0bfd40e2
TP/quantization/weight loading refactor part 1 - Simplify parallel linear logic ( #1181 )
2023-10-02 15:36:09 -07:00
Zhuohan Li and JiCheng
a60b353005
support sharding llama2-70b on more than 8 GPUs ( #1209 )
...
Co-authored-by: JiCheng <247153481@qq.com >
2023-10-02 15:26:33 -07:00
Zhuohan Li
f187877945
[FIX] Simplify sampler logic ( #1156 )
2023-09-23 17:21:56 -07:00
Zhuohan Li and Antoni Baum
947b794146
[Sampler] Vectorized sampling (simplified) ( #1048 )
...
Co-authored-by: Antoni Baum <antoni.baum@protonmail.com >
2023-09-22 17:48:04 -07:00
Zhuohan Li
c1026311b5
[Community] Add vLLM Discord server ( #1086 )
2023-09-18 12:23:35 -07:00
Zhuohan Li
f029ef94d7
Fix get_max_num_running_seqs for waiting and swapped seq groups ( #1068 )
2023-09-18 11:49:40 -07:00
Zhuohan Li
90979c38f8
[FIX] Don't initialize parameter by default ( #1067 )
2023-09-17 17:15:38 -07:00
Zhuohan Li
f04908cae7
[FIX] Minor bug fixes ( #1035 )
...
* [FIX] Minor bug fixes
* Address review comments
2023-09-13 16:38:12 -07:00
Zhuohan Li
1117aa1411
Bump up the version to v0.1.6 ( #989 )
2023-09-08 00:07:46 -07:00
db09d4ad83
[FIX] Fix Alibi implementation in PagedAttention kernel ( #945 )
...
* [FIX] Fix Alibi implementation in PagedAttention kernel
* Fix test_attention
* Fix
---------
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu >
Co-authored-by: Oliver-ss <yuansongwx@outlook.com >
2023-09-07 15:53:14 -07:00
Zhuohan Li
c957c741d9
Enable safetensors loading for all models ( #974 )
2023-09-07 15:49:52 -07:00
Zhuohan Li
002800f081
Align vLLM's beam search implementation with HF generate ( #857 )
2023-09-04 17:29:42 -07:00
Zhuohan Li
c128d69856
Fix README.md Link ( #927 )
2023-08-31 17:18:34 -07:00
Zhuohan Li
0080d8329d
Add acknowledgement to a16z grant
2023-08-30 02:26:47 -07:00
Zhuohan Li
d2b2eed67c
[Fix] Fix a condition for ignored sequences ( #867 )
2023-08-27 23:00:56 -07:00
Zhuohan Li
14f9c72bfd
Update Supported Model List ( #825 )
2023-08-22 11:51:44 -07:00
Zhuohan Li
aa84c92ef6
Bump up version to 0.1.3 ( #657 )
2023-08-02 16:46:53 -07:00
Zhuohan Li
f7389f4763
[Doc] Add Baichuan 13B to supported models ( #656 )
2023-08-02 16:45:12 -07:00
Zhuohan Li
1b0bd0fe8a
Add Falcon support (new) ( #592 )
2023-08-02 14:04:39 -07:00
Zhuohan Li
58a072be15
[Fix] Add model sequence length into model config ( #575 )
2023-07-25 23:46:30 -07:00
Zhuohan Li
82ad323dee
[Fix] Add chat completion Example and simplify dependencies ( #576 )
2023-07-25 23:45:48 -07:00
Zhuohan Li
df5dd3c68e
Add Baichuan-7B to README ( #494 )
2023-07-25 15:25:12 -07:00
Zhuohan Li
7d5a155e4a
[Fix] Fix GPTBigcoder for distributed execution ( #503 )
2023-07-24 18:36:33 -07:00
Zhuohan Li
6fc2a38b11
Add support for LLaMA-2 ( #505 )
2023-07-20 11:38:27 -07:00
Zhuohan Li
2bdea7ac11
[Fix] Fix the condition of max_seq_len ( #477 )
2023-07-17 00:33:48 -04:00
Zhuohan Li
96853af5a8
Optimize MQA Kernel ( #452 )
2023-07-14 20:06:40 -04:00
Zhuohan Li
98fe8cb542
[Server] Add option to specify chat template for chat endpoint ( #345 )
2023-07-03 23:01:56 -07:00
Zhuohan Li
42e0c1df78
[Quality] Add CI for formatting ( #343 )
2023-07-03 14:50:56 -07:00
Zhuohan Li
d6fa1be3a8
[Quality] Add code formatter and linter ( #326 )
2023-07-03 11:31:55 -07:00
Zhuohan Li
0ffded812a
[Fix] Better error message for batched prompts ( #342 )
2023-07-03 09:27:31 -07:00
Zhuohan Li
598dc4b79a
[Fix] Weight loading for GPTBigCode ( #313 )
2023-06-29 22:14:17 -07:00
Zhuohan Li
85de093472
[Fix] Do not pin memory when in WSL ( #312 )
2023-06-29 15:00:21 -07:00
Zhuohan Li
43710e8d09
[Fix] Fix default port number in benchmark scripts ( #265 )
2023-06-26 13:15:35 -07:00
Zhuohan Li
2cf1a333b6
[Doc] Documentation for distributed inference ( #261 )
2023-06-26 11:34:23 -07:00
Zhuohan Li
0b7db411b5
[Bug] Fix the OOM condition for CPU cache ( #260 )
2023-06-26 11:16:13 -07:00
Zhuohan Li
83658c8ace
Bump up version to 0.1.1 ( #204 )
2023-06-22 15:33:32 +08:00
Zhuohan Li
1d24ccb96c
[Fix] Better error message when there is OOM during cache initialization ( #203 )
2023-06-22 15:30:06 +08:00
Zhuohan Li
2e0d314384
fix-ray ( #193 )
2023-06-22 00:21:41 +08:00
Zhuohan Li
033f5c78f5
Remove e.g. in README ( #167 )
2023-06-20 14:00:28 +08:00
Zhuohan Li
fc72e39de3
Change image urls ( #164 )
2023-06-20 11:15:15 +08:00
Zhuohan Li
0b32a987dd
Add and list supported models in README ( #161 )
2023-06-20 10:57:46 +08:00
Zhuohan Li
a255885f83
Add logo and polish readme ( #156 )
2023-06-19 16:31:13 +08:00
Zhuohan Li
0370afa2e5
Remove benchmark_async_llm_server.py ( #155 )
2023-06-19 11:12:37 +08:00
Zhuohan Li
bf5f121c02
Reduce GPU memory utilization to make sure OOM doesn't happen ( #153 )
2023-06-18 17:33:50 +08:00
Zhuohan Li
bec7b2dc26
Add quickstart guide ( #148 )
2023-06-18 01:26:12 +08:00
Zhuohan Li
e5464ee484
Rename servers to engines ( #152 )
2023-06-17 17:25:21 +08:00
Zhuohan Li
eedb46bf03
Rename servers and change port numbers to reduce confusion ( #149 )
2023-06-17 00:13:02 +08:00
Zhuohan Li
5020e1e80c
Non-streaming simple fastapi server ( #144 )
2023-06-10 10:43:07 -07:00
Zhuohan Li
4298374265
Add docstrings for LLMServer and related classes and examples ( #142 )
2023-06-07 18:25:20 +08:00
Zhuohan Li
1a956e136b
Fix various issues of async servers ( #135 )
2023-06-05 23:44:50 +08:00
Zhuohan Li
057daef778
OpenAI Compatible Frontend ( #116 )
2023-05-23 21:39:50 -07:00