Commit Graph
100 Commits
Author SHA1 Message Date
Zhuohan Li 326fcc8b9f [Doc] Deprecate Discord (#12668) 2025-02-02 19:19:56 -08:00
Zhuohan Li 49628fe13e [Doc] Update README.md with Ray Summit talk links (#10610) 2024-11-24 16:45:09 -08:00
Zhuohan Li a95354a36e [Doc] Update README.md with Ray summit slides (#9088) 2024-10-05 02:54:45 +00:00
Zhuohan Li bd0e7802e0 [Bugfix] Add warmup for prefix caching example (#5235) 2024-06-03 19:36:41 -07:00
Zhuohan Li 8279078e21 [Bugfix] Remove deprecated @abstractproperty (#5174) 2024-06-01 22:40:25 +00:00
1102bef219 [Bugfix / Core] Prefix Caching Guards (merged with main) (#4846)
Co-authored-by: rsnm2 <rshaw@neuralmagic.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com>
2024-05-27 15:18:17 -07:00
Zhuohan Li 361c461a12 [Doc] Highlight the fourth meetup in the README (#4842) 2024-05-15 11:38:49 -07:00
Zhuohan Li c579b750a0 [Doc] Add meetups to the doc (#4798) 2024-05-13 18:48:00 -07:00
Zhuohan Li ac1fbf7fd2 [Doc] Shorten README by removing supported model list (#4796) 2024-05-13 16:23:54 -07:00
Zhuohan Li e11e200736 [Bugfix] Fix filelock version requirement (#4075) 2024-04-14 21:50:08 -07:00
Zhuohan Li d4ec9ffb95 [Misc] Fix typo in scheduler.py (#4022) 2024-04-12 13:56:04 -07:00
Zhuohan Li e90fc21f2e [Hardware][Neuron] Refactor neuron support (#3471) 2024-03-22 01:22:17 +00:00
Zhuohan Li 523e30ea0c [BugFix] Hot fix in setup.py for neuron build (#3537) 2024-03-20 17:59:52 -07:00
Zhuohan Li b30880a762 [Misc] Update README for the Third vLLM Meetup (#3479) 2024-03-18 15:58:38 -07:00
Zhuohan Li c0c17d4896 [Misc] Fix PR Template (#3478) 2024-03-18 15:00:31 -07:00
Zhuohan Li eeab52a4ff [FIX] Simpler fix for async engine running on ray (#3371) 2024-03-13 14:18:40 -07:00
Zhuohan Li 4c922709b6 Add distributed model executor abstraction (#3191) 2024-03-11 11:03:45 -07:00
Zhuohan Li 2f8844ba08 Re-enable the 80 char line width limit (#3305) 2024-03-10 19:49:14 -07:00
Zhuohan Li f48c6791b7 [FIX] Fix prefix test error on main (#3286) 2024-03-08 17:16:14 -08:00
Zhuohan Li 996d095c54 [FIX] Fix styles in automatic prefix caching & add a automatic prefix caching benchmark (#3158) 2024-03-03 14:37:18 -08:00
Zhuohan Li 8fbd84bf78 Bump up version to v0.3.2 (#2968)
This version is for more model support. Add support for Gemma models (#2964) and OLMo models (#2832).
2024-02-21 11:47:25 -08:00
Zhuohan Li a9c8212895 [FIX] Add Gemma model to the doc (#2966) 2024-02-21 09:46:15 -08:00
Zhuohan Li 63e2a6419d [FIX] Fix beam search test (#2930) 2024-02-20 14:37:39 -08:00
Zhuohan Li a61f0521b8 [Test] Add basic correctness test (#2908) 2024-02-18 16:44:50 -08:00
Zhuohan Li 537c9755a7 [Minor] Small fix to make distributed init logic in worker looks cleaner (#2905) 2024-02-18 14:39:00 -08:00
Zhuohan Li 1af090b57d Bump up version to v0.3.0 (#2656) 2024-01-31 00:07:07 -08:00
Zhuohan Li ef9b636e2d Simplify broadcast logic for control messages (#2501) 2024-01-19 11:23:30 -08:00
Zhuohan Li f745847ef7 [Minor] Fix the format in quick start guide related to Model Scope (#2425) 2024-01-11 19:44:01 -08:00
Zhuohan Li fd4ea8ef5c Use NCCL instead of ray for control-plane communication to remove serialization overhead (#2221) 2024-01-03 11:30:22 -08:00
Zhuohan Li 358c328d69 [BUGFIX] Fix communication test (#2285) 2023-12-27 17:18:11 -05:00
Zhuohan Li 4aaafdd289 [BUGFIX] Fix the path of test prompts (#2273) 2023-12-26 10:37:21 -08:00
Zhuohan Li 66b108d142 [BUGFIX] Fix API server test (#2270) 2023-12-26 10:37:06 -08:00
Zhuohan Li e0ff920001 [BUGFIX] Do not return ignored sentences twice in async llm engine (#2258) 2023-12-26 13:41:09 +08:00
Zhuohan Li e19a64c7ef [FIX] Fix formatting error in main branch (#1822) 2023-11-28 16:56:43 -08:00
Zhuohan Li 1cb4ad8de9 [FIX] Fix formatting error 2023-11-29 00:40:19 +00:00
Zhuohan Li 708e6c18b0 [FIX] Fix class naming (#1803) 2023-11-28 14:08:01 -08:00
Zhuohan Li 7d761fe3c1 [FIX] Fix the case when input_is_parallel=False for ScaledActivation (#1737) 2023-11-20 23:56:48 -08:00
Zhuohan Li 32c927b53f [FIX] Update the doc link in README.md (#1730) 2023-11-20 12:46:24 -08:00
Zhuohan Li 0fc280b06c Update the adding-model doc according to the new refactor (#1692) 2023-11-16 18:46:26 -08:00
Zhuohan Li 20d0699d49 [Fix] Fix comm test (#1691) 2023-11-16 16:28:39 -08:00
Zhuohan Li 415d109527 [Fix] Update Supported Models List (#1690) 2023-11-16 14:47:26 -08:00
Zhuohan Li 7076fa1c9f TP/quantization/weight loading refactor part 2 - Refactor quantized linear logic and extend quantization support to all models (#1622)
Refactor the tensor parallelism, quantization, and weight-loading codes.

Summary of the new features enabled by this PR:
- **All models** are able to be quantized with AWQ and SqueezeLLM, and [soon GPTQ](https://github.com/vllm-project/vllm/pull/1580).
- Model loading code became much simpler.
- Support model parallelism for all MQA/GQA models when the number of key/value heads is smaller than the tensor parallel size.
2023-11-15 22:50:41 -08:00
Zhuohan LiandWoosuk Kwon 06458a0b42 Upgrade to CUDA 12 (#1527)
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
2023-11-08 14:17:49 -08:00
Zhuohan Li 7b895c5976 [Fix] Fix duplicated logging messages (#1524) 2023-10-31 09:04:47 -07:00
Zhuohan Li 651c614aa4 Bump up the version to v0.2.1 (#1355) 2023-10-16 12:58:57 -07:00
Zhuohan LiandYunmo Chen 9d9072a069 Implement prompt logprobs & Batched topk for computing logprobs (#1328)
Co-authored-by: Yunmo Chen <16273544+wanmok@users.noreply.github.com>
2023-10-16 10:56:50 -07:00
Zhuohan Li b95ee898fe [Minor] Fix comment in mistral.py (#1303) 2023-10-09 19:44:37 -07:00
Zhuohan Li 9eed4d1f3e Update README.md (#1292) 2023-10-08 23:15:50 -07:00
Zhuohan Li 6b5296aa3a [FIX] Explain why the finished_reason of ignored sequences are length (#1289) 2023-10-08 15:22:38 -07:00
Zhuohan Li ba0bfd40e2 TP/quantization/weight loading refactor part 1 - Simplify parallel linear logic (#1181) 2023-10-02 15:36:09 -07:00
Zhuohan LiandJiCheng a60b353005 support sharding llama2-70b on more than 8 GPUs (#1209)
Co-authored-by: JiCheng <247153481@qq.com>
2023-10-02 15:26:33 -07:00
Zhuohan Li f187877945 [FIX] Simplify sampler logic (#1156) 2023-09-23 17:21:56 -07:00
Zhuohan LiandAntoni Baum 947b794146 [Sampler] Vectorized sampling (simplified) (#1048)
Co-authored-by: Antoni Baum <antoni.baum@protonmail.com>
2023-09-22 17:48:04 -07:00
Zhuohan Li c1026311b5 [Community] Add vLLM Discord server (#1086) 2023-09-18 12:23:35 -07:00
Zhuohan Li f029ef94d7 Fix get_max_num_running_seqs for waiting and swapped seq groups (#1068) 2023-09-18 11:49:40 -07:00
Zhuohan Li 90979c38f8 [FIX] Don't initialize parameter by default (#1067) 2023-09-17 17:15:38 -07:00
Zhuohan Li f04908cae7 [FIX] Minor bug fixes (#1035)
* [FIX] Minor bug fixes

* Address review comments
2023-09-13 16:38:12 -07:00
Zhuohan Li 1117aa1411 Bump up the version to v0.1.6 (#989) 2023-09-08 00:07:46 -07:00
db09d4ad83 [FIX] Fix Alibi implementation in PagedAttention kernel (#945)
* [FIX] Fix Alibi implementation in PagedAttention kernel

* Fix test_attention

* Fix

---------

Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: Oliver-ss <yuansongwx@outlook.com>
2023-09-07 15:53:14 -07:00
Zhuohan Li c957c741d9 Enable safetensors loading for all models (#974) 2023-09-07 15:49:52 -07:00
Zhuohan Li 002800f081 Align vLLM's beam search implementation with HF generate (#857) 2023-09-04 17:29:42 -07:00
Zhuohan Li c128d69856 Fix README.md Link (#927) 2023-08-31 17:18:34 -07:00
Zhuohan Li 0080d8329d Add acknowledgement to a16z grant 2023-08-30 02:26:47 -07:00
Zhuohan Li d2b2eed67c [Fix] Fix a condition for ignored sequences (#867) 2023-08-27 23:00:56 -07:00
Zhuohan Li 14f9c72bfd Update Supported Model List (#825) 2023-08-22 11:51:44 -07:00
Zhuohan Li aa84c92ef6 Bump up version to 0.1.3 (#657) 2023-08-02 16:46:53 -07:00
Zhuohan Li f7389f4763 [Doc] Add Baichuan 13B to supported models (#656) 2023-08-02 16:45:12 -07:00
Zhuohan Li 1b0bd0fe8a Add Falcon support (new) (#592) 2023-08-02 14:04:39 -07:00
Zhuohan Li 58a072be15 [Fix] Add model sequence length into model config (#575) 2023-07-25 23:46:30 -07:00
Zhuohan Li 82ad323dee [Fix] Add chat completion Example and simplify dependencies (#576) 2023-07-25 23:45:48 -07:00
Zhuohan Li df5dd3c68e Add Baichuan-7B to README (#494) 2023-07-25 15:25:12 -07:00
Zhuohan Li 7d5a155e4a [Fix] Fix GPTBigcoder for distributed execution (#503) 2023-07-24 18:36:33 -07:00
Zhuohan Li 6fc2a38b11 Add support for LLaMA-2 (#505) 2023-07-20 11:38:27 -07:00
Zhuohan Li 2bdea7ac11 [Fix] Fix the condition of max_seq_len (#477) 2023-07-17 00:33:48 -04:00
Zhuohan Li 96853af5a8 Optimize MQA Kernel (#452) 2023-07-14 20:06:40 -04:00
Zhuohan Li 98fe8cb542 [Server] Add option to specify chat template for chat endpoint (#345) 2023-07-03 23:01:56 -07:00
Zhuohan Li 42e0c1df78 [Quality] Add CI for formatting (#343) 2023-07-03 14:50:56 -07:00
Zhuohan Li d6fa1be3a8 [Quality] Add code formatter and linter (#326) 2023-07-03 11:31:55 -07:00
Zhuohan Li 0ffded812a [Fix] Better error message for batched prompts (#342) 2023-07-03 09:27:31 -07:00
Zhuohan Li 598dc4b79a [Fix] Weight loading for GPTBigCode (#313) 2023-06-29 22:14:17 -07:00
Zhuohan Li 85de093472 [Fix] Do not pin memory when in WSL (#312) 2023-06-29 15:00:21 -07:00
Zhuohan Li 43710e8d09 [Fix] Fix default port number in benchmark scripts (#265) 2023-06-26 13:15:35 -07:00
Zhuohan Li 2cf1a333b6 [Doc] Documentation for distributed inference (#261) 2023-06-26 11:34:23 -07:00
Zhuohan Li 0b7db411b5 [Bug] Fix the OOM condition for CPU cache (#260) 2023-06-26 11:16:13 -07:00
Zhuohan Li 83658c8ace Bump up version to 0.1.1 (#204) 2023-06-22 15:33:32 +08:00
Zhuohan Li 1d24ccb96c [Fix] Better error message when there is OOM during cache initialization (#203) 2023-06-22 15:30:06 +08:00
Zhuohan Li 2e0d314384 fix-ray (#193) 2023-06-22 00:21:41 +08:00
Zhuohan Li 033f5c78f5 Remove e.g. in README (#167) 2023-06-20 14:00:28 +08:00
Zhuohan Li fc72e39de3 Change image urls (#164) 2023-06-20 11:15:15 +08:00
Zhuohan Li 0b32a987dd Add and list supported models in README (#161) 2023-06-20 10:57:46 +08:00
Zhuohan Li a255885f83 Add logo and polish readme (#156) 2023-06-19 16:31:13 +08:00
Zhuohan Li 0370afa2e5 Remove benchmark_async_llm_server.py (#155) 2023-06-19 11:12:37 +08:00
Zhuohan Li bf5f121c02 Reduce GPU memory utilization to make sure OOM doesn't happen (#153) 2023-06-18 17:33:50 +08:00
Zhuohan Li bec7b2dc26 Add quickstart guide (#148) 2023-06-18 01:26:12 +08:00
Zhuohan Li e5464ee484 Rename servers to engines (#152) 2023-06-17 17:25:21 +08:00
Zhuohan Li eedb46bf03 Rename servers and change port numbers to reduce confusion (#149) 2023-06-17 00:13:02 +08:00
Zhuohan Li 5020e1e80c Non-streaming simple fastapi server (#144) 2023-06-10 10:43:07 -07:00
Zhuohan Li 4298374265 Add docstrings for LLMServer and related classes and examples (#142) 2023-06-07 18:25:20 +08:00
Zhuohan Li 1a956e136b Fix various issues of async servers (#135) 2023-06-05 23:44:50 +08:00
Zhuohan Li 057daef778 OpenAI Compatible Frontend (#116) 2023-05-23 21:39:50 -07:00