703e42ee4b
Add guided decoding for OpenAI API server ( #2819 )
...
Co-authored-by: br3no <breno@veltefaria.de >
Co-authored-by: simon-mo <simon.mo@hey.com >
2024-02-29 22:13:08 +00:00
zspo
c664b0e683
fix some bugs ( #2689 )
2024-01-31 10:09:23 -08:00
Wen Sun
d79ced3292
Fix 'Actor methods cannot be called directly' when using --engine-use-ray ( #2664 )
...
* fix: engine-useray complain
* fix: typo
2024-01-30 17:17:05 +01:00
Murali Andoorveedu
89be30fa7d
Small async_llm_engine refactor ( #2618 )
2024-01-27 23:28:37 -08:00
9b945daaf1
[Experimental] Add multi-LoRA support ( #1804 )
...
Co-authored-by: Chen Shen <scv119@gmail.com >
Co-authored-by: Shreyas Krishnaswamy <shrekris@anyscale.com >
Co-authored-by: Avnish Narayan <avnish@anyscale.com >
2024-01-23 15:26:37 -08:00
d10f8e1d43
[Experimental] Prefix Caching Support ( #1669 )
...
Co-authored-by: DouHappy <2278958187@qq.com >
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com >
2024-01-17 16:32:10 -08:00
Jiaxiang
6549aef245
[DOC] Add additional comments for LLMEngine and AsyncLLMEngine ( #1011 )
2024-01-11 19:26:49 -08:00
Iskren Ivov Chernev
d0215a58e7
Ensure metrics are logged regardless of requests ( #2347 )
2024-01-05 05:24:42 -08:00
Zhuohan Li
fd4ea8ef5c
Use NCCL instead of ray for control-plane communication to remove serialization overhead ( #2221 )
2024-01-03 11:30:22 -08:00
Zhuohan Li
e0ff920001
[BUGFIX] Do not return ignored sentences twice in async llm engine ( #2258 )
2023-12-26 13:41:09 +08:00
mezuzza
6774bd50b0
Fix typing in AsyncLLMEngine & add toml to requirements-dev ( #2100 )
2023-12-14 00:19:41 -08:00
Woosuk Kwon
464dd985e3
Fix num_gpus when TP > 1 ( #1852 )
2023-12-03 12:24:30 -08:00
Zhuohan Li
7076fa1c9f
TP/quantization/weight loading refactor part 2 - Refactor quantized linear logic and extend quantization support to all models ( #1622 )
...
Refactor the tensor parallelism, quantization, and weight-loading codes.
Summary of the new features enabled by this PR:
- **All models** are able to be quantized with AWQ and SqueezeLLM, and [soon GPTQ](https://github.com/vllm-project/vllm/pull/1580 ).
- Model loading code became much simpler.
- Support model parallelism for all MQA/GQA models when the number of key/value heads is smaller than the tensor parallel size.
2023-11-15 22:50:41 -08:00
Dominik Schwabe
1b290ace4f
Run default _AsyncLLMEngine._run_workers_async in threadpool ( #1628 )
2023-11-11 14:50:44 -08:00
ljss
5687d584fe
[BugFix] Set engine_use_ray=True when TP>1 ( #1531 )
2023-11-01 02:14:18 -07:00
Antoni Baum
acbed3ef40
Use monotonic time where appropriate ( #1249 )
2023-10-02 19:22:05 -07:00
Roy
95592fa00a
align llm_engine and async_engine. ( #1081 )
2023-09-18 11:49:10 -07:00
Antoni Baum
ff36139ffc
Remove AsyncLLMEngine busy loop, shield background task ( #1059 )
2023-09-17 00:29:08 -07:00
Jerry Yang
b9fe4616f9
Abort when coroutine is cancelled ( #1020 )
2023-09-14 17:40:18 -07:00
d6545ad22e
add option to shorten prompt print in log ( #991 )
...
Signed-off-by: Lei Wen <wenlei03@qiyi.com >
Co-authored-by: Lei Wen <wenlei03@qiyi.com >
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com >
2023-09-12 15:10:14 -07:00
Antoni Baum and Zhuohan Li
080438477f
Start background task in AsyncLLMEngine.generate ( #988 )
...
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com >
2023-09-08 00:03:39 -07:00
Antoni Baum and Avnish Narayan
c07ece5ca4
Make AsyncLLMEngine more robust & fix batched abort ( #969 )
...
Signed-off-by: Antoni Baum <antoni.baum@protonmail.com >
Co-authored-by: Avnish Narayan <38871737+avnishn@users.noreply.github.com >
2023-09-07 13:43:45 -07:00
Antoni Baum
c9927c1a6a
Use queue for finished requests ( #957 )
2023-09-05 19:27:23 -07:00
Wen Sun
22379d5513
fix: typo ( #948 )
2023-09-04 23:22:30 -07:00
Antoni Baum
1696725879
Initialize AsyncLLMEngine bg loop correctly ( #943 )
2023-09-04 17:41:22 -07:00
Antoni Baum
ce741ba3e4
Refactor AsyncLLMEngine ( #880 )
2023-09-03 21:43:43 -07:00
Antoni Baum
9925c17940
Ray placement group support ( #397 )
2023-07-19 22:49:31 -07:00
coolcloudcol
7717d0838b
Fix an endless loop issue when engine_step throws a RuntimeError ( #339 )
2023-07-03 15:22:28 -07:00
Zhuohan Li
42e0c1df78
[Quality] Add CI for formatting ( #343 )
2023-07-03 14:50:56 -07:00
Zhuohan Li
d6fa1be3a8
[Quality] Add code formatter and linter ( #326 )
2023-07-03 11:31:55 -07:00
Woosuk Kwon
14f0b39cda
[Bugfix] Fix a bug in RequestOutput.finished ( #202 )
2023-06-22 00:17:24 -07:00
Woosuk Kwon
0b98ba15c7
Change the name to vLLM ( #150 )
2023-06-17 03:07:40 -07:00
