iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
1,173 Commits 3 Branches 0 Tags
3cd9b5bb2d4a0d5eed07186ae140f5dc8f839708
Commit Graph
91 Commits
Author SHA1 Message Date
Cyrus Leung 1e8f4252aa [Bugfix][Frontend] Raise exception when file-like chat template fails to be opened (#4292) 2024-04-23 18:19:03 +00:00
Jack Gordley d3c8180ac4 [Bugfix] Fixing max token error message for openai compatible server (#4016) 2024-04-23 19:06:29 +08:00
SangBin Cho 0ae11f78ab [Mypy] Part 3 fix typing for nested directories for most of directory (#4161) 2024-04-22 21:32:44 -07:00
Chirag Jain bc9df1571b Pass tokenizer_revision when getting tokenizer in openai serving (#4214) 2024-04-19 17:13:56 -07:00
James Whedbee e1bb2fd52d [Bugfix] Support logprobs when using guided_json and other constrained decoding fields (#4149) 2024-04-18 21:12:55 +00:00
Harry MellorandAlexandre Payot 66ded03067 Allow model to be served under multiple names (#2894) 
Co-authored-by: Alexandre Payot <alexandrep@graphcore.ai>
 2024-04-18 00:16:26 -07:00
Noam GatandSimon Mo 05434764cd LM Format Enforcer Guided Decoding Support (#3868) 
Co-authored-by: Simon Mo <simon.mo@hey.com>
 2024-04-16 05:54:57 +00:00
Cyrus Leung 7fd3949a0b [Frontend][Core] Move merge_async_iterators to utils (#4026) 2024-04-12 05:30:54 +00:00
Dylan HawkandDylan Hawk 95e7d4a97c Fix echo/logprob OpenAI completion bug (#3441) 
Co-authored-by: Dylan Hawk <dylanwawk@gmail.com>
 2024-04-11 22:15:50 +00:00
Thomas Parnell 1d7c940d74 Add option to completion API to truncate prompt tokens (#3144) 2024-04-05 10:15:42 -07:00
A-MahlaandA-Mahla <> 0739b1947f [Frontend][Bugfix] allow using the default middleware with a root path (#3788) 
Co-authored-by: A-Mahla <>
 2024-04-02 01:20:28 -07:00
Roy f510395bbf [BugFix][Frontend] Fix completion logprobs=0 error (#3731) 2024-03-29 09:38:21 -07:00
Roy 6110c39dc8 [BugFix] Fix tokenizer out of vocab size (#3685) 2024-03-29 08:18:59 -07:00
yhu422 d8658c8cc1 Usage Stats Collection (#2852) 2024-03-28 22:16:12 -07:00
Nick HillandSahil Suneja dfeb2ecc3a [Misc] Include matched stop string/token in responses (#2976) 
Co-authored-by: Sahil Suneja <sahilsuneja@gmail.com>
 2024-03-25 17:31:32 -07:00
Dylan HawkandDylan Hawk 0b4997e05c [Bugfix] API stream returning two stops (#3450) 
Co-authored-by: Dylan Hawk <dylanwawk@gmail.com>
 2024-03-25 10:14:34 -07:00
Travis JohnsonandNick Hill c13ad1b7bd feat: implement the min_tokens sampling parameter (#3124) 
Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
Co-authored-by: Nick Hill <nickhill@us.ibm.com>
 2024-03-25 10:14:26 -07:00
SangBin Cho 01bfb22b41 [CI] Try introducing isort. (#3495) 2024-03-25 07:59:47 -07:00
Roy 865732342b [Misc][Log] Add log for tokenizer length not equal to vocabulary size (#3500) 2024-03-21 18:07:48 +08:00
Simon Mo ef65dcfa6f [Doc] Add docs about OpenAI compatible server (#3288) 2024-03-18 22:05:34 -07:00
Simon Mo 120157fd2a Support arbitrary json_object in OpenAI and Context Free Grammar (#3211) 2024-03-16 13:35:27 -07:00
Robert Shaw 10585e035e Removed Extraneous Print Message From OAI Server (#3440) 2024-03-16 00:35:36 +00:00
Tao Heandsimon-mo 14b8ae02e7 Fixes the misuse/mixuse of time.time()/time.monotonic() (#3220) 
Signed-off-by: Tao He <sighingnow@gmail.com>
Co-authored-by: simon-mo <simon.mo@hey.com>
 2024-03-15 18:25:43 +00:00
Dan Clarkanddeclark1 03d37f2441 [Fix] Add args for mTLS support (#3430) 
Co-authored-by: declark1 <daniel.clark@ibm.com>
 2024-03-15 09:56:13 -07:00
Zhuohan Li 2f8844ba08 Re-enable the 80 char line width limit (#3305) 2024-03-10 19:49:14 -07:00
Roy 9e8744a545 [BugFix] Fix get tokenizer when using ray (#3301) 2024-03-10 19:17:16 -07:00
Nick Hill d2339d6840 Connect engine healthcheck to openai server (#3260) 2024-03-07 16:38:12 -08:00
Antoni BaumandAvnish Narayan 22de45235c Push logprob generation to LLMEngine (#3065) 
Co-authored-by: Avnish Narayan <avnish@anyscale.com>
 2024-03-04 19:54:06 +00:00
Jason Cox d65fac2738 Add vLLM version info to logs and openai API server (#3161) 2024-03-02 21:00:29 -08:00
Allen.DouzixiaoSimon Mo
 		29e70e3e88 allow user chose log level by --log-level instead of fixed 'info'. (#3109) 
Co-authored-by: zixiao <shunli.dsl@alibaba-inc.com>
Co-authored-by: Simon Mo <simon.mo@hey.com>
 2024-03-01 23:28:41 +00:00
Huarongandhuohuarong 90fbf12540 fix relative import path of protocol.py (#3134) 
Co-authored-by: huohuarong <huohuarong@zuoshouyisheng.com>
 2024-03-01 19:42:06 +00:00
Seonghyeon 27ca23dc00 Remove exclude_unset in streaming response (#3143) 2024-03-01 09:59:06 -08:00
felixzhu555br3nosimon-mo
 		703e42ee4b Add guided decoding for OpenAI API server (#2819) 
Co-authored-by: br3no <breno@veltefaria.de>
Co-authored-by: simon-mo <simon.mo@hey.com>
 2024-02-29 22:13:08 +00:00
Jae-Won Chung a6d471c759 Fix: AttributeError in OpenAI-compatible server (#3018) 2024-02-28 22:04:07 -08:00
Dylan Hawk e0ade06d63 Support logit bias for OpenAI API (#3027) 2024-02-27 11:51:53 +08:00
Jared Moore 70f3e8e3a1 Add LogProbs for Chat Completions in OpenAI (#2918) 2024-02-26 10:39:34 +08:00
Harry Mellor ef978fe411 Port metrics from aioprometheus to prometheus_client (#2730) 2024-02-25 11:54:00 -08:00
Mustafa Eyceoz 5574081c49 Added early stopping to completion APIs (#2939) 2024-02-21 18:24:01 -08:00
Nick Hill 7d2dcce175 Support per-request seed (#2514) 2024-02-21 11:47:00 -08:00
jvmncs 8f36444c4f multi-LoRA as extra models in OpenAI server (#2775) 
how to serve the loras (mimicking the [multilora inference example](https://github.com/vllm-project/vllm/blob/main/examples/multilora_inference.py)):
```terminal
$ export LORA_PATH=~/.cache/huggingface/hub/models--yard1--llama-2-7b-sql-lora-test/
$ python -m vllm.entrypoints.api_server \
 --model meta-llama/Llama-2-7b-hf \
 --enable-lora \
 --lora-modules sql-lora=$LORA_PATH sql-lora2=$LORA_PATH
```
the above server will list 3 separate values if the user queries `/models`: one for the base served model, and one each for the specified lora modules. in this case sql-lora and sql-lora2 point to the same underlying lora, but this need not be the case. lora config values take the same values they do in EngineArgs

no work has been done here to scope client permissions to specific models
 2024-02-17 12:00:48 -08:00
Simon Mo b9e96b17de fix python 3.8 syntax (#2716) 2024-02-01 14:00:58 -08:00
Simon Mo 3a7dd7e367 Support Batch Completion in Server (#2529) 2024-01-24 17:11:07 -08:00
Federico Galatolo f1f6cc10c7 Added include_stop_str_in_output and length_penalty parameters to OpenAI API (#2562) 2024-01-24 10:21:56 -08:00
Erfan Al-Hossami 9c1352eb57 [Feature] Simple API token authentication and pluggable middlewares (#1106) 2024-01-23 15:13:00 -08:00
Jannis Schönleber 71d63ed72e migrate pydantic from v1 to v2 (#2531) 2024-01-21 16:05:56 -08:00
Simon Mo dd7e8f5f64 refactor complemention api for readability (#2499) 2024-01-18 16:45:14 -08:00
FlorianJoncour 14cc317ba4 OpenAI Server refactoring (#2360) 2024-01-16 21:33:14 -08:00
Chirag Jain ce036244c9 Allow setting fastapi root_path argument (#2341) 2024-01-12 10:59:59 -08:00
Iskren Ivov Chernev d0215a58e7 Ensure metrics are logged regardless of requests (#2347) 2024-01-05 05:24:42 -08:00
Harry Mellor 08133c4d1a Add SSL arguments to API servers (#2109) 2023-12-18 10:56:23 +08:00