iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
3,987 Commits 3 Branches 0 Tags
07064cb1d49d2b04ec58d8876bee2cd8281eedf5
Commit Graph
424 Commits
Author SHA1 Message Date
Nathan Azrak 68d37809b9 [Misc] Minimum requirements for SageMaker compatibility (#11576) 2025-01-02 15:59:25 -08:00
Joe Runde 4db72e57f6 [Bugfix][Refactor] Unify model management in frontend (#11660) 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
 2025-01-01 02:21:51 +00:00
Robert Shaw 5886aa496e [V1] [6/N] API Server: Better Shutdown (#11586) 2024-12-30 15:51:02 +00:00
Robert Shaw df04dffade [V1] [4/N] API Server: ZMQ/MP Utilities (#11541) 2024-12-28 01:45:08 +00:00
Cyrus Leung 7af553ea30 [Misc] Abstract the logic for reading and writing media content (#11527) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-12-27 19:21:23 +08:00
Robert Shaw 55fb97f7bd [2/N] API Server: Avoid ulimit footgun (#11530) 2024-12-26 23:43:05 +00:00
Robert Shaw 720b10fdc6 [1/N] API Server (Remove Proxy) (#11529) 2024-12-26 23:03:43 +00:00
Cyrus Leung 9edca6bf8f [Frontend] Online Pooling API (#11457) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-12-24 17:54:30 +08:00
dpxaandshiquan <shiquan> 4f074fbf53 [Misc]Suppress irrelevant exception stack trace information when CUDA… (#11438) 
Co-authored-by: shiquan <shiquan>
 2024-12-24 08:43:39 +00:00
Rafael Vasquez 32aa2059ad [Docs] Convert rST to MyST (Markdown) (#11145) 
Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>
 2024-12-23 22:35:38 +00:00
Ricky Xu 584f0ae40d [V1] Make AsyncLLMEngine v1-v0 opaque (#11383) 
Signed-off-by: Ricky Xu <xuchen727@hotmail.com>
 2024-12-21 15:14:08 +08:00
Michael Goin d573aeadcc [Bugfix] Don't log OpenAI field aliases as ignored (#11378) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-12-20 19:03:50 +00:00
Yanyi LiuandCyrus Leung 5aef49806d [Feature] Add load generation config from model (#11164) 
Signed-off-by: liuyanyi <wolfsonliu@163.com>
Signed-off-by: Yanyi Liu <wolfsonliu@163.com>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: Cyrus Leung <tlleungac@connect.ust.hk>
 2024-12-19 10:50:38 +00:00
Travis Johnson 17ca964273 [Model] IBM Granite 3.1 (#11307) 
Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
 2024-12-19 11:27:24 +08:00
Joe Runde 2d1b9baa8f [Bugfix] Fix request cancellation without polling (#11190) 2024-12-17 12:26:32 -08:00
kYLeandDarkLight1337 66d4b16724 [Frontend] Add OpenAI API support for input_audio (#11027) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-12-16 22:09:58 -08:00
Michael Goin 0064f697d3 [CI] Add test case with JSON schema using references + use xgrammar by default with OpenAI parse (#10935) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-12-17 11:39:58 +08:00
Isotr0pyandCyrus Leung d927dbcd88 [Model] Refactor Ultravox to use merged input processor (#11198) 
Signed-off-by: Isotr0py <2037008807@qq.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2024-12-16 10:09:53 +00:00
yansh97 17138af7c4 [Bugfix] Fix the default value for temperature in ChatCompletionRequest (#11219) 2024-12-16 00:15:40 -08:00
Brad Hilton 9c3dadd1c9 [Frontend] Add logits_processors as an extra completion argument (#11150) 
Signed-off-by: Brad Hilton <brad.hilton.nw@gmail.com>
 2024-12-14 16:46:42 +00:00
Russell Bryant 4863e5fba5 [Core] V1: Use multiprocessing by default (#11074) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2024-12-13 16:27:32 -08:00
Cyrus Leung 0920ab9131 [Doc] Reorganize online pooling APIs (#11172) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-12-14 00:22:22 +08:00
zhangjf 5b0ed8391d [Bugfix] using len(tokenizer) instead of tokenizer.vocab_size in AllowedTokenIdsLogitsProcessor (#11156) 2024-12-13 15:56:19 +00:00
Cyrus Leung eeec9e3390 [Frontend] Separate pooling APIs in offline inference (#11129) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-12-13 10:40:07 +00:00
Jiaxin ShanandJee Jee Li 85362f028c [Misc][LoRA] Ensure Lora Adapter requests return adapter name (#11094) 
Signed-off-by: Jiaxin Shan <seedjeffwan@gmail.com>
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
Co-authored-by: Jee Jee Li <pandaleefree@gmail.com>
 2024-12-12 09:25:16 +00:00
Russell Bryant ccede2b264 [Core] cleanup zmq ipc sockets on exit (#11115) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2024-12-11 19:12:24 -08:00
Clayton 7439a8b5fc [Bugfix] Multiple fixes to tool streaming with hermes and mistral (#10979) 
Signed-off-by: cedonley <clayton@donley.io>
 2024-12-12 01:10:12 +00:00
Cyrus Leung cad5c0a6ed [Doc] Update docs to refer to pooling models (#11093) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-12-11 13:36:27 +00:00
Cyrus Leung 8f10d5e393 [Misc] Split up pooling tasks (#10820) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-12-11 01:28:00 -08:00
Rafael Vasquez 40766ca1b8 [Bugfix]: Clamp -inf logprob values in prompt_logprobs (#11073) 
Signed-off-by: Rafael Vasquez <rafvasq21@gmail.com>
 2024-12-11 01:27:39 -08:00
Maximilien de Bayser e39400a4b6 Fix streaming for granite tool call when <|tool_call|> is present (#11069) 
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
 2024-12-11 04:51:40 +00:00
Travis Johnson beb16b2c81 [Bugfix] Handle <|tool_call|> token in granite tool parser (#11039) 
Signed-off-by: Travis Johnson <tsjohnso@us.ibm.com>
 2024-12-10 10:27:11 +00:00
Joe Runde 980ad394a8 [Frontend] Use request id from header (#10968) 
Signed-off-by: Joe Runde <Joseph.Runde@ibm.com>
 2024-12-10 13:46:29 +08:00
Cyrus Leung 391d7b2763 [Bugfix] Fix usage of deprecated decorator (#11025) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-12-10 13:45:47 +08:00
youkaichao fd57d2b534 [torch.compile] allow candidate compile sizes (#10984) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2024-12-08 11:05:21 +00:00
Russell Bryant 69d357ba12 [Core] Cleanup startup logging a bit (#10961) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2024-12-07 02:30:23 +00:00
Cyrus Leung 3257d449fa [Misc] Remove deprecated names (#10817) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-12-03 06:52:57 +00:00
Cyrus Leung d2f058e76c [Misc] Rename embedding classes to pooling (#10801) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2024-12-01 14:36:51 +08:00
tomeras91 395b1c7454 [Frontend] don't block event loop in tokenization (preprocess) in OpenAI compatible server (#10635) 
Signed-off-by: Tomer Asida <tomera@ai21.com>
 2024-11-27 13:21:10 -08:00
Michael Goin 9a99273b48 [Bugfix] Fix using -O[0,3] with LLM entrypoint (#10677) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-11-26 10:44:01 -08:00
Ricky Xu 519e8e4182 [v1] EngineArgs for better config handling for v1 (#10382) 
Signed-off-by: rickyx <rickyx@anyscale.com>
 2024-11-25 21:09:43 -08:00
zhou fanandIsotr0py b1d920531f [Model]: Add support for Aria model (#10514) 
Signed-off-by: xffxff <1247714429@qq.com>
Co-authored-by: Isotr0py <2037008807@qq.com>
 2024-11-25 18:10:55 +00:00
Chauncey d04b13a380 [Bug]: Authorization ignored when root_path is set (#10606) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
 2024-11-25 16:21:41 +00:00
Maximilien de BayserandFlavia Beo 214efc2c3c Support Cross encoder models (#10400) 
Signed-off-by: Max de Bayser <maxdebayser@gmail.com>
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
Signed-off-by: Flavia Beo <flavia.beo@ibm.com>
Co-authored-by: Flavia Beo <flavia.beo@ibm.com>
 2024-11-24 18:56:20 -08:00
Varun Vinayak Shenoy 7d8ffb344f [Bugfix] Internal Server Error when tool_choice is incorrect. (#10567) 
Signed-off-by: Varun Shenoy <varun.vinayak.shenoy@oracle.com>
 2024-11-22 21:13:29 -08:00
Noam Gat 11fcf0e066 Remove token-adding chat embedding params (#10551) 
Signed-off-by: Noam Gat <noamgat@gmail.com>
 2024-11-21 23:59:47 -08:00
Woosuk Kwon 446c7806b2 [Minor] Fix line-too-long (#10563) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2024-11-21 19:40:40 -08:00
youkaichao 33e0a2540a [9/N] torch.compile LLM usage (#10552) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2024-11-21 19:13:31 -08:00
Chauncey da7e702c6f [Bug]: When apply continue_final_message for OpenAI server, the "echo":false is ignored (#10180) 
Signed-off-by: chaunceyjiang <chaunceyjiang@gmail.com>
 2024-11-21 16:24:32 +00:00
Zhong Qishuai f0e0238016 [Doc] fix a small typo in docstring of llama_tool_parser (#10513) 2024-11-21 09:05:23 +00:00