Commit Graph

Select branches

Hide Pull Requests

main

patches

sm86

Mono Color

• b5853f9963 [ROCm][AMD][Bugfix] adding a missing triton autotune config (#4845) Hongxia Yang 2024-05-16 13:46:52 -04:00
• f09edd8a25 Add JSON output support for benchmark_latency and benchmark_throughput (#4848) Simon Mo 2024-05-16 10:02:56 -07:00
• 6979ade384 Add GPTQ Marlin 2:4 sparse structured support (#4790) Alexander MatveevandRobert Shaw 2024-05-16 12:56:15 -04:00
• 9216b9cc38 [Bugfix] Bypass authorization API token for preflight requests (#4862) Pierre Dulac 2024-05-16 18:42:21 +02:00
• 5e0391c040 [Frontend] Separate OpenAI Batch Runner usage from API Server (#4851) Alex Wu 2024-05-16 11:42:41 -04:00
• dbc0754ddf [docs] Fix typo in examples filename openi -> openai (#4864) Alex Wu 2024-05-16 11:42:17 -04:00
• 99caa49106 [Kernel] add bfloat16 support for gptq marlin kernel (#4788) Jinzhen Lin 2024-05-16 21:55:29 +08:00
• 5c342570d7 Add marlin unit tests and marlin benchmark script (#4815) alexm-nm 2024-05-16 09:36:49 -04:00
• 973617ae02 [Speculative decoding][Re-take] Enable TP>1 speculative decoding (#4840) 3 people 2024-05-16 00:53:51 -07:00
• 30e754390c [Core] Implement sharded state loader (#4690) Aurick QiaoandWoosuk Kwon 2024-05-16 01:11:54 -04:00
• 52f8107cf2 [Frontend] Support OpenAI batch file format (#4794) Alex WuandRobert Shaw 2024-05-15 19:13:36 -04:00
• fc0d9dfc3a [Frontend] Re-enable custom roles in Chat Completions API (#4758) Cyrus Leung 2024-05-16 05:58:46 +08:00
• 361c461a12 [Doc] Highlight the fourth meetup in the README (#4842) Zhuohan Li 2024-05-15 11:38:49 -07:00
• a5675d348b [Bugfix] Properly set distributed_executor_backend in ParallelConfig (#4816) zifeitong 2024-05-15 07:22:09 -07:00
• e9cdd2b1e2 [CI/Build] Further decouple HuggingFace implementation from ours during tests (#4166) Cyrus Leung 2024-05-15 14:38:40 +08:00
• 65bf2ac165 [Core][2/N] Model runner refactoring part 2. Combine prepare prefill / decode to a single API (#4681) SangBin Cho 2024-05-15 14:00:10 +09:00
• 8a7cc254a0 Revert "[Kernel] Use flash-attn for decoding (#3648)" (#4820) SangBin Cho 2024-05-15 11:52:45 +09:00
• 29bc01bf3b Add 4th meetup announcement to readme (#4817) Simon Mo 2024-05-14 15:33:06 -07:00
• 676a99982f [Core] Add MultiprocessingGPUExecutor (#4539) Nick HillandSAHIL SUNEJA 2024-05-14 10:38:59 -07:00
• dc72402b57 [Bugfix][Doc] Fix CI failure in docs (#4804) Cyrus Leung 2024-05-15 00:57:08 +08:00
• ccb63a8245 [Core][Hash][Automatic Prefix caching] Accelerating the hashing function by avoiding deep copies (#4696) Kuntai Du 2024-05-14 05:34:33 -07:00
• c579b750a0 [Doc] Add meetups to the doc (#4798) Zhuohan Li 2024-05-13 18:48:00 -07:00
• 4bfa7e7f75 [Doc] Add API reference for offline inference (#4710) Cyrus Leung 2024-05-14 08:47:42 +08:00
• ac1fbf7fd2 [Doc] Shorten README by removing supported model list (#4796) Zhuohan Li 2024-05-13 16:23:54 -07:00
• 33d3914b1e [Bugfix] Fix dynamic FP8 quantization for Mixtral (#4793) Philipp Moritz 2024-05-13 16:00:27 -07:00
• 1356df53bd [Kernel] Use flash-attn for decoding (#3648) 3 people 2024-05-13 15:50:33 -07:00
• ce532ff45c [Speculative decoding] Improve n-gram efficiency (#4724) Cody Yu 2024-05-13 15:00:13 -07:00
• 8bc68e198c [Frontend] [Core] perf: Automatically detect vLLM-tensorized model, update tensorizer to version 2.9.0 (#4208) Sanger Steel 2024-05-13 17:57:07 -04:00
• 0fca3cdcf2 [Misc] Enhance attention selector (#4751) Woosuk Kwon 2024-05-13 10:47:25 -07:00
• e7c46b9527 [Scheduler] Warning upon preemption and Swapping (#4647) SangBin ChoandRobert Shaw 2024-05-13 23:50:44 +09:00
• 350f9e107f [CI/Build] Move test_utils.py to tests/utils.py (#4425) Cyrus Leung 2024-05-13 22:50:09 +08:00
• 702bee461f [Core][Distributed] refactor custom allreduce to support multiple tp groups (#4754) youkaichao 2024-05-12 17:47:59 -07:00
• a7be4d0072 [CORE] Improvement in ranks code (#4718) Swapnil Parekh 2024-05-12 20:47:47 -04:00
• a709e87a4f [CI/Build] Tweak Marlin Nondeterminism Issues (#4713) Robert Shaw 2024-05-12 20:46:31 -04:00
• 6eaccb7353 [Model] Add support for IBM Granite Code models (#4636) Yikang Shen 2024-05-12 00:27:24 -04:00
• e254497b66 [Model][Misc] Add e5-mistral-7b-instruct and Embedding API (#3734) Chang Su 2024-05-11 11:30:37 -07:00
• 4e12131089 [Core][Test] fix function name typo in custom allreduce (#4750) youkaichao 2024-05-10 15:14:40 -07:00
• fcc2994be6 [CI] Nits for bad initialization of SeqGroup in testing (#4748) Robert Shaw 2024-05-10 16:01:01 -06:00
• 2e7796f2cf [Speculative decoding] CUDA graph support (#4295) heeju-kim2andCade Daniel 2024-05-11 02:36:25 +09:00
• 706588a77d [Bugfix] Fix CLI arguments in OpenAI server docs (#4729) Allen.Dou 2024-05-10 23:00:56 +08:00
• 6a0f617210 [Core] Fix circular reference which leaked llm instance in local dev env (#4737) SangBin Cho 2024-05-10 23:54:32 +09:00
• dac6a3f6ed [Misc] Apply a couple g++ cleanups (#4719) Steve Grubb 2024-05-10 09:37:05 -04:00
• 64b77dfd7e [Core]fix type annotation for swap_blocks (#4726) Kunshang Ji 2024-05-10 20:52:48 +08:00
• 51d4094fda chunked-prefill-doc-syntax (#4603) Simon Moandsang 2024-05-09 22:13:23 -07:00
• e965d46184 [Misc] Keep only one implementation of the create_dummy_prompt function. (#4716) Allen.Dou 2024-05-10 12:42:38 +08:00
• 208b71bcc1 [Core][Distributed] refactor pynccl (#4591) youkaichao 2024-05-09 19:48:43 -07:00
• c833101740 [Kernel] Refactor FP8 kv-cache with NVIDIA float8_e4m3 support (#4535) Cody Yu 2024-05-09 17:04:17 -07:00
• 379da6dcb5 [Kernel] [FP8] Improve FP8 linear layer performance (#4691) Philipp Moritz 2024-05-09 16:38:07 -07:00
• ebce310b74 [Model] Snowflake arctic model implementation (#4652) 6 people 2024-05-09 15:37:14 -07:00
• be0c5180ac [Bugfix] Add logs for all model dtype casting (#4717) Michael Goin 2024-05-09 14:36:25 -04:00
• cea64430f6 [Bugfix] Update grafana.json (#4711) Robert Shaw 2024-05-09 11:10:13 -06:00
• a3c124570a [Bugfix] Fix CLI arguments in OpenAI server docs (#4709) Cyrus Leung 2024-05-10 00:53:14 +08:00
• ff5abcd746 [ROCm] Add support for Punica kernels on AMD GPUs (#3140) kliuaeandmiloice 2024-05-10 00:19:50 +08:00
• 0ee535b294 [Misc] Set block size at initialization & Fix test_model_runner (#4705) Woosuk Kwon 2024-05-09 09:04:59 -07:00
• 190bc838e1 [Misc] Remove unnecessary ModelRunner imports (#4703) Woosuk Kwon 2024-05-09 00:17:17 -07:00
• f12b20decc [Frontend] Move async logic outside of constructor (#4674) Cyrus Leung 2024-05-09 13:48:33 +08:00
• 16bc0a098f [Frontend] add tok/s speed metric to llm class when using tqdm (#4400) Mahmoud AshrafandMichael Goin 2024-05-09 08:02:31 +03:00
• e288df0632 [Bugfix] Fine-tune gptq_marlin configs to be more similar to marlin (#4626) alexm-nm 2024-05-08 20:14:31 -04:00
• 8b9241be3a [Speculative decoding] [Bugfix] Fix overallocation in ngram + spec logprobs (#4672) Cade Daniel 2024-05-08 16:24:46 -07:00
• f942efb5a3 [Dynamic Spec Decoding] Auto-disable by the running queue size (#4592) Cody YuandCade Daniel 2024-05-08 14:44:00 -07:00
• 89579a201f [Misc] Use vllm-flash-attn instead of flash-attn (#4686) Woosuk Kwon 2024-05-08 13:15:34 -07:00
• 230c4b38c1 [CI/Test] fix swap test for multi gpu (#4689) youkaichao 2024-05-08 13:14:02 -07:00
• 20cfcdec99 [Core][Optimization] change python dict to pytorch tensor for blocks to swap (#4659) youkaichao 2024-05-08 12:07:05 -07:00
• ad932a221d [Core] Faster startup for LoRA enabled models (#4634) Antoni Baum 2024-05-08 10:33:18 -07:00
• 5510cf0e8a [Misc] Add get_name method to attention backends (#4685) Woosuk Kwon 2024-05-08 09:59:31 -07:00
• 0f9a6e3d22 [Bugfix][Kernel] allow non-power-of-2 for prefix prefill with alibi (#4573) DefTruth 2024-05-09 00:19:58 +08:00
• f6a593093a [CI] Make mistral tests pass (#4596) SangBin Cho 2024-05-09 00:44:35 +09:00
• d7740ea4dc [Core] Optimize sampler get_logprobs (#4594) SangBin Cho 2024-05-09 00:42:28 +09:00
• cc466a3290 [Core][Distributed] support cpu&device in broadcast tensor dict (#4660) youkaichao 2024-05-07 19:34:47 -07:00
• 8344f7742b [Bug fix][Core] fixup ngram not setup correctly (#4551) 4 people 2024-05-08 02:40:18 +08:00
• 469f85c782 [Core][Optimization] change copy-on-write from dict[int, list] to list (#4648) youkaichao 2024-05-07 11:06:32 -07:00
• 10760da800 [Bugfix] Fixed error in slice_lora_b for MergedQKVParallelLinearWithLora (#4609) Austin Veselka 2024-05-07 12:59:07 -05:00
• 478aed5827 [Build/CI] Fixing 'docker run' to re-enable AMD CI tests. (#4642) Alexei-V-Ivanov-AMD 2024-05-07 11:23:17 -05:00
• 63575bc2e1 [Core][Optimization] change python dict to pytorch tensor (#4607) youkaichao 2024-05-06 21:30:27 -07:00
• a98187cf72 [Kernel] Make static FP8 scaling more robust (#4570) Philipp Moritz 2024-05-06 17:39:28 -07:00
• bd99d22629 Update lm-format-enforcer to 0.10.1 (#4631) Noam Gat 2024-05-07 02:51:59 +03:00
• 19cb4716ee [CI] Add retry for agent lost (#4633) Cade Daniel 2024-05-06 16:18:57 -07:00
• e186d37cb1 [CI] use ccache actions properly in release workflow (#4629) Simon Mo 2024-05-06 15:23:36 -07:00
• 323f27b904 [Bugfix] Fix asyncio.Task not being subscriptable (#4623) Cyrus Leung 2024-05-07 00:31:05 +08:00
• 0650e5935b Disable cuda version check in vllm-openai image (#4530) zhaoyang-star 2024-05-06 07:58:55 +08:00
• c7f2cf2b7f [CI] Reduce wheel size by not shipping debug symbols (#4602) Simon Mo 2024-05-04 21:28:58 -07:00
• 8d8357c8ed bump version to v0.4.2 (#4600) Simon Mo 2024-05-04 17:09:49 -07:00
• 4302987069 [Bugfix] Fix inappropriate content of model_name tag in Prometheus metrics (#3937) DearPlanet 2024-05-05 06:39:34 +08:00
• 021b1a2ab7 [CI] check size of the wheels (#4319) Simon Mo 2024-05-04 13:44:36 -07:00
• 2a052011ca [Kernel] Support MoE Fp8 Checkpoints for Mixtral (Static Weights with Dynamic/Static Activations) (#4527) Michael Goin 2024-05-04 14:45:16 -04:00
• 36fb68f947 [Doc] Chunked Prefill Documentation (#4580) SangBin Cho 2024-05-04 16:18:00 +09:00
• bc8ad68455 [Misc][Refactor] Introduce ExecuteModelData (#4540) Cody Yu 2024-05-03 17:47:07 -07:00
• 344bf7cd2d [Misc] add installation time env vars (#4574) youkaichao 2024-05-03 15:55:56 -07:00
• ab50275111 [Speculative decoding] Support target-model logprobs (#4378) Cade Daniel 2024-05-03 15:52:01 -07:00
• 43c413ec57 [Kernel] Use flashinfer for decoding (#4353) Lily LiuandLiuXiaoxuanPKU 2024-05-03 15:51:27 -07:00
• f8e7adda21 Fix/async chat serving (#2727) Sebastian Schoennenbeck 2024-05-03 20:04:14 +02:00
• 7e65477e5e [Bugfix] Allow "None" or "" to be passed to CLI for string args that default to None (#4586) Michael Goin 2024-05-03 13:32:21 -04:00
• 3521ba4f25 [Core][Model runner refactoring 1/N] Refactor attn metadata term (#4518) SangBin Cho 2024-05-04 02:20:12 +09:00
• 2d7bce9cd5 [Doc] add env vars to the doc (#4572) youkaichao 2024-05-02 22:13:49 -07:00
• ce3f1eedf8 [Misc] remove chunk detected debug logs (#4571) DefTruth 2024-05-03 12:48:08 +08:00
• 808632d3b4 [BugFix] Prevent the task of _force_log from being garbage collected (#4567) Yang, Bo 2024-05-02 18:35:18 -07:00
• 344a5d0c33 [Core][Distributed] enable allreduce for multiple tp groups (#4566) youkaichao 2024-05-02 17:32:33 -07:00
• 0f8a91401c [Core] Ignore infeasible swap requests. (#4557) SangBin Cho 2024-05-03 06:31:20 +09:00
• 9b5c9f9484 [CI/Build] AMD CI pipeline with extended set of tests. (#4267) Alexei-V-Ivanov-AMDandsimon-mo 2024-05-02 14:29:07 -05:00
• 32881f3f31 [kernel] fix sliding window in prefix prefill Triton kernel (#4405) Michał MoskalandSangBin Cho 2024-05-02 11:23:37 -07:00