Commit Graph

Select branches

Hide Pull Requests

main

patches

sm86

Mono Color

• 4df417d059 fix: fix some args desc (#2487) zspo 2024-01-19 01:41:44 +08:00
• 5d80a9178b Minor fix in prefill cache example (#2494) Jason Zhu 2024-01-18 09:40:34 -08:00
• 8a25d3a71a fix stablelm.py tensor-parallel-size bug (#2482) YingchaoX 2024-01-19 01:39:46 +08:00
• d10f8e1d43 [Experimental] Prefix Caching Support (#1669) 3 people 2024-01-17 16:32:10 -08:00
• 14cc317ba4 OpenAI Server refactoring (#2360) FlorianJoncour 2024-01-17 05:33:14 +00:00
• e1957c6ebd Add StableLM3B model (#2372) Hyunsung Lee 2024-01-17 13:32:40 +09:00
• 8cd5a992bf ci: retry on build failure as well (#2457) Simon Mo 2024-01-16 12:51:04 -08:00
• 947f0b23cc CI: make sure benchmark script exit on error (#2449) Simon Mo 2024-01-16 09:50:13 -08:00
• f780504d12 fix weigit loading for GQA with TP (#2379) Chenhui Zhang 2024-01-16 07:43:59 +08:00
• bfc072addf Allow buildkite to retry build on agent lost (#2446) Simon Mo 2024-01-15 15:43:15 -08:00
• 2a18da257c Announce the second vLLM meetup (#2444) Woosuk Kwon 2024-01-15 14:11:59 -08:00
• 6e01e8c1c8 [CI] Add Buildkite (#2355) Simon Mo 2024-01-14 12:37:58 -08:00
• 9f659bf07f [Minor] Optimize cuda graph memory usage (#2437) Roy 2024-01-15 01:40:51 +08:00
• 35c4bc20d9 [Minor] Fix err msg (#2431) Woosuk Kwon 2024-01-12 14:02:52 -08:00
• 218dc2ccda Aligning top_p and top_k Sampling (#1885) 陈序 2024-01-13 05:51:03 +08:00
• 827cbcd37c Update quickstart.rst (#2369) Simon 2024-01-12 14:56:18 -06:00
• cb7a1c1cbf Suggest using dtype=half when OOM. Ben 2024-01-13 04:33:29 +08:00
• 7878958c0d Address Phi modeling update 2 (#2428) Gary Hui 2024-01-13 04:16:49 +08:00
• ce036244c9 Allow setting fastapi root_path argument (#2341) Chirag Jain 2024-01-13 00:29:59 +05:30
• 48cf1e413c fix: deque mutated during iteration in abort_seq_group (#2371) 陈序 2024-01-13 00:44:18 +08:00
• 97460585d9 Add gradio chatbot for openai webserver (#2307) arkohut 2024-01-12 11:45:56 +08:00
• f745847ef7 [Minor] Fix the format in quick start guide related to Model Scope (#2425) Zhuohan Li 2024-01-11 19:44:01 -08:00
• 6549aef245 [DOC] Add additional comments for LLMEngine and AsyncLLMEngine (#1011) Jiaxiang 2024-01-12 11:26:49 +08:00
• 50376faa7b Rename phi_1_5 -> phi (#2385) Woosuk Kwon 2024-01-11 16:23:43 -08:00
• 4b61c6b669 get_ip(): Fix ipv4 ipv6 dualstack (#2408) Yunfeng Bai 2024-01-10 11:39:58 -08:00
• 79d64c4954 [Speculative decoding 1/9] Optimized rejection sampler (#2336) Cade Daniel 2024-01-09 15:38:41 -08:00
• 74cd5abdd1 Add baichuan chat template jinjia file (#2390) KKY 2024-01-09 11:13:02 -06:00
• 28c3f12104 [Minor] Remove unused code in attention (#2384) Woosuk Kwon 2024-01-08 13:13:08 -08:00
• c884819135 Fix eager mode performance (#2377) Woosuk Kwon 2024-01-08 10:11:06 -08:00
• 05921a9a7a Changed scheduler to use deques instead of lists (#2290) Nadav ShmayovitsandWoosuk Kwon 2024-01-07 19:48:07 +02:00
• d0215a58e7 Ensure metrics are logged regardless of requests (#2347) Iskren Ivov Chernev 2024-01-05 15:24:42 +02:00
• 937e7b7d7c Build docker image with shared objects from "build" step (#2237) Alexandre Payot 2024-01-04 18:35:18 +01:00
• aee8ef661a Miner fix of type hint (#2340) ljss 2024-01-04 13:27:56 +08:00
• 2e0b6e7757 Bump up to v0.2.7 (#2337) Woosuk Kwon 2024-01-03 17:35:56 -08:00
• 941767127c Revert the changes in test_cache (#2335) Woosuk Kwon 2024-01-03 17:32:05 -08:00
• 74d8d77626 Remove unused const TIMEOUT_TO_PREVENT_DEADLOCK (#2321) Ronen Schaffer 2024-01-04 01:49:07 +02:00
• fd4ea8ef5c Use NCCL instead of ray for control-plane communication to remove serialization overhead (#2221) Zhuohan Li 2024-01-04 03:30:22 +08:00
• 1066cbd152 Remove deprecated parameter: concurrency_count (#2315) Ronen Schaffer 2024-01-03 19:56:21 +02:00
• 6ef00b03a2 Enable CUDA graph for GPTQ & SqueezeLLM (#2318) Woosuk Kwon 2024-01-03 09:52:29 -08:00
• 9140561059 [Minor] Fix typo and remove unused code (#2305) Roy 2024-01-03 11:23:15 +08:00
• 77af974b40 [FIX] Support non-zero CUDA devices in custom kernels (#1959) Jee Li 2024-01-03 11:09:59 +08:00
• 4934d49274 Support GPT-NeoX Models without attention biases (#2301) Jong-hun Shin 2023-12-31 01:42:04 +09:00
• 358c328d69 [BUGFIX] Fix communication test (#2285) Zhuohan Li 2023-12-28 06:18:11 +08:00
• 4aaafdd289 [BUGFIX] Fix the path of test prompts (#2273) Zhuohan Li 2023-12-27 02:37:21 +08:00
• 66b108d142 [BUGFIX] Fix API server test (#2270) Zhuohan Li 2023-12-27 02:37:06 +08:00
• e0ff920001 [BUGFIX] Do not return ignored sentences twice in async llm engine (#2258) Zhuohan Li 2023-12-26 13:41:09 +08:00
• face83c7ec [Docs] Add "About" Heading to README.md (#2260) blueceiling 2023-12-25 17:37:07 -07:00
• 1db83e31a2 [Docs] Update installation instructions to include CUDA 11.8 xFormers (#2246) Shivam Thakkar 2023-12-23 12:50:02 +05:30
• a1b9cb2a34 [BugFix] Fix recovery logic for sequence group (#2186) Woosuk Kwon 2023-12-20 21:52:37 -08:00
• 3a4fd5ca59 Disable Ray usage stats collection (#2206) Woosuk Kwon 2023-12-20 21:52:08 -08:00
• c17daa9f89 [Docs] Fix broken links (#2222) Ronen Schaffer 2023-12-20 22:43:42 +02:00
• bd29cf3d3a Remove Sampler copy stream (#2209) Antoni Baum 2023-12-20 00:04:33 -08:00
• 31bff69151 Make _prepare_sample non-blocking and use pinned memory for input buffers (#2207) Hanzhi Zhou 2023-12-19 16:52:46 -08:00
• ba4f826738 [BugFix] Fix weight loading for Mixtral with TP (#2208) Woosuk Kwon 2023-12-19 16:16:11 -08:00
• de60a3fb93 Added DeciLM-7b and DeciLM-7b-instruct (#2062) avideci 2023-12-19 12:29:33 +02:00
• 21d5daa4ac Add warning on CUDA graph memory usage (#2182) Woosuk Kwon 2023-12-18 18:16:17 -08:00
• 290e015c6c Update Help Text for --gpu-memory-utilization Argument (#2183) Suhong Moon 2023-12-18 14:33:24 -05:00
• 1b7c791d60 [ROCm] Fixes for GPTQ on ROCm (#2180) kliuae 2023-12-19 02:41:04 +08:00
• bbe4466fd9 [Minor] Fix typo (#2166) JohnSaxonandJohn-Saxon 2023-12-18 15:28:49 +08:00
• 08133c4d1a Add SSL arguments to API servers (#2109) Harry Mellor 2023-12-18 02:56:23 +00:00
• 76a7983b23 [BugFix] Fix RoPE kernel on long sequences(#2164) Woosuk Kwon 2023-12-17 17:09:10 -08:00
• 8041b7305e [BugFix] Raise error when max_model_len is larger than KV cache (#2163) Woosuk Kwon 2023-12-17 17:08:23 -08:00
• 3ec8c25cd0 [Docs] Update documentation for gpu-memory-utilization option (#2162) Suhong Moon 2023-12-17 13:51:57 -05:00
• 671af2b1c0 Bump up to v0.2.6 (#2157) Woosuk Kwon 2023-12-17 10:34:56 -08:00
• 6f41f0e377 Disable CUDA graph for SqueezeLLM (#2161) Woosuk Kwon 2023-12-17 10:24:25 -08:00
• 2c9b638065 [Minor] Fix a typo in .pt weight support (#2160) Woosuk Kwon 2023-12-17 10:12:44 -08:00
• a7347d9a6d Make sampler less blocking (#1889) Antoni Baum 2023-12-17 07:03:49 -08:00
• f8c688d746 [Minor] Add Phi 2 to supported models (#2159) Woosuk Kwon 2023-12-17 02:54:57 -08:00
• c9fadda543 [Minor] Fix xformers version (#2158) Woosuk Kwon 2023-12-17 02:28:02 -08:00
• 30fb0956df [Minor] Add more detailed explanation on quantization argument (#2145) Woosuk Kwon 2023-12-17 01:56:16 -08:00
• 3a765bd5e1 Temporarily enforce eager mode for GPTQ models (#2154) Woosuk Kwon 2023-12-17 01:51:12 -08:00
• 26c52a5ea6 [Docs] Add CUDA graph support to docs (#2148) Woosuk Kwon 2023-12-17 01:49:20 -08:00
• c3372e87be Remove dependency on CuPy (#2152) Woosuk Kwon 2023-12-17 01:49:07 -08:00
• b0a1d667b0 Pin PyTorch & xformers versions (#2155) Woosuk Kwon 2023-12-17 01:46:54 -08:00
• e1d5402238 Fix all-reduce memory usage (#2151) Woosuk Kwon 2023-12-17 01:44:45 -08:00
• 3d1cfbfc74 [Minor] Delete Llama tokenizer warnings (#2146) Woosuk Kwon 2023-12-16 22:05:18 -08:00
• 37ca558103 Optimize model execution with CUDA graph (#1926) 3 people 2023-12-16 21:12:08 -08:00
• eed74a558f Simplify weight loading logic (#2133) Roy 2023-12-17 04:41:23 +08:00
• 2acd76f346 [ROCm] Temporarily remove GPTQ ROCm support (#2138) Woosuk Kwon 2023-12-15 17:13:58 -08:00
• b81a6a6bb3 [Docs] Add supported quantization methods to docs (#2135) Woosuk Kwon 2023-12-15 13:29:22 -08:00
• 0fbfc4b81b Add GPTQ support (#916) CHU Tianxiang 2023-12-15 19:04:22 +08:00
• c06170cc8e Add a flag to include stop string in output text (#1976) Yunfeng Bai 2023-12-15 00:45:58 -08:00
• 614856da25 Avoid multiple redefinition (#1817) Mingcan Xiang 2023-12-14 12:35:58 -05:00
• 05bdf4eaf3 Fix Dockerfile.rocm (#2101) TJianandmiloice 2023-12-14 16:45:58 +08:00
• 6774bd50b0 Fix typing in AsyncLLMEngine & add toml to requirements-dev (#2100) mezuzza 2023-12-14 03:19:41 -05:00
• 31c1f3255e Bump up to v0.2.5 (#2095) Woosuk Kwon 2023-12-13 23:56:15 -08:00
• 21d93c140d Optimize Mixtral with expert parallelism (#2090) Antoni Baum 2023-12-13 23:55:07 -08:00
• f1c8520146 [BugFix] Fix input positions for long context with sliding window (#2088) Woosuk Kwon 2023-12-13 12:28:13 -08:00
• 096827c284 [Docs] Add notes on ROCm-supported models (#2087) Woosuk Kwon 2023-12-13 09:45:34 -08:00
• 6565d9e33e Update installation instruction for vLLM + CUDA 11.8 (#2086) Woosuk Kwon 2023-12-13 09:25:59 -08:00
• f375ec8440 [ROCm] Upgrade xformers version for ROCm & update doc (#2079) TJianandmiloice 2023-12-13 16:56:05 +08:00
• 518369d78c Implement lazy model loader (#2044) Woosuk Kwon 2023-12-12 22:21:45 -08:00
• 30bad5c492 Fix peak memory profiling (#2031) Woosuk Kwon 2023-12-12 22:01:53 -08:00
• 3fefe271ec Update Dockerfile to build Megablocks (#2042) Simon Mo 2023-12-12 17:34:17 -08:00
• 6428f1d051 Support MPT with GQA (#1938) Megha AgarwalandWoosuk Kwon 2023-12-12 10:16:05 -08:00
• 7e1b21daac Remove einops from requirements (#2049) Woosuk Kwon 2023-12-12 09:34:09 -08:00
• cb3f30c600 Upgrade transformers version to 4.36.0 (#2046) Woosuk Kwon 2023-12-11 18:39:14 -08:00
• f3e024bece [CI/CD] Upgrade PyTorch version to v2.1.1 (#2045) Woosuk Kwon 2023-12-11 17:48:11 -08:00
• 31d2ab4aff Remove python 3.10 requirement (#2040) Woosuk Kwon 2023-12-11 12:26:42 -08:00
• eb17212858 Update Dockerfile to support Mixtral (#2027) Simon Mo 2023-12-11 11:59:08 -08:00