Commit Graph

Select branches

Hide Pull Requests

main

patches

sm86

Mono Color

• 0eda2e0953 Add .readthedocs.yaml (#136) Woosuk Kwon 2023-06-02 22:27:44 -07:00
• 211318d44a Add throughput benchmarking script (#133) Woosuk Kwon 2023-05-28 03:20:05 -07:00
• 337871c6fd Enable LLaMA fast tokenizer (#132) Woosuk Kwon 2023-05-28 02:51:42 -07:00
• 56b7f0efa4 Add a doc for installation (#128) Woosuk Kwon 2023-05-27 01:13:06 -07:00
• d721168449 Improve setup script & Add a guard for bfloat16 kernels (#130) Woosuk Kwon 2023-05-27 00:59:32 -07:00
• 4a151dd453 Add activation registry (#126) Woosuk Kwon 2023-05-25 00:09:07 -07:00
• 057daef778 OpenAI Compatible Frontend (#116) Zhuohan Li 2023-05-23 21:39:50 -07:00
• e86717833d Incrementally decode output tokens (#121) Woosuk Kwon 2023-05-23 20:46:32 -07:00
• aedba6d5ec Print warnings/errors for large swap space (#123) Woosuk Kwon 2023-05-23 18:22:26 -07:00
• a283ec2eec Add contributing guideline and mypy config (#122) Woosuk Kwon 2023-05-23 17:58:51 -07:00
• 3f942acfe1 Fix latency benchmark script (#118) Woosuk Kwon 2023-05-22 17:03:40 -07:00
• 19d2899439 Add initial sphinx docs (#120) Woosuk Kwon 2023-05-22 17:02:44 -07:00
• 655a5e48df Introduce LLM class for offline inference (#115) Woosuk Kwon 2023-05-21 17:04:18 -07:00
• f746ced08d Implement stop strings and best_of (#114) Woosuk Kwon 2023-05-21 11:18:00 -07:00
• c3442c1f6f Refactor system architecture (#109) Woosuk Kwon 2023-05-20 13:06:59 -07:00
• 7297fa6f7c Remove unused parts in Megatron-LM code and add copyright notice (#110) Zhuohan Li 2023-05-20 09:11:34 -06:00
• b7955ef17b Fix timeout error in the FastAPI frontend (#34) Zhuohan Li 2023-05-19 14:00:46 -06:00
• f756799b84 Use runtime profiling to replace manual memory analyzers (#81) Zhuohan Li 2023-05-19 11:35:44 -06:00
• 825d8892b5 Use pytest format for unit tests (#107) Woosuk Kwon 2023-05-17 17:11:23 -07:00
• b322fd1607 Add docstrings to some modules and classes (#100) Woosuk Kwon 2023-05-14 22:32:38 -07:00
• 667ba3995c Add copyright headers to source files adapted from FT (#104) Woosuk Kwon 2023-05-14 22:19:19 -07:00
• 707ec647bb Add copyright headers for HF models (#103) Woosuk Kwon 2023-05-14 21:54:32 -07:00
• 89988ec8c2 Add Apache-2.0 license (#102) Woosuk Kwon 2023-05-14 18:05:19 -07:00
• 6208d622ca Minor code cleaning for SamplingParams (#99) Woosuk Kwon 2023-05-12 18:07:09 -07:00
• 42f1042e1c Enhance SamplingParams (#96) Woosuk Kwon 2023-05-11 15:45:30 -07:00
• 55f8b0a5de Implement presence and frequency penalties (#95) Woosuk Kwon 2023-05-10 23:39:12 -07:00
• 9f88db35da Support top-k sampling (#94) Woosuk Kwon 2023-05-10 12:51:36 -07:00
• ae356774ab Avoid sorting waiting queue & Minor code cleaning (#93) Woosuk Kwon 2023-05-10 01:57:07 -07:00
• e331957784 Log system stats (#90) Woosuk Kwon 2023-05-10 01:06:53 -07:00
• 8d66a7b6d7 Rename variables and methods (#91) Woosuk Kwon 2023-05-10 00:58:31 -07:00
• ce26e57fd3 Update sample prompts in simple_server.py (#89) Woosuk Kwon 2023-05-09 16:47:39 -07:00
• 85eb631839 Use slow tokenizer for LLaMA (#84) Woosuk Kwon 2023-05-09 16:03:44 -07:00
• add055e151 Enhance model loader (#83) Woosuk Kwon 2023-05-09 15:46:42 -07:00
• 7c041ab578 Refactor system architecture (#82) Woosuk Kwon 2023-05-09 15:30:12 -07:00
• 8917782af6 Add a system logger (#85) Woosuk Kwon 2023-05-08 23:03:35 -07:00
• 7addca5935 Specify python package dependencies in requirements.txt (#78) Woosuk Kwon 2023-05-07 16:30:43 -07:00
• c84e924287 [Minor] Fix a dtype bug (#79) Woosuk Kwon 2023-05-06 02:12:12 -07:00
• c9d5b6d4a8 Replace FlashAttention with xformers (#70) Woosuk Kwon 2023-05-05 02:01:08 -07:00
• 189ae23133 Use dtype from model config & Add Dolly V2 (#63) Woosuk Kwon 2023-05-04 03:05:37 -07:00
• e548c1488a Add support for GPT-2 (#60) Woosuk Kwon 2023-05-04 02:59:56 -07:00
• 130d5fd8c7 Fix a bug in attention kernel (#68) Woosuk Kwon 2023-05-04 02:56:09 -07:00
• e070829ae8 Support bfloat16 data type (#54) Woosuk Kwon 2023-05-03 14:09:44 -07:00
• 436e523bf1 Refactor attention kernels (#53) Woosuk Kwon 2023-05-03 13:40:13 -07:00
• 27f1410d06 New weight loader without np copy (#52) Zhuohan Li 2023-05-03 15:32:04 +08:00
• 4858f3bb45 Add an option to launch cacheflow without ray (#51) Zhuohan Li 2023-04-30 15:42:17 +08:00
• a96d63c21d Add support for GPT-NeoX (Pythia) (#50) Woosuk Kwon 2023-04-28 00:32:10 -07:00
• aa50b17ca7 Change plotting script Woosuk Kwon 2023-04-17 04:49:14 +00:00
• 0f4b32199e Support various block sizes & Change default block size to 16 (#38) Woosuk Kwon 2023-04-15 09:03:24 -07:00
• 84eee24e20 Collect system stats in scheduler & Add scripts for experiments (#30) Woosuk Kwon 2023-04-12 15:03:49 -07:00
• e3cec88aa5 Memcpy kernel for flash attention (#29) Siyuan (Ryans) Zhuang 2023-04-10 18:22:49 -07:00
• b9926f7f66 Support block size 32 (#35) Woosuk Kwon 2023-04-09 23:07:18 -07:00
• ee88a7e5f3 Add an option to use dummy model weights (#33) Woosuk Kwon 2023-04-08 23:36:12 -07:00
• c267b1a02c Add query stride to multi_query_cached_kv_attention & Add kernel benchmark script (#27) Woosuk Kwon 2023-04-08 13:36:09 -07:00
• 0f40557af6 Implement block copy kernel to optimize beam search (#32) Woosuk Kwon 2023-04-07 17:45:07 -07:00
• a490aafa36 Fix potential bugs in FastAPI frontend and add comments (#28) Zhuohan Li 2023-04-06 13:44:24 +08:00
• 12659a0bd7 Add CUDA graph-based all reduce launcher (#26) Woosuk Kwon 2023-04-05 11:16:57 -07:00
• 21b3671bbc Basic attention kernel that supports cached KV + (multi-)prompts (#24) Siyuan (Ryans) Zhuang 2023-04-04 20:34:46 -07:00
• 897cb2ae28 Optimize data movement (#20) Woosuk Kwon 2023-04-02 00:30:17 -07:00
• 1f01a18d39 Merge QKV into one linear layer (#15) Zhuohan Li 2023-04-02 15:23:29 +08:00
• 2c5cd0defe Add ninja to dependency (#21) Woosuk Kwon 2023-04-01 19:00:20 -07:00
• a90c97d727 Use FP32 for log probabilities (#19) Woosuk Kwon 2023-03-31 23:33:43 -07:00
• e3f00d191e Modify README to include info on loading LLaMA (#18) Zhuohan Li 2023-04-01 01:07:57 +08:00
• 09e9245478 Add custom kernel for RMS normalization (#16) Woosuk Kwon 2023-03-31 09:51:22 -07:00
• c45f3c3ab6 Optimize tensor parallel execution speed (#17) Zhuohan Li 2023-04-01 00:51:08 +08:00
• 7a7929abe8 Implement preemption via recomputation & Refactor scheduling logic (#12) Woosuk Kwon 2023-03-30 14:51:46 -07:00
• 88c0268a18 Implement custom kernel for LLaMA rotary embedding (#14) Woosuk Kwon 2023-03-30 11:04:21 -07:00
• 80a2f812f1 Implement LLaMA (#9) Woosuk KwonandZhuohan Li 2023-03-29 21:25:32 -07:00
• a1b3de86cd Refactor the test code for attention kernels (#13) Woosuk Kwon 2023-03-29 18:59:27 -07:00
• 64e0e38314 Add cache watermark to avoid frequent cache eviction (#11) Woosuk Kwon 2023-03-29 16:38:48 -07:00
• 721fa3df15 FastAPI-based working frontend (#10) Zhuohan Li 2023-03-29 14:48:56 +08:00
• d359cda5fa Minor Woosuk Kwon 2023-03-26 08:00:39 +00:00
• 2f49f15585 Support tensor parallel (#2) Zhuohan Li 2023-03-22 04:45:42 +08:00
• cfae35b861 Add miscellaneous updates (#8) Woosuk Kwon 2023-03-13 13:48:38 -07:00
• e9d3f2ff77 Add memory analyzer & utomatically configure KV cache size (#6) Woosuk Kwon 2023-03-11 23:23:14 -08:00
• 1a7eb7da61 Support beam search & parallel generation (#7) Woosuk Kwon 2023-03-10 09:58:21 -08:00
• 04e5acc08e Fix a bug in 1D input shape (#5) Woosuk Kwon 2023-03-06 10:05:27 -08:00
• 3e9f991d6a Use FlashAttention for multi_query_kv_attention (#4) Woosuk Kwon 2023-03-01 21:13:08 -08:00
• 0deacbce6e Implement single_query_cached_kv_attention kernel (#3) Woosuk Kwon 2023-03-01 15:02:19 -08:00
• cbf8779afa Fix a bug in tying OPT embeddings (#1) Woosuk Kwon 2023-02-24 16:29:36 -08:00
• c84c708a1d Add README Woosuk Kwon 2023-02-24 12:04:49 +00:00
• fa16389a2e Clean up the server script Woosuk Kwon 2023-02-24 11:56:21 +00:00
• 6aef2278f4 [Minor] Fix printing format Woosuk Kwon 2023-02-24 11:56:06 +00:00
• 1132fae0ca Add Frontend Woosuk Kwon 2023-02-24 11:46:43 +00:00
• 46ce1356f7 Add max_num_steps to SamplingParams Woosuk Kwon 2023-02-24 11:44:40 +00:00
• b39f149a08 Add is_finished Woosuk Kwon 2023-02-24 11:44:21 +00:00
• ef6098ec51 Merge pre_step and step Woosuk Kwon 2023-02-24 10:36:08 +00:00
• 53f70e7334 Reduce the number of states in scheduler Woosuk Kwon 2023-02-24 10:22:39 +00:00
• 762fd1c3fa Refactor and annotate types for attention Woosuk Kwon 2023-02-24 08:58:46 +00:00
• 7f22f90e8c Remove xformers Woosuk Kwon 2023-02-24 08:36:16 +00:00
• afdbe5d373 [WIP] Add server script Woosuk Kwon 2023-02-24 01:33:37 +00:00
• 932844f1cd Fix attention Woosuk Kwon 2023-02-23 23:02:25 +00:00
• ba84b8728a Fix attention Woosuk Kwon 2023-02-23 22:29:46 +00:00
• 87e0bcd426 Fix attention Woosuk Kwon 2023-02-23 21:32:02 +00:00
• 1ce1333573 Set default dtype to half Woosuk Kwon 2023-02-23 21:31:39 +00:00
• de0fabbc5c Fix sampler Woosuk Kwon 2023-02-23 20:30:12 +00:00
• fdd0f2f472 Minor Woosuk Kwon 2023-02-23 20:23:47 +00:00
• 7f985166f7 Consider pempty tensor Woosuk Kwon 2023-02-23 20:20:33 +00:00
• 86f9eb6d39 Fix typo Woosuk Kwon 2023-02-23 20:19:24 +00:00
• 1f6c7ef437 Add controller Woosuk Kwon 2023-02-23 09:32:19 +00:00
• d4bc1a4d24 Add unoptimized OPT Attention Woosuk Kwon 2023-02-23 09:31:55 +00:00