Commit Graph

Select branches

Hide Pull Requests

main

patches

sm86

Mono Color

• d88c458f44 [Doc][AMD][ROCm]Added tips to refer to mi300x tuning guide for mi300x users (#6754) Hongxia Yang 2024-07-24 17:32:57 -04:00
• 421e218b37 [Bugfix] Bump transformers to 4.43.2 (#6752) Michael Goin 2024-07-24 16:22:16 -04:00
• 5448f67635 [Core] Tweaks to model runner/input builder developer APIs (#6712) Antoni Baum 2024-07-24 12:17:12 -07:00
• 0e63494cf3 Add fp8 support to reshape_and_cache_flash (#6667) Antoni Baum 2024-07-24 11:36:52 -07:00
• ee812580f7 [Frontend] split run_server into build_server and run_server (#6740) Daniele 2024-07-24 19:36:04 +02:00
• 40468b13fa [Bugfix] Miscalculated latency lead to time_to_first_token_seconds inaccurate. (#6686) Allen.Dou 2024-07-24 23:58:42 +08:00
• 2cf0df3381 [Bugfix] Fix speculative decode seeded test (#6743) Nick Hill 2024-07-24 08:58:31 -07:00
• 545146349c Adding f-string to validation error which is missing (#6748) LF Marques 2024-07-24 16:55:53 +01:00
• f4f8a9d892 [Bugfix]fix modelscope compatible issue (#6730) liuyhwangyh 2024-07-24 20:04:46 +08:00
• b570811706 [Build/CI] Update run-amd-test.sh. Enable Docker Hub login. (#6711) Alexei-V-Ivanov-AMD 2024-07-24 07:01:14 -05:00
• ccc4a73257 [Docs][ROCm] Detailed instructions to build from source (#6680) Woosuk Kwon 2024-07-24 01:07:23 -07:00
• 0a740a11ba [Bugfix] Fix token padding for chameleon (#6724) Roger Wang 2024-07-24 01:05:09 -07:00
• c882a7f5b3 [SpecDecoding] Update MLPSpeculator CI tests to use smaller model (#6714) Nick Hill 2024-07-24 00:34:22 -07:00
• 5e8ca973eb [Bugfix] fix flashinfer cudagraph capture for PP (#6708) William Lin 2024-07-23 18:49:44 -07:00
• 87525fab92 [bitsandbytes]: support read bnb pre-quantized model (#5753) dongmao zhangandMichael Goin 2024-07-23 16:45:09 -07:00
• 2f808e69ab [Bugfix] StatLoggers: cache spec decode metrics when they get collected. (#6645) Thomas Parnell 2024-07-24 01:05:05 +02:00
• 01c16ede6b [CI] Add smoke test for non-uniform AutoFP8 quantization (#6702) Michael Goin 2024-07-23 18:45:12 -04:00
• 72fc704803 [build] relax wheel size limit (#6704) youkaichao 2024-07-23 14:03:49 -07:00
• 1bedf210e3 Bump transformers version for Llama 3.1 hotfix and patch Chameleon (#6690) Roger Wang 2024-07-23 13:47:48 -07:00
• 507ef787d8 [Model] Pipeline Parallel Support for DeepSeek v2 (#6519) Travis Johnson 2024-07-23 13:22:09 -06:00
• 58f53034ad [Frontend] Add Usage data in each chunk for chat_serving. #6540 (#6652) Yehoshua Cohen 2024-07-23 21:41:55 +03:00
• 0eb0757bef [Misc] Add ignored layers for fp8 quantization (#6657) Michael Goin 2024-07-23 14:04:04 -04:00
• 38c4b7e863 Bump version to 0.5.3.post1 (#6696) Simon Mo 2024-07-23 10:08:59 -07:00
• a112a84aad [BugFix] Fix RoPE error in Llama 3.1 (#6693) Woosuk Kwon 2024-07-23 09:46:05 -07:00
• 461089a21a [Bugfix] Fix a log error in chunked prefill (#6694) Woosuk Kwon 2024-07-23 09:27:58 -07:00
• 71950af726 [doc][distributed] fix doc argument order (#6691) youkaichao 2024-07-23 08:55:33 -07:00
• cb1362a889 [Docs] Announce llama3.1 support (#6688) Woosuk Kwon 2024-07-23 08:18:15 -07:00
• bb2fc08072 Bump version to v0.5.3 (#6674) Simon Mo 2024-07-23 00:00:08 -07:00
• 3eda4ec780 support ignore patterns in model loader (#6673) Simon Mo 2024-07-22 23:59:42 -07:00
• 22fa2e35cb [VLM][Model] Support image input for Chameleon (#6633) Roger Wang 2024-07-22 23:50:48 -07:00
• c5201240a4 [misc] only tqdm for first rank (#6672) youkaichao 2024-07-22 21:57:27 -07:00
• 97234be0ec [Misc] Manage HTTP connections in one place (#6600) Cyrus Leung 2024-07-23 12:32:02 +08:00
• c051bfe4eb [doc][distributed] doc for setting up multi-node environment (#6529) youkaichao 2024-07-22 21:22:09 -07:00
• 9e0b558a09 [Misc] Support FP8 kv cache scales from compressed-tensors (#6528) Michael Goin 2024-07-23 00:11:50 -04:00
• e519ae097a add tqdm when loading checkpoint shards (#6569) 3 people 2024-07-23 11:48:01 +08:00
• 7c2749a4fd [misc] add start loading models for users information (#6670) youkaichao 2024-07-22 20:08:02 -07:00
• 729171ae58 [Misc] Enable chunked prefill by default for long context models (#6666) Woosuk Kwon 2024-07-22 20:03:13 -07:00
• c5e8330997 [Bugfix] Fix null modules_to_not_convert in FBGEMM Fp8 quantization (#6665) Cheng Li 2024-07-22 19:25:05 -07:00
• e0c15758b8 [Core] Modulize prepare input and attention metadata builder (#6596) Cody Yu 2024-07-22 17:45:24 -07:00
• bdf5fd1386 [Misc] Remove deprecation warning for beam search (#6659) Woosuk Kwon 2024-07-22 17:21:58 -07:00
• 5a96ee52a3 [ci][build] add back vim in docker (#6661) youkaichao 2024-07-22 16:26:29 -07:00
• 42c7f66a38 [Core] Support dynamically loading Lora adapter from HuggingFace (#6234) Jiaxin ShanandAntoni Baum 2024-07-22 15:42:40 -07:00
• 69d5ae38dc [ci] Use different sccache bucket for CUDA 11.8 wheel build (#6656) Kevin H. Luu 2024-07-22 14:20:41 -07:00
• fea59c7712 [Bugfix][Kernel] Use int64_t for indices in fp8 quant kernels (#6649) Tyler Michael Smith 2024-07-22 16:08:30 -04:00
• 739b61a348 [Frontend] Refactor prompt processing (#4028) Cyrus LeungandRoger Wang 2024-07-23 01:13:53 +08:00
• 89c1c6a196 [Bugfix] Fix vocab_size field access in llava_next.py (#6624) Jae-Won Chung 2024-07-22 01:02:51 -04:00
• 42de2cefcb [Misc] Add a wrapper for torch.inference_mode (#6618) Woosuk Kwon 2024-07-21 18:43:11 -07:00
• c9eef37f32 [Model] Initial Support for Chameleon (#5770) Roger Wang 2024-07-21 17:37:51 -07:00
• 396d92d5e0 [Kernel][Core] Add AWQ support to the Marlin kernel (#6612) Alexander Matveev 2024-07-21 19:41:42 -04:00
• 25e778aa16 [Model] Refactor and decouple phi3v image embedding (#6621) Isotr0py 2024-07-22 07:07:58 +08:00
• b6df37f943 [Misc] Remove abused noqa (#6619) Woosuk Kwon 2024-07-21 08:47:04 -07:00
• 14f91fe67c [Spec Decode] Disable Log Prob serialization to CPU for spec decoding for both draft and target models. (#6485) sroy745 2024-07-20 23:58:58 -07:00
• d7f4178dd9 [Frontend] Move chat utils (#6602) Cyrus LeungandRoger Wang 2024-07-21 08:38:17 +08:00
• 082ecd80d5 [ Bugfix ] Fix AutoFP8 fp8 marlin (#6609) Robert Shaw 2024-07-20 19:25:56 -04:00
• f952bbc8ff [Misc] Fix input_scale typing in w8a8_utils.py (#6579) Michael Goin 2024-07-20 19:11:13 -04:00
• 9364f74eee [ Kernel ] Enable fp8-marlin for fbgemm-fp8 models (#6606) Robert Shaw 2024-07-20 14:50:10 -04:00
• 06d6c5fe9f [Bugfix][CI/Build][Hardware][AMD] Fix AMD tests, add HF cache, update CK FA, add partially supported model notes (#6543) Matt Wong 2024-07-20 11:39:07 -05:00
• 683e3cb9c4 [ Misc ] fbgemm checkpoints (#6559) Robert Shaw 2024-07-20 12:36:57 -04:00
• 9042d68362 [Misc] Consolidate and optimize logic for building padded tensors (#6541) Cyrus Leung 2024-07-20 12:17:24 +08:00
• 3f8d42c81f Pipeline Parallel: Guard for KeyErrors at request abort (#6587) Travis Johnson 2024-07-19 20:18:19 -06:00
• 7bd82002ae [Core] Allow specifying custom Executor (#6557) Antoni Baum 2024-07-19 18:25:06 -07:00
• 2e26564259 [ Kernel ] FP8 Dynamic Per Token Quant - Add scale_ub (#6593) Varun Sundar RabindranathandVarun Sundar Rabindranth 2024-07-19 21:15:26 -04:00
• e81522e879 [build] add ib in image for out-of-the-box infiniband support (#6599) youkaichao 2024-07-19 17:16:57 -07:00
• 45ceb85a0c [Docs] Update PP docs (#6598) Murali Andoorveedu 2024-07-19 19:38:21 -04:00
• 4cc24f01b1 [ Kernel ] Enable Dynamic Per Token fp8 (#6547) Robert Shaw 2024-07-19 19:08:15 -04:00
• 07eb6f19f3 [bugfix][distributed] fix multi-node bug for shared memory (#6597) youkaichao 2024-07-19 15:34:34 -07:00
• f0bbfaf917 [Bugfix] [SpecDecode] AsyncMetricsCollector: update time since last collection (#6578) Thomas Parnell 2024-07-19 23:01:03 +02:00
• 30efe41532 [Docs] Update docs for wheel location (#6580) Simon Mo 2024-07-19 12:14:11 -07:00
• 9ed82e7074 [Misc] Small perf improvements (#6520) Antoni Baum 2024-07-19 12:10:56 -07:00
• 51f8aa90ad [Bugfix][Frontend] remove duplicate init logger (#6581) Daniele 2024-07-19 19:16:27 +02:00
• a5314e8698 [Model] RowParallelLinear: pass bias to quant_method.apply (#6327) Thomas Parnell 2024-07-19 15:15:22 +02:00
• a921e86392 [BUGFIX] Raise an error for no draft token case when draft_tp>1 (#6369) Woo-Yeon Lee 2024-07-19 22:01:09 +09:00
• 6366efc67b [Bugfix][Frontend] Fix missing /metrics endpoint (#6463) Cyrus Leung 2024-07-19 11:55:13 +08:00
• dbe5588554 [ Misc ] non-uniform quantization via compressed-tensors for Llama (#6515) Robert Shaw 2024-07-18 22:39:18 -04:00
• d4201e06d5 [Bugfix] Make spec. decode respect per-request seed. (#6034) Thomas ParnellandNick Hill 2024-07-19 04:22:08 +02:00
• b5672a112c [Core] Multiprocessing Pipeline Parallel support (#6130) Nick HillandMurali Andoorveedu 2024-07-18 19:15:52 -07:00
• c5df56f88b Add support for a rope extension method (#6553) Simon Mo 2024-07-18 18:53:03 -07:00
• 1689219ebf [CI/Build] Build on Ubuntu 20.04 instead of 22.04 (#6517) Tyler Michael Smith 2024-07-18 20:29:25 -04:00
• 4ffffccb7e [Kernel] Implement fallback for FP8 channelwise using torch._scaled_mm (#6552) Tyler Michael Smith 2024-07-18 19:52:22 -04:00
• f53b8f0d05 [ci][test] add correctness test for cpu offloading (#6549) youkaichao 2024-07-18 16:41:06 -07:00
• 2d4733ba2d Fix PR comment bot (#6554) Kevin H. Luu 2024-07-18 14:48:29 -07:00
• 15c6a079b1 [Model] Support Mistral-Nemo (#6548) Michael Goin 2024-07-18 16:31:50 -04:00
• ecdb462c24 [ci] Reword Github bot comment (#6534) Kevin H. Luu 2024-07-18 08:01:45 -07:00
• 58ca663224 [ Misc ] Improve Min Capability Checking in compressed-tensors (#6522) Robert Shaw 2024-07-18 10:39:12 -04:00
• 4634c8728b [TPU] Refactor TPU worker & model runner (#6506) Woosuk Kwon 2024-07-18 01:34:16 -07:00
• c8a7d51c49 [Bugfix] Update flashinfer.py with PagedAttention forwards - Fixes Gemma2 OpenAI Server Crash (#6501) Noam Gat 2024-07-18 10:47:13 +03:00
• e2fbaee725 [BugFix][Frontend] Use LoRA tokenizer in OpenAI APIs (#6227) Nick HillandCyrus Leung 2024-07-18 00:13:30 -07:00
• 8a74c68bd1 [Misc] Minor patch for draft model runner (#6523) Cody Yu 2024-07-17 23:06:21 -07:00
• 61e592747c [Core] Introduce SPMD worker execution using Ray accelerated DAG (#6032) Rui QiaoandStephanie Wang 2024-07-17 22:27:09 -07:00
• d25877dd9b [BugFix] Avoid secondary error in ShmRingBuffer destructor (#6530) Nick Hill 2024-07-17 22:24:43 -07:00
• 1c27d25fb5 [core][model] yet another cpu offload implementation (#6496) youkaichaoandMichael Goin 2024-07-17 20:54:35 -07:00
• 18fecc3559 [ Kernel ] Fp8 Channelwise Weight Support (#6487) Robert Shaw 2024-07-17 23:18:13 -04:00
• b5af8c223c [Model] Pipeline parallel support for Mixtral (#6516) Cody Yu 2024-07-17 19:26:04 -07:00
• b5241e41d9 [ Kernel ] FP8 Dynamic-Per-Token Quant Kernel (#6511) Varun Sundar RabindranathandVarun Sundar Rabindranath 2024-07-17 21:38:35 -04:00
• e76466dde2 [Core] draft_model_runner: Implement prepare_inputs on GPU for advance_step (#6338) Alexander Matveev 2024-07-17 17:30:28 -04:00
• 5f0b9933e6 [Bugfix] Fix Ray Metrics API usage (#6354) Antoni Baum 2024-07-17 12:40:10 -07:00
• a38524f338 [DOC] - Add docker image to Cerebrium Integration (#6510) milo157 2024-07-17 13:22:53 -04:00
• 2fa4623d9e [Core] Refactor _prepare_model_input_tensors - take 2 (#6164) Cody Yu 2024-07-17 09:37:16 -07:00
• a9a2e74d21 [Misc] Use torch.Tensor for type annotation (#6505) Woosuk Kwon 2024-07-17 06:01:10 -07:00
• e09ce759aa [TPU] Remove multi-modal args in TPU backend (#6504) Woosuk Kwon 2024-07-17 04:02:53 -07:00