Commit Graph

Select branches

Hide Pull Requests

main

patches

sm86

Mono Color

• b0925b3878 docs: Add BentoML deployment doc (#3336) Sherlock Xu 2024-03-13 01:34:30 +08:00
• 654865e21d Support Mistral Model Inference with transformers-neuronx (#3153) DAIZHENWEI 2024-03-11 13:19:51 -07:00
• c9415c19d3 [ROCm] Fix warp and lane calculation in blockReduceSum (#3321) kliuae 2024-03-12 04:14:07 +08:00
• 4c922709b6 Add distributed model executor abstraction (#3191) Zhuohan Li 2024-03-11 11:03:45 -07:00
• 657061fdce [docs] Add LoRA support information for models (#3299) Philipp Moritz 2024-03-11 00:54:51 -07:00
• 2f8844ba08 Re-enable the 80 char line width limit (#3305) Zhuohan Li 2024-03-10 19:49:14 -07:00
• 4b59f00e91 [Fix] Fix best_of behavior when n=1 (#3298) Nick Hill 2024-03-10 19:17:46 -07:00
• 9e8744a545 [BugFix] Fix get tokenizer when using ray (#3301) Roy 2024-03-11 10:17:16 +08:00
• e4a28e5316 [ROCM] Fix blockReduceSum to use correct warp counts for ROCm and CUDA (#3262) Douglas Lehr 2024-03-10 17:27:45 -05:00
• 0bba88df03 Enhance lora tests with more layer and rank variations (#3243) Terry 2024-03-09 17:14:16 -08:00
• 8437bae6ef [Speculative decoding 3/9] Worker which speculates, scores, and applies rejection sampling (#3103) Cade Daniel 2024-03-08 23:32:46 -08:00
• f48c6791b7 [FIX] Fix prefix test error on main (#3286) Zhuohan Li 2024-03-08 17:16:14 -08:00
• c2c5e0909a Move model filelocks from /tmp/ to ~/.cache/vllm/locks/ dir (#3241) Michael Goin 2024-03-08 13:33:10 -08:00
• 1cb0cc2975 [FIX] Make flash_attn optional (#3269) Woosuk Kwon 2024-03-08 10:52:20 -08:00
• 99c3cfb83c [Docs] Fix Unmocked Imports (#3275) Roger Wang 2024-03-08 09:58:01 -08:00
• 1ece1ae829 [Minor Fix] Fix comments in benchmark_serving (#3252) TianYu GUO 2024-03-08 14:22:59 +08:00
• c59e120c55 Feature add lora support for Qwen2 (#3177) whyiug 2024-03-08 13:58:24 +08:00
• d2339d6840 Connect engine healthcheck to openai server (#3260) Nick Hill 2024-03-07 16:38:12 -08:00
• b35cc93420 Fix auto prefix bug (#3239) ElizaWszola 2024-03-08 01:37:28 +01:00
• 8cbba4622c Possible fix for conflict between Automated Prefix Caching (#2762) and multi-LoRA support (#1804) (#3263) jacobthebanana 2024-03-07 18:03:22 -05:00
• 385da2dae2 Measure model memory usage (#3120) Michael Goin 2024-03-07 11:42:42 -08:00
• 2daf23ab0c Separate attention backends (#3005) Woosuk Kwon 2024-03-07 01:45:50 -08:00
• cbf4c05b15 Update requirements-dev.txt to include package for benchmarking scripts. (#3181) Chen WangandZhuohan Li 2024-03-07 03:39:28 -05:00
• d3c04b6a39 Add GPTQ support for Gemma (#3200) TechxGenus 2024-03-07 08:19:14 +08:00
• 4cb3b924cd Add tqdm dynamic_ncols=True (#3242) Chujie Zheng 2024-03-06 14:41:42 -08:00
• a33ce60c66 [Testing] Fix core tests (#3224) Cade Daniel 2024-03-06 01:04:23 -08:00
• 24aecf421a [Tests] Add block manager and scheduler tests (#3108) SangBin Cho 2024-03-06 11:23:34 +09:00
• 2efce05dc3 [Fix] Avoid pickling entire LLMEngine for Ray workers (#3207) Nick HillandAntoni Baum 2024-03-05 16:17:20 -08:00
• 8999ec3c16 Store eos_token_id in Sequence for easy access (#3166) Nick Hill 2024-03-05 15:35:43 -08:00
• 05af6da8d9 [ROCm] enable cupy in order to enable cudagraph mode for AMD GPUs (#3123) Hongxia Yangandlcskrishna 2024-03-04 21:14:53 -05:00
• 9a4548bae7 Fix the openai benchmarking requests to work with latest OpenAI apis (#2992) Chen WangandRoger Wang 2024-03-04 18:51:56 -05:00
• ff578cae54 Add health check, make async Engine more robust (#3015) Antoni BaumandZhuohan Li 2024-03-04 14:01:40 -08:00
• 22de45235c Push logprob generation to LLMEngine (#3065) Antoni BaumandAvnish Narayan 2024-03-04 11:54:06 -08:00
• 76e8a70476 [Minor fix] The domain dns.google may cause a socket.gaierror exception (#3176) ttbachyinsdaandguofangze 2024-03-05 03:17:12 +08:00
• 9cbc7e5f3b enable --gpu-memory-utilization in benchmark_throughput.py (#3175) Allen.Douandzixiao 2024-03-05 02:37:58 +08:00
• 27a7b070db Add document for vllm paged attention kernel. (#2978) Jialun Lyu 2024-03-04 09:23:34 -08:00
• 901cf4c52b [Minor Fix] Remove unused code in benchmark_prefix_caching.py (#3171) TianYu GUO 2024-03-04 14:48:27 +08:00
• d0fae88114 [DOC] add setup document to support neuron backend (#2777) Liangfu Chen 2024-03-03 17:03:51 -08:00
• 17c3103c56 Make it easy to profile workers with nsight (#3162) Philipp MoritzandRoger Wang 2024-03-03 16:19:13 -08:00
• 996d095c54 [FIX] Fix styles in automatic prefix caching & add a automatic prefix caching benchmark (#3158) Zhuohan Li 2024-03-03 14:37:18 -08:00
• d65fac2738 Add vLLM version info to logs and openai API server (#3161) Jason Cox 2024-03-03 00:00:29 -05:00
• ce4f5a29fb Add Automatic Prefix Caching (#2762) 3 people 2024-03-02 03:50:01 -05:00
• baee28c46c Reorder kv dtype check to avoid nvcc not found error on AMD platform (#3104) cloudhan 2024-03-02 14:34:48 +08:00
• 29e70e3e88 allow user chose log level by --log-level instead of fixed 'info'. (#3109) 3 people 2024-03-02 07:28:41 +08:00
• 82091b864a Bump up to v0.3.3 (#3129) Woosuk Kwon 2024-03-01 12:58:06 -08:00
• c0c2335ce0 Integrate Marlin Kernels for Int4 GPTQ inference (#2497) 3 people 2024-03-01 14:47:51 -06:00
• 90fbf12540 fix relative import path of protocol.py (#3134) Huarongandhuohuarong 2024-03-02 03:42:06 +08:00
• 49d849b3ab docs: Add tutorial on deploying vLLM model with KServe (#2586) Yuan Tang 2024-03-01 14:04:14 -05:00
• 27ca23dc00 Remove exclude_unset in streaming response (#3143) Seonghyeon 2024-03-02 02:59:06 +09:00
• 54d3544784 Fix: Output text is always truncated in some models (#3016) Sherry 2024-03-01 15:52:22 +08:00
• 703e42ee4b Add guided decoding for OpenAI API server (#2819) 3 people 2024-02-29 14:13:08 -08:00
• 29a8d6a554 [Fix] Don't deep-copy LogitsProcessors when copying SamplingParams (#3099) Nick Hill 2024-02-29 11:20:42 -08:00
• 2c08ff23c0 Fix building from source on WSL (#3112) Billy Cao 2024-03-01 03:13:58 +08:00
• bfdcfa6a05 Support starcoder2 architecture (#3089) Seonghyeon 2024-02-29 17:51:48 +09:00
• 9289e577ec add cache_config's info to prometheus metrics. (#3100) Allen.Dou 2024-02-29 14:15:18 +08:00
• a6d471c759 Fix: AttributeError in OpenAI-compatible server (#3018) Jae-Won Chung 2024-02-29 01:04:07 -05:00
• 01a5d18a53 Add Support for 2/3/8-bit GPTQ Quantization Models (#2330) CHU Tianxiang 2024-02-29 13:52:23 +08:00
• 929b4f2973 Add LoRA support for Gemma (#3050) Woosuk Kwon 2024-02-28 13:03:28 -08:00
• 3b7178cfa4 [Neuron] Support inference with transformers-neuronx (#2569) Liangfu Chen 2024-02-28 09:34:34 -08:00
• e46fa5d52e Restrict prometheus_client >= 0.18.0 to prevent errors when importing pkgs (#3070) Allen.Dou 2024-02-28 13:38:26 +08:00
• a8683102cc multi-lora documentation fix (#3064) Ganesh Jagadeesan 2024-02-28 00:26:15 -05:00
• 71bcaf99e2 Enable GQA support in the prefix prefill kernels (#3007) Tao He 2024-02-27 17:14:31 +08:00
• 8b430d7dea [Minor] Fix StableLMEpochForCausalLM -> StableLmForCausalLM (#3046) Woosuk Kwon 2024-02-26 20:23:50 -08:00
• e0ade06d63 Support logit bias for OpenAI API (#3027) Dylan Hawk 2024-02-26 19:51:53 -08:00
• 4bd18ec0c7 [Minor] Fix type annotation in fused moe (#3045) Woosuk Kwon 2024-02-26 19:44:29 -08:00
• 2410e320b3 fix get_ip error in pure ipv6 environment (#2931) Jingru 2024-02-27 11:22:16 +08:00
• 48a8f4a7fd Support Orion model (#2539) 3 people 2024-02-27 11:17:06 +08:00
• 4dd6416faf Fix stablelm (#3038) Roy 2024-02-27 10:31:10 +08:00
• c1c0d00b88 Don't use cupy when enforce_eager=True (#3037) Roy 2024-02-27 09:33:38 +08:00
• d9f726c4d0 [Minor] Remove unused config files (#3039) Roy 2024-02-27 09:25:22 +08:00
• d6e4a130b0 [Minor] Remove gather_cached_kv kernel (#3043) Woosuk Kwon 2024-02-26 15:00:54 -08:00
• cfc15a1031 Optimize Triton MoE Kernel (#2979) Philipp MoritzandCade Daniel 2024-02-26 13:48:56 -08:00
• 70f3e8e3a1 Add LogProbs for Chat Completions in OpenAI (#2918) Jared Moore 2024-02-25 18:39:34 -08:00
• ef978fe411 Port metrics from aioprometheus to prometheus_client (#2730) Harry Mellor 2024-02-25 19:54:00 +00:00
• f7c1234990 [Fix] Fissertion on YaRN model len (#2984) Woosuk Kwon 2024-02-23 12:57:48 -08:00
• 57f044945f Fix nvcc not found in vlm-openai image (#2781) zhaoyang-star 2024-02-23 06:25:07 +08:00
• 4caf7044e0 Include tokens from prompt phase in counter_generation_tokens (#2802) Ronen Schaffer 2024-02-23 00:00:12 +02:00
• 6f32cddf1c Remove Flash Attention in test env (#2982) Woosuk Kwon 2024-02-22 09:58:29 -08:00
• c530e2cfe3 [FIX] Fix a bug in initializing Yarn RoPE (#2983) 44670 2024-02-22 17:40:05 +08:00
• fd5dcc5c81 Optimize GeGLU layer in Gemma (#2975) Woosuk Kwon 2024-02-21 20:17:52 -08:00
• 93dc5a2870 chore(vllm): codespell for spell checking (#2820) Massimiliano Pronesti 2024-02-22 02:56:01 +00:00
• 95529e3253 Use Llama RMSNorm custom op for Gemma (#2974) Woosuk Kwon 2024-02-21 18:28:23 -08:00
• 344020c926 Migrate MistralForCausalLM to LlamaForCausalLM (#2868) Roy 2024-02-22 10:25:05 +08:00
• 5574081c49 Added early stopping to completion APIs (#2939) Mustafa Eyceoz 2024-02-21 21:24:01 -05:00
• d7f396486e Update comment (#2934) Ronen Schaffer 2024-02-22 04:18:37 +02:00
• 8fbd84bf78 Bump up version to v0.3.2 (#2968) Zhuohan Li 2024-02-21 11:47:25 -08:00
• 7d2dcce175 Support per-request seed (#2514) Nick Hill 2024-02-21 11:47:00 -08:00
• dc903e70ac [ROCm] Upgrade transformers to v4.38.0 (#2967) Woosuk Kwon 2024-02-21 09:46:57 -08:00
• a9c8212895 [FIX] Add Gemma model to the doc (#2966) Zhuohan Li 2024-02-21 09:46:15 -08:00
• c20ecb6a51 Upgrade transformers to v4.38.0 (#2965) Woosuk Kwon 2024-02-21 09:38:03 -08:00
• 5253edaacb Add Gemma model (#2964) Xiang Xu 2024-02-21 09:34:30 -08:00
• 017d9f1515 Add metrics to RequestOutput (#2876) Antoni Baum 2024-02-20 21:55:57 -08:00
• 181b27d881 Make vLLM logging formatting optional (#2877) Antoni Baum 2024-02-20 14:38:55 -08:00
• 63e2a6419d [FIX] Fix beam search test (#2930) Zhuohan Li 2024-02-20 14:37:39 -08:00
• 264017a2bf [ROCm] include gfx908 as supported (#2792) James Whedbee 2024-02-19 19:58:59 -06:00
• e433c115bc Fix vllm:prompt_tokens_total metric calculation (#2869) Ronen Schaffer 2024-02-19 09:55:41 +02:00
• 86fd8bb0ac Add warning to prevent changes to benchmark api server (#2858) Simon Mo 2024-02-18 21:36:19 -08:00
• ab3a5a8259 Support OLMo models. (#2832) Isotr0py 2024-02-19 13:05:15 +08:00
• a61f0521b8 [Test] Add basic correctness test (#2908) Zhuohan Li 2024-02-18 16:44:50 -08:00
• 537c9755a7 [Minor] Small fix to make distributed init logic in worker looks cleaner (#2905) Zhuohan Li 2024-02-18 14:39:00 -08:00