iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
14,386 Commits 3 Branches 0 Tags
fa6a6be51978bd4b49ba0da17039e60f96dc5b13
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
Zhuohan Li d707678dfb Fix num_logprobs parameter description in sampler.py (#34451) 
Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>
 2026-02-12 18:18:03 -08:00
Zhuohan LiandNick Hill d0cd728907 [Core] Support reseting all running requests' KV while calling reset_prefix_cache (#28827) 
Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-12-02 02:25:05 +00:00
Zhuohan Li 552cac95b5 [Misc] Fix wrong comment in scheduler (#28880) 
Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>
 2025-11-17 15:32:22 -08:00
Zhuohan Li dd6ac1c2bb [RL] [V1] Remove unused device argument from reset_kv_cache (#28766) 
Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>
 2025-11-14 23:59:42 -08:00
Zhuohan Li 8d706cca90 [Misc] FlattenLogprobs -> FlatLogprobs (#28335) 2025-11-11 03:41:23 +00:00
Zhuohan Li 300a265978 [Core] Enable StatLogger in LLMEngine (#28020) 
Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>
 2025-11-04 04:13:35 -08:00
Zhuohan Li 56ed7609a9 Revert "[Misc] Remove use of CUDA_VISIBLE_DEVICES for device selectio… (#27502) 2025-10-25 05:31:43 +00:00
Zhuohan Li d29483b58a [Minor] Remove unnecessary error message (#27115) 
Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>
 2025-10-17 20:02:12 +00:00
Zhuohan Li ebf6ef1a9b [Minor] Change warning->warning_once in preprocess (#26455) 
Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>
 2025-10-08 21:09:06 -07:00
Zhuohan Li d3bd171123 [Benchmark] Support benchmark throughput for external launcher DP (#25913) 
Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>
 2025-09-30 01:43:57 +00:00
Zhuohan Li 8eb0a1d906 [Doc] Polish example for torchrun dp (#25899) 2025-09-29 21:31:34 +00:00
Zhuohan Li 8bf8f45822 [Core] Don't count preempted tokens in prefix cache hit rate (#25787) 
Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>
 2025-09-27 00:16:40 +00:00
Zhuohan LiandYe Qi 8c435c9bce [Core] Enable command line logging for LLMEngine (#25610) 
Co-authored-by: Ye (Charlotte) Qi <yeq@meta.com>
Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>
 2025-09-25 15:31:17 -07:00
Zhuohan Li 6c47f6bfa4 [Core] Remove tokenizer group in vLLM (#24078) 
Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>
 2025-09-17 08:42:59 +00:00
Zhuohan Li 886ccbe5ba [CI/Build] Reduce the number of redundant cases to test for LoRA (#24276) 
Signed-off-by: Zhuohan Li <zhuohan123@gmail.com>
 2025-09-04 21:58:44 +00:00
Zhuohan Li 326fcc8b9f [Doc] Deprecate Discord (#12668) 2025-02-02 19:19:56 -08:00
Zhuohan Li 49628fe13e [Doc] Update README.md with Ray Summit talk links (#10610) 2024-11-24 16:45:09 -08:00
Zhuohan Li a95354a36e [Doc] Update README.md with Ray summit slides (#9088) 2024-10-05 02:54:45 +00:00
Zhuohan Li bd0e7802e0 [Bugfix] Add warmup for prefix caching example (#5235) 2024-06-03 19:36:41 -07:00
Zhuohan Li 8279078e21 [Bugfix] Remove deprecated @abstractproperty (#5174) 2024-06-01 22:40:25 +00:00
Zhuohan Lirsnm2Robert Shaw
 		1102bef219 [Bugfix / Core] Prefix Caching Guards (merged with main) (#4846) 
Co-authored-by: rsnm2 <rshaw@neuralmagic.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-neuralmagic@users.noreply.github.com>
 2024-05-27 15:18:17 -07:00
Zhuohan Li 361c461a12 [Doc] Highlight the fourth meetup in the README (#4842) 2024-05-15 11:38:49 -07:00
Zhuohan Li c579b750a0 [Doc] Add meetups to the doc (#4798) 2024-05-13 18:48:00 -07:00
Zhuohan Li ac1fbf7fd2 [Doc] Shorten README by removing supported model list (#4796) 2024-05-13 16:23:54 -07:00
Zhuohan Li e11e200736 [Bugfix] Fix filelock version requirement (#4075) 2024-04-14 21:50:08 -07:00
Zhuohan Li d4ec9ffb95 [Misc] Fix typo in scheduler.py (#4022) 2024-04-12 13:56:04 -07:00
Zhuohan Li e90fc21f2e [Hardware][Neuron] Refactor neuron support (#3471) 2024-03-22 01:22:17 +00:00
Zhuohan Li 523e30ea0c [BugFix] Hot fix in setup.py for neuron build (#3537) 2024-03-20 17:59:52 -07:00
Zhuohan Li b30880a762 [Misc] Update README for the Third vLLM Meetup (#3479) 2024-03-18 15:58:38 -07:00
Zhuohan Li c0c17d4896 [Misc] Fix PR Template (#3478) 2024-03-18 15:00:31 -07:00
Zhuohan Li eeab52a4ff [FIX] Simpler fix for async engine running on ray (#3371) 2024-03-13 14:18:40 -07:00
Zhuohan Li 4c922709b6 Add distributed model executor abstraction (#3191) 2024-03-11 11:03:45 -07:00
Zhuohan Li 2f8844ba08 Re-enable the 80 char line width limit (#3305) 2024-03-10 19:49:14 -07:00
Zhuohan Li f48c6791b7 [FIX] Fix prefix test error on main (#3286) 2024-03-08 17:16:14 -08:00
Zhuohan Li 996d095c54 [FIX] Fix styles in automatic prefix caching & add a automatic prefix caching benchmark (#3158) 2024-03-03 14:37:18 -08:00
Zhuohan Li 8fbd84bf78 Bump up version to v0.3.2 (#2968) 
This version is for more model support. Add support for Gemma models (#2964) and OLMo models (#2832).
 2024-02-21 11:47:25 -08:00
Zhuohan Li a9c8212895 [FIX] Add Gemma model to the doc (#2966) 2024-02-21 09:46:15 -08:00
Zhuohan Li 63e2a6419d [FIX] Fix beam search test (#2930) 2024-02-20 14:37:39 -08:00
Zhuohan Li a61f0521b8 [Test] Add basic correctness test (#2908) 2024-02-18 16:44:50 -08:00
Zhuohan Li 537c9755a7 [Minor] Small fix to make distributed init logic in worker looks cleaner (#2905) 2024-02-18 14:39:00 -08:00
Zhuohan Li 1af090b57d Bump up version to v0.3.0 (#2656) 2024-01-31 00:07:07 -08:00
Zhuohan Li ef9b636e2d Simplify broadcast logic for control messages (#2501) 2024-01-19 11:23:30 -08:00
Zhuohan Li f745847ef7 [Minor] Fix the format in quick start guide related to Model Scope (#2425) 2024-01-11 19:44:01 -08:00
Zhuohan Li fd4ea8ef5c Use NCCL instead of ray for control-plane communication to remove serialization overhead (#2221) 2024-01-03 11:30:22 -08:00
Zhuohan Li 358c328d69 [BUGFIX] Fix communication test (#2285) 2023-12-27 17:18:11 -05:00
Zhuohan Li 4aaafdd289 [BUGFIX] Fix the path of test prompts (#2273) 2023-12-26 10:37:21 -08:00
Zhuohan Li 66b108d142 [BUGFIX] Fix API server test (#2270) 2023-12-26 10:37:06 -08:00
Zhuohan Li e0ff920001 [BUGFIX] Do not return ignored sentences twice in async llm engine (#2258) 2023-12-26 13:41:09 +08:00
Zhuohan Li e19a64c7ef [FIX] Fix formatting error in main branch (#1822) 2023-11-28 16:56:43 -08:00
Zhuohan Li 1cb4ad8de9 [FIX] Fix formatting error 2023-11-29 00:40:19 +00:00
Zhuohan Li 708e6c18b0 [FIX] Fix class naming (#1803) 2023-11-28 14:08:01 -08:00
Zhuohan Li 7d761fe3c1 [FIX] Fix the case when input_is_parallel=False for ScaledActivation (#1737) 2023-11-20 23:56:48 -08:00
Zhuohan Li 32c927b53f [FIX] Update the doc link in README.md (#1730) 2023-11-20 12:46:24 -08:00
Zhuohan Li 0fc280b06c Update the adding-model doc according to the new refactor (#1692) 2023-11-16 18:46:26 -08:00
Zhuohan Li 20d0699d49 [Fix] Fix comm test (#1691) 2023-11-16 16:28:39 -08:00
Zhuohan Li 415d109527 [Fix] Update Supported Models List (#1690) 2023-11-16 14:47:26 -08:00
Zhuohan Li 7076fa1c9f TP/quantization/weight loading refactor part 2 - Refactor quantized linear logic and extend quantization support to all models (#1622) 
Refactor the tensor parallelism, quantization, and weight-loading codes.

Summary of the new features enabled by this PR:
- **All models** are able to be quantized with AWQ and SqueezeLLM, and [soon GPTQ](https://github.com/vllm-project/vllm/pull/1580).
- Model loading code became much simpler.
- Support model parallelism for all MQA/GQA models when the number of key/value heads is smaller than the tensor parallel size.
 2023-11-15 22:50:41 -08:00
Zhuohan LiandWoosuk Kwon 06458a0b42 Upgrade to CUDA 12 (#1527) 
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2023-11-08 14:17:49 -08:00
Zhuohan Li 7b895c5976 [Fix] Fix duplicated logging messages (#1524) 2023-10-31 09:04:47 -07:00
Zhuohan Li 651c614aa4 Bump up the version to v0.2.1 (#1355) 2023-10-16 12:58:57 -07:00
Zhuohan LiandYunmo Chen 9d9072a069 Implement prompt logprobs & Batched topk for computing logprobs (#1328) 
Co-authored-by: Yunmo Chen <16273544+wanmok@users.noreply.github.com>
 2023-10-16 10:56:50 -07:00
Zhuohan Li b95ee898fe [Minor] Fix comment in mistral.py (#1303) 2023-10-09 19:44:37 -07:00
Zhuohan Li 9eed4d1f3e Update README.md (#1292) 2023-10-08 23:15:50 -07:00
Zhuohan Li 6b5296aa3a [FIX] Explain why the finished_reason of ignored sequences are length (#1289) 2023-10-08 15:22:38 -07:00
Zhuohan Li ba0bfd40e2 TP/quantization/weight loading refactor part 1 - Simplify parallel linear logic (#1181) 2023-10-02 15:36:09 -07:00
Zhuohan LiandJiCheng a60b353005 support sharding llama2-70b on more than 8 GPUs (#1209) 
Co-authored-by: JiCheng <247153481@qq.com>
 2023-10-02 15:26:33 -07:00
Zhuohan Li f187877945 [FIX] Simplify sampler logic (#1156) 2023-09-23 17:21:56 -07:00
Zhuohan LiandAntoni Baum 947b794146 [Sampler] Vectorized sampling (simplified) (#1048) 
Co-authored-by: Antoni Baum <antoni.baum@protonmail.com>
 2023-09-22 17:48:04 -07:00
Zhuohan Li c1026311b5 [Community] Add vLLM Discord server (#1086) 2023-09-18 12:23:35 -07:00
Zhuohan Li f029ef94d7 Fix get_max_num_running_seqs for waiting and swapped seq groups (#1068) 2023-09-18 11:49:40 -07:00
Zhuohan Li 90979c38f8 [FIX] Don't initialize parameter by default (#1067) 2023-09-17 17:15:38 -07:00
Zhuohan Li f04908cae7 [FIX] Minor bug fixes (#1035) 
* [FIX] Minor bug fixes

* Address review comments
 2023-09-13 16:38:12 -07:00
Zhuohan Li 1117aa1411 Bump up the version to v0.1.6 (#989) 2023-09-08 00:07:46 -07:00
Zhuohan LiWoosuk KwonOliver-ss
 		db09d4ad83 [FIX] Fix Alibi implementation in PagedAttention kernel (#945) 
* [FIX] Fix Alibi implementation in PagedAttention kernel

* Fix test_attention

* Fix

---------

Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: Oliver-ss <yuansongwx@outlook.com>
 2023-09-07 15:53:14 -07:00
Zhuohan Li c957c741d9 Enable safetensors loading for all models (#974) 2023-09-07 15:49:52 -07:00
Zhuohan Li 002800f081 Align vLLM's beam search implementation with HF generate (#857) 2023-09-04 17:29:42 -07:00
Zhuohan Li c128d69856 Fix README.md Link (#927) 2023-08-31 17:18:34 -07:00
Zhuohan Li 0080d8329d Add acknowledgement to a16z grant 2023-08-30 02:26:47 -07:00
Zhuohan Li d2b2eed67c [Fix] Fix a condition for ignored sequences (#867) 2023-08-27 23:00:56 -07:00
Zhuohan Li 14f9c72bfd Update Supported Model List (#825) 2023-08-22 11:51:44 -07:00
Zhuohan Li aa84c92ef6 Bump up version to 0.1.3 (#657) 2023-08-02 16:46:53 -07:00
Zhuohan Li f7389f4763 [Doc] Add Baichuan 13B to supported models (#656) 2023-08-02 16:45:12 -07:00
Zhuohan Li 1b0bd0fe8a Add Falcon support (new) (#592) 2023-08-02 14:04:39 -07:00
Zhuohan Li 58a072be15 [Fix] Add model sequence length into model config (#575) 2023-07-25 23:46:30 -07:00
Zhuohan Li 82ad323dee [Fix] Add chat completion Example and simplify dependencies (#576) 2023-07-25 23:45:48 -07:00
Zhuohan Li df5dd3c68e Add Baichuan-7B to README (#494) 2023-07-25 15:25:12 -07:00
Zhuohan Li 7d5a155e4a [Fix] Fix GPTBigcoder for distributed execution (#503) 2023-07-24 18:36:33 -07:00
Zhuohan Li 6fc2a38b11 Add support for LLaMA-2 (#505) 2023-07-20 11:38:27 -07:00
Zhuohan Li 2bdea7ac11 [Fix] Fix the condition of max_seq_len (#477) 2023-07-17 00:33:48 -04:00
Zhuohan Li 96853af5a8 Optimize MQA Kernel (#452) 2023-07-14 20:06:40 -04:00
Zhuohan Li 98fe8cb542 [Server] Add option to specify chat template for chat endpoint (#345) 2023-07-03 23:01:56 -07:00
Zhuohan Li 42e0c1df78 [Quality] Add CI for formatting (#343) 2023-07-03 14:50:56 -07:00
Zhuohan Li d6fa1be3a8 [Quality] Add code formatter and linter (#326) 2023-07-03 11:31:55 -07:00
Zhuohan Li 0ffded812a [Fix] Better error message for batched prompts (#342) 2023-07-03 09:27:31 -07:00
Zhuohan Li 598dc4b79a [Fix] Weight loading for GPTBigCode (#313) 2023-06-29 22:14:17 -07:00
Zhuohan Li 85de093472 [Fix] Do not pin memory when in WSL (#312) 2023-06-29 15:00:21 -07:00
Zhuohan Li 43710e8d09 [Fix] Fix default port number in benchmark scripts (#265) 2023-06-26 13:15:35 -07:00
Zhuohan Li 2cf1a333b6 [Doc] Documentation for distributed inference (#261) 2023-06-26 11:34:23 -07:00
Zhuohan Li 0b7db411b5 [Bug] Fix the OOM condition for CPU cache (#260) 2023-06-26 11:16:13 -07:00
Zhuohan Li 83658c8ace Bump up version to 0.1.1 (#204) 2023-06-22 15:33:32 +08:00