iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
7,161 Commits 3 Branches 0 Tags
ede5c4ebdfbbcbcd22d2b3fdb4da317ecc7254d4
Commit Graph
725 Commits
Author SHA1 Message Date
Li, Jiang 6458721108 [CPU] Refine default config for the CPU backend (#19539) 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
 2025-06-13 13:27:39 +08:00
rasmith c7ea0b56cd [AMD] [Quantization] Add override flag for attention dtype instead of using kv_cache_dtype trigger (#17331) 
Signed-off-by: Randall Smith <Randall.Smith@amd.com>
 2025-06-11 15:53:28 -04:00
Cyrus Leung 68b4a26149 [Doc] Update V1 User Guide for Hardware and Models (#19474) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-06-11 00:49:06 -07:00
Gregory Shtrasberg 5241ca50d6 [ROCm][V1] Adding ROCm to the list of plaforms using V1 by default (#19440) 
Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>
 2025-06-10 22:06:15 +00:00
Isotr0py 5f1ac1e1d1 Revert "[v1] Add fp32 support to v1 engine through flex attn" (#19404) 2025-06-10 01:30:20 -07:00
22quinn c1c7dbbeeb [Bugfix][Core] Prevent token lengths exceeding max_model_len in V0 (#19348) 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
 2025-06-09 23:01:29 +08:00
Isotr0py b8089195b4 [v1] Add fp32 support to v1 engine through flex attn (#19319) 
Signed-off-by: Isotr0py <2037008807@qq.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2025-06-09 22:10:44 +08:00
Cyrus Leung e31ae3de36 [Deprecation] Remove inputs arg fallback in Engine classes (#18799) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-06-09 10:19:56 +08:00
Driss Guessous cf02f9b283 Add FlexAttention to V1 (#16078) 
Signed-off-by: drisspg <drisspguessous@gmail.com>
 2025-06-06 21:58:55 -07:00
jmswen 7353492a47 [Core] Raise when non-multi-instance DP clients target a DP rank (#19227) 
Signed-off-by: Jon Swenson <jmswen@gmail.com>
 2025-06-06 19:03:01 +08:00
Chen Zhang f8a1a2d108 [v1] Hybrid Memory Allocator (#17996) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-06-05 20:47:09 -07:00
Kebe ef3f98b59f [Bugfix] fix v1 cpu worker fails on macOS (#19121) 2025-06-04 20:17:38 +00:00
jmswen c8dcc15921 Allow AsyncLLMEngine.generate to target a specific DP rank (#19102) 
Signed-off-by: Jon Swenson <jmswen@gmail.com>
 2025-06-04 08:26:47 -07:00
Kaixi Hou 41aa578428 [NVIDIA] Add Cutlass MLA backend (#17625) 2025-06-03 21:40:26 -07:00
Li, Jiang 4555143ea7 [CPU] V1 support for the CPU backend (#16441) 2025-06-03 18:43:01 -07:00
Harry MellorandYikun Jiang 6865fe0074 Fix interaction between Optional and Annotated in CLI typing (#19093) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Yikun Jiang <yikun@apache.org>
 2025-06-03 21:07:19 +00:00
Simon Mo 02f0c7b220 [Misc] Add SPDX-FileCopyrightText (#19100) 
Signed-off-by: simon-mo <simon.mo@hey.com>
 2025-06-03 11:20:17 -07:00
Reidandreidliu41 17430e3653 [bugfix] small fix logic issue (#18999) 
Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>
 2025-06-03 05:35:12 +00:00
Rui Qiao bdce64f236 [V1] Support DP with Ray (#18779) 2025-06-02 21:15:13 -07:00
jennyyyyzhenyZhenyzhen
 		ebb1ec9318 [Model] enable data parallel for Llama4 vision encoder (#18368) 
Signed-off-by: yzhen <yzhen@devgpu093.cco2.facebook.com>
Co-authored-by: yZhen <yZhen@fb.com>
Co-authored-by: yzhen <yzhen@devgpu093.cco2.facebook.com>
 2025-06-02 19:22:54 +08:00
Michael Goin 2ad6194a02 Let max_num_batched_tokens use human_readable_int for large numbers (#18968) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-06-01 11:41:29 +08:00
Michael Goin d54af615d5 [Bugfix] Fix PP default fallback behavior for V1 (#18915) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-05-30 10:13:17 +08:00
Mark McLoughlin 0e98964e94 [V1][Metrics] Remove metrics that were deprecated in 0.8 (#18837) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2025-05-28 18:54:12 +00:00
Alex Brooks 321331b8ae [Core] Add Lora Support to Beam Search (#18346) 
Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
 2025-05-28 08:58:24 -07:00
Harry Mellor 4c2b38ce9e Enable Pydantic mypy checks and convert configs to Pydantic dataclasses (#17599) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-05-28 12:46:04 +00:00
Cyrus Leung 0f0926b43f [Deprecation] Remove unused sync methods in async_timeout (#18792) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-05-28 15:08:48 +08:00
Hyogeun Oh (오효근) a68e293cb9 [Doc] Convert Sphinx directives ( {class}, {meth}, {attr}, ...) to MkDocs format for better documentation linking (#18663) 
Signed-off-by: Zerohertz <ohg3417@gmail.com>
 2025-05-27 01:44:20 -07:00
Isotr0py 1f1b1bc03b [V1][Quantization] Add CUDA graph compatible v1 GGUF support (#18646) 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Signed-off-by: Isotr0py <2037008807@qq.com>
 2025-05-27 04:40:28 +00:00
Yuqi ZhangandCyrus Leung f2faac745d [Bugfix] Fix cpu usage and cache hit stats reporting on cpu environment (#18674) 
Signed-off-by: zzzyq <zhangyuqi94@gmail.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2025-05-25 02:36:06 -07:00
Feng XiaoLongandCrucifixion-Fxl 4fc1bf813a [Bugfix] Migrate to REGEX Library to prevent catastrophic backtracking (#18454) 
Signed-off-by: Crucifixion-Fxl <xmufxl@gmail.com>
Co-authored-by: Crucifixion-Fxl <xmufxl@gmail.com>
 2025-05-23 16:16:26 -07:00
Cyrus Leung 7d9216495c [Doc] Update references to doc files (#18637) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-05-23 15:49:21 -07:00
Jiayi YaoandRui Qiao 2628a69e35 [V1] Support Deepseek MTP (#18435) 
Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
Signed-off-by: YaoJiayi <120040070@link.cuhk.edu.cn>
Co-authored-by: Rui Qiao <ruisearch42@gmail.com>
 2025-05-23 10:26:28 -07:00
Cyrus Leung 273cb3b4d9 [Doc] Fix top-level API links/docs (#18621) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-05-23 09:46:56 -07:00
Harry Mellor a1fe24d961 Migrate docs from Sphinx to MkDocs (#18145) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-05-23 02:09:53 -07:00
Sanger Steel c32e249a23 [Frontend] [Core] Add Tensorizer support for V1, LoRA adapter serialization and deserialization (#17926) 
Signed-off-by: Sanger Steel <sangersteel@gmail.com>
 2025-05-22 18:44:18 -07:00
Hyogeun Oh (오효근) 2b16104557 [Misc] Update deprecation message for --enable-reasoning (#18404) 2025-05-21 07:33:11 -07:00
Kebe 5d7f545204 [Frontend] deprecate --device arg (#18399) 
Signed-off-by: Kebe <mail@kebe7jun.com>
 2025-05-21 01:21:17 -07:00
Woosuk Kwon fabe89bbc4 [Spec Decode] Don't fall back to V0 when spec decoding is enabled (#18265) 2025-05-16 16:10:27 -07:00
Lucia Fang 3d2779c29a [Feature] Support Pipeline Parallism in torchrun SPMD offline inference for V1 (#17827) 
Signed-off-by: Lucia Fang <fanglu@fb.com>
 2025-05-15 22:28:27 -07:00
Harry Mellor b18201fe06 Allow users to pass arbitrary JSON keys from CLI (#18208) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-05-15 21:05:34 -07:00
Sky Leeandlisiqi23 f4937a51c1 [Model] vLLM v1 supports Medusa (#17956) 
Signed-off-by: lisiqi23 <lisiqi23@xiaomi.com>
Signed-off-by: skylee-01 <497627264@qq.com>
Co-authored-by: lisiqi23 <lisiqi23@xiaomi.com>
 2025-05-15 21:05:31 -07:00
Nick Hill 55aa7af994 [V1] DP scale-out (2/N): Decouple engine process management and comms (#15977) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-05-13 10:48:21 -07:00
Cyrus Leung 61e0a506a3 [Bugfix] Avoid repeatedly creating dummy data during engine startup (#17935) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-05-12 22:40:19 -07:00
Tao He 60f7624334 Implements dual-chunk-flash-attn backend for dual chunk attention with sparse attention support (#11844) 2025-05-12 19:52:47 -07:00
Reidandreidliu41 009b3d5382 [Misc] not show --model in vllm serve --help (#16691) 
Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>
 2025-05-11 08:47:58 +00:00
Gregory Shtrasberg 06c0922a69 [FP8][ROCm][Attention] Enable FP8 KV cache on ROCm for V1 (#17870) 
Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>
 2025-05-11 15:58:45 +08:00
Kuntai Du 9112155283 [Perf] Use small max_num_batched_tokens for A100 (#17885) 
Signed-off-by: KuntaiDu <kuntai@uchicago.edu>
 2025-05-11 07:53:23 +00:00
Harry Mellor 4b2ed7926a Improve configs - the rest! (#17562) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-05-09 15:18:44 -07:00
vllmellmqli88Hongxia Yang
 		3c9396a64f [FEAT][ROCm]: Support AITER MLA on V1 Engine (#17523) 
Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com>
Co-authored-by: qli88 <qiang.li2@amd.com>
Co-authored-by: Hongxia Yang <62075498+hongxiayang@users.noreply.github.com>
 2025-05-09 10:42:05 +08:00
Vadim Markovtsev b2da14a05a Improve exception reporting in MP engine (#17800) 
Signed-off-by: Vadim Markovtsev <vadim@poolside.ai>
 2025-05-08 05:32:39 +00:00