iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity
7,507 Commits 3 Branches 0 Tags
1fd471e957526a34a0cb4b60d2e830cd6ca79fdc
Commit Graph
740 Commits
Author SHA1 Message Date
Isotr0py 32c9be2200 [v1] Re-add fp32 support to v1 engine through FlexAttention (#19754) 
Signed-off-by: Isotr0py <2037008807@qq.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2025-07-05 09:41:10 +00:00
Cyrus Leung b024a42e93 [Core] Move multimodal placeholder from chat utils to model definition (#20355) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-07-03 08:18:30 +00:00
Nick Hill 657f2f301a [DP] Support external DP Load Balancer mode (#19790) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-07-02 10:21:52 -07:00
Chenheli Hua 2e7cbf2d7d [Frontend] Support configurable mm placeholder strings & flexible video sampling policies via CLI flags. (#20105) 
Signed-off-by: Chenheli Hua <huachenheli@outlook.com>
 2025-07-01 23:34:03 -07:00
Luka Govedič 6d42ce8315 [CLI] Improve CLI arg parsing for -O/--compilation-config (#20156) 
Signed-off-by: luka <luka@neuralmagic.com>
 2025-07-01 01:03:13 +00:00
Chales Xu e53be6f00a [Misc] Add type assertion of request_id for LLMEngine.add_request (#19700) 
Signed-off-by: n2ptr <xuzhanchaomail@163.com>
 2025-06-27 22:47:36 -07:00
Bowen Wang e9fd658a73 [Feature] Expert Parallelism Load Balancer (EPLB) (#18343) 
Signed-off-by: Bowen Wang <abmfy@icloud.com>
 2025-06-26 15:30:21 -07:00
amit 981eeca41a [Fix][V1] Remove --scheduling-policy oracle (#20010) 
Signed-off-by: amit <amit.man@gmail.com>
 2025-06-24 09:52:15 -07:00
Aaron Pham c4cf260677 [Perf][CLI] Improve overall startup time (#19941) 2025-06-22 23:11:22 +00:00
Li, Jiang 79f2f1c2a1 [CPU][CI] Fallback sliding window to v0 and fix CPU pooling model tests (#19901) 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
 2025-06-20 15:30:36 +00:00
Alex Brooks ead2110297 [Core][Bugfix] Fix Online MM Beam Search (#19688) 
Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
 2025-06-19 17:18:07 +00:00
Maximilien de Bayserand22quinn 799397ee4f Support embedding models in V1 (#16188) 
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
Signed-off-by: Max de Bayser <maxdebayser@gmail.com>
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
Co-authored-by: 22quinn <33176974+22quinn@users.noreply.github.com>
 2025-06-18 21:36:33 -07:00
Chen Zhang a89209b78d [v1] Support mamba2 (#19327) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-06-18 20:34:15 +00:00
wangxiyuan 257ab95439 [Platform] Allow platform use V1 Engine by default (#19792) 
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
 2025-06-18 13:03:36 +00:00
Ning Xie 6e9cc73f67 [MISC] correct DeviceConfig device field static type analysis (#19699) 
Signed-off-by: Andy Xie <andy.xning@gmail.com>
 2025-06-17 17:21:50 -07:00
Li, Jiang 6458721108 [CPU] Refine default config for the CPU backend (#19539) 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
 2025-06-13 13:27:39 +08:00
rasmith c7ea0b56cd [AMD] [Quantization] Add override flag for attention dtype instead of using kv_cache_dtype trigger (#17331) 
Signed-off-by: Randall Smith <Randall.Smith@amd.com>
 2025-06-11 15:53:28 -04:00
Cyrus Leung 68b4a26149 [Doc] Update V1 User Guide for Hardware and Models (#19474) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-06-11 00:49:06 -07:00
Gregory Shtrasberg 5241ca50d6 [ROCm][V1] Adding ROCm to the list of plaforms using V1 by default (#19440) 
Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>
 2025-06-10 22:06:15 +00:00
Isotr0py 5f1ac1e1d1 Revert "[v1] Add fp32 support to v1 engine through flex attn" (#19404) 2025-06-10 01:30:20 -07:00
22quinn c1c7dbbeeb [Bugfix][Core] Prevent token lengths exceeding max_model_len in V0 (#19348) 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
 2025-06-09 23:01:29 +08:00
Isotr0py b8089195b4 [v1] Add fp32 support to v1 engine through flex attn (#19319) 
Signed-off-by: Isotr0py <2037008807@qq.com>
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
 2025-06-09 22:10:44 +08:00
Cyrus Leung e31ae3de36 [Deprecation] Remove inputs arg fallback in Engine classes (#18799) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-06-09 10:19:56 +08:00
Driss Guessous cf02f9b283 Add FlexAttention to V1 (#16078) 
Signed-off-by: drisspg <drisspguessous@gmail.com>
 2025-06-06 21:58:55 -07:00
jmswen 7353492a47 [Core] Raise when non-multi-instance DP clients target a DP rank (#19227) 
Signed-off-by: Jon Swenson <jmswen@gmail.com>
 2025-06-06 19:03:01 +08:00
Chen Zhang f8a1a2d108 [v1] Hybrid Memory Allocator (#17996) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-06-05 20:47:09 -07:00
Kebe ef3f98b59f [Bugfix] fix v1 cpu worker fails on macOS (#19121) 2025-06-04 20:17:38 +00:00
jmswen c8dcc15921 Allow AsyncLLMEngine.generate to target a specific DP rank (#19102) 
Signed-off-by: Jon Swenson <jmswen@gmail.com>
 2025-06-04 08:26:47 -07:00
Kaixi Hou 41aa578428 [NVIDIA] Add Cutlass MLA backend (#17625) 2025-06-03 21:40:26 -07:00
Li, Jiang 4555143ea7 [CPU] V1 support for the CPU backend (#16441) 2025-06-03 18:43:01 -07:00
Harry MellorandYikun Jiang 6865fe0074 Fix interaction between Optional and Annotated in CLI typing (#19093) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
Co-authored-by: Yikun Jiang <yikun@apache.org>
 2025-06-03 21:07:19 +00:00
Simon Mo 02f0c7b220 [Misc] Add SPDX-FileCopyrightText (#19100) 
Signed-off-by: simon-mo <simon.mo@hey.com>
 2025-06-03 11:20:17 -07:00
Reidandreidliu41 17430e3653 [bugfix] small fix logic issue (#18999) 
Signed-off-by: reidliu41 <reid201711@gmail.com>
Co-authored-by: reidliu41 <reid201711@gmail.com>
 2025-06-03 05:35:12 +00:00
Rui Qiao bdce64f236 [V1] Support DP with Ray (#18779) 2025-06-02 21:15:13 -07:00
jennyyyyzhenyZhenyzhen
 		ebb1ec9318 [Model] enable data parallel for Llama4 vision encoder (#18368) 
Signed-off-by: yzhen <yzhen@devgpu093.cco2.facebook.com>
Co-authored-by: yZhen <yZhen@fb.com>
Co-authored-by: yzhen <yzhen@devgpu093.cco2.facebook.com>
 2025-06-02 19:22:54 +08:00
Michael Goin 2ad6194a02 Let max_num_batched_tokens use human_readable_int for large numbers (#18968) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-06-01 11:41:29 +08:00
Michael Goin d54af615d5 [Bugfix] Fix PP default fallback behavior for V1 (#18915) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-05-30 10:13:17 +08:00
Mark McLoughlin 0e98964e94 [V1][Metrics] Remove metrics that were deprecated in 0.8 (#18837) 
Signed-off-by: Mark McLoughlin <markmc@redhat.com>
 2025-05-28 18:54:12 +00:00
Alex Brooks 321331b8ae [Core] Add Lora Support to Beam Search (#18346) 
Signed-off-by: Alex-Brooks <Alex.Brooks@ibm.com>
 2025-05-28 08:58:24 -07:00
Harry Mellor 4c2b38ce9e Enable Pydantic mypy checks and convert configs to Pydantic dataclasses (#17599) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-05-28 12:46:04 +00:00
Cyrus Leung 0f0926b43f [Deprecation] Remove unused sync methods in async_timeout (#18792) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-05-28 15:08:48 +08:00
Hyogeun Oh (오효근) a68e293cb9 [Doc] Convert Sphinx directives ( {class}, {meth}, {attr}, ...) to MkDocs format for better documentation linking (#18663) 
Signed-off-by: Zerohertz <ohg3417@gmail.com>
 2025-05-27 01:44:20 -07:00
Isotr0py 1f1b1bc03b [V1][Quantization] Add CUDA graph compatible v1 GGUF support (#18646) 
Signed-off-by: Isotr0py <mozf@mail2.sysu.edu.cn>
Signed-off-by: Isotr0py <2037008807@qq.com>
 2025-05-27 04:40:28 +00:00
Yuqi ZhangandCyrus Leung f2faac745d [Bugfix] Fix cpu usage and cache hit stats reporting on cpu environment (#18674) 
Signed-off-by: zzzyq <zhangyuqi94@gmail.com>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2025-05-25 02:36:06 -07:00
Feng XiaoLongandCrucifixion-Fxl 4fc1bf813a [Bugfix] Migrate to REGEX Library to prevent catastrophic backtracking (#18454) 
Signed-off-by: Crucifixion-Fxl <xmufxl@gmail.com>
Co-authored-by: Crucifixion-Fxl <xmufxl@gmail.com>
 2025-05-23 16:16:26 -07:00
Cyrus Leung 7d9216495c [Doc] Update references to doc files (#18637) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-05-23 15:49:21 -07:00
Jiayi YaoandRui Qiao 2628a69e35 [V1] Support Deepseek MTP (#18435) 
Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
Signed-off-by: YaoJiayi <120040070@link.cuhk.edu.cn>
Co-authored-by: Rui Qiao <ruisearch42@gmail.com>
 2025-05-23 10:26:28 -07:00
Cyrus Leung 273cb3b4d9 [Doc] Fix top-level API links/docs (#18621) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-05-23 09:46:56 -07:00
Harry Mellor a1fe24d961 Migrate docs from Sphinx to MkDocs (#18145) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-05-23 02:09:53 -07:00
Sanger Steel c32e249a23 [Frontend] [Core] Add Tensorizer support for V1, LoRA adapter serialization and deserialization (#17926) 
Signed-off-by: Sanger Steel <sangersteel@gmail.com>
 2025-05-22 18:44:18 -07:00