iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity
8,621 Commits 3 Branches 0 Tags
070da660c1bf9e7a7be8b9efeff4b06f91c7342f
Commit Graph
78 Commits
Author SHA1 Message Date
fhl2000Luka GovedičLucas WilkinsonLucas Wilkinson
 		74f441f4b5 [Core] Allow full cudagraph with separate attention routines and orthogonal to compilation, add support for FA2 and FlashInfer (#20059) 
Signed-off-by: fhl <2410591650@qq.com>
Signed-off-by: fhl2000 <63384265+fhl2000@users.noreply.github.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
Co-authored-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Lucas Wilkinson <LucasWilkinson@users.noreply.github.com>
 2025-08-15 10:01:39 -04:00
nvjullinandmgoin 279a5f31b3 [Kernel] Add nvfp4 gemm flashinfer backends (#22346) 
Signed-off-by: Julien Lin <jullin@nvidia.com>
Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: mgoin <mgoin64@gmail.com>
 2025-08-14 16:03:55 -04:00
Varun Sundar RabindranathandVarun Sundar Rabindranath f703b923f3 [Misc] DeepGEMM : Avoid JIT generation in the hot-path (#22215) 
Signed-off-by: Varun Sundar Rabindranath <vsundarr@redhat.com>
Co-authored-by: Varun Sundar Rabindranath <vsundarr@redhat.com>
 2025-08-08 16:09:59 -07:00
David Ben-DavidandDavid Ben-David aefeea0fde [V1] [P/D] Refactor KV Connector Path (#21980) 
Signed-off-by: David Ben-David <davidb@pliops.com>
Co-authored-by: David Ben-David <davidb@pliops.com>
 2025-08-03 04:03:40 -07:00
fhl2000 23322431c8 [V1][CUDA] Full cudagraph support for FlashInfer (#21367) 2025-08-01 21:49:34 -04:00
Csrayz b917da442b Expose PyTorch profiler configuration to environment variables (#21803) 
Signed-off-by: Csrayz <33659823+Csrayz@users.noreply.github.com>
 2025-07-29 19:46:31 -07:00
Kuntai Du b18b417fbf Revert "[V1] Exception Handling when Loading KV Cache from Remote Store" (#21778) 
Signed-off-by: KuntaiDu <kuntai@uchicago.edu>
 2025-07-28 20:15:18 +00:00
Adelineandliuyumoye 15a72ac478 [V1] Exception Handling when Loading KV Cache from Remote Store (#21534) 
Signed-off-by: liuyumoye <adeline_ly2023@outlook.com>
Co-authored-by: liuyumoye <adeline_ly2023@outlook.com>
 2025-07-27 20:34:17 -07:00
Ye (Charlotte) Qi a40a8506df [Misc] Improve memory profiling debug message (#21429) 
Signed-off-by: Ye (Charlotte) Qi <yeq@meta.com>
 2025-07-26 07:07:21 -07:00
Cyrus Leung 46d81d6951 [V1] Get supported tasks from model runner instead of model config (#21585) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-07-25 05:36:45 -07:00
Nick Hill eec6942014 [BugFix] Fix KVConnector TP worker aggregation (#21473) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-07-23 20:56:49 -07:00
22quinn 5c9b807b34 [Core] Add reload_weights RPC method (#20096) 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
 2025-07-23 14:24:52 -07:00
kourosh hakhamaneshi 9f414a12ad [BugFix] Make PD work with Ray (#21072) 
Signed-off-by: Kourosh Hakhamaneshi <kourosh@anyscale.com>
 2025-07-19 08:46:50 -07:00
Rui Qiao 217937221b Elastic Expert Parallel Initial Support (#20775) 
Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
 2025-07-18 17:46:09 -07:00
Cyrus Leung 45badd05d0 [Core] Set pooling params based on task and model (#21128) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-07-18 05:41:17 -07:00
22quinn 8632e831ba [Core] Add update_config RPC method (#20095) 
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
 2025-07-14 00:49:18 +00:00
Nick HillandDavid Ben-David 574ad60db9 [KVConnector] Always call connector clear_metadata() at end of step (#20756) 
Signed-off-by: Nick Hill <nhill@redhat.com>
Co-authored-by: David Ben-David <sdavidbd@gmail.com>
 2025-07-10 22:37:27 +01:00
Or Ozeri cc876d0f29 [KVConnector] Aggregate finished requests on the scheduler (#19555) 
Signed-off-by: Or Ozeri <oro@il.ibm.com>
 2025-07-10 09:22:18 +01:00
Nick Hill 59389c927b [BugFix][CPU] Fix CPU worker dependency on cumem_allocator (#20696) 
Signed-off-by: Nick Hill <nhill@redhat.com>
 2025-07-10 14:24:20 +08:00
Kunshang Ji 0b407479ef [misc]refactor Platform.set_device method (#20262) 
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
 2025-07-09 01:39:47 +00:00
Yang YangandKunshang Ji 6e2c19ce22 [Refactor]Abstract Platform Interface for Distributed Backend and Add xccl Support for Intel XPU (#19410) 
Signed-off-by: dbyoung18 <yang5.yang@intel.com>
Signed-off-by: Kunshang Ji <kunshang.ji@intel.com>
Co-authored-by: Kunshang Ji <kunshang.ji@intel.com>
 2025-07-07 04:32:32 +00:00
Bowen Wang e9fd658a73 [Feature] Expert Parallelism Load Balancer (EPLB) (#18343) 
Signed-off-by: Bowen Wang <abmfy@icloud.com>
 2025-06-26 15:30:21 -07:00
Maximilien de Bayserand22quinn 799397ee4f Support embedding models in V1 (#16188) 
Signed-off-by: Max de Bayser <mbayser@br.ibm.com>
Signed-off-by: Max de Bayser <maxdebayser@gmail.com>
Signed-off-by: 22quinn <33176974+22quinn@users.noreply.github.com>
Co-authored-by: 22quinn <33176974+22quinn@users.noreply.github.com>
 2025-06-18 21:36:33 -07:00
Isotr0py 1173804dca [Bugfix] Fix TP inference for Flex attention backend (#19657) 
Signed-off-by: Isotr0py <2037008807@qq.com>
 2025-06-16 11:21:37 +00:00
Ye (Charlotte) Qi cc867be19c [V1] Reuse V0's memory_profiling util for gpu worker memory profiling (#19312) 
Signed-off-by: Ye (Charlotte) Qi <yeq@meta.com>
 2025-06-10 08:40:01 +08:00
Luka Govedič 2d8476e465 [BugFix][V1] Fix memory profiling bug (#18974) 
Signed-off-by: luka <luka@neuralmagic.com>
 2025-06-07 10:34:51 -07:00
Li, Jiang 4555143ea7 [CPU] V1 support for the CPU backend (#16441) 2025-06-03 18:43:01 -07:00
Simon Mo 02f0c7b220 [Misc] Add SPDX-FileCopyrightText (#19100) 
Signed-off-by: simon-mo <simon.mo@hey.com>
 2025-06-03 11:20:17 -07:00
Divakar Verma 774c5fde30 [V1] fix torch profiling for V1 offline scenarios (#18445) 
Signed-off-by: Divakar Verma <divakar.verma@amd.com>
 2025-05-28 04:16:30 +00:00
youkaichao 6a7988c55b Refactor pplx init logic to make it modular (prepare for deepep) (#18200) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2025-05-23 23:43:43 +08:00
Harry Mellor a1fe24d961 Migrate docs from Sphinx to MkDocs (#18145) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-05-23 02:09:53 -07:00
Sanger Steel c32e249a23 [Frontend] [Core] Add Tensorizer support for V1, LoRA adapter serialization and deserialization (#17926) 
Signed-off-by: Sanger Steel <sangersteel@gmail.com>
 2025-05-22 18:44:18 -07:00
Lucia Fang 3d2779c29a [Feature] Support Pipeline Parallism in torchrun SPMD offline inference for V1 (#17827) 
Signed-off-by: Lucia Fang <fanglu@fb.com>
 2025-05-15 22:28:27 -07:00
bnellnm f9c069c85e Modularize fused experts and integrate PPLX kernels (#15956) 2025-05-14 13:11:54 -07:00
Jee Jee Li 822de7fb94 [Misc] Split model loader (#17712) 
Signed-off-by: Jee Jee Li <pandaleefree@gmail.com>
 2025-05-07 12:42:26 +08:00
Li, Jiang a6fed02068 [V1][PP] Support PP for MultiprocExecutor (#14219) 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
Signed-off-by: jiang.li <jiang1.li@intel.com>
 2025-05-06 07:58:05 -07:00
Harry Mellor d6484ef3c3 Add full API docs and improve the UX of navigating them (#17485) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-05-03 19:42:43 -07:00
Daniel Li 48cb2109b6 [V1] Move usage stats to worker and start logging TPU hardware (#16211) 2025-04-25 14:06:01 -06:00
Han Zhang d41faaf9df Restore buffers when wake up from level 2 sleep (#16564) (#16889) 
Signed-off-by: Han <zh950713@gmail.com>
 2025-04-21 20:18:28 +08:00
Yihua Chengrshaw@neuralmagic.com <robertgshaw2@gmail.com>Robert ShawRémi DelacourtTyler Michael Smith
 		3408e47159 [P/D][V1] KV Connector API V1 (#15960) 
Signed-off-by: ApostaC <yihua98@uchicago.edu>
Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>
Signed-off-by: remi <remi@mistral.ai>
Co-authored-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Rémi Delacourt <54138269+Flechman@users.noreply.github.com>
Co-authored-by: Tyler Michael Smith <tysmith@redhat.com>
 2025-04-17 13:22:40 -07:00
wwl2755 463bbb1835 [Bugfix][V1] Fix bug from putting llm_engine.model_executor in a background process (#15367) 
Signed-off-by: wwl2755 <wangwenlong2755@gmail.com>
 2025-04-03 07:32:10 +00:00
Eric Tang ddb94c2605 [core] Add tags parameter to wake_up() (#15500) 
Signed-off-by: Eric <erictang000@gmail.com>
 2025-04-02 01:59:27 -07:00
Chen Zhang 93a00d7dde [v1] Refactor KVCacheConfig (#14079) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-03-21 04:56:27 -07:00
Woosuk KwonCody YuNick Hill
 		0c6f5023c3 [V1] Scheduler Refactoring [1/N] - Add Scheduler Interface (#15250) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: Cody Yu <hao.yu.cody@gmail.com>
Co-authored-by: Nick Hill <nhill@redhat.com>
 2025-03-20 17:50:43 -07:00
Roger WangandSimon Mo ad19c8a003 [V1] Move OOM check into sampler run (#14728) 
Signed-off-by: Roger Wang <ywang@roblox.com>
Co-authored-by: Simon Mo <simon.mo@hey.com>
 2025-03-13 20:40:23 -07:00
Cody Yu b706d898af [Bugfix][V1][PP] Only warmup sampler at last PP rank (#14643) 
Signed-off-by: Cody Yu <hao.yu.cody@gmail.com>
 2025-03-11 23:40:07 +00:00
Roger WangandVarun Sundar Rabindranath 1fc973c0b5 [V1][Core] Fix memory issue with logits & sampling (#14508) 
Signed-off-by: Roger Wang <ywang@roblox.com>
Co-authored-by: Varun Sundar Rabindranath <3337719+varun-sundar-rabindranath@users.noreply.github.com>
 2025-03-11 04:03:41 +00:00
Robert ShawandRoger Wang 5f0b53c6ea Revert "[V1][Core] Fix memory issue with logits & sampling" (#14504) 
Signed-off-by: Roger Wang <ywang@roblox.com>
Co-authored-by: Roger Wang <ywang@roblox.com>
 2025-03-08 17:43:37 -08:00
Roger Wang 8d5aa466fb [V1][Core] Fix memory issue with logits & sampling (#13776) 
Signed-off-by: Roger Wang <ywang@roblox.com>
 2025-03-08 06:11:04 -08:00
Harry Mellor cf069aa8aa Update deprecated Python 3.8 typing (#13971) 2025-03-02 17:34:51 -08:00