iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
6,768 Commits 3 Branches 0 Tags
6881107948c00a8564bc2fa85308f6fc2f065d64
Commit Graph
91 Commits
Author SHA1 Message Date
Hosang dd5fa7e04f [ROCm][Kernel][V1] Enable AMD Radeon GPU Custom Paged Attention on v1 (#17004) 
Signed-off-by: Hosang Yoon <hosang.yoon@amd.com>
 2025-05-21 08:35:00 -07:00
Percy 980a172474 [Kernel] update comment for KV shape in unified triton attn (#18099) 
Signed-off-by: haochengxia <xhc_1007@163.com>
 2025-05-20 11:19:34 -07:00
Thomas ParnellandLucas Wilkinson 01c22335ba [Kernel] [V1] Fix performance regression for triton unified attention (#18161) 
Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>
Co-authored-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
 2025-05-15 06:39:00 -07:00
Thomas ParnellandLucas Wilkinson e6b8e65d2d [Bugfix] Fix fp8 tests for triton_unified_attention for Triton 3.3 (#18013) 
Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>
Co-authored-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
 2025-05-15 13:26:34 +08:00
qli88 4f8b373225 [BugFix][AMD] Compatible patch for AITER lib after 04/20 (#17912) 
Signed-off-by: Qiang Li <qiang.li2@amd.com>
 2025-05-13 23:05:20 -07:00
Gregory Shtrasberg 06c0922a69 [FP8][ROCm][Attention] Enable FP8 KV cache on ROCm for V1 (#17870) 
Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>
 2025-05-11 15:58:45 +08:00
Michael Goin 85b72cb7b1 Revert "[BugFix][AMD] Compatible patch for latest AITER(05/07/2025)" (#17910) 2025-05-09 08:58:18 -07:00
qli88 9f64e93415 [BugFix][AMD] Compatible patch for latest AITER(05/07/2025) (#17864) 
Signed-off-by: Qiang Li <qiang.li2@amd.com>
 2025-05-09 08:59:36 -06:00
vllmellmqli88Hongxia Yang
 		3c9396a64f [FEAT][ROCm]: Support AITER MLA on V1 Engine (#17523) 
Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com>
Co-authored-by: qli88 <qiang.li2@amd.com>
Co-authored-by: Hongxia Yang <62075498+hongxiayang@users.noreply.github.com>
 2025-05-09 10:42:05 +08:00
Agata Dobrzyniewicz 843b222723 [Hardware][Intel-Gaudi] Support Automatic Prefix Caching on HPU (#17648) 
Signed-off-by: Agata Dobrzyniewicz <adobrzyniewicz@habana.ai>
 2025-05-07 22:37:03 -07:00
Thomas ParnellandLucas Wilkinson 2f925e5777 [Kernel] Unified Triton kernel that doesn't distinguish between prefill + decode (#16828) 
Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Co-authored-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
 2025-05-06 18:21:48 -04:00
Mengqing Cao f9bc5a0693 [Bugfix] Fix triton import with local TritonPlaceholder (#17446) 
Signed-off-by: Mengqing Cao <cmq0113@163.com>
 2025-05-06 17:53:09 +08:00
Hui Liu 4c33d67321 [Bugfix] fix tmp_out and exp_sums dimensions (#17438) 
Signed-off-by: Hui Liu <96135754+hliuca@users.noreply.github.com>
 2025-05-02 16:44:07 +00:00
Hongxia Yang 28566d73b3 [ROCm] remove unsupported archs from rocm triton flash-attention supported list (#17536) 
Signed-off-by: Hongxia Yang <hongxia.yang@amd.com>
 2025-05-01 07:54:25 -07:00
Huy Do 2c4f59afc3 Update PyTorch to 2.7.0 (#16859) 2025-04-29 19:08:04 -07:00
rasmith 8e4b351a0c [Kernel][Triton][FP8] Adding fp8 and variable length sequence support to Triton FAv2 kernel (#12591) 
Signed-off-by: Randall Smith <Randall.Smith@amd.com>
 2025-04-27 00:35:08 +00:00
Agata Dobrzyniewicz c48334d405 [Hardware][Intel-Gaudi] Update hpu-extension and update bucketing system for HPU device (#17186) 
Signed-off-by: Agata Dobrzyniewicz <adobrzyniewicz@habana.ai>
 2025-04-26 05:55:14 -07:00
Aleksandr MalyshevSage MoorerootAleksandr Malyshevqli88root
 		bc7c4d206b [Kernel][ROCM] Upstream prefix prefill speed up for vLLM V1 (#13305) 
Signed-off-by: Sage Moore <sage@neuralmagic.com>
Signed-off-by: root <root@banff-cyxtera-s73-5.ctr.dcgpu>
Signed-off-by: Aleksandr Malyshev <maleksan@amd.com>
Signed-off-by: root <root@banff-cyxtera-s65-4.amd.com>
Signed-off-by: maleksan85 <maleksan@amd.com>
Signed-off-by: <>
Co-authored-by: Sage Moore <sage@neuralmagic.com>
Co-authored-by: root <root@banff-cyxtera-s73-5.ctr.dcgpu>
Co-authored-by: Aleksandr Malyshev <maleksan@amd.com>
Co-authored-by: qli88 <qiang.li2@amd.com>
Co-authored-by: root <root@banff-cyxtera-s65-4.amd.com>
 2025-04-22 19:11:56 -07:00
vllmellmandqli88 30bc3e0f66 [FEAT][ROCm]: Support AITER MLA (#15893) 
Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com>
Co-authored-by: qli88 <qiang.li2@amd.com>
 2025-04-22 09:31:13 -07:00
vllmellmandtjtanaa 0e237f0035 [FEAT][ROCm] Integrate Paged Attention Kernel from AITER (#15001) 
Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com>
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
Co-authored-by: tjtanaa <tunjian.tan@embeddedllm.com>
 2025-04-22 02:46:28 -07:00
DefTruth 280d62b8a2 [Kernel] Remove redundant Exp calculations (#16123) 
Signed-off-by: DefTruth <qiustudent_r@163.com>
 2025-04-15 12:58:37 +00:00
DefTruth ce4ddd2d1a [Misc] remove warning if triton>=3.2.0 (#16553) 
Signed-off-by: DefTruth <qiustudent_r@163.com>
 2025-04-14 02:39:47 -07:00
DefTruth e9528f6dc6 [Kernel] support merge_attn_states CUDA kernel, 3x speedup (#16173) 
Signed-off-by: DefTruth <qiustudent_r@163.com>
 2025-04-11 06:50:50 -06:00
yihong 04149cce27 [BugFix] fix some typos found by typos. (#16314) 
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
 2025-04-09 03:43:59 -07:00
Gregory Shtrasberg 40a36ccfeb [ROCm][Bugfix] Use platform specific FP8 dtype (#15717) 
Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>
 2025-04-04 09:40:20 -07:00
Liangfu Chen d2b58ca203 [Neuron][kernel] Fuse kv cache into a single tensor (#15911) 
Signed-off-by: Liangfu Chen <liangfc@amazon.com>
 2025-04-03 09:51:32 -07:00
Aleksandr MalyshevAleksandr Malyshevroot
 		e73ff24e31 [ROCM][KERNEL] Paged attention for V1 (#15720) 
Signed-off-by: Aleksandr Malyshev <maleksan@amd.com>
Signed-off-by: root <root@banff-cyxtera-s65-4.amd.com>
Co-authored-by: Aleksandr Malyshev <maleksan@amd.com>
Co-authored-by: root <root@banff-cyxtera-s65-4.amd.com>
 2025-04-02 19:48:00 -07:00
Lucas Wilkinson 33437bc6e7 [BugFix] Fix nightly MLA failure (FA2 + MLA chunked prefill, i.e. V1, producing bad results) (#15492) 
Signed-off-by: LucasWilkinson <lwilkinson@neuralmagic.com>
 2025-03-25 20:33:22 -07:00
Thomas ParnellJan van LunterenBurkhard RingleinChih-Chieh Yang
 		fb4c7f8ef0 [Kernel] [V1] Further optimizations to ROCm (Triton) Backend to better handle GQA. (#14431) 
Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>
Co-authored-by: Jan van Lunteren <jvl@zurich.ibm.com>
Co-authored-by: Burkhard Ringlein <ngl@zurich.ibm.com>
Co-authored-by: Chih-Chieh Yang <chih.chieh.yang@ibm.com>
 2025-03-13 20:42:27 -07:00
Li, JiangandIsotr0py ff47aab056 [CPU] Upgrade CPU backend to torch-2.6 (#13381) 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
Co-authored-by: Isotr0py <2037008807@qq.com>
 2025-03-12 10:41:13 +00:00
Liangfu Chen c91b64f749 [neuron] add reshape_and_cache (#14391) 2025-03-10 18:37:29 -07:00
Thomas ParnellBurkhard RingleinJan van Lunteren
 		e642ec962c Add authors to license header. (#14371) 
Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>
Co-authored-by: Burkhard Ringlein <ngl@zurich.ibm.com>
Co-authored-by: Jan van Lunteren <jvl@zurich.ibm.com>
 2025-03-06 08:43:09 -08:00
Thomas Parnell 6bd1dd9d26 [Kernel] [V1] Improved performance for V1 Triton (ROCm) backend (#14152) 2025-03-06 07:39:16 -08:00
qli88 8294773e48 [core] Perf improvement for DSv3 on AMD GPUs (#13718) 
Signed-off-by: qli88 <qiang.li2@amd.com>
 2025-02-27 22:14:30 +00:00
Lucas Wilkinson f95903909f [Kernel] FlashMLA integration (#13747) 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
 2025-02-27 10:35:08 +08:00
Sage Moore 558db8083c [V1][Kernel] Refactor the prefix_prefill kernel so that the caller no longer has to pass in the context lengths (#13095) 2025-02-22 05:25:41 -08:00
Lucas WilkinsonPatrick Hornsimon-moTyler Michael Smith
 		288cc6c234 [Attention] MLA with chunked prefill (#12639) 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Patrick Horn <patrick.horn@gmail.com>
Co-authored-by: simon-mo <xmo@berkeley.edu>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
 2025-02-21 15:30:12 -08:00
Lingfan Yu 33170081f1 [Neuron][Kernel] Vectorize KV cache load in FlashPagedAttention to maximize DMA bandwidth (#13245) 
Signed-off-by: Lingfan Yu <lingfany@amazon.com>
 2025-02-20 17:45:45 -08:00
Yu-ZhouandCody Yu d0a7a2769d [Hardware][Gaudi][Feature] Support Contiguous Cache Fetch (#12139) 
Signed-off-by: yuzhou <yuzhou@habana.ai>
Signed-off-by: zhouyu5 <yu.zhou@intel.com>
Co-authored-by: Cody Yu <hao.yu.cody@gmail.com>
 2025-02-18 19:40:19 -08:00
Sage Moore ba59b78a9c [ROCm][V1] Add intial ROCm support to V1 (#12790) 2025-02-13 22:21:50 -08:00
Lingfan Yu e92694b6fe [Neuron][Kernel] Support Longer Sequences in NKI-based Flash PagedAttention and Improve Efficiency (#12921) 
Signed-off-by: Lingfan Yu <lingfany@amazon.com>
 2025-02-11 21:12:37 -08:00
Aleksandr MalyshevandAleksandr Malyshev 64862d106e [ROCM][AMD][TRITON] Halving warps number for fw_prefill to reduce spilling (#12713) 
Signed-off-by: Aleksandr Malyshev <maleksan@amd.com>
Co-authored-by: Aleksandr Malyshev <maleksan@amd.com>
 2025-02-05 03:58:22 +00:00
Lucas Wilkinsonandsimon-mo 75e94309e8 [Perf] Mem align KV caches for CUDA devices (MLA perf improvement) (#12676) 
Signed-off-by: simon-mo <xmo@berkeley.edu>
Signed-off-by: Lucas Wilkinson <lcwilkins@redhat.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Co-authored-by: simon-mo <xmo@berkeley.edu>
 2025-02-04 18:22:24 -08:00
Russell Bryant 33e0602e59 [Misc] Fix improper placement of SPDX header in scripts (#12694) 
Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-02-03 11:16:59 -08:00
Russell Bryant e489ad7a21 [Misc] Add SPDX-License-Identifier headers to python source files (#12628) 
- **Add SPDX license headers to python source files**
- **Check for SPDX headers using pre-commit**

commit 9d7ef44c3cfb72ca4c32e1c677d99259d10d4745
Author: Russell Bryant <rbryant@redhat.com>
Date:   Fri Jan 31 14:18:24 2025 -0500

    Add SPDX license headers to python source files
    
This commit adds SPDX license headers to python source files as
recommended to
the project by the Linux Foundation. These headers provide a concise way
that is
both human and machine readable for communicating license information
for each
source file. It helps avoid any ambiguity about the license of the code
and can
    also be easily used by tools to help manage license compliance.
    
The Linux Foundation runs license scans against the codebase to help
ensure
    we are in compliance with the licenses of the code we use, including
dependencies. Having these headers in place helps that tool do its job.
    
    More information can be found on the SPDX site:
    
    - https://spdx.dev/learn/handling-license-info/
    
    Signed-off-by: Russell Bryant <rbryant@redhat.com>

commit 5a1cf1cb3b80759131c73f6a9dddebccac039dea
Author: Russell Bryant <rbryant@redhat.com>
Date:   Fri Jan 31 14:36:32 2025 -0500

    Check for SPDX headers using pre-commit
    
    Signed-off-by: Russell Bryant <rbryant@redhat.com>

---------

Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-02-02 11:58:18 -08:00
Lucas WilkinsonWoosuk Kwonsimon-moMichael GoinZhuohan LiTyler Michael SmithAlexander Matveevsimon-mo
 		cabaf4eff3 [Attention] MLA decode optimizations (#12528) 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Signed-off-by: simon-mo <xmo@berkeley.edu>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: simon-mo <simon.mo@hey.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
Co-authored-by: Tyler Michael Smith <tysmith@redhat.com>
Co-authored-by: Alexander Matveev <59768536+alexm-neuralmagic@users.noreply.github.com>
Co-authored-by: simon-mo <xmo@berkeley.edu>
 2025-01-30 23:49:37 -08:00
Mengqing Cao dd66fd2b01 [CI] fix pre-commit error (#12494) 
Signed-off-by: Mengqing Cao <cmq0113@163.com>
 2025-01-28 06:11:05 +00:00
Liangfu ChenandJiangfei Duan ddee88d0ff [Neuron][Kernel] NKI-based flash-attention kernel with paged KV cache (#11277) 
Signed-off-by: Liangfu Chen <liangfc@amazon.com>
Co-authored-by: Jiangfei Duan <jfduan@outlook.com>
 2025-01-27 17:31:16 -08:00
Harry Mellor 823ab79633 Update pre-commit hooks (#12475) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-01-27 17:23:08 -07:00
Gregory ShtrasbergandMicah Williamson e97f802b2d [FP8][Kernel] Dynamic kv cache scaling factors computation (#11906) 
Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>
Co-authored-by: Micah Williamson <micah.williamson@amd.com>
 2025-01-23 18:04:03 +00:00