iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 2 Packages Projects Releases Wiki Activity
6,768 Commits 3 Branches 0 Tags
6881107948c00a8564bc2fa85308f6fc2f065d64
Commit Graph
67 Commits
Author SHA1 Message Date
Andrew Sansom临景Bryce1010Nan2018Cyrus LeungDarkLight1337
 		cc2a77d7f1 [Core] [Bugfix] Add Input Embeddings (#15428) 
Signed-off-by: Andrew Sansom <andrew@protopia.ai>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: 临景 <linjing.yx@alibaba-inc.com>
Co-authored-by: Bryce1010 <bryceyx@gmail.com>
Co-authored-by: Nan2018 <nan@protopia.ai>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-05-02 01:06:39 -07:00
Chen Zhang 838cedade7 [Bugfix] Get a specific type of layer from forward context (#17222) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-04-27 00:58:05 -07:00
Shu Wang 9e96f56efb Allocate kv_cache with stride order (#16605) 
Signed-off-by: shuw <shuw@nvidia.com>
 2025-04-25 22:03:31 -07:00
Zhengyuan Su (苏政渊)and苏政渊 f961d7f6ef [BugFix] Pass in correct VLLM config in FlashInfer backend (#13207) (#16973) 
Signed-off-by: 苏政渊 <suzhengyuan@moonshot.cn>
Co-authored-by: 苏政渊 <suzhengyuan@moonshot.cn>
 2025-04-22 06:44:10 -07:00
Yong Hoon Shin 05a015d6a5 Add warning for Attention backends that do not support irope yet (#16212) 2025-04-08 03:59:26 +00:00
cjackal 51010a1807 [Misc] set single whitespace between log sentences (#13771) 
Signed-off-by: cjackal <44624812+cjackal@users.noreply.github.com>
 2025-02-25 10:26:12 +08:00
Russell Bryant e489ad7a21 [Misc] Add SPDX-License-Identifier headers to python source files (#12628) 
- **Add SPDX license headers to python source files**
- **Check for SPDX headers using pre-commit**

commit 9d7ef44c3cfb72ca4c32e1c677d99259d10d4745
Author: Russell Bryant <rbryant@redhat.com>
Date:   Fri Jan 31 14:18:24 2025 -0500

    Add SPDX license headers to python source files
    
This commit adds SPDX license headers to python source files as
recommended to
the project by the Linux Foundation. These headers provide a concise way
that is
both human and machine readable for communicating license information
for each
source file. It helps avoid any ambiguity about the license of the code
and can
    also be easily used by tools to help manage license compliance.
    
The Linux Foundation runs license scans against the codebase to help
ensure
    we are in compliance with the licenses of the code we use, including
dependencies. Having these headers in place helps that tool do its job.
    
    More information can be found on the SPDX site:
    
    - https://spdx.dev/learn/handling-license-info/
    
    Signed-off-by: Russell Bryant <rbryant@redhat.com>

commit 5a1cf1cb3b80759131c73f6a9dddebccac039dea
Author: Russell Bryant <rbryant@redhat.com>
Date:   Fri Jan 31 14:36:32 2025 -0500

    Check for SPDX headers using pre-commit
    
    Signed-off-by: Russell Bryant <rbryant@redhat.com>

---------

Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-02-02 11:58:18 -08:00
Bowen Wangandyoukaichao 2bc3fbba0c [FlashInfer] Upgrade to 0.2.0 (#11194) 
Signed-off-by: Bowen Wang <abmfy@icloud.com>
Signed-off-by: youkaichao <youkaichao@gmail.com>
Co-authored-by: youkaichao <youkaichao@gmail.com>
 2025-01-27 18:19:24 +00:00
Gregory ShtrasbergandMicah Williamson e97f802b2d [FP8][Kernel] Dynamic kv cache scaling factors computation (#11906) 
Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>
Co-authored-by: Micah Williamson <micah.williamson@amd.com>
 2025-01-23 18:04:03 +00:00
youkaichao 66818e5b63 [core] separate builder init and builder prepare for each batch (#12253) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2025-01-22 14:13:52 +08:00
wangxiyuan 86bfb6dba7 [Misc] Pass attention to impl backend (#12218) 
Signed-off-by: wangxiyuan <wangxiyuan1007@gmail.com>
 2025-01-20 23:25:28 +08:00
Elfie Guo 0794e7446e [Misc] Add multipstep chunked-prefill support for FlashInfer (#10467) 2025-01-15 12:47:49 +08:00
Chen Zhang e20c92bb61 [Kernel] Move attn_type to Attention.__init__() (#11690) 
Signed-off-by: Chen Zhang <zhangch99@outlook.com>
 2025-01-07 00:11:28 +08:00
youkaichao a4c4daf364 [misc] use out argument for flash attention (#10822) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2024-12-02 10:50:10 +00:00
youkaichao eebad39f26 [torch.compile] support all attention backends (#10558) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2024-11-22 14:04:42 -08:00
Pavani Majety 6c1208d083 [Core] Add Sliding Window Support with Flashinfer (#10462) 
Signed-off-by: Pavani Majety <pmajety@nvidia.com>
 2024-11-20 19:56:47 -08:00
Peter Salas 6c0b7f548d [Core][VLM] Add precise multi-modal placeholder tracking (#8346) 
Signed-off-by: Peter Salas <peter@fixie.ai>
 2024-11-01 16:21:10 -07:00
Pavani Majety 598b6d7b07 [Bugfix/Core] Flashinfer k_scale and v_scale (#9861) 2024-11-01 12:15:05 -07:00
youkaichao 96e0c9cbbd [torch.compile] directly register custom op (#9896) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2024-10-31 21:56:09 -07:00
Elfie Guo 9ff4511e43 [Misc] Add chunked-prefill support on FlashInfer. (#9781) 2024-10-30 09:33:53 -07:00
Thomas Parnell 496e991da8 [Doc] Consistent naming of attention backends (#9498) 
Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>
 2024-10-21 22:29:57 +08:00
Thomas ParnellChih-Chieh YangCody Yu
 		0c9a5258f9 [Kernel] Add env variable to force flashinfer backend to enable tensor cores (#9497) 
Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>
Co-authored-by: Chih-Chieh Yang <chih.chieh.yang@ibm.com>
Co-authored-by: Cody Yu <hao.yu.cody@gmail.com>
 2024-10-18 17:55:48 -07:00
Kuntai Du 81ede99ca4 [Core] Deprecating block manager v1 and make block manager v2 default (#8704) 
Removing the block manager v1. This is the initial piece of prefix-caching-centric design. In order to achieve prefix-caching-centric design, we need to simplify the code path so that we only use v2 block manager (which has much higher performance on prefix caching).
 2024-10-17 11:38:15 -05:00
youkaichao f4dd830e09 [core] use forward context for flash infer (#9097) 2024-10-05 19:37:31 -07:00
youkaichao 9aaf14c62e [misc] add forward context for attention (#9029) 2024-10-03 12:09:42 -07:00
Lily Liu 1570203864 [Spec Decode] (1/2) Remove batch expansion (#8839) 2024-10-01 16:04:42 -07:00
Varun Sundar RabindranathandVarun Sundar Rabindranath c2ec430ab5 [Core] Multi-Step + Single Step Prefills via Chunked Prefill code path (#8378) 
Co-authored-by: Varun Sundar Rabindranath <varun@neuralmagic.com>
 2024-09-27 13:32:07 -07:00
youkaichao a9b15c606f [torch.compile] use empty tensor instead of None for profiling (#8875) 2024-09-27 08:11:32 -07:00
sroy745 1009e93c5d [Encoder decoder] Add cuda graph support during decoding for encoder-decoder models (#7631) 2024-09-17 07:35:01 -07:00
Alexander Matveev 019877253b [Bugfix] multi-step + flashinfer: ensure cuda graph compatible (#8427) 2024-09-12 21:01:50 +00:00
William Lin a6c0f3658d [multi-step] add flashinfer backend (#7928) 2024-09-12 11:16:22 -07:00
Elfie Guo e39ebf5cf5 [Core/Bugfix] Add query dtype as per FlashInfer API requirements. (#8173) 2024-09-05 05:12:26 +00:00
Pavani Majety 622f8abff8 [Bugfix] bugfix and add model test for flashinfer fp8 kv cache. (#8013) 2024-08-30 22:18:50 -07:00
Pavani MajetySimon MoCody Yu
 		6b3421567d [Core][Kernels] Enable FP8 KV Cache with Flashinfer backend. + BugFix for kv_cache_dtype=auto (#7985) 
Co-authored-by: Simon Mo <simon.mo@hey.com>
Co-authored-by: Cody Yu <hao.yu.cody@gmail.com>
 2024-08-29 14:53:11 -04:00
youkaichao ef99a78760 Revert "[Core][Kernels] Use FlashInfer backend for FP8 KV Cache when available." (#7982) 2024-08-28 21:27:06 -07:00
Pavani MajetyandSimon Mo b98cc28f91 [Core][Kernels] Use FlashInfer backend for FP8 KV Cache when available. (#7798) 
Co-authored-by: Simon Mo <simon.mo@hey.com>
 2024-08-28 10:01:22 -07:00
Cody Yu 9606c7197d Revert #7509 (#7887) 2024-08-27 00:16:31 -07:00
LI MOU 53328d7536 [BUG] fix crash on flashinfer backend with cudagraph disabled, when attention group_size not in [1,2,4,8] (#7509) 2024-08-21 08:54:31 -07:00
Antoni Baum 3b682179dd [Core] Add AttentionState abstraction (#7663) 2024-08-20 18:50:45 +00:00
youkaichao 54bd9a03c4 register custom op for flash attn and use from torch.ops (#7536) 2024-08-15 22:38:56 -07:00
Lily Liu ec2affa8ae [Kernel] Flashinfer correctness fix for v0.1.3 (#7319) 2024-08-12 07:59:17 +00:00
Lily Liu e53dfd3eaf [Kernel] Fix Flashinfer Correctness (#7284) 2024-08-07 16:26:52 -07:00
Cody Yu ef527be06c [MISC] Use non-blocking transfer in prepare_input (#7172) 2024-08-05 23:41:27 +00:00
Lily Liu 954f7305a1 [Kernel] Fix input for flashinfer prefill wrapper. (#7008) 2024-08-01 18:44:16 -07:00
Woosuk Kwon 805a8a75f2 [Misc] Support attention logits soft-capping with flash-attn (#7022) 2024-08-01 13:14:37 -07:00
Cody Yu 309aaef825 [Bugfix] Fix decode tokens w. CUDA graph (#6757) 2024-07-24 22:33:56 -07:00
Antoni Baum 5448f67635 [Core] Tweaks to model runner/input builder developer APIs (#6712) 2024-07-24 12:17:12 -07:00
Antoni Baum 0e63494cf3 Add fp8 support to reshape_and_cache_flash (#6667) 2024-07-24 18:36:52 +00:00
Cody Yu e0c15758b8 [Core] Modulize prepare input and attention metadata builder (#6596) 2024-07-23 00:45:24 +00:00
Cyrus Leung 9042d68362 [Misc] Consolidate and optimize logic for building padded tensors (#6541) 2024-07-20 04:17:24 +00:00