iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
6,768 Commits 3 Branches 0 Tags
6881107948c00a8564bc2fa85308f6fc2f065d64
Commit Graph
34 Commits
Author SHA1 Message Date
Luka Govedič 176a95c670 [Fix] Support CUDAGraph capture for encoder-decoder on ROCm (#18104) 
Signed-off-by: Luka Govedič <lgovedic@redhat.com>
 2025-05-13 19:31:42 -07:00
Harry Mellor d6484ef3c3 Add full API docs and improve the UX of navigating them (#17485) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-05-03 19:42:43 -07:00
Lucas Wilkinson 183dad7a85 [Attention] Update to lastest FA3 code (#13111) 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
 2025-04-17 15:14:07 -07:00
Mickaël Seznec a597a57595 [Attention] Flash Attention 3 - fp8 (#14570) 
Signed-off-by: Mickael Seznec <mickael@mistral.ai>
 2025-03-20 01:14:20 -04:00
Pavani Majety ed6ea06577 [Hardware] Update the flash attn tag to support Blackwell (#14244) 2025-03-05 22:01:37 -08:00
Andy Lo 322d2a27d6 [BugFix] Minor: logger import in attention backend (#13706) 
Signed-off-by: Andy Lo <andy@mistral.ai>
 2025-02-22 16:51:13 -08:00
youkaichao fe743b798d [bugfix] fix early import of flash attention (#12959) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2025-02-09 00:06:56 +08:00
Isotr0py 8108ac841d [Bugfix] Fix unsupported FA version check for Turing GPU (#12828) 2025-02-06 09:18:22 -08:00
Lucas Wilkinson c786e757fa [Attention] Use FA3 for MLA on Hopper (#12807) 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
 2025-02-06 11:43:12 +00:00
Russell Bryant e489ad7a21 [Misc] Add SPDX-License-Identifier headers to python source files (#12628) 
- **Add SPDX license headers to python source files**
- **Check for SPDX headers using pre-commit**

commit 9d7ef44c3cfb72ca4c32e1c677d99259d10d4745
Author: Russell Bryant <rbryant@redhat.com>
Date:   Fri Jan 31 14:18:24 2025 -0500

    Add SPDX license headers to python source files
    
This commit adds SPDX license headers to python source files as
recommended to
the project by the Linux Foundation. These headers provide a concise way
that is
both human and machine readable for communicating license information
for each
source file. It helps avoid any ambiguity about the license of the code
and can
    also be easily used by tools to help manage license compliance.
    
The Linux Foundation runs license scans against the codebase to help
ensure
    we are in compliance with the licenses of the code we use, including
dependencies. Having these headers in place helps that tool do its job.
    
    More information can be found on the SPDX site:
    
    - https://spdx.dev/learn/handling-license-info/
    
    Signed-off-by: Russell Bryant <rbryant@redhat.com>

commit 5a1cf1cb3b80759131c73f6a9dddebccac039dea
Author: Russell Bryant <rbryant@redhat.com>
Date:   Fri Jan 31 14:36:32 2025 -0500

    Check for SPDX headers using pre-commit
    
    Signed-off-by: Russell Bryant <rbryant@redhat.com>

---------

Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-02-02 11:58:18 -08:00
Lucas WilkinsonWoosuk Kwonsimon-moMichael GoinZhuohan LiTyler Michael SmithAlexander Matveevsimon-mo
 		cabaf4eff3 [Attention] MLA decode optimizations (#12528) 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Signed-off-by: simon-mo <xmo@berkeley.edu>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: simon-mo <simon.mo@hey.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
Co-authored-by: Tyler Michael Smith <tysmith@redhat.com>
Co-authored-by: Alexander Matveev <59768536+alexm-neuralmagic@users.noreply.github.com>
Co-authored-by: simon-mo <xmo@berkeley.edu>
 2025-01-30 23:49:37 -08:00
Gregory ShtrasbergandMicah Williamson e97f802b2d [FP8][Kernel] Dynamic kv cache scaling factors computation (#11906) 
Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>
Co-authored-by: Micah Williamson <micah.williamson@amd.com>
 2025-01-23 18:04:03 +00:00
youkaichao 66818e5b63 [core] separate builder init and builder prepare for each batch (#12253) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2025-01-22 14:13:52 +08:00
youkaichao eebad39f26 [torch.compile] support all attention backends (#10558) 
Signed-off-by: youkaichao <youkaichao@gmail.com>
 2024-11-22 14:04:42 -08:00
Yang ZhengandYang Zheng 4dbcbbeb09 [Misc] Compute query_start_loc/seq_start_loc on CPU (#9447) 
Co-authored-by: Yang Zheng(SW)(Alex) <you@example.com>
 2024-11-04 08:54:37 +00:00
sroy745 a78dd3303e [Encoder Decoder] Add flash_attn kernel support for encoder-decoder models (#9559) 2024-11-01 23:22:49 -07:00
Peter Salas 6c0b7f548d [Core][VLM] Add precise multi-modal placeholder tracking (#8346) 
Signed-off-by: Peter Salas <peter@fixie.ai>
 2024-11-01 16:21:10 -07:00
sasha0552 55650c83a0 [Bugfix] Fix illegal memory access error with chunked prefill, prefix caching, block manager v2 and xformers enabled together (#9532) 
Signed-off-by: sasha0552 <admin@sasha0552.org>
 2024-10-31 11:46:36 -07:00
Thomas Parnell 496e991da8 [Doc] Consistent naming of attention backends (#9498) 
Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>
 2024-10-21 22:29:57 +08:00
Kuntai Du 81ede99ca4 [Core] Deprecating block manager v1 and make block manager v2 default (#8704) 
Removing the block manager v1. This is the initial piece of prefix-caching-centric design. In order to achieve prefix-caching-centric design, we need to simplify the code path so that we only use v2 block manager (which has much higher performance on prefix caching).
 2024-10-17 11:38:15 -05:00
Lily Liu 89feb4c84d [SpecDec] Remove Batch Expansion (2/3) (#9298) 2024-10-12 05:13:37 +00:00
Lily Liu 1570203864 [Spec Decode] (1/2) Remove batch expansion (#8839) 2024-10-01 16:04:42 -07:00
Aaron PhamandCyrus Leung 9d104b5beb [CI/Build] Update Ruff version (#8469) 
Signed-off-by: Aaron Pham <contact@aarnphm.xyz>
Co-authored-by: Cyrus Leung <cyrus.tl.leung@gmail.com>
 2024-09-18 11:00:56 +00:00
sroy745 1009e93c5d [Encoder decoder] Add cuda graph support during decoding for encoder-decoder models (#7631) 2024-09-17 07:35:01 -07:00
Antoni Baum 3b682179dd [Core] Add AttentionState abstraction (#7663) 2024-08-20 18:50:45 +00:00
Antoni Baum 999ef0b917 [Misc] Add numpy implementation of compute_slot_mapping (#7377) 2024-08-09 22:52:29 +00:00
Alexander Matveev e02ac55617 [Performance] Optimize e2e overheads: Reduce python allocations (#7162) 2024-08-08 21:34:28 -07:00
Cody Yu ef527be06c [MISC] Use non-blocking transfer in prepare_input (#7172) 2024-08-05 23:41:27 +00:00
Woosuk Kwon 805a8a75f2 [Misc] Support attention logits soft-capping with flash-attn (#7022) 2024-08-01 13:14:37 -07:00
Cody Yu 309aaef825 [Bugfix] Fix decode tokens w. CUDA graph (#6757) 2024-07-24 22:33:56 -07:00
Cody Yu e0c15758b8 [Core] Modulize prepare input and attention metadata builder (#6596) 2024-07-23 00:45:24 +00:00
Cyrus Leung 9042d68362 [Misc] Consolidate and optimize logic for building padded tensors (#6541) 2024-07-20 04:17:24 +00:00
Cody Yu 2fa4623d9e [Core] Refactor _prepare_model_input_tensors - take 2 (#6164) 2024-07-17 09:37:16 -07:00
afeldman-nmandWoosuk Kwon 543aa48573 [Kernel] Correctly invoke prefill & decode kernels for cross-attention (towards eventual encoder/decoder model support) (#4888) 
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2024-07-08 17:12:15 +00:00