iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
4,774 Commits 3 Branches 0 Tags
db986c19ea35d7f3522a45d5205bf5d3ffab14e4
Commit Graph
68 Commits
Author SHA1 Message Date
Gregory Shtrasberg c904fdddf6 [ROCm] Apply FP8 weights padding to values not divisible by 512 bytes on ROCm (#13231) 2025-02-22 05:54:38 -08:00
Lucas WilkinsonPatrick Hornsimon-moTyler Michael Smith
 		288cc6c234 [Attention] MLA with chunked prefill (#12639) 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Signed-off-by: Lucas Wilkinson <lwilkins@redhat.com>
Co-authored-by: Patrick Horn <patrick.horn@gmail.com>
Co-authored-by: simon-mo <xmo@berkeley.edu>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
 2025-02-21 15:30:12 -08:00
Tyler Michael Smith b3942e157e [Bugfix][CI][V1] Work around V1 + CUDA Graph + torch._scaled_mm fallback issue (#13425) 
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
 2025-02-18 00:32:48 +00:00
rasmith ed0de3e4b8 [AMD] [Model] DeepSeek tunings (#13199) 2025-02-15 03:58:09 -08:00
Michael Goin 09972e716c [Bugfix] Allow fallback to AWQ from AWQMarlin at per-layer granularity (#13119) 2025-02-12 09:19:53 -08:00
Qubitium-ModelCloud 36a08630e8 [CORE] [QUANT] Support for GPTQModel's dynamic quantization per module override/control (#7086) 2025-02-12 09:19:43 -08:00
TJian eaa92d4437 [ROCm] [Feature] [Doc] [Dockerfile] [BugFix] Support Per-Token-Activation Per-Channel-Weight FP8 Quantization Inferencing (#12501) 2025-02-07 08:13:43 -08:00
Lu Fang 741429a4cd [MISC] Check space in the file names in the pre commit checks (#12804) 
Signed-off-by: Lu Fang <lufang@fb.com>
 2025-02-06 15:36:21 -08:00
Lu Fang 9cdea30b4f [Misc][Easy] Remove the space from the file name 2025-02-05 19:23:35 -08:00
Lucas Wilkinson 76abd0c881 [Bugfix] Better FP8 supported defaults 2025-02-05 19:22:19 -08:00
Kyle Sayersandmgoin 7ff7a638b6 [Model][Quant] Fix GLM, Fix fused module mappings for quantization (#12634) 
Signed-off-by: mgoin <michael@neuralmagic.com>
Signed-off-by: Kyle Sayers <kylesayrs@gmail.com>
Co-authored-by: mgoin <michael@neuralmagic.com>
 2025-02-05 05:32:06 +00:00
Hongxia YangandMatthew Wong c36ac98d01 [AMD][ROCm] Enable DeepSeek model on ROCm (#12662) 
Signed-off-by: Hongxia Yang <hongxia.yang@amd.com>
Co-authored-by: Matthew Wong <Matthew.Wong2@amd.com>
 2025-02-04 08:24:11 +00:00
kushanam 4797dad3ec [Model] Add Deepseek V3 fp8_w8a8 configs for B200 (#12707) 2025-02-03 13:30:39 -08:00
Russell Bryant e489ad7a21 [Misc] Add SPDX-License-Identifier headers to python source files (#12628) 
- **Add SPDX license headers to python source files**
- **Check for SPDX headers using pre-commit**

commit 9d7ef44c3cfb72ca4c32e1c677d99259d10d4745
Author: Russell Bryant <rbryant@redhat.com>
Date:   Fri Jan 31 14:18:24 2025 -0500

    Add SPDX license headers to python source files
    
This commit adds SPDX license headers to python source files as
recommended to
the project by the Linux Foundation. These headers provide a concise way
that is
both human and machine readable for communicating license information
for each
source file. It helps avoid any ambiguity about the license of the code
and can
    also be easily used by tools to help manage license compliance.
    
The Linux Foundation runs license scans against the codebase to help
ensure
    we are in compliance with the licenses of the code we use, including
dependencies. Having these headers in place helps that tool do its job.
    
    More information can be found on the SPDX site:
    
    - https://spdx.dev/learn/handling-license-info/
    
    Signed-off-by: Russell Bryant <rbryant@redhat.com>

commit 5a1cf1cb3b80759131c73f6a9dddebccac039dea
Author: Russell Bryant <rbryant@redhat.com>
Date:   Fri Jan 31 14:36:32 2025 -0500

    Check for SPDX headers using pre-commit
    
    Signed-off-by: Russell Bryant <rbryant@redhat.com>

---------

Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-02-02 11:58:18 -08:00
Lucas WilkinsonWoosuk Kwonsimon-moMichael GoinZhuohan LiTyler Michael SmithAlexander Matveev
 		baeded2569 [Attention] Deepseek v3 MLA support with FP8 compute (#12601) 
This PR implements the Deepseek V3 support by performing matrix absorption the fp8 weights 

---------

Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
Co-authored-by: simon-mo <simon.mo@hey.com>
Co-authored-by: Michael Goin <mgoin64@gmail.com>
Co-authored-by: Zhuohan Li <zhuohan123@gmail.com>
Co-authored-by: Tyler Michael Smith <tysmith@redhat.com>
Co-authored-by: Alexander Matveev <59768536+alexm-neuralmagic@users.noreply.github.com>
 2025-01-31 21:52:51 -08:00
Tyler Michael Smith eb5741ad42 [Kernel][Quantization] Integrate block-quantized CUTLASS kernels for DeepSeekV3 (#12587) 
Integrates the block-quantized kernels introduced in
https://github.com/vllm-project/vllm/pull/11868 for use in linear
layers.

Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
 2025-01-31 15:29:11 -08:00
Robert Shawmgoinsimon-mo
 		9b0c4bab36 [Kernel] Triton Configs for Fp8 Block Quantization (#11589) 
Signed-off-by: rshaw@neuralmagic.com <rshaw@neuralmagic.com>
Signed-off-by: mgoin <michael@neuralmagic.com>
Co-authored-by: mgoin <michael@neuralmagic.com>
Co-authored-by: simon-mo <xmo@berkeley.edu>
 2025-01-30 11:53:22 -08:00
Harry Mellor 823ab79633 Update pre-commit hooks (#12475) 
Signed-off-by: Harry Mellor <19981378+hmellor@users.noreply.github.com>
 2025-01-27 17:23:08 -07:00
Gregory Shtrasberg b5b57e301e [AMD][FP8] Using MI300 FP8 format on ROCm for block_quant (#12134) 
Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>
 2025-01-17 17:12:26 +00:00
Cyrus LeungandMaxime Fournioux d848800e88 [Misc] Move print_*_once from utils to logger (#11298) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: Maxime Fournioux <55544262+mfournioux@users.noreply.github.com>
Co-authored-by: Maxime Fournioux <55544262+mfournioux@users.noreply.github.com>
 2025-01-09 12:48:12 +08:00
Robert ShawandWoosuk Kwon 56fe4c297c [TPU][Quantization] TPU W8A8 (#11785) 
Co-authored-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-01-08 19:33:29 +00:00
Michael Goinsimon-mosimon-moHandH1998
 		2072924d14 [Model] [Quantization] Support deepseek_v3 w8a8 fp8 block-wise quantization (#11523) 
Signed-off-by: mgoin <michael@neuralmagic.com>
Signed-off-by: simon-mo <simon.mo@hey.com>
Signed-off-by: simon-mo <xmo@berkeley.edu>
Co-authored-by: simon-mo <simon.mo@hey.com>
Co-authored-by: simon-mo <xmo@berkeley.edu>
Co-authored-by: HandH1998 <1335248067@qq.com>
 2024-12-26 15:33:30 -08:00
Isotr0py 276738ce0f [Bugfix] Fix broken CPU compressed-tensors test (#11338) 
Signed-off-by: Isotr0py <2037008807@qq.com>
 2024-12-19 17:37:31 +00:00
Tyler Michael Smith 5a9da2e6e9 [Bugfix][Build/CI] Fix sparse CUTLASS compilation on CUDA [12.0, 12.2) (#11311) 
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
 2024-12-19 02:43:30 +00:00
ElizaWszola b00b33d77e [Model][Quantization] HQQ support through Marlin kernel expansion (#9766) 
Signed-off-by: ElizaWszola <eliza@neuralmagic.com>
 2024-11-19 13:31:12 -08:00
Lucas Wilkinson 96d999fbe8 [Kernel] Initial Machete W4A8 support + Refactors (#9855) 
Signed-off-by: Lucas Wilkinson <lwilkinson@neuralmagic.com>
 2024-11-18 12:59:29 -07:00
Luka GovedičandDaniël de Kok bf2ddc6610 [bugfix] Fix static asymmetric quantization case (#10334) 
Signed-off-by: Daniël de Kok <me@danieldk.eu>
Signed-off-by: luka <luka@neuralmagic.com>
Co-authored-by: Daniël de Kok <me@danieldk.eu>
 2024-11-15 09:35:11 +08:00
Michael Goin 4ab3256644 [Bugfix] Fix FP8 torch._scaled_mm fallback for torch>2.5 with CUDA<12.4 (#10095) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-11-07 00:54:13 +00:00
Michael Goin 226688bd61 [Bugfix][VLM] Make apply_fp8_linear work with >2D input (#9812) 2024-10-29 19:49:44 -07:00
wangshuai09 4e2d95e372 [Hardware][ROCM] using current_platform.is_rocm (#9642) 
Signed-off-by: wangshuai09 <391746016@qq.com>
 2024-10-28 04:07:00 +00:00
Lucas Wilkinson e312e52b44 [Kernel] Add Exllama as a backend for compressed-tensors (#9395) 2024-10-17 09:48:26 -04:00
ElizaWszolaDipikaDipika Sikka
 		05d686432f [Kernel] Zero point support in fused MarlinMoE kernel + AWQ Fused MoE (#8973) 
Co-authored-by: Dipika <dipikasikka1@gmail.com>
Co-authored-by: Dipika Sikka <ds3822@columbia.edu>
 2024-10-04 12:34:44 -06:00
Cyrus LeungandRoger Wang e1a3f5e831 [CI/Build] Update models tests & examples (#8874) 
Co-authored-by: Roger Wang <ywang@roblox.com>
 2024-09-28 09:54:35 -07:00
Luka Govedič 172d1cd276 [Kernel] AQ AZP 4/4: Integrate asymmetric quantization to linear method (#7271) 2024-09-27 14:25:10 -04:00
Lucas Wilkinson 72fc97a0f1 [Bugfix] Fix torch dynamo fixes caused by replace_parameters (#8748) 2024-09-24 14:33:21 -04:00
Lucas WilkinsonmgoinDivakar VermaTyler Michael Smith
 		86e9c8df29 [Kernel] (2/N) Machete - Integrate into CompressedTensorsWNA16 and GPTQMarlin (#7701) 
Co-authored-by: mgoin <michael@neuralmagic.com>
Co-authored-by: Divakar Verma <137818590+divakar-amd@users.noreply.github.com>
Co-authored-by: Tyler Michael Smith <tyler@neuralmagic.com>
 2024-09-23 13:46:26 -04:00
Gregory ShtrasbergAlexei-V-Ivanov-AMDMichael Goin
 		b3195bc9e4 [AMD][ROCm]Quantization methods on ROCm; Fix _scaled_mm call (#8380) 
Co-authored-by: Alexei-V-Ivanov-AMD <156011006+Alexei-V-Ivanov-AMD@users.noreply.github.com>
Co-authored-by: Michael Goin <michael@neuralmagic.com>
 2024-09-18 10:41:08 -07:00
Cyrus Leung 6ffa3f314c [CI/Build] Avoid CUDA initialization (#8534) 2024-09-18 10:38:11 +00:00
Luka Govedič 5d73ae49d6 [Kernel] AQ AZP 3/4: Asymmetric quantization kernels (#7270) 2024-09-16 11:52:40 -07:00
Dipika Sikka 6cd5e5b07e [Misc] Fused MoE Marlin support for GPTQ (#8217) 2024-09-09 23:02:52 -04:00
Dipika SikkaandMichael Goin e16fa99a6a [Misc] Update fbgemmfp8 to use vLLMParameters (#7972) 
Co-authored-by: Michael Goin <michael@neuralmagic.com>
 2024-09-03 20:12:41 -06:00
Lucas Wilkinson 5288c06aa0 [Kernel] (1/N) Machete - Hopper Optimized Mixed Precision Linear Kernel (#7174) 2024-08-20 07:09:33 -06:00
Charlie Fu e837b624f2 [Feature][Hardware][Amd] Add fp8 Linear Layer for Rocm (#7210) 2024-08-16 10:06:30 -07:00
Lucas Wilkinson 311f743831 [Bugfix] Fix gptq failure on T4s (#7264) 2024-08-07 20:05:37 +00:00
Lucas Wilkinson a8d604ca2a [Misc] Disambiguate quantized types via a new ScalarType (#6396) 2024-08-02 13:51:58 -07:00
HandH1998 6512937de1 Support W4A8 quantization for vllm (#5218) 2024-07-31 07:55:21 -06:00
Tyler Michael Smith d7a299edaa [Kernel] Remove scaled_fp8_quant kernel padding footgun (#6842) 2024-07-30 16:37:01 -04:00
Alexander Matveev 75acdaa4b6 [Kernel] Increase precision of GPTQ/AWQ Marlin kernel (#6795) 2024-07-27 17:52:33 -04:00
Robert Shawandmgoin 889da130e7 [ Misc ] fp8-marlin channelwise via compressed-tensors (#6524) 
Co-authored-by: mgoin <michael@neuralmagic.com>
 2024-07-25 09:46:04 -07:00
Michael Goin 0eb0757bef [Misc] Add ignored layers for fp8 quantization (#6657) 2024-07-23 14:04:04 -04:00