iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity
5,799 Commits 3 Branches 0 Tags
57504a4bcf82cf544a4261778fc0505e492e4484
Commit Graph
72 Commits
Author SHA1 Message Date
Isotr0py 40b4284fe3 [Bugfix] Handle process_weights_after_loading for QKVCrossParallelLinear (#15328) 
Signed-off-by: Isotr0py <2037008807@qq.com>
 2025-04-08 10:02:23 -07:00
Michael Goin 21802c4b6d [ROCm][Bugfix][FP8] Make fp8 quant respect fused modules mapping (#16031) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2025-04-07 21:28:14 -04:00
Lu FangLu FangRoger WangDarkLight1337
 		55dcce91df Upstream Llama4 Support to Main (#16113) 
Signed-off-by: Aston Zhang <22279212+astonzhang@users.noreply.github.com>
Signed-off-by: Chris Thi <chris.c.thi@gmail.com>
Signed-off-by: drisspg <drisspguessous@gmail.com>
Signed-off-by: Jon Swenson <jmswen@gmail.com>
Signed-off-by: Keyun Tong <tongkeyun@gmail.com>
Signed-off-by: Lu Fang <fanglu@meta.com>
Signed-off-by: Xiaodong Wang <xdwang@meta.com>
Signed-off-by: Yang Chen <yangche@fb.com>
Signed-off-by: Ye (Charlotte) Qi <yeq@meta.com>
Signed-off-by: Yong Hoon Shin <yhshin@meta.com>
Signed-off-by: Zijing Liu <liuzijing2014@gmail.com>
Signed-off-by: Lu Fang <lufang@fb.com>
Signed-off-by: Lu Fang <fanglu@fb.com>
Signed-off-by: Lucia Fang <fanglu@fb.com>
Signed-off-by: Roger Wang <ywang@roblox.com>
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Co-authored-by: Lu Fang <fanglu@fb.com>
Co-authored-by: Roger Wang <ywang@roblox.com>
Co-authored-by: DarkLight1337 <tlleungac@connect.ust.hk>
 2025-04-07 08:06:27 -07:00
bnellnm e59ca942f5 Add option to use DeepGemm contiguous grouped gemm kernel for fused MoE operations. (#13932) 
Signed-off-by: Bill Nell <bnell@redhat.com>
 2025-04-01 12:07:43 -04:00
Gregory ShtrasbergandLuka Govedič 4d0ec37267 [Quantization][FP8] Adding support for fp8 gemm layer input in fp8 (#14578) 
Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>
Co-authored-by: Luka Govedič <ProExpertProg@users.noreply.github.com>
 2025-03-28 02:58:16 +00:00
Robert Shaw e1e0fd7543 [TPU] Avoid Triton Import (#15589) 
Signed-off-by: rshaw@neuralmagic.com <robertgshaw2@gmail.com>
 2025-03-27 06:43:02 +00:00
vllmellmandtjtanaa 5ebf66748b [FEAT][ROCm] Integrate Fused MoE Kernels from AITER (#14967) 
Signed-off-by: vllmellm <vllm.ellm@embeddedllm.com>
Signed-off-by: tjtanaa <tunjian.tan@embeddedllm.com>
Co-authored-by: tjtanaa <tunjian.tan@embeddedllm.com>
 2025-03-26 16:30:30 +08:00
Gregory Shtrasbergandcharlifu f533b5837f [ROCm][Kernel] MoE weights padding (#14454) 
Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>
Signed-off-by: charlifu <charlifu@amd.com>
Co-authored-by: charlifu <charlifu@amd.com>
 2025-03-24 23:45:30 +00:00
Jeff Daily a1c8f3796c dynamic distpatch of fp8 kernels (#14245) 
Signed-off-by: Jeff Daily <jeff.daily@amd.com>
 2025-03-11 10:54:56 -04:00
Luka Govedič e1744502c2 [FP8] Refactor apply_fp8_linear and apply_fp8_linear_generic into an object (#14390) 
Signed-off-by: luka <luka@neuralmagic.com>
 2025-03-07 05:20:16 +00:00
Woosuk Kwon b382a7f28f [BugFix] Make FP8 Linear compatible with torch.compile (#13918) 
Signed-off-by: Woosuk Kwon <woosuk.kwon@berkeley.edu>
 2025-02-26 13:48:55 -08:00
Michael Goin 07c4353057 [Model] Support Grok1 (#13795) 
Signed-off-by: mgoin <mgoin64@gmail.com>
 2025-02-26 01:07:12 +00:00
Tyler Michael Smith 1e15aaef56 [Bugfix][Quantization] Fix FP8 + EP (#13784) 
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
 2025-02-25 10:54:17 +08:00
Jongseok Park 781096e385 Expert Parallelism (EP) Support for DeepSeek V2 (#12583) 2025-02-24 07:33:20 -08:00
Gregory Shtrasberg c904fdddf6 [ROCm] Apply FP8 weights padding to values not divisible by 512 bytes on ROCm (#13231) 2025-02-22 05:54:38 -08:00
Tyler Michael Smith b3942e157e [Bugfix][CI][V1] Work around V1 + CUDA Graph + torch._scaled_mm fallback issue (#13425) 
Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
 2025-02-18 00:32:48 +00:00
Kyle Sayers 12913d17ba [Quant] Add SupportsQuant to phi3 and clip (#13104) 2025-02-15 19:28:33 -08:00
Russell Bryant e489ad7a21 [Misc] Add SPDX-License-Identifier headers to python source files (#12628) 
- **Add SPDX license headers to python source files**
- **Check for SPDX headers using pre-commit**

commit 9d7ef44c3cfb72ca4c32e1c677d99259d10d4745
Author: Russell Bryant <rbryant@redhat.com>
Date:   Fri Jan 31 14:18:24 2025 -0500

    Add SPDX license headers to python source files
    
This commit adds SPDX license headers to python source files as
recommended to
the project by the Linux Foundation. These headers provide a concise way
that is
both human and machine readable for communicating license information
for each
source file. It helps avoid any ambiguity about the license of the code
and can
    also be easily used by tools to help manage license compliance.
    
The Linux Foundation runs license scans against the codebase to help
ensure
    we are in compliance with the licenses of the code we use, including
dependencies. Having these headers in place helps that tool do its job.
    
    More information can be found on the SPDX site:
    
    - https://spdx.dev/learn/handling-license-info/
    
    Signed-off-by: Russell Bryant <rbryant@redhat.com>

commit 5a1cf1cb3b80759131c73f6a9dddebccac039dea
Author: Russell Bryant <rbryant@redhat.com>
Date:   Fri Jan 31 14:36:32 2025 -0500

    Check for SPDX headers using pre-commit
    
    Signed-off-by: Russell Bryant <rbryant@redhat.com>

---------

Signed-off-by: Russell Bryant <rbryant@redhat.com>
 2025-02-02 11:58:18 -08:00
Tyler Michael Smith eb5741ad42 [Kernel][Quantization] Integrate block-quantized CUTLASS kernels for DeepSeekV3 (#12587) 
Integrates the block-quantized kernels introduced in
https://github.com/vllm-project/vllm/pull/11868 for use in linear
layers.

Signed-off-by: Tyler Michael Smith <tyler@neuralmagic.com>
 2025-01-31 15:29:11 -08:00
Robert Shawandsimon-mo 325f679f32 [BugFix] Fix Torch.Compile For DeepSeek (#12594) 
Co-authored-by: simon-mo <xmo@berkeley.edu>
 2025-01-31 12:06:39 -08:00
Dipika SikkaElizaWszolaMichael Goin
 		eb5cb5e528 [BugFix] Fix parameter names and process_after_weight_loading for W4A16 MoE Group Act Order (#11528) 
Signed-off-by: ElizaWszola <eliza@neuralmagic.com>
Co-authored-by: ElizaWszola <eliza@neuralmagic.com>
Co-authored-by: Michael Goin <michael@neuralmagic.com>
 2025-01-23 21:40:33 +00:00
Gregory Shtrasberg b5b57e301e [AMD][FP8] Using MI300 FP8 format on ROCm for block_quant (#12134) 
Signed-off-by: Gregory Shtrasberg <Gregory.Shtrasberg@amd.com>
 2025-01-17 17:12:26 +00:00
Elfie GuoMichael Goinmgoin
 		fa0050db08 [Core] Default to using per_token quantization for fp8 when cutlass is supported. (#8651) 
Signed-off-by: mgoin <michael@neuralmagic.com>
Co-authored-by: Michael Goin <mgoin@redhat.com>
Co-authored-by: mgoin <michael@neuralmagic.com>
 2025-01-16 04:31:27 +00:00
Cyrus LeungandMaxime Fournioux d848800e88 [Misc] Move print_*_once from utils to logger (#11298) 
Signed-off-by: DarkLight1337 <tlleungac@connect.ust.hk>
Signed-off-by: Maxime Fournioux <55544262+mfournioux@users.noreply.github.com>
Co-authored-by: Maxime Fournioux <55544262+mfournioux@users.noreply.github.com>
 2025-01-09 12:48:12 +08:00
Li, Jiang 5dbf854553 [CI/Build][CPU] Fix CPU CI by lazy importing triton FP8 kernels (#11618) 
Signed-off-by: jiang1.li <jiang1.li@intel.com>
 2024-12-30 10:17:04 +00:00
Robert Shaw 2339d59f92 [BugFix] Fix quantization for all other methods (#11547) 2024-12-26 22:23:29 -08:00
Simon Momgoinrobertgshaw2-neuralmagic
 		f49777ba62 Deepseek v3 (#11502) 
Signed-off-by: mgoin <michael@neuralmagic.com>
Co-authored-by: mgoin <michael@neuralmagic.com>
Co-authored-by: robertgshaw2-neuralmagic <rshaw@neuralmagic.com>
 2024-12-26 16:09:44 -08:00
Michael Goinsimon-mosimon-moHandH1998
 		2072924d14 [Model] [Quantization] Support deepseek_v3 w8a8 fp8 block-wise quantization (#11523) 
Signed-off-by: mgoin <michael@neuralmagic.com>
Signed-off-by: simon-mo <simon.mo@hey.com>
Signed-off-by: simon-mo <xmo@berkeley.edu>
Co-authored-by: simon-mo <simon.mo@hey.com>
Co-authored-by: simon-mo <xmo@berkeley.edu>
Co-authored-by: HandH1998 <1335248067@qq.com>
 2024-12-26 15:33:30 -08:00
Michael Goin 399c798608 Remove ScaledActivation for AWQ (#10057) 
Signed-off-by: mgoin <michael@neuralmagic.com>
 2024-11-06 14:27:06 +00:00
wangshuai09 4e2d95e372 [Hardware][ROCM] using current_platform.is_rocm (#9642) 
Signed-off-by: wangshuai09 <391746016@qq.com>
 2024-10-28 04:07:00 +00:00
Cyrus Leung 6ffa3f314c [CI/Build] Avoid CUDA initialization (#8534) 2024-09-18 10:38:11 +00:00
WenxiangYour NameZeqi LinZeqi Lin
 		1248e8506a [Model] Adding support for MSFT Phi-3.5-MoE (#7729) 
Co-authored-by: Your Name <you@example.com>
Co-authored-by: Zeqi Lin <zelin@microsoft.com>
Co-authored-by: Zeqi Lin <Zeqi.Lin@microsoft.com>
 2024-08-30 13:42:57 -06:00
Dipika SikkaandElizaWszola fc911880cc [Kernel] Expand MoE weight loading + Add Fused Marlin MoE Kernel (#7766) 
Co-authored-by: ElizaWszola <eliza@neuralmagic.com>
 2024-08-27 15:07:09 -07:00
Dipika Sikka 955b5191c9 [Misc] update fp8 to use vLLMParameter (#7437) 2024-08-22 08:36:18 -04:00
Michael Goin aae74ef95c Revert "[Kernel] Expand MoE weight loading + Add Fused Marlin MoE Kernel (#7527)" (#7764) 2024-08-22 03:42:14 +00:00
Dipika SikkaandElizaWszola 8678a69ab5 [Kernel] Expand MoE weight loading + Add Fused Marlin MoE Kernel (#7527) 
Co-authored-by: ElizaWszola <eliza@neuralmagic.com>
 2024-08-21 16:17:10 -07:00
Mor Zusman 7fc23be81c [Kernel] W8A16 Int8 inside FusedMoE (#7415) 2024-08-16 10:06:51 -07:00
Charlie Fu e837b624f2 [Feature][Hardware][Amd] Add fp8 Linear Layer for Rocm (#7210) 2024-08-16 10:06:30 -07:00
Dipika Sikka d3bdfd3ab9 [Misc] Update Fused MoE weight loading (#7334) 2024-08-13 14:57:45 -04:00
Michael Goin 5223199e03 [Bugfix][FP8] Fix dynamic FP8 Marlin quantization (#7219) 2024-08-07 11:23:12 -07:00
Thomas Parnell 9a7e2d0534 [Bugfix] Allow vllm to still work if triton is not installed. (#6786) 
Signed-off-by: Thomas Parnell <tpa@zurich.ibm.com>
 2024-07-29 14:51:27 -07:00
Robert Shawandmgoin 889da130e7 [ Misc ] fp8-marlin channelwise via compressed-tensors (#6524) 
Co-authored-by: mgoin <michael@neuralmagic.com>
 2024-07-25 09:46:04 -07:00
Michael Goin 0eb0757bef [Misc] Add ignored layers for fp8 quantization (#6657) 2024-07-23 14:04:04 -04:00
Michael Goin 9e0b558a09 [Misc] Support FP8 kv cache scales from compressed-tensors (#6528) 2024-07-23 04:11:50 +00:00
Robert Shaw 683e3cb9c4 [ Misc ] fbgemm checkpoints (#6559) 2024-07-20 09:36:57 -07:00
Robert Shaw 4cc24f01b1 [ Kernel ] Enable Dynamic Per Token fp8 (#6547) 2024-07-19 23:08:15 +00:00
Michael Goin 978aed5300 [Kernel][Attention] Separate Attention.kv_scale into k_scale and v_scale (#6081) 2024-07-16 15:31:32 -07:00
Robert Shaw fb6af8bc08 [ Misc ] Apply MoE Refactor to Deepseekv2 To Support Fp8 (#6417) 2024-07-13 20:03:58 -07:00
Robert ShawandRobert Shaw b675069d74 [ Misc ] Refactor Marlin Python Utilities (#6082) 
Co-authored-by: Robert Shaw <rshaw@neuralmagic.com>
 2024-07-11 15:40:11 +00:00
Robert ShawandRobert Shaw abfe705a02 [ Misc ] Support Fp8 via llm-compressor (#6110) 
Co-authored-by: Robert Shaw <rshaw@neuralmagic>
 2024-07-07 20:42:11 +00:00