Integrate Marlin Kernels for Int4 GPTQ inference (#2497)

Co-authored-by: Robert Shaw <114415538+rib-2@users.noreply.github.com>
Co-authored-by: alexm <alexm@neuralmagic.com>
This commit is contained in:
Robert Shaw
2024-03-01 12:47:51 -08:00
committed by GitHub
co-authored by Robert Shaw alexm
parent 90fbf12540
commit c0c2335ce0
12 changed files with 1752 additions and 6 deletions
+2
View File
@@ -342,6 +342,8 @@ vllm_extension_sources = [
if _is_cuda():
vllm_extension_sources.append("csrc/quantization/awq/gemm_kernels.cu")
vllm_extension_sources.append(
"csrc/quantization/marlin/marlin_cuda_kernel.cu")
vllm_extension_sources.append("csrc/custom_all_reduce.cu")
# Add MoE kernels.