[ROCm][Perf] Enable medium-skinny dispatch in RDNAHybridW4A16LinearKernel (#52619)

Signed-off-by: tangzzycc <3081129260@qq.com>
This commit is contained in:
00CC
2026-10-07 13:21:43 +00:00
committed by GitHub
parent efb8d85f24
commit 5281e49908
2 changed files with 88 additions and 9 deletions
@@ -2,9 +2,10 @@
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Hybrid W4A16 kernel: Triton for prefill, HIP skinny for decode.
Routes based on batch size M:
M <= MAX_SKINNY_BATCH_SIZE: HIP skinny GEMM (wvSplitK_int4_g)
M > MAX_SKINNY_BATCH_SIZE: Triton W4A16 fused dequant GEMM
Routes based on the activation shape:
M <= MAX_SKINNY_BATCH_SIZE and K*M <= MEDIUM_SKINNY_LIMIT_ELEMENTS:
HIP skinny GEMM (wvSplitK_int4_g)
otherwise: Triton W4A16 fused dequant GEMM
Stores the weights ONCE as int8 [N, K//2] (ExLlama shuffle packed). Both
paths read this single buffer: the HIP skinny kernel uses it directly, and
@@ -56,13 +57,14 @@ def _on_gfx1151() -> bool:
return on_gfx1151()
# Maximum batch size M for the HIP skinny kernel path (C++ supports N_in
# up to 5). When M is below this AND K*M fits in LDS, the skinny kernel is
# used; otherwise the Triton prefill path handles the GEMM.
# Maximum activation rows supported by HIP skinny (Python M maps to C++ N_in).
# When K*M exceeds regular LDS capacity, the C++ medium kernel caches the
# prefix in LDS and reads the remaining activation elements from global memory.
MAX_SKINNY_BATCH_SIZE = 5
# 64 KiB per-workgroup LDS limit expressed in fp16 elements.
# (AMD RDNA has 128 KiB total LDS per CU, but 64 KiB per workgroup.)
LDS_CAPACITY_ELEMENTS = 64 * 1024 // 2 # 32768 fp16 elements
MEDIUM_SKINNY_LIMIT_ELEMENTS = int(LDS_CAPACITY_ELEMENTS * 1.2)
# ---------------------------------------------------------------------------
@@ -464,7 +466,7 @@ def _rdna_hybrid_w4a16_apply_impl(
K = x_2d.shape[1]
N = w_q.shape[0]
if M <= MAX_SKINNY_BATCH_SIZE and K * M <= LDS_CAPACITY_ELEMENTS:
if M <= MAX_SKINNY_BATCH_SIZE and K * M <= MEDIUM_SKINNY_LIMIT_ELEMENTS:
# record_function is not torch.compile-safe; use nullcontext when
# compiling to keep the op traceable.
ctx = (