[ROCm][Perf] Extend QK-norm/RoPE/KV-cache fusion to MRoPE (#50212)

Signed-off-by: Vorapol Assavasangthong <Vorapol.Assavasangthong@amd.com>
Co-authored-by: Santosh Hiremath <Santosh.Hiremath@amd.com>
Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com>
This commit is contained in:
vorapolsiloai
2026-09-23 11:11:57 -05:00
committed by GitHub
co-authored by Santosh Hiremath Douglas Lehr
parent 0549e8d0ab
commit b715699df4
11 changed files with 1314 additions and 118 deletions
+4 -4
View File
@@ -154,15 +154,15 @@ class PassConfig:
fuse_rope_kvcache: bool = None # type: ignore[assignment]
"""Fuse the QK rope + KV cache ops."""
fuse_qk_norm_rope_kvcache: bool = Field(default=None) # type: ignore[assignment]
"""Fuse QK RMSNorm + RoPE + KV cache update into a single AITER HIP
"""Fuse QK RMSNorm + RoPE/MRoPE + KV cache update into an AITER HIP
kernel. Supersedes both enable_qk_norm_rope_fusion and fuse_rope_kvcache
for layers that support it. Auto-enabled at O1+ on ROCm for models
with QK-norm (e.g. Qwen3-MoE)."""
for layers that support it. Auto-enabled at O2+ on ROCm for models
with QK-norm (e.g. Qwen3-MoE and Qwen3-VL-class architectures)."""
rope_kvcache_fusion_max_token_num: int = 256
"""The threshold for ROCm AITER RoPE+KVCache fusion e.g. for small batch decode.
Larger batch sizes e.g. during prefill will use the unfused kernels.
Also applies to the fused QK-Norm+RoPE+KVCache pass.
Also applies to the fused QK-Norm+RoPE/MRoPE+KVCache pass.
"""
fi_allreduce_fusion_max_size_mb: float | None = None