[ROCm][Perf] Extend QK-norm/RoPE/KV-cache fusion to MRoPE (#50212)
Signed-off-by: Vorapol Assavasangthong <Vorapol.Assavasangthong@amd.com> Co-authored-by: Santosh Hiremath <Santosh.Hiremath@amd.com> Co-authored-by: Douglas Lehr <91553416+dllehr-amd@users.noreply.github.com>
This commit is contained in:
co-authored by
Santosh Hiremath
Douglas Lehr
parent
0549e8d0ab
commit
b715699df4
@@ -154,15 +154,15 @@ class PassConfig:
|
||||
fuse_rope_kvcache: bool = None # type: ignore[assignment]
|
||||
"""Fuse the QK rope + KV cache ops."""
|
||||
fuse_qk_norm_rope_kvcache: bool = Field(default=None) # type: ignore[assignment]
|
||||
"""Fuse QK RMSNorm + RoPE + KV cache update into a single AITER HIP
|
||||
"""Fuse QK RMSNorm + RoPE/MRoPE + KV cache update into an AITER HIP
|
||||
kernel. Supersedes both enable_qk_norm_rope_fusion and fuse_rope_kvcache
|
||||
for layers that support it. Auto-enabled at O1+ on ROCm for models
|
||||
with QK-norm (e.g. Qwen3-MoE)."""
|
||||
for layers that support it. Auto-enabled at O2+ on ROCm for models
|
||||
with QK-norm (e.g. Qwen3-MoE and Qwen3-VL-class architectures)."""
|
||||
|
||||
rope_kvcache_fusion_max_token_num: int = 256
|
||||
"""The threshold for ROCm AITER RoPE+KVCache fusion e.g. for small batch decode.
|
||||
Larger batch sizes e.g. during prefill will use the unfused kernels.
|
||||
Also applies to the fused QK-Norm+RoPE+KVCache pass.
|
||||
Also applies to the fused QK-Norm+RoPE/MRoPE+KVCache pass.
|
||||
"""
|
||||
|
||||
fi_allreduce_fusion_max_size_mb: float | None = None
|
||||
|
||||
Reference in New Issue
Block a user