[ROCm][Bugfix][Kimi-K3] Preserve MoE correction bias in FP32 (#50761)

Signed-off-by: Fangzhou Ai <Fangzhou.Ai@amd.com>
Co-authored-by: OpenAI Codex <codex@openai.com>
This commit is contained in:
Fangzhou Ai
2026-08-02 20:30:54 +00:00
committed by GitHub
co-authored by OpenAI Codex
parent 55c98e370a
commit c666810676
+4 -1
View File
@@ -210,7 +210,10 @@ class KimiMoE(nn.Module):
prefix=f"{prefix}.gate",
)
self.gate.e_score_correction_bias = nn.Parameter(torch.empty(num_experts))
# Preserve FP32 checkpoint values and match FP32 router logits.
self.gate.e_score_correction_bias = nn.Parameter(
torch.empty(num_experts, dtype=torch.float32)
)
if self.num_shared_experts is not None:
shared_intermediate_size = moe_intermediate_size * self.num_shared_experts