[ROCm][Bugfix][Kimi-K3] Preserve MoE correction bias in FP32 (#50761)
Signed-off-by: Fangzhou Ai <Fangzhou.Ai@amd.com> Co-authored-by: OpenAI Codex <codex@openai.com>
This commit is contained in:
co-authored by
OpenAI Codex
parent
55c98e370a
commit
c666810676
@@ -210,7 +210,10 @@ class KimiMoE(nn.Module):
|
||||
prefix=f"{prefix}.gate",
|
||||
)
|
||||
|
||||
self.gate.e_score_correction_bias = nn.Parameter(torch.empty(num_experts))
|
||||
# Preserve FP32 checkpoint values and match FP32 router logits.
|
||||
self.gate.e_score_correction_bias = nn.Parameter(
|
||||
torch.empty(num_experts, dtype=torch.float32)
|
||||
)
|
||||
|
||||
if self.num_shared_experts is not None:
|
||||
shared_intermediate_size = moe_intermediate_size * self.num_shared_experts
|
||||
|
||||
Reference in New Issue
Block a user