[Model] Fix Gemma4 MoE activation mismatch (#41574)

Signed-off-by: Luciano Martins <lucianommartins@users.noreply.github.com>
Co-authored-by: Luciano Martins <lucianommartins@users.noreply.github.com>
This commit is contained in:
Luciano Martins
2026-05-05 04:34:11 +00:00
committed by GitHub
co-authored by Luciano Martins
parent eaec7be446
commit 6bb924bbf3
6 changed files with 23 additions and 1 deletions
+1 -1
View File
@@ -360,7 +360,7 @@ class Gemma4MoE(nn.Module):
quant_config=quant_config,
prefix=f"{prefix}.experts",
custom_routing_function=routing_function,
activation="gelu",
activation="gelu_tanh",
)
def forward(self, x: torch.Tensor, router_logits: torch.Tensor) -> torch.Tensor: