[MoE Refactor] Add more MoE layer tests (#39349)

Signed-off-by: Bill Nell <bnell@redhat.com>
Signed-off-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
Co-authored-by: Robert Shaw <114415538+robertgshaw2-redhat@users.noreply.github.com>
This commit is contained in:
bnellnm
2026-04-21 18:12:36 -04:00
committed by GitHub
co-authored by Robert Shaw
parent 5e584ce9ec
commit 9db4650e5e
3 changed files with 98 additions and 44 deletions
@@ -77,8 +77,8 @@ def _worker_parallel_launch(
*args: Any,
) -> None:
rank = node_rank * world_local_size + local_rank
torch.accelerator.set_device_index(local_rank)
device = torch.device("cuda", local_rank)
torch.accelerator.set_device_index(device)
torch.distributed.init_process_group(
backend="cpu:gloo,cuda:nccl",
init_method=init_method,