[Data-parallel] Allow DP>1 for world_size > num_gpus on node (8) (#26367)

Signed-off-by: Patrick von Platen <patrick.v.platen@gmail.com>
Signed-off-by: Rui Qiao <ruisearch42@gmail.com>
Co-authored-by: gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com>
Co-authored-by: Rui Qiao <ruisearch42@gmail.com>
This commit is contained in:
Patrick von Platen
2025-10-17 08:24:42 -07:00
committed by GitHub
co-authored by gemini-code-assist[bot] <176961590+gemini-code-assist[bot]@users.noreply.github.com> Rui Qiao
parent 2ba60ec7fe
commit b038d9c40c
4 changed files with 96 additions and 22 deletions
+9 -2
View File
@@ -1403,8 +1403,15 @@ class EngineArgs:
"data_parallel_size_local must be set to use data_parallel_hybrid_lb."
)
# Local DP size defaults to global DP size if not set.
data_parallel_size_local = self.data_parallel_size
if self.data_parallel_backend == "ray" and (
envs.VLLM_RAY_DP_PACK_STRATEGY == "span"
):
# Data parallel size defaults to 1 if DP ranks are spanning
# multiple nodes
data_parallel_size_local = 1
else:
# Otherwise local DP size defaults to global DP size if not set
data_parallel_size_local = self.data_parallel_size
# DP address, used in multi-node case for torch distributed group
# and ZMQ sockets.