[WideEP] Explain how to fix a too-old NCCL for DeepEP v2 (#60449)
Signed-off-by: mgoin <mgoin64@gmail.com> Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
+18
-10
@@ -11,12 +11,16 @@ Step 2 is necessary for multi-node deployment.
|
||||
|
||||
All scripts accept a positional argument as workspace path for staging the build, defaulting to `$(pwd)/ep_kernels_workspace`.
|
||||
|
||||
## NCCL version requirement (CUDA 13+)
|
||||
## NCCL version requirement for DeepEPv2
|
||||
|
||||
DeepEPv2 uses the NCCL GIN (GPU-Initiated Networking) backend, which requires
|
||||
NCCL >= 2.30.4 at both compile time and runtime. PyTorch 2.11 pins
|
||||
`nvidia-nccl-cu13==2.28.9` as a transitive dependency, so you need to
|
||||
override it.
|
||||
DeepEPv2 (`--all2all-backend deepep_v2`) uses the NCCL GIN (GPU-Initiated
|
||||
Networking) backend, which requires NCCL >= 2.30.4 both when DeepEP is built
|
||||
and at runtime. PyTorch pins an older release as a dependency (PyTorch 2.13
|
||||
pins `nvidia-nccl-cu13==2.29.7`), so a plain `pip install` or `uv pip install`
|
||||
of vLLM is not enough. The vLLM Docker images already override it.
|
||||
|
||||
Upgrade NCCL before running `install_python_libraries.sh`. Use
|
||||
`nvidia-nccl-cu12` on CUDA 12.
|
||||
|
||||
**With uv** (recommended):
|
||||
|
||||
@@ -29,18 +33,22 @@ export UV_OVERRIDE=/tmp/nccl-override.txt
|
||||
uv pip install vllm
|
||||
```
|
||||
|
||||
**With pip**:
|
||||
Keep `UV_OVERRIDE` set for later installs into the same environment. Without
|
||||
it, any `uv pip install` that resolves PyTorch again restores the pinned NCCL.
|
||||
|
||||
**With pip**, or to fix an existing environment:
|
||||
|
||||
```bash
|
||||
pip install vllm
|
||||
pip install "nvidia-nccl-cu13>=2.30.4" --no-deps
|
||||
```
|
||||
|
||||
The override / reinstall must happen before building DeepEP (for GIN device
|
||||
headers) and must remain in place at runtime. You can verify with:
|
||||
`install_python_libraries.sh` warns when the installed NCCL is too old. If
|
||||
DeepEP was built against an older NCCL, upgrade NCCL and run the script again.
|
||||
When NCCL is too old at runtime, vLLM logs the reason at startup and
|
||||
`deepep_v2` fails with the same message. You can check with:
|
||||
|
||||
```bash
|
||||
python -c "from vllm.utils.import_utils import has_deep_ep_v2; print(has_deep_ep_v2())"
|
||||
python -c "from vllm.utils.import_utils import deep_ep_v2_unavailable_reason; print(deep_ep_v2_unavailable_reason() or 'deepep_v2 is available')"
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
@@ -78,10 +78,25 @@ mkdir -p "$WHEEL_DIR"
|
||||
pushd "$WORKSPACE"
|
||||
|
||||
# install dependencies if not installed
|
||||
UV_PIP_TARGET=()
|
||||
if [ -z "$VIRTUAL_ENV" ]; then
|
||||
uv pip install --system cmake torch ninja
|
||||
else
|
||||
uv pip install cmake torch ninja
|
||||
UV_PIP_TARGET=(--system)
|
||||
fi
|
||||
uv pip install "${UV_PIP_TARGET[@]}" cmake torch ninja
|
||||
|
||||
# DeepEPv2 needs NCCL >= 2.30.4 at both build time and runtime, but PyTorch
|
||||
# pins an older release.
|
||||
DEEPEP_V2_MIN_NCCL="2.30.4"
|
||||
NCCL_PACKAGE="nvidia-nccl-cu${CUDA_VERSION_MAJOR}"
|
||||
NCCL_WARNING=""
|
||||
NCCL_INSTALLED=$(uv pip show "${UV_PIP_TARGET[@]}" "$NCCL_PACKAGE" 2>/dev/null | sed -n 's/^Version: //p')
|
||||
if [ -n "$NCCL_INSTALLED" ] && \
|
||||
[ "$(printf '%s\n' "$DEEPEP_V2_MIN_NCCL" "$NCCL_INSTALLED" | sort -V | head -n1)" != "$DEEPEP_V2_MIN_NCCL" ]; then
|
||||
NCCL_WARNING="WARNING: ${NCCL_PACKAGE} ${NCCL_INSTALLED} is installed, but the deepep_v2 backend
|
||||
requires NCCL >= ${DEEPEP_V2_MIN_NCCL} when DeepEP is built and when it runs. To use deepep_v2, run
|
||||
uv pip install \"${NCCL_PACKAGE}>=${DEEPEP_V2_MIN_NCCL}\" --no-deps
|
||||
and then run this script again. See tools/ep_kernels/README.md."
|
||||
echo "$NCCL_WARNING" >&2
|
||||
fi
|
||||
|
||||
# fetch nvshmem
|
||||
@@ -236,3 +251,8 @@ if [ "$MODE" = "wheel" ]; then
|
||||
echo "All wheels written to $WHEEL_DIR"
|
||||
ls -l "$WHEEL_DIR"
|
||||
fi
|
||||
|
||||
# Repeat after the verbose build output, where the first copy is easy to miss.
|
||||
if [ -n "$NCCL_WARNING" ]; then
|
||||
echo "$NCCL_WARNING" >&2
|
||||
fi
|
||||
|
||||
Reference in New Issue
Block a user