[WideEP] Explain how to fix a too-old NCCL for DeepEP v2 (#60449)

Signed-off-by: mgoin <mgoin64@gmail.com>
Co-authored-by: Claude <noreply@anthropic.com>
This commit is contained in:
Misha Goin
2026-10-09 12:34:58 -07:00
committed by GitHub
co-authored by Claude
parent 04462732a4
commit 3766a67a32
4 changed files with 70 additions and 31 deletions
+18 -10
View File
@@ -11,12 +11,16 @@ Step 2 is necessary for multi-node deployment.
All scripts accept a positional argument as workspace path for staging the build, defaulting to `$(pwd)/ep_kernels_workspace`.
## NCCL version requirement (CUDA 13+)
## NCCL version requirement for DeepEPv2
DeepEPv2 uses the NCCL GIN (GPU-Initiated Networking) backend, which requires
NCCL >= 2.30.4 at both compile time and runtime. PyTorch 2.11 pins
`nvidia-nccl-cu13==2.28.9` as a transitive dependency, so you need to
override it.
DeepEPv2 (`--all2all-backend deepep_v2`) uses the NCCL GIN (GPU-Initiated
Networking) backend, which requires NCCL >= 2.30.4 both when DeepEP is built
and at runtime. PyTorch pins an older release as a dependency (PyTorch 2.13
pins `nvidia-nccl-cu13==2.29.7`), so a plain `pip install` or `uv pip install`
of vLLM is not enough. The vLLM Docker images already override it.
Upgrade NCCL before running `install_python_libraries.sh`. Use
`nvidia-nccl-cu12` on CUDA 12.
**With uv** (recommended):
@@ -29,18 +33,22 @@ export UV_OVERRIDE=/tmp/nccl-override.txt
uv pip install vllm
```
**With pip**:
Keep `UV_OVERRIDE` set for later installs into the same environment. Without
it, any `uv pip install` that resolves PyTorch again restores the pinned NCCL.
**With pip**, or to fix an existing environment:
```bash
pip install vllm
pip install "nvidia-nccl-cu13>=2.30.4" --no-deps
```
The override / reinstall must happen before building DeepEP (for GIN device
headers) and must remain in place at runtime. You can verify with:
`install_python_libraries.sh` warns when the installed NCCL is too old. If
DeepEP was built against an older NCCL, upgrade NCCL and run the script again.
When NCCL is too old at runtime, vLLM logs the reason at startup and
`deepep_v2` fails with the same message. You can check with:
```bash
python -c "from vllm.utils.import_utils import has_deep_ep_v2; print(has_deep_ep_v2())"
python -c "from vllm.utils.import_utils import deep_ep_v2_unavailable_reason; print(deep_ep_v2_unavailable_reason() or 'deepep_v2 is available')"
```
## Usage
+23 -3
View File
@@ -78,10 +78,25 @@ mkdir -p "$WHEEL_DIR"
pushd "$WORKSPACE"
# install dependencies if not installed
UV_PIP_TARGET=()
if [ -z "$VIRTUAL_ENV" ]; then
uv pip install --system cmake torch ninja
else
uv pip install cmake torch ninja
UV_PIP_TARGET=(--system)
fi
uv pip install "${UV_PIP_TARGET[@]}" cmake torch ninja
# DeepEPv2 needs NCCL >= 2.30.4 at both build time and runtime, but PyTorch
# pins an older release.
DEEPEP_V2_MIN_NCCL="2.30.4"
NCCL_PACKAGE="nvidia-nccl-cu${CUDA_VERSION_MAJOR}"
NCCL_WARNING=""
NCCL_INSTALLED=$(uv pip show "${UV_PIP_TARGET[@]}" "$NCCL_PACKAGE" 2>/dev/null | sed -n 's/^Version: //p')
if [ -n "$NCCL_INSTALLED" ] && \
[ "$(printf '%s\n' "$DEEPEP_V2_MIN_NCCL" "$NCCL_INSTALLED" | sort -V | head -n1)" != "$DEEPEP_V2_MIN_NCCL" ]; then
NCCL_WARNING="WARNING: ${NCCL_PACKAGE} ${NCCL_INSTALLED} is installed, but the deepep_v2 backend
requires NCCL >= ${DEEPEP_V2_MIN_NCCL} when DeepEP is built and when it runs. To use deepep_v2, run
uv pip install \"${NCCL_PACKAGE}>=${DEEPEP_V2_MIN_NCCL}\" --no-deps
and then run this script again. See tools/ep_kernels/README.md."
echo "$NCCL_WARNING" >&2
fi
# fetch nvshmem
@@ -236,3 +251,8 @@ if [ "$MODE" = "wheel" ]; then
echo "All wheels written to $WHEEL_DIR"
ls -l "$WHEEL_DIR"
fi
# Repeat after the verbose build output, where the first copy is easy to miss.
if [ -n "$NCCL_WARNING" ]; then
echo "$NCCL_WARNING" >&2
fi