Merge branch 'main' into patches
This commit is contained in:
+1
-1
@@ -136,7 +136,7 @@ vllm bench {forward, latency, serve, throughput}
|
||||
|
||||
### forward
|
||||
|
||||
Benchmark the model forward of exact batches: each case lists its requests as `[N*]Q[@K][c][:D]` (query length Q, KV length K including the query, D draft tokens). A prefill's prefix is a prefix-cache hit unless `c` marks a chunked prefill; they differ only on models with SWA bounded replay, which recompute the last window of a hit. Each request's prefix length varies by ±50% by default (`--kv-len-jitter`), identically on every DP rank. Each forward is captured as a one-shot CUDA graph; sampling and CPU overhead aren't measured. Use `torchrun`, one process per GPU, for DP/TP/EP. CUDA only for now.
|
||||
Benchmark the model forward of exact batches: each case lists its requests as `[N*]Q[@K][c][:D]` (query length Q, KV length K including the query, D draft tokens). A prefill's prefix is a prefix-cache hit unless `c` marks a chunked prefill; they differ only on models with SWA bounded replay, which recompute the last window of a hit. Each request's prefix length varies by ±50% by default (`--kv-len-jitter`), identically on every DP rank. Each forward is captured as a one-shot CUDA graph; sampling and CPU overhead aren't measured. Use `torchrun`, one process per GPU, for DP/TP/EP. CUDA only for now. Add `--profile-dir DIR` to write a Perfetto trace and a per-kernel GPU time summary of each case; `python -m vllm.benchmarks.forward.profile A.kernels.json [B.kernels.json]` prints a summary or compares two. Under Nsight Systems, `nsys profile --capture-range=cudaProfilerApi --capture-range-end=repeat --cuda-graph-trace=node` records just those replays, one report per case.
|
||||
|
||||
```bash
|
||||
vllm bench forward \
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
<!-- markdownlint-disable MD041 MD051 -->
|
||||
--8<-- [start:installation]
|
||||
|
||||
vLLM supports AMD GPUs with ROCm 6.3 or above. Pre-built wheels are available for ROCm 7.0 and ROCm 7.2.1.
|
||||
vLLM supports AMD GPUs with ROCm 6.3 or above. Pre-built wheels are available for ROCm 10.0 (the default) and ROCm 7.2.
|
||||
|
||||
#### Prebuilt Wheels
|
||||
|
||||
@@ -9,6 +9,8 @@ vLLM supports AMD GPUs with ROCm 6.3 or above. Pre-built wheels are available fo
|
||||
| ------------ | -------------- | ------------ | ----------------- | ------------------ |
|
||||
| `rocm700` | 3.12 | 7.0 | >= 2.35 | `0.14.0` to `0.18.0` |
|
||||
| `rocm721` | 3.12 | 7.2.1 | >= 2.35 | Nightly releases after commit `171775f306a333a9cf105bfd533bf3e113d401d9` |
|
||||
| `rocm72` | 3.12 | 7.2 | >= 2.39 | Nightly releases on the legacy ROCm 7.2 stack |
|
||||
| `rocm100` | 3.12 | 10.0 | >= 2.35 | Nightly releases (default) |
|
||||
|
||||
--8<-- [end:installation]
|
||||
--8<-- [start:requirements]
|
||||
@@ -40,12 +42,17 @@ If you need a different ROCm version or want to use an existing PyTorch installa
|
||||
source .venv/bin/activate
|
||||
```
|
||||
|
||||
To install the latest version of vLLM for Python 3.12, ROCm 7.0 and `glibc >= 2.35`.
|
||||
To install the latest version of vLLM for Python 3.12, ROCm 10.0 and `glibc >= 2.35`:
|
||||
|
||||
```bash
|
||||
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/ --upgrade
|
||||
```
|
||||
|
||||
!!! tip
|
||||
The ROCm 10.0 wheels install the ROCm SDK from pip ([TheRock](https://github.com/ROCm/TheRock)), including the GPU kernels for every supported architecture.
|
||||
|
||||
Wheels for the legacy ROCm 7.2 stack, which use a system ROCm installation, remain available at `https://wheels.vllm.ai/rocm/${VLLM_VERSION}/rocm72`.
|
||||
|
||||
!!! tip
|
||||
You can find out about which ROCm version the latest vLLM supports by checking the `vllm` package in index in extra-index-url <https://wheels.vllm.ai/rocm/> at [https://wheels.vllm.ai/rocm/vllm](https://wheels.vllm.ai/rocm/vllm) .
|
||||
|
||||
@@ -195,9 +202,9 @@ install it without re-running the full installation:
|
||||
and place it in `vllm/vllm-rs`.
|
||||
|
||||
If you see an error about a wheel not being found, the wheel for your base
|
||||
commit and ROCm patch version might not be available. Check the available
|
||||
variants under `https://wheels.vllm.ai/rocm/<commit>/`. For example, ROCm 7.2.1
|
||||
uses the `rocm721` variant.
|
||||
commit and ROCm version might not be available. Check the available
|
||||
variants under `https://wheels.vllm.ai/rocm/<commit>/`. Variants are named by ROCm
|
||||
major and minor version, e.g. ROCm 7.2.3 uses the `rocm72` variant.
|
||||
|
||||
There are more environment variables to control the behavior of Python-only
|
||||
build:
|
||||
@@ -208,7 +215,7 @@ build:
|
||||
- `VLLM_PRECOMPILED_WHEEL_COMMIT`: override the full commit hash used to
|
||||
download the pre-compiled wheel.
|
||||
- `VLLM_PRECOMPILED_WHEEL_VARIANT`: specify the ROCm variant subdirectory, e.g.,
|
||||
`rocm700` or `rocm721`. If not specified, the variant is auto-detected based
|
||||
`rocm72` or `rocm100`. If not specified, the variant is auto-detected based
|
||||
on your system's ROCm version. An explicitly specified variant must match the
|
||||
detected environment.
|
||||
|
||||
@@ -226,23 +233,34 @@ You can find more information about vLLM's wheels in
|
||||
#### Full build (with compilation) {#full-build}
|
||||
|
||||
!!! tip
|
||||
- If you found that the following installation step does not work for you, please refer to [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base). Dockerfile is a form of installation steps.
|
||||
- The steps below follow [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base), which pins the validated version of every component. If a step does not work for you, refer to it; a Dockerfile is a form of installation steps.
|
||||
- The legacy ROCm 7.2 stack uses a system ROCm installation instead; its steps are in [docker/Dockerfile.rocm_72_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_72_base).
|
||||
|
||||
0. Install prerequisites (skip if you are already in an environment/docker with the following installed):
|
||||
0. Install the ROCm SDK and PyTorch (skip if you are already in an environment or docker image with them installed, e.g. `vllm/vllm-openai-rocm:base-nightly`).
|
||||
|
||||
- [ROCm](https://rocm.docs.amd.com/en/latest/deploy/linux/index.html)
|
||||
- [PyTorch](https://pytorch.org/)
|
||||
|
||||
For installing PyTorch, you can start from a fresh docker image, e.g, `rocm/pytorch:rocm7.0_ubuntu22.04_py3.10_pytorch_release_2.8.0`, `rocm/pytorch-nightly`. If you are using docker image, you can skip to Step 3.
|
||||
|
||||
Alternatively, you can install PyTorch using PyTorch wheels. You can check PyTorch installation guide in PyTorch [Getting Started](https://pytorch.org/get-started/locally/). Example:
|
||||
ROCm 10.0 is installed from pip ([TheRock](https://github.com/ROCm/TheRock)), together with PyTorch and the GPU kernels for your GPU's architecture. In a Python 3.12 environment:
|
||||
|
||||
```bash
|
||||
# Install PyTorch
|
||||
pip uninstall torch -y
|
||||
pip install --no-cache-dir torch torchvision --index-url https://download.pytorch.org/whl/nightly/rocm7.0
|
||||
# Your GPU's architecture, e.g. gfx942 for MI300 or gfx950 for MI350
|
||||
export GPU_ARCH=$(uvx --from rocm-bootstrap rocm-bootstrap-detect --unique)
|
||||
export ROCM_INDEX=https://stable.repo.amd.com/rocm/whl-next/
|
||||
|
||||
pip install --index-url ${ROCM_INDEX} \
|
||||
"rocm[libraries,devel,device-${GPU_ARCH}]==10.0.0" \
|
||||
"torch[device-${GPU_ARCH}]==2.12.0+rocm10.0.0" \
|
||||
"torchvision[device-${GPU_ARCH}]==0.27.0+rocm10.0.0"
|
||||
rocm-sdk init
|
||||
|
||||
# Point builds at the pip-installed ROCm SDK
|
||||
export ROCM_PATH=$(rocm-sdk path --root)
|
||||
export ROCM_HOME=${ROCM_PATH} HIP_PATH=${ROCM_PATH}
|
||||
export PATH=$(rocm-sdk path --bin):${PATH}
|
||||
```
|
||||
|
||||
!!! note
|
||||
- The validated `ROCM_SDK_VERSION`, `TORCH_VERSION` and `TORCHVISION_VERSION` can be found in [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base).
|
||||
- To support several GPU architectures, add a `device-<arch>` extra for each of them.
|
||||
|
||||
1. Install [Triton for ROCm](https://github.com/ROCm/triton.git)
|
||||
|
||||
Install ROCm's Triton following the instructions from [ROCm/triton](https://github.com/ROCm/triton.git)
|
||||
@@ -252,8 +270,7 @@ You can find more information about vLLM's wheels in
|
||||
pip uninstall -y triton
|
||||
git clone https://github.com/ROCm/triton.git
|
||||
cd triton
|
||||
# git checkout $TRITON_BRANCH
|
||||
git checkout f9e5bf54
|
||||
git checkout $TRITON_BRANCH
|
||||
if [ ! -f setup.py ]; then cd python; fi
|
||||
python3 setup.py install
|
||||
cd ../..
|
||||
@@ -265,17 +282,14 @@ You can find more information about vLLM's wheels in
|
||||
|
||||
2. Optionally, if you choose to use CK flash attention, you can install [flash attention for ROCm](https://github.com/Dao-AILab/flash-attention.git)
|
||||
|
||||
Install ROCm's flash attention (v2.8.0) following the instructions from [ROCm/flash-attention](https://github.com/Dao-AILab/flash-attention#amd-rocm-support)
|
||||
|
||||
For example, for ROCm 7.0, suppose your gfx arch is `gfx942`. To get your gfx architecture, run `rocminfo |grep gfx`.
|
||||
Install ROCm's flash attention following the instructions from [ROCm/flash-attention](https://github.com/Dao-AILab/flash-attention#amd-rocm-support)
|
||||
|
||||
```bash
|
||||
git clone https://github.com/Dao-AILab/flash-attention.git
|
||||
cd flash-attention
|
||||
# git checkout $FA_BRANCH
|
||||
git checkout 0e60e394
|
||||
git checkout $FA_BRANCH
|
||||
git submodule update --init
|
||||
GPU_ARCHS="gfx942" python3 setup.py install
|
||||
GPU_ARCHS="${GPU_ARCH}" python3 setup.py install
|
||||
cd ..
|
||||
```
|
||||
|
||||
@@ -290,7 +304,8 @@ You can find more information about vLLM's wheels in
|
||||
cd aiter
|
||||
git checkout $AITER_BRANCH_OR_COMMIT
|
||||
git submodule sync; git submodule update --init --recursive
|
||||
python3 setup.py develop
|
||||
# Keep the Triton installed in step 1
|
||||
AITER_USE_SYSTEM_TRITON=1 python3 setup.py develop
|
||||
```
|
||||
|
||||
!!! note
|
||||
@@ -300,32 +315,34 @@ You can find more information about vLLM's wheels in
|
||||
4. Optionally, if you want to use MORI for EP or PD disaggregation, you can install [MORI](https://github.com/ROCm/mori) using the following steps:
|
||||
|
||||
```bash
|
||||
pip install "setuptools_scm>=6.2"
|
||||
git clone https://github.com/ROCm/mori.git
|
||||
cd mori
|
||||
git checkout $MORI_BRANCH_OR_COMMIT
|
||||
git submodule sync; git submodule update --init --recursive
|
||||
MORI_GPU_ARCHS="gfx942;gfx950" python3 setup.py install
|
||||
# The ROCm SDK keeps its CMake configs for system dependencies (e.g. NUMA) under rocm_sysdeps
|
||||
CMAKE_PREFIX_PATH="${ROCM_PATH}:${ROCM_PATH}/lib/rocm_sysdeps" \
|
||||
MORI_GPU_ARCHS="gfx942;gfx950" python3 setup.py install
|
||||
```
|
||||
|
||||
!!! note
|
||||
- You will need to config the `$MORI_BRANCH_OR_COMMIT` for your purpose.
|
||||
- The validated `$MORI_BRANCH_OR_COMMIT` can be found in the [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base).
|
||||
|
||||
5. Build vLLM. For example, vLLM on ROCM 7.0 can be built with the following steps:
|
||||
5. Build vLLM. For example, vLLM on ROCm 10.0 can be built with the following steps:
|
||||
|
||||
???+ console "Commands"
|
||||
|
||||
```bash
|
||||
pip install --upgrade pip
|
||||
|
||||
# Build & install AMD SMI
|
||||
pip install /opt/rocm/share/amd_smi
|
||||
# Install AMD SMI, which the ROCm SDK ships as sources
|
||||
SDK_CORE=$(python3 -c "import _rocm_sdk_core, os; print(os.path.dirname(_rocm_sdk_core.__file__))")
|
||||
pip install ${SDK_CORE}/share/amd_smi
|
||||
# It loads the SDK's libamd_smi relative to its own directory
|
||||
export PYTHONPATH=${SDK_CORE}/share/amd_smi
|
||||
|
||||
# Install dependencies
|
||||
pip install --upgrade numba \
|
||||
scipy \
|
||||
huggingface-hub \
|
||||
setuptools_scm
|
||||
pip install -r requirements/rocm.txt
|
||||
|
||||
# To build for a single architecture (e.g., MI300) for faster installation (recommended):
|
||||
@@ -352,8 +369,9 @@ You can find more information about vLLM's wheels in
|
||||
vLLM offers official Docker images for deployment.
|
||||
The images can be used to run OpenAI compatible server and are available on Docker Hub as [vllm/vllm-openai-rocm](https://hub.docker.com/r/vllm/vllm-openai-rocm/tags).
|
||||
|
||||
- `vllm/vllm-openai-rocm:latest` — stable release
|
||||
- `vllm/vllm-openai-rocm:nightly` — preview build from the latest development branch, use this if you want the latest features and fixes
|
||||
- `vllm/vllm-openai-rocm:latest` — stable release, built on ROCm 10.0 (also tagged `latest-rocm100`)
|
||||
- `vllm/vllm-openai-rocm:nightly` — preview build from the latest development branch on ROCm 10.0 (also tagged `nightly-rocm100`), use this if you want the latest features and fixes
|
||||
- `vllm/vllm-openai-rocm:latest-rocm72` / `nightly-rocm72` — the same builds on the legacy ROCm 7.2 stack, kept for a transition period
|
||||
|
||||
```bash
|
||||
docker run --rm \
|
||||
@@ -409,7 +427,7 @@ You can build and run vLLM from source via the provided [docker/Dockerfile.rocm]
|
||||
??? info "(Optional) Build an image with ROCm software stack"
|
||||
|
||||
Build a docker image from [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base) which setup ROCm software stack needed by the vLLM.
|
||||
**This step is optional as this rocm_base image is usually prebuilt and store at [Docker Hub](https://hub.docker.com/r/rocm/vllm-dev) under tag `rocm/vllm-dev:base` to speed up user experience.**
|
||||
**This step is optional as this rocm_base image is usually prebuilt and store at [Docker Hub](https://hub.docker.com/r/rocm/vllm-dev) under tag `rocm/vllm-dev:base` to speed up user experience.** For the legacy ROCm 7.2 stack, the base image is built from [docker/Dockerfile.rocm_72_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_72_base) and published as `vllm/vllm-openai-rocm:base-nightly-rocm72`.
|
||||
If you choose to build this rocm_base image yourself, the steps are as follows.
|
||||
|
||||
It is important that the user kicks off the docker build using buildkit. Either the user put `DOCKER_BUILDKIT=1` as environment variable when calling docker build command, or the user needs to set up buildkit in the docker daemon configuration `/etc/docker/daemon.json` as follows and restart the daemon:
|
||||
@@ -422,7 +440,7 @@ You can build and run vLLM from source via the provided [docker/Dockerfile.rocm]
|
||||
}
|
||||
```
|
||||
|
||||
To build vllm on ROCm 7.0 for MI200 and MI300 series, you can use the default:
|
||||
To build vllm on ROCm 10.0 for all supported GPUs, you can use the default:
|
||||
|
||||
```bash
|
||||
DOCKER_BUILDKIT=1 docker build \
|
||||
@@ -441,15 +459,15 @@ It is important that the user kicks off the docker build using buildkit. Either
|
||||
}
|
||||
```
|
||||
|
||||
[docker/Dockerfile.rocm](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm) uses ROCm 7.0 by default, but also supports ROCm 5.7, 6.0, 6.1, 6.2, 6.3, and 6.4, in older vLLM branches.
|
||||
[docker/Dockerfile.rocm](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm) uses ROCm 10.0 (installed from TheRock wheels) by default. The legacy ROCm 7.2 stack is available as [docker/Dockerfile.rocm_72](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_72) with [docker/Dockerfile.rocm_72_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_72_base); older vLLM branches support ROCm 5.7 through 7.0.
|
||||
It provides flexibility to customize the build of docker image using the following arguments:
|
||||
|
||||
- `BASE_IMAGE`: specifies the base image used when running `docker build`. The default value `rocm/vllm-dev:base` is an image published and maintained by AMD. It is being built using [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base)
|
||||
- `BASE_IMAGE`: specifies the base image used when running `docker build`. The default value `rocm/vllm-dev:base` is an image published and maintained by AMD. It is being built using [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base). For [docker/Dockerfile.rocm_72](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_72), the default is `vllm/vllm-openai-rocm:base-nightly-rocm72`
|
||||
- `ARG_PYTORCH_ROCM_ARCH`: Allows to override the gfx architecture values from the base docker image
|
||||
|
||||
Their values can be passed in when running `docker build` with `--build-arg` options.
|
||||
|
||||
To build vllm on ROCm 7.0 for MI200 and MI300 series, you can use the default (which build a docker image with `vllm serve` as entrypoint):
|
||||
To build vllm on ROCm 10.0 for all supported GPUs, you can use the default (which build a docker image with `vllm serve` as entrypoint):
|
||||
|
||||
```bash
|
||||
DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.rocm -t vllm/vllm-openai-rocm .
|
||||
|
||||
Reference in New Issue
Block a user