[Frontend][Build] Add vllm download-kernels for FlashInfer precompiled kernels (#58765)
Signed-off-by: mgoin <mgoin64@gmail.com> Signed-off-by: Misha Goin <mgoin64@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
be0823357f
commit
29e4d2d42e
@@ -23,14 +23,28 @@ Therefore, it is recommended to install vLLM with a **fresh new** environment. I
|
||||
|
||||
```bash
|
||||
uv pip install vllm --torch-backend=auto
|
||||
vllm download-kernels
|
||||
```
|
||||
|
||||
??? console "pip"
|
||||
```bash
|
||||
# Install vLLM with CUDA 12.9.
|
||||
pip install vllm --extra-index-url https://download.pytorch.org/whl/cu129
|
||||
vllm download-kernels
|
||||
```
|
||||
|
||||
`vllm download-kernels` installs [FlashInfer](https://docs.flashinfer.ai/)'s
|
||||
precompiled kernels for the installed FlashInfer and CUDA versions. Without them,
|
||||
vLLM downloads and compiles kernels at startup, which can add several minutes on
|
||||
Hopper and newer GPUs, **especially Blackwell**, and logs a warning. Run it in the
|
||||
same Python environment after every vLLM install or upgrade, and in custom
|
||||
container images. The kernels are tied to the FlashInfer version, and FlashInfer
|
||||
refuses to import when they do not match, so an upgrade needs them refreshed.
|
||||
With a PyTorch built for CUDA older than 12.9, only `flashinfer-cubin` is
|
||||
installed, because FlashInfer does not publish `flashinfer-jit-cache` for it.
|
||||
For CUDA overrides and nightly kernels, use the
|
||||
[FlashInfer CLI](https://docs.flashinfer.ai/cli.html#download-kernels) directly.
|
||||
|
||||
We recommend leveraging `uv` to [automatically select the appropriate PyTorch index at runtime](https://docs.astral.sh/uv/guides/integration/pytorch/#automatic-backend-selection) by inspecting the installed CUDA driver version via `--torch-backend=auto` (or `UV_TORCH_BACKEND=auto`). To select a specific backend (e.g., `cu130`), set `--torch-backend=cu130` (or `UV_TORCH_BACKEND=cu130`). If this doesn't work, try running `uv self update` to update `uv` first.
|
||||
|
||||
!!! note
|
||||
@@ -94,9 +108,10 @@ If you only need to change Python code, you can build and install vLLM without c
|
||||
git clone https://github.com/vllm-project/vllm.git
|
||||
cd vllm
|
||||
VLLM_USE_PRECOMPILED=1 uv pip install --editable . --torch-backend=auto
|
||||
vllm download-kernels
|
||||
```
|
||||
|
||||
This command will do the following:
|
||||
The install command will do the following:
|
||||
|
||||
1. Look for the current branch in your vLLM clone.
|
||||
1. Identify the corresponding base commit in the main branch.
|
||||
|
||||
@@ -25,8 +25,11 @@ This guide will help you quickly get started with vLLM to perform:
|
||||
uv venv --python 3.12 --seed
|
||||
source .venv/bin/activate
|
||||
uv pip install vllm --torch-backend=auto
|
||||
vllm download-kernels
|
||||
```
|
||||
|
||||
`vllm download-kernels` installs precompiled kernels that vLLM otherwise downloads and compiles at startup, which can add several minutes on Hopper and newer GPUs. See the [installation guide](installation/gpu.md) for details.
|
||||
|
||||
`uv` can [automatically select the appropriate PyTorch index at runtime](https://docs.astral.sh/uv/guides/integration/pytorch/#automatic-backend-selection) by inspecting the installed CUDA driver version via `--torch-backend=auto` (or `UV_TORCH_BACKEND=auto`). To select a specific backend (e.g., `cu126`), set `--torch-backend=cu126` (or `UV_TORCH_BACKEND=cu126`).
|
||||
|
||||
Another delightful way is to use `uv run` with `--with [dependency]` option, which allows you to run commands such as `vllm serve` without creating any permanent environment:
|
||||
@@ -329,5 +332,5 @@ Some of the available backend options include:
|
||||
- On AMD ROCm: `TRITON_ATTN`, `ROCM_ATTN`, `ROCM_AITER_FA`, `ROCM_AITER_UNIFIED_ATTN`, `TRITON_MLA`, `ROCM_AITER_MLA` or `ROCM_AITER_TRITON_MLA`.
|
||||
- On Intel XPU: `FLASH_ATTN`, `TRITON_ATTN`, `TRITON_MLA`, `XPU_MLA_SPARSE`, `TORCH_SDPA` or `TURBOQUANT`.
|
||||
|
||||
!!! warning
|
||||
There are no pre-built vllm wheels containing Flash Infer, so you must install it in your environment first. Refer to the [Flash Infer official docs](https://docs.flashinfer.ai/) or see [docker/Dockerfile](../../docker/Dockerfile) for instructions on how to install it.
|
||||
!!! tip
|
||||
FlashInfer is installed with vLLM, but its precompiled kernels are not. Run `vllm download-kernels` after installing or upgrading vLLM so they are not downloaded and compiled at startup.
|
||||
|
||||
Reference in New Issue
Block a user