[Frontend][Build] Add vllm download-kernels for FlashInfer precompiled kernels (#58765)

Signed-off-by: mgoin <mgoin64@gmail.com>
Signed-off-by: Misha Goin <mgoin64@gmail.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Misha Goin
2026-10-09 21:22:35 -07:00
committed by GitHub
co-authored by Claude Opus 5.5
parent be0823357f
commit 29e4d2d42e
15 changed files with 309 additions and 20 deletions
@@ -23,14 +23,28 @@ Therefore, it is recommended to install vLLM with a **fresh new** environment. I
```bash
uv pip install vllm --torch-backend=auto
vllm download-kernels
```
??? console "pip"
```bash
# Install vLLM with CUDA 12.9.
pip install vllm --extra-index-url https://download.pytorch.org/whl/cu129
vllm download-kernels
```
`vllm download-kernels` installs [FlashInfer](https://docs.flashinfer.ai/)'s
precompiled kernels for the installed FlashInfer and CUDA versions. Without them,
vLLM downloads and compiles kernels at startup, which can add several minutes on
Hopper and newer GPUs, **especially Blackwell**, and logs a warning. Run it in the
same Python environment after every vLLM install or upgrade, and in custom
container images. The kernels are tied to the FlashInfer version, and FlashInfer
refuses to import when they do not match, so an upgrade needs them refreshed.
With a PyTorch built for CUDA older than 12.9, only `flashinfer-cubin` is
installed, because FlashInfer does not publish `flashinfer-jit-cache` for it.
For CUDA overrides and nightly kernels, use the
[FlashInfer CLI](https://docs.flashinfer.ai/cli.html#download-kernels) directly.
We recommend leveraging `uv` to [automatically select the appropriate PyTorch index at runtime](https://docs.astral.sh/uv/guides/integration/pytorch/#automatic-backend-selection) by inspecting the installed CUDA driver version via `--torch-backend=auto` (or `UV_TORCH_BACKEND=auto`). To select a specific backend (e.g., `cu130`), set `--torch-backend=cu130` (or `UV_TORCH_BACKEND=cu130`). If this doesn't work, try running `uv self update` to update `uv` first.
!!! note
@@ -94,9 +108,10 @@ If you only need to change Python code, you can build and install vLLM without c
git clone https://github.com/vllm-project/vllm.git
cd vllm
VLLM_USE_PRECOMPILED=1 uv pip install --editable . --torch-backend=auto
vllm download-kernels
```
This command will do the following:
The install command will do the following:
1. Look for the current branch in your vLLM clone.
1. Identify the corresponding base commit in the main branch.
+5 -2
View File
@@ -25,8 +25,11 @@ This guide will help you quickly get started with vLLM to perform:
uv venv --python 3.12 --seed
source .venv/bin/activate
uv pip install vllm --torch-backend=auto
vllm download-kernels
```
`vllm download-kernels` installs precompiled kernels that vLLM otherwise downloads and compiles at startup, which can add several minutes on Hopper and newer GPUs. See the [installation guide](installation/gpu.md) for details.
`uv` can [automatically select the appropriate PyTorch index at runtime](https://docs.astral.sh/uv/guides/integration/pytorch/#automatic-backend-selection) by inspecting the installed CUDA driver version via `--torch-backend=auto` (or `UV_TORCH_BACKEND=auto`). To select a specific backend (e.g., `cu126`), set `--torch-backend=cu126` (or `UV_TORCH_BACKEND=cu126`).
Another delightful way is to use `uv run` with `--with [dependency]` option, which allows you to run commands such as `vllm serve` without creating any permanent environment:
@@ -329,5 +332,5 @@ Some of the available backend options include:
- On AMD ROCm: `TRITON_ATTN`, `ROCM_ATTN`, `ROCM_AITER_FA`, `ROCM_AITER_UNIFIED_ATTN`, `TRITON_MLA`, `ROCM_AITER_MLA` or `ROCM_AITER_TRITON_MLA`.
- On Intel XPU: `FLASH_ATTN`, `TRITON_ATTN`, `TRITON_MLA`, `XPU_MLA_SPARSE`, `TORCH_SDPA` or `TURBOQUANT`.
!!! warning
There are no pre-built vllm wheels containing Flash Infer, so you must install it in your environment first. Refer to the [Flash Infer official docs](https://docs.flashinfer.ai/) or see [docker/Dockerfile](../../docker/Dockerfile) for instructions on how to install it.
!!! tip
FlashInfer is installed with vLLM, but its precompiled kernels are not. Run `vllm download-kernels` after installing or upgrading vLLM so they are not downloaded and compiled at startup.