[Frontend][Build] Add vllm download-kernels for FlashInfer precompiled kernels (#58765)
Signed-off-by: mgoin <mgoin64@gmail.com> Signed-off-by: Misha Goin <mgoin64@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
be0823357f
commit
29e4d2d42e
+16
-1
@@ -9,7 +9,7 @@ vllm --help
|
||||
Available Commands:
|
||||
|
||||
```bash
|
||||
vllm {chat,complete,serve,launch,bench,collect-env,run-batch,preload}
|
||||
vllm {chat,complete,serve,launch,bench,collect-env,download-kernels,run-batch,preload}
|
||||
```
|
||||
|
||||
## serve
|
||||
@@ -188,6 +188,21 @@ Start collecting environment information.
|
||||
vllm collect-env
|
||||
```
|
||||
|
||||
## download-kernels
|
||||
|
||||
Install the precompiled kernels that vLLM's dependencies would otherwise download
|
||||
or compile at startup. Run it after installing or upgrading vLLM. It currently
|
||||
installs FlashInfer's kernels and does nothing when FlashInfer is not installed.
|
||||
|
||||
```bash
|
||||
vllm download-kernels
|
||||
|
||||
# Print the install commands without running them
|
||||
vllm download-kernels --dry-run
|
||||
```
|
||||
|
||||
See [vllm download-kernels](./download-kernels.md) for the full reference of all available arguments.
|
||||
|
||||
## run-batch
|
||||
|
||||
Run batch prompts and write results to file.
|
||||
|
||||
@@ -41,6 +41,7 @@ If you are only developing vLLM's Python code, install vLLM using:
|
||||
|
||||
```bash
|
||||
VLLM_USE_PRECOMPILED=1 uv pip install -e .
|
||||
vllm download-kernels
|
||||
```
|
||||
|
||||
To rebuild only the Rust frontend binary:
|
||||
|
||||
@@ -23,14 +23,28 @@ Therefore, it is recommended to install vLLM with a **fresh new** environment. I
|
||||
|
||||
```bash
|
||||
uv pip install vllm --torch-backend=auto
|
||||
vllm download-kernels
|
||||
```
|
||||
|
||||
??? console "pip"
|
||||
```bash
|
||||
# Install vLLM with CUDA 12.9.
|
||||
pip install vllm --extra-index-url https://download.pytorch.org/whl/cu129
|
||||
vllm download-kernels
|
||||
```
|
||||
|
||||
`vllm download-kernels` installs [FlashInfer](https://docs.flashinfer.ai/)'s
|
||||
precompiled kernels for the installed FlashInfer and CUDA versions. Without them,
|
||||
vLLM downloads and compiles kernels at startup, which can add several minutes on
|
||||
Hopper and newer GPUs, **especially Blackwell**, and logs a warning. Run it in the
|
||||
same Python environment after every vLLM install or upgrade, and in custom
|
||||
container images. The kernels are tied to the FlashInfer version, and FlashInfer
|
||||
refuses to import when they do not match, so an upgrade needs them refreshed.
|
||||
With a PyTorch built for CUDA older than 12.9, only `flashinfer-cubin` is
|
||||
installed, because FlashInfer does not publish `flashinfer-jit-cache` for it.
|
||||
For CUDA overrides and nightly kernels, use the
|
||||
[FlashInfer CLI](https://docs.flashinfer.ai/cli.html#download-kernels) directly.
|
||||
|
||||
We recommend leveraging `uv` to [automatically select the appropriate PyTorch index at runtime](https://docs.astral.sh/uv/guides/integration/pytorch/#automatic-backend-selection) by inspecting the installed CUDA driver version via `--torch-backend=auto` (or `UV_TORCH_BACKEND=auto`). To select a specific backend (e.g., `cu130`), set `--torch-backend=cu130` (or `UV_TORCH_BACKEND=cu130`). If this doesn't work, try running `uv self update` to update `uv` first.
|
||||
|
||||
!!! note
|
||||
@@ -94,9 +108,10 @@ If you only need to change Python code, you can build and install vLLM without c
|
||||
git clone https://github.com/vllm-project/vllm.git
|
||||
cd vllm
|
||||
VLLM_USE_PRECOMPILED=1 uv pip install --editable . --torch-backend=auto
|
||||
vllm download-kernels
|
||||
```
|
||||
|
||||
This command will do the following:
|
||||
The install command will do the following:
|
||||
|
||||
1. Look for the current branch in your vLLM clone.
|
||||
1. Identify the corresponding base commit in the main branch.
|
||||
|
||||
@@ -25,8 +25,11 @@ This guide will help you quickly get started with vLLM to perform:
|
||||
uv venv --python 3.12 --seed
|
||||
source .venv/bin/activate
|
||||
uv pip install vllm --torch-backend=auto
|
||||
vllm download-kernels
|
||||
```
|
||||
|
||||
`vllm download-kernels` installs precompiled kernels that vLLM otherwise downloads and compiles at startup, which can add several minutes on Hopper and newer GPUs. See the [installation guide](installation/gpu.md) for details.
|
||||
|
||||
`uv` can [automatically select the appropriate PyTorch index at runtime](https://docs.astral.sh/uv/guides/integration/pytorch/#automatic-backend-selection) by inspecting the installed CUDA driver version via `--torch-backend=auto` (or `UV_TORCH_BACKEND=auto`). To select a specific backend (e.g., `cu126`), set `--torch-backend=cu126` (or `UV_TORCH_BACKEND=cu126`).
|
||||
|
||||
Another delightful way is to use `uv run` with `--with [dependency]` option, which allows you to run commands such as `vllm serve` without creating any permanent environment:
|
||||
@@ -329,5 +332,5 @@ Some of the available backend options include:
|
||||
- On AMD ROCm: `TRITON_ATTN`, `ROCM_ATTN`, `ROCM_AITER_FA`, `ROCM_AITER_UNIFIED_ATTN`, `TRITON_MLA`, `ROCM_AITER_MLA` or `ROCM_AITER_TRITON_MLA`.
|
||||
- On Intel XPU: `FLASH_ATTN`, `TRITON_ATTN`, `TRITON_MLA`, `XPU_MLA_SPARSE`, `TORCH_SDPA` or `TURBOQUANT`.
|
||||
|
||||
!!! warning
|
||||
There are no pre-built vllm wheels containing Flash Infer, so you must install it in your environment first. Refer to the [Flash Infer official docs](https://docs.flashinfer.ai/) or see [docker/Dockerfile](../../docker/Dockerfile) for instructions on how to install it.
|
||||
!!! tip
|
||||
FlashInfer is installed with vLLM, but its precompiled kernels are not. Run `vllm download-kernels` after installing or upgrading vLLM so they are not downloaded and compiled at startup.
|
||||
|
||||
@@ -178,6 +178,9 @@ LaunchSubcommandBase = auto_mock("vllm.entrypoints.cli.launch", "LaunchSubcomman
|
||||
launch_description = auto_mock("vllm.entrypoints.cli.launch", "DESCRIPTION")
|
||||
RenderSubcommand = auto_mock("vllm.entrypoints.cli.launch", "RenderSubcommand")
|
||||
PreloadSubcommand = auto_mock("vllm.entrypoints.cli.preload", "PreloadSubcommand")
|
||||
DownloadKernelsSubcommand = auto_mock(
|
||||
"vllm.entrypoints.cli.download_kernels", "DownloadKernelsSubcommand"
|
||||
)
|
||||
sweep_subcommands = auto_mock("vllm.benchmarks.sweep.cli", "SUBCOMMANDS")
|
||||
openai_cli_args = auto_mock("vllm.entrypoints.launchers", "cli_args")
|
||||
run_batch = auto_mock("vllm.entrypoints.launchers", "run_batch")
|
||||
@@ -332,6 +335,10 @@ pages = {
|
||||
"cli/complete.md": (create_parser(CompleteCommand.add_cli_args), False),
|
||||
"cli/run-batch.md": (create_parser(run_batch.make_arg_parser), True),
|
||||
"cli/preload.md": (create_parser(PreloadSubcommand.add_cli_args), True),
|
||||
"cli/download-kernels.md": (
|
||||
create_parser(DownloadKernelsSubcommand.add_cli_args),
|
||||
False,
|
||||
),
|
||||
"cli/launch/render.md": (create_parser(RenderSubcommand.add_cli_args), True),
|
||||
"cli/bench/latency.md": (create_parser(bench_latency.add_cli_args), True),
|
||||
# URL kept as `mm_processor` for back-compat; command name is `mm-processor`
|
||||
|
||||
Reference in New Issue
Block a user