[UX][Frontend] Introduce vllm preload cli for fast restart (#56680)

Signed-off-by: Isotr0py <Isotr0py@outlook.com>
This commit is contained in:
Isotr0py
2026-09-25 06:56:39 +00:00
committed by GitHub
parent 5ff3bbfb08
commit 24928a4d62
8 changed files with 496 additions and 193 deletions
+28 -1
View File
@@ -9,7 +9,7 @@ vllm --help
Available Commands:
```bash
vllm {chat,complete,serve,launch,bench,collect-env,run-batch}
vllm {chat,complete,serve,launch,bench,collect-env,run-batch,preload}
```
## serve
@@ -212,6 +212,33 @@ vllm run-batch \
See [vllm run-batch](./run-batch.md) for the full reference of all available arguments.
## preload
Launch weight cache daemons (one per GPU) that hold the post-quantized,
TP-sharded weights in GPU memory and serve CUDA IPC handles to vLLM engines
over a Unix domain socket. Restarting engines then map the weights via
zero-copy IPC instead of reloading from disk, enabling fast engine restarts.
```bash
# Launch one daemon per GPU
vllm preload --model meta-llama/Llama-3.2-1B-Instruct --tensor-parallel-size 4
# Engines then load from the daemons
vllm serve meta-llama/Llama-3.2-1B-Instruct --tensor-parallel-size 4 \
--load-format ipc_cache
```
The daemon accepts the standard engine arguments (model, dtype, quantization,
tensor-parallel-size, ...) plus `--weight-cache-socket-dir` to override the
directory holding the per-GPU Unix sockets, and `--weight-cache-master-port` /
`--weight-cache-draft-master-port` to pin the daemon rendezvous ports for
multi-node and speculative-decoding setups. Tensor, expert and data
parallelism are supported; pipeline parallelism is rejected at launch.
See [Preload](../features/preload.md) for how it works, cache modes, and
limitations, and [vllm preload](./preload.md) for the full reference of all
available arguments.
## More Help
For detailed options of any subcommand, use: