[UX][Frontend] Introduce vllm preload cli for fast restart (#56680)
Signed-off-by: Isotr0py <Isotr0py@outlook.com>
This commit is contained in:
+28
-1
@@ -9,7 +9,7 @@ vllm --help
|
||||
Available Commands:
|
||||
|
||||
```bash
|
||||
vllm {chat,complete,serve,launch,bench,collect-env,run-batch}
|
||||
vllm {chat,complete,serve,launch,bench,collect-env,run-batch,preload}
|
||||
```
|
||||
|
||||
## serve
|
||||
@@ -212,6 +212,33 @@ vllm run-batch \
|
||||
|
||||
See [vllm run-batch](./run-batch.md) for the full reference of all available arguments.
|
||||
|
||||
## preload
|
||||
|
||||
Launch weight cache daemons (one per GPU) that hold the post-quantized,
|
||||
TP-sharded weights in GPU memory and serve CUDA IPC handles to vLLM engines
|
||||
over a Unix domain socket. Restarting engines then map the weights via
|
||||
zero-copy IPC instead of reloading from disk, enabling fast engine restarts.
|
||||
|
||||
```bash
|
||||
# Launch one daemon per GPU
|
||||
vllm preload --model meta-llama/Llama-3.2-1B-Instruct --tensor-parallel-size 4
|
||||
|
||||
# Engines then load from the daemons
|
||||
vllm serve meta-llama/Llama-3.2-1B-Instruct --tensor-parallel-size 4 \
|
||||
--load-format ipc_cache
|
||||
```
|
||||
|
||||
The daemon accepts the standard engine arguments (model, dtype, quantization,
|
||||
tensor-parallel-size, ...) plus `--weight-cache-socket-dir` to override the
|
||||
directory holding the per-GPU Unix sockets, and `--weight-cache-master-port` /
|
||||
`--weight-cache-draft-master-port` to pin the daemon rendezvous ports for
|
||||
multi-node and speculative-decoding setups. Tensor, expert and data
|
||||
parallelism are supported; pipeline parallelism is rejected at launch.
|
||||
|
||||
See [Preload](../features/preload.md) for how it works, cache modes, and
|
||||
limitations, and [vllm preload](./preload.md) for the full reference of all
|
||||
available arguments.
|
||||
|
||||
## More Help
|
||||
|
||||
For detailed options of any subcommand, use:
|
||||
|
||||
Reference in New Issue
Block a user