connects vLLM Recipes with vLLM's native config-based deployment and benchmark (#51308)
Signed-off-by: louie-tsai <louie.tsai@intel.com>
This commit is contained in:
@@ -8,6 +8,27 @@ toc_depth: 2
|
||||
|
||||
--8<-- "docs/getting_started/installation/gpu.md:pre-built-images"
|
||||
|
||||
## Run a vLLM Recipes configuration
|
||||
|
||||
[vLLM Recipes](https://recipes.vllm.ai/) can be converted into `config.yaml`
|
||||
and `env.sh`. See the
|
||||
[Recipes conversion tool README](../../tools/recipes/README.md) for usage.
|
||||
|
||||
For Docker, mount both files and source `env.sh` inside the container before
|
||||
starting vLLM:
|
||||
|
||||
```bash
|
||||
docker run --rm --gpus all \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
-v "$PWD/config.yaml:/recipe/config.yaml:ro" \
|
||||
-v "$PWD/env.sh:/recipe/env.sh:ro" \
|
||||
-p 8000:8000 \
|
||||
--ipc=host \
|
||||
--entrypoint /bin/bash \
|
||||
vllm/vllm-openai:latest \
|
||||
-lc 'source /recipe/env.sh && exec vllm serve --config /recipe/config.yaml'
|
||||
```
|
||||
|
||||
## Persist the compile cache across containers
|
||||
|
||||
Mounting the Hugging Face cache keeps model weights across containers, but each
|
||||
|
||||
Reference in New Issue
Block a user