[CPU] Include vLLM Recipes tooling in release image to deploy models using vLLM Recipes (#58796)
Signed-off-by: louie-tsai <louie.tsai@intel.com>
This commit is contained in:
@@ -482,5 +482,6 @@ LABEL ai.vllm.build.python-version="${PYTHON_VERSION:-3.12}"
|
||||
# stage above adds examples/ for testing only, so without this the published
|
||||
# vllm-openai-cpu image would not ship examples/*.jinja.
|
||||
COPY examples examples
|
||||
COPY tools/recipes tools/recipes
|
||||
|
||||
ENTRYPOINT ["vllm", "serve"]
|
||||
|
||||
@@ -183,6 +183,40 @@ docker run \
|
||||
vllm/vllm-openai-cpu:latest-x86_64 <args...>
|
||||
```
|
||||
|
||||
### Serve with vLLM Recipes
|
||||
|
||||
The CPU image includes the [vLLM Recipes](https://recipes.vllm.ai/) deployment
|
||||
tool. Recipe-assisted serving queries the site at runtime, so the image can use
|
||||
the latest published recipe for the selected model and hardware.
|
||||
|
||||
To find the value for `--hardware`, open vLLM Recipes, select the model, and
|
||||
choose the target in the **Hardware** picker. The selected page URL also
|
||||
contains the hardware key; for example, `?hardware=xeon6` maps to
|
||||
`--hardware xeon6`.
|
||||
|
||||
Keep the normal image entrypoint for regular serving. For recipe-assisted
|
||||
serving, override it with `serve_with_recipe.sh`:
|
||||
|
||||
```bash
|
||||
docker run --rm \
|
||||
--shm-size=4g \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
-p 8000:8000 \
|
||||
--env "HF_TOKEN=<secret>" \
|
||||
--entrypoint tools/recipes/serve_with_recipe.sh \
|
||||
vllm/vllm-openai-cpu:latest-x86_64 \
|
||||
--model meta-llama/Llama-3.1-8B-Instruct \
|
||||
--hardware xeon6
|
||||
```
|
||||
|
||||
For Xeon 6, the script enables hardware detection automatically, generates
|
||||
`config.yml` and `env.sh`, and starts `vllm serve`. The image keeps
|
||||
`/vllm-workspace` as its working directory so relative recipe assets under
|
||||
`examples/` continue to resolve.
|
||||
|
||||
See the [Recipes tool documentation](../../../tools/recipes/README.md) for
|
||||
interactive discovery and additional options.
|
||||
|
||||
--8<-- [end:pre-built-images]
|
||||
--8<-- [start:build-image-from-source]
|
||||
|
||||
|
||||
@@ -12,17 +12,17 @@
|
||||
|
||||
## Deploy from a vLLM Recipe
|
||||
|
||||
The **Recipe** column below links to validated Xeon 6 configurations when
|
||||
available. Use the Recipe conversion tool to generate `config.yaml` and
|
||||
`env.sh`; see the
|
||||
[Recipes conversion tool README](../../../tools/recipes/README.md) for usage.
|
||||
The **Recipe** column below links to published Xeon 6 configurations when
|
||||
available. Open a recipe to review the latest deployment configuration for the
|
||||
model and hardware.
|
||||
|
||||
Load the generated environment before starting vLLM:
|
||||
For one-step deployment with the pre-built CPU image, see
|
||||
[Serve with vLLM Recipes](../../getting_started/installation/cpu.md#serve-with-vllm-recipes).
|
||||
The image includes the Recipes tool, which can retrieve the latest published
|
||||
recipe, apply Xeon hardware detection, and start `vllm serve`.
|
||||
|
||||
```bash
|
||||
source env.sh
|
||||
vllm serve --config config.yaml
|
||||
```
|
||||
For converter usage and interactive model/hardware discovery, see the
|
||||
[Recipes tool documentation](../../../tools/recipes/README.md).
|
||||
|
||||
## Recommended Models
|
||||
|
||||
|
||||
+20
-2
@@ -23,8 +23,9 @@ refine the initial configuration. Sweep tuning is an optional validation step.
|
||||
|
||||
## Getting Started
|
||||
|
||||
Use `serve_with_recipe.sh` to generate the recipe configuration and start vLLM
|
||||
in one step:
|
||||
Use `serve_with_recipe.sh` to fetch the selected deployment from
|
||||
[vLLM Recipes](https://recipes.vllm.ai/), generate the configuration, and start
|
||||
vLLM in one step:
|
||||
|
||||
```bash
|
||||
tools/recipes/serve_with_recipe.sh \
|
||||
@@ -32,6 +33,23 @@ tools/recipes/serve_with_recipe.sh \
|
||||
--hardware xeon6
|
||||
```
|
||||
|
||||
The converter queries `recipes.vllm.ai` at runtime, so it can use the latest
|
||||
published recipe for the requested model and hardware without rebuilding the
|
||||
vLLM image.
|
||||
|
||||
### Find the hardware name
|
||||
|
||||
Open [vLLM Recipes](https://recipes.vllm.ai/), select the model, and choose the
|
||||
target in the **Hardware** picker. The selected page URL contains the hardware
|
||||
key. For example, `?hardware=xeon6` maps to `--hardware xeon6`.
|
||||
|
||||
You can also run the converter without arguments for interactive model,
|
||||
hardware, and strategy discovery:
|
||||
|
||||
```bash
|
||||
python3 tools/recipes/recipe_json_to_vllm_config.py
|
||||
```
|
||||
|
||||
For `xeon6`, the script enables hardware detection automatically. Generated
|
||||
`config.yml` and `env.sh` files remain in the directory where the script was
|
||||
invoked.
|
||||
|
||||
Reference in New Issue
Block a user