connects vLLM Recipes with vLLM's native config-based deployment and benchmark (#51308)

Signed-off-by: louie-tsai <louie.tsai@intel.com>
This commit is contained in:
Louie Tsai
2026-08-11 13:53:51 +00:00
committed by GitHub
parent f863387250
commit 5b184f775a
6 changed files with 773 additions and 43 deletions
+2 -1
View File
@@ -1,7 +1,8 @@
# Benchmark Suites
vLLM provides comprehensive benchmarking tools for performance testing and evaluation:
vLLM provides comprehensive benchmarking tools for performance testing and evaluation.
- **[Benchmark CLI](./cli.md)**: `vllm bench` CLI tools and specialized benchmark scripts for interactive performance testing.
- **[Parameter Sweeps](./sweeps.md)**: Automate `vllm bench` runs for multiple configurations, useful for [optimization and tuning](../configuration/optimization.md).
- **[Performance Dashboard](./dashboard.md)**: Automated CI that publishes benchmarks on each commit.
- **[Benchmark using vLLM Recipes](../../tools/recipes/README.md)**: Use the Recipe conversion tool to generate `config.yaml` and `env.sh` from [vLLM Recipes](https://recipes.vllm.ai/) for Benchmarking.
+13
View File
@@ -29,6 +29,19 @@ To use the above config file:
vllm serve --config config.yaml
```
### Generate a configuration from vLLM Recipes
[vLLM Recipes](https://recipes.vllm.ai/) can be converted into `config.yaml`
and `env.sh`. See the
[Recipes conversion tool README](../../tools/recipes/README.md) for usage.
Source the generated environment before starting vLLM:
```bash
source env.sh
vllm serve --config config.yaml
```
!!! note
In case an argument is supplied simultaneously using command line and the config file, the value from the command line will take precedence.
The order of priorities is `command line > config file values > defaults`.
+21
View File
@@ -8,6 +8,27 @@ toc_depth: 2
--8<-- "docs/getting_started/installation/gpu.md:pre-built-images"
## Run a vLLM Recipes configuration
[vLLM Recipes](https://recipes.vllm.ai/) can be converted into `config.yaml`
and `env.sh`. See the
[Recipes conversion tool README](../../tools/recipes/README.md) for usage.
For Docker, mount both files and source `env.sh` inside the container before
starting vLLM:
```bash
docker run --rm --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-v "$PWD/config.yaml:/recipe/config.yaml:ro" \
-v "$PWD/env.sh:/recipe/env.sh:ro" \
-p 8000:8000 \
--ipc=host \
--entrypoint /bin/bash \
vllm/vllm-openai:latest \
-lc 'source /recipe/env.sh && exec vllm serve --config /recipe/config.yaml'
```
## Persist the compile cache across containers
Mounting the Hugging Face cache keeps model weights across containers, but each
+62 -42
View File
@@ -10,54 +10,74 @@
| [Intel® Xeon® 6 Processors](https://www.intel.com/content/www/us/en/products/details/processors/xeon.html) |
| [Intel® Xeon® 5 Processors](https://www.intel.com/content/www/us/en/products/docs/processors/xeon/5th-gen-xeon-scalable-processors.html) |
## Deploy from a vLLM Recipe
The **Recipe** column below links to validated Xeon 6 configurations when
available. Use the Recipe conversion tool to generate `config.yaml` and
`env.sh`; see the
[Recipes conversion tool README](../../../tools/recipes/README.md) for usage.
Load the generated environment before starting vLLM:
```bash
source env.sh
vllm serve --config config.yaml
```
## Recommended Models
### Text-only Language Models
| Model | Architecture | Supported |
| ------------------------------------ | ---------------------------------------- | --------- |
| unsloth/gpt-oss-20b | GptOssForCausalLM | ✅ |
| meta-llama/Llama-3.1-8B-Instruct | LlamaForCausalLM | ✅ |
| meta-llama/Llama-3.2-1B | LlamaForCausalLM | ✅ |
| meta-llama/Llama-3.2-3B-Instruct | LlamaForCausalLM | ✅ |
| meta-llama/Llama-3.3-70B-Instruct | LlamaForCausalLM | ✅ |
| RedHatAI/Meta-Llama-3.1-8B-quantized.w8a8 | LlamaForCausalLM | ✅ |
| RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8 | LlamaForCausalLM | ✅ |
| RedHatAI/Llama-3.2-1B-Instruct-quantized.w8a8 | LlamaForCausalLM | ✅ |
| RedHatAI/Llama-3.2-3B-Instruct-quantized.w8a8 | LlamaForCausalLM | ✅ |
| RedHatAI/DeepSeek-R1-Distill-Llama-70B-quantized.w8a8 | LlamaForCausalLM | ✅ |
| hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4 | LlamaForCausalLM | ✅ |
| AMead10/Llama-3.2-1B-Instruct-AWQ | LlamaForCausalLM | ✅ |
| AMead10/Llama-3.2-3B-Instruct-AWQ | LlamaForCausalLM | ✅ |
| TheBloke/TinyLlama-1.1B-Chat-v1.0-AWQ | LlamaForCausalLM | ✅ |
| TheBloke/TinyLlama-1.1B-Chat-v1.0-GPTQ | LlamaForCausalLM | ✅ |
| ibm-granite/granite-3.2-2b-instruct | GraniteForCausalLM | ✅ |
| Qwen/Qwen3-1.7B | Qwen3ForCausalLM | ✅ |
| Qwen/Qwen3-4B | Qwen3ForCausalLM | ✅ |
| Qwen/Qwen3-8B | Qwen3ForCausalLM | ✅ |
| Qwen/Qwen3-14B | Qwen3ForCausalLM | ✅ |
| Qwen/Qwen3-14B-AWQ | Qwen3ForCausalLM | ✅ |
| Qwen/Qwen3-30B-A3B | Qwen3MoeForCausalLM | ✅ |
| Qwen/QwQ-32B-AWQ | Qwen2ForCausalLM | ✅ |
| Qwen/Qwen1.5-0.5B-Chat-GPTQ-Int4 | Qwen2ForCausalLM | ✅ |
| RedHatAI/QwQ-32B-quantized.w8a8 | Qwen2ForCausalLM | ✅ |
| zai-org/glm-4-9b-hf | GLMForCausalLM | ✅ |
| google/gemma-7b | GemmaForCausalLM | ✅ |
| microsoft/Phi-4-reasoning | Phi3ForCausalLM | ✅ |
| TheBloke/Mistral-7B-Instruct-v0.2-AWQ | MistralForCausalLM | ✅ |
| Model | Architecture | Supported | Recipe |
| ------------------------------------ | ---------------------------------------- | --------- | ------ |
| unsloth/gpt-oss-20b | GptOssForCausalLM | ✅ | — |
| openai/gpt-oss-20b | GptOssForCausalLM | ✅ | — |
| meta-llama/Llama-3.1-8B-Instruct | LlamaForCausalLM | ✅ | [Xeon 6](https://recipes.vllm.ai/meta-llama/Llama-3.1-8B-Instruct?hardware=xeon6) |
| meta-llama/Llama-3.2-1B | LlamaForCausalLM | ✅ | — |
| meta-llama/Llama-3.2-3B-Instruct | LlamaForCausalLM | ✅ | — |
| meta-llama/Llama-3.3-70B-Instruct | LlamaForCausalLM | ✅ | [Xeon 6](https://recipes.vllm.ai/meta-llama/Llama-3.3-70B-Instruct?hardware=xeon6) |
| RedHatAI/Meta-Llama-3.1-8B-quantized.w8a8 | LlamaForCausalLM | ✅ | — |
| RedHatAI/Meta-Llama-3.1-8B-Instruct-quantized.w8a8 | LlamaForCausalLM | ✅ | — |
| RedHatAI/Llama-3.2-1B-Instruct-quantized.w8a8 | LlamaForCausalLM | ✅ | — |
| RedHatAI/Llama-3.2-3B-Instruct-quantized.w8a8 | LlamaForCausalLM | ✅ | — |
| RedHatAI/DeepSeek-R1-Distill-Llama-70B-quantized.w8a8 | LlamaForCausalLM | ✅ | — |
| hugging-quants/Meta-Llama-3.1-8B-Instruct-AWQ-INT4 | LlamaForCausalLM | ✅ | — |
| AMead10/Llama-3.2-1B-Instruct-AWQ | LlamaForCausalLM | ✅ | — |
| AMead10/Llama-3.2-3B-Instruct-AWQ | LlamaForCausalLM | ✅ | — |
| TheBloke/TinyLlama-1.1B-Chat-v1.0-AWQ | LlamaForCausalLM | ✅ | — |
| TheBloke/TinyLlama-1.1B-Chat-v1.0-GPTQ | LlamaForCausalLM | ✅ | — |
| ibm-granite/granite-3.2-2b-instruct | GraniteForCausalLM | ✅ | — |
| Qwen/Qwen3-1.7B | Qwen3ForCausalLM | ✅ | — |
| Qwen/Qwen3-4B | Qwen3ForCausalLM | ✅ | [Xeon 6](https://recipes.vllm.ai/Qwen/Qwen3-4B?hardware=xeon6) |
| Qwen/Qwen3-8B | Qwen3ForCausalLM | ✅ | — |
| Qwen/Qwen3-14B | Qwen3ForCausalLM | ✅ | — |
| Qwen/Qwen3-14B-FP8 | Qwen3ForCausalLM | ✅ | — |
| Qwen/Qwen3-14B-AWQ | Qwen3ForCausalLM | ✅ | — |
| Qwen/Qwen3-30B-A3B | Qwen3MoeForCausalLM | ✅ | — |
| Qwen/Qwen3-30B-A3B-Instruct-2507-FP8 | Qwen3MoeForCausalLM | ✅ | — |
| Qwen/QwQ-32B | Qwen2ForCausalLM | ✅ | — |
| Qwen/QwQ-32B-AWQ | Qwen2ForCausalLM | ✅ | — |
| Qwen/Qwen1.5-0.5B-Chat-GPTQ-Int4 | Qwen2ForCausalLM | ✅ | — |
| RedHatAI/QwQ-32B-quantized.w8a8 | Qwen2ForCausalLM | ✅ | — |
| zai-org/glm-4-9b-hf | GLMForCausalLM | ✅ | — |
| google/gemma-7b | GemmaForCausalLM | ✅ | — |
| microsoft/Phi-4-reasoning | Phi3ForCausalLM | ✅ | — |
| mistralai/Mistral-7B-Instruct-v0.2 | MistralForCausalLM | ✅ | — |
| TheBloke/Mistral-7B-Instruct-v0.2-AWQ | MistralForCausalLM | ✅ | — |
### Multimodal Language Models
| Model | Architecture | Supported |
| ------------------------------------ | ---------------------------------------- | --------- |
| meta-llama/Llama-4-Scout-17B-16E-Instruct | Llama4ForConditionalGeneration | ✅ |
| google/gemma-3-4b-it | Gemma3ForConditionalGeneration | ✅ |
| google/gemma-3-12b-it | Gemma3ForConditionalGeneration | ✅ |
| google/gemma-4-E4B-it | Gemma4ForConditionalGeneration | ✅ |
| google/gemma-4-E2B-it | Gemma4ForConditionalGeneration | ✅ |
| google/gemma-4-26B-A4B-it | Gemma4ForConditionalGeneration | ✅ |
| microsoft/Phi-4-multimodal-instruct | Phi4MMForCausalLM | ✅ |
| Qwen/Qwen2.5-VL-7B-Instruct | Qwen2VLForConditionalGeneration | ✅ |
| openai/whisper-large-v3 | WhisperForConditionalGeneration | ✅ |
| Model | Architecture | Supported | Recipe |
| ------------------------------------ | ---------------------------------------- | --------- | ------ |
| meta-llama/Llama-4-Scout-17B-16E-Instruct | Llama4ForConditionalGeneration | ✅ | [Xeon 6](https://recipes.vllm.ai/meta-llama/Llama-4-Scout-17B-16E-Instruct?hardware=xeon6) |
| google/gemma-3-4b-it | Gemma3ForConditionalGeneration | ✅ | — |
| google/gemma-3-12b-it | Gemma3ForConditionalGeneration | ✅ | — |
| google/gemma-4-E4B-it | Gemma4ForConditionalGeneration | ✅ | [Xeon 6](https://recipes.vllm.ai/Google/gemma-4-E4B-it?hardware=xeon6) |
| google/gemma-4-E2B-it | Gemma4ForConditionalGeneration | ✅ | [Xeon 6](https://recipes.vllm.ai/Google/gemma-4-E2B-it?hardware=xeon6) |
| google/gemma-4-26B-A4B-it | Gemma4ForConditionalGeneration | ✅ | [Xeon 6](https://recipes.vllm.ai/Google/gemma-4-26B-A4B-it?hardware=xeon6) |
| microsoft/Phi-4-multimodal-instruct | Phi4MMForCausalLM | ✅ | — |
| Qwen/Qwen2.5-VL-7B-Instruct | Qwen2VLForConditionalGeneration | ✅ | — |
| Qwen/Qwen3-VL-30B-A3B-Instruct | Qwen3VLMoeForConditionalGeneration | ✅ | — |
| openai/whisper-large-v3 | WhisperForConditionalGeneration | ✅ | — |
✅ Runs and optimized.