[Doc] Host model recipes in recipes/ (#61302)
Signed-off-by: esmeetu <jasonailu87@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
co-authored by
Claude Opus 5.5
parent
f98ae6a9fd
commit
b515648641
@@ -0,0 +1,178 @@
|
||||
meta:
|
||||
title: "Phi-4"
|
||||
slug: "phi-4"
|
||||
provider: "Microsoft"
|
||||
description: "Microsoft's Phi-4 family of lightweight dense models (mini-instruct, reasoning, multimodal) with 128K context"
|
||||
date_added: 2026-04-02
|
||||
date_updated: 2026-04-17
|
||||
tasks:
|
||||
- text
|
||||
- multimodal
|
||||
related_recipes: []
|
||||
hardware:
|
||||
h100: verified
|
||||
|
||||
model:
|
||||
model_id: "microsoft/Phi-4-mini-instruct"
|
||||
min_vllm_version: "0.7.0"
|
||||
architecture: dense
|
||||
parameter_count: "4B"
|
||||
active_parameters: "4B"
|
||||
context_length: 131072
|
||||
base_args: []
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
text_only:
|
||||
description: "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
|
||||
args:
|
||||
- "--language-model-only"
|
||||
encoder_parallel:
|
||||
description: "Run the vision encoder in data-parallel mode — avoids TP comm overhead on the small encoder. Mutually exclusive with text_only."
|
||||
args:
|
||||
- "--mm-encoder-tp-mode"
|
||||
- "data"
|
||||
|
||||
opt_in_features:
|
||||
- text_only
|
||||
- encoder_parallel
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 10
|
||||
description: "Phi-4-mini-instruct, conversational instruction-tuned"
|
||||
|
||||
mini_reasoning:
|
||||
model_id: "microsoft/Phi-4-mini-reasoning"
|
||||
precision: bf16
|
||||
vram_minimum_gb: 10
|
||||
description: "Optimized for reasoning tasks"
|
||||
|
||||
reasoning:
|
||||
model_id: "microsoft/Phi-4-reasoning"
|
||||
precision: bf16
|
||||
vram_minimum_gb: 30
|
||||
description: "Advanced reasoning capabilities (14B)"
|
||||
|
||||
multimodal:
|
||||
model_id: "microsoft/Phi-4-multimodal-instruct"
|
||||
min_vllm_version: "0.8.0"
|
||||
precision: bf16
|
||||
vram_minimum_gb: 16
|
||||
description: "Multimodal instruction-following (text + image)"
|
||||
extra_args:
|
||||
- "--trust-remote-code"
|
||||
|
||||
fp8:
|
||||
model_id: "RedHatAI/Phi-4-mini-instruct-FP8-dynamic"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 5
|
||||
description: "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
|
||||
hardware_overrides: {}
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
The Phi-4 family includes several lightweight, open models from Microsoft. These models
|
||||
can process text and, in some variants, multimodal inputs like images, to generate
|
||||
text outputs. They come with a 128K token context length.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Hardware: 1x GPU with >=16 GB VRAM (A100, L40S, H100, etc.)
|
||||
- vLLM >= 0.7.0 (the `multimodal` variant serves `Phi-4-multimodal-instruct`, which needs vLLM >= 0.8.0)
|
||||
|
||||
### Install vLLM
|
||||
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend auto
|
||||
```
|
||||
|
||||
## Launch commands
|
||||
|
||||
Phi-4-mini-instruct on a single GPU:
|
||||
|
||||
```bash
|
||||
vllm serve microsoft/Phi-4-mini-instruct \
|
||||
--host 0.0.0.0 \
|
||||
--max-model-len 4000
|
||||
```
|
||||
|
||||
Phi-4-multimodal-instruct (requires --trust-remote-code for LoRA modules):
|
||||
|
||||
```bash
|
||||
vllm serve microsoft/Phi-4-multimodal-instruct \
|
||||
--host 0.0.0.0 \
|
||||
--max-model-len 4000 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600)
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="microsoft/Phi-4-mini-instruct",
|
||||
messages=[{"role": "user", "content": "Write a short story"}],
|
||||
temperature=0.0,
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
Multimodal (requires Phi-4-multimodal-instruct):
|
||||
|
||||
```python
|
||||
response = client.chat.completions.create(
|
||||
model="microsoft/Phi-4-multimodal-instruct",
|
||||
messages=[{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "Describe this image."},
|
||||
{"type": "image_url", "image_url": {"url": "https://example.com/image.jpg"}},
|
||||
],
|
||||
}],
|
||||
)
|
||||
```
|
||||
|
||||
## Available Phi-4 Variants
|
||||
|
||||
- `microsoft/Phi-4-mini-instruct` — conversational instruction-tuned
|
||||
- `microsoft/Phi-4-mini-reasoning` — optimized for reasoning
|
||||
- `microsoft/Phi-4-reasoning` — advanced reasoning
|
||||
- `microsoft/Phi-4-multimodal-instruct` — multimodal (text + image)
|
||||
|
||||
## Benchmarking
|
||||
|
||||
```bash
|
||||
vllm bench serve \
|
||||
--model microsoft/Phi-4-mini-instruct \
|
||||
--dataset-name random \
|
||||
--random-input-len 2000 \
|
||||
--random-output-len 512 \
|
||||
--num-prompts 100
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- Multimodal variant fails to load: add `--trust-remote-code`.
|
||||
|
||||
## References
|
||||
|
||||
- [Phi-4-mini-instruct](https://huggingface.co/microsoft/Phi-4-mini-instruct)
|
||||
- [Phi-4-multimodal-instruct](https://huggingface.co/microsoft/Phi-4-multimodal-instruct)
|
||||
- [Phi-4-reasoning](https://huggingface.co/microsoft/Phi-4-reasoning)
|
||||
@@ -0,0 +1,103 @@
|
||||
meta:
|
||||
title: "Phi-4-multimodal-instruct"
|
||||
slug: "phi-4-multimodal-instruct"
|
||||
provider: "Microsoft"
|
||||
description: "Microsoft Phi-4 multimodal model supporting text, image, and audio inputs with text outputs."
|
||||
date_added: 2026-08-19
|
||||
date_updated: 2026-09-12
|
||||
tasks:
|
||||
- multimodal
|
||||
performance_headline: "Validated on Intel Xeon 6"
|
||||
hardware:
|
||||
xeon6: verified
|
||||
|
||||
model:
|
||||
model_id: "microsoft/Phi-4-multimodal-instruct"
|
||||
min_vllm_version: "0.8.0"
|
||||
architecture: dense
|
||||
parameter_count: "5.6B"
|
||||
active_parameters: "5.6B"
|
||||
context_length: 131072
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
base_env: {}
|
||||
|
||||
features: {}
|
||||
|
||||
opt_in_features: []
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 14
|
||||
description: "Full precision BF16"
|
||||
supported_hardware: [xeon6]
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
|
||||
hardware_overrides: {}
|
||||
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
Phi-4-multimodal-instruct processes text, image, and audio inputs and
|
||||
generates text. `--trust-remote-code` is retained because the model's
|
||||
multimodal LoRA components require it.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Hardware: Intel Xeon 6/Xeon 5 CPUs
|
||||
- vLLM >= 0.8.0
|
||||
|
||||
### Docker (Intel Xeon 6 CPUs)
|
||||
|
||||
```bash
|
||||
docker pull vllm/vllm-openai-cpu:latest-x86_64
|
||||
```
|
||||
|
||||
## Intel Xeon 6
|
||||
|
||||
```bash
|
||||
vllm serve microsoft/Phi-4-multimodal-instruct \
|
||||
--trust-remote-code \
|
||||
--tensor-parallel-size 1
|
||||
```
|
||||
|
||||
Docker (the image entrypoint is `vllm serve`):
|
||||
|
||||
```bash
|
||||
docker run \
|
||||
--privileged --ipc=host -p 8000:8000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai-cpu:latest-x86_64 microsoft/Phi-4-multimodal-instruct \
|
||||
--trust-remote-code \
|
||||
--tensor-parallel-size 1
|
||||
```
|
||||
|
||||
The example uses TP=1. Adjust TP/DP for the deployment topology;
|
||||
host-specific CPU placement is intentionally not hard-coded.
|
||||
|
||||
## Runtime and Platform Tuning
|
||||
|
||||
The following settings are intentionally not prescribed as portable CPU model
|
||||
defaults because they depend on the workload, hardware, or runtime environment:
|
||||
|
||||
- `--max-num-batched-tokens`: scheduler/throughput tuning.
|
||||
- `--max-num-seqs`: concurrency and scheduler-capacity tuning.
|
||||
- `--gpu-memory-utilization`: platform memory-budget tuning.
|
||||
- `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning.
|
||||
- `VLLM_ENGINE_ITERATION_TIMEOUT_S`: operational runtime timeout.
|
||||
|
||||
These settings may still appear in validated hardware-specific overrides. Tune
|
||||
them at deployment time based on platform resources, workload shape, and
|
||||
latency/throughput goals.
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/microsoft/Phi-4-multimodal-instruct)
|
||||
- [vLLM Phi-4 usage guide](https://docs.vllm.ai/projects/recipes/en/stable/Microsoft/Phi-4.html)
|
||||
- [vLLM CPU installation](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/)
|
||||
@@ -0,0 +1,98 @@
|
||||
meta:
|
||||
title: "Phi-4-reasoning"
|
||||
slug: "phi-4-reasoning"
|
||||
provider: "Microsoft"
|
||||
description: "Microsoft Phi-4 14B dense reasoning model with 32K context."
|
||||
date_added: 2026-08-19
|
||||
date_updated: 2026-09-12
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "Validated on Intel Xeon 6 with BF16 serving"
|
||||
hardware:
|
||||
xeon6: verified
|
||||
|
||||
model:
|
||||
model_id: "microsoft/Phi-4-reasoning"
|
||||
min_vllm_version: "0.4.2"
|
||||
architecture: dense
|
||||
parameter_count: "14B"
|
||||
active_parameters: "14B"
|
||||
context_length: 32768
|
||||
base_args: []
|
||||
base_env: {}
|
||||
|
||||
features: {}
|
||||
|
||||
opt_in_features: []
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 34
|
||||
description: "Full precision BF16"
|
||||
supported_hardware: [xeon6]
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
|
||||
hardware_overrides: {}
|
||||
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
Phi-4-reasoning is Microsoft's 14B dense reasoning model. Intel Xeon 6 CPU
|
||||
serving was validated with BF16.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Hardware: Intel Xeon 6/Xeon 5 CPUs
|
||||
- vLLM >= 0.4.2
|
||||
|
||||
### Docker (Intel Xeon 6 CPUs)
|
||||
|
||||
```bash
|
||||
docker pull vllm/vllm-openai-cpu:latest-x86_64
|
||||
```
|
||||
|
||||
## Intel Xeon 6
|
||||
|
||||
```bash
|
||||
vllm serve microsoft/Phi-4-reasoning \
|
||||
--tensor-parallel-size 1
|
||||
```
|
||||
|
||||
Docker (the image entrypoint is `vllm serve`):
|
||||
|
||||
```bash
|
||||
docker run \
|
||||
--privileged --ipc=host -p 8000:8000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai-cpu:latest-x86_64 microsoft/Phi-4-reasoning \
|
||||
--tensor-parallel-size 1
|
||||
```
|
||||
|
||||
The example uses TP=1. Adjust TP/DP for the deployment topology;
|
||||
host-specific CPU placement is intentionally not hard-coded.
|
||||
|
||||
## Runtime and Platform Tuning
|
||||
|
||||
The following settings are intentionally not prescribed as portable CPU model
|
||||
defaults because they depend on the workload, hardware, or runtime environment:
|
||||
|
||||
- `--max-num-batched-tokens`: scheduler/throughput tuning.
|
||||
- `--max-num-seqs`: concurrency and scheduler-capacity tuning.
|
||||
- `--gpu-memory-utilization`: platform memory-budget tuning.
|
||||
- `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning.
|
||||
- `VLLM_ENGINE_ITERATION_TIMEOUT_S`: operational runtime timeout.
|
||||
|
||||
These settings may still appear in validated hardware-specific overrides. Tune
|
||||
them at deployment time based on platform resources, workload shape, and
|
||||
latency/throughput goals.
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/microsoft/Phi-4-reasoning)
|
||||
- [vLLM CPU installation](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/)
|
||||
Reference in New Issue
Block a user