[Doc] Host model recipes in recipes/ (#61302)
Signed-off-by: esmeetu <jasonailu87@gmail.com> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Roy Wang
co-authored by
Claude Opus 5.5
Claude Opus 5.5
parent
f98ae6a9fd
commit
b515648641
@@ -5,9 +5,12 @@ description: Find vLLM deployment recipes and configs for LLMs and Hugging Face
|
||||
|
||||
# Model Recipes
|
||||
|
||||
Model serving recipes are maintained in the
|
||||
[models/ directory of vllm-project/recipes](https://github.com/vllm-project/recipes/tree/main/models).
|
||||
The published catalog is at
|
||||
Model serving recipes are maintained in this repository's
|
||||
[recipes/models/ directory](../../../recipes/models). Omni (image, video,
|
||||
audio) recipes stay in vllm-project/recipes for now (moving to vllm-omni),
|
||||
and Ascend NPU content is inline in these recipes (moving to vllm-ascend
|
||||
overlays). The published catalog, rendered by
|
||||
[vllm-project/recipes](https://github.com/vllm-project/recipes), is at
|
||||
[recipes.vllm.ai](https://recipes.vllm.ai).
|
||||
|
||||
## Usage notes
|
||||
@@ -146,6 +149,9 @@ uploads, untested hardware claims, or non-standard flags that only work for one
|
||||
setup. Before opening a pull request, confirm that the user ran the
|
||||
configuration and can share the results.
|
||||
|
||||
Follow the recipes repository's
|
||||
Edit the YAML under [recipes/models/](../../../recipes/models) in this
|
||||
repository, following the recipes repository's
|
||||
[add-recipe skill](https://github.com/vllm-project/recipes/blob/main/.agents/skills/add-recipe/SKILL.md)
|
||||
in a checkout of `vllm-project/recipes` or the user's intended fork.
|
||||
for the schema. Validate and preview it from a checkout of
|
||||
`vllm-project/recipes` pointed at this directory, as
|
||||
[recipes/README.md](../../../recipes/README.md) describes.
|
||||
|
||||
@@ -6,6 +6,8 @@
|
||||
dist
|
||||
vllm/*.so
|
||||
vllm/vllm-rs
|
||||
# Model recipes are rendered by vllm-project/recipes, never part of the image.
|
||||
/recipes
|
||||
|
||||
# Byte-compiled / optimized / DLL files
|
||||
__pycache__/
|
||||
|
||||
@@ -165,6 +165,9 @@
|
||||
.readthedocs.yaml @hmellor
|
||||
mkdocs.yaml @hmellor
|
||||
|
||||
# Model recipes
|
||||
/recipes @esmeetu
|
||||
|
||||
# Linting
|
||||
/tools/pre_commit/.markdownlint.yaml @hmellor
|
||||
.pre-commit-config.yaml @hmellor
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
name: Recipes
|
||||
|
||||
# Validate recipe changes against the renderer that publishes them
|
||||
# (recipes.vllm.ai). The check logic lives in vllm-project/recipes and tracks
|
||||
# its main branch, so a PR is held to the same validation that runs when
|
||||
# vllm-project/recipes picks up this directory after merge.
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
paths:
|
||||
- 'recipes/**'
|
||||
- '.github/workflows/recipes.yml'
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
validate:
|
||||
name: Recipes
|
||||
if: github.repository_owner == 'vllm-project'
|
||||
uses: vllm-project/recipes/.github/workflows/validate-recipes.yml@main
|
||||
with:
|
||||
source: vllm
|
||||
path: recipes
|
||||
+2
-1
@@ -201,7 +201,8 @@ extend-exclude = ["tests/models/fixtures/*", "tests/prompts/*", "tests/tokenizer
|
||||
"tests/v1/engine/test_fast_incdec_prefix_err.py", ".git/*", "csrc/cpu/sgl-kernels/*",
|
||||
"rust/src/chat/src/renderer/deepseek_v32/fixtures/*", "rust/src/parser/**",
|
||||
"rust/src/text/src/output/decoded.rs",
|
||||
"rust/src/tokenizer/src/incremental.rs"]
|
||||
"rust/src/tokenizer/src/incremental.rs",
|
||||
"recipes/models/*"]
|
||||
ignore-hidden = false
|
||||
|
||||
[tool.typos.default]
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
# vLLM model recipes
|
||||
|
||||
This directory holds the deployment recipes for models served by vLLM, one
|
||||
YAML per model at `models/<hf_org>/<hf_repo>.yaml`. The path mirrors the
|
||||
Hugging Face ID, so `models/Qwen/Qwen3.8-27B.yaml` renders at
|
||||
[recipes.vllm.ai/Qwen/Qwen3.8-27B](https://recipes.vllm.ai/Qwen/Qwen3.8-27B).
|
||||
|
||||
The site, the command builder, and the JSON API are built by
|
||||
[vllm-project/recipes](https://github.com/vllm-project/recipes). That
|
||||
repository owns the schema and the shared vocabulary (hardware profiles,
|
||||
serving strategies, KV-offload deployments) and pins this directory to a
|
||||
commit. Omni (image, video, audio) recipes stay in vllm-project/recipes for
|
||||
now (moving to vllm-omni), and Ascend NPU content is inline in these recipes
|
||||
(moving to vllm-ascend overlays).
|
||||
|
||||
## Editing a recipe
|
||||
|
||||
Start a new recipe from
|
||||
[`templates/recipe.yaml`](https://github.com/vllm-project/recipes/blob/main/templates/recipe.yaml).
|
||||
Fields are defined in
|
||||
[SCHEMA.md](https://github.com/vllm-project/recipes/blob/main/SCHEMA.md) and
|
||||
checked against [`recipe.schema.json`](https://recipes.vllm.ai/recipe.schema.json)
|
||||
on every build; an unknown or misspelled key is an error. For completion and
|
||||
inline errors in YAML-aware editors, start the file with
|
||||
`# yaml-language-server: $schema=https://recipes.vllm.ai/recipe.schema.json`.
|
||||
|
||||
Follow the recipes repository's
|
||||
[add-recipe skill](https://github.com/vllm-project/recipes/blob/main/.agents/skills/add-recipe/SKILL.md)
|
||||
for the schema and the validation steps. To validate a change exactly as this
|
||||
repository's `Recipes` PR check does, run from a checkout of
|
||||
`vllm-project/recipes`:
|
||||
|
||||
```bash
|
||||
node scripts/validate-source.mjs --source vllm --dir ~/vllm/recipes
|
||||
```
|
||||
|
||||
To preview the site with your changes, add a gitignored `sources.local.yaml`
|
||||
to that checkout with `overrides: { vllm: ~/vllm/recipes }` and run `pnpm dev`.
|
||||
This works once vllm-project/recipes reads this directory as its `vllm`
|
||||
source (vllm-project/recipes#1095).
|
||||
@@ -0,0 +1,4 @@
|
||||
# Recipe source manifest, read by the vllm-project/recipes renderer
|
||||
# (scripts/assemble-content.mjs). Its sources.yaml grants this directory's
|
||||
# role and scope; this file only declares the schema the recipes target.
|
||||
schema_version: 1
|
||||
@@ -0,0 +1,164 @@
|
||||
meta:
|
||||
title: "Seed-OSS-36B"
|
||||
slug: "seed-oss-36b"
|
||||
provider: "Seed (ByteDance)"
|
||||
description: "ByteDance Seed-OSS 36B dense model with unique 'thinking budget' control and 512K context support"
|
||||
date_added: 2025-08-28
|
||||
date_updated: 2026-04-17
|
||||
tasks:
|
||||
- text
|
||||
related_recipes: []
|
||||
|
||||
model:
|
||||
model_id: "ByteDance-Seed/Seed-OSS-36B-Instruct"
|
||||
min_vllm_version: "0.11.0"
|
||||
architecture: dense
|
||||
parameter_count: "36B"
|
||||
active_parameters: "36B"
|
||||
context_length: 524288
|
||||
base_args: []
|
||||
base_env: {}
|
||||
|
||||
dependencies:
|
||||
- note: "Pinned transformers commit required for Seed-OSS tokenizer compatibility"
|
||||
command: "uv pip install git+https://github.com/huggingface/transformers.git@56d68c6706ee052b445e1e476056ed92ac5eb383"
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Seed-OSS tool-call parser with automatic tool choice"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "seed_oss"
|
||||
|
||||
opt_in_features: []
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 86
|
||||
description: "Native BF16 weights on 8x GPU (TP=8)"
|
||||
extra_args:
|
||||
- "--tensor-parallel-size"
|
||||
- "8"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
|
||||
kv_offload_support:
|
||||
offloading_cpu: verified
|
||||
offloading_fs: verified
|
||||
|
||||
hardware_overrides:
|
||||
amd:
|
||||
extra_args: []
|
||||
extra_env:
|
||||
VLLM_ROCM_USE_AITER: "1"
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
Seed-OSS-36B is a dense language model from ByteDance Seed with a unique **thinking
|
||||
budget** feature for controlled reasoning, and up to 512K context. Users can choose
|
||||
tensor-parallel (low latency) or data-parallel (high throughput).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Hardware: 8x GPU recommended (TP=8); also runs on consumer hardware like RTX 3090
|
||||
- vLLM >= 0.11.0 (support may require main branch)
|
||||
- Latest transformers for compatibility
|
||||
|
||||
### Install vLLM (NVIDIA)
|
||||
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend auto
|
||||
uv pip install git+https://github.com/huggingface/transformers.git@56d68c6706ee052b445e1e476056ed92ac5eb383
|
||||
```
|
||||
|
||||
### Install vLLM (AMD ROCm)
|
||||
|
||||
```bash
|
||||
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
|
||||
```
|
||||
|
||||
## Launch commands
|
||||
|
||||
NVIDIA:
|
||||
|
||||
```bash
|
||||
vllm serve ByteDance-Seed/Seed-OSS-36B-Instruct \
|
||||
--host localhost --port 8000 \
|
||||
--tensor-parallel-size 8 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser seed_oss
|
||||
```
|
||||
|
||||
AMD:
|
||||
|
||||
```bash
|
||||
export VLLM_ROCM_USE_AITER=1
|
||||
vllm serve ByteDance-Seed/Seed-OSS-36B-Instruct \
|
||||
--tensor-parallel-size 8 \
|
||||
--enable-auto-tool-choice --tool-call-parser seed_oss \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
Tuning:
|
||||
- `--max-model-len=65536` works well; max is 512K.
|
||||
- `--max-num-batched-tokens=32768` for prompt-heavy; reduce to 8K–16K for latency.
|
||||
- `--gpu-memory-utilization=0.95` to maximize KV cache.
|
||||
|
||||
## Thinking Budget
|
||||
|
||||
Control the model's chain-of-thought length via `chat_template_kwargs`. Recommended
|
||||
values are multiples of 512 (512, 1K, 2K, 4K, 8K, 16K). Use `0` for direct answers.
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
|
||||
model = client.models.list().data[0].id
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model=model,
|
||||
messages=[
|
||||
{"role": "system", "content": "You are a helpful assistant"},
|
||||
{"role": "user", "content": "Janet's ducks lay 16 eggs per day..."},
|
||||
],
|
||||
extra_body={"chat_template_kwargs": {"thinking_budget": 512}},
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
curl:
|
||||
|
||||
```bash
|
||||
curl http://localhost:8000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "ByteDance-Seed/Seed-OSS-36B-Instruct",
|
||||
"messages": [{"role": "user", "content": "Explain quantum computing"}],
|
||||
"chat_template_kwargs": {"thinking_budget": 512}
|
||||
}'
|
||||
```
|
||||
|
||||
The model emits `<seed:think>` blocks with `<seed:cot_budget_reflect>` markers that
|
||||
report token usage against the budget.
|
||||
|
||||
## Benchmarking
|
||||
|
||||
```bash
|
||||
vllm bench serve \
|
||||
--backend vllm --model ByteDance-Seed/Seed-OSS-36B-Instruct \
|
||||
--endpoint /v1/completions --host localhost --port 8000 \
|
||||
--dataset-name random --random-input 800 --random-output 100 \
|
||||
--request-rate 2 --num-prompt 100
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [Seed-OSS-36B-Instruct on Hugging Face](https://huggingface.co/ByteDance-Seed/Seed-OSS-36B-Instruct)
|
||||
@@ -0,0 +1,358 @@
|
||||
meta:
|
||||
title: "DiffusionGemma 26B-A4B IT"
|
||||
slug: "diffusiongemma-26b-a4b-it"
|
||||
provider: "Google"
|
||||
description: "Google's DiffusionGemma — a block-diffusion language model built on Gemma 4's MoE backbone (26B total / 4B active). Generates tokens via iterative denoising over a fixed-length canvas rather than left-to-right autoregressive decoding, enabling higher throughput with parallel block generation."
|
||||
date_added: 2026-06-11
|
||||
date_updated: 2026-09-23
|
||||
tasks:
|
||||
- multimodal
|
||||
- text
|
||||
performance_headline: "Block-diffusion MoE — 26B total / 4B active, canvas-based parallel generation with ~1.9x throughput vs autoregressive baseline"
|
||||
related_recipes:
|
||||
- google/gemma-4-26B-A4B-it
|
||||
- google/gemma-4-31B-it
|
||||
hardware:
|
||||
h100: verified
|
||||
h200: verified
|
||||
dgx_spark_gb10: verified
|
||||
xeon6: verified
|
||||
arc_pro_b60: verified
|
||||
arc_pro_b70: verified
|
||||
|
||||
model:
|
||||
model_id: "google/diffusiongemma-26B-A4B-it"
|
||||
min_vllm_version: "0.24.0"
|
||||
docker_image: "vllm/vllm-openai:gemma"
|
||||
architecture: moe
|
||||
parameter_count: "26B"
|
||||
active_parameters: "4B"
|
||||
context_length: 262144
|
||||
base_args:
|
||||
- "--max-num-seqs"
|
||||
- "4"
|
||||
base_env: {}
|
||||
|
||||
dependencies: []
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with Gemma 4 parser and chat template"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "gemma4"
|
||||
- "--chat-template"
|
||||
- "examples/tool_chat_template_gemma4.jinja"
|
||||
reasoning:
|
||||
description: "Enable structured thinking/reasoning output"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "gemma4"
|
||||
|
||||
opt_in_features: []
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 64
|
||||
description: "Full BF16 — single 80 GB NVIDIA GPU (H100/H200). Requires --max-num-seqs 4 to avoid OOM from diffusion state buffers."
|
||||
supported_hardware: [h100, h200, dgx_spark_gb10, xeon6]
|
||||
fp8:
|
||||
model_id: "RedHatAI/diffusiongemma-26B-A4B-it-FP8-dynamic"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 32
|
||||
description: "FP8 (E4M3) weights with dynamic per-token activation quantization"
|
||||
supported_hardware: [h100, h200, dgx_spark_gb10]
|
||||
nvfp4:
|
||||
model_id: "RedHatAI/diffusiongemma-26B-A4B-it-NVFP4"
|
||||
precision: nvfp4
|
||||
label: "NVFP4 - RedHatAI"
|
||||
vram_minimum_gb: 24
|
||||
description: "NVFP4 (4-bit) quantized weights"
|
||||
supported_hardware: [dgx_spark_gb10]
|
||||
nvidia_nvfp4:
|
||||
model_id: "nvidia/diffusiongemma-26B-A4B-it-NVFP4"
|
||||
precision: nvfp4
|
||||
label: "NVFP4 - NVIDIA"
|
||||
vram_minimum_gb: 24
|
||||
description: "NVIDIA NVFP4 (4-bit) quantized weights"
|
||||
supported_hardware: [dgx_spark_gb10]
|
||||
hardware_overrides:
|
||||
dgx_spark_gb10:
|
||||
extra_args:
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.8"
|
||||
- "--max-num-seqs"
|
||||
- "8"
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "gemma4"
|
||||
- "--load-format"
|
||||
- "fastsafetensors"
|
||||
- "--attention-backend"
|
||||
- "TRITON_ATTN"
|
||||
- "--enable-prefix-caching"
|
||||
- "--diffusion-config"
|
||||
- '{"canvas_length":256}'
|
||||
- "--override-generation-config"
|
||||
- '{"max_new_tokens": null}'
|
||||
- "--reasoning-parser"
|
||||
- "gemma4"
|
||||
- "--default-chat-template-kwargs"
|
||||
- '{"enable_thinking": true}'
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
|
||||
hardware_overrides:
|
||||
cpu: {}
|
||||
xpu:
|
||||
extra_args:
|
||||
- "--enforce-eager"
|
||||
- "--attention-backend"
|
||||
- "TRITON_ATTN"
|
||||
- "--tensor-parallel-size"
|
||||
- "4"
|
||||
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
tp: 1
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[DiffusionGemma 26B-A4B](https://huggingface.co/google/diffusiongemma-26B-A4B-it) is a block-diffusion language model built on Gemma 4's MoE backbone. Instead of generating tokens one at a time (autoregressive), it generates blocks of 256 tokens simultaneously via iterative denoising over a fixed canvas — trading higher time-to-first-token for significantly higher per-request generation throughput.
|
||||
|
||||
### Key Architecture Features
|
||||
- **Block Diffusion**: Generates tokens in 256-token canvas blocks via iterative denoising (up to 48 steps per block).
|
||||
- **MoE Backbone**: Same Gemma 4 architecture — 128 fine-grained experts with top-8 routing, 26B total / 4B active parameters.
|
||||
- **Entropy-Bound Sampling**: Uses an entropy-based sampler (`diffusion_sampler: entropy_bound`, `entropy_bound: 0.1`) for the denoising process.
|
||||
- **Multimodal**: Supports text + images via the Gemma 4 vision encoder.
|
||||
- **Thinking Mode**: Supports structured reasoning via `<|channel>thought\n...<channel|>` delimiters.
|
||||
- **Function Calling**: Supports Gemma 4's tool-call protocol (works best in thinking mode).
|
||||
|
||||
### Important Deployment Notes
|
||||
|
||||
DiffusionGemma requires several specific flags that differ from standard Gemma 4:
|
||||
|
||||
| Flag | Why |
|
||||
|------|-----|
|
||||
| `--max-num-seqs 4` | The diffusion state buffers (`self_conditioning_probs`) pre-allocate `max_seqs × canvas_length × vocab_size` tensors. With Gemma's 262K vocab and canvas_length=256, higher values cause OOM. |
|
||||
| `--generation-config vllm` | The checkpoint's `generation_config.json` sets `max_tokens: 256`, which would override per-request limits. This flag ignores it. |
|
||||
| `--gpu-memory-utilization 0.85` | Leaves headroom for activation memory during denoising. |
|
||||
| `--hf-overrides '{"diffusion_sampler":"entropy_bound","diffusion_entropy_bound":0.1}'` | Configures the entropy-bound denoising sampler. |
|
||||
| `--diffusion-config '{"canvas_length": 256}'` | Sets the canvas block size for generation. |
|
||||
|
||||
## Prerequisites
|
||||
|
||||
DiffusionGemma requires a vLLM build with diffusion model support, available in the
|
||||
Gemma docker image:
|
||||
|
||||
```bash
|
||||
docker pull vllm/vllm-openai:gemma
|
||||
```
|
||||
|
||||
## Deployment
|
||||
|
||||
### Single GPU (H100/H200, BF16)
|
||||
|
||||
```bash
|
||||
docker run -itd --name diffusiongemma \
|
||||
--ipc=host --network host --gpus all \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai:gemma \
|
||||
--model google/diffusiongemma-26B-A4B-it \
|
||||
--max-model-len 262144 \
|
||||
--max-num-seqs 4 \
|
||||
--gpu-memory-utilization 0.85 \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
### Intel Xeon 6 (CPU, BF16)
|
||||
|
||||
DiffusionGemma runs on Intel Xeon 6 CPUs using vLLM's CPU backend with AMX BF16 acceleration.
|
||||
The generated example uses TP=1. Adjust `--tensor-parallel-size` based on the
|
||||
detected NUMA topology and deployment requirements.
|
||||
|
||||
```bash
|
||||
docker run \
|
||||
--privileged --ipc=host -p 8000:8000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai-cpu:latest-x86_64 google/diffusiongemma-26B-A4B-it \
|
||||
--max-num-seqs 4 \
|
||||
--tensor-parallel-size 1 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser gemma4 \
|
||||
--chat-template examples/tool_chat_template_gemma4.jinja \
|
||||
--reasoning-parser gemma4
|
||||
```
|
||||
|
||||
|
||||
### Intel XPU (B60 / B70, BF16)
|
||||
|
||||
Validated on 4x Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card)
|
||||
with the official vLLM XPU image `vllm/vllm-openai-xpu:latest`. `--enforce-eager`
|
||||
and `--attention-backend TRITON_ATTN` are required on this backend. Select Arc Pro
|
||||
B60/B70 in the hardware picker above for the exact command.
|
||||
|
||||
### Full-Featured Server (text + image + thinking + tools)
|
||||
|
||||
```bash
|
||||
docker run -itd --name diffusiongemma \
|
||||
--ipc=host --network host --gpus all \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai:gemma \
|
||||
--model google/diffusiongemma-26B-A4B-it \
|
||||
--max-model-len 262144 \
|
||||
--max-num-seqs 4 \
|
||||
--gpu-memory-utilization 0.85 \
|
||||
--mm-processor-kwargs '{"max_soft_tokens": 1120}' \
|
||||
--limit-mm-per-prompt '{"image": 7}' \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser gemma4 \
|
||||
--reasoning-parser gemma4 \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
### Jev-Style Structured Decisions (Nightly)
|
||||
|
||||
vLLM's structured-read support for DiffusionGemma can answer yes/no, multiple-choice,
|
||||
and scored questions with a confidence value. The example server seeds a canvas with
|
||||
the response template, leaves only the answer slots noisy, and reads a probability
|
||||
distribution from every slot after one denoising step. It exposes the Jev-compatible
|
||||
`/v1/systemone` endpoint; this is an example server rather than a standard vLLM endpoint.
|
||||
|
||||
Use the pinned nightly image that contains the merged structured-read support:
|
||||
|
||||
```bash
|
||||
docker run -d --name dgemma \
|
||||
--gpus '"device=0"' \
|
||||
--ipc=host \
|
||||
-p 8000:8000 \
|
||||
-p 8011:8011 \
|
||||
-v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
|
||||
vllm/vllm-openai:nightly-e9757321527ca1ecd514c07c1418dd2c53da3d19 \
|
||||
google/diffusiongemma-26B-A4B-it \
|
||||
--served-model-name dgemma \
|
||||
--diffusion-config '{"canvas_length":64}' \
|
||||
--max-logprobs 32 \
|
||||
--enable-prefix-caching
|
||||
```
|
||||
|
||||
Wait for the vLLM server to become healthy, then start the example server in the same
|
||||
container:
|
||||
|
||||
```bash
|
||||
until curl -fsS localhost:8000/health >/dev/null; do sleep 2; done
|
||||
|
||||
docker exec -d dgemma \
|
||||
python /vllm-workspace/examples/features/structured_diffusion/structured_server.py \
|
||||
--upstream http://127.0.0.1:8000 \
|
||||
--tokenizer google/diffusiongemma-26B-A4B-it \
|
||||
--canvas 64
|
||||
```
|
||||
|
||||
After the example server becomes healthy, send a Jev-style decision request:
|
||||
|
||||
```bash
|
||||
until curl -fsS localhost:8011/health >/dev/null; do sleep 2; done
|
||||
|
||||
curl -sS localhost:8011/v1/systemone \
|
||||
-H 'content-type: application/json' \
|
||||
-d '{
|
||||
"model": "jev-latest",
|
||||
"state": {"ticket": "Everything is down, demo at noon."},
|
||||
"questions": {
|
||||
"urgent": {
|
||||
"type": "noul",
|
||||
"instructions": "Needs a reply within the hour?"
|
||||
}
|
||||
}
|
||||
}'
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
### Text Generation
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||||
response = client.chat.completions.create(
|
||||
model="google/diffusiongemma-26B-A4B-it",
|
||||
messages=[{"role": "user", "content": "Write a poem about the ocean."}],
|
||||
max_tokens=512,
|
||||
temperature=0.7,
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Offline Inference
|
||||
|
||||
```python
|
||||
from vllm import LLM, SamplingParams
|
||||
from transformers import AutoTokenizer
|
||||
|
||||
model_path = "google/diffusiongemma-26B-A4B-it"
|
||||
tokenizer = AutoTokenizer.from_pretrained(model_path)
|
||||
|
||||
llm = LLM(
|
||||
model=model_path,
|
||||
max_num_seqs=4,
|
||||
gpu_memory_utilization=0.85,
|
||||
hf_overrides={
|
||||
"diffusion_sampler": "entropy_bound",
|
||||
"diffusion_entropy_bound": 0.1,
|
||||
},
|
||||
diffusion_config={"canvas_length": 256},
|
||||
)
|
||||
|
||||
messages = [{"role": "user", "content": "Explain quantum computing in simple terms."}]
|
||||
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
|
||||
outputs = llm.generate(prompt, SamplingParams(temperature=0.0, max_tokens=1024))
|
||||
|
||||
print(outputs[0].outputs[0].text)
|
||||
```
|
||||
|
||||
### Thinking Mode
|
||||
|
||||
Launch the server with `--reasoning-parser gemma4`, then enable per-request:
|
||||
|
||||
```python
|
||||
response = client.chat.completions.create(
|
||||
model="google/diffusiongemma-26B-A4B-it",
|
||||
messages=[{"role": "user", "content": "What is the derivative of x^3 * ln(x)?"}],
|
||||
max_tokens=32768,
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": True}},
|
||||
)
|
||||
```
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
Compared to the autoregressive Gemma 4 26B-A4B baseline (single H100, concurrency=1, SPEED-Bench):
|
||||
|
||||
| Metric | Gemma 4 26B-A4B (baseline) | DiffusionGemma 26B-A4B |
|
||||
|--------|---------------------------|------------------------|
|
||||
| Output TPS | 199 tok/s | 375 tok/s (**1.9×**) |
|
||||
| E2E Request Time (mean) | 2.87s | 0.88s (**3.3× faster**) |
|
||||
| TTFT (mean) | 53ms | 489ms (higher due to canvas denoising setup) |
|
||||
| Per-request Gen TPS (mean) | 205 tok/s | 1,282 tok/s (**6.2×**) |
|
||||
|
||||
The diffusion model trades higher time-to-first-token for significantly higher generation throughput. It generates fewer total tokens per prompt on average (shorter outputs) but completes requests much faster.
|
||||
|
||||
## Known Limitations
|
||||
|
||||
- **TTFT**: ~10× higher than autoregressive baseline — the model must denoise an entire canvas block before emitting the first token.
|
||||
- **`--max-num-seqs`**: Must be kept low (≤4) due to the large diffusion state tensors. Higher values cause CUDA OOM.
|
||||
- **Audio**: Not supported — the diffusion checkpoints do not include an audio encoder.
|
||||
- **Function Calling**: Works best in thinking mode. Non-thinking mode may answer directly instead of emitting tool calls.
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/google/diffusiongemma-26B-A4B-it)
|
||||
- [vLLM structured-read support for DiffusionGemma](https://github.com/vllm-project/vllm/pull/57250)
|
||||
- [djev-run Cloud Run deployment](https://github.com/taeold/djev-run)
|
||||
- [Gemma 4 26B-A4B (autoregressive baseline)](https://huggingface.co/google/gemma-4-26B-A4B-it)
|
||||
- [Gemma docs](https://ai.google.dev/gemma/docs)
|
||||
@@ -0,0 +1,316 @@
|
||||
meta:
|
||||
title: "Gemma 4 12B IT"
|
||||
slug: "gemma-4-12b-it"
|
||||
provider: "Google"
|
||||
description: "Google's encoder-free unified Gemma 4 dense model (12B) with native text, image, and audio, plus thinking mode and tool-use protocol."
|
||||
date_added: 2026-06-03
|
||||
date_updated: 2026-09-09
|
||||
tasks:
|
||||
- multimodal
|
||||
- text
|
||||
performance_headline: "Encoder-free unified multimodal model with audio, structured thinking, and function calling — runs on a single 40 GB+ GPU"
|
||||
related_recipes:
|
||||
- google/gemma-4-E4B-it
|
||||
- google/gemma-4-26B-A4B-it
|
||||
- google/gemma-4-31B-it
|
||||
hardware:
|
||||
h100: verified
|
||||
arc_pro_b60: verified
|
||||
arc_pro_b70: verified
|
||||
|
||||
model:
|
||||
model_id: "google/gemma-4-12B-it"
|
||||
min_vllm_version: "0.23.0"
|
||||
docker_image: "vllm/vllm-openai:gemma4-unified"
|
||||
install:
|
||||
docker: {}
|
||||
pip: {}
|
||||
architecture: dense
|
||||
parameter_count: "12B"
|
||||
active_parameters: "12B"
|
||||
context_length: 131072
|
||||
base_args: []
|
||||
base_env: {}
|
||||
|
||||
dependencies:
|
||||
- note: "Audio extras — only needed when serving the audio modality"
|
||||
command: 'uv pip install "vllm[audio]"'
|
||||
optional: true
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with Gemma 4 parser and chat template"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "gemma4"
|
||||
- "--chat-template"
|
||||
- "examples/tool_chat_template_gemma4.jinja"
|
||||
reasoning:
|
||||
description: "Enable structured thinking/reasoning output"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "gemma4"
|
||||
spec_decoding:
|
||||
description: "MTP speculative decoding for accelerated inference"
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"model":"google/gemma-4-12B-it-assistant","num_speculative_tokens":4}'
|
||||
|
||||
opt_in_features:
|
||||
- spec_decoding
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 29
|
||||
description: "Full BF16 — single 40 GB+ NVIDIA GPU"
|
||||
nvfp4:
|
||||
model_id: "RedHatAI/gemma-4-12B-it-NVFP4"
|
||||
precision: nvfp4
|
||||
vram_minimum_gb: 8
|
||||
description: "RedHatAI compressed-tensors NVFP4 quantization for Blackwell GPUs"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
w4a16:
|
||||
model_id: "google/gemma-4-12B-it-qat-w4a16-ct"
|
||||
precision: int4
|
||||
vram_minimum_gb: 9
|
||||
description: "W4A16 QAT (4-bit weights, 16-bit activations, group_size=32) via compressed-tensors — runs on any GPU"
|
||||
|
||||
fp8:
|
||||
model_id: "RedHatAI/gemma-4-12B-it-FP8-Dynamic"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 15
|
||||
description: "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
|
||||
hardware_overrides:
|
||||
xpu:
|
||||
extra_args:
|
||||
- "--enforce-eager"
|
||||
- "--attention-backend"
|
||||
- "flash_attn"
|
||||
- "--trust-remote-code"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.9"
|
||||
- "--no-enable-prefix-caching"
|
||||
- "--max-num-batched-tokens"
|
||||
- "8192"
|
||||
- "--max-model-len"
|
||||
- "5120"
|
||||
- "--tensor-parallel-size"
|
||||
- "2"
|
||||
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
tp: 1
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[Gemma 4](https://ai.google.dev/gemma/docs) is Google's most capable open model family, featuring a unified multimodal architecture that natively processes text, images, and audio. The **12B Unified** model is *encoder-free*: instead of dedicated vision/audio encoders, it projects raw image patches and audio waveforms directly into the decoder's embedding space through lightweight linear layers, so every modality flows into a single decoder-only transformer. Gemma 4 models support structured thinking/reasoning, function calling with a custom tool-use protocol, and dynamic vision resolution — all available through vLLM's OpenAI-compatible API.
|
||||
|
||||
### Key Features
|
||||
- **Encoder-free multimodality**: Text, image, and audio inputs handled by a single decoder-only transformer — no separate encoders to load, reducing multimodal latency.
|
||||
- **Dual Attention**: Alternating sliding-window (local, 1024 tokens) and global attention; the final layer is always global, with unified Keys/Values and Proportional RoPE on global layers for long-context efficiency.
|
||||
- **Thinking Mode**: Structured reasoning via `<|channel>thought\n...<channel|>` delimiters.
|
||||
- **Function Calling**: Custom tool-call protocol with dedicated special tokens.
|
||||
- **Native System Prompt Support**: First-class `system` role.
|
||||
|
||||
### Supported Variants
|
||||
|
||||
Dense:
|
||||
- `google/gemma-4-E2B-it` (effective 2B)
|
||||
- `google/gemma-4-E4B-it` (effective 4B)
|
||||
- `google/gemma-4-12B-it` (11.95B, encoder-free)
|
||||
- `google/gemma-4-31B-it` (31B)
|
||||
|
||||
MoE:
|
||||
- `google/gemma-4-26B-A4B-it` (26B total / 4B active)
|
||||
|
||||
> **Context length**: this recipe pins `context_length` to the value in `config.json` (`max_position_embeddings = 131072`, i.e. 128K). The Gemma 4 model card markets the 12B at up to 256K — raise `--max-model-len` only if your vLLM build accepts it.
|
||||
|
||||
TPU support is provided through [vLLM TPU](https://github.com/vllm-project/tpu-inference) with recipes for [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) and [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/).
|
||||
|
||||
> **Nightly required**: support for the encoder-free 12B Unified model landed in [vllm-project/vllm#44429](https://github.com/vllm-project/vllm/pull/44429) and has not yet shipped in a stable release. Use the pinned Docker image (recommended) or a nightly pip wheel.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### Docker (recommended)
|
||||
```bash
|
||||
docker pull vllm/vllm-openai:gemma4-unified # NVIDIA (CUDA 13; append -cu129 for CUDA 12.9 hosts)
|
||||
docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70)
|
||||
```
|
||||
TPU images are published separately by [vllm-project/tpu-inference](https://github.com/vllm-project/tpu-inference); see the Trillium / Ironwood tpu-recipes below for the pinned tag.
|
||||
|
||||
### pip (nightly, NVIDIA CUDA)
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --pre \
|
||||
--extra-index-url https://wheels.vllm.ai/nightly/cu129 \
|
||||
--extra-index-url https://download.pytorch.org/whl/cu129 \
|
||||
--index-strategy unsafe-best-match
|
||||
```
|
||||
|
||||
## Deployment Configurations
|
||||
|
||||
### Quick Start (Single GPU, BF16)
|
||||
The 12B fits comfortably on a single 40 GB+ GPU.
|
||||
```bash
|
||||
vllm serve google/gemma-4-12B-it \
|
||||
--max-model-len 32768 \
|
||||
--gpu-memory-utilization 0.90
|
||||
```
|
||||
|
||||
### Full-Featured Server Launch
|
||||
Enables text, image, audio, thinking, and tool calling:
|
||||
```bash
|
||||
vllm serve google/gemma-4-12B-it \
|
||||
--max-model-len 16384 \
|
||||
--gpu-memory-utilization 0.90 \
|
||||
--enable-auto-tool-choice \
|
||||
--reasoning-parser gemma4 \
|
||||
--tool-call-parser gemma4 \
|
||||
--chat-template examples/tool_chat_template_gemma4.jinja \
|
||||
--limit-mm-per-prompt '{"image": 4, "audio": 1}' \
|
||||
--async-scheduling \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
### Docker (NVIDIA)
|
||||
```bash
|
||||
docker run -itd --name gemma4 \
|
||||
--ipc=host --network host --shm-size 16G --gpus all \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai:gemma4-unified \
|
||||
--model google/gemma-4-12B-it \
|
||||
--max-model-len 32768 \
|
||||
--gpu-memory-utilization 0.90 \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
On CUDA 12.9 hosts, use the `vllm/vllm-openai:gemma4-unified-cu129` tag instead.
|
||||
|
||||
### Docker (Intel XPU B60 / B70, TP=2, Full-Featured)
|
||||
|
||||
Validated on 2× Intel Arc Pro B60 / B70 with the official vLLM XPU image.
|
||||
|
||||
```bash
|
||||
docker run --device /dev/dri \
|
||||
-v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
|
||||
--privileged --ipc=host -p 8000:8000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai-xpu:latest \
|
||||
google/gemma-4-12B-it \
|
||||
--enforce-eager \
|
||||
--attention-backend flash_attn \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000 \
|
||||
--trust-remote-code \
|
||||
--enable-auto-tool-choice \
|
||||
--reasoning-parser gemma4 \
|
||||
--tool-call-parser gemma4 \
|
||||
--chat-template examples/tool_chat_template_gemma4.jinja \
|
||||
--limit-mm-per-prompt '{"image": 4, "audio": 1}' \
|
||||
--async-scheduling \
|
||||
--gpu-memory-utilization 0.9 \
|
||||
--no-enable-prefix-caching \
|
||||
--max-num-batched-tokens 8192 \
|
||||
--max-model-len 5120 \
|
||||
--tensor-parallel-size 2
|
||||
```
|
||||
|
||||
### Docker (Cloud TPU — Trillium / Ironwood)
|
||||
TPU uses the separate `vllm/vllm-tpu` image (no pip wheel). Pull the tag specified by the upstream [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) or [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/) recipe, then run:
|
||||
```bash
|
||||
docker run -itd --name gemma4-tpu \
|
||||
--privileged --network host --shm-size 16G \
|
||||
-v /dev/shm:/dev/shm -e HF_TOKEN=$HF_TOKEN \
|
||||
vllm/vllm-tpu:latest \
|
||||
--model google/gemma-4-12B-it \
|
||||
--max-model-len 16384 \
|
||||
--disable_chunked_mm_input \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
### Text Generation
|
||||
```python
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||||
response = client.chat.completions.create(
|
||||
model="google/gemma-4-12B-it",
|
||||
messages=[{"role": "user", "content": "Write a poem about the ocean."}],
|
||||
max_tokens=512, temperature=0.7,
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Image Understanding
|
||||
```python
|
||||
response = client.chat.completions.create(
|
||||
model="google/gemma-4-12B-it",
|
||||
messages=[{"role": "user", "content": [
|
||||
{"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg"}},
|
||||
{"type": "text", "text": "Describe this image in detail."},
|
||||
]}],
|
||||
max_tokens=1024,
|
||||
)
|
||||
```
|
||||
|
||||
### Audio
|
||||
Requires `uv pip install "vllm[audio]"`.
|
||||
```bash
|
||||
vllm serve google/gemma-4-12B-it \
|
||||
--max-model-len 8192 \
|
||||
--limit-mm-per-prompt '{"image": 4, "audio": 1}'
|
||||
```
|
||||
|
||||
### Thinking Mode
|
||||
```bash
|
||||
vllm serve google/gemma-4-12B-it \
|
||||
--max-model-len 16384 \
|
||||
--enable-auto-tool-choice \
|
||||
--reasoning-parser gemma4 \
|
||||
--tool-call-parser gemma4 \
|
||||
--chat-template examples/tool_chat_template_gemma4.jinja
|
||||
```
|
||||
Enable thinking per-request via `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`, or default-on with `--default-chat-template-kwargs '{"enable_thinking": true}'`.
|
||||
|
||||
### Structured Outputs
|
||||
vLLM guided decoding constrains output to a JSON schema. Include semantic instructions in the system prompt — the model does not see schema descriptions.
|
||||
|
||||
## Configuration Tips
|
||||
|
||||
- Set `--max-model-len` to match your workload.
|
||||
- `--gpu-memory-utilization 0.90–0.95` maximizes KV cache.
|
||||
- Image-only workloads: pass `--limit-mm-per-prompt.audio 0`.
|
||||
- Text-only workloads: pass `--limit-mm-per-prompt '{"image": 0, "audio": 0}'` to skip MM profiling.
|
||||
- `--async-scheduling` improves throughput.
|
||||
- FP8 KV cache (`--kv-cache-dtype fp8`) saves ~50% KV memory.
|
||||
|
||||
## Speculative Decoding (MTP)
|
||||
|
||||
Enable the **Spec Decoding** feature toggle (above) or add `--speculative-config` manually to use MTP drafting with the [assistant model](https://huggingface.co/google/gemma-4-12B-it-assistant). Recommended `num_speculative_tokens`: 4–8 for this model.
|
||||
|
||||
> **Note:** MTP speculative decoding for Gemma 4 requires the nightly build — the pinned `vllm/vllm-openai:gemma4-unified` image or a nightly pip wheel. The standard `:latest` stable tag does not include this feature.
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/google/gemma-4-12B-it)
|
||||
- [vLLM support PR (#44429)](https://github.com/vllm-project/vllm/pull/44429)
|
||||
- [Assistant (MTP draft) model](https://huggingface.co/google/gemma-4-12B-it-assistant)
|
||||
- [Gemma docs](https://ai.google.dev/gemma/docs)
|
||||
- [vLLM Gemma 4 tool-call template](https://github.com/vllm-project/vllm/blob/main/examples/tool_chat_template_gemma4.jinja)
|
||||
- [TPU recipes: Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4)
|
||||
- [TPU recipes: Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/)
|
||||
@@ -0,0 +1,602 @@
|
||||
meta:
|
||||
title: "Gemma 4 26B-A4B IT"
|
||||
slug: "gemma-4-26b-a4b-it"
|
||||
provider: "Google"
|
||||
description: "Google's Gemma 4 MoE multimodal model (26B total / 4B active) with 128 fine-grained experts, top-8 routing, thinking mode, and tool-use protocol."
|
||||
date_added: 2026-04-02
|
||||
date_updated: 2026-09-27
|
||||
tasks:
|
||||
- multimodal
|
||||
- text
|
||||
performance_headline: "MoE multimodal model — 26B total / 4B active, 128 experts with top-8 routing"
|
||||
related_recipes:
|
||||
- google/gemma-4-E2B-it
|
||||
- google/gemma-4-E4B-it
|
||||
- google/gemma-4-31B-it
|
||||
platforms:
|
||||
- id: modal
|
||||
blurb: "Serverless deploy via the Gemma 4 Modal script bundled with this recipe."
|
||||
script: https://github.com/vllm-project/recipes/blob/main/Google/gemma4-modal.py
|
||||
install: |
|
||||
curl -O https://raw.githubusercontent.com/vllm-project/recipes/main/Google/gemma4-modal.py
|
||||
pip install modal
|
||||
modal setup
|
||||
modal deploy gemma4-modal.py
|
||||
modal run gemma4-modal.py
|
||||
hardware:
|
||||
h100: verified
|
||||
h200: verified
|
||||
mi300x: verified
|
||||
mi325x: verified
|
||||
mi355x: verified
|
||||
trillium: verified
|
||||
ironwood: verified
|
||||
xeon6: verified
|
||||
dgx_spark_gb10: verified
|
||||
dgx_station_gb300: verified
|
||||
rtx_pro_6000: verified
|
||||
rtx_5090: verified
|
||||
arc_pro_b60: verified
|
||||
arc_pro_b70: verified
|
||||
|
||||
model:
|
||||
model_id: "google/gemma-4-26B-A4B-it"
|
||||
min_vllm_version: "0.25.0"
|
||||
architecture: moe
|
||||
parameter_count: "26B"
|
||||
active_parameters: "4B"
|
||||
context_length: 131072
|
||||
base_args: []
|
||||
base_env: {}
|
||||
|
||||
dependencies: []
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with Gemma 4 parser and chat template"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "gemma4"
|
||||
- "--chat-template"
|
||||
- "examples/tool_chat_template_gemma4.jinja"
|
||||
hardware_overrides:
|
||||
dgx_spark_gb10:
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "gemma4"
|
||||
reasoning:
|
||||
description: "Enable structured thinking/reasoning output"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "gemma4"
|
||||
text_only:
|
||||
description: "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
|
||||
args:
|
||||
- "--language-model-only"
|
||||
spec_decoding:
|
||||
description: "MTP speculative decoding for accelerated inference"
|
||||
default_mode: mtp
|
||||
modes:
|
||||
mtp:
|
||||
label: "MTP"
|
||||
description: "Built-in Multi-Token Prediction, 4 draft tokens."
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"model":"google/gemma-4-26B-A4B-it-assistant","num_speculative_tokens":4}'
|
||||
# The single-GPU Blackwell desktops run the draft MoE on the Triton
|
||||
# backend — the fused CUTLASS path the drafter picks by default is not
|
||||
# built for GB10/GB300/RTX PRO in the validated images.
|
||||
hardware_overrides:
|
||||
dgx_spark_gb10: &gemma4_mtp_triton
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"mtp","model":"google/gemma-4-26B-A4B-it-assistant","num_speculative_tokens":4,"moe_backend":"triton"}'
|
||||
dgx_station_gb300: *gemma4_mtp_triton
|
||||
rtx_pro_6000: *gemma4_mtp_triton
|
||||
eagle3:
|
||||
label: "Eagle3"
|
||||
description: "Eagle3 speculative decoding with externally trained draft model; the served checkpoint is unchanged."
|
||||
variants:
|
||||
- default
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"model": "RedHatAI/gemma-4-26B-A4B-it-speculator.eagle3", "num_speculative_tokens": 3, "method": "eagle3"}'
|
||||
opt_in_features:
|
||||
- text_only
|
||||
- spec_decoding
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 64
|
||||
description: "Full BF16 — single 80 GB NVIDIA GPU or 1x MI300X/MI325X/MI350X/MI355X or 2x Xeon6/Xeon5 NUMA nodes"
|
||||
supported_hardware: [h100, h200, mi300x, mi325x, mi355x, trillium, ironwood, xeon6, dgx_spark_gb10, dgx_station_gb300, rtx_pro_6000, rtx_5090]
|
||||
hardware_overrides:
|
||||
dgx_spark_gb10:
|
||||
extra_env:
|
||||
VLLM_USE_V2_MODEL_RUNNER: "1"
|
||||
dgx_station_gb300:
|
||||
extra_env:
|
||||
VLLM_USE_V2_MODEL_RUNNER: "1"
|
||||
rtx_pro_6000:
|
||||
extra_env:
|
||||
VLLM_USE_V2_MODEL_RUNNER: "1"
|
||||
extra_args:
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.9"
|
||||
fp8:
|
||||
model_id: "RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 32
|
||||
description: "FP8 (E4M3) weights with dynamic per-token activation quantization — Hopper or Blackwell"
|
||||
supported_hardware: [h100, h200, dgx_spark_gb10, dgx_station_gb300, rtx_pro_6000, rtx_5090]
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
hardware_overrides:
|
||||
dgx_spark_gb10:
|
||||
extra_env:
|
||||
VLLM_USE_V2_MODEL_RUNNER: "1"
|
||||
dgx_station_gb300:
|
||||
extra_env:
|
||||
VLLM_USE_V2_MODEL_RUNNER: "1"
|
||||
rtx_pro_6000:
|
||||
extra_env:
|
||||
VLLM_USE_V2_MODEL_RUNNER: "1"
|
||||
extra_args:
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.9"
|
||||
rtx_5090:
|
||||
extra_env:
|
||||
VLLM_USE_V2_MODEL_RUNNER: "1"
|
||||
extra_args:
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.92"
|
||||
- "--max-model-len"
|
||||
- "131072"
|
||||
- "--max-num-seqs"
|
||||
- "1"
|
||||
nvfp4:
|
||||
model_id: "RedHatAI/gemma-4-26B-A4B-it-NVFP4"
|
||||
precision: nvfp4
|
||||
label: "NVFP4 - RedHatAI"
|
||||
vram_minimum_gb: 16
|
||||
description: "NVFP4 (4-bit) quantized weights — requires Blackwell (B200/B300)"
|
||||
supported_hardware: [dgx_spark_gb10, dgx_station_gb300, rtx_pro_6000, rtx_5090]
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
hardware_overrides:
|
||||
dgx_spark_gb10:
|
||||
extra_env:
|
||||
VLLM_USE_V2_MODEL_RUNNER: "1"
|
||||
dgx_station_gb300:
|
||||
extra_env:
|
||||
VLLM_USE_V2_MODEL_RUNNER: "1"
|
||||
rtx_pro_6000:
|
||||
extra_env:
|
||||
VLLM_USE_V2_MODEL_RUNNER: "1"
|
||||
extra_args:
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.9"
|
||||
rtx_5090:
|
||||
extra_env:
|
||||
VLLM_USE_V2_MODEL_RUNNER: "1"
|
||||
extra_args:
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.9"
|
||||
- "--max-num-batched-tokens"
|
||||
- "4096"
|
||||
- "--max-num-seqs"
|
||||
- "4"
|
||||
nvidia_nvfp4:
|
||||
model_id: "nvidia/Gemma-4-26B-A4B-NVFP4"
|
||||
precision: nvfp4
|
||||
label: "NVFP4 - NVIDIA"
|
||||
vram_minimum_gb: 16
|
||||
description: "NVIDIA ModelOpt NVFP4 (4-bit) quantized weights — validated on the single-GPU Blackwell desktops (DGX Spark GB10, DGX Station GB300, RTX PRO 6000)"
|
||||
supported_hardware: [dgx_spark_gb10, dgx_station_gb300, rtx_pro_6000]
|
||||
hardware_overrides:
|
||||
dgx_spark_gb10:
|
||||
# vLLM 0.24 and the generic nightly both fail on GB10 with this
|
||||
# checkpoint; the dated ARM64 community build is the tested runtime, so
|
||||
# the wheel tab is hidden here rather than offering a command that
|
||||
# won't start.
|
||||
docker_image: "eugr/spark-vllm:nightly-20260704"
|
||||
# This image's entrypoint is nvidia_entrypoint.sh, not `vllm serve`.
|
||||
docker_serve_command: "vllm serve"
|
||||
install:
|
||||
pip: false
|
||||
extra_env:
|
||||
VLLM_USE_V2_MODEL_RUNNER: "1"
|
||||
extra_args:
|
||||
# GB10 shares its 128 GB with the CPU, so leave headroom for the host.
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.8"
|
||||
- "--max-model-len"
|
||||
- "262144"
|
||||
- "--max-num-seqs"
|
||||
- "8"
|
||||
- "--max-num-batched-tokens"
|
||||
- "8192"
|
||||
- "--enable-prefix-caching"
|
||||
- "--load-format"
|
||||
- "fastsafetensors"
|
||||
dgx_station_gb300:
|
||||
extra_env:
|
||||
VLLM_USE_V2_MODEL_RUNNER: "1"
|
||||
extra_args:
|
||||
# 252 GB on one GB300 — full context at high utilization and a 64-way
|
||||
# batch.
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.92"
|
||||
- "--max-model-len"
|
||||
- "262144"
|
||||
- "--max-num-seqs"
|
||||
- "64"
|
||||
- "--max-num-batched-tokens"
|
||||
- "8192"
|
||||
- "--enable-chunked-prefill"
|
||||
- "--enable-prefix-caching"
|
||||
- "--load-format"
|
||||
- "fastsafetensors"
|
||||
rtx_pro_6000:
|
||||
extra_env:
|
||||
VLLM_USE_V2_MODEL_RUNNER: "1"
|
||||
extra_args:
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.92"
|
||||
- "--max-model-len"
|
||||
- "262144"
|
||||
- "--max-num-seqs"
|
||||
- "8"
|
||||
- "--max-num-batched-tokens"
|
||||
- "8192"
|
||||
- "--enable-prefix-caching"
|
||||
- "--load-format"
|
||||
- "fastsafetensors"
|
||||
rtx_5090:
|
||||
extra_env:
|
||||
VLLM_USE_V2_MODEL_RUNNER: "1"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.9"
|
||||
- "--max-num-batched-tokens"
|
||||
- "4096"
|
||||
- "--enable-prefix-caching"
|
||||
- "--enable-chunked-prefill"
|
||||
- "--async-scheduling"
|
||||
- "--max-num-seqs"
|
||||
- "4"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- single_node_tep
|
||||
- single_node_dep
|
||||
- multi_node_tp
|
||||
- multi_node_tep
|
||||
- multi_node_dep
|
||||
|
||||
hardware_overrides:
|
||||
xpu:
|
||||
extra_args:
|
||||
- "--enforce-eager"
|
||||
- "--attention-backend"
|
||||
- "flash_attn"
|
||||
- "--trust-remote-code"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.9"
|
||||
- "--no-enable-prefix-caching"
|
||||
- "--max-num-batched-tokens"
|
||||
- "8192"
|
||||
- "--max-model-len"
|
||||
- "5120"
|
||||
- "--tensor-parallel-size"
|
||||
- "4"
|
||||
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
tp: 1
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[Gemma 4 26B-A4B](https://huggingface.co/google/gemma-4-26B-A4B-it) is the Mixture-of-Experts member of Google's Gemma 4 family — 26B total parameters with only 4B active per token via 128 fine-grained experts and top-8 routing. It supports text + images natively, structured thinking, function calling, and dynamic vision resolution.
|
||||
|
||||
Intel Xeon 6 CPU support remains verified. Benchmark-specific TP/DP, CPU binding,
|
||||
sequence limits, and repro-only timeouts remain deployment choices.
|
||||
|
||||
### Key Features
|
||||
- **MoE**: 128 fine-grained experts with top-8 routing and custom GELU-activated FFN.
|
||||
- **Multimodal**: Text + images natively (video via custom frame-extraction pipeline). Audio is only supported on the smaller E2B/E4B variants.
|
||||
- **Dual Attention**: Alternating sliding-window (local) and global attention with different head dimensions.
|
||||
- **Thinking Mode**: Structured reasoning via `<|channel>thought\n...<channel|>` delimiters.
|
||||
- **Function Calling**: Custom tool-call protocol with dedicated special tokens.
|
||||
- **Dynamic Vision Resolution**: Per-request configurable vision token budget (70, 140, 280, 560, 1120 tokens).
|
||||
|
||||
TPU support is provided through [vLLM TPU](https://github.com/vllm-project/tpu-inference) with recipes for [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) and [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Hardware: Intel Xeon 6/Xeon 5 CPUs
|
||||
- vLLM >= 0.25.0
|
||||
|
||||
### pip (NVIDIA CUDA)
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --pre \
|
||||
--extra-index-url https://wheels.vllm.ai/nightly/cu129 \
|
||||
--extra-index-url https://download.pytorch.org/whl/cu129 \
|
||||
--index-strategy unsafe-best-match
|
||||
```
|
||||
|
||||
### pip (AMD ROCm: MI300X, MI325X, MI350X, MI355X)
|
||||
Requires Python 3.12, ROCm 7.2.1, glibc >= 2.35 (Ubuntu 22.04+).
|
||||
```bash
|
||||
uv venv --python 3.12
|
||||
source .venv/bin/activate
|
||||
uv pip install vllm --pre \
|
||||
--extra-index-url https://wheels.vllm.ai/rocm/nightly/rocm721 --upgrade
|
||||
```
|
||||
|
||||
### pip (Intel Xeon 6 CPUs)
|
||||
For Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions.
|
||||
|
||||
### Docker
|
||||
```bash
|
||||
docker pull vllm/vllm-openai:gemma4-0505-cu129 # NVIDIA Hopper (H100/H200, CUDA 12.9)
|
||||
docker pull vllm/vllm-openai:gemma4-0505-cu130 # NVIDIA Blackwell (B200/B300, CUDA 13.0)
|
||||
docker pull vllm/vllm-openai-rocm:latest # AMD
|
||||
docker pull vllm/vllm-openai-cpu:latest-x86_64 # For Intel Xeon 6
|
||||
docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70)
|
||||
```
|
||||
TPU images are published separately by [vllm-project/tpu-inference](https://github.com/vllm-project/tpu-inference); see the Trillium / Ironwood tpu-recipes below for the pinned tag.
|
||||
|
||||
## Deployment Configurations
|
||||
|
||||
### 26B MoE on 1x A100/H100 (BF16)
|
||||
```bash
|
||||
vllm serve google/gemma-4-26B-A4B-it \
|
||||
--max-model-len 32768 \
|
||||
--gpu-memory-utilization 0.90
|
||||
```
|
||||
|
||||
### Full-Featured Server Launch
|
||||
Enables text, image, thinking, and tool calling:
|
||||
```bash
|
||||
vllm serve google/gemma-4-26B-A4B-it \
|
||||
--max-model-len 16384 \
|
||||
--gpu-memory-utilization 0.90 \
|
||||
--enable-auto-tool-choice \
|
||||
--reasoning-parser gemma4 \
|
||||
--tool-call-parser gemma4 \
|
||||
--chat-template examples/tool_chat_template_gemma4.jinja \
|
||||
--limit-mm-per-prompt.image 4 \
|
||||
--async-scheduling \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
### Docker (NVIDIA)
|
||||
```bash
|
||||
docker run -itd --name gemma4-moe \
|
||||
--ipc=host --network host --shm-size 16G --gpus all \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai:gemma4-0505-cu129 \
|
||||
--model google/gemma-4-26B-A4B-it \
|
||||
--max-model-len 32768 \
|
||||
--gpu-memory-utilization 0.90 \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
Swap `vllm/vllm-openai:gemma4-0505-cu129` for `vllm/vllm-openai:gemma4-0505-cu130` on Blackwell (B200/B300).
|
||||
|
||||
### Docker (AMD MI300X/MI325X/MI350X/MI355X)
|
||||
```bash
|
||||
docker run -itd --name gemma4-rocm \
|
||||
--ipc=host --network=host --privileged \
|
||||
--cap-add=CAP_SYS_ADMIN --device=/dev/kfd --device=/dev/dri \
|
||||
--group-add=video --cap-add=SYS_PTRACE \
|
||||
--security-opt=seccomp=unconfined --shm-size 16G \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai-rocm:latest \
|
||||
--model google/gemma-4-26B-A4B-it \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
### Docker (Cloud TPU — Trillium / Ironwood)
|
||||
TPU uses the separate `vllm/vllm-tpu` image (no pip wheel). Pull the tag specified by the upstream [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) or [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/) recipe, then run:
|
||||
```bash
|
||||
docker run -itd --name gemma4-tpu \
|
||||
--privileged --network host --shm-size 16G \
|
||||
-v /dev/shm:/dev/shm -e HF_TOKEN=$HF_TOKEN \
|
||||
vllm/vllm-tpu:latest \
|
||||
--model google/gemma-4-26B-A4B-it \
|
||||
--tensor-parallel-size 8 \
|
||||
--max-model-len 16384 \
|
||||
--disable_chunked_mm_input \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
Trillium requires a 4-chip slice minimum; Ironwood runs on a single chip.
|
||||
|
||||
### Intel Xeon 6 Deployment via Docker
|
||||
|
||||
Launch the x86 CPU vLLM Docker container for `google/gemma-4-26B-A4B-it`:
|
||||
|
||||
```bash
|
||||
docker run \
|
||||
--privileged --ipc=host -p 8000:8000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai-cpu:latest-x86_64 google/gemma-4-26B-A4B-it \
|
||||
--tensor-parallel-size 1 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser gemma4 \
|
||||
--chat-template examples/tool_chat_template_gemma4.jinja \
|
||||
--reasoning-parser gemma4
|
||||
```
|
||||
|
||||
For additional Intel Xeon 6 deployment details, see the Intel Software Catalog entries for [Gemma 4 26B-A4B IT](https://aiswcatalog.intel.com/models/google-gemma-4-26b-a4b-it).
|
||||
|
||||
The following settings are intentionally not prescribed as portable CPU model
|
||||
defaults because they depend on the workload, hardware, or runtime environment:
|
||||
|
||||
- `--max-num-batched-tokens`: scheduler/throughput tuning.
|
||||
- `--max-num-seqs`: concurrency and scheduler-capacity tuning.
|
||||
- `--gpu-memory-utilization`: platform memory-budget tuning.
|
||||
- `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning.
|
||||
- `VLLM_ENGINE_ITERATION_TIMEOUT_S`: operational runtime timeout.
|
||||
|
||||
These settings may still appear in validated hardware-specific overrides. Tune
|
||||
them at deployment time based on platform resources, workload shape, and
|
||||
latency/throughput goals.
|
||||
|
||||
### Docker (Intel XPU B60 / B70, TP=4, Full-Featured)
|
||||
|
||||
Validated on 4x Intel Arc Pro B60 / B70 with the official vLLM XPU image.
|
||||
|
||||
```bash
|
||||
docker run --device /dev/dri \
|
||||
-v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
|
||||
--privileged --ipc=host -p 8000:8000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai-xpu:latest \
|
||||
google/gemma-4-26B-A4B-it \
|
||||
--enforce-eager \
|
||||
--attention-backend flash_attn \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000 \
|
||||
--trust-remote-code \
|
||||
--enable-auto-tool-choice \
|
||||
--reasoning-parser gemma4 \
|
||||
--tool-call-parser gemma4 \
|
||||
--chat-template examples/tool_chat_template_gemma4.jinja \
|
||||
--limit-mm-per-prompt.image 4 \
|
||||
--async-scheduling \
|
||||
--gpu-memory-utilization 0.9 \
|
||||
--no-enable-prefix-caching \
|
||||
--max-num-batched-tokens 8192 \
|
||||
--max-model-len 5120 \
|
||||
--tensor-parallel-size 4
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
### Text Generation
|
||||
```python
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||||
response = client.chat.completions.create(
|
||||
model="google/gemma-4-26B-A4B-it",
|
||||
messages=[{"role": "user", "content": "Write a poem about the ocean."}],
|
||||
max_tokens=512, temperature=0.7,
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Image Understanding
|
||||
```python
|
||||
response = client.chat.completions.create(
|
||||
model="google/gemma-4-26B-A4B-it",
|
||||
messages=[{"role": "user", "content": [
|
||||
{"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg"}},
|
||||
{"type": "text", "text": "Describe this image in detail."},
|
||||
]}],
|
||||
max_tokens=1024,
|
||||
)
|
||||
```
|
||||
|
||||
### Thinking Mode
|
||||
```bash
|
||||
vllm serve google/gemma-4-26B-A4B-it \
|
||||
--max-model-len 16384 \
|
||||
--reasoning-parser gemma4 \
|
||||
--tool-call-parser gemma4 \
|
||||
--enable-auto-tool-choice \
|
||||
--chat-template examples/tool_chat_template_gemma4.jinja
|
||||
```
|
||||
Enable per-request via `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`.
|
||||
|
||||
### Dynamic Vision Resolution
|
||||
Supported values: 70, 140, 280 (default), 560, 1120 tokens/image.
|
||||
```bash
|
||||
vllm serve google/gemma-4-26B-A4B-it \
|
||||
--mm-processor-kwargs '{"max_soft_tokens": 560}'
|
||||
```
|
||||
|
||||
## Configuration Tips
|
||||
|
||||
- Set `--max-model-len` to match your workload.
|
||||
- `--gpu-memory-utilization 0.90-0.95` maximizes KV cache.
|
||||
- Text-only workloads: `--limit-mm-per-prompt '{"image": 0, "audio": 0}'`.
|
||||
- `--async-scheduling` improves throughput.
|
||||
- FP8 KV cache (`--kv-cache-dtype fp8`) saves ~50% KV memory.
|
||||
- For MoE, TEP (tensor-expert parallelism) and DEP (data-expert parallelism) strategies scale better than pure TP at large node counts.
|
||||
|
||||
## Quantized Variants
|
||||
|
||||
Three pre-quantized checkpoints are available:
|
||||
|
||||
- [`RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic`](https://huggingface.co/RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic) — FP8 (E4M3) weights with dynamic per-token activation quantization; runs on Hopper and Blackwell.
|
||||
- [`RedHatAI/gemma-4-26B-A4B-it-NVFP4`](https://huggingface.co/RedHatAI/gemma-4-26B-A4B-it-NVFP4) — NVFP4 (4-bit) weights; requires Blackwell (B200/B300).
|
||||
- [`nvidia/Gemma-4-26B-A4B-NVFP4`](https://huggingface.co/nvidia/Gemma-4-26B-A4B-NVFP4) — NVIDIA ModelOpt NVFP4 (4-bit) weights; this is the checkpoint validated on the single-GPU Blackwell desktops below.
|
||||
|
||||
Pick them from the **Variant** dropdown above, or pass the repo id directly to `vllm serve`.
|
||||
|
||||
### NVFP4 on the Blackwell desktops
|
||||
|
||||
[`nvidia/Gemma-4-26B-A4B-NVFP4`](https://huggingface.co/nvidia/Gemma-4-26B-A4B-NVFP4)
|
||||
has been run end to end on DGX Spark (GB10), DGX Station (GB300) and RTX PRO
|
||||
6000, in each case on one GPU at TP1 with MTP drafting against
|
||||
[`google/gemma-4-26B-A4B-it-assistant`](https://huggingface.co/google/gemma-4-26B-A4B-it-assistant).
|
||||
|
||||
Select the **NVIDIA NVFP4** variant, enable **Spec Decoding**, and switch the
|
||||
hardware pill — memory utilization, batch size, prefill scheduling and the
|
||||
drafter's MoE backend all differ per box and are encoded per hardware, so the
|
||||
generated command is the tested one. Highlights:
|
||||
|
||||
- **DGX Spark (GB10)** — 128 GB shared with the CPU, so `--gpu-memory-utilization 0.8`
|
||||
and an 8-way batch. Serve from `eugr/spark-vllm:nightly-20260704`: vLLM 0.24
|
||||
and the generic nightly both fail on this checkpoint, so the pip tab is
|
||||
hidden on this hardware.
|
||||
- **DGX Station (GB300)** — 252 GB on one GPU, so `--gpu-memory-utilization 0.92`
|
||||
with chunked prefill and a 64-way batch.
|
||||
- **RTX PRO 6000** — 96 GB, `--gpu-memory-utilization 0.92` and an 8-way batch.
|
||||
|
||||
All three run the full 262,144-token context, above the 131,072 in
|
||||
`config.json`, and pass `"moe_backend":"triton"` in `--speculative-config`
|
||||
because the drafter's default fused MoE path is not built in these images.
|
||||
|
||||
The GB300 and RTX PRO 6000 runs used an internal NVIDIA nightly vLLM build.
|
||||
As noted under Speculative Decoding below, Gemma 4 MTP has not landed in a
|
||||
stable release, so the default `vllm/vllm-openai:latest` tag will not serve
|
||||
these commands with Spec Decoding enabled — use a nightly wheel or a nightly
|
||||
image on those two boxes.
|
||||
|
||||
## Throughput vs Latency
|
||||
|
||||
| Goal | TP | `--max-num-seqs` | Notes |
|
||||
|------|----|------------------|-------|
|
||||
| Max throughput | 1-2 | 256-512 | Best tok/s per GPU |
|
||||
| Min latency | 4-8 | 8-16 | Best TTFT/TPOT |
|
||||
| Balanced | 2 | 128 | Mixed workloads |
|
||||
|
||||
## Speculative Decoding (MTP)
|
||||
|
||||
Enable the **Spec Decoding** feature toggle (above) or add `--speculative-config` manually to use MTP drafting with the [assistant model](https://huggingface.co/google/gemma-4-26B-A4B-it-assistant). Recommended `num_speculative_tokens`: 4 for this model. See the [Gemma 4 usage guide](../../Google/Gemma4) for details and benchmarks.
|
||||
|
||||
> **Note:** MTP speculative decoding for Gemma 4 is only available on the vLLM nightly build — it has not yet landed in a stable release. Install via the nightly wheel (`uv pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly/cu129 …`) or use the `vllm/vllm-openai:gemma4-0505-cu129` / `vllm/vllm-openai:gemma4-0505-cu130` images above; the standard `:latest` stable tag does not include this feature.
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/google/gemma-4-26B-A4B-it)
|
||||
- [FP8 variant](https://huggingface.co/RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic)
|
||||
- [NVFP4 variant](https://huggingface.co/RedHatAI/gemma-4-26B-A4B-it-NVFP4)
|
||||
- [NVIDIA ModelOpt NVFP4 variant](https://huggingface.co/nvidia/Gemma-4-26B-A4B-NVFP4)
|
||||
- [Gemma docs](https://ai.google.dev/gemma/docs)
|
||||
- [vLLM Gemma 4 tool-call template](https://github.com/vllm-project/vllm/blob/main/examples/tool_chat_template_gemma4.jinja)
|
||||
- [TPU recipes: Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4)
|
||||
- [TPU recipes: Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/)
|
||||
- [Xeon 6 recipe: Gemma 4 26B-A4B IT](https://aiswcatalog.intel.com/models/google-gemma-4-26b-a4b-it)
|
||||
@@ -0,0 +1,425 @@
|
||||
meta:
|
||||
title: "Gemma 4 31B IT"
|
||||
slug: "gemma-4-31b-it"
|
||||
provider: "Google"
|
||||
description: "Google's unified multimodal Gemma 4 dense model (31B) with native text, image, and audio, plus thinking mode and tool-use protocol."
|
||||
date_added: 2026-04-02
|
||||
date_updated: 2026-09-09
|
||||
tasks:
|
||||
- multimodal
|
||||
- text
|
||||
performance_headline: "Unified multimodal model with structured thinking, function calling, dynamic vision resolution"
|
||||
related_recipes: []
|
||||
hardware:
|
||||
h100: verified
|
||||
mi300x: verified
|
||||
mi325x: verified
|
||||
mi355x: verified
|
||||
trillium: verified
|
||||
ironwood: verified
|
||||
arc_pro_b60: verified
|
||||
arc_pro_b70: verified
|
||||
|
||||
model:
|
||||
model_id: "google/gemma-4-31B-it"
|
||||
min_vllm_version: "0.19.1"
|
||||
architecture: dense
|
||||
parameter_count: "31B"
|
||||
active_parameters: "31B"
|
||||
context_length: 262144
|
||||
base_args: []
|
||||
base_env: {}
|
||||
|
||||
dependencies:
|
||||
- note: "Audio extras — only needed when serving the audio modality"
|
||||
command: 'uv pip install "vllm[audio]"'
|
||||
optional: true
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with Gemma 4 parser and chat template"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "gemma4"
|
||||
- "--chat-template"
|
||||
- "examples/tool_chat_template_gemma4.jinja"
|
||||
reasoning:
|
||||
description: "Enable structured thinking/reasoning output"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "gemma4"
|
||||
text_only:
|
||||
description: "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
|
||||
args:
|
||||
- "--language-model-only"
|
||||
spec_decoding:
|
||||
description: "MTP speculative decoding for accelerated inference"
|
||||
default_mode: mtp
|
||||
modes:
|
||||
mtp:
|
||||
label: "MTP"
|
||||
description: "Built-in Multi-Token Prediction, 4 draft tokens."
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"model":"google/gemma-4-31B-it-assistant","num_speculative_tokens":4}'
|
||||
dflash:
|
||||
label: "DFlash"
|
||||
description: "DFlash speculative decoding with an externally trained draft model; the served checkpoint is unchanged."
|
||||
variants:
|
||||
- default
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"model": "RedHatAI/gemma-4-31B-it-speculator.dflash", "num_speculative_tokens": 7, "method": "dflash"}'
|
||||
dspark:
|
||||
label: "DSpark"
|
||||
description: "DSpark drafting with an externally trained draft model; the served checkpoint is unchanged."
|
||||
variants:
|
||||
- default
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"model": "RedHatAI/gemma-4-31B-it-speculator.dspark", "num_speculative_tokens": 7, "method": "dspark"}'
|
||||
eagle3:
|
||||
label: "Eagle3"
|
||||
description: "Eagle3 speculative decoding with externally trained draft model; the served checkpoint is unchanged."
|
||||
variants:
|
||||
- default
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"model": "RedHatAI/gemma-4-31B-it-speculator.eagle3", "num_speculative_tokens": 3, "method": "eagle3"}'
|
||||
opt_in_features:
|
||||
- text_only
|
||||
- spec_decoding
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 75
|
||||
description: "Full BF16 — single 80 GB NVIDIA GPU or 1x MI300X/MI325X/MI350X/MI355X"
|
||||
fp8:
|
||||
model_id: "RedHatAI/gemma-4-31B-it-FP8-dynamic"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 38
|
||||
description: "FP8 (E4M3) linear weights with dynamic per-token activation quantization (vision tower stays BF16) — Hopper or Blackwell"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
nvfp4:
|
||||
model_id: "nvidia/gemma-4-31B-it-NVFP4"
|
||||
precision: nvfp4
|
||||
vram_minimum_gb: 19
|
||||
description: "NVIDIA NVFP4 quantized weights for Blackwell GPUs"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
redhat_nvfp4:
|
||||
model_id: "RedHatAI/gemma-4-31B-it-NVFP4"
|
||||
precision: nvfp4
|
||||
label: "NVFP4 (RedHat)"
|
||||
vram_minimum_gb: 19
|
||||
description: "RedHatAI compressed-tensors NVFP4 quantization for Blackwell GPUs"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
w4a16:
|
||||
model_id: "google/gemma-4-31B-it-qat-w4a16-ct"
|
||||
precision: int4
|
||||
vram_minimum_gb: 20
|
||||
description: "W4A16 QAT (4-bit weights, 16-bit activations, group_size=32) via compressed-tensors — runs on any GPU"
|
||||
|
||||
redhat_fp8:
|
||||
model_id: "RedHatAI/gemma-4-31B-it-FP8-block"
|
||||
precision: fp8
|
||||
label: "FP8 Block (RedHat)"
|
||||
vram_minimum_gb: 38
|
||||
description: "RedHatAI FP8 block quantization"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
|
||||
hardware_overrides:
|
||||
xpu:
|
||||
extra_args:
|
||||
- "--enforce-eager"
|
||||
- "--attention-backend"
|
||||
- "flash_attn"
|
||||
- "--trust-remote-code"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.9"
|
||||
- "--no-enable-prefix-caching"
|
||||
- "--max-num-batched-tokens"
|
||||
- "8192"
|
||||
- "--max-model-len"
|
||||
- "5120"
|
||||
- "--tensor-parallel-size"
|
||||
- "4"
|
||||
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
tp: 1
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[Gemma 4](https://ai.google.dev/gemma/docs) is Google's most capable open model family, featuring a unified multimodal architecture that natively processes text, images, and audio. Gemma 4 models support structured thinking/reasoning, function calling with a custom tool-use protocol, and dynamic vision resolution — all available through vLLM's OpenAI-compatible API.
|
||||
|
||||
### Key Features
|
||||
- **Multimodal**: Text + images natively (video via custom frame-extraction pipeline). The smaller E2B and E4B models also support audio.
|
||||
- **MoE variant**: 128 fine-grained experts with top-8 routing and custom GELU-activated FFN (Gemma 4 26B-A4B).
|
||||
- **Dual Attention**: Alternating sliding-window (local) and global attention with different head dimensions.
|
||||
- **Thinking Mode**: Structured reasoning via `<|channel>thought\n...<channel|>` delimiters.
|
||||
- **Function Calling**: Custom tool-call protocol with dedicated special tokens.
|
||||
- **Dynamic Vision Resolution**: Per-request configurable vision token budget (70, 140, 280, 560, 1120 tokens).
|
||||
|
||||
### Supported Variants
|
||||
|
||||
Dense:
|
||||
- `google/gemma-4-E2B-it` (effective 2B)
|
||||
- `google/gemma-4-E4B-it` (effective 4B)
|
||||
- `google/gemma-4-31B-it` (31B)
|
||||
|
||||
MoE:
|
||||
- `google/gemma-4-26B-A4B-it` (26B total / 4B active)
|
||||
|
||||
TPU support is provided through [vLLM TPU](https://github.com/vllm-project/tpu-inference) with recipes for [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) and [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### pip (NVIDIA CUDA)
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --pre \
|
||||
--extra-index-url https://wheels.vllm.ai/nightly/cu129 \
|
||||
--extra-index-url https://download.pytorch.org/whl/cu129 \
|
||||
--index-strategy unsafe-best-match
|
||||
```
|
||||
|
||||
### pip (AMD ROCm: MI300X, MI325X, MI350X, MI355X)
|
||||
Requires Python 3.12, ROCm 7.2.1, glibc >= 2.35 (Ubuntu 22.04+).
|
||||
```bash
|
||||
uv venv --python 3.12
|
||||
source .venv/bin/activate
|
||||
uv pip install vllm --pre \
|
||||
--extra-index-url https://wheels.vllm.ai/rocm/nightly/rocm721 --upgrade
|
||||
```
|
||||
|
||||
### Docker
|
||||
```bash
|
||||
docker pull vllm/vllm-openai:gemma4-0505-cu129 # NVIDIA Hopper (H100/H200, CUDA 12.9)
|
||||
docker pull vllm/vllm-openai:gemma4-0505-cu130 # NVIDIA Blackwell (B200/B300, CUDA 13.0)
|
||||
docker pull vllm/vllm-openai-rocm:latest # AMD
|
||||
docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70)
|
||||
```
|
||||
TPU images are published separately by [vllm-project/tpu-inference](https://github.com/vllm-project/tpu-inference); see the Trillium / Ironwood tpu-recipes below for the pinned tag.
|
||||
|
||||
## Deployment Configurations
|
||||
|
||||
### Quick Start (Single GPU)
|
||||
```bash
|
||||
vllm serve google/gemma-4-E4B-it \
|
||||
--max-model-len <n_of_tokens> # up to 131072
|
||||
```
|
||||
|
||||
### 31B Dense on 2xA100/H100 (TP=2, BF16)
|
||||
```bash
|
||||
vllm serve google/gemma-4-31B-it \
|
||||
--tensor-parallel-size 2 \
|
||||
--max-model-len 32768 \
|
||||
--gpu-memory-utilization 0.90
|
||||
```
|
||||
|
||||
### 26B MoE on 1xA100/H100 (BF16)
|
||||
```bash
|
||||
vllm serve google/gemma-4-26B-A4B-it \
|
||||
--max-model-len 32768 \
|
||||
--gpu-memory-utilization 0.90
|
||||
```
|
||||
|
||||
### Full-Featured Server Launch
|
||||
Enables text, image, audio, thinking, and tool calling:
|
||||
```bash
|
||||
vllm serve google/gemma-4-31B-it \
|
||||
--tensor-parallel-size 2 \
|
||||
--max-model-len 16384 \
|
||||
--gpu-memory-utilization 0.90 \
|
||||
--enable-auto-tool-choice \
|
||||
--reasoning-parser gemma4 \
|
||||
--tool-call-parser gemma4 \
|
||||
--chat-template examples/tool_chat_template_gemma4.jinja \
|
||||
--limit-mm-per-prompt '{"image": 4, "audio": 1}' \
|
||||
--async-scheduling \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
### Docker (NVIDIA)
|
||||
```bash
|
||||
docker run -itd --name gemma4 \
|
||||
--ipc=host --network host --shm-size 16G --gpus all \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai:gemma4-0505-cu129 \
|
||||
--model google/gemma-4-31B-it \
|
||||
--tensor-parallel-size 2 \
|
||||
--max-model-len 32768 \
|
||||
--gpu-memory-utilization 0.90 \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
Swap `vllm/vllm-openai:gemma4-0505-cu129` for `vllm/vllm-openai:gemma4-0505-cu130` on Blackwell (B200/B300).
|
||||
|
||||
### Docker (AMD MI300X/MI325X/MI350X/MI355X)
|
||||
```bash
|
||||
docker run -itd --name gemma4-rocm \
|
||||
--ipc=host --network=host --privileged \
|
||||
--cap-add=CAP_SYS_ADMIN --device=/dev/kfd --device=/dev/dri \
|
||||
--group-add=video --cap-add=SYS_PTRACE \
|
||||
--security-opt=seccomp=unconfined --shm-size 16G \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai-rocm:latest \
|
||||
--model google/gemma-4-31B-it \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
### Docker (Intel XPU B60 / B70, TP=4, Full-Featured)
|
||||
|
||||
Validated on 4× Intel Arc Pro B60 / B70 with the official vLLM XPU image.
|
||||
|
||||
```bash
|
||||
docker run --device /dev/dri \
|
||||
-v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
|
||||
--privileged --ipc=host -p 8000:8000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai-xpu:latest \
|
||||
google/gemma-4-31B-it \
|
||||
--enforce-eager \
|
||||
--attention-backend flash_attn \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000 \
|
||||
--trust-remote-code \
|
||||
--enable-auto-tool-choice \
|
||||
--reasoning-parser gemma4 \
|
||||
--tool-call-parser gemma4 \
|
||||
--chat-template examples/tool_chat_template_gemma4.jinja \
|
||||
--limit-mm-per-prompt '{"image": 4, "audio": 1}' \
|
||||
--async-scheduling \
|
||||
--gpu-memory-utilization 0.9 \
|
||||
--no-enable-prefix-caching \
|
||||
--max-num-batched-tokens 8192 \
|
||||
--max-model-len 5120 \
|
||||
--tensor-parallel-size 4
|
||||
```
|
||||
|
||||
### Docker (Cloud TPU — Trillium / Ironwood)
|
||||
TPU uses the separate `vllm/vllm-tpu` image (no pip wheel). Pull the tag specified by the upstream [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) or [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/) recipe, then run:
|
||||
```bash
|
||||
docker run -itd --name gemma4-tpu \
|
||||
--privileged --network host --shm-size 16G \
|
||||
-v /dev/shm:/dev/shm -e HF_TOKEN=$HF_TOKEN \
|
||||
vllm/vllm-tpu:latest \
|
||||
--model google/gemma-4-31B-it \
|
||||
--tensor-parallel-size 8 \
|
||||
--max-model-len 16384 \
|
||||
--disable_chunked_mm_input \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
### Text Generation
|
||||
```python
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||||
response = client.chat.completions.create(
|
||||
model="google/gemma-4-31B-it",
|
||||
messages=[{"role": "user", "content": "Write a poem about the ocean."}],
|
||||
max_tokens=512, temperature=0.7,
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Image Understanding
|
||||
```python
|
||||
response = client.chat.completions.create(
|
||||
model="google/gemma-4-31B-it",
|
||||
messages=[{"role": "user", "content": [
|
||||
{"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg"}},
|
||||
{"type": "text", "text": "Describe this image in detail."},
|
||||
]}],
|
||||
max_tokens=1024,
|
||||
)
|
||||
```
|
||||
|
||||
### Dynamic Vision Resolution
|
||||
Supported values: 70, 140, 280 (default), 560, 1120 tokens/image.
|
||||
```bash
|
||||
vllm serve google/gemma-4-31B-it \
|
||||
--mm-processor-kwargs '{"max_soft_tokens": 560}'
|
||||
```
|
||||
|
||||
### Audio (E2B / E4B)
|
||||
Requires `uv pip install "vllm[audio]"`.
|
||||
```bash
|
||||
vllm serve google/gemma-4-E2B-it \
|
||||
--max-model-len 8192 \
|
||||
--limit-mm-per-prompt '{"image": 4, "audio": 1}'
|
||||
```
|
||||
|
||||
### Thinking Mode
|
||||
```bash
|
||||
vllm serve google/gemma-4-31B-it \
|
||||
--max-model-len 16384 \
|
||||
--enable-auto-tool-choice \
|
||||
--reasoning-parser gemma4 \
|
||||
--tool-call-parser gemma4 \
|
||||
--chat-template examples/tool_chat_template_gemma4.jinja
|
||||
```
|
||||
Enable thinking per-request via `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`, or default-on with `--default-chat-template-kwargs '{"enable_thinking": true}'`.
|
||||
|
||||
### Structured Outputs
|
||||
vLLM guided decoding constrains output to a JSON schema. Include semantic instructions in the system prompt — the model does not see schema descriptions.
|
||||
|
||||
## Configuration Tips
|
||||
|
||||
- Set `--max-model-len` to match your workload.
|
||||
- `--gpu-memory-utilization 0.90–0.95` maximizes KV cache.
|
||||
- Image-only workloads: pass `--limit-mm-per-prompt.audio 0`.
|
||||
- Text-only workloads: pass `--limit-mm-per-prompt '{"image": 0, "audio": 0}'` to skip MM profiling.
|
||||
- `--async-scheduling` improves throughput.
|
||||
- FP8 KV cache (`--kv-cache-dtype fp8`) saves ~50% KV memory.
|
||||
|
||||
## Quantized Variants
|
||||
|
||||
Two pre-quantized checkpoints are available:
|
||||
|
||||
- [`RedHatAI/gemma-4-31B-it-FP8-dynamic`](https://huggingface.co/RedHatAI/gemma-4-31B-it-FP8-dynamic) — FP8 (E4M3) linear weights with dynamic per-token activation quantization (vision tower stays BF16); runs on Hopper and Blackwell.
|
||||
- [`nvidia/gemma-4-31B-it-NVFP4`](https://huggingface.co/nvidia/gemma-4-31B-it-NVFP4) — NVFP4 (4-bit) weights; requires Blackwell (B200/B300).
|
||||
|
||||
Pick them from the **Variant** dropdown above, or pass the repo id directly to `vllm serve`.
|
||||
|
||||
## Throughput vs Latency
|
||||
|
||||
| Goal | TP | `--max-num-seqs` | Notes |
|
||||
|------|----|------------------|-------|
|
||||
| Max throughput | 1-2 | 256-512 | Best tok/s per GPU |
|
||||
| Min latency | 4-8 | 8-16 | Best TTFT/TPOT |
|
||||
| Balanced | 2 | 128 | Mixed workloads |
|
||||
|
||||
## Speculative Decoding (MTP)
|
||||
|
||||
Enable the **Spec Decoding** feature toggle (above) or add `--speculative-config` manually to use MTP drafting with the [assistant model](https://huggingface.co/google/gemma-4-31B-it-assistant). Recommended `num_speculative_tokens`: 4–8 for this model. See the [Gemma 4 usage guide](../../Google/Gemma4) for details and benchmarks.
|
||||
|
||||
> **Note:** MTP speculative decoding for Gemma 4 is only available on the vLLM nightly build — it has not yet landed in a stable release. Install via the nightly wheel (`uv pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly/cu129 …`) or use the `vllm/vllm-openai:gemma4-0505-cu129` / `vllm/vllm-openai:gemma4-0505-cu130` images above; the standard `:latest` stable tag does not include this feature.
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/google/gemma-4-31B-it)
|
||||
- [FP8 variant](https://huggingface.co/RedHatAI/gemma-4-31B-it-FP8-dynamic)
|
||||
- [NVFP4 variant](https://huggingface.co/nvidia/gemma-4-31B-it-NVFP4)
|
||||
- [Gemma docs](https://ai.google.dev/gemma/docs)
|
||||
- [vLLM Gemma 4 tool-call template](https://github.com/vllm-project/vllm/blob/main/examples/tool_chat_template_gemma4.jinja)
|
||||
- [TPU recipes: Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4)
|
||||
- [TPU recipes: Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/)
|
||||
@@ -0,0 +1,322 @@
|
||||
meta:
|
||||
title: "Gemma 4 E2B IT"
|
||||
slug: "gemma-4-e2b-it"
|
||||
provider: "Google"
|
||||
description: "Google's compact Gemma 4 multimodal model (effective 2B) with native text, image, and audio, plus thinking mode and tool-use protocol."
|
||||
date_added: 2026-04-02
|
||||
date_updated: 2026-09-12
|
||||
tasks:
|
||||
- multimodal
|
||||
- text
|
||||
performance_headline: "Compact unified multimodal model with audio, thinking, and function calling — runs on a single 24 GB+ GPU"
|
||||
related_recipes:
|
||||
- google/gemma-4-E4B-it
|
||||
- google/gemma-4-31B-it
|
||||
- google/gemma-4-26B-A4B-it
|
||||
hardware:
|
||||
h100: verified
|
||||
mi300x: verified
|
||||
mi325x: verified
|
||||
mi355x: verified
|
||||
trillium: verified
|
||||
ironwood: verified
|
||||
xeon6: verified
|
||||
|
||||
model:
|
||||
model_id: "google/gemma-4-E2B-it"
|
||||
min_vllm_version: "0.19.1"
|
||||
architecture: dense
|
||||
parameter_count: "5B"
|
||||
active_parameters: "5B"
|
||||
context_length: 131072
|
||||
base_args: []
|
||||
base_env: {}
|
||||
|
||||
dependencies:
|
||||
- note: "Audio extras — needed to serve the audio modality"
|
||||
command: 'uv pip install "vllm[audio]"'
|
||||
optional: true
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with Gemma 4 parser and chat template"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "gemma4"
|
||||
- "--chat-template"
|
||||
- "examples/tool_chat_template_gemma4.jinja"
|
||||
reasoning:
|
||||
description: "Enable structured thinking/reasoning output"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "gemma4"
|
||||
text_only:
|
||||
description: "Skip loading the vision and audio encoders for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
|
||||
args:
|
||||
- "--language-model-only"
|
||||
spec_decoding:
|
||||
description: "MTP speculative decoding with centroids masking for accelerated inference"
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"model":"google/gemma-4-E2B-it-assistant","num_speculative_tokens":2}'
|
||||
|
||||
opt_in_features:
|
||||
- text_only
|
||||
- spec_decoding
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 13
|
||||
description: "Full BF16 — single 24 GB+ NVIDIA GPU or 1x MI300X/MI325X/MI350X/MI355X or 1x Xeon6/Xeon5 NUMA node"
|
||||
supported_hardware: [h100, mi300x, mi325x, mi355x, trillium, ironwood, xeon6]
|
||||
fp8:
|
||||
model_id: "RedHatAI/gemma-4-E2B-it-FP8-dynamic"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 6
|
||||
description: "FP8 (E4M3) linear weights with dynamic per-token activation quantization (vision/audio encoders stay BF16) — Hopper or Blackwell"
|
||||
supported_hardware: [h100]
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
w4a16:
|
||||
model_id: "google/gemma-4-E2B-it-qat-w4a16-ct"
|
||||
precision: int4
|
||||
vram_minimum_gb: 8
|
||||
description: "W4A16 QAT (4-bit weights, 16-bit activations, group_size=32) via compressed-tensors — runs on any GPU"
|
||||
supported_hardware: [h100, mi300x, mi325x, mi355x]
|
||||
mobile:
|
||||
model_id: "google/gemma-4-E2B-it-qat-mobile-ct"
|
||||
precision: int2/4/8
|
||||
vram_minimum_gb: 4
|
||||
description: "Mobile QAT — mixed int2/4/8 compressed-tensors (int2 embeddings + lm_head + deep MLP, int4 attention + early MLP, int8 vision + gates). 3.6× weight compression vs BF16, up to 1.25× faster decode at low batch"
|
||||
supported_hardware: [h100, mi300x, mi325x, mi355x]
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
tp: 1
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[Gemma 4 E2B](https://huggingface.co/google/gemma-4-E2B-it) is the smallest member of Google's Gemma 4 family — an effective-2B unified multimodal model that natively processes text, images, and audio, with structured thinking/reasoning, function calling, and dynamic vision resolution. It runs comfortably on a single 24 GB+ GPU.
|
||||
|
||||
Intel Xeon 6 CPU support remains verified. No additional benchmark-specific CPU
|
||||
tuning is prescribed for general deployment.
|
||||
|
||||
### Key Features
|
||||
- **Multimodal**: Text + images + audio natively (video via custom frame-extraction pipeline).
|
||||
- **Dual Attention**: Alternating sliding-window (local) and global attention with different head dimensions.
|
||||
- **Thinking Mode**: Structured reasoning via `<|channel>thought\n...<channel|>` delimiters.
|
||||
- **Function Calling**: Custom tool-call protocol with dedicated special tokens.
|
||||
- **Dynamic Vision Resolution**: Per-request configurable vision token budget (70, 140, 280, 560, 1120 tokens).
|
||||
|
||||
TPU support is provided through [vLLM TPU](https://github.com/vllm-project/tpu-inference) with recipes for [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) and [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Hardware: Intel Xeon 6/Xeon 5 CPUs
|
||||
- vLLM >= 0.19.1
|
||||
|
||||
### pip (NVIDIA CUDA)
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --pre \
|
||||
--extra-index-url https://wheels.vllm.ai/nightly/cu129 \
|
||||
--extra-index-url https://download.pytorch.org/whl/cu129 \
|
||||
--index-strategy unsafe-best-match
|
||||
```
|
||||
|
||||
### pip (AMD ROCm: MI300X, MI325X, MI350X, MI355X)
|
||||
Requires Python 3.12, ROCm 7.2.1, glibc >= 2.35 (Ubuntu 22.04+).
|
||||
```bash
|
||||
uv venv --python 3.12
|
||||
source .venv/bin/activate
|
||||
uv pip install vllm --pre \
|
||||
--extra-index-url https://wheels.vllm.ai/rocm/nightly/rocm721 --upgrade
|
||||
```
|
||||
### pip (Intel Xeon 6 CPUs)
|
||||
For Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions.
|
||||
|
||||
|
||||
|
||||
### Docker
|
||||
```bash
|
||||
docker pull vllm/vllm-openai:gemma4-0505-cu129 # NVIDIA Hopper (H100/H200, CUDA 12.9)
|
||||
docker pull vllm/vllm-openai:gemma4-0505-cu130 # NVIDIA Blackwell (B200/B300, CUDA 13.0)
|
||||
docker pull vllm/vllm-openai-rocm:latest # AMD
|
||||
docker pull vllm/vllm-openai-cpu:latest-x86_64 # For Intel Xeon 6
|
||||
```
|
||||
|
||||
## Deployment Configurations
|
||||
|
||||
### Quick Start (Single GPU)
|
||||
```bash
|
||||
vllm serve google/gemma-4-E2B-it \
|
||||
--max-model-len 32768
|
||||
```
|
||||
|
||||
### With Audio Support
|
||||
```bash
|
||||
vllm serve google/gemma-4-E2B-it \
|
||||
--max-model-len 8192 \
|
||||
--limit-mm-per-prompt '{"image": 4, "audio": 1}'
|
||||
```
|
||||
|
||||
### Full-Featured Server Launch
|
||||
Enables text, image, audio, thinking, and tool calling:
|
||||
```bash
|
||||
vllm serve google/gemma-4-E2B-it \
|
||||
--max-model-len 16384 \
|
||||
--gpu-memory-utilization 0.90 \
|
||||
--enable-auto-tool-choice \
|
||||
--reasoning-parser gemma4 \
|
||||
--tool-call-parser gemma4 \
|
||||
--chat-template examples/tool_chat_template_gemma4.jinja \
|
||||
--limit-mm-per-prompt '{"image": 4, "audio": 1}' \
|
||||
--async-scheduling \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
### Docker (NVIDIA)
|
||||
```bash
|
||||
docker run -itd --name gemma4-e2b \
|
||||
--ipc=host --network host --shm-size 16G --gpus all \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai:gemma4-0505-cu129 \
|
||||
--model google/gemma-4-E2B-it \
|
||||
--max-model-len 32768 \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
Swap `vllm/vllm-openai:gemma4-0505-cu129` for `vllm/vllm-openai:gemma4-0505-cu130` on Blackwell (B200/B300).
|
||||
|
||||
### Docker (AMD MI300X/MI325X/MI350X/MI355X)
|
||||
```bash
|
||||
docker run -itd --name gemma4-rocm \
|
||||
--ipc=host --network=host --privileged \
|
||||
--cap-add=CAP_SYS_ADMIN --device=/dev/kfd --device=/dev/dri \
|
||||
--group-add=video --cap-add=SYS_PTRACE \
|
||||
--security-opt=seccomp=unconfined --shm-size 16G \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai-rocm:latest \
|
||||
--model google/gemma-4-E2B-it \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
### Docker (Cloud TPU — Trillium / Ironwood)
|
||||
TPU uses the separate `vllm/vllm-tpu` image (no pip wheel). Pull the tag specified by the upstream [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) or [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/) recipe, then run:
|
||||
```bash
|
||||
docker run -itd --name gemma4-tpu \
|
||||
--privileged --network host --shm-size 16G \
|
||||
-v /dev/shm:/dev/shm -e HF_TOKEN=$HF_TOKEN \
|
||||
vllm/vllm-tpu:latest \
|
||||
--model google/gemma-4-E2B-it \
|
||||
--max-model-len 16384 \
|
||||
--disable_chunked_mm_input \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
### Intel Xeon 6 Deployment via Docker
|
||||
|
||||
Launch the x86 CPU vLLM Docker container for `google/gemma-4-E2B-it`:
|
||||
|
||||
```bash
|
||||
docker run \
|
||||
--privileged --ipc=host -p 8000:8000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai-cpu:latest-x86_64 google/gemma-4-E2B-it \
|
||||
--tensor-parallel-size 1 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser gemma4 \
|
||||
--chat-template examples/tool_chat_template_gemma4.jinja \
|
||||
--reasoning-parser gemma4
|
||||
```
|
||||
|
||||
|
||||
The following settings are intentionally not prescribed as portable CPU model
|
||||
defaults because they depend on the workload, hardware, or runtime environment:
|
||||
|
||||
- `--max-num-batched-tokens`: scheduler/throughput tuning.
|
||||
- `--max-num-seqs`: concurrency and scheduler-capacity tuning.
|
||||
- `--gpu-memory-utilization`: platform memory-budget tuning.
|
||||
- `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning.
|
||||
- `VLLM_ENGINE_ITERATION_TIMEOUT_S`: operational runtime timeout.
|
||||
|
||||
These settings may still appear in validated hardware-specific overrides. Tune
|
||||
them at deployment time based on platform resources, workload shape, and
|
||||
latency/throughput goals.
|
||||
|
||||
## Client Usage
|
||||
|
||||
### Audio Transcription
|
||||
```python
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||||
response = client.chat.completions.create(
|
||||
model="google/gemma-4-E2B-it",
|
||||
messages=[{"role": "user", "content": [
|
||||
{"type": "audio_url", "audio_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/2/22/Beatbox_by_Wikipedia_user_Wikipedia_Brown.ogg"}},
|
||||
{"type": "text", "text": "Provide a verbatim, word-for-word transcription of the audio."},
|
||||
]}],
|
||||
max_tokens=512,
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Image Understanding
|
||||
```python
|
||||
response = client.chat.completions.create(
|
||||
model="google/gemma-4-E2B-it",
|
||||
messages=[{"role": "user", "content": [
|
||||
{"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg"}},
|
||||
{"type": "text", "text": "Describe this image in detail."},
|
||||
]}],
|
||||
max_tokens=1024,
|
||||
)
|
||||
```
|
||||
|
||||
### Thinking Mode
|
||||
Launch with reasoning parser, then enable per-request:
|
||||
```bash
|
||||
vllm serve google/gemma-4-E2B-it \
|
||||
--max-model-len 16384 \
|
||||
--reasoning-parser gemma4 \
|
||||
--tool-call-parser gemma4 \
|
||||
--enable-auto-tool-choice \
|
||||
--chat-template examples/tool_chat_template_gemma4.jinja
|
||||
```
|
||||
Enable per-request via `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`.
|
||||
|
||||
## Configuration Tips
|
||||
|
||||
- Set `--max-model-len` to match your workload (max 131072).
|
||||
- Image-only workloads: `--limit-mm-per-prompt.audio 0`.
|
||||
- Text-only workloads: `--limit-mm-per-prompt '{"image": 0, "audio": 0}'` to skip MM profiling.
|
||||
- `--async-scheduling` improves throughput.
|
||||
- FP8 KV cache (`--kv-cache-dtype fp8`) saves ~50% KV memory.
|
||||
|
||||
## Quantized Variant
|
||||
|
||||
[`RedHatAI/gemma-4-E2B-it-FP8-dynamic`](https://huggingface.co/RedHatAI/gemma-4-E2B-it-FP8-dynamic) is a pre-quantized FP8 (E4M3) checkpoint — linear weights with dynamic per-token activation quantization, vision/audio encoders kept in BF16. Runs on Hopper and Blackwell. Pick the **fp8** variant above, or pass the repo id directly to `vllm serve`.
|
||||
|
||||
## Speculative Decoding (MTP)
|
||||
|
||||
Enable the **Spec Decoding** feature toggle (above) or add `--speculative-config` manually to use MTP drafting with the [assistant model](https://huggingface.co/google/gemma-4-E2B-it-assistant). Recommended `num_speculative_tokens`: 2 for this model. The E2B assistant uses centroids masking for efficient sparse logit computation. See the [Gemma 4 usage guide](../../Google/Gemma4) for details and benchmarks.
|
||||
|
||||
> **Note:** MTP speculative decoding for Gemma 4 is only available on the vLLM nightly build — it has not yet landed in a stable release. Install via the nightly wheel (`uv pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly/cu129 …`) or use the `vllm/vllm-openai:gemma4-0505-cu129` / `vllm/vllm-openai:gemma4-0505-cu130` images above; the standard `:latest` stable tag does not include this feature.
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/google/gemma-4-E2B-it)
|
||||
- [FP8 variant](https://huggingface.co/RedHatAI/gemma-4-E2B-it-FP8-dynamic)
|
||||
- [Gemma docs](https://ai.google.dev/gemma/docs)
|
||||
- [vLLM Gemma 4 tool-call template](https://github.com/vllm-project/vllm/blob/main/examples/tool_chat_template_gemma4.jinja)
|
||||
- [TPU recipes: Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4)
|
||||
- [TPU recipes: Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/)
|
||||
@@ -0,0 +1,322 @@
|
||||
meta:
|
||||
title: "Gemma 4 E4B IT"
|
||||
slug: "gemma-4-e4b-it"
|
||||
provider: "Google"
|
||||
description: "Google's compact Gemma 4 multimodal model (effective 4B) with native text, image, and audio, plus thinking mode and tool-use protocol."
|
||||
date_added: 2026-04-02
|
||||
date_updated: 2026-09-12
|
||||
tasks:
|
||||
- multimodal
|
||||
- text
|
||||
performance_headline: "Effective-4B unified multimodal model with audio, thinking, and function calling — runs on a single 24 GB+ GPU"
|
||||
related_recipes:
|
||||
- google/gemma-4-E2B-it
|
||||
- google/gemma-4-31B-it
|
||||
- google/gemma-4-26B-A4B-it
|
||||
hardware:
|
||||
h100: verified
|
||||
mi300x: verified
|
||||
mi325x: verified
|
||||
mi355x: verified
|
||||
trillium: verified
|
||||
ironwood: verified
|
||||
xeon6: verified
|
||||
|
||||
model:
|
||||
model_id: "google/gemma-4-E4B-it"
|
||||
min_vllm_version: "0.19.1"
|
||||
architecture: dense
|
||||
parameter_count: "8B"
|
||||
active_parameters: "8B"
|
||||
context_length: 131072
|
||||
base_args: []
|
||||
base_env: {}
|
||||
|
||||
dependencies:
|
||||
- note: "Audio extras — needed to serve the audio modality"
|
||||
command: 'uv pip install "vllm[audio]"'
|
||||
optional: true
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with Gemma 4 parser and chat template"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "gemma4"
|
||||
- "--chat-template"
|
||||
- "examples/tool_chat_template_gemma4.jinja"
|
||||
reasoning:
|
||||
description: "Enable structured thinking/reasoning output"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "gemma4"
|
||||
text_only:
|
||||
description: "Skip loading the vision and audio encoders for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
|
||||
args:
|
||||
- "--language-model-only"
|
||||
spec_decoding:
|
||||
description: "MTP speculative decoding with centroids masking for accelerated inference"
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"model":"google/gemma-4-E4B-it-assistant","num_speculative_tokens":4}'
|
||||
|
||||
opt_in_features:
|
||||
- text_only
|
||||
- spec_decoding
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 20
|
||||
description: "Full BF16 — single 24 GB+ NVIDIA GPU or 1x MI300X/MI325X/MI350X/MI355Xi or 2x Xeon6/Xeon5 NUMA nodes"
|
||||
supported_hardware: [h100, mi300x, mi325x, mi355x, trillium, ironwood, xeon6]
|
||||
fp8:
|
||||
model_id: "RedHatAI/gemma-4-E4B-it-FP8-dynamic"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 10
|
||||
description: "FP8 (E4M3) linear weights with dynamic per-token activation quantization (vision/audio encoders stay BF16) — Hopper or Blackwell"
|
||||
supported_hardware: [h100]
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
w4a16:
|
||||
model_id: "google/gemma-4-E4B-it-qat-w4a16-ct"
|
||||
precision: int4
|
||||
vram_minimum_gb: 10
|
||||
description: "W4A16 QAT (4-bit weights, 16-bit activations, group_size=32) via compressed-tensors — runs on any GPU"
|
||||
supported_hardware: [h100, mi300x, mi325x, mi355x]
|
||||
mobile:
|
||||
model_id: "google/gemma-4-E4B-it-qat-mobile-ct"
|
||||
precision: int2/4/8
|
||||
vram_minimum_gb: 5
|
||||
description: "Mobile QAT — mixed int2/4/8 compressed-tensors (int2 embeddings + lm_head + audio, int4 LLM attention + MLP, int8 vision + gates). 4.1× weight compression vs BF16, up to 1.5× faster decode at low batch"
|
||||
supported_hardware: [h100, mi300x, mi325x, mi355x]
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
tp: 1
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[Gemma 4 E4B](https://huggingface.co/google/gemma-4-E4B-it) is Google's effective-4B unified multimodal model — text + images + audio in a single model, with structured thinking/reasoning, function calling, and dynamic vision resolution. It fits on a single 24 GB+ GPU.
|
||||
|
||||
Intel Xeon 6 CPU support remains verified. No additional benchmark-specific CPU
|
||||
tuning is prescribed for general deployment.
|
||||
|
||||
### Key Features
|
||||
- **Multimodal**: Text + images + audio natively (video via custom frame-extraction pipeline).
|
||||
- **Dual Attention**: Alternating sliding-window (local) and global attention with different head dimensions.
|
||||
- **Thinking Mode**: Structured reasoning via `<|channel>thought\n...<channel|>` delimiters.
|
||||
- **Function Calling**: Custom tool-call protocol with dedicated special tokens.
|
||||
- **Dynamic Vision Resolution**: Per-request configurable vision token budget (70, 140, 280, 560, 1120 tokens).
|
||||
|
||||
TPU support is provided through [vLLM TPU](https://github.com/vllm-project/tpu-inference) with recipes for [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) and [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Hardware: Intel Xeon 6/Xeon 5 CPUs
|
||||
- vLLM >= 0.19.1
|
||||
|
||||
### pip (NVIDIA CUDA)
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --pre \
|
||||
--extra-index-url https://wheels.vllm.ai/nightly/cu129 \
|
||||
--extra-index-url https://download.pytorch.org/whl/cu129 \
|
||||
--index-strategy unsafe-best-match
|
||||
```
|
||||
|
||||
### pip (AMD ROCm: MI300X, MI325X, MI350X, MI355X)
|
||||
Requires Python 3.12, ROCm 7.2.1, glibc >= 2.35 (Ubuntu 22.04+).
|
||||
```bash
|
||||
uv venv --python 3.12
|
||||
source .venv/bin/activate
|
||||
uv pip install vllm --pre \
|
||||
--extra-index-url https://wheels.vllm.ai/rocm/nightly/rocm721 --upgrade
|
||||
```
|
||||
|
||||
### pip (Intel Xeon 6 CPUs)
|
||||
For Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions.
|
||||
|
||||
### Docker
|
||||
```bash
|
||||
docker pull vllm/vllm-openai:gemma4-0505-cu129 # NVIDIA Hopper (H100/H200, CUDA 12.9)
|
||||
docker pull vllm/vllm-openai:gemma4-0505-cu130 # NVIDIA Blackwell (B200/B300, CUDA 13.0)
|
||||
docker pull vllm/vllm-openai-rocm:latest # AMD
|
||||
docker pull vllm/vllm-openai-cpu:latest-x86_64 # For Intel Xeon 6
|
||||
```
|
||||
|
||||
## Deployment Configurations
|
||||
|
||||
### Quick Start (Single GPU)
|
||||
```bash
|
||||
vllm serve google/gemma-4-E4B-it \
|
||||
--max-model-len 32768
|
||||
```
|
||||
|
||||
### With Audio Support
|
||||
```bash
|
||||
vllm serve google/gemma-4-E4B-it \
|
||||
--max-model-len 8192 \
|
||||
--limit-mm-per-prompt '{"image": 4, "audio": 1}'
|
||||
```
|
||||
|
||||
### Full-Featured Server Launch
|
||||
Enables text, image, audio, thinking, and tool calling:
|
||||
```bash
|
||||
vllm serve google/gemma-4-E4B-it \
|
||||
--max-model-len 16384 \
|
||||
--gpu-memory-utilization 0.90 \
|
||||
--enable-auto-tool-choice \
|
||||
--reasoning-parser gemma4 \
|
||||
--tool-call-parser gemma4 \
|
||||
--chat-template examples/tool_chat_template_gemma4.jinja \
|
||||
--limit-mm-per-prompt '{"image": 4, "audio": 1}' \
|
||||
--async-scheduling \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
### Docker (NVIDIA)
|
||||
```bash
|
||||
docker run -itd --name gemma4-e4b \
|
||||
--ipc=host --network host --shm-size 16G --gpus all \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai:gemma4-0505-cu129 \
|
||||
--model google/gemma-4-E4B-it \
|
||||
--max-model-len 32768 \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
Swap `vllm/vllm-openai:gemma4-0505-cu129` for `vllm/vllm-openai:gemma4-0505-cu130` on Blackwell (B200/B300).
|
||||
|
||||
### Docker (AMD MI300X/MI325X/MI350X/MI355X)
|
||||
```bash
|
||||
docker run -itd --name gemma4-rocm \
|
||||
--ipc=host --network=host --privileged \
|
||||
--cap-add=CAP_SYS_ADMIN --device=/dev/kfd --device=/dev/dri \
|
||||
--group-add=video --cap-add=SYS_PTRACE \
|
||||
--security-opt=seccomp=unconfined --shm-size 16G \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai-rocm:latest \
|
||||
--model google/gemma-4-E4B-it \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
### Docker (Cloud TPU — Trillium / Ironwood)
|
||||
TPU uses the separate `vllm/vllm-tpu` image (no pip wheel). Pull the tag specified by the upstream [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) or [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/) recipe, then run:
|
||||
```bash
|
||||
docker run -itd --name gemma4-tpu \
|
||||
--privileged --network host --shm-size 16G \
|
||||
-v /dev/shm:/dev/shm -e HF_TOKEN=$HF_TOKEN \
|
||||
vllm/vllm-tpu:latest \
|
||||
--model google/gemma-4-E4B-it \
|
||||
--max-model-len 16384 \
|
||||
--disable_chunked_mm_input \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
### Intel Xeon 6 Deployment via Docker
|
||||
|
||||
Launch the x86 CPU vLLM Docker container for `google/gemma-4-E4B-it`:
|
||||
|
||||
```bash
|
||||
docker run \
|
||||
--privileged --ipc=host -p 8000:8000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai-cpu:latest-x86_64 google/gemma-4-E4B-it \
|
||||
--tensor-parallel-size 1 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser gemma4 \
|
||||
--chat-template examples/tool_chat_template_gemma4.jinja \
|
||||
--reasoning-parser gemma4
|
||||
```
|
||||
|
||||
For additional Intel Xeon 6 deployment details, see the Intel Software Catalog entries for [Gemma 4 E4B IT](https://aiswcatalog.intel.com/models/google-gemma-4-e4b-it).
|
||||
|
||||
The following settings are intentionally not prescribed as portable CPU model
|
||||
defaults because they depend on the workload, hardware, or runtime environment:
|
||||
|
||||
- `--max-num-batched-tokens`: scheduler/throughput tuning.
|
||||
- `--max-num-seqs`: concurrency and scheduler-capacity tuning.
|
||||
- `--gpu-memory-utilization`: platform memory-budget tuning.
|
||||
- `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning.
|
||||
- `VLLM_ENGINE_ITERATION_TIMEOUT_S`: operational runtime timeout.
|
||||
|
||||
These settings may still appear in validated hardware-specific overrides. Tune
|
||||
them at deployment time based on platform resources, workload shape, and
|
||||
latency/throughput goals.
|
||||
|
||||
## Client Usage
|
||||
|
||||
### Audio Transcription
|
||||
```python
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||||
response = client.chat.completions.create(
|
||||
model="google/gemma-4-E4B-it",
|
||||
messages=[{"role": "user", "content": [
|
||||
{"type": "audio_url", "audio_url": {"url": "https://example.com/audio.wav"}},
|
||||
{"type": "text", "text": "Transcribe this audio."},
|
||||
]}],
|
||||
max_tokens=512,
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Image Understanding
|
||||
```python
|
||||
response = client.chat.completions.create(
|
||||
model="google/gemma-4-E4B-it",
|
||||
messages=[{"role": "user", "content": [
|
||||
{"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg"}},
|
||||
{"type": "text", "text": "Describe this image in detail."},
|
||||
]}],
|
||||
max_tokens=1024,
|
||||
)
|
||||
```
|
||||
|
||||
### Thinking Mode
|
||||
```bash
|
||||
vllm serve google/gemma-4-E4B-it \
|
||||
--max-model-len 16384 \
|
||||
--reasoning-parser gemma4 \
|
||||
--tool-call-parser gemma4 \
|
||||
--enable-auto-tool-choice \
|
||||
--chat-template examples/tool_chat_template_gemma4.jinja
|
||||
```
|
||||
Enable per-request via `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`.
|
||||
|
||||
## Configuration Tips
|
||||
|
||||
- Set `--max-model-len` to match your workload (max 131072).
|
||||
- Image-only workloads: `--limit-mm-per-prompt.audio 0`.
|
||||
- Text-only workloads: `--limit-mm-per-prompt '{"image": 0, "audio": 0}'` to skip MM profiling.
|
||||
- `--async-scheduling` improves throughput.
|
||||
- FP8 KV cache (`--kv-cache-dtype fp8`) saves ~50% KV memory.
|
||||
|
||||
## Quantized Variant
|
||||
|
||||
[`RedHatAI/gemma-4-E4B-it-FP8-dynamic`](https://huggingface.co/RedHatAI/gemma-4-E4B-it-FP8-dynamic) is a pre-quantized FP8 (E4M3) checkpoint — linear weights with dynamic per-token activation quantization, vision/audio encoders kept in BF16. Runs on Hopper and Blackwell. Pick the **fp8** variant above, or pass the repo id directly to `vllm serve`.
|
||||
|
||||
## Speculative Decoding (MTP)
|
||||
|
||||
Enable the **Spec Decoding** feature toggle (above) or add `--speculative-config` manually to use MTP drafting with the [assistant model](https://huggingface.co/google/gemma-4-E4B-it-assistant). Recommended `num_speculative_tokens`: 4 for this model. The E4B assistant uses centroids masking for efficient sparse logit computation. See the [Gemma 4 usage guide](../../Google/Gemma4) for details and benchmarks.
|
||||
|
||||
> **Note:** MTP speculative decoding for Gemma 4 is only available on the vLLM nightly build — it has not yet landed in a stable release. Install via the nightly wheel (`uv pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly/cu129 …`) or use the `vllm/vllm-openai:gemma4-0505-cu129` / `vllm/vllm-openai:gemma4-0505-cu130` images above; the standard `:latest` stable tag does not include this feature.
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/google/gemma-4-E4B-it)
|
||||
- [FP8 variant](https://huggingface.co/RedHatAI/gemma-4-E4B-it-FP8-dynamic)
|
||||
- [Gemma docs](https://ai.google.dev/gemma/docs)
|
||||
- [vLLM Gemma 4 tool-call template](https://github.com/vllm-project/vllm/blob/main/examples/tool_chat_template_gemma4.jinja)
|
||||
- [TPU recipes: Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4)
|
||||
- [TPU recipes: Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/)
|
||||
- [Xeon 6 recipe: Gemma 4 E4B IT](https://aiswcatalog.intel.com/models/google-gemma-4-e4b-it)
|
||||
@@ -0,0 +1,122 @@
|
||||
meta:
|
||||
title: "TranslateGemma 27B IT"
|
||||
slug: "translategemma-27b-it"
|
||||
provider: "Google"
|
||||
description: "Lightweight open translation model from Google (based on Gemma 3) supporting 55 languages. Served via the vLLM-optimized Infomaniak-AI checkpoint."
|
||||
date_added: 2026-03-06
|
||||
date_updated: 2026-04-17
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "Deployable on laptops/desktops and cloud GPUs; vLLM-optimized checkpoint removes custom JSON inputs"
|
||||
related_recipes: []
|
||||
|
||||
model:
|
||||
model_id: "google/translategemma-27b-it"
|
||||
min_vllm_version: "0.14.1"
|
||||
architecture: dense
|
||||
parameter_count: "27B"
|
||||
active_parameters: "27B"
|
||||
context_length: 131072
|
||||
base_args: []
|
||||
base_env: {}
|
||||
|
||||
features: {}
|
||||
|
||||
opt_in_features: []
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 65
|
||||
description: "Original Google checkpoint in BF16 (has vLLM compatibility issues — prefer vllm-optimized variant)"
|
||||
vllm_optimized:
|
||||
model_id: "Infomaniak-AI/vllm-translategemma-27b-it"
|
||||
precision: bf16
|
||||
vram_minimum_gb: 65
|
||||
description: "Infomaniak-AI vLLM-optimized checkpoint — recommended. Fixes RoPE config, EOS token, and replaces custom JSON inputs with string delimiters."
|
||||
small_4b:
|
||||
model_id: "google/translategemma-4b-it"
|
||||
precision: bf16
|
||||
vram_minimum_gb: 10
|
||||
description: "4B variant for lower-resource deployments (prefer Infomaniak-AI/vllm-translategemma-4b-it)"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
|
||||
hardware_overrides: {}
|
||||
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[TranslateGemma](https://huggingface.co/collections/google/translategemma) is a family of lightweight, state-of-the-art open translation models from Google, based on the Gemma 3 family. TranslateGemma models handle translation across 55 languages and are small enough to deploy on laptops, desktops, and modest cloud GPU environments.
|
||||
|
||||
### Original Models
|
||||
- [google/translategemma-27b-it](https://huggingface.co/google/translategemma-27b-it)
|
||||
- [google/translategemma-4b-it](https://huggingface.co/google/translategemma-4b-it)
|
||||
|
||||
### Optimized vLLM Models
|
||||
- [Infomaniak-AI/vllm-translategemma-27b-it](https://huggingface.co/Infomaniak-AI/vllm-translategemma-27b-it)
|
||||
- [Infomaniak-AI/vllm-translategemma-4b-it](https://huggingface.co/Infomaniak-AI/vllm-translategemma-4b-it)
|
||||
|
||||
### Why use the vLLM-optimized models?
|
||||
|
||||
The original Google models have compatibility issues with standard inference engines like vLLM. The optimized versions from Infomaniak-AI ([detailed changes](https://huggingface.co/Infomaniak-AI/vllm-translategemma-27b-it#changes-from-original-model)):
|
||||
|
||||
- **vLLM Compatibility**: Originals require custom JSON parameters (`source_lang_code`/`target_lang_code`). The optimized version uses string delimiters.
|
||||
- **RoPE Simplification**: Originals use a complex RoPE configuration for sliding attention. Optimized uses a standard linear RoPE format (`factor: 8.0`).
|
||||
- **EOS Token Fix**: Corrects the EOS token from `<end_of_turn>` to `<eos>`.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### Docker
|
||||
```bash
|
||||
docker pull vllm/vllm-openai:latest
|
||||
```
|
||||
|
||||
## Deployment Configurations
|
||||
|
||||
Verified for both 4B and 27B:
|
||||
```bash
|
||||
docker run -itd --name google-translategemma-27b-it \
|
||||
--ipc=host \
|
||||
--network host \
|
||||
--shm-size 16G \
|
||||
--gpus all \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai:latest \
|
||||
Infomaniak-AI/vllm-translategemma-27b-it \
|
||||
--served-model-name translategemma-27b-it \
|
||||
--gpu-memory-utilization 0.8 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
Tips:
|
||||
- **Prompt Delimiters**: Encode language metadata directly in the content string:
|
||||
`<<<source>>>{src_lang}<<<target>>>{tgt_lang}<<<text>>>{text}`
|
||||
- **Language Codes**: ISO 639-1 Alpha-2 (e.g. `en`, `zh`) and regional variants (e.g. `en_US`, `zh_CN`).
|
||||
- **Context Limit**: ~2K tokens.
|
||||
|
||||
### cURL
|
||||
```bash
|
||||
curl -X POST http://localhost:8000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "translategemma-27b-it",
|
||||
"messages": [{
|
||||
"role": "user",
|
||||
"content": "<<<source>>>en<<<target>>>zh<<<text>>>We distribute two models for language identification, which can recognize 176 languages."
|
||||
}]
|
||||
}'
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [TranslateGemma collection](https://huggingface.co/collections/google/translategemma)
|
||||
- [Infomaniak-AI optimized 27B](https://huggingface.co/Infomaniak-AI/vllm-translategemma-27b-it)
|
||||
- [Infomaniak-AI optimized 4B](https://huggingface.co/Infomaniak-AI/vllm-translategemma-4b-it)
|
||||
@@ -0,0 +1,109 @@
|
||||
meta:
|
||||
title: "Jais-3-0.9B"
|
||||
slug: "jais-3-0.9b"
|
||||
provider: "IFM"
|
||||
description: "Compact dense model for mathematics, code, instruction following, and STEM tasks"
|
||||
date_added: 2026-10-05
|
||||
date_updated: 2026-10-05
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "1.08B stored parameters including embeddings, with 131K context and native K2-Horizon parsing"
|
||||
related_recipes:
|
||||
- "IFM/Jais-3-3.7B"
|
||||
hardware:
|
||||
h200: verified
|
||||
model:
|
||||
model_id: "IFM/Jais-3-0.9B"
|
||||
architecture: dense
|
||||
parameter_count: "1.08B"
|
||||
active_parameters: "1.08B"
|
||||
context_length: 131072
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
- "--chat-template-content-format"
|
||||
- "string"
|
||||
- "--dtype"
|
||||
- "bfloat16"
|
||||
- "--max-model-len"
|
||||
- "131072"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.85"
|
||||
base_env: {}
|
||||
features:
|
||||
reasoning:
|
||||
description: "Extract Jais-3 reasoning into message.reasoning"
|
||||
args: ["--reasoning-parser", "k2_horizon"]
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with the K2-Horizon parser"
|
||||
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 3
|
||||
description: "Released BF16 checkpoint; the command starts at an 8K runtime context"
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
- multi_node_tp_pp
|
||||
hardware_overrides: {}
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
Jais-3-0.9B is a compact dense model in the IFM Jais-3
|
||||
family, with 1.08B stored parameters including embeddings and a 131,072-token
|
||||
context window.
|
||||
|
||||
Jais-3 is an open-weight IFM model family built for transparent
|
||||
foundation-model research, staged checkpoint analysis, and practical
|
||||
deployment. The series spans compact dense models such as Jais-3-0.9B
|
||||
for local experimentation, dense mid-size and large models for high-quality
|
||||
research workloads, and mixture-of-expert (MoE) models for higher-capacity
|
||||
serving.
|
||||
|
||||
## Launch Commands
|
||||
|
||||
Jais-3-0.9B's `config.json` already enables YaRN, so the full 131,072-token context needs no RoPE override.
|
||||
|
||||
```bash
|
||||
vllm serve IFM/Jais-3-0.9B \
|
||||
--trust-remote-code \
|
||||
--chat-template-content-format string \
|
||||
--dtype bfloat16 \
|
||||
--max-model-len 131072 \
|
||||
--gpu-memory-utilization 0.85 \
|
||||
--reasoning-parser k2_horizon \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser k2_horizon
|
||||
```
|
||||
|
||||
Lower `--max-model-len` (e.g. 8192) to reduce KV-cache memory.
|
||||
|
||||
## Client usage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
|
||||
)
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="IFM/Jais-3-0.9B",
|
||||
messages=[{"role": "user", "content": "لدينا عددان مجموعهما 50 وحاصل ضربهما 600. ما هما العددان؟ اشرح خطوات الحل."}],
|
||||
temperature=1.0, top_p=0.95, max_tokens=2048,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Thinking modes
|
||||
|
||||
The model supports selectable thinking through `chat_template_kwargs`, per
|
||||
request or server-wide via `--default-chat-template-kwargs`:
|
||||
|
||||
- `{"reasoning_effort": "high"}` — full thinking (default).
|
||||
- `{"reasoning_effort": "low"}` — no thinking.
|
||||
|
||||
## References
|
||||
|
||||
- [IFM/Jais-3-0.9B on Hugging Face](https://huggingface.co/IFM/Jais-3-0.9B)
|
||||
- [Jais 3 Collection](https://huggingface.co/collections/IFM/jais-3)
|
||||
@@ -0,0 +1,88 @@
|
||||
meta:
|
||||
title: "Jais-3-3.7B"
|
||||
slug: "jais-3-3.7b"
|
||||
provider: "IFM"
|
||||
description: "Dense model for long-context research, serving, and downstream adaptation"
|
||||
date_added: 2026-10-05
|
||||
date_updated: 2026-10-05
|
||||
tasks: [text]
|
||||
performance_headline: "5.06B stored parameters including embeddings, with up to 512K context"
|
||||
related_recipes: ["IFM/Jais-3-0.9B", "IFM/Jais-3-7B"]
|
||||
hardware:
|
||||
h200: verified
|
||||
model:
|
||||
model_id: "IFM/Jais-3-3.7B"
|
||||
architecture: dense
|
||||
parameter_count: "5.06B"
|
||||
active_parameters: "5.06B"
|
||||
context_length: 524288
|
||||
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16"]
|
||||
base_env: {}
|
||||
features:
|
||||
reasoning:
|
||||
description: "Extract Jais-3 reasoning into message.reasoning"
|
||||
args: ["--reasoning-parser", "k2_horizon"]
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with the K2-Horizon parser"
|
||||
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 13
|
||||
description: "BF16 serving runtime with a 131K context baseline"
|
||||
compatible_strategies: [single_node_tp, multi_node_tp, multi_node_tp_pp]
|
||||
hardware_overrides: {}
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
Jais-3-3.7B is a small dense model in the IFM Jais-3 family,
|
||||
with 5.06B stored parameters including embeddings and support for up to
|
||||
524,288 tokens of context.
|
||||
|
||||
Jais-3 is an open-weight IFM model family built for transparent
|
||||
foundation-model research, staged checkpoint analysis, and practical
|
||||
deployment. The series spans compact dense models for local experimentation,
|
||||
dense mid-size and large models for high-quality research workloads, and
|
||||
mixture-of-expert (MoE) models for higher-capacity serving.
|
||||
|
||||
## Launch Commands
|
||||
|
||||
```bash
|
||||
vllm serve IFM/Jais-3-3.7B \
|
||||
--trust-remote-code \
|
||||
--chat-template-content-format string \
|
||||
--dtype bfloat16 \
|
||||
--reasoning-parser k2_horizon \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser k2_horizon
|
||||
```
|
||||
|
||||
## Client usage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
|
||||
)
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="IFM/Jais-3-3.7B",
|
||||
messages=[{"role": "user", "content": "لدينا عددان مجموعهما 50 وحاصل ضربهما 600. ما هما العددان؟ اشرح خطوات الحل."}],
|
||||
temperature=1.0, top_p=0.95, max_tokens=2048,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Thinking modes
|
||||
|
||||
The model supports selectable thinking through `chat_template_kwargs`, per
|
||||
request or server-wide via `--default-chat-template-kwargs`:
|
||||
|
||||
- `{"reasoning_effort": "high"}` — full thinking (default).
|
||||
- `{"reasoning_effort": "low"}` — no thinking.
|
||||
|
||||
## References
|
||||
|
||||
- [IFM/Jais-3-3.7B on Hugging Face](https://huggingface.co/IFM/Jais-3-3.7B)
|
||||
- [Jais 3 Collection](https://huggingface.co/collections/IFM/jais-3)
|
||||
@@ -0,0 +1,93 @@
|
||||
meta:
|
||||
title: "Jais-3-32B"
|
||||
slug: "jais-3-32b"
|
||||
provider: "IFM"
|
||||
description: "Dense long-context model for reasoning research and high-capacity serving"
|
||||
date_added: 2026-10-05
|
||||
date_updated: 2026-10-05
|
||||
tasks: [text]
|
||||
performance_headline: "34.78B stored parameters including embeddings, with native vLLM execution and up to 512K context"
|
||||
related_recipes: ["IFM/Jais-3-7B", "IFM/Jais-3-MoVA-36B-A4B"]
|
||||
hardware:
|
||||
h200: verified
|
||||
model:
|
||||
model_id: "IFM/Jais-3-32B"
|
||||
architecture: dense
|
||||
parameter_count: "34.78B"
|
||||
active_parameters: "34.78B"
|
||||
context_length: 524288
|
||||
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16"]
|
||||
base_env: {}
|
||||
features:
|
||||
reasoning:
|
||||
description: "Extract Jais-3 reasoning into message.reasoning"
|
||||
args: ["--reasoning-parser", "k2_horizon"]
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with the K2-Horizon parser"
|
||||
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 84
|
||||
description: "BF16 serving runtime with native vLLM execution"
|
||||
compatible_strategies: [single_node_tp, multi_node_tp, multi_node_tp_pp]
|
||||
hardware_overrides: {}
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
tp: 2
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
Jais-3-32B is a large dense model in the IFM Jais-3 family,
|
||||
with 34.78B stored parameters including embeddings and support for up to
|
||||
524,288 tokens of context.
|
||||
|
||||
Jais-3 is an open-weight IFM model family built for transparent
|
||||
foundation-model research, staged checkpoint analysis, and practical
|
||||
deployment. The series spans compact dense models for local experimentation,
|
||||
dense mid-size and large models such as Jais-3-32B for high-quality
|
||||
research workloads, and mixture-of-expert (MoE) models for higher-capacity
|
||||
serving.
|
||||
|
||||
## Launch Commands
|
||||
|
||||
```bash
|
||||
vllm serve IFM/Jais-3-32B \
|
||||
--tensor-parallel-size 2 \
|
||||
--trust-remote-code \
|
||||
--chat-template-content-format string \
|
||||
--dtype bfloat16 \
|
||||
--reasoning-parser k2_horizon \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser k2_horizon
|
||||
```
|
||||
|
||||
## Client usage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
|
||||
)
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="IFM/Jais-3-32B",
|
||||
messages=[{"role": "user", "content": "لدينا عددان مجموعهما 50 وحاصل ضربهما 600. ما هما العددان؟ اشرح خطوات الحل."}],
|
||||
temperature=1.0, top_p=0.95, max_tokens=2048,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Thinking modes
|
||||
|
||||
The model supports selectable thinking through `chat_template_kwargs`, per
|
||||
request or server-wide via `--default-chat-template-kwargs`:
|
||||
|
||||
- `{"reasoning_effort": "high"}` — full thinking (default).
|
||||
- `{"reasoning_effort": "low"}` — no thinking.
|
||||
|
||||
## References
|
||||
|
||||
- [IFM/Jais-3-32B on Hugging Face](https://huggingface.co/IFM/Jais-3-32B)
|
||||
- [Jais 3 Collection](https://huggingface.co/collections/IFM/jais-3)
|
||||
@@ -0,0 +1,148 @@
|
||||
meta:
|
||||
title: "Jais-3-375B-A23B"
|
||||
slug: "jais-3-375b-a23b"
|
||||
provider: "IFM"
|
||||
description: "Frontier-scale sparse MoE model for long-context research and high-capacity serving"
|
||||
date_added: 2026-10-05
|
||||
date_updated: 2026-10-05
|
||||
tasks: [text]
|
||||
performance_headline: "379.17B stored parameters including embeddings / 26.67B active parameters per token"
|
||||
related_recipes: ["IFM/Jais-3-MoVA-36B-A4B"]
|
||||
hardware:
|
||||
h200: verified
|
||||
mi355x: verified
|
||||
model:
|
||||
model_id: "IFM/Jais-3-375B-A23B"
|
||||
architecture: moe
|
||||
parameter_count: "379.17B"
|
||||
active_parameters: "26.67B"
|
||||
context_length: 524288
|
||||
# --enable-expert-parallel is not here: in base_args it would reach the AMD
|
||||
# path too, which deliberately runs pure TP (see strategy_hardware). The TEP
|
||||
# strategies emit it themselves; multi_node_tp_pp does not, so it carries its
|
||||
# own copy in strategy_overrides below.
|
||||
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16"]
|
||||
base_env: {}
|
||||
features:
|
||||
reasoning:
|
||||
description: "Extract Jais-3 reasoning into message.reasoning"
|
||||
args: ["--reasoning-parser", "k2_horizon"]
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with the K2-Horizon parser"
|
||||
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 911
|
||||
description: "BF16 serving runtime with an eight-GPU tensor-parallel baseline"
|
||||
compatible_strategies: [single_node_tep, single_node_tp, multi_node_tep, multi_node_tp_pp]
|
||||
default_strategy: single_node_tep
|
||||
|
||||
# On AMD the expert-parallel dispatch is slower than replicating the experts
|
||||
# across a pure tensor-parallel shard, so the TEP strategies are opted out and
|
||||
# single-node TP is the offered path. multi_node_tp_pp is left available: EP
|
||||
# works there, it is just not the fast default at single-node scale.
|
||||
strategy_hardware:
|
||||
single_node_tep:
|
||||
amd: unsupported
|
||||
multi_node_tep:
|
||||
amd: unsupported
|
||||
hardware_overrides:
|
||||
amd:
|
||||
extra_args:
|
||||
- "--attention-backend"
|
||||
- "ROCM_AITER_UNIFIED_ATTN"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.9"
|
||||
extra_env:
|
||||
VLLM_ROCM_USE_AITER: "1"
|
||||
NCCL_MIN_NCHANNELS: "112"
|
||||
strategy_overrides:
|
||||
single_node_tep:
|
||||
tp: 8
|
||||
# multi_node_tp_pp emits no EP flag of its own, and this recipe has always
|
||||
# run it with expert parallel on. Restored here rather than in base_args so
|
||||
# it stays off the AMD single-node TP path.
|
||||
multi_node_tp_pp:
|
||||
extra_args: ["--enable-expert-parallel"]
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
Jais-3-375B-A23B is a frontier-scale mixture-of-expert (MoE) model in the IFM
|
||||
Jais-3 family, with 379.17B stored parameters including embeddings,
|
||||
26.67B active parameters per token, and support for up to 524,288 tokens of
|
||||
context.
|
||||
|
||||
Jais-3 is an open-weight IFM model family built for transparent
|
||||
foundation-model research, staged checkpoint analysis, and practical
|
||||
deployment. The series spans compact dense models for local experimentation,
|
||||
dense mid-size and large models for high-quality research workloads, and
|
||||
mixture-of-expert (MoE) models such as Jais-3-375B-A23B for frontier-scale
|
||||
research and higher-capacity serving.
|
||||
|
||||
## Launch Commands
|
||||
|
||||
```bash
|
||||
vllm serve IFM/Jais-3-375B-A23B \
|
||||
--tensor-parallel-size 8 \
|
||||
--trust-remote-code \
|
||||
--chat-template-content-format string \
|
||||
--dtype bfloat16 \
|
||||
--enable-expert-parallel \
|
||||
--reasoning-parser k2_horizon \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser k2_horizon
|
||||
```
|
||||
|
||||
### AMD MI355X
|
||||
|
||||
Verified on an 8-GPU MI355X (gfx950) node. AITER supplies the fused MoE and
|
||||
the unified attention backend:
|
||||
|
||||
```bash
|
||||
export VLLM_ROCM_USE_AITER=1
|
||||
export NCCL_MIN_NCHANNELS=112
|
||||
|
||||
vllm serve IFM/Jais-3-375B-A23B \
|
||||
--tensor-parallel-size 8 \
|
||||
--trust-remote-code \
|
||||
--chat-template-content-format string \
|
||||
--dtype bfloat16 \
|
||||
--max-model-len 131072 \
|
||||
--gpu-memory-utilization 0.9 \
|
||||
--attention-backend ROCM_AITER_UNIFIED_ATTN \
|
||||
--generation-config auto \
|
||||
--reasoning-parser k2_horizon \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser k2_horizon
|
||||
```
|
||||
|
||||
## Client usage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
|
||||
)
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="IFM/Jais-3-375B-A23B",
|
||||
messages=[{"role": "user", "content": "لدينا عددان مجموعهما 50 وحاصل ضربهما 600. ما هما العددان؟ اشرح خطوات الحل."}],
|
||||
temperature=1.0, top_p=0.95, max_tokens=2048,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Thinking modes
|
||||
|
||||
The model supports selectable thinking through `chat_template_kwargs`, per
|
||||
request or server-wide via `--default-chat-template-kwargs`:
|
||||
|
||||
- `{"reasoning_effort": "high"}` — full thinking (default).
|
||||
- `{"reasoning_effort": "low"}` — no thinking.
|
||||
|
||||
## References
|
||||
|
||||
- [IFM/Jais-3-375B-A23B on Hugging Face](https://huggingface.co/IFM/Jais-3-375B-A23B)
|
||||
- [Jais 3 Collection](https://huggingface.co/collections/IFM/jais-3)
|
||||
@@ -0,0 +1,89 @@
|
||||
meta:
|
||||
title: "Jais-3-7B"
|
||||
slug: "jais-3-7b"
|
||||
provider: "IFM"
|
||||
description: "Dense model for long-context research, fine-tuning, and cost-conscious deployment"
|
||||
date_added: 2026-10-05
|
||||
date_updated: 2026-10-05
|
||||
tasks: [text]
|
||||
performance_headline: "9B stored parameters including embeddings, with up to 512K context"
|
||||
related_recipes: ["IFM/Jais-3-3.7B", "IFM/Jais-3-32B"]
|
||||
hardware:
|
||||
h200: verified
|
||||
model:
|
||||
model_id: "IFM/Jais-3-7B"
|
||||
architecture: dense
|
||||
parameter_count: "9B"
|
||||
active_parameters: "9B"
|
||||
context_length: 524288
|
||||
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16"]
|
||||
base_env: {}
|
||||
features:
|
||||
reasoning:
|
||||
description: "Extract Jais-3 reasoning into message.reasoning"
|
||||
args: ["--reasoning-parser", "k2_horizon"]
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with the K2-Horizon parser"
|
||||
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 22
|
||||
description: "BF16 serving runtime with a 131K context baseline"
|
||||
compatible_strategies: [single_node_tp, multi_node_tp, multi_node_tp_pp]
|
||||
hardware_overrides: {}
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
Jais-3-7B is a compact dense model in the IFM Jais-3 family,
|
||||
with 9B stored parameters including embeddings and support for up to 524,288
|
||||
tokens of context.
|
||||
|
||||
Jais-3 is an open-weight IFM model family built for transparent
|
||||
foundation-model research, staged checkpoint analysis, and practical
|
||||
deployment. The series spans compact dense models such as Jais-3-7B for
|
||||
local experimentation, dense mid-size and large models for high-quality
|
||||
research workloads, and mixture-of-expert (MoE) models for higher-capacity
|
||||
serving.
|
||||
|
||||
## Launch Commands
|
||||
|
||||
```bash
|
||||
vllm serve IFM/Jais-3-7B \
|
||||
--trust-remote-code \
|
||||
--chat-template-content-format string \
|
||||
--dtype bfloat16 \
|
||||
--reasoning-parser k2_horizon \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser k2_horizon
|
||||
```
|
||||
|
||||
## Client usage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
|
||||
)
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="IFM/Jais-3-7B",
|
||||
messages=[{"role": "user", "content": "لدينا عددان مجموعهما 50 وحاصل ضربهما 600. ما هما العددان؟ اشرح خطوات الحل."}],
|
||||
temperature=1.0, top_p=0.95, max_tokens=2048,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Thinking modes
|
||||
|
||||
The model supports selectable thinking through `chat_template_kwargs`, per
|
||||
request or server-wide via `--default-chat-template-kwargs`:
|
||||
|
||||
- `{"reasoning_effort": "high"}` — full thinking (default).
|
||||
- `{"reasoning_effort": "low"}` — no thinking.
|
||||
|
||||
## References
|
||||
|
||||
- [IFM/Jais-3-7B on Hugging Face](https://huggingface.co/IFM/Jais-3-7B)
|
||||
- [Jais 3 Collection](https://huggingface.co/collections/IFM/jais-3)
|
||||
@@ -0,0 +1,96 @@
|
||||
meta:
|
||||
title: "Jais-3-MoVA-36B-A4B"
|
||||
slug: "jais-3-mova-36b-a4b"
|
||||
provider: "IFM"
|
||||
description: "Sparse MoVA and MoE long-context model for research and production-style serving experiments"
|
||||
date_added: 2026-10-05
|
||||
date_updated: 2026-10-05
|
||||
tasks: [text]
|
||||
performance_headline: "37.44B stored parameters including embeddings / 5.95B active parameters per token"
|
||||
related_recipes: ["IFM/Jais-3-32B", "IFM/Jais-3-375B-A23B"]
|
||||
hardware:
|
||||
h200: verified
|
||||
model:
|
||||
model_id: "IFM/Jais-3-MoVA-36B-A4B"
|
||||
architecture: moe
|
||||
parameter_count: "37.44B"
|
||||
active_parameters: "5.95B"
|
||||
context_length: 524288
|
||||
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16", "--enable-expert-parallel"]
|
||||
base_env: {}
|
||||
features:
|
||||
reasoning:
|
||||
description: "Extract Jais-3 reasoning into message.reasoning"
|
||||
args: ["--reasoning-parser", "k2_horizon"]
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with the K2-Horizon parser"
|
||||
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 90
|
||||
description: "BF16 serving runtime with native vLLM execution"
|
||||
compatible_strategies: [single_node_tep, multi_node_tep, multi_node_tp_pp]
|
||||
default_strategy: single_node_tep
|
||||
hardware_overrides: {}
|
||||
strategy_overrides:
|
||||
single_node_tep:
|
||||
tp: 2
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
Jais-3-MoVA-36B-A4B is a mixture-of-expert (MoE) model in the IFM Jais-3
|
||||
family. It combines MoVA attention and MoE layers, with 37.44B stored parameters
|
||||
including embeddings, 5.95B active parameters per token, and support for up
|
||||
to 524,288 tokens of context.
|
||||
|
||||
Jais-3 is an open-weight IFM model family built for transparent
|
||||
foundation-model research, staged checkpoint analysis, and practical
|
||||
deployment. The series spans compact dense models for local experimentation,
|
||||
dense mid-size and large models for high-quality research workloads, and
|
||||
mixture-of-expert (MoE) models such as Jais-3-MoVA-36B-A4B for higher-capacity
|
||||
serving.
|
||||
|
||||
## Launch Commands
|
||||
|
||||
```bash
|
||||
vllm serve IFM/Jais-3-MoVA-36B-A4B \
|
||||
--tensor-parallel-size 2 \
|
||||
--trust-remote-code \
|
||||
--chat-template-content-format string \
|
||||
--dtype bfloat16 \
|
||||
--enable-expert-parallel \
|
||||
--reasoning-parser k2_horizon \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser k2_horizon
|
||||
```
|
||||
|
||||
## Client usage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
|
||||
)
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="IFM/Jais-3-MoVA-36B-A4B",
|
||||
messages=[{"role": "user", "content": "لدينا عددان مجموعهما 50 وحاصل ضربهما 600. ما هما العددان؟ اشرح خطوات الحل."}],
|
||||
temperature=1.0, top_p=0.95, max_tokens=2048,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Thinking modes
|
||||
|
||||
The model supports selectable thinking through `chat_template_kwargs`, per
|
||||
request or server-wide via `--default-chat-template-kwargs`:
|
||||
|
||||
- `{"reasoning_effort": "high"}` — full thinking (default).
|
||||
- `{"reasoning_effort": "low"}` — no thinking.
|
||||
|
||||
## References
|
||||
|
||||
- [IFM/Jais-3-MoVA-36B-A4B on Hugging Face](https://huggingface.co/IFM/Jais-3-MoVA-36B-A4B)
|
||||
- [Jais 3 Collection](https://huggingface.co/collections/IFM/jais-3)
|
||||
@@ -0,0 +1,110 @@
|
||||
meta:
|
||||
title: "K2-Horizon-0.9B"
|
||||
slug: "k2-horizon-0.9b"
|
||||
provider: "IFM"
|
||||
description: "Compact dense model for mathematics, code, instruction following, and STEM tasks"
|
||||
date_added: 2026-08-31
|
||||
date_updated: 2026-08-31
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "1.08B stored parameters including embeddings, with 131K context and native K2-Horizon parsing"
|
||||
related_recipes:
|
||||
- "IFM/K2-Horizon-3.7B"
|
||||
hardware:
|
||||
h200: verified
|
||||
model:
|
||||
model_id: "IFM/K2-Horizon-0.9B"
|
||||
architecture: dense
|
||||
parameter_count: "1.08B"
|
||||
active_parameters: "1.08B"
|
||||
context_length: 131072
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
- "--chat-template-content-format"
|
||||
- "string"
|
||||
- "--dtype"
|
||||
- "bfloat16"
|
||||
- "--max-model-len"
|
||||
- "131072"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.85"
|
||||
base_env: {}
|
||||
features:
|
||||
reasoning:
|
||||
description: "Extract K2-Horizon reasoning into message.reasoning"
|
||||
args: ["--reasoning-parser", "k2_horizon"]
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with the K2-Horizon parser"
|
||||
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 3
|
||||
description: "Released BF16 checkpoint; the command starts at an 8K runtime context"
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
- multi_node_tp_pp
|
||||
hardware_overrides: {}
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
K2-Horizon-0.9B is a compact dense model in the IFM K2-Horizon
|
||||
family, with 1.08B stored parameters including embeddings and a 131,072-token
|
||||
context window.
|
||||
|
||||
K2-Horizon is an open-weight IFM model family built for transparent
|
||||
foundation-model research, staged checkpoint analysis, and practical
|
||||
deployment. The series spans compact dense models such as K2-Horizon-0.9B
|
||||
for local experimentation, dense mid-size and large models for high-quality
|
||||
research workloads, and mixture-of-expert (MoE) models for higher-capacity
|
||||
serving.
|
||||
|
||||
## Launch Commands
|
||||
|
||||
K2-Horizon-0.9B's `config.json` already enables YaRN, so the full 131,072-token context needs no RoPE override.
|
||||
|
||||
```bash
|
||||
vllm serve IFM/K2-Horizon-0.9B \
|
||||
--trust-remote-code \
|
||||
--chat-template-content-format string \
|
||||
--dtype bfloat16 \
|
||||
--max-model-len 131072 \
|
||||
--gpu-memory-utilization 0.85 \
|
||||
--reasoning-parser k2_horizon \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser k2_horizon
|
||||
```
|
||||
|
||||
Lower `--max-model-len` (e.g. 8192) to reduce KV-cache memory.
|
||||
|
||||
## Client usage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
|
||||
)
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="IFM/K2-Horizon-0.9B",
|
||||
messages=[{"role": "user", "content": "Give me three primes above 100."}],
|
||||
temperature=1.0, top_p=0.95, max_tokens=2048,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Thinking modes
|
||||
|
||||
The model supports selectable thinking through `chat_template_kwargs`, per
|
||||
request or server-wide via `--default-chat-template-kwargs`:
|
||||
|
||||
- `{"reasoning_effort": "high"}` — full thinking (default).
|
||||
- `{"reasoning_effort": "medium"}` — faster thinking.
|
||||
- `{"reasoning_effort": "low"}` — fastest thinking.
|
||||
|
||||
## References
|
||||
|
||||
- [IFM/K2-Horizon-0.9B on Hugging Face](https://huggingface.co/IFM/K2-Horizon-0.9B)
|
||||
- [K2 Horizon Collection](https://huggingface.co/collections/IFM/k2-horizon)
|
||||
@@ -0,0 +1,89 @@
|
||||
meta:
|
||||
title: "K2-Horizon-3.7B"
|
||||
slug: "k2-horizon-3.7b"
|
||||
provider: "IFM"
|
||||
description: "Dense model for long-context research, serving, and downstream adaptation"
|
||||
date_added: 2026-08-31
|
||||
date_updated: 2026-09-02
|
||||
tasks: [text]
|
||||
performance_headline: "5.06B stored parameters including embeddings, with up to 512K context"
|
||||
related_recipes: ["IFM/K2-Horizon-0.9B", "IFM/K2-Horizon-7B"]
|
||||
hardware:
|
||||
h200: verified
|
||||
model:
|
||||
model_id: "IFM/K2-Horizon-3.7B"
|
||||
architecture: dense
|
||||
parameter_count: "5.06B"
|
||||
active_parameters: "5.06B"
|
||||
context_length: 524288
|
||||
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16"]
|
||||
base_env: {}
|
||||
features:
|
||||
reasoning:
|
||||
description: "Extract K2-Horizon reasoning into message.reasoning"
|
||||
args: ["--reasoning-parser", "k2_horizon"]
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with the K2-Horizon parser"
|
||||
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 13
|
||||
description: "BF16 serving runtime with a 131K context baseline"
|
||||
compatible_strategies: [single_node_tp, multi_node_tp, multi_node_tp_pp]
|
||||
hardware_overrides: {}
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
K2-Horizon-3.7B is a small dense model in the IFM K2-Horizon family,
|
||||
with 5.06B stored parameters including embeddings and support for up to
|
||||
524,288 tokens of context.
|
||||
|
||||
K2-Horizon is an open-weight IFM model family built for transparent
|
||||
foundation-model research, staged checkpoint analysis, and practical
|
||||
deployment. The series spans compact dense models for local experimentation,
|
||||
dense mid-size and large models for high-quality research workloads, and
|
||||
mixture-of-expert (MoE) models for higher-capacity serving.
|
||||
|
||||
## Launch Commands
|
||||
|
||||
```bash
|
||||
vllm serve IFM/K2-Horizon-3.7B \
|
||||
--trust-remote-code \
|
||||
--chat-template-content-format string \
|
||||
--dtype bfloat16 \
|
||||
--reasoning-parser k2_horizon \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser k2_horizon
|
||||
```
|
||||
|
||||
## Client usage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
|
||||
)
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="IFM/K2-Horizon-3.7B",
|
||||
messages=[{"role": "user", "content": "Give me three primes above 100."}],
|
||||
temperature=1.0, top_p=0.95, max_tokens=2048,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Thinking modes
|
||||
|
||||
The model supports selectable thinking through `chat_template_kwargs`, per
|
||||
request or server-wide via `--default-chat-template-kwargs`:
|
||||
|
||||
- `{"reasoning_effort": "high"}` — full thinking (default).
|
||||
- `{"reasoning_effort": "medium"}` — faster thinking.
|
||||
- `{"reasoning_effort": "low"}` — fastest thinking.
|
||||
|
||||
## References
|
||||
|
||||
- [IFM/K2-Horizon-3.7B on Hugging Face](https://huggingface.co/IFM/K2-Horizon-3.7B)
|
||||
- [K2 Horizon Collection](https://huggingface.co/collections/IFM/k2-horizon)
|
||||
@@ -0,0 +1,94 @@
|
||||
meta:
|
||||
title: "K2-Horizon-32B"
|
||||
slug: "k2-horizon-32b"
|
||||
provider: "IFM"
|
||||
description: "Dense long-context model for reasoning research and high-capacity serving"
|
||||
date_added: 2026-08-31
|
||||
date_updated: 2026-09-02
|
||||
tasks: [text]
|
||||
performance_headline: "34.78B stored parameters including embeddings, with native vLLM execution and up to 512K context"
|
||||
related_recipes: ["IFM/K2-Horizon-7B", "IFM/K2-Horizon-MoVA-36B-A4B"]
|
||||
hardware:
|
||||
h200: verified
|
||||
model:
|
||||
model_id: "IFM/K2-Horizon-32B"
|
||||
architecture: dense
|
||||
parameter_count: "34.78B"
|
||||
active_parameters: "34.78B"
|
||||
context_length: 524288
|
||||
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16"]
|
||||
base_env: {}
|
||||
features:
|
||||
reasoning:
|
||||
description: "Extract K2-Horizon reasoning into message.reasoning"
|
||||
args: ["--reasoning-parser", "k2_horizon"]
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with the K2-Horizon parser"
|
||||
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 84
|
||||
description: "BF16 serving runtime with native vLLM execution"
|
||||
compatible_strategies: [single_node_tp, multi_node_tp, multi_node_tp_pp]
|
||||
hardware_overrides: {}
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
tp: 2
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
K2-Horizon-32B is a large dense model in the IFM K2-Horizon family,
|
||||
with 34.78B stored parameters including embeddings and support for up to
|
||||
524,288 tokens of context.
|
||||
|
||||
K2-Horizon is an open-weight IFM model family built for transparent
|
||||
foundation-model research, staged checkpoint analysis, and practical
|
||||
deployment. The series spans compact dense models for local experimentation,
|
||||
dense mid-size and large models such as K2-Horizon-32B for high-quality
|
||||
research workloads, and mixture-of-expert (MoE) models for higher-capacity
|
||||
serving.
|
||||
|
||||
## Launch Commands
|
||||
|
||||
```bash
|
||||
vllm serve IFM/K2-Horizon-32B \
|
||||
--tensor-parallel-size 2 \
|
||||
--trust-remote-code \
|
||||
--chat-template-content-format string \
|
||||
--dtype bfloat16 \
|
||||
--reasoning-parser k2_horizon \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser k2_horizon
|
||||
```
|
||||
|
||||
## Client usage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
|
||||
)
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="IFM/K2-Horizon-32B",
|
||||
messages=[{"role": "user", "content": "Give me three primes above 100."}],
|
||||
temperature=1.0, top_p=0.95, max_tokens=2048,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Thinking modes
|
||||
|
||||
The model supports selectable thinking through `chat_template_kwargs`, per
|
||||
request or server-wide via `--default-chat-template-kwargs`:
|
||||
|
||||
- `{"reasoning_effort": "high"}` — full thinking (default).
|
||||
- `{"reasoning_effort": "medium"}` — faster thinking.
|
||||
- `{"reasoning_effort": "low"}` — fastest thinking.
|
||||
|
||||
## References
|
||||
|
||||
- [IFM/K2-Horizon-32B on Hugging Face](https://huggingface.co/IFM/K2-Horizon-32B)
|
||||
- [K2 Horizon Collection](https://huggingface.co/collections/IFM/k2-horizon)
|
||||
@@ -0,0 +1,149 @@
|
||||
meta:
|
||||
title: "K2-Horizon-375B-A23B"
|
||||
slug: "k2-horizon-375b-a23b"
|
||||
provider: "IFM"
|
||||
description: "Frontier-scale sparse MoE model for long-context research and high-capacity serving"
|
||||
date_added: 2026-08-31
|
||||
date_updated: 2026-09-08
|
||||
tasks: [text]
|
||||
performance_headline: "379.17B stored parameters including embeddings / 26.67B active parameters per token"
|
||||
related_recipes: ["IFM/K2-Horizon-MoVA-36B-A4B"]
|
||||
hardware:
|
||||
h200: verified
|
||||
mi355x: verified
|
||||
model:
|
||||
model_id: "IFM/K2-Horizon-375B-A23B"
|
||||
architecture: moe
|
||||
parameter_count: "379.17B"
|
||||
active_parameters: "26.67B"
|
||||
context_length: 524288
|
||||
# --enable-expert-parallel is not here: in base_args it would reach the AMD
|
||||
# path too, which deliberately runs pure TP (see strategy_hardware). The TEP
|
||||
# strategies emit it themselves; multi_node_tp_pp does not, so it carries its
|
||||
# own copy in strategy_overrides below.
|
||||
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16"]
|
||||
base_env: {}
|
||||
features:
|
||||
reasoning:
|
||||
description: "Extract K2-Horizon reasoning into message.reasoning"
|
||||
args: ["--reasoning-parser", "k2_horizon"]
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with the K2-Horizon parser"
|
||||
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 911
|
||||
description: "BF16 serving runtime with an eight-GPU tensor-parallel baseline"
|
||||
compatible_strategies: [single_node_tep, single_node_tp, multi_node_tep, multi_node_tp_pp]
|
||||
default_strategy: single_node_tep
|
||||
|
||||
# On AMD the expert-parallel dispatch is slower than replicating the experts
|
||||
# across a pure tensor-parallel shard, so the TEP strategies are opted out and
|
||||
# single-node TP is the offered path. multi_node_tp_pp is left available: EP
|
||||
# works there, it is just not the fast default at single-node scale.
|
||||
strategy_hardware:
|
||||
single_node_tep:
|
||||
amd: unsupported
|
||||
multi_node_tep:
|
||||
amd: unsupported
|
||||
hardware_overrides:
|
||||
amd:
|
||||
extra_args:
|
||||
- "--attention-backend"
|
||||
- "ROCM_AITER_UNIFIED_ATTN"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.9"
|
||||
extra_env:
|
||||
VLLM_ROCM_USE_AITER: "1"
|
||||
NCCL_MIN_NCHANNELS: "112"
|
||||
strategy_overrides:
|
||||
single_node_tep:
|
||||
tp: 8
|
||||
# multi_node_tp_pp emits no EP flag of its own, and this recipe has always
|
||||
# run it with expert parallel on. Restored here rather than in base_args so
|
||||
# it stays off the AMD single-node TP path.
|
||||
multi_node_tp_pp:
|
||||
extra_args: ["--enable-expert-parallel"]
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
K2-Horizon-375B-A23B is a frontier-scale mixture-of-expert (MoE) model in the IFM
|
||||
K2-Horizon family, with 379.17B stored parameters including embeddings,
|
||||
26.67B active parameters per token, and support for up to 524,288 tokens of
|
||||
context.
|
||||
|
||||
K2-Horizon is an open-weight IFM model family built for transparent
|
||||
foundation-model research, staged checkpoint analysis, and practical
|
||||
deployment. The series spans compact dense models for local experimentation,
|
||||
dense mid-size and large models for high-quality research workloads, and
|
||||
mixture-of-expert (MoE) models such as K2-Horizon-375B-A23B for frontier-scale
|
||||
research and higher-capacity serving.
|
||||
|
||||
## Launch Commands
|
||||
|
||||
```bash
|
||||
vllm serve IFM/K2-Horizon-375B-A23B \
|
||||
--tensor-parallel-size 8 \
|
||||
--trust-remote-code \
|
||||
--chat-template-content-format string \
|
||||
--dtype bfloat16 \
|
||||
--enable-expert-parallel \
|
||||
--reasoning-parser k2_horizon \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser k2_horizon
|
||||
```
|
||||
|
||||
### AMD MI355X
|
||||
|
||||
Verified on an 8-GPU MI355X (gfx950) node. AITER supplies the fused MoE and
|
||||
the unified attention backend:
|
||||
|
||||
```bash
|
||||
export VLLM_ROCM_USE_AITER=1
|
||||
export NCCL_MIN_NCHANNELS=112
|
||||
|
||||
vllm serve IFM/K2-Horizon-375B-A23B \
|
||||
--tensor-parallel-size 8 \
|
||||
--trust-remote-code \
|
||||
--chat-template-content-format string \
|
||||
--dtype bfloat16 \
|
||||
--max-model-len 131072 \
|
||||
--gpu-memory-utilization 0.9 \
|
||||
--attention-backend ROCM_AITER_UNIFIED_ATTN \
|
||||
--generation-config auto \
|
||||
--reasoning-parser k2_horizon \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser k2_horizon
|
||||
```
|
||||
|
||||
## Client usage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
|
||||
)
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="IFM/K2-Horizon-375B-A23B",
|
||||
messages=[{"role": "user", "content": "Give me three primes above 100."}],
|
||||
temperature=1.0, top_p=0.95, max_tokens=2048,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Thinking modes
|
||||
|
||||
The model supports selectable thinking through `chat_template_kwargs`, per
|
||||
request or server-wide via `--default-chat-template-kwargs`:
|
||||
|
||||
- `{"reasoning_effort": "high"}` — full thinking (default).
|
||||
- `{"reasoning_effort": "medium"}` — faster thinking.
|
||||
- `{"reasoning_effort": "low"}` — fastest thinking.
|
||||
|
||||
## References
|
||||
|
||||
- [IFM/K2-Horizon-375B-A23B on Hugging Face](https://huggingface.co/IFM/K2-Horizon-375B-A23B)
|
||||
- [K2 Horizon Collection](https://huggingface.co/collections/IFM/k2-horizon)
|
||||
@@ -0,0 +1,90 @@
|
||||
meta:
|
||||
title: "K2-Horizon-7B"
|
||||
slug: "k2-horizon-7b"
|
||||
provider: "IFM"
|
||||
description: "Dense model for long-context research, fine-tuning, and cost-conscious deployment"
|
||||
date_added: 2026-08-31
|
||||
date_updated: 2026-09-02
|
||||
tasks: [text]
|
||||
performance_headline: "9B stored parameters including embeddings, with up to 512K context"
|
||||
related_recipes: ["IFM/K2-Horizon-3.7B", "IFM/K2-Horizon-32B"]
|
||||
hardware:
|
||||
h200: verified
|
||||
model:
|
||||
model_id: "IFM/K2-Horizon-7B"
|
||||
architecture: dense
|
||||
parameter_count: "9B"
|
||||
active_parameters: "9B"
|
||||
context_length: 524288
|
||||
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16"]
|
||||
base_env: {}
|
||||
features:
|
||||
reasoning:
|
||||
description: "Extract K2-Horizon reasoning into message.reasoning"
|
||||
args: ["--reasoning-parser", "k2_horizon"]
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with the K2-Horizon parser"
|
||||
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 22
|
||||
description: "BF16 serving runtime with a 131K context baseline"
|
||||
compatible_strategies: [single_node_tp, multi_node_tp, multi_node_tp_pp]
|
||||
hardware_overrides: {}
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
K2-Horizon-7B is a compact dense model in the IFM K2-Horizon family,
|
||||
with 9B stored parameters including embeddings and support for up to 524,288
|
||||
tokens of context.
|
||||
|
||||
K2-Horizon is an open-weight IFM model family built for transparent
|
||||
foundation-model research, staged checkpoint analysis, and practical
|
||||
deployment. The series spans compact dense models such as K2-Horizon-7B for
|
||||
local experimentation, dense mid-size and large models for high-quality
|
||||
research workloads, and mixture-of-expert (MoE) models for higher-capacity
|
||||
serving.
|
||||
|
||||
## Launch Commands
|
||||
|
||||
```bash
|
||||
vllm serve IFM/K2-Horizon-7B \
|
||||
--trust-remote-code \
|
||||
--chat-template-content-format string \
|
||||
--dtype bfloat16 \
|
||||
--reasoning-parser k2_horizon \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser k2_horizon
|
||||
```
|
||||
|
||||
## Client usage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
|
||||
)
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="IFM/K2-Horizon-7B",
|
||||
messages=[{"role": "user", "content": "Give me three primes above 100."}],
|
||||
temperature=1.0, top_p=0.95, max_tokens=2048,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Thinking modes
|
||||
|
||||
The model supports selectable thinking through `chat_template_kwargs`, per
|
||||
request or server-wide via `--default-chat-template-kwargs`:
|
||||
|
||||
- `{"reasoning_effort": "high"}` — full thinking (default).
|
||||
- `{"reasoning_effort": "medium"}` — faster thinking.
|
||||
- `{"reasoning_effort": "low"}` — fastest thinking.
|
||||
|
||||
## References
|
||||
|
||||
- [IFM/K2-Horizon-7B on Hugging Face](https://huggingface.co/IFM/K2-Horizon-7B)
|
||||
- [K2 Horizon Collection](https://huggingface.co/collections/IFM/k2-horizon)
|
||||
@@ -0,0 +1,97 @@
|
||||
meta:
|
||||
title: "K2-Horizon-MoVA-36B-A4B"
|
||||
slug: "k2-horizon-mova-36b-a4b"
|
||||
provider: "IFM"
|
||||
description: "Sparse MoVA and MoE long-context model for research and production-style serving experiments"
|
||||
date_added: 2026-08-31
|
||||
date_updated: 2026-09-03
|
||||
tasks: [text]
|
||||
performance_headline: "37.44B stored parameters including embeddings / 5.95B active parameters per token"
|
||||
related_recipes: ["IFM/K2-Horizon-32B", "IFM/K2-Horizon-375B-A23B"]
|
||||
hardware:
|
||||
h200: verified
|
||||
model:
|
||||
model_id: "IFM/K2-Horizon-MoVA-36B-A4B"
|
||||
architecture: moe
|
||||
parameter_count: "37.44B"
|
||||
active_parameters: "5.95B"
|
||||
context_length: 524288
|
||||
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16", "--enable-expert-parallel"]
|
||||
base_env: {}
|
||||
features:
|
||||
reasoning:
|
||||
description: "Extract K2-Horizon reasoning into message.reasoning"
|
||||
args: ["--reasoning-parser", "k2_horizon"]
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with the K2-Horizon parser"
|
||||
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 90
|
||||
description: "BF16 serving runtime with native vLLM execution"
|
||||
compatible_strategies: [single_node_tep, multi_node_tep, multi_node_tp_pp]
|
||||
default_strategy: single_node_tep
|
||||
hardware_overrides: {}
|
||||
strategy_overrides:
|
||||
single_node_tep:
|
||||
tp: 2
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
K2-Horizon-MoVA-36B-A4B is a mixture-of-expert (MoE) model in the IFM K2-Horizon
|
||||
family. It combines MoVA attention and MoE layers, with 37.44B stored parameters
|
||||
including embeddings, 5.95B active parameters per token, and support for up
|
||||
to 524,288 tokens of context.
|
||||
|
||||
K2-Horizon is an open-weight IFM model family built for transparent
|
||||
foundation-model research, staged checkpoint analysis, and practical
|
||||
deployment. The series spans compact dense models for local experimentation,
|
||||
dense mid-size and large models for high-quality research workloads, and
|
||||
mixture-of-expert (MoE) models such as K2-Horizon-MoVA-36B-A4B for higher-capacity
|
||||
serving.
|
||||
|
||||
## Launch Commands
|
||||
|
||||
```bash
|
||||
vllm serve IFM/K2-Horizon-MoVA-36B-A4B \
|
||||
--tensor-parallel-size 2 \
|
||||
--trust-remote-code \
|
||||
--chat-template-content-format string \
|
||||
--dtype bfloat16 \
|
||||
--enable-expert-parallel \
|
||||
--reasoning-parser k2_horizon \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser k2_horizon
|
||||
```
|
||||
|
||||
## Client usage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
|
||||
)
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="IFM/K2-Horizon-MoVA-36B-A4B",
|
||||
messages=[{"role": "user", "content": "Give me three primes above 100."}],
|
||||
temperature=1.0, top_p=0.95, max_tokens=2048,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Thinking modes
|
||||
|
||||
The model supports selectable thinking through `chat_template_kwargs`, per
|
||||
request or server-wide via `--default-chat-template-kwargs`:
|
||||
|
||||
- `{"reasoning_effort": "high"}` — full thinking (default).
|
||||
- `{"reasoning_effort": "medium"}` — faster thinking.
|
||||
- `{"reasoning_effort": "low"}` — fastest thinking.
|
||||
|
||||
## References
|
||||
|
||||
- [IFM/K2-Horizon-MoVA-36B-A4B on Hugging Face](https://huggingface.co/IFM/K2-Horizon-MoVA-36B-A4B)
|
||||
- [K2 Horizon Collection](https://huggingface.co/collections/IFM/k2-horizon)
|
||||
@@ -0,0 +1,153 @@
|
||||
meta:
|
||||
title: "Mellum2-12B-A2.5B-Instruct"
|
||||
slug: "mellum2-12b-a2.5b-instruct"
|
||||
provider: "JetBrains"
|
||||
description: "JetBrains' instruction-tuned code MoE (12B total / 2.5B active) that answers directly without an externalized chain of thought — low-latency coding and tool use"
|
||||
date_added: 2026-06-02
|
||||
date_updated: 2026-06-02
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "78.4 EvalPlus, 67.1 MultiPL-E — direct answers, fits on a single GPU"
|
||||
related_recipes:
|
||||
- "JetBrains/Mellum2-12B-A2.5B-Thinking"
|
||||
hardware:
|
||||
h200: verified
|
||||
mi300x: verified
|
||||
mi325x: verified
|
||||
mi355x: verified
|
||||
|
||||
model:
|
||||
model_id: "JetBrains/Mellum2-12B-A2.5B-Instruct"
|
||||
min_vllm_version: "0.23.0"
|
||||
architecture: moe
|
||||
parameter_count: "12B"
|
||||
active_parameters: "2.5B"
|
||||
context_length: 131072
|
||||
base_args:
|
||||
- "--max-model-len"
|
||||
- "131072"
|
||||
- "--trust-remote-code"
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Hermes tool-call parser for function calling"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "hermes"
|
||||
|
||||
opt_in_features: []
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 29
|
||||
description: "Native bfloat16 weights; fits comfortably on a single H200/H100/A100"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- single_node_tep
|
||||
- single_node_dep
|
||||
|
||||
hardware_overrides: {}
|
||||
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
tp: 1
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[Mellum2-12B-A2.5B-Instruct](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Instruct)
|
||||
is JetBrains' instruction-tuned code assistant. It shares the same Mixture-of-Experts
|
||||
backbone as the rest of the Mellum2 family — 64 experts (8 activated per token), 12B total
|
||||
/ 2.5B active parameters, sliding-window + full-attention layers, 131,072-token context —
|
||||
but is post-trained (SFT + RLVR on math, coding, tool use, instruction following, reasoning,
|
||||
and knowledge) to **answer directly, without an externalized chain of thought**. For complex
|
||||
debugging, multi-step planning, or math/reasoning-heavy tasks where you want explicit
|
||||
reasoning traces, use the
|
||||
[Thinking](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Thinking) variant instead.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Hardware: a single H200, H100, or A100 (~29 GB at bf16) is plenty. On AMD,
|
||||
use MI300X / MI325X / MI355X with TP=1.
|
||||
- vLLM **nightly** — `MellumForCausalLM` support landed after v0.22.0 and is not yet in a
|
||||
stable release. Install the nightly wheels until the next tagged release ships.
|
||||
|
||||
### Install vLLM (nightly)
|
||||
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --extra-index-url https://wheels.vllm.ai/nightly
|
||||
```
|
||||
|
||||
## Launch command
|
||||
|
||||
Unlike the Thinking checkpoint, Instruct does not emit `<think>` blocks, so no
|
||||
`--reasoning-parser` is needed.
|
||||
|
||||
### Single GPU
|
||||
|
||||
```bash
|
||||
vllm serve JetBrains/Mellum2-12B-A2.5B-Instruct \
|
||||
--max-model-len 131072 \
|
||||
--trust-remote-code \
|
||||
--tensor-parallel-size 1
|
||||
|
||||
# Optional: add tool calling.
|
||||
vllm serve JetBrains/Mellum2-12B-A2.5B-Instruct \
|
||||
--max-model-len 131072 \
|
||||
--trust-remote-code \
|
||||
--tensor-parallel-size 1 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser hermes
|
||||
```
|
||||
|
||||
### Docker (AMD MI300X / MI325X / MI355X)
|
||||
|
||||
MI300X / MI325X / MI355X GPUs have larger per-GPU HBM, so you can use
|
||||
`--max-model-len auto`.
|
||||
|
||||
```bash
|
||||
# Plain serving
|
||||
vllm serve JetBrains/Mellum2-12B-A2.5B-Instruct \
|
||||
--max-model-len auto \
|
||||
--trust-remote-code \
|
||||
--tensor-parallel-size 1
|
||||
|
||||
# Add tool calling.
|
||||
vllm serve JetBrains/Mellum2-12B-A2.5B-Instruct \
|
||||
--max-model-len auto \
|
||||
--trust-remote-code \
|
||||
--tensor-parallel-size 1 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser hermes
|
||||
```
|
||||
|
||||
## Client usage
|
||||
|
||||
JetBrains recommends sampling at `temperature=0.6`, `top_p=0.95`, `top_k=20`.
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="JetBrains/Mellum2-12B-A2.5B-Instruct",
|
||||
messages=[{"role": "user", "content": "Write a Python function to reverse a string."}],
|
||||
max_tokens=81920,
|
||||
temperature=0.6,
|
||||
top_p=0.95,
|
||||
extra_body={"top_k": 20},
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Instruct)
|
||||
- [vLLM support PR #43992](https://github.com/vllm-project/vllm/pull/43992)
|
||||
- [Mellum2 Technical Report](https://arxiv.org/abs/2605.31268)
|
||||
@@ -0,0 +1,146 @@
|
||||
meta:
|
||||
title: "Mellum2-12B-A2.5B-Thinking"
|
||||
slug: "mellum2-12b-a2.5b-thinking"
|
||||
provider: "JetBrains"
|
||||
description: "JetBrains' reasoning-augmented code MoE (12B total / 2.5B active) that emits explicit <think> chains for debugging, planning, and agentic coding"
|
||||
date_added: 2026-06-02
|
||||
date_updated: 2026-06-02
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "69.9 LiveCodeBench v6, 58.4 AIME — fits on a single GPU"
|
||||
related_recipes:
|
||||
- "JetBrains/Mellum2-12B-A2.5B-Instruct"
|
||||
hardware:
|
||||
h200: verified
|
||||
|
||||
model:
|
||||
model_id: "JetBrains/Mellum2-12B-A2.5B-Thinking"
|
||||
min_vllm_version: "0.23.0"
|
||||
architecture: moe
|
||||
parameter_count: "12B"
|
||||
active_parameters: "2.5B"
|
||||
context_length: 131072
|
||||
base_args:
|
||||
- "--max-model-len"
|
||||
- "131072"
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
reasoning:
|
||||
description: "Parse the model's <think>...</think> reasoning blocks (Qwen3-style parser)"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "qwen3"
|
||||
tool_calling:
|
||||
description: "Hermes tool-call parser for function calling"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "hermes"
|
||||
|
||||
spec_decoding:
|
||||
description: "DFlash speculative decoding with an externally trained draft model; the served checkpoint is unchanged."
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"model": "RedHatAI/Mellum2-12B-A2.5B-Thinking-speculator.dflash", "num_speculative_tokens": 7, "method": "dflash"}'
|
||||
|
||||
opt_in_features:
|
||||
- spec_decoding
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 29
|
||||
description: "Native bfloat16 weights; fits comfortably on a single H200/H100/A100"
|
||||
|
||||
fp8:
|
||||
model_id: "RedHatAI/Mellum2-12B-A2.5B-Thinking-FP8-Dynamic"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 15
|
||||
description: "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
|
||||
nvfp4:
|
||||
model_id: "RedHatAI/Mellum2-12B-A2.5B-Thinking-NVFP4"
|
||||
precision: nvfp4
|
||||
vram_minimum_gb: 8
|
||||
description: "RedHatAI NVFP4 quantization"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- single_node_tep
|
||||
- single_node_dep
|
||||
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
tp: 1
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[Mellum2-12B-A2.5B-Thinking](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Thinking)
|
||||
is JetBrains' reasoning-augmented code assistant. It uses a Mixture-of-Experts architecture
|
||||
with 64 experts (8 activated per token) — 12B total parameters, 2.5B active — combining
|
||||
sliding-window and full-attention layers for a 131,072-token context. The model emits its
|
||||
chain-of-thought inside `<think>...</think>` blocks before the final answer, making it
|
||||
suited to complex debugging, multi-step planning, and agentic workflows. For direct,
|
||||
low-latency answers without reasoning traces, use the
|
||||
[Instruct](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Instruct) variant instead.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Hardware: a single H200, H100, or A100 (~29 GB at bf16) is plenty
|
||||
- vLLM **nightly** — `MellumForCausalLM` support landed after v0.22.0 and is not yet in a
|
||||
stable release. Install the nightly wheels until the next tagged release ships.
|
||||
|
||||
### Install vLLM (nightly)
|
||||
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --extra-index-url https://wheels.vllm.ai/nightly
|
||||
```
|
||||
|
||||
## Launch command
|
||||
|
||||
```bash
|
||||
# With reasoning (recommended for the Thinking checkpoint)
|
||||
vllm serve JetBrains/Mellum2-12B-A2.5B-Thinking \
|
||||
--max-model-len 131072 \
|
||||
--reasoning-parser qwen3
|
||||
|
||||
# Add tool calling
|
||||
vllm serve JetBrains/Mellum2-12B-A2.5B-Thinking \
|
||||
--max-model-len 131072 \
|
||||
--reasoning-parser qwen3 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser hermes
|
||||
```
|
||||
|
||||
## Client usage
|
||||
|
||||
JetBrains recommends sampling at `temperature=0.6`, `top_p=0.95`, `top_k=20` for the
|
||||
Thinking checkpoint.
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="JetBrains/Mellum2-12B-A2.5B-Thinking",
|
||||
messages=[{"role": "user", "content": "Is 1024 a power of 2? Explain your reasoning."}],
|
||||
max_tokens=81920,
|
||||
temperature=0.6,
|
||||
top_p=0.95,
|
||||
extra_body={"top_k": 20},
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Thinking)
|
||||
- [vLLM support PR #43992](https://github.com/vllm-project/vllm/pull/43992)
|
||||
- [Mellum2 Technical Report](https://arxiv.org/abs/2605.31268)
|
||||
@@ -0,0 +1,137 @@
|
||||
meta:
|
||||
title: "LFM2.5 1.2B Base"
|
||||
slug: "lfm2.5-1.2b-base"
|
||||
provider: "LiquidAI"
|
||||
description: "Liquid AI's 1.2B pretrained base model on the LFM2 hybrid conv+attention backbone — a text-completion and fine-tuning foundation (no chat template)."
|
||||
date_added: 2026-06-25
|
||||
date_updated: 2026-06-24
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "1.2B pretrained base (no chat template) — completions endpoint, ideal as a fine-tuning base"
|
||||
related_recipes:
|
||||
- LiquidAI/LFM2.5-1.2B-Instruct
|
||||
- LiquidAI/LFM2.5-350M
|
||||
hardware:
|
||||
h100: verified
|
||||
h200: verified
|
||||
b200: verified
|
||||
|
||||
model:
|
||||
model_id: "LiquidAI/LFM2.5-1.2B-Base"
|
||||
min_vllm_version: "0.23.0"
|
||||
architecture: dense
|
||||
parameter_count: "1.2B"
|
||||
active_parameters: "1.2B"
|
||||
context_length: 32768
|
||||
base_args: []
|
||||
base_env: {}
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 3
|
||||
description: "Full BF16 — single 8 GB+ NVIDIA GPU with room for KV cache"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
|
||||
hardware_overrides:
|
||||
# Blackwell (B200) only: route RMSNorm to the bundled oink kernels (measured ~+2.6% on B200,
|
||||
# output identical). Inert (falls back to native) on non-Blackwell GPUs.
|
||||
blackwell:
|
||||
extra_env:
|
||||
VLLM_USE_OINK_OPS: "1"
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
tp: 1
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[LFM2.5-1.2B-Base](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Base) is the pretrained base
|
||||
checkpoint behind [Liquid AI's](https://www.liquid.ai/) LFM2.5-1.2B instruct models, built on
|
||||
the **LFM2 hybrid backbone** (short-range gated convolution blocks interleaved with grouped-query
|
||||
attention). It is **not** instruction-tuned and has no chat template — use the
|
||||
`/v1/completions` endpoint (raw text continuation), or fine-tune it for your own task.
|
||||
|
||||
### Key Features
|
||||
- **Pretrained base**: No chat template, no instruction tuning — a clean foundation for fine-tuning or raw completion.
|
||||
- **Hybrid backbone**: Gated short convolutions interleaved with grouped-query attention — a smaller KV cache and lower decode latency than a same-size full-attention transformer.
|
||||
- **32K context**: 32,768-token context window.
|
||||
- **Native vLLM support**: Served via the `Lfm2ForCausalLM` architecture.
|
||||
|
||||
### Supported Variants
|
||||
|
||||
Dense:
|
||||
- `LiquidAI/LFM2.5-230M` (230M)
|
||||
- `LiquidAI/LFM2.5-350M` (350M)
|
||||
- `LiquidAI/LFM2.5-1.2B-Instruct` (1.2B)
|
||||
- `LiquidAI/LFM2.5-1.2B-Thinking` (1.2B, reasoning)
|
||||
- `LiquidAI/LFM2.5-1.2B-JP` / `LiquidAI/LFM2.5-1.2B-JP-202606` (Japanese)
|
||||
- `LiquidAI/LFM2.5-1.2B-Base` (pretrained base)
|
||||
|
||||
MoE:
|
||||
- `LiquidAI/LFM2.5-8B-A1B` (8B total / ~1B active)
|
||||
|
||||
Vision-Language:
|
||||
- `LiquidAI/LFM2.5-VL-450M`, `LiquidAI/LFM2.5-VL-1.6B`
|
||||
|
||||
See the [LFM2.5 usage guide](../../LiquidAI/LFM2.5) for the full family.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Hardware**: 1× GPU with ≥8 GB VRAM.
|
||||
- **vLLM**: ≥ 0.23.0.
|
||||
|
||||
### pip (NVIDIA CUDA)
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend auto
|
||||
```
|
||||
|
||||
## Deployment Configurations
|
||||
|
||||
### Quick Start (Single GPU, BF16)
|
||||
```bash
|
||||
vllm serve LiquidAI/LFM2.5-1.2B-Base
|
||||
```
|
||||
|
||||
### Docker (NVIDIA)
|
||||
```bash
|
||||
docker run -itd --name lfm2.5-base \
|
||||
--ipc=host --network host --shm-size 16G --gpus all \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai:latest \
|
||||
--model LiquidAI/LFM2.5-1.2B-Base \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
### Text Completion
|
||||
This is a base model — use the **completions** endpoint, not chat. Recommended sampling:
|
||||
temperature `0.3`, `min_p 0.15`, `repetition_penalty 1.05`.
|
||||
```python
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||||
response = client.completions.create(
|
||||
model="LiquidAI/LFM2.5-1.2B-Base",
|
||||
prompt="The three laws of thermodynamics are:",
|
||||
temperature=0.3,
|
||||
extra_body={"min_p": 0.15, "repetition_penalty": 1.05},
|
||||
)
|
||||
print(response.choices[0].text)
|
||||
```
|
||||
|
||||
## Configuration Tips
|
||||
- Use `/v1/completions` (raw continuation) — there is no chat template on a base model.
|
||||
- For fine-tuning, this checkpoint is the recommended starting point for custom LFM2.5-1.2B tasks.
|
||||
- Set `--max-model-len` to match your workload (up to 32K).
|
||||
- Sampling presets are per-request client defaults — don't bake them into `vllm serve`.
|
||||
|
||||
## References
|
||||
- [Model card](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Base)
|
||||
- [LFM2.5 usage guide](../../LiquidAI/LFM2.5)
|
||||
- [Liquid AI](https://www.liquid.ai/)
|
||||
- [vLLM supported models](https://docs.vllm.ai/en/latest/models/supported_models.html)
|
||||
@@ -0,0 +1,186 @@
|
||||
meta:
|
||||
title: "LFM2.5 1.2B Instruct"
|
||||
slug: "lfm2.5-1.2b-instruct"
|
||||
provider: "LiquidAI"
|
||||
description: "Liquid AI's 1.2B instruction-tuned model on the LFM2 hybrid conv+attention backbone, with tool calling and a 32K context window on a single small GPU."
|
||||
date_added: 2026-06-25
|
||||
date_updated: 2026-06-24
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "1.2B hybrid (gated conv + GQA) instruct model with tool calling — runs on a single 8 GB+ GPU"
|
||||
related_recipes:
|
||||
- LiquidAI/LFM2.5-350M
|
||||
- LiquidAI/LFM2.5-1.2B-Thinking
|
||||
- LiquidAI/LFM2.5-8B-A1B
|
||||
hardware:
|
||||
h100: verified
|
||||
h200: verified
|
||||
b200: verified
|
||||
|
||||
model:
|
||||
model_id: "LiquidAI/LFM2.5-1.2B-Instruct"
|
||||
min_vllm_version: "0.23.0"
|
||||
architecture: dense
|
||||
parameter_count: "1.2B"
|
||||
active_parameters: "1.2B"
|
||||
context_length: 32768
|
||||
base_args: []
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Surface LFM2.5's Pythonic tool calls (wrapped in <|tool_call_start|>…<|tool_call_end|>) as OpenAI tool_calls via vLLM's native lfm2 parser"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "lfm2"
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 3
|
||||
description: "Full BF16 — single 8 GB+ NVIDIA GPU with room for KV cache"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
|
||||
hardware_overrides:
|
||||
# Blackwell (B200) only: route RMSNorm to the bundled oink kernels (measured ~+2.6% on B200,
|
||||
# output identical). Inert (falls back to native) on non-Blackwell GPUs.
|
||||
blackwell:
|
||||
extra_env:
|
||||
VLLM_USE_OINK_OPS: "1"
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
tp: 1
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[LFM2.5-1.2B-Instruct](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Instruct) is a 1.2B
|
||||
instruction-tuned model from [Liquid AI](https://www.liquid.ai/), built on the **LFM2 hybrid
|
||||
backbone**: short-range gated convolution blocks interleaved with grouped-query attention.
|
||||
The hybrid design keeps the KV cache small and decode fast, so the model serves comfortably on
|
||||
a single small GPU while supporting a 32K context window and Pythonic tool calling — all
|
||||
through vLLM's OpenAI-compatible API.
|
||||
|
||||
### Key Features
|
||||
- **Hybrid backbone**: Gated short convolutions interleaved with grouped-query attention — a smaller KV cache and lower decode latency than a same-size full-attention transformer.
|
||||
- **32K context**: 32,768-token context window.
|
||||
- **Tool calling**: Pythonic tool calls (`<|tool_call_start|>…<|tool_call_end|>`) surfaced as OpenAI `tool_calls` by vLLM's native `lfm2` parser.
|
||||
- **Native vLLM support**: Served via the `Lfm2ForCausalLM` architecture.
|
||||
- **Edge-ready family**: Day-one support across llama.cpp, MLX, ONNX, vLLM, and SGLang.
|
||||
|
||||
### Supported Variants
|
||||
|
||||
Dense:
|
||||
- `LiquidAI/LFM2.5-230M` (230M)
|
||||
- `LiquidAI/LFM2.5-350M` (350M)
|
||||
- `LiquidAI/LFM2.5-1.2B-Instruct` (1.2B)
|
||||
- `LiquidAI/LFM2.5-1.2B-Thinking` (1.2B, reasoning)
|
||||
- `LiquidAI/LFM2.5-1.2B-JP` / `LiquidAI/LFM2.5-1.2B-JP-202606` (Japanese)
|
||||
- `LiquidAI/LFM2.5-1.2B-Base` (pretrained base)
|
||||
|
||||
MoE:
|
||||
- `LiquidAI/LFM2.5-8B-A1B` (8B total / ~1B active)
|
||||
|
||||
Vision-Language:
|
||||
- `LiquidAI/LFM2.5-VL-450M`, `LiquidAI/LFM2.5-VL-1.6B`
|
||||
|
||||
See the [LFM2.5 usage guide](../../LiquidAI/LFM2.5) for the full family.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Hardware**: 1× GPU with ≥8 GB VRAM (e.g. L4, A10, RTX 4090, H100).
|
||||
- **vLLM**: ≥ 0.23.0.
|
||||
|
||||
### pip (NVIDIA CUDA)
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend auto
|
||||
```
|
||||
|
||||
## Deployment Configurations
|
||||
|
||||
### Quick Start (Single GPU, BF16)
|
||||
```bash
|
||||
vllm serve LiquidAI/LFM2.5-1.2B-Instruct
|
||||
```
|
||||
Cap the context to fit a smaller GPU (the model supports up to 32K):
|
||||
```bash
|
||||
vllm serve LiquidAI/LFM2.5-1.2B-Instruct --max-model-len 32768
|
||||
```
|
||||
|
||||
### Full-Featured Server Launch
|
||||
Enables tool calling:
|
||||
```bash
|
||||
vllm serve LiquidAI/LFM2.5-1.2B-Instruct \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser lfm2 \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
### Docker (NVIDIA)
|
||||
```bash
|
||||
docker run -itd --name lfm2.5-1.2b \
|
||||
--ipc=host --network host --shm-size 16G --gpus all \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai:latest \
|
||||
--model LiquidAI/LFM2.5-1.2B-Instruct \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
### Text Generation
|
||||
The model card recommends temperature `0.1`, `top_k 50`, `repetition_penalty 1.05`. `top_k` and
|
||||
`repetition_penalty` are vLLM extra sampling params — pass them via `extra_body`.
|
||||
```python
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||||
response = client.chat.completions.create(
|
||||
model="LiquidAI/LFM2.5-1.2B-Instruct",
|
||||
messages=[{"role": "user", "content": "What is C. elegans? Answer in one sentence."}],
|
||||
temperature=0.1,
|
||||
extra_body={"top_k": 50, "repetition_penalty": 1.05},
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Tool Calling
|
||||
Launch with `--enable-auto-tool-choice --tool-call-parser lfm2`, then pass `tools=[…]`. The
|
||||
`lfm2` parser converts the model's Pythonic call into a standard `tool_calls` array.
|
||||
```python
|
||||
response = client.chat.completions.create(
|
||||
model="LiquidAI/LFM2.5-1.2B-Instruct",
|
||||
messages=[{"role": "user", "content": "What's the weather in Paris?"}],
|
||||
tools=[{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get current weather for a location",
|
||||
"parameters": {"type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"]},
|
||||
},
|
||||
}],
|
||||
temperature=0.1,
|
||||
extra_body={"top_k": 50, "repetition_penalty": 1.05},
|
||||
)
|
||||
print(response.choices[0].message.tool_calls)
|
||||
```
|
||||
|
||||
### Structured Outputs
|
||||
vLLM's guided decoding constrains output to a JSON schema via `response_format`. The model does
|
||||
not see the schema itself — put semantic instructions (units, formatting) in the system prompt.
|
||||
|
||||
## Configuration Tips
|
||||
- Set `--max-model-len` to match your workload (up to 32K); lowering it frees VRAM for KV cache.
|
||||
- `--gpu-memory-utilization 0.90–0.95` maximizes KV cache capacity.
|
||||
- FP8 KV cache (`--kv-cache-dtype fp8`) roughly halves KV memory.
|
||||
- Sampling presets are per-request client defaults — don't bake them into `vllm serve`.
|
||||
|
||||
## References
|
||||
- [Model card](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Instruct)
|
||||
- [LFM2.5 usage guide](../../LiquidAI/LFM2.5)
|
||||
- [Liquid AI](https://www.liquid.ai/)
|
||||
- [vLLM supported models](https://docs.vllm.ai/en/latest/models/supported_models.html)
|
||||
@@ -0,0 +1,158 @@
|
||||
meta:
|
||||
title: "LFM2.5 1.2B JP (202606)"
|
||||
slug: "lfm2.5-1.2b-jp-202606"
|
||||
provider: "LiquidAI"
|
||||
description: "Liquid AI's updated (2026-06) 1.2B Japanese chat model on the LFM2 hybrid conv+attention backbone — adds tool calling, with a 32K context window."
|
||||
date_added: 2026-06-25
|
||||
date_updated: 2026-06-24
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "Updated Japanese-tuned 1.2B hybrid chat model with tool calling — single small GPU"
|
||||
related_recipes:
|
||||
- LiquidAI/LFM2.5-1.2B-JP
|
||||
- LiquidAI/LFM2.5-1.2B-Instruct
|
||||
hardware:
|
||||
h100: verified
|
||||
h200: verified
|
||||
b200: verified
|
||||
|
||||
model:
|
||||
model_id: "LiquidAI/LFM2.5-1.2B-JP-202606"
|
||||
min_vllm_version: "0.23.0"
|
||||
architecture: dense
|
||||
parameter_count: "1.2B"
|
||||
active_parameters: "1.2B"
|
||||
context_length: 32768
|
||||
base_args: []
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Surface LFM2.5's Pythonic tool calls (wrapped in <|tool_call_start|>…<|tool_call_end|>) as OpenAI tool_calls via vLLM's native lfm2 parser"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "lfm2"
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 3
|
||||
description: "Full BF16 — single 8 GB+ NVIDIA GPU with room for KV cache"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
|
||||
hardware_overrides:
|
||||
# Blackwell (B200) only: route RMSNorm to the bundled oink kernels (measured ~+2.6% on B200,
|
||||
# output identical). Inert (falls back to native) on non-Blackwell GPUs.
|
||||
blackwell:
|
||||
extra_env:
|
||||
VLLM_USE_OINK_OPS: "1"
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
tp: 1
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[LFM2.5-1.2B-JP-202606](https://huggingface.co/LiquidAI/LFM2.5-1.2B-JP-202606) is the updated
|
||||
(June 2026) Japanese-specialized 1.2B chat model from [Liquid AI](https://www.liquid.ai/),
|
||||
adding tool calling on top of the **LFM2 hybrid backbone** (short-range gated convolution
|
||||
blocks interleaved with grouped-query attention). It serves on a single small GPU through
|
||||
vLLM's OpenAI-compatible API.
|
||||
|
||||
### Key Features
|
||||
- **Japanese-specialized**: Updated June 2026 checkpoint tuned for Japanese chat.
|
||||
- **Hybrid backbone**: Gated short convolutions interleaved with grouped-query attention — a smaller KV cache and lower decode latency than a same-size full-attention transformer.
|
||||
- **Tool calling**: Pythonic tool calls (`<|tool_call_start|>…<|tool_call_end|>`) surfaced as OpenAI `tool_calls` by vLLM's native `lfm2` parser.
|
||||
- **32K context**: 32,768-token context window.
|
||||
- **Native vLLM support**: Served via the `Lfm2ForCausalLM` architecture.
|
||||
|
||||
### Supported Variants
|
||||
|
||||
Dense:
|
||||
- `LiquidAI/LFM2.5-230M` (230M)
|
||||
- `LiquidAI/LFM2.5-350M` (350M)
|
||||
- `LiquidAI/LFM2.5-1.2B-Instruct` (1.2B)
|
||||
- `LiquidAI/LFM2.5-1.2B-Thinking` (1.2B, reasoning)
|
||||
- `LiquidAI/LFM2.5-1.2B-JP` / `LiquidAI/LFM2.5-1.2B-JP-202606` (Japanese)
|
||||
- `LiquidAI/LFM2.5-1.2B-Base` (pretrained base)
|
||||
|
||||
MoE:
|
||||
- `LiquidAI/LFM2.5-8B-A1B` (8B total / ~1B active)
|
||||
|
||||
Vision-Language:
|
||||
- `LiquidAI/LFM2.5-VL-450M`, `LiquidAI/LFM2.5-VL-1.6B`
|
||||
|
||||
See the [LFM2.5 usage guide](../../LiquidAI/LFM2.5) for the full family.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Hardware**: 1× GPU with ≥8 GB VRAM.
|
||||
- **vLLM**: ≥ 0.23.0.
|
||||
|
||||
### pip (NVIDIA CUDA)
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend auto
|
||||
```
|
||||
|
||||
## Deployment Configurations
|
||||
|
||||
### Quick Start (Single GPU, BF16)
|
||||
```bash
|
||||
vllm serve LiquidAI/LFM2.5-1.2B-JP-202606
|
||||
```
|
||||
|
||||
### Full-Featured Server Launch
|
||||
Enables tool calling:
|
||||
```bash
|
||||
vllm serve LiquidAI/LFM2.5-1.2B-JP-202606 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser lfm2 \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
### Docker (NVIDIA)
|
||||
```bash
|
||||
docker run -itd --name lfm2.5-jp-202606 \
|
||||
--ipc=host --network host --shm-size 16G --gpus all \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai:latest \
|
||||
--model LiquidAI/LFM2.5-1.2B-JP-202606 \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
### Text Generation
|
||||
The updated checkpoint uses temperature `0.1`, `top_k 50`, `repetition_penalty 1.05` (`top_k`
|
||||
and `repetition_penalty` ride in `extra_body`).
|
||||
```python
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||||
response = client.chat.completions.create(
|
||||
model="LiquidAI/LFM2.5-1.2B-JP-202606",
|
||||
messages=[{"role": "user", "content": "京都でおすすめの観光スポットを3つ教えてください。"}],
|
||||
temperature=0.1,
|
||||
extra_body={"top_k": 50, "repetition_penalty": 1.05},
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Tool Calling
|
||||
Launch with `--enable-auto-tool-choice --tool-call-parser lfm2`, then pass `tools=[…]`; the
|
||||
`lfm2` parser converts the model's Pythonic call into a standard `tool_calls` array.
|
||||
|
||||
## Configuration Tips
|
||||
- Set `--max-model-len` to match your workload (up to 32K); lowering it frees VRAM for KV cache.
|
||||
- `--gpu-memory-utilization 0.90–0.95` maximizes KV cache capacity.
|
||||
- Sampling presets are per-request client defaults — don't bake them into `vllm serve`.
|
||||
|
||||
## References
|
||||
- [Model card](https://huggingface.co/LiquidAI/LFM2.5-1.2B-JP-202606)
|
||||
- [LFM2.5 usage guide](../../LiquidAI/LFM2.5)
|
||||
- [Liquid AI](https://www.liquid.ai/)
|
||||
- [vLLM supported models](https://docs.vllm.ai/en/latest/models/supported_models.html)
|
||||
@@ -0,0 +1,137 @@
|
||||
meta:
|
||||
title: "LFM2.5 1.2B JP"
|
||||
slug: "lfm2.5-1.2b-jp"
|
||||
provider: "LiquidAI"
|
||||
description: "Liquid AI's 1.2B Japanese-specialized chat model on the LFM2 hybrid conv+attention backbone, with a 32K context window on a single small GPU."
|
||||
date_added: 2026-06-25
|
||||
date_updated: 2026-06-24
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "Japanese-tuned 1.2B hybrid chat model — single small GPU"
|
||||
related_recipes:
|
||||
- LiquidAI/LFM2.5-1.2B-JP-202606
|
||||
- LiquidAI/LFM2.5-1.2B-Instruct
|
||||
hardware:
|
||||
h100: verified
|
||||
h200: verified
|
||||
b200: verified
|
||||
|
||||
model:
|
||||
model_id: "LiquidAI/LFM2.5-1.2B-JP"
|
||||
min_vllm_version: "0.23.0"
|
||||
architecture: dense
|
||||
parameter_count: "1.2B"
|
||||
active_parameters: "1.2B"
|
||||
context_length: 32768
|
||||
base_args: []
|
||||
base_env: {}
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 3
|
||||
description: "Full BF16 — single 8 GB+ NVIDIA GPU with room for KV cache"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
|
||||
hardware_overrides:
|
||||
# Blackwell (B200) only: route RMSNorm to the bundled oink kernels (measured ~+2.6% on B200,
|
||||
# output identical). Inert (falls back to native) on non-Blackwell GPUs.
|
||||
blackwell:
|
||||
extra_env:
|
||||
VLLM_USE_OINK_OPS: "1"
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
tp: 1
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[LFM2.5-1.2B-JP](https://huggingface.co/LiquidAI/LFM2.5-1.2B-JP) is the Japanese-specialized
|
||||
1.2B chat model from [Liquid AI](https://www.liquid.ai/), built on the **LFM2 hybrid backbone**
|
||||
(short-range gated convolution blocks interleaved with grouped-query attention). It serves on a
|
||||
single small GPU through vLLM's OpenAI-compatible API. For the latest Japanese checkpoint with
|
||||
tool calling, see
|
||||
[LFM2.5-1.2B-JP-202606](https://huggingface.co/LiquidAI/LFM2.5-1.2B-JP-202606).
|
||||
|
||||
### Key Features
|
||||
- **Japanese-specialized**: Tuned for Japanese chat and instruction following.
|
||||
- **Hybrid backbone**: Gated short convolutions interleaved with grouped-query attention — a smaller KV cache and lower decode latency than a same-size full-attention transformer.
|
||||
- **32K context**: 32,768-token context window.
|
||||
- **Native vLLM support**: Served via the `Lfm2ForCausalLM` architecture.
|
||||
|
||||
### Supported Variants
|
||||
|
||||
Dense:
|
||||
- `LiquidAI/LFM2.5-230M` (230M)
|
||||
- `LiquidAI/LFM2.5-350M` (350M)
|
||||
- `LiquidAI/LFM2.5-1.2B-Instruct` (1.2B)
|
||||
- `LiquidAI/LFM2.5-1.2B-Thinking` (1.2B, reasoning)
|
||||
- `LiquidAI/LFM2.5-1.2B-JP` / `LiquidAI/LFM2.5-1.2B-JP-202606` (Japanese)
|
||||
- `LiquidAI/LFM2.5-1.2B-Base` (pretrained base)
|
||||
|
||||
MoE:
|
||||
- `LiquidAI/LFM2.5-8B-A1B` (8B total / ~1B active)
|
||||
|
||||
Vision-Language:
|
||||
- `LiquidAI/LFM2.5-VL-450M`, `LiquidAI/LFM2.5-VL-1.6B`
|
||||
|
||||
See the [LFM2.5 usage guide](../../LiquidAI/LFM2.5) for the full family.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Hardware**: 1× GPU with ≥8 GB VRAM.
|
||||
- **vLLM**: ≥ 0.23.0.
|
||||
|
||||
### pip (NVIDIA CUDA)
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend auto
|
||||
```
|
||||
|
||||
## Deployment Configurations
|
||||
|
||||
### Quick Start (Single GPU, BF16)
|
||||
```bash
|
||||
vllm serve LiquidAI/LFM2.5-1.2B-JP
|
||||
```
|
||||
|
||||
### Docker (NVIDIA)
|
||||
```bash
|
||||
docker run -itd --name lfm2.5-jp \
|
||||
--ipc=host --network host --shm-size 16G --gpus all \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai:latest \
|
||||
--model LiquidAI/LFM2.5-1.2B-JP \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
### Text Generation
|
||||
The card recommends temperature `0.3`, `min_p 0.15`, `repetition_penalty 1.05` for the JP
|
||||
checkpoint (`min_p` and `repetition_penalty` ride in `extra_body`).
|
||||
```python
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||||
response = client.chat.completions.create(
|
||||
model="LiquidAI/LFM2.5-1.2B-JP",
|
||||
messages=[{"role": "user", "content": "日本の四季について簡単に説明してください。"}],
|
||||
temperature=0.3,
|
||||
extra_body={"min_p": 0.15, "repetition_penalty": 1.05},
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
## Configuration Tips
|
||||
- Set `--max-model-len` to match your workload (up to 32K); lowering it frees VRAM for KV cache.
|
||||
- `--gpu-memory-utilization 0.90–0.95` maximizes KV cache capacity.
|
||||
- Sampling presets are per-request client defaults — don't bake them into `vllm serve`.
|
||||
|
||||
## References
|
||||
- [Model card](https://huggingface.co/LiquidAI/LFM2.5-1.2B-JP)
|
||||
- [LFM2.5 usage guide](../../LiquidAI/LFM2.5)
|
||||
- [Liquid AI](https://www.liquid.ai/)
|
||||
- [vLLM supported models](https://docs.vllm.ai/en/latest/models/supported_models.html)
|
||||
@@ -0,0 +1,180 @@
|
||||
meta:
|
||||
title: "LFM2.5 1.2B Thinking"
|
||||
slug: "lfm2.5-1.2b-thinking"
|
||||
provider: "LiquidAI"
|
||||
description: "Liquid AI's 1.2B reasoning model on the LFM2 hybrid conv+attention backbone — <think> chain-of-thought, tool calling, and 32K context on a single small GPU."
|
||||
date_added: 2026-06-25
|
||||
date_updated: 2026-06-24
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "1.2B hybrid reasoning model with <think> chain-of-thought and tool calling — single small GPU"
|
||||
related_recipes:
|
||||
- LiquidAI/LFM2.5-1.2B-Instruct
|
||||
- LiquidAI/LFM2.5-8B-A1B
|
||||
hardware:
|
||||
h100: verified
|
||||
h200: verified
|
||||
b200: verified
|
||||
|
||||
model:
|
||||
model_id: "LiquidAI/LFM2.5-1.2B-Thinking"
|
||||
min_vllm_version: "0.23.0"
|
||||
architecture: dense
|
||||
parameter_count: "1.2B"
|
||||
active_parameters: "1.2B"
|
||||
context_length: 32768
|
||||
base_args: []
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
reasoning:
|
||||
description: "Split the model's <think>…</think> chain-of-thought into a separate reasoning field via vLLM's qwen3 parser"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "qwen3"
|
||||
tool_calling:
|
||||
description: "Surface LFM2.5's Pythonic tool calls (wrapped in <|tool_call_start|>…<|tool_call_end|>) as OpenAI tool_calls via vLLM's native lfm2 parser"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "lfm2"
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 3
|
||||
description: "Full BF16 — single 8 GB+ NVIDIA GPU with room for KV cache"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
|
||||
hardware_overrides:
|
||||
# Blackwell (B200) only: route RMSNorm to the bundled oink kernels (measured ~+2.6% on B200,
|
||||
# output identical). Inert (falls back to native) on non-Blackwell GPUs.
|
||||
blackwell:
|
||||
extra_env:
|
||||
VLLM_USE_OINK_OPS: "1"
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
tp: 1
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[LFM2.5-1.2B-Thinking](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Thinking) is the reasoning
|
||||
variant of [Liquid AI's](https://www.liquid.ai/) 1.2B LFM2.5 model, built on the **LFM2 hybrid
|
||||
backbone** (short-range gated convolution blocks interleaved with grouped-query attention). On
|
||||
reasoning-heavy tasks it produces an explicit `<think>…</think>` chain-of-thought before its
|
||||
final answer, so it reasons well above its weight class while still serving on a single small
|
||||
GPU through vLLM's OpenAI-compatible API.
|
||||
|
||||
### Key Features
|
||||
- **Hybrid backbone**: Gated short convolutions interleaved with grouped-query attention — a smaller KV cache and lower decode latency than a same-size full-attention transformer.
|
||||
- **`<think>` reasoning**: Emits an explicit `<think>…</think>` chain-of-thought on non-trivial problems; vLLM's `qwen3` parser splits it into a separate `reasoning` field.
|
||||
- **32K context**: 32,768-token context window.
|
||||
- **Tool calling**: Pythonic tool calls surfaced as OpenAI `tool_calls` by vLLM's native `lfm2` parser.
|
||||
- **Native vLLM support**: Served via the `Lfm2ForCausalLM` architecture.
|
||||
|
||||
### Supported Variants
|
||||
|
||||
Dense:
|
||||
- `LiquidAI/LFM2.5-230M` (230M)
|
||||
- `LiquidAI/LFM2.5-350M` (350M)
|
||||
- `LiquidAI/LFM2.5-1.2B-Instruct` (1.2B)
|
||||
- `LiquidAI/LFM2.5-1.2B-Thinking` (1.2B, reasoning)
|
||||
- `LiquidAI/LFM2.5-1.2B-JP` / `LiquidAI/LFM2.5-1.2B-JP-202606` (Japanese)
|
||||
- `LiquidAI/LFM2.5-1.2B-Base` (pretrained base)
|
||||
|
||||
MoE:
|
||||
- `LiquidAI/LFM2.5-8B-A1B` (8B total / ~1B active, also a reasoning model)
|
||||
|
||||
Vision-Language:
|
||||
- `LiquidAI/LFM2.5-VL-450M`, `LiquidAI/LFM2.5-VL-1.6B`
|
||||
|
||||
See the [LFM2.5 usage guide](../../LiquidAI/LFM2.5) for the full family.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Hardware**: 1× GPU with ≥8 GB VRAM.
|
||||
- **vLLM**: ≥ 0.23.0.
|
||||
|
||||
### pip (NVIDIA CUDA)
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend auto
|
||||
```
|
||||
|
||||
## Deployment Configurations
|
||||
|
||||
### Quick Start (Single GPU, BF16)
|
||||
Enable the reasoning parser so the chain-of-thought is returned in `reasoning`:
|
||||
```bash
|
||||
vllm serve LiquidAI/LFM2.5-1.2B-Thinking \
|
||||
--reasoning-parser qwen3
|
||||
```
|
||||
|
||||
### Full-Featured Server Launch
|
||||
Reasoning **and** tool calling:
|
||||
```bash
|
||||
vllm serve LiquidAI/LFM2.5-1.2B-Thinking \
|
||||
--reasoning-parser qwen3 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser lfm2 \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
### Docker (NVIDIA)
|
||||
```bash
|
||||
docker run -itd --name lfm2.5-thinking \
|
||||
--ipc=host --network host --shm-size 16G --gpus all \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai:latest \
|
||||
--model LiquidAI/LFM2.5-1.2B-Thinking \
|
||||
--reasoning-parser qwen3 \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
### Reasoning Mode
|
||||
The card recommends a low temperature for the reasoning model — temperature `0.05`, `top_k 50`,
|
||||
`repetition_penalty 1.05`. Do **not** set `max_tokens` — capping it can truncate the
|
||||
chain-of-thought mid-stream.
|
||||
```python
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||||
response = client.chat.completions.create(
|
||||
model="LiquidAI/LFM2.5-1.2B-Thinking",
|
||||
messages=[{"role": "user", "content": "A snail climbs 3 ft up a 20 ft well each day and slides 2 ft back each night. How many days to reach the top?"}],
|
||||
temperature=0.05,
|
||||
extra_body={"top_k": 50, "repetition_penalty": 1.05},
|
||||
)
|
||||
msg = response.choices[0].message
|
||||
print("reasoning:", msg.reasoning)
|
||||
print("answer:", msg.content)
|
||||
```
|
||||
|
||||
> **Note**: the model opens the `<think>` channel for non-trivial problems; a simple prompt may
|
||||
> be answered directly, in which case `reasoning` is empty. That is expected behavior —
|
||||
> the `qwen3` parser extracts the block whenever it is present.
|
||||
|
||||
### Tool Calling
|
||||
Add `--enable-auto-tool-choice --tool-call-parser lfm2` at launch, then pass `tools=[…]`; the
|
||||
model can reason about which tool to call before emitting it.
|
||||
|
||||
### Structured Outputs
|
||||
vLLM's guided decoding constrains output to a JSON schema via `response_format`. The model does
|
||||
not see the schema itself — put semantic instructions in the system prompt.
|
||||
|
||||
## Configuration Tips
|
||||
- Give reasoning room: leave `max_tokens` unset so the `<think>` block and answer aren't truncated.
|
||||
- Set `--max-model-len` to match your workload (up to 32K).
|
||||
- `--gpu-memory-utilization 0.90–0.95` maximizes KV cache capacity.
|
||||
- Sampling presets are per-request client defaults — don't bake them into `vllm serve`.
|
||||
|
||||
## References
|
||||
- [Model card](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Thinking)
|
||||
- [LFM2.5 usage guide](../../LiquidAI/LFM2.5)
|
||||
- [Liquid AI](https://www.liquid.ai/)
|
||||
- [vLLM supported models](https://docs.vllm.ai/en/latest/models/supported_models.html)
|
||||
@@ -0,0 +1,184 @@
|
||||
meta:
|
||||
title: "LFM2.5 230M"
|
||||
slug: "lfm2.5-230m"
|
||||
provider: "LiquidAI"
|
||||
description: "Liquid AI's most compact LFM2.5 chat model (230M) on the LFM2 hybrid conv+attention backbone — distilled from LFM2.5-350M, tool calling and 32K context, built for edge / on-device serving."
|
||||
date_added: 2026-06-25
|
||||
date_updated: 2026-06-25
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "230M hybrid instruct model with tool calling — runs from low-cost CPUs to production GPUs, ideal for tightest on-device budgets"
|
||||
related_recipes:
|
||||
- LiquidAI/LFM2.5-350M
|
||||
- LiquidAI/LFM2.5-1.2B-Instruct
|
||||
- LiquidAI/LFM2.5-8B-A1B
|
||||
hardware:
|
||||
h100: verified
|
||||
h200: verified
|
||||
b200: verified
|
||||
|
||||
model:
|
||||
model_id: "LiquidAI/LFM2.5-230M"
|
||||
min_vllm_version: "0.23.0"
|
||||
architecture: dense
|
||||
parameter_count: "230M"
|
||||
active_parameters: "230M"
|
||||
context_length: 32768
|
||||
base_args: []
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Surface LFM2.5's Pythonic tool calls (wrapped in <|tool_call_start|>…<|tool_call_end|>) as OpenAI tool_calls via vLLM's native lfm2 parser"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "lfm2"
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 1
|
||||
description: "Full BF16 — fits essentially any GPU (~0.5 GB of weights)"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
|
||||
hardware_overrides: {}
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
tp: 1
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[LFM2.5-230M](https://huggingface.co/LiquidAI/LFM2.5-230M) is the most compact chat model in
|
||||
[Liquid AI's](https://www.liquid.ai/) LFM2.5 family, built on the **LFM2 hybrid backbone**:
|
||||
short-range gated convolution blocks interleaved with grouped-query attention. It was
|
||||
**distilled from LFM2.5-350M** and refined with multi-stage reinforcement learning, then
|
||||
trained on a 19T-token budget — keeping memory and latency low enough for the tightest edge and
|
||||
on-device budgets while still supporting tool calling and a 32K context window through vLLM's
|
||||
OpenAI-compatible API.
|
||||
|
||||
### Key Features
|
||||
- **Hybrid backbone**: 14 layers — 8 double-gated short-convolution blocks interleaved with 6 grouped-query attention blocks — for a smaller KV cache and lower decode latency than a same-size full-attention transformer.
|
||||
- **32K context**: Long-context support (`context_length = 32768`).
|
||||
- **Tool calling**: Pythonic tool calls (`<|tool_call_start|>…<|tool_call_end|>`) surfaced as OpenAI `tool_calls` by vLLM's native `lfm2` parser.
|
||||
- **Native vLLM support**: Served via the `Lfm2ForCausalLM` architecture — no `--trust-remote-code` required.
|
||||
- **Edge-ready**: ~0.5 GB of BF16 weights — runs from low-cost CPUs to production GPUs (213 tok/s decode on a Galaxy S25 Ultra, 42 tok/s on a Raspberry Pi 5).
|
||||
|
||||
Best suited for data extraction and lightweight on-device agentic pipelines; the model card
|
||||
notes it is **not** recommended for reasoning-heavy workloads (advanced math, code generation,
|
||||
creative writing) — reach for a larger sibling there.
|
||||
|
||||
### Supported Variants
|
||||
|
||||
Dense:
|
||||
- `LiquidAI/LFM2.5-230M` (230M)
|
||||
- `LiquidAI/LFM2.5-350M` (350M)
|
||||
- `LiquidAI/LFM2.5-1.2B-Instruct` (1.2B)
|
||||
- `LiquidAI/LFM2.5-1.2B-Thinking` (1.2B, reasoning)
|
||||
- `LiquidAI/LFM2.5-1.2B-JP` / `LiquidAI/LFM2.5-1.2B-JP-202606` (Japanese)
|
||||
- `LiquidAI/LFM2.5-1.2B-Base` (pretrained base)
|
||||
|
||||
MoE:
|
||||
- `LiquidAI/LFM2.5-8B-A1B` (8B total / ~1B active)
|
||||
|
||||
Vision-Language:
|
||||
- `LiquidAI/LFM2.5-VL-450M`, `LiquidAI/LFM2.5-VL-1.6B`
|
||||
|
||||
See the [LFM2.5 usage guide](../../LiquidAI/LFM2.5) for the full family.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Hardware**: 1× GPU (~1 GB VRAM for weights; any modern GPU works). Verified on H100.
|
||||
- **vLLM**: ≥ 0.23.0 — the LFM2 architecture ships in the 0.23.0 stable release.
|
||||
|
||||
### pip (NVIDIA CUDA)
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend auto
|
||||
```
|
||||
|
||||
## Deployment Configurations
|
||||
|
||||
### Quick Start (Single GPU, BF16)
|
||||
```bash
|
||||
vllm serve LiquidAI/LFM2.5-230M
|
||||
```
|
||||
|
||||
### Full-Featured Server Launch
|
||||
Enables tool calling:
|
||||
```bash
|
||||
vllm serve LiquidAI/LFM2.5-230M \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser lfm2 \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
### Docker (NVIDIA)
|
||||
```bash
|
||||
docker run -itd --name lfm2.5-230m \
|
||||
--ipc=host --network host --shm-size 16G --gpus all \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai:latest \
|
||||
--model LiquidAI/LFM2.5-230M \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
### Text Generation
|
||||
The model card recommends temperature `0.1`, `top_k 50`, `repetition_penalty 1.05` (`top_k` and
|
||||
`repetition_penalty` ride in `extra_body`).
|
||||
```python
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||||
response = client.chat.completions.create(
|
||||
model="LiquidAI/LFM2.5-230M",
|
||||
messages=[{"role": "user", "content": "Give me three uses for baking soda."}],
|
||||
temperature=0.1,
|
||||
extra_body={"top_k": 50, "repetition_penalty": 1.05},
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Tool Calling
|
||||
Launch with `--enable-auto-tool-choice --tool-call-parser lfm2`, then pass `tools=[…]`; the
|
||||
`lfm2` parser converts the model's Pythonic call into a standard `tool_calls` array.
|
||||
```python
|
||||
response = client.chat.completions.create(
|
||||
model="LiquidAI/LFM2.5-230M",
|
||||
messages=[{"role": "user", "content": "What's the weather in Paris?"}],
|
||||
tools=[{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get current weather for a location",
|
||||
"parameters": {"type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"]},
|
||||
},
|
||||
}],
|
||||
temperature=0.1,
|
||||
extra_body={"top_k": 50, "repetition_penalty": 1.05},
|
||||
)
|
||||
print(response.choices[0].message.tool_calls)
|
||||
```
|
||||
|
||||
### Structured Outputs
|
||||
vLLM's guided decoding constrains output to a JSON schema via `response_format`. The model does
|
||||
not see the schema itself — put semantic instructions in the system prompt. Data extraction is a
|
||||
primary use case for this model, so structured outputs pair naturally with it.
|
||||
|
||||
## Configuration Tips
|
||||
- At 230M the model fits any GPU; raise `--max-num-seqs` to push batch throughput.
|
||||
- Set `--max-model-len` to match your workload (up to 32K).
|
||||
- `--gpu-memory-utilization 0.90–0.95` maximizes KV cache capacity.
|
||||
- Sampling presets are per-request client defaults — don't bake them into `vllm serve`.
|
||||
|
||||
## References
|
||||
- [Model card](https://huggingface.co/LiquidAI/LFM2.5-230M)
|
||||
- [Blog post](https://www.liquid.ai/blog/lfm2-5-230m)
|
||||
- [LFM2.5 usage guide](../../LiquidAI/LFM2.5)
|
||||
- [Liquid AI](https://www.liquid.ai/)
|
||||
- [vLLM supported models](https://docs.vllm.ai/en/latest/models/supported_models.html)
|
||||
@@ -0,0 +1,181 @@
|
||||
meta:
|
||||
title: "LFM2.5 350M"
|
||||
slug: "lfm2.5-350m"
|
||||
provider: "LiquidAI"
|
||||
description: "Liquid AI's smallest LFM2.5 chat model (350M) on the LFM2 hybrid conv+attention backbone — tool calling and 32K context, light enough for edge GPUs."
|
||||
date_added: 2026-06-25
|
||||
date_updated: 2026-06-24
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "350M hybrid instruct model with tool calling — fits any GPU, ideal for edge / on-device serving"
|
||||
related_recipes:
|
||||
- LiquidAI/LFM2.5-1.2B-Instruct
|
||||
- LiquidAI/LFM2.5-8B-A1B
|
||||
hardware:
|
||||
h100: verified
|
||||
h200: verified
|
||||
b200: verified
|
||||
|
||||
model:
|
||||
model_id: "LiquidAI/LFM2.5-350M"
|
||||
min_vllm_version: "0.23.0"
|
||||
architecture: dense
|
||||
parameter_count: "350M"
|
||||
active_parameters: "350M"
|
||||
context_length: 32768
|
||||
base_args: []
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Surface LFM2.5's Pythonic tool calls (wrapped in <|tool_call_start|>…<|tool_call_end|>) as OpenAI tool_calls via vLLM's native lfm2 parser"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "lfm2"
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 1
|
||||
description: "Full BF16 — fits essentially any GPU (~0.7 GB of weights)"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
|
||||
hardware_overrides:
|
||||
# Blackwell (B200) only: route RMSNorm to the bundled oink kernels (measured ~+2.6% on B200,
|
||||
# output identical). Inert (falls back to native) on non-Blackwell GPUs.
|
||||
blackwell:
|
||||
extra_env:
|
||||
VLLM_USE_OINK_OPS: "1"
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
tp: 1
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[LFM2.5-350M](https://huggingface.co/LiquidAI/LFM2.5-350M) is the smallest chat model in
|
||||
[Liquid AI's](https://www.liquid.ai/) LFM2.5 family, built on the **LFM2 hybrid backbone**:
|
||||
short-range gated convolution blocks interleaved with grouped-query attention. It shares that
|
||||
backbone with its larger siblings, which keeps memory and latency low enough for edge and
|
||||
on-device deployment while still supporting tool calling and a 32K context window — all
|
||||
through vLLM's OpenAI-compatible API.
|
||||
|
||||
### Key Features
|
||||
- **Hybrid backbone**: Gated short convolutions interleaved with grouped-query attention — a smaller KV cache and lower decode latency than a same-size full-attention transformer.
|
||||
- **32K context**: 32,768-token context window.
|
||||
- **Tool calling**: Pythonic tool calls (`<|tool_call_start|>…<|tool_call_end|>`) surfaced as OpenAI `tool_calls` by vLLM's native `lfm2` parser.
|
||||
- **Native vLLM support**: Served via the `Lfm2ForCausalLM` architecture.
|
||||
- **Edge-ready**: ~0.7 GB of BF16 weights — runs on commodity and on-device GPUs.
|
||||
|
||||
### Supported Variants
|
||||
|
||||
Dense:
|
||||
- `LiquidAI/LFM2.5-230M` (230M)
|
||||
- `LiquidAI/LFM2.5-350M` (350M)
|
||||
- `LiquidAI/LFM2.5-1.2B-Instruct` (1.2B)
|
||||
- `LiquidAI/LFM2.5-1.2B-Thinking` (1.2B, reasoning)
|
||||
- `LiquidAI/LFM2.5-1.2B-JP` / `LiquidAI/LFM2.5-1.2B-JP-202606` (Japanese)
|
||||
- `LiquidAI/LFM2.5-1.2B-Base` (pretrained base)
|
||||
|
||||
MoE:
|
||||
- `LiquidAI/LFM2.5-8B-A1B` (8B total / ~1B active)
|
||||
|
||||
Vision-Language:
|
||||
- `LiquidAI/LFM2.5-VL-450M`, `LiquidAI/LFM2.5-VL-1.6B`
|
||||
|
||||
See the [LFM2.5 usage guide](../../LiquidAI/LFM2.5) for the full family.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Hardware**: 1× GPU (~1 GB VRAM for weights; any modern GPU works).
|
||||
- **vLLM**: ≥ 0.23.0.
|
||||
|
||||
### pip (NVIDIA CUDA)
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend auto
|
||||
```
|
||||
|
||||
## Deployment Configurations
|
||||
|
||||
### Quick Start (Single GPU, BF16)
|
||||
```bash
|
||||
vllm serve LiquidAI/LFM2.5-350M
|
||||
```
|
||||
|
||||
### Full-Featured Server Launch
|
||||
Enables tool calling:
|
||||
```bash
|
||||
vllm serve LiquidAI/LFM2.5-350M \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser lfm2 \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
### Docker (NVIDIA)
|
||||
```bash
|
||||
docker run -itd --name lfm2.5-350m \
|
||||
--ipc=host --network host --shm-size 16G --gpus all \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai:latest \
|
||||
--model LiquidAI/LFM2.5-350M \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
### Text Generation
|
||||
The model card recommends temperature `0.1`, `top_k 50`, `repetition_penalty 1.05` (`top_k` and
|
||||
`repetition_penalty` ride in `extra_body`).
|
||||
```python
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||||
response = client.chat.completions.create(
|
||||
model="LiquidAI/LFM2.5-350M",
|
||||
messages=[{"role": "user", "content": "Give me three uses for baking soda."}],
|
||||
temperature=0.1,
|
||||
extra_body={"top_k": 50, "repetition_penalty": 1.05},
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Tool Calling
|
||||
Launch with `--enable-auto-tool-choice --tool-call-parser lfm2`, then pass `tools=[…]`; the
|
||||
`lfm2` parser converts the model's Pythonic call into a standard `tool_calls` array.
|
||||
```python
|
||||
response = client.chat.completions.create(
|
||||
model="LiquidAI/LFM2.5-350M",
|
||||
messages=[{"role": "user", "content": "What's the weather in Paris?"}],
|
||||
tools=[{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get current weather for a location",
|
||||
"parameters": {"type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"]},
|
||||
},
|
||||
}],
|
||||
temperature=0.1,
|
||||
extra_body={"top_k": 50, "repetition_penalty": 1.05},
|
||||
)
|
||||
print(response.choices[0].message.tool_calls)
|
||||
```
|
||||
|
||||
### Structured Outputs
|
||||
vLLM's guided decoding constrains output to a JSON schema via `response_format`. The model does
|
||||
not see the schema itself — put semantic instructions in the system prompt.
|
||||
|
||||
## Configuration Tips
|
||||
- At 350M the model fits any GPU; raise `--max-num-seqs` to push batch throughput.
|
||||
- Set `--max-model-len` to match your workload (up to 32K).
|
||||
- `--gpu-memory-utilization 0.90–0.95` maximizes KV cache capacity.
|
||||
- Sampling presets are per-request client defaults — don't bake them into `vllm serve`.
|
||||
|
||||
## References
|
||||
- [Model card](https://huggingface.co/LiquidAI/LFM2.5-350M)
|
||||
- [LFM2.5 usage guide](../../LiquidAI/LFM2.5)
|
||||
- [Liquid AI](https://www.liquid.ai/)
|
||||
- [vLLM supported models](https://docs.vllm.ai/en/latest/models/supported_models.html)
|
||||
@@ -0,0 +1,189 @@
|
||||
meta:
|
||||
title: "LFM2.5 8B A1B"
|
||||
slug: "lfm2.5-8b-a1b"
|
||||
provider: "LiquidAI"
|
||||
description: "Liquid AI's 8B mixture-of-experts model (~1B active) on the LFM2 hybrid conv+attention backbone, with <think> reasoning and tool calling at ~1B decode cost."
|
||||
date_added: 2026-06-25
|
||||
date_updated: 2026-06-24
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "8B-total / ~1B-active hybrid MoE with reasoning and tool calling — 8B quality at ~1B decode cost on a single GPU"
|
||||
related_recipes:
|
||||
- LiquidAI/LFM2.5-1.2B-Instruct
|
||||
- LiquidAI/LFM2.5-1.2B-Thinking
|
||||
hardware:
|
||||
h100: verified
|
||||
h200: verified
|
||||
b200: verified
|
||||
|
||||
model:
|
||||
model_id: "LiquidAI/LFM2.5-8B-A1B"
|
||||
min_vllm_version: "0.23.0"
|
||||
architecture: moe
|
||||
parameter_count: "8B"
|
||||
active_parameters: "1B"
|
||||
context_length: 128000
|
||||
base_args: []
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
reasoning:
|
||||
description: "Split the model's <think>…</think> chain-of-thought into a separate reasoning field via vLLM's qwen3 parser"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "qwen3"
|
||||
tool_calling:
|
||||
description: "Surface LFM2.5's Pythonic tool calls (wrapped in <|tool_call_start|>…<|tool_call_end|>) as OpenAI tool_calls via vLLM's native lfm2 parser"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "lfm2"
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 21
|
||||
description: "Full BF16 — all 8B of experts live in VRAM; single 24 GB+ NVIDIA GPU"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
|
||||
hardware_overrides:
|
||||
# Blackwell (B200) only: route RMSNorm to the bundled oink kernels (measured ~+2.6% on B200,
|
||||
# output identical). Inert (falls back to native) on non-Blackwell GPUs.
|
||||
blackwell:
|
||||
extra_env:
|
||||
VLLM_USE_OINK_OPS: "1"
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
tp: 1
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[LFM2.5-8B-A1B](https://huggingface.co/LiquidAI/LFM2.5-8B-A1B) is [Liquid AI's](https://www.liquid.ai/)
|
||||
mixture-of-experts model: ~8B total parameters with only ~1B activated per token. It pairs the
|
||||
**LFM2 hybrid backbone** (short-range gated convolution blocks interleaved with grouped-query
|
||||
attention) with sparse MoE feed-forward layers, so it reaches 8B-class quality at roughly 1B
|
||||
decode cost. It supports explicit `<think>` reasoning and Pythonic tool calling through vLLM's
|
||||
OpenAI-compatible API.
|
||||
|
||||
### Key Features
|
||||
- **Hybrid + MoE**: Gated short convolutions + grouped-query attention with sparse expert FFNs — ~1B active parameters per token out of ~8B total.
|
||||
- **`<think>` reasoning**: Emits an explicit `<think>…</think>` chain-of-thought on non-trivial problems; vLLM's `qwen3` parser splits it into `reasoning`.
|
||||
- **Tool calling**: Pythonic tool calls surfaced as OpenAI `tool_calls` by vLLM's native `lfm2` parser.
|
||||
- **128K context**: Long-context support (`max_position_embeddings = 128000`).
|
||||
- **Native vLLM support**: Served via the `Lfm2MoeForCausalLM` architecture.
|
||||
|
||||
### Supported Variants
|
||||
|
||||
Dense:
|
||||
- `LiquidAI/LFM2.5-230M` (230M)
|
||||
- `LiquidAI/LFM2.5-350M` (350M)
|
||||
- `LiquidAI/LFM2.5-1.2B-Instruct` (1.2B)
|
||||
- `LiquidAI/LFM2.5-1.2B-Thinking` (1.2B, reasoning)
|
||||
- `LiquidAI/LFM2.5-1.2B-JP` / `LiquidAI/LFM2.5-1.2B-JP-202606` (Japanese)
|
||||
- `LiquidAI/LFM2.5-1.2B-Base` (pretrained base)
|
||||
|
||||
MoE:
|
||||
- `LiquidAI/LFM2.5-8B-A1B` (8B total / ~1B active)
|
||||
|
||||
Vision-Language:
|
||||
- `LiquidAI/LFM2.5-VL-450M`, `LiquidAI/LFM2.5-VL-1.6B`
|
||||
|
||||
See the [LFM2.5 usage guide](../../LiquidAI/LFM2.5) for the full family.
|
||||
|
||||
> **Sizing**: MoE keeps every expert resident in VRAM, so size the GPU for the full ~8B of
|
||||
> weights (≈17 GB BF16 + KV cache) even though only ~1B is active per token.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Hardware**: 1× GPU with ≥24 GB VRAM.
|
||||
- **vLLM**: ≥ 0.23.0.
|
||||
|
||||
### pip (NVIDIA CUDA)
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend auto
|
||||
```
|
||||
|
||||
## Deployment Configurations
|
||||
|
||||
### Quick Start (Single GPU, BF16)
|
||||
```bash
|
||||
vllm serve LiquidAI/LFM2.5-8B-A1B
|
||||
```
|
||||
|
||||
### Full-Featured Server Launch
|
||||
Reasoning **and** tool calling:
|
||||
```bash
|
||||
vllm serve LiquidAI/LFM2.5-8B-A1B \
|
||||
--reasoning-parser qwen3 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser lfm2 \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
### Multi-GPU
|
||||
This model fits on a single 24 GB+ GPU, so tensor parallelism gives no meaningful speedup on
|
||||
basically any GPU — there's nothing to split that helps. You can still try it on a multi-GPU
|
||||
node if you want, but expect no throughput gain:
|
||||
```bash
|
||||
vllm serve LiquidAI/LFM2.5-8B-A1B --tensor-parallel-size 2
|
||||
```
|
||||
|
||||
### Docker (NVIDIA)
|
||||
```bash
|
||||
docker run -itd --name lfm2.5-8b-a1b \
|
||||
--ipc=host --network host --shm-size 16G --gpus all \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai:latest \
|
||||
--model LiquidAI/LFM2.5-8B-A1B \
|
||||
--reasoning-parser qwen3 \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
### Reasoning Mode
|
||||
The model card recommends temperature `0.2`, `top_k 80`, `repetition_penalty 1.05`. Leave
|
||||
`max_tokens` unset so the `<think>` block and answer aren't truncated mid-stream.
|
||||
```python
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||||
response = client.chat.completions.create(
|
||||
model="LiquidAI/LFM2.5-8B-A1B",
|
||||
messages=[{"role": "user", "content": "A snail climbs 3 ft up a 20 ft well each day and slides 2 ft back each night. How many days to reach the top?"}],
|
||||
temperature=0.2,
|
||||
extra_body={"top_k": 80, "repetition_penalty": 1.05},
|
||||
)
|
||||
msg = response.choices[0].message
|
||||
print("reasoning:", msg.reasoning)
|
||||
print("answer:", msg.content)
|
||||
```
|
||||
|
||||
> **Note**: the model opens the `<think>` channel for non-trivial problems; a simple prompt may
|
||||
> be answered directly, in which case `reasoning` is empty. That is expected behavior —
|
||||
> the `qwen3` parser extracts the block whenever it is present.
|
||||
|
||||
### Tool Calling
|
||||
Add `--enable-auto-tool-choice --tool-call-parser lfm2` at launch, then pass `tools=[…]`; the
|
||||
`lfm2` parser converts the model's Pythonic call into a standard `tool_calls` array.
|
||||
|
||||
### Structured Outputs
|
||||
vLLM's guided decoding constrains output to a JSON schema via `response_format`. The model does
|
||||
not see the schema itself — put semantic instructions in the system prompt.
|
||||
|
||||
## Configuration Tips
|
||||
- Size the GPU for the full ~8B of weights — all experts are resident even though ~1B is active.
|
||||
- It fits on one GPU; tensor parallelism gives no real speedup on basically any GPU (you can still try `--tensor-parallel-size N`).
|
||||
- Set `--max-model-len` to match your workload (up to 128K).
|
||||
- `--gpu-memory-utilization 0.90–0.95` maximizes KV cache capacity.
|
||||
- Sampling presets are per-request client defaults — don't bake them into `vllm serve`.
|
||||
|
||||
## References
|
||||
- [Model card](https://huggingface.co/LiquidAI/LFM2.5-8B-A1B)
|
||||
- [LFM2.5 usage guide](../../LiquidAI/LFM2.5)
|
||||
- [Liquid AI](https://www.liquid.ai/)
|
||||
- [vLLM supported models](https://docs.vllm.ai/en/latest/models/supported_models.html)
|
||||
@@ -0,0 +1,155 @@
|
||||
meta:
|
||||
title: "LFM2.5 VL 1.6B"
|
||||
slug: "lfm2.5-vl-1.6b"
|
||||
provider: "LiquidAI"
|
||||
description: "Liquid AI's 1.6B vision-language model — LFM2 hybrid LM backbone plus a SigLIP2 vision tower for image+text chat on a single small GPU."
|
||||
date_added: 2026-06-25
|
||||
date_updated: 2026-06-24
|
||||
tasks:
|
||||
- multimodal
|
||||
performance_headline: "1.6B vision-language model (hybrid LM + SigLIP2 vision) — image understanding on a single small GPU"
|
||||
related_recipes:
|
||||
- LiquidAI/LFM2.5-VL-450M
|
||||
- LiquidAI/LFM2.5-1.2B-Instruct
|
||||
hardware:
|
||||
h100: verified
|
||||
h200: verified
|
||||
b200: verified
|
||||
|
||||
model:
|
||||
model_id: "LiquidAI/LFM2.5-VL-1.6B"
|
||||
min_vllm_version: "0.23.0"
|
||||
architecture: dense
|
||||
parameter_count: "1.6B"
|
||||
active_parameters: "1.6B"
|
||||
context_length: 32768
|
||||
base_args: []
|
||||
base_env: {}
|
||||
# TODO(vincentzed): the Lfm2VL encoder CUDA graph (vllm-project/vllm#44930, ~10-20% lower e2e
|
||||
# latency at low batch) is NOT in 0.23.0 — add the encoder-CG compilation flag here once it
|
||||
# ships in a stable release (>0.23.0).
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 4
|
||||
description: "Full BF16 (LM + vision encoder) — single 8 GB+ NVIDIA GPU"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
|
||||
hardware_overrides:
|
||||
# Blackwell (B200) only: route RMSNorm to the bundled oink kernels (measured ~+2.6% on B200,
|
||||
# output identical). Inert (falls back to native) on non-Blackwell GPUs.
|
||||
blackwell:
|
||||
extra_env:
|
||||
VLLM_USE_OINK_OPS: "1"
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
tp: 1
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[LFM2.5-VL-1.6B](https://huggingface.co/LiquidAI/LFM2.5-VL-1.6B) is the larger vision-language
|
||||
model in [Liquid AI's](https://www.liquid.ai/) LFM2.5-VL family. It pairs the **LFM2 hybrid
|
||||
language backbone** (short-range gated convolution blocks interleaved with grouped-query
|
||||
attention) with a SigLIP2 vision encoder for image understanding, while remaining small enough
|
||||
to serve on a single commodity GPU through vLLM's OpenAI-compatible API.
|
||||
|
||||
### Key Features
|
||||
- **Vision-language**: SigLIP2 vision encoder on top of the LFM2 hybrid language backbone — single- and multi-image prompts.
|
||||
- **Hybrid LM backbone**: Gated short convolutions interleaved with grouped-query attention — a smaller KV cache and lower decode latency than a same-size full-attention transformer.
|
||||
- **32K context**: 32,768-token context window.
|
||||
- **Native vLLM support**: Served via the `Lfm2VlForConditionalGeneration` architecture.
|
||||
|
||||
### Supported Variants
|
||||
|
||||
Vision-Language:
|
||||
- `LiquidAI/LFM2.5-VL-450M` (450M)
|
||||
- `LiquidAI/LFM2.5-VL-1.6B` (1.6B)
|
||||
|
||||
Text (same LFM2 family):
|
||||
- Dense: `LiquidAI/LFM2.5-230M`, `LiquidAI/LFM2.5-350M`, `LiquidAI/LFM2.5-1.2B-Instruct`, `LiquidAI/LFM2.5-1.2B-Thinking`, `LiquidAI/LFM2.5-1.2B-JP`, `LiquidAI/LFM2.5-1.2B-JP-202606`, `LiquidAI/LFM2.5-1.2B-Base`
|
||||
- MoE: `LiquidAI/LFM2.5-8B-A1B`
|
||||
|
||||
See the [LFM2.5 usage guide](../../LiquidAI/LFM2.5) for the full family.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Hardware**: 1× GPU with ≥8 GB VRAM.
|
||||
- **vLLM**: ≥ 0.23.0.
|
||||
|
||||
### pip (NVIDIA CUDA)
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend auto
|
||||
```
|
||||
|
||||
## Deployment Configurations
|
||||
|
||||
### Quick Start (Single GPU, BF16)
|
||||
```bash
|
||||
vllm serve LiquidAI/LFM2.5-VL-1.6B
|
||||
```
|
||||
|
||||
### Multiple Images per Request
|
||||
```bash
|
||||
vllm serve LiquidAI/LFM2.5-VL-1.6B \
|
||||
--limit-mm-per-prompt '{"image": 4}' \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
### Docker (NVIDIA)
|
||||
```bash
|
||||
docker run -itd --name lfm2.5-vl-1.6b \
|
||||
--ipc=host --network host --shm-size 16G --gpus all \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai:latest \
|
||||
--model LiquidAI/LFM2.5-VL-1.6B \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
### Image Understanding
|
||||
Send an image + text turn via the OpenAI chat API. The card recommends temperature `0.1`,
|
||||
`min_p 0.15`, `repetition_penalty 1.05` (`min_p` and `repetition_penalty` ride in `extra_body`).
|
||||
```python
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||||
response = client.chat.completions.create(
|
||||
model="LiquidAI/LFM2.5-VL-1.6B",
|
||||
messages=[{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "image_url", "image_url": {"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"}},
|
||||
{"type": "text", "text": "What is in this image?"},
|
||||
],
|
||||
}],
|
||||
temperature=0.1,
|
||||
extra_body={"min_p": 0.15, "repetition_penalty": 1.05},
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Multiple Images
|
||||
Launch with `--limit-mm-per-prompt '{"image": N}'`, then include several `image_url` blocks in
|
||||
one message to compare or reason across images.
|
||||
|
||||
## Performance
|
||||
|
||||
Tensor parallelism gives no speedup — the model fits on a single GPU.
|
||||
|
||||
## Configuration Tips
|
||||
- `--limit-mm-per-prompt '{"image": N}'` caps images per request (default 1).
|
||||
- Set `--max-model-len` to match your workload (up to 32K).
|
||||
- `--gpu-memory-utilization 0.90–0.95` maximizes KV cache capacity.
|
||||
- Sampling presets are per-request client defaults — don't bake them into `vllm serve`.
|
||||
|
||||
## References
|
||||
- [Model card](https://huggingface.co/LiquidAI/LFM2.5-VL-1.6B)
|
||||
- [LFM2.5 usage guide](../../LiquidAI/LFM2.5)
|
||||
- [Liquid AI](https://www.liquid.ai/)
|
||||
- [vLLM supported models](https://docs.vllm.ai/en/latest/models/supported_models.html)
|
||||
@@ -0,0 +1,149 @@
|
||||
meta:
|
||||
title: "LFM2.5 VL 450M"
|
||||
slug: "lfm2.5-vl-450m"
|
||||
provider: "LiquidAI"
|
||||
description: "Liquid AI's smallest vision-language model (450M) — LFM2 hybrid LM backbone plus a SigLIP2 vision tower for image+text chat, light enough for edge GPUs."
|
||||
date_added: 2026-06-25
|
||||
date_updated: 2026-06-24
|
||||
tasks:
|
||||
- multimodal
|
||||
performance_headline: "450M vision-language model (hybrid LM + SigLIP2 vision) — image understanding light enough for edge / on-device serving"
|
||||
related_recipes:
|
||||
- LiquidAI/LFM2.5-VL-1.6B
|
||||
- LiquidAI/LFM2.5-350M
|
||||
hardware:
|
||||
h100: verified
|
||||
h200: verified
|
||||
b200: verified
|
||||
|
||||
model:
|
||||
model_id: "LiquidAI/LFM2.5-VL-450M"
|
||||
min_vllm_version: "0.23.0"
|
||||
architecture: dense
|
||||
parameter_count: "450M"
|
||||
active_parameters: "450M"
|
||||
context_length: 32768
|
||||
base_args: []
|
||||
base_env: {}
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 2
|
||||
description: "Full BF16 (LM + vision encoder) — fits essentially any GPU"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
|
||||
hardware_overrides:
|
||||
# Blackwell (B200) only: route RMSNorm to the bundled oink kernels (measured ~+2.6% on B200,
|
||||
# output identical). Inert (falls back to native) on non-Blackwell GPUs.
|
||||
blackwell:
|
||||
extra_env:
|
||||
VLLM_USE_OINK_OPS: "1"
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
tp: 1
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[LFM2.5-VL-450M](https://huggingface.co/LiquidAI/LFM2.5-VL-450M) is the smallest vision-language
|
||||
model in [Liquid AI's](https://www.liquid.ai/) LFM2.5-VL family — a SigLIP2 vision encoder on
|
||||
top of the **LFM2 hybrid language backbone** (short-range gated convolution blocks interleaved
|
||||
with grouped-query attention). It is light enough for edge and on-device image understanding,
|
||||
served through vLLM's OpenAI-compatible API.
|
||||
|
||||
### Key Features
|
||||
- **Vision-language**: SigLIP2 vision encoder on top of the LFM2 hybrid language backbone — single- and multi-image prompts.
|
||||
- **Edge-ready**: ~1 GB of BF16 weights — runs on commodity and on-device GPUs.
|
||||
- **Hybrid LM backbone**: Gated short convolutions interleaved with grouped-query attention — a smaller KV cache and lower decode latency than a same-size full-attention transformer.
|
||||
- **32K context**: 32,768-token context window.
|
||||
- **Native vLLM support**: Served via the `Lfm2VlForConditionalGeneration` architecture.
|
||||
|
||||
### Supported Variants
|
||||
|
||||
Vision-Language:
|
||||
- `LiquidAI/LFM2.5-VL-450M` (450M)
|
||||
- `LiquidAI/LFM2.5-VL-1.6B` (1.6B)
|
||||
|
||||
Text (same LFM2 family):
|
||||
- Dense: `LiquidAI/LFM2.5-230M`, `LiquidAI/LFM2.5-350M`, `LiquidAI/LFM2.5-1.2B-Instruct`, `LiquidAI/LFM2.5-1.2B-Thinking`, `LiquidAI/LFM2.5-1.2B-JP`, `LiquidAI/LFM2.5-1.2B-JP-202606`, `LiquidAI/LFM2.5-1.2B-Base`
|
||||
- MoE: `LiquidAI/LFM2.5-8B-A1B`
|
||||
|
||||
See the [LFM2.5 usage guide](../../LiquidAI/LFM2.5) for the full family.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Hardware**: 1× GPU (~1–2 GB VRAM for weights; any modern GPU works).
|
||||
- **vLLM**: ≥ 0.23.0.
|
||||
|
||||
### pip (NVIDIA CUDA)
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend auto
|
||||
```
|
||||
|
||||
## Deployment Configurations
|
||||
|
||||
### Quick Start (Single GPU, BF16)
|
||||
```bash
|
||||
vllm serve LiquidAI/LFM2.5-VL-450M
|
||||
```
|
||||
|
||||
### Multiple Images per Request
|
||||
```bash
|
||||
vllm serve LiquidAI/LFM2.5-VL-450M \
|
||||
--limit-mm-per-prompt '{"image": 4}' \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
### Docker (NVIDIA)
|
||||
```bash
|
||||
docker run -itd --name lfm2.5-vl-450m \
|
||||
--ipc=host --network host --shm-size 16G --gpus all \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai:latest \
|
||||
--model LiquidAI/LFM2.5-VL-450M \
|
||||
--host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
### Image Understanding
|
||||
Send an image + text turn via the OpenAI chat API. The card recommends temperature `0.1`,
|
||||
`min_p 0.15`, `repetition_penalty 1.05` (`min_p` and `repetition_penalty` ride in `extra_body`).
|
||||
```python
|
||||
from openai import OpenAI
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||||
response = client.chat.completions.create(
|
||||
model="LiquidAI/LFM2.5-VL-450M",
|
||||
messages=[{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "image_url", "image_url": {"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"}},
|
||||
{"type": "text", "text": "What is in this image?"},
|
||||
],
|
||||
}],
|
||||
temperature=0.1,
|
||||
extra_body={"min_p": 0.15, "repetition_penalty": 1.05},
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Multiple Images
|
||||
Launch with `--limit-mm-per-prompt '{"image": N}'`, then include several `image_url` blocks in
|
||||
one message to compare or reason across images.
|
||||
|
||||
## Configuration Tips
|
||||
- At 450M the model fits any GPU; ideal for edge / on-device image understanding.
|
||||
- `--limit-mm-per-prompt '{"image": N}'` caps images per request (default 1).
|
||||
- Set `--max-model-len` to match your workload (up to 32K).
|
||||
- Sampling presets are per-request client defaults — don't bake them into `vllm serve`.
|
||||
|
||||
## References
|
||||
- [Model card](https://huggingface.co/LiquidAI/LFM2.5-VL-450M)
|
||||
- [LFM2.5 usage guide](../../LiquidAI/LFM2.5)
|
||||
- [Liquid AI](https://www.liquid.ai/)
|
||||
- [vLLM supported models](https://docs.vllm.ai/en/latest/models/supported_models.html)
|
||||
@@ -0,0 +1,179 @@
|
||||
meta:
|
||||
title: "MiniMax-M2.1"
|
||||
slug: "minimax-m2.1"
|
||||
provider: "MiniMax"
|
||||
description: "MiniMax M2.1 MoE language model (230B total / 10B active) for coding, agent toolchains, and long-context reasoning — native FP8 checkpoint"
|
||||
date_added: 2025-12-20
|
||||
date_updated: 2026-04-17
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "Updated M2 series MoE with strong SWE-Bench and Terminal-Bench performance, 196K context"
|
||||
related_recipes: []
|
||||
hardware:
|
||||
h100: verified
|
||||
mi300x: verified
|
||||
mi325x: verified
|
||||
mi355x: verified
|
||||
|
||||
model:
|
||||
model_id: "MiniMaxAI/MiniMax-M2.1"
|
||||
min_vllm_version: "0.11.1"
|
||||
architecture: moe
|
||||
parameter_count: "230B"
|
||||
active_parameters: "10B"
|
||||
context_length: 196608
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "MiniMax M2 tool call parser with automatic tool choice"
|
||||
args:
|
||||
- "--tool-call-parser"
|
||||
- "minimax_m2"
|
||||
- "--enable-auto-tool-choice"
|
||||
reasoning:
|
||||
description: "MiniMax M2 reasoning parser for chain-of-thought extraction"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "minimax_m2"
|
||||
|
||||
opt_in_features: []
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: fp8
|
||||
vram_minimum_gb: 276
|
||||
description: "Native FP8 checkpoint — 4x H200/H20/H100 or 4x A100/A800 for weights, plus KV cache headroom"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- single_node_tep
|
||||
- single_node_dep
|
||||
- multi_node_tp
|
||||
- multi_node_tp_pp
|
||||
- multi_node_tep
|
||||
- multi_node_dep
|
||||
- pd_cluster
|
||||
|
||||
kv_offload_support:
|
||||
offloading_cpu: verified
|
||||
offloading_fs: verified
|
||||
|
||||
hardware_overrides:
|
||||
amd:
|
||||
extra_args:
|
||||
- "--tensor-parallel-size"
|
||||
- "2"
|
||||
- "--attention-backend"
|
||||
- "ROCM_AITER_FA"
|
||||
extra_env:
|
||||
VLLM_ROCM_USE_AITER: "1"
|
||||
VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT: "1"
|
||||
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
# Pure TP=8 fails on the FP8 block-quantized gate_up weights:
|
||||
# intermediate_size / 8 = 192 is not divisible by block_n = 128.
|
||||
# Pin TP=4 on 8-GPU nodes; for full-node utilization use
|
||||
# single_node_tep (TP+EP) or single_node_dep (DP+EP).
|
||||
tp: 4
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[MiniMax-M2.1](https://huggingface.co/MiniMaxAI/MiniMax-M2.1) is part of the MiniMax
|
||||
M2 series of advanced MoE language models. It retains the M2 architecture (10B active,
|
||||
230B total) with improvements over the original M2 release. Supports 196K context per sequence.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **OS:** Linux
|
||||
- **Python:** 3.10 - 3.13
|
||||
- **NVIDIA:** compute capability >= 7.0; ~220 GB for weights + 240 GB per 1M context tokens
|
||||
- **AMD:** MI300X / MI325X / MI350X / MI355X with ROCm 7.0+
|
||||
|
||||
### Install vLLM (NVIDIA)
|
||||
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend auto
|
||||
```
|
||||
|
||||
### Docker (dedicated M2-series image)
|
||||
|
||||
```bash
|
||||
docker run --gpus all \
|
||||
-p 8000:8000 \
|
||||
--ipc=host \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai:minimax27 MiniMaxAI/MiniMax-M2.1 \
|
||||
--tensor-parallel-size 4 \
|
||||
--tool-call-parser minimax_m2 \
|
||||
--reasoning-parser minimax_m2 \
|
||||
--enable-auto-tool-choice \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
## Launching the Server
|
||||
|
||||
### NVIDIA — TP4
|
||||
|
||||
```bash
|
||||
vllm serve MiniMaxAI/MiniMax-M2.1 \
|
||||
--tensor-parallel-size 4 \
|
||||
--tool-call-parser minimax_m2 \
|
||||
--reasoning-parser minimax_m2 \
|
||||
--enable-auto-tool-choice \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
Pure TP8 is not supported. For >4 GPUs use DP+EP or TP+EP.
|
||||
|
||||
### TP4+EP (recommended for H100)
|
||||
|
||||
```bash
|
||||
vllm serve MiniMaxAI/MiniMax-M2.1 \
|
||||
--tensor-parallel-size 4 \
|
||||
--enable-expert-parallel \
|
||||
--tool-call-parser minimax_m2 \
|
||||
--reasoning-parser minimax_m2 \
|
||||
--enable-auto-tool-choice
|
||||
```
|
||||
|
||||
### AMD ROCm — TP2
|
||||
|
||||
```bash
|
||||
VLLM_ROCM_USE_AITER=1 vllm serve MiniMaxAI/MiniMax-M2.1 \
|
||||
--tensor-parallel-size 2 \
|
||||
--tool-call-parser minimax_m2 \
|
||||
--reasoning-parser minimax_m2 \
|
||||
--enable-auto-tool-choice \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
## Benchmarking
|
||||
|
||||
```bash
|
||||
vllm bench serve \
|
||||
--backend vllm \
|
||||
--model MiniMaxAI/MiniMax-M2.1 \
|
||||
--endpoint /v1/completions \
|
||||
--dataset-name random \
|
||||
--random-input 2048 \
|
||||
--random-output 1024 \
|
||||
--max-concurrency 10 \
|
||||
--num-prompt 100
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- See [MiniMax-M2](./MiniMax-M2.yaml) for shared troubleshooting notes
|
||||
(latest vLLM fusion behavior, nightly vs stable, DeepGEMM, AITER).
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/MiniMaxAI/MiniMax-M2.1)
|
||||
- [MiniMax](https://www.minimax.io/)
|
||||
@@ -0,0 +1,249 @@
|
||||
meta:
|
||||
title: "MiniMax-M2.5"
|
||||
slug: "minimax-m2.5"
|
||||
provider: "MiniMax"
|
||||
description: "MiniMax M2.5 MoE language model (230B total / 10B active) for coding, agent toolchains, and long-context reasoning — native FP8 checkpoint"
|
||||
date_added: 2026-02-13
|
||||
date_updated: 2026-09-14
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "Refreshed M2 series MoE with strong SWE-Bench and Terminal-Bench performance, 196K context"
|
||||
related_recipes: []
|
||||
hardware:
|
||||
h100: verified
|
||||
h200: verified
|
||||
mi300x: verified
|
||||
mi325x: verified
|
||||
mi355x: verified
|
||||
|
||||
model:
|
||||
model_id: "MiniMaxAI/MiniMax-M2.5"
|
||||
min_vllm_version: "0.20.2"
|
||||
default_frontend: rust
|
||||
docker_image: "vllm/vllm-openai:v0.28.0"
|
||||
architecture: moe
|
||||
parameter_count: "230B"
|
||||
active_parameters: "10B"
|
||||
context_length: 196608
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "MiniMax M2 tool call parser with automatic tool choice"
|
||||
args:
|
||||
- "--tool-call-parser"
|
||||
- "minimax_m2"
|
||||
- "--enable-auto-tool-choice"
|
||||
reasoning:
|
||||
description: "MiniMax M2 reasoning parser for chain-of-thought extraction"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "minimax_m2"
|
||||
|
||||
opt_in_features: []
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: fp8
|
||||
vram_minimum_gb: 276
|
||||
description: "Native FP8 checkpoint — 4x H200/H20/H100 or 4x A100/A800 for weights, plus KV cache headroom"
|
||||
nvfp4:
|
||||
model_id: "nvidia/MiniMax-M2.5-NVFP4"
|
||||
precision: nvfp4
|
||||
vram_minimum_gb: 138
|
||||
description: "NVIDIA NVFP4 quantized weights for Blackwell GPUs"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
|
||||
redhat_nvfp4:
|
||||
model_id: "RedHatAI/MiniMax-M2.5-NVFP4"
|
||||
precision: nvfp4
|
||||
label: "NVFP4 (RedHat)"
|
||||
vram_minimum_gb: 146
|
||||
description: "RedHatAI NVFP4 quantization"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- single_node_tep
|
||||
- single_node_dep
|
||||
- multi_node_tp
|
||||
- multi_node_tp_pp
|
||||
- multi_node_tep
|
||||
- multi_node_dep
|
||||
- pd_cluster
|
||||
|
||||
kv_offload_support:
|
||||
offloading_cpu: verified
|
||||
offloading_fs: verified
|
||||
|
||||
hardware_overrides:
|
||||
nvidia:
|
||||
extra_args:
|
||||
- "--compilation-config"
|
||||
- '{"cudagraph_mode":"PIECEWISE"}'
|
||||
hopper:
|
||||
extra_args:
|
||||
- "--compilation-config"
|
||||
- '{"cudagraph_mode":"PIECEWISE"}'
|
||||
- "--max-num-seqs"
|
||||
- "512"
|
||||
- "--max-num-batched-tokens"
|
||||
- "32768"
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
- "--moe-backend"
|
||||
- "triton"
|
||||
- "--attention-backend"
|
||||
- "FLASHINFER"
|
||||
- "--enable-flashinfer-autotune"
|
||||
extra_env:
|
||||
PYTHONNOUSERSITE: "1"
|
||||
VLLM_USE_DEEP_GEMM: "0"
|
||||
VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER: "0"
|
||||
VLLM_FLOAT32_MATMUL_PRECISION: "high"
|
||||
amd:
|
||||
extra_args:
|
||||
- "--tensor-parallel-size"
|
||||
- "2"
|
||||
- "--attention-backend"
|
||||
- "ROCM_AITER_FA"
|
||||
extra_env:
|
||||
VLLM_ROCM_USE_AITER: "1"
|
||||
VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT: "1"
|
||||
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
# Pure TP=8 fails on the FP8 block-quantized gate_up weights:
|
||||
# intermediate_size / 8 = 192 is not divisible by block_n = 128.
|
||||
# Pin TP=4 on 8-GPU nodes; for full-node utilization use
|
||||
# single_node_tep (TP+EP) or single_node_dep (DP+EP).
|
||||
tp: 4
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[MiniMax-M2.5](https://huggingface.co/MiniMaxAI/MiniMax-M2.5) is part of the MiniMax
|
||||
M2 series of advanced MoE language models. It retains the M2 architecture (10B active,
|
||||
230B total) and a 196K context per sequence.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **OS:** Linux
|
||||
- **Python:** 3.10 - 3.13
|
||||
- **NVIDIA:** compute capability >= 7.0; ~220 GB for weights + 240 GB per 1M context tokens
|
||||
- **AMD:** MI300X / MI325X / MI350X / MI355X with ROCm 7.0+
|
||||
|
||||
### Install vLLM (NVIDIA)
|
||||
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend auto
|
||||
```
|
||||
|
||||
### Docker (dedicated M2-series image)
|
||||
|
||||
```bash
|
||||
docker run --gpus all \
|
||||
-p 8000:8000 \
|
||||
--ipc=host \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
-e PYTHONNOUSERSITE=1 \
|
||||
-e VLLM_USE_DEEP_GEMM=0 \
|
||||
-e VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER=0 \
|
||||
-e VLLM_FLOAT32_MATMUL_PRECISION=high \
|
||||
vllm/vllm-openai:v0.28.0 MiniMaxAI/MiniMax-M2.5 \
|
||||
--tensor-parallel-size 4 \
|
||||
--max-num-seqs 512 \
|
||||
--max-num-batched-tokens 32768 \
|
||||
--kv-cache-dtype fp8 \
|
||||
--moe-backend triton \
|
||||
--attention-backend FLASHINFER \
|
||||
--enable-flashinfer-autotune \
|
||||
--tool-call-parser minimax_m2 \
|
||||
--reasoning-parser minimax_m2 \
|
||||
--enable-auto-tool-choice \
|
||||
--compilation-config '{"cudagraph_mode":"PIECEWISE"}' \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
## Launching the Server
|
||||
|
||||
### NVIDIA H200 — TP4
|
||||
|
||||
```bash
|
||||
PYTHONNOUSERSITE=1 \
|
||||
VLLM_USE_DEEP_GEMM=0 \
|
||||
VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER=0 \
|
||||
VLLM_FLOAT32_MATMUL_PRECISION=high \
|
||||
vllm serve MiniMaxAI/MiniMax-M2.5 \
|
||||
--tensor-parallel-size 4 \
|
||||
--max-num-seqs 512 \
|
||||
--max-num-batched-tokens 32768 \
|
||||
--kv-cache-dtype fp8 \
|
||||
--moe-backend triton \
|
||||
--attention-backend FLASHINFER \
|
||||
--enable-flashinfer-autotune \
|
||||
--tool-call-parser minimax_m2 \
|
||||
--reasoning-parser minimax_m2 \
|
||||
--compilation-config '{"cudagraph_mode":"PIECEWISE"}' \
|
||||
--enable-auto-tool-choice \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
For H200 FP8, choose the attention backend by sequence length for best performance:
|
||||
shorter sequences (e.g. 1024) should keep the command above with `--attention-backend FLASHINFER`
|
||||
and `--enable-flashinfer-autotune`, while longer input sequences (e.g. 8192)
|
||||
should remove those flags to use the auto-selected FlashAttention backend.
|
||||
|
||||
Pure TP8 is not supported. For >4 GPUs use DP+EP or TP+EP.
|
||||
|
||||
### TP4+EP (recommended for H100)
|
||||
|
||||
```bash
|
||||
vllm serve MiniMaxAI/MiniMax-M2.5 \
|
||||
--tensor-parallel-size 4 \
|
||||
--enable-expert-parallel \
|
||||
--tool-call-parser minimax_m2 \
|
||||
--reasoning-parser minimax_m2 \
|
||||
--compilation-config '{"cudagraph_mode":"PIECEWISE"}' \
|
||||
--enable-auto-tool-choice
|
||||
```
|
||||
|
||||
### AMD ROCm — TP2
|
||||
|
||||
```bash
|
||||
VLLM_ROCM_USE_AITER=1 vllm serve MiniMaxAI/MiniMax-M2.5 \
|
||||
--tensor-parallel-size 2 \
|
||||
--tool-call-parser minimax_m2 \
|
||||
--reasoning-parser minimax_m2 \
|
||||
--enable-auto-tool-choice \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
## Benchmarking
|
||||
|
||||
```bash
|
||||
vllm bench serve \
|
||||
--backend vllm \
|
||||
--model MiniMaxAI/MiniMax-M2.5 \
|
||||
--endpoint /v1/completions \
|
||||
--dataset-name random \
|
||||
--random-input 2048 \
|
||||
--random-output 1024 \
|
||||
--max-concurrency 10 \
|
||||
--num-prompt 100
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- See [MiniMax-M2](./MiniMax-M2.yaml) for shared troubleshooting notes
|
||||
(latest vLLM fusion behavior, nightly vs stable, DeepGEMM, AITER).
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/MiniMaxAI/MiniMax-M2.5)
|
||||
- [MiniMax](https://www.minimax.io/)
|
||||
@@ -0,0 +1,252 @@
|
||||
meta:
|
||||
title: "MiniMax-M2.7"
|
||||
slug: "minimax-m2.7"
|
||||
provider: "MiniMax"
|
||||
description: "MiniMax M2.7 MoE language model (230B total / 10B active) — latest M2 release for coding, agent toolchains, and long-context reasoning with native FP8"
|
||||
date_added: 2026-04-12
|
||||
date_updated: 2026-09-14
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "Latest M2 series release; verified accuracy on AIME25, GPQA-D, GSM8K; 196K context"
|
||||
related_recipes: []
|
||||
hardware:
|
||||
h100: verified
|
||||
h200: verified
|
||||
mi300x: verified
|
||||
mi325x: verified
|
||||
mi355x: verified
|
||||
dgx_station_gb300: verified
|
||||
rtx_pro_6000_2x: verified
|
||||
|
||||
model:
|
||||
model_id: "MiniMaxAI/MiniMax-M2.7"
|
||||
min_vllm_version: "0.20.0"
|
||||
default_frontend: rust
|
||||
architecture: moe
|
||||
parameter_count: "230B"
|
||||
active_parameters: "10B"
|
||||
context_length: 196608
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "MiniMax M2 tool call parser with automatic tool choice"
|
||||
args:
|
||||
- "--tool-call-parser"
|
||||
- "minimax_m2"
|
||||
- "--enable-auto-tool-choice"
|
||||
reasoning:
|
||||
description: "MiniMax M2 reasoning parser for chain-of-thought extraction"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "minimax_m2"
|
||||
|
||||
opt_in_features: []
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: fp8
|
||||
vram_minimum_gb: 276
|
||||
description: "Native FP8 checkpoint — 4x H200/H20/H100 or 4x A100/A800 for weights, plus KV cache headroom"
|
||||
nvfp4:
|
||||
model_id: "nvidia/MiniMax-M2.7-NVFP4"
|
||||
precision: nvfp4
|
||||
vram_minimum_gb: 138
|
||||
description: "NVIDIA NVFP4 quantized weights for Blackwell GPUs"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
hardware_overrides:
|
||||
dgx_station_gb300:
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.92"
|
||||
- "--load-format"
|
||||
- "fastsafetensors"
|
||||
- "--max-num-batched-tokens"
|
||||
- "8192"
|
||||
- "--enable-chunked-prefill"
|
||||
- "--enable-prefix-caching"
|
||||
rtx_pro_6000_2x:
|
||||
extra_args:
|
||||
- "--load-format"
|
||||
- "fastsafetensors"
|
||||
- "--enable-chunked-prefill"
|
||||
- "--enable-prefix-caching"
|
||||
- "--max-num-batched-tokens"
|
||||
- "8192"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.95"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- single_node_tep
|
||||
- single_node_dep
|
||||
- multi_node_tp
|
||||
- multi_node_tp_pp
|
||||
- multi_node_tep
|
||||
- multi_node_dep
|
||||
- pd_cluster
|
||||
|
||||
kv_offload_support:
|
||||
offloading_cpu: verified
|
||||
offloading_fs: verified
|
||||
|
||||
hardware_overrides:
|
||||
amd:
|
||||
extra_args:
|
||||
- "--tensor-parallel-size"
|
||||
- "2"
|
||||
- "--attention-backend"
|
||||
- "ROCM_AITER_FA"
|
||||
extra_env:
|
||||
VLLM_ROCM_USE_AITER: "1"
|
||||
VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT: "1"
|
||||
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
# Pure TP=8 fails on the FP8 block-quantized gate_up weights:
|
||||
# intermediate_size / 8 = 192 is not divisible by block_n = 128.
|
||||
# Pin TP=4 on 8-GPU nodes; for full-node utilization use
|
||||
# single_node_tep (TP+EP) or single_node_dep (DP+EP).
|
||||
tp: 4
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[MiniMax-M2.7](https://huggingface.co/MiniMaxAI/MiniMax-M2.7) is the latest release
|
||||
in the MiniMax M2 series. Like earlier M2 variants, it ships with 10B active
|
||||
parameters out of 230B total, and supports a 196K context per sequence. MiniMax has
|
||||
verified M2.7 accuracy on AIME25, GPQA-D, and GSM8K at vLLM commit
|
||||
`0f3ce4c74b1875791d6604e006b6e905fde9f698`.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **OS:** Linux
|
||||
- **Python:** 3.10 - 3.13
|
||||
- **NVIDIA:** compute capability >= 7.0; ~220 GB for weights + 240 GB per 1M context tokens
|
||||
- **AMD:** MI300X / MI325X / MI350X / MI355X with ROCm 7.0+
|
||||
|
||||
### Install vLLM (NVIDIA)
|
||||
|
||||
```bash
|
||||
uv pip install -U vllm --extra-index-url https://wheels.vllm.ai/nightly
|
||||
```
|
||||
|
||||
### Install vLLM (AMD ROCm)
|
||||
|
||||
```bash
|
||||
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
|
||||
```
|
||||
|
||||
### Docker (dedicated M2-series image)
|
||||
|
||||
```bash
|
||||
docker run --gpus all \
|
||||
-p 8000:8000 \
|
||||
--ipc=host \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai:minimax27 MiniMaxAI/MiniMax-M2.7 \
|
||||
--tensor-parallel-size 4 \
|
||||
--tool-call-parser minimax_m2 \
|
||||
--reasoning-parser minimax_m2 \
|
||||
--enable-auto-tool-choice \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
## Launching the Server
|
||||
|
||||
### NVIDIA — TP4 (4x H200/H20/H100 or 4x A100/A800)
|
||||
|
||||
```bash
|
||||
vllm serve MiniMaxAI/MiniMax-M2.7 \
|
||||
--tensor-parallel-size 4 \
|
||||
--tool-call-parser minimax_m2 \
|
||||
--reasoning-parser minimax_m2 \
|
||||
--enable-auto-tool-choice \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
Pure TP8 is not supported. For >4 GPUs use DP+EP or TP+EP:
|
||||
|
||||
### DP8+EP
|
||||
|
||||
```bash
|
||||
vllm serve MiniMaxAI/MiniMax-M2.7 \
|
||||
--data-parallel-size 8 \
|
||||
--enable-expert-parallel \
|
||||
--tool-call-parser minimax_m2 \
|
||||
--reasoning-parser minimax_m2 \
|
||||
--enable-auto-tool-choice
|
||||
```
|
||||
|
||||
### TP4+EP (recommended for H100)
|
||||
|
||||
```bash
|
||||
vllm serve MiniMaxAI/MiniMax-M2.7 \
|
||||
--tensor-parallel-size 4 \
|
||||
--enable-expert-parallel \
|
||||
--tool-call-parser minimax_m2 \
|
||||
--reasoning-parser minimax_m2 \
|
||||
--enable-auto-tool-choice
|
||||
```
|
||||
|
||||
### TP8+EP
|
||||
|
||||
```bash
|
||||
vllm serve MiniMaxAI/MiniMax-M2.7 \
|
||||
--tensor-parallel-size 8 \
|
||||
--enable-expert-parallel \
|
||||
--tool-call-parser minimax_m2 \
|
||||
--reasoning-parser minimax_m2 \
|
||||
--enable-auto-tool-choice
|
||||
```
|
||||
|
||||
### AMD ROCm — TP2
|
||||
|
||||
```bash
|
||||
VLLM_ROCM_USE_AITER=1 vllm serve MiniMaxAI/MiniMax-M2.7 \
|
||||
--tensor-parallel-size 2 \
|
||||
--tool-call-parser minimax_m2 \
|
||||
--reasoning-parser minimax_m2 \
|
||||
--enable-auto-tool-choice \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
## Benchmarking
|
||||
|
||||
```bash
|
||||
vllm bench serve \
|
||||
--backend vllm \
|
||||
--model MiniMaxAI/MiniMax-M2.7 \
|
||||
--endpoint /v1/completions \
|
||||
--dataset-name random \
|
||||
--random-input 2048 \
|
||||
--random-output 1024 \
|
||||
--max-concurrency 10 \
|
||||
--num-prompt 100
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **MiniMax QK norm fusion:** Latest vLLM applies the fusion automatically at
|
||||
runtime when the fused CUDA op is available; older vLLM builds used an
|
||||
explicit `fuse_minimax_qk_norm` compile pass.
|
||||
- **Corrupted output on stable release:** Upgrade to a nightly after commit
|
||||
`cf3eacfe58fa9e745c2854782ada884a9f992cf7`.
|
||||
- **Verified accuracy:** Use the pinned commit
|
||||
`0f3ce4c74b1875791d6604e006b6e905fde9f698` if reproducing MiniMax's reported results.
|
||||
- **DeepGEMM:** vLLM uses DeepGEMM by default; install via
|
||||
[install_deepgemm.sh](https://github.com/vllm-project/vllm/blob/main/tools/install_deepgemm.sh)
|
||||
if missing.
|
||||
- **AITER first launch:** Initial AMD launch JIT-compiles optimized kernels; subsequent launches reuse cached kernels.
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/MiniMaxAI/MiniMax-M2.7)
|
||||
- [MiniMax](https://www.minimax.io/)
|
||||
- [vLLM PR #43410 (automatic MiniMax QK norm fusion)](https://github.com/vllm-project/vllm/pull/43410)
|
||||
@@ -0,0 +1,244 @@
|
||||
meta:
|
||||
title: "MiniMax-M2"
|
||||
slug: "minimax-m2"
|
||||
provider: "MiniMax"
|
||||
description: "MiniMax M2 MoE language model (230B total / 10B active) for coding, agent toolchains, and long-context reasoning — native FP8 checkpoint, with an NVFP4 variant for Blackwell"
|
||||
date_added: 2025-10-26
|
||||
date_updated: 2026-05-11
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "Open-source MoE with strong SWE-Bench and Terminal-Bench performance, 196K context"
|
||||
related_recipes: []
|
||||
hardware:
|
||||
h100: verified
|
||||
h200: verified
|
||||
mi300x: verified
|
||||
mi325x: verified
|
||||
mi355x: verified
|
||||
|
||||
model:
|
||||
model_id: "MiniMaxAI/MiniMax-M2"
|
||||
min_vllm_version: "0.11.1"
|
||||
architecture: moe
|
||||
parameter_count: "230B"
|
||||
active_parameters: "10B"
|
||||
context_length: 196608
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
base_env: {}
|
||||
|
||||
dependencies:
|
||||
- note: "Optional: DeepGEMM FP8 MoE kernels for throughput (skip on B200 — known FlashInfer FP8 MoE error)"
|
||||
command: "export VLLM_USE_DEEP_GEMM=1"
|
||||
optional: true
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "MiniMax M2 tool call parser with automatic tool choice"
|
||||
args:
|
||||
- "--tool-call-parser"
|
||||
- "minimax_m2"
|
||||
- "--enable-auto-tool-choice"
|
||||
reasoning:
|
||||
description: "MiniMax M2 reasoning parser for chain-of-thought extraction"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "minimax_m2"
|
||||
|
||||
opt_in_features: []
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: fp8
|
||||
vram_minimum_gb: 276
|
||||
description: "Native FP8 checkpoint — 4x H200/H20/H100 or 4x A100/A800 for weights, plus KV cache headroom"
|
||||
nvfp4:
|
||||
model_id: "RedHatAI/MiniMax-M2-NVFP4"
|
||||
precision: nvfp4
|
||||
vram_minimum_gb: 138
|
||||
description: "NVFP4 (4-bit) quantized weights — requires Blackwell (B200/B300); roughly half the VRAM of the FP8 checkpoint"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- single_node_tep
|
||||
- single_node_dep
|
||||
- multi_node_tp
|
||||
- multi_node_tp_pp
|
||||
- multi_node_tep
|
||||
- multi_node_dep
|
||||
- pd_cluster
|
||||
|
||||
kv_offload_support:
|
||||
offloading_cpu: verified
|
||||
offloading_fs: verified
|
||||
|
||||
hardware_overrides:
|
||||
amd:
|
||||
# First launch JIT-compiles AITER kernels (CK FP8 MoE, RMSNorm, activation);
|
||||
# expect a slow cold start on MI300X/MI325X/MI350X/MI355X.
|
||||
extra_args:
|
||||
- "--tensor-parallel-size"
|
||||
- "2"
|
||||
- "--attention-backend"
|
||||
- "ROCM_AITER_FA"
|
||||
extra_env:
|
||||
VLLM_ROCM_USE_AITER: "1"
|
||||
VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT: "1"
|
||||
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
# Pure TP=8 fails on the FP8 block-quantized gate_up weights:
|
||||
# intermediate_size / 8 = 192 is not divisible by block_n = 128.
|
||||
# Pin TP=4 on 8-GPU nodes; for full-node utilization use
|
||||
# single_node_tep (TP+EP) or single_node_dep (DP+EP).
|
||||
tp: 4
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[MiniMax-M2](https://huggingface.co/MiniMaxAI/MiniMax-M2) is an advanced MoE language
|
||||
model from [MiniMax](https://www.minimax.io/). Highlights:
|
||||
|
||||
- Superior intelligence — #1 among open-source models globally on math, science, coding, tool use
|
||||
- Advanced coding — multi-file edits, run-fix loops, test-validated repairs (SWE-Bench, Terminal-Bench)
|
||||
- Agent performance — plans and executes complex toolchains across shell, browser, and code
|
||||
- Efficient design — 10B active / 230B total for low latency and high throughput
|
||||
- 196K context length per sequence
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **OS:** Linux
|
||||
- **Python:** 3.10 - 3.13
|
||||
- **NVIDIA:** compute capability >= 7.0; ~220 GB for weights + 240 GB per 1M context tokens
|
||||
- **AMD:** MI300X / MI325X / MI350X / MI355X with ROCm 7.0+
|
||||
|
||||
### Install vLLM (NVIDIA, stable)
|
||||
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend auto
|
||||
```
|
||||
|
||||
### Install vLLM (NVIDIA, nightly)
|
||||
|
||||
If you hit corrupted output, upgrade to a nightly after commit
|
||||
`cf3eacfe58fa9e745c2854782ada884a9f992cf7`:
|
||||
|
||||
```bash
|
||||
uv pip install -U vllm --extra-index-url https://wheels.vllm.ai/nightly
|
||||
```
|
||||
|
||||
### Install vLLM (AMD ROCm)
|
||||
|
||||
```bash
|
||||
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
|
||||
```
|
||||
|
||||
### Docker (NVIDIA, dedicated M2 image)
|
||||
|
||||
```bash
|
||||
docker run --gpus all \
|
||||
-p 8000:8000 \
|
||||
--ipc=host \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai:minimax27 MiniMaxAI/MiniMax-M2 \
|
||||
--tensor-parallel-size 4 \
|
||||
--tool-call-parser minimax_m2 \
|
||||
--reasoning-parser minimax_m2 \
|
||||
--enable-auto-tool-choice \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
## Launching the Server
|
||||
|
||||
### NVIDIA — TP4 (4x H200/H20/H100 or 4x A100/A800)
|
||||
|
||||
```bash
|
||||
vllm serve MiniMaxAI/MiniMax-M2 \
|
||||
--tensor-parallel-size 4 \
|
||||
--tool-call-parser minimax_m2 \
|
||||
--reasoning-parser minimax_m2 \
|
||||
--enable-auto-tool-choice \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
Pure TP8 is not supported. For >4 GPUs use DP+EP or TP+EP:
|
||||
|
||||
### TP4+EP (recommended for H100)
|
||||
|
||||
```bash
|
||||
vllm serve MiniMaxAI/MiniMax-M2 \
|
||||
--tensor-parallel-size 4 \
|
||||
--enable-expert-parallel \
|
||||
--tool-call-parser minimax_m2 \
|
||||
--reasoning-parser minimax_m2 \
|
||||
--enable-auto-tool-choice
|
||||
```
|
||||
|
||||
### DP8+EP
|
||||
|
||||
```bash
|
||||
vllm serve MiniMaxAI/MiniMax-M2 \
|
||||
--data-parallel-size 8 \
|
||||
--enable-expert-parallel \
|
||||
--tool-call-parser minimax_m2 \
|
||||
--reasoning-parser minimax_m2 \
|
||||
--enable-auto-tool-choice
|
||||
```
|
||||
|
||||
### AMD ROCm — TP2
|
||||
|
||||
```bash
|
||||
VLLM_ROCM_USE_AITER=1 vllm serve MiniMaxAI/MiniMax-M2 \
|
||||
--tensor-parallel-size 2 \
|
||||
--tool-call-parser minimax_m2 \
|
||||
--reasoning-parser minimax_m2 \
|
||||
--enable-auto-tool-choice \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
## Benchmarking
|
||||
|
||||
```bash
|
||||
vllm bench serve \
|
||||
--backend vllm \
|
||||
--model MiniMaxAI/MiniMax-M2 \
|
||||
--endpoint /v1/completions \
|
||||
--dataset-name random \
|
||||
--random-input 2048 \
|
||||
--random-output 1024 \
|
||||
--max-concurrency 10 \
|
||||
--num-prompt 100
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **MiniMax QK norm fusion:** Latest vLLM applies the fusion automatically at
|
||||
runtime when the fused CUDA op is available; older vLLM builds used an
|
||||
explicit `fuse_minimax_qk_norm` compile pass.
|
||||
- **Corrupted output on stable release:** Upgrade to a nightly after commit
|
||||
`cf3eacfe58fa9e745c2854782ada884a9f992cf7`.
|
||||
- **DeepGEMM:** vLLM uses DeepGEMM by default; install via
|
||||
[install_deepgemm.sh](https://github.com/vllm-project/vllm/blob/main/tools/install_deepgemm.sh)
|
||||
when missing.
|
||||
- **AITER first launch:** AMD initial launch JIT-compiles CK-based FP8 MoE, RMSNorm,
|
||||
and activation kernels. Subsequent launches use cached kernels.
|
||||
|
||||
## Quantized Variant (NVFP4)
|
||||
|
||||
[`RedHatAI/MiniMax-M2-NVFP4`](https://huggingface.co/RedHatAI/MiniMax-M2-NVFP4) is an NVFP4 (4-bit)
|
||||
checkpoint — roughly half the VRAM of the native FP8 checkpoint. Select the **nvfp4** variant
|
||||
above (it sets `--kv-cache-dtype fp8`; on vLLM v0.28.0+ the FlashInfer NVFP4 MoE
|
||||
kernels are selected automatically), or pass the repo id
|
||||
directly to `vllm serve`. NVFP4 requires a Blackwell GPU (compute capability 10.0+).
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/MiniMaxAI/MiniMax-M2)
|
||||
- [NVFP4 variant](https://huggingface.co/RedHatAI/MiniMax-M2-NVFP4)
|
||||
- [MiniMax](https://www.minimax.io/)
|
||||
- [vLLM PR #43410 (automatic MiniMax QK norm fusion)](https://github.com/vllm-project/vllm/pull/43410)
|
||||
@@ -0,0 +1,973 @@
|
||||
meta:
|
||||
title: "MiniMax-M3"
|
||||
slug: "minimax-m3"
|
||||
provider: "MiniMax"
|
||||
description: "MiniMax M3 vision-language MoE (427B total / 26B active) for frontier coding, agent toolchains, and 1M-token reasoning via MSA sparse attention — native multimodal (image + video + computer use); BF16 plus MXFP8, NVIDIA Blackwell NVFP4, and AMD MI355X MXFP4 variants. Runs on NVIDIA (Hopper/Blackwell) and AMD CDNA4/CDNA3."
|
||||
date_added: 2026-06-12
|
||||
date_updated: 2026-10-06
|
||||
tasks:
|
||||
- text
|
||||
- multimodal
|
||||
performance_headline: "Frontier coding and agent (SWE-Bench Pro 59.0, Terminal-Bench 2.1 66.0); MSA sparse attention; 1M context"
|
||||
related_recipes: []
|
||||
hardware:
|
||||
h200: verified
|
||||
b200: verified
|
||||
b300: verified
|
||||
mi300x: verified
|
||||
mi355x: verified
|
||||
|
||||
model:
|
||||
model_id: "MiniMaxAI/MiniMax-M3"
|
||||
# Opt in to SMG gRPC PD; parsing runs on the router, not vLLM.
|
||||
smg_grpc:
|
||||
reasoning_parser: minimax_m3
|
||||
tool_call_parser: minimax_m3
|
||||
min_vllm_version: "0.24.0"
|
||||
nightly_required: true
|
||||
docker_image:
|
||||
nvidia: "vllm/vllm-openai:minimax-m3"
|
||||
amd: "vllm/vllm-openai-rocm:nightly"
|
||||
# Docker-only: MiniMax-M3 support hasn't shipped in a stable wheel, and the
|
||||
# dedicated image is the supported path — hide the pip tab.
|
||||
architecture: moe
|
||||
# 427.04B total from safetensors metadata (BF16: 426.99B + F32: 46.3M).
|
||||
# Active: 57 MoE layers × (4 routed + 1 shared) × 56.6M + 60-layer attn
|
||||
# 6.6B + 3 dense FFN 0.68B + embeddings 2.46B ≈ 25.9B per token.
|
||||
parameter_count: "427B"
|
||||
active_parameters: "26B"
|
||||
# config.max_position_embeddings is 524288; MiniMax officially advertises 1M
|
||||
# via RoPE scaling — using the advertised number here.
|
||||
context_length: 1048576
|
||||
base_args:
|
||||
# MSA's sparse_block_size is 128 — KV cache block size must match.
|
||||
- "--block-size"
|
||||
- "128"
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "MiniMax M3 tool call parser with automatic tool choice"
|
||||
args:
|
||||
- "--tool-call-parser"
|
||||
- "minimax_m3"
|
||||
- "--enable-auto-tool-choice"
|
||||
reasoning:
|
||||
description: "MiniMax M3 reasoning parser (supports `thinking` and `non-thinking` modes, switchable per request)"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "minimax_m3"
|
||||
spec_decoding:
|
||||
description: "Eagle3 speculative decoding with an Inferact MiniMax-M3 draft head — accelerates decoding. Choose between the standard MHA head and the GQA head (16× smaller draft KV cache — better for long-context or high-batch deployments)."
|
||||
default_mode: eagle3
|
||||
modes:
|
||||
eagle3:
|
||||
label: "MHA"
|
||||
description: "Standard Eagle3 draft head (Multi-Head Attention, 64 KV heads)."
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method": "eagle3", "model": "Inferact/MiniMax-M3-EAGLE3", "num_speculative_tokens": 3, "attention_backend": "FLASH_ATTN"}'
|
||||
hardware_overrides:
|
||||
amd:
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method": "eagle3", "model": "Inferact/MiniMax-M3-EAGLE3", "num_speculative_tokens": 3, "attention_backend": "TRITON_ATTN"}'
|
||||
eagle3_gqa:
|
||||
label: "GQA"
|
||||
description: "GQA Eagle3 draft head (Grouped Query Attention, 4 KV heads vs 64) — 16× smaller draft KV cache. Prefer for long-context or large-batch deployments where draft KV memory is the constraint."
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method": "eagle3", "model": "Inferact/MiniMax-M3-EAGLE3-GQA", "num_speculative_tokens": 3, "attention_backend": "FLASH_ATTN"}'
|
||||
hardware_overrides:
|
||||
amd:
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method": "eagle3", "model": "Inferact/MiniMax-M3-EAGLE3-GQA", "num_speculative_tokens": 3, "attention_backend": "TRITON_ATTN"}'
|
||||
text_only:
|
||||
description: "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
|
||||
args:
|
||||
- "--language-model-only"
|
||||
thinking_always_on:
|
||||
description: "Pin `thinking_mode: enabled` server-side — every turn reasons before answering, including after tool results. Recommended for agentic workflows."
|
||||
args:
|
||||
- "--default-chat-template-kwargs"
|
||||
- '{"thinking_mode": "enabled"}'
|
||||
encoder_parallel:
|
||||
description: "Run the vision encoder data-parallel instead of tensor-parallel — avoids TP comm overhead on the small encoder. Also enables the host-shared-memory multimodal processor cache and the per-brand encoder attention backend (FlashInfer on NVIDIA, AITER FlashAttention on AMD). Mutually exclusive with text_only."
|
||||
# No platform-agnostic `args`: this recipe targets only NVIDIA + AMD, both
|
||||
# covered by the per-brand overrides below. (A feature's top-level `args`
|
||||
# is just the fallback used when no hardware_overrides.<gen> matches.)
|
||||
hardware_overrides:
|
||||
# Per-brand vision-encoder config — each block carries shm + tp-mode plus
|
||||
# its attention backend: FlashInfer on NVIDIA, AITER FlashAttention on AMD.
|
||||
nvidia:
|
||||
args:
|
||||
- "--mm-encoder-attn-backend"
|
||||
- "FLASHINFER"
|
||||
- "--mm-processor-cache-type"
|
||||
- "shm"
|
||||
- "--mm-encoder-tp-mode"
|
||||
- "data"
|
||||
amd:
|
||||
args:
|
||||
- "--mm-encoder-attn-backend"
|
||||
- "ROCM_AITER_FA"
|
||||
- "--mm-processor-cache-type"
|
||||
- "shm"
|
||||
- "--mm-encoder-tp-mode"
|
||||
- "data"
|
||||
|
||||
opt_in_features:
|
||||
- spec_decoding
|
||||
- text_only
|
||||
- thinking_always_on
|
||||
- encoder_parallel
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
# 427B × 2 bytes (BF16) + 46M × 4 (F32 norms), × 1.2 headroom ≈ 1025 GB.
|
||||
# 8x H200/H20 (1128 GB) is a tight fit; multi-node TP recommended for
|
||||
# KV-cache headroom at long context.
|
||||
# On AMD, 8x MI350X/MI355X fits comfortably.
|
||||
vram_minimum_gb: 1025
|
||||
description: "Native BF16 checkpoint — multi-node TP recommended for long-context headroom"
|
||||
hardware_overrides:
|
||||
h200:
|
||||
# After ~1 TB of BF16 weights, 8x H200 has ~3.5 GiB/GPU left for KV;
|
||||
# vLLM estimates ~83K max tokens, so the 1M default fails at startup
|
||||
# ("KV cache is needed"). Cap at 64K; use MXFP8 or multi-node for more.
|
||||
extra_args:
|
||||
- "--max-model-len"
|
||||
- "65536"
|
||||
mxfp8:
|
||||
model_id: "MiniMaxAI/MiniMax-M3-MXFP8"
|
||||
precision: mxfp8
|
||||
# 427B × 1 byte (MXFP8) + F32 norms, × 1.2 ≈ 513 GB → fits 8x H200/H20
|
||||
# comfortably, ~4 GPUs for weights alone on Blackwell (B200/B300) or AMD MI350X/MI355X (gfx950)).
|
||||
vram_minimum_gb: 513
|
||||
description: "NVIDIA-quantized MXFP8 weights — Blackwell (B200/B300) for native MX tensor cores, and AMD CDNA4 (MI350X/MI355X, gfx950) for native MXFP8 Matrix Cores."
|
||||
extra_args:
|
||||
- "--attention_config.indexer_kv_dtype"
|
||||
- "fp8"
|
||||
hardware_overrides:
|
||||
blackwell:
|
||||
extra_env:
|
||||
VLLM_FLOAT32_MATMUL_PRECISION: "high"
|
||||
h100:
|
||||
# 8x H100 80GB (640 GB node) fits the ~513 GB MXFP8 weights but not the
|
||||
# full 1M-token KV cache — cap the context to 128K so the engine starts.
|
||||
# H200 (1128 GB node) has the headroom and keeps the full window.
|
||||
extra_args:
|
||||
- "--max-model-len"
|
||||
- "131072"
|
||||
h200:
|
||||
# Humming runs the dense MXFP8 linear layers; the atomic-add env var
|
||||
# enables Marlin's atomic reduction on the MXFP8 MoE path. This image
|
||||
# also supports the FP8 MSA indexer cache on H200.
|
||||
docker_image: "vllm/vllm-openai:nightly-1a001d584244bd21829a5e5bd4a376ea141c0ce3"
|
||||
extra_args:
|
||||
- "--attention-backend"
|
||||
- "FLASH_ATTN"
|
||||
- "--linear-backend"
|
||||
- "humming"
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.95"
|
||||
- "--enable-prefix-caching"
|
||||
extra_env:
|
||||
VLLM_MARLIN_USE_ATOMIC_ADD: "1"
|
||||
mi300x:
|
||||
docker_image: "vllm/vllm-openai-rocm:nightly"
|
||||
extra_args:
|
||||
- "--language-model-only"
|
||||
- "--async-scheduling"
|
||||
- "--max-num-batched-tokens"
|
||||
- "16384"
|
||||
extra_env:
|
||||
VLLM_ENGINE_READY_TIMEOUT_S: "3600"
|
||||
VLLM_USE_BREAKABLE_CUDAGRAPH: "0"
|
||||
VLLM_ROCM_USE_AITER: "1"
|
||||
VLLM_ROCM_USE_AITER_MHA: "0"
|
||||
TORCH_BLAS_PREFER_HIPBLASLT: "1"
|
||||
NCCL_MIN_NCHANNELS: "112"
|
||||
GPU_MAX_HW_QUEUES: "2"
|
||||
mi355x:
|
||||
docker_image: "vllm/vllm-openai-rocm:nightly"
|
||||
extra_args:
|
||||
# run linear-layer GEMMs through the bf16 hipblasLT on MI355X.
|
||||
- "--linear-backend"
|
||||
- "emulation"
|
||||
extra_env:
|
||||
VLLM_ROCM_USE_AITER: "1"
|
||||
VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS: "1"
|
||||
VLLM_ROCM_QUICK_REDUCE_QUANTIZATION: "INT4"
|
||||
nvfp4:
|
||||
model_id: "nvidia/MiniMax-M3-NVFP4"
|
||||
precision: nvfp4
|
||||
vram_minimum_gb: 257
|
||||
description: "NVIDIA-quantized NVFP4 weights (ModelOpt) for Blackwell (B200/B300) — native FP4 tensor cores, ~half the VRAM of MXFP8."
|
||||
extra_args:
|
||||
- "--trust-remote-code"
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
# FlashInfer TRT-LLM attention kernels with an FP8 indexer KV cache.
|
||||
- "--attention_config.backend"
|
||||
- "FLASHINFER"
|
||||
- "--attention_config.use_trtllm_attention"
|
||||
- "true"
|
||||
- "--attention_config.indexer_kv_dtype"
|
||||
- "fp8"
|
||||
- "--attention_config.minimax_m3_msa_decode_backend"
|
||||
- "cutlass"
|
||||
extra_env:
|
||||
VLLM_FLOAT32_MATMUL_PRECISION: "high"
|
||||
VLLM_FLASHINFER_ALLREDUCE_BACKEND: "trtllm"
|
||||
mxfp4:
|
||||
model_id: "amd/MiniMax-M3-MXFP4"
|
||||
precision: mxfp4
|
||||
# 427B × 0.5 bytes (MXFP4) × 1.2 headroom = 256.2 GB.
|
||||
vram_minimum_gb: 257
|
||||
description: "AMD Quark OCP MXFP4 weights and dynamic MXFP4 activations for MI355X (gfx950)."
|
||||
supported_hardware:
|
||||
- mi355x
|
||||
# Match the validated InferenceX single-node baseline. TP8 also leaves
|
||||
# practical headroom for KV cache and the multimodal encoder.
|
||||
tp: 8
|
||||
hardware_overrides:
|
||||
mi355x:
|
||||
docker_image: "vllm/vllm-openai-rocm:nightly"
|
||||
extra_args:
|
||||
- "--trust-remote-code"
|
||||
- "--compilation-config"
|
||||
- '{"cudagraph_mode":"FULL_DECODE_ONLY"}'
|
||||
# Keep the multimodal encoder loaded and run it data-parallel on MI355X.
|
||||
- "--mm-encoder-tp-mode"
|
||||
- "data"
|
||||
- "--mm-encoder-attn-backend"
|
||||
- "ROCM_AITER_FA"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- single_node_tep
|
||||
- single_node_dep
|
||||
- multi_node_tp
|
||||
- multi_node_tp_pp
|
||||
- multi_node_tep
|
||||
- multi_node_dep
|
||||
- pd_cluster
|
||||
|
||||
hardware_overrides:
|
||||
amd:
|
||||
# ROCm MI3xx: MSA sparse attention runs on the Triton backend — this is the
|
||||
# main-model attention and is always needed on AMD. The vision-encoder
|
||||
# flags (data-parallel + AITER FlashAttention) are gated behind the
|
||||
# encoder_parallel feature, mirroring NVIDIA.
|
||||
extra_args:
|
||||
- "--attention-backend"
|
||||
- "TRITON_ATTN"
|
||||
extra_env:
|
||||
# Run MiniMax-M3 with CUDA graphs on AMD MI3xx; avoids the decode
|
||||
# breakable-cudagraph path that otherwise forces eager (per @hongxiayang).
|
||||
VLLM_USE_BREAKABLE_CUDAGRAPH: "0"
|
||||
VLLM_ROCM_USE_AITER_MOE: "1"
|
||||
VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS: "1"
|
||||
|
||||
strategy_overrides:
|
||||
pd_cluster:
|
||||
hardware_overrides:
|
||||
blackwell:
|
||||
extra_args:
|
||||
- "--attention_config.backend"
|
||||
- "FLASHINFER"
|
||||
- "--attention_config.use_trtllm_attention"
|
||||
- "true"
|
||||
- "--attention_config.indexer_kv_dtype"
|
||||
- "fp8"
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[MiniMax-M3](https://huggingface.co/MiniMaxAI/MiniMax-M3) is a frontier
|
||||
vision-language MoE model from [MiniMax](https://www.minimax.io/).
|
||||
|
||||
- **MSA (MiniMax Sparse Attention)** — scalable sparse-attention architecture
|
||||
that lifts the context window to **1M tokens**. MiniMax reports per-token
|
||||
compute at 1M context reduced to ~1/20 of the previous generation, with
|
||||
>9× prefill and >15× decode speedup vs dense baselines.
|
||||
- **Frontier coding and agent capabilities** — SWE-Bench Pro 59.0%,
|
||||
Terminal-Bench 2.1 66.0%, SWE-fficiency 34.8%, KernelBench Hard 28.8%,
|
||||
MCP Atlas 74.2%.
|
||||
- **Native multimodal** — image + video inputs, plus computer-use; trained
|
||||
multimodally from step 0.
|
||||
- **Two reasoning modes** — `thinking` (complex reasoning / agents) and
|
||||
`non-thinking` (latency-sensitive), switchable per request.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **OS:** Linux
|
||||
- **Python:** 3.10 - 3.13
|
||||
- **NVIDIA:** compute capability >= 9.0 (Hopper) recommended; 8x H200 / H20
|
||||
for a tight single-node BF16 fit, or multi-node TP for long-context headroom
|
||||
- **AMD:** MI350X/MI355X (gfx950), MI300X/MI325X (gfx942), ROCm 7.2+. BF16
|
||||
needs TP=8; the MXFP8 variant runs from TP=4.
|
||||
- **`--block-size 128` is mandatory on every platform** (MSA sparse/index cache).
|
||||
|
||||
### Docker (NVIDIA)
|
||||
|
||||
MiniMax-M3 support has not yet shipped in a stable vLLM release — use the
|
||||
dedicated Docker image:
|
||||
|
||||
```bash
|
||||
docker pull vllm/vllm-openai:minimax-m3
|
||||
```
|
||||
|
||||
### Docker (AMD ROCm)
|
||||
|
||||
Use the nightly image, which is the image the AMD variants of this recipe pin:
|
||||
|
||||
```bash
|
||||
docker pull vllm/vllm-openai-rocm:nightly
|
||||
```
|
||||
|
||||
```bash
|
||||
docker run --rm -it --device /dev/kfd --device /dev/dri --group-add video \
|
||||
--cap-add SYS_PTRACE --security-opt seccomp=unconfined --ipc=host \
|
||||
--shm-size=16g -p 8000:8000 \
|
||||
--entrypoint /bin/bash \
|
||||
vllm/vllm-openai-rocm:nightly
|
||||
```
|
||||
|
||||
## Launching the Server
|
||||
|
||||
### NVIDIA — TP8 (8x H200 / H20)
|
||||
|
||||
```bash
|
||||
vllm serve MiniMaxAI/MiniMax-M3 \
|
||||
--tensor-parallel-size 8 \
|
||||
--block-size 128 \
|
||||
--tool-call-parser minimax_m3 \
|
||||
--reasoning-parser minimax_m3 \
|
||||
--enable-auto-tool-choice
|
||||
```
|
||||
|
||||
#### TP8 + Expert Parallel
|
||||
|
||||
```bash
|
||||
vllm serve MiniMaxAI/MiniMax-M3 \
|
||||
--tensor-parallel-size 8 \
|
||||
--enable-expert-parallel \
|
||||
--block-size 128 \
|
||||
--tool-call-parser minimax_m3 \
|
||||
--reasoning-parser minimax_m3 \
|
||||
--enable-auto-tool-choice
|
||||
```
|
||||
|
||||
#### DP8 + Expert Parallel
|
||||
|
||||
```bash
|
||||
vllm serve MiniMaxAI/MiniMax-M3 \
|
||||
--data-parallel-size 8 \
|
||||
--enable-expert-parallel \
|
||||
--block-size 128 \
|
||||
--tool-call-parser minimax_m3 \
|
||||
--reasoning-parser minimax_m3 \
|
||||
--enable-auto-tool-choice
|
||||
```
|
||||
|
||||
### AMD ROCm (MI350X/MI355X (gfx950), MI300X/MI325X (gfx942))
|
||||
|
||||
On AMD MI300X / MI325X / MI355X, run with CUDA graphs and set the following
|
||||
before any of the serve commands below. It avoids the MiniMax-M3 decode breakable-cudagraph path that would
|
||||
otherwise force eager execution (per @hongxiayang):
|
||||
|
||||
```bash
|
||||
export VLLM_USE_BREAKABLE_CUDAGRAPH=0
|
||||
```
|
||||
|
||||
For gfx950: Prefer using the MXFP8 variant `MiniMaxAI/MiniMax-M3-MXFP8` for TP=4 and a smaller
|
||||
footprint. Use TP=8 for lower latency or long context length, or the default bf16 model.
|
||||
|
||||
#### TP8 (Text or Vision)
|
||||
|
||||
```bash
|
||||
vllm serve MiniMaxAI/MiniMax-M3 \
|
||||
--tensor-parallel-size 8 \
|
||||
--block-size 128 \
|
||||
--attention-backend TRITON_ATTN \
|
||||
--mm-encoder-tp-mode data \
|
||||
--mm-encoder-attn-backend ROCM_AITER_FA \
|
||||
--tool-call-parser minimax_m3 \
|
||||
--reasoning-parser minimax_m3 \
|
||||
--enable-auto-tool-choice
|
||||
```
|
||||
|
||||
Add the vision-encoder flags (`--mm-encoder-tp-mode data
|
||||
--mm-encoder-attn-backend ROCM_AITER_FA`) for multimodal serving.
|
||||
|
||||
#### TP8 + `--quantization fp8_per_channel`
|
||||
|
||||
Online **per-channel FP8 (PTPC)** quantization of the BF16 checkpoint halves the
|
||||
weight footprint (≈100 → **≈50 GiB/GPU**, ~1.75× more KV cache) and lifts
|
||||
batched throughput with **gsm8k unchanged from BF16** (lossless);
|
||||
These env vars and the flag are for the **BF16 checkpoint only**.
|
||||
|
||||
```bash
|
||||
export VLLM_USE_BREAKABLE_CUDAGRAPH=0
|
||||
export VLLM_ROCM_USE_AITER=1
|
||||
export VLLM_ROCM_USE_AITER_MOE=0
|
||||
|
||||
vllm serve MiniMaxAI/MiniMax-M3 \
|
||||
--tensor-parallel-size 8 \
|
||||
--block-size 128 \
|
||||
--attention-backend TRITON_ATTN \
|
||||
--quantization fp8_per_channel \
|
||||
--tool-call-parser minimax_m3 \
|
||||
--reasoning-parser minimax_m3 \
|
||||
--enable-auto-tool-choice
|
||||
```
|
||||
|
||||
#### TP8 + Expert Parallel
|
||||
|
||||
```bash
|
||||
vllm serve MiniMaxAI/MiniMax-M3 \
|
||||
--tensor-parallel-size 8 \
|
||||
--enable-expert-parallel \
|
||||
--block-size 128 \
|
||||
--attention-backend TRITON_ATTN \
|
||||
--mm-encoder-tp-mode data \
|
||||
--mm-encoder-attn-backend ROCM_AITER_FA \
|
||||
--tool-call-parser minimax_m3 \
|
||||
--reasoning-parser minimax_m3 \
|
||||
--enable-auto-tool-choice
|
||||
```
|
||||
|
||||
#### DP8 + Expert Parallel
|
||||
|
||||
```bash
|
||||
vllm serve MiniMaxAI/MiniMax-M3 \
|
||||
--data-parallel-size 8 \
|
||||
--enable-expert-parallel \
|
||||
--block-size 128 \
|
||||
--attention-backend TRITON_ATTN \
|
||||
--mm-encoder-tp-mode data \
|
||||
--mm-encoder-attn-backend ROCM_AITER_FA \
|
||||
--tool-call-parser minimax_m3 \
|
||||
--reasoning-parser minimax_m3 \
|
||||
--enable-auto-tool-choice
|
||||
```
|
||||
|
||||
## FP8 KV Cache
|
||||
|
||||
Add `--kv-cache-dtype fp8` to any command for ~1.5× the KV pool — lossless
|
||||
in our testing across the full native context. Especially worth it for high
|
||||
concurrency or long context, where KV is the binding constraint.
|
||||
|
||||
## Context Length & GPU Memory
|
||||
|
||||
The full **1M-token** window (`context_length: 1048576`) needs a large KV
|
||||
cache. To save GPU memory, you can optionally cap the context with
|
||||
`--max-model-len`:
|
||||
|
||||
```bash
|
||||
vllm serve MiniMaxAI/MiniMax-M3 \
|
||||
--tensor-parallel-size 8 \
|
||||
--block-size 128 \
|
||||
--max-model-len 131072 # 128K instead of the full 1M
|
||||
```
|
||||
|
||||
AMD ROCm notes: Native context is 512K. To go past it, supply a YaRN rope_scaling on the text config
|
||||
(a top-level override silently misses the decoder's config) and allow the long max length.
|
||||
TP=8 + fp8 KV is the practical combo at 1M:
|
||||
|
||||
```bash
|
||||
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
|
||||
vllm serve MiniMaxAI/MiniMax-M3 \
|
||||
--block-size 128 \
|
||||
--kv-cache-dtype fp8 \
|
||||
--tensor-parallel-size 8 \
|
||||
--max-model-len 1048576 \
|
||||
--attention-backend TRITON_ATTN \
|
||||
--mm-encoder-tp-mode data \
|
||||
--mm-encoder-attn-backend ROCM_AITER_FA \
|
||||
--tool-call-parser minimax_m3 \
|
||||
--enable-auto-tool-choice \
|
||||
--reasoning-parser minimax_m3 \
|
||||
--hf-overrides '{"text_config":{"rope_scaling":{"rope_type":"yarn","factor":2.0,"original_max_position_embeddings":524288}}}'
|
||||
```
|
||||
|
||||
- Set `--max-model-len` to the longest prompt + output you actually serve
|
||||
(e.g. `32768`, `131072`, `262144`). A smaller window frees KV-pool headroom
|
||||
for higher concurrency and lets the model fit on fewer GPUs; if you need the
|
||||
full 1M window, consider scaling out with multi-node TP instead.
|
||||
|
||||
## Client Usage
|
||||
|
||||
Recommended sampling parameters (from the model card):
|
||||
|
||||
- `temperature = 1.0`
|
||||
- `top_p = 0.95`
|
||||
- `top_k = 40`
|
||||
|
||||
Default system prompt:
|
||||
|
||||
```
|
||||
You are a helpful assistant. Your name is MiniMax-M3 and is built by MiniMax.
|
||||
```
|
||||
|
||||
Example chat request:
|
||||
|
||||
```bash
|
||||
curl http://localhost:8000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "MiniMaxAI/MiniMax-M3",
|
||||
"temperature": 1.0,
|
||||
"top_p": 0.95,
|
||||
"messages": [
|
||||
{"role": "system", "content": "You are a helpful assistant. Your name is MiniMax-M3 and is built by MiniMax."},
|
||||
{"role": "user", "content": "Explain MSA sparse attention in 3 bullets."}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
## Thinking Modes
|
||||
|
||||
M3 reasoning is controlled by the `thinking_mode`, there are three values:
|
||||
|
||||
- **`enabled`** — the model thinks before every response, including after tool results.
|
||||
Use for complex reasoning and agents.
|
||||
- **`disabled`** — no thinking; the model answers directly. Use for
|
||||
latency-sensitive turns.
|
||||
- **`adaptive`** *(default when unset)* — the model decides whether to think
|
||||
based on the task.
|
||||
|
||||
Pass it per request through `chat_template_kwargs`. The same value also tunes
|
||||
the `minimax_m3` reasoning parser, so `reasoning` and `content` are
|
||||
split correctly in every mode.
|
||||
|
||||
```python
|
||||
|
||||
# Start the MiniMax-M3 model by referring to the command above first.
|
||||
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="MiniMaxAI/MiniMax-M3",
|
||||
messages=[{"role": "user", "content": "Prove there are infinitely many primes."}],
|
||||
extra_body={"chat_template_kwargs": {"thinking_mode": "enabled"}},
|
||||
)
|
||||
msg = response.choices[0].message
|
||||
print(msg.reasoning) # the <mm:think> block
|
||||
print(msg.content) # the final answer
|
||||
```
|
||||
|
||||
## Benchmarking
|
||||
|
||||
```bash
|
||||
vllm bench serve \
|
||||
--backend vllm \
|
||||
--model MiniMaxAI/MiniMax-M3 \
|
||||
--endpoint /v1/completions \
|
||||
--dataset-name random \
|
||||
--random-input 2048 \
|
||||
--random-output 1024 \
|
||||
--max-concurrency 10 \
|
||||
--num-prompt 100
|
||||
```
|
||||
|
||||
## Quantized Variant (MXFP8)
|
||||
|
||||
[`MiniMaxAI/MiniMax-M3-MXFP8`](https://huggingface.co/MiniMaxAI/MiniMax-M3-MXFP8)
|
||||
is an MXFP8 checkpoint quantized by NVIDIA from the original FP16 weights —
|
||||
roughly half the VRAM of the BF16 release. Select the **mxfp8** variant above,
|
||||
or pass the repo id directly to `vllm serve`:
|
||||
|
||||
```bash
|
||||
vllm serve MiniMaxAI/MiniMax-M3-MXFP8 \
|
||||
--tensor-parallel-size 8 \
|
||||
--block-size 128 \
|
||||
--tool-call-parser minimax_m3 \
|
||||
--reasoning-parser minimax_m3 \
|
||||
--enable-auto-tool-choice
|
||||
```
|
||||
|
||||
For best MXFP8 throughput, prefer Blackwell (B200/B300) for native MX tensor cores,
|
||||
or AMD CDNA4 (MI350X/MI355X, gfx950) for native MXFP8 matrix cores.
|
||||
|
||||
On AMD MI300X, the **mxfp8** variant stacks two accuracy-safe scheduling
|
||||
levers from the InferenceX MI300X tuning recipe and avoids the high-concurrency
|
||||
expert-parallel regression by using plain TP8:
|
||||
|
||||
```bash
|
||||
export VLLM_ENGINE_READY_TIMEOUT_S=3600
|
||||
export VLLM_USE_BREAKABLE_CUDAGRAPH=0
|
||||
export VLLM_ROCM_USE_AITER=1
|
||||
export VLLM_ROCM_USE_AITER_MHA=0
|
||||
export TORCH_BLAS_PREFER_HIPBLASLT=1
|
||||
export NCCL_MIN_NCHANNELS="${NCCL_MIN_NCHANNELS:-112}"
|
||||
export GPU_MAX_HW_QUEUES="${GPU_MAX_HW_QUEUES:-2}"
|
||||
|
||||
vllm serve MiniMaxAI/MiniMax-M3-MXFP8 \
|
||||
--tensor-parallel-size 8 \
|
||||
--block-size 128 \
|
||||
--language-model-only \
|
||||
--max-model-len "$MAX_MODEL_LEN" \
|
||||
--attention-backend TRITON_ATTN \
|
||||
--async-scheduling \
|
||||
--max-num-batched-tokens 16384 \
|
||||
--tool-call-parser minimax_m3 \
|
||||
--reasoning-parser minimax_m3 \
|
||||
--enable-auto-tool-choice
|
||||
```
|
||||
|
||||
On AMD MI355X the **mxfp8** variant additionally enables the AITER kernels,
|
||||
the fused shared-experts MoE, and INT4 QuickReduce, and pulls the nightly ROCm
|
||||
image (`vllm/vllm-openai-rocm:nightly`) — selected automatically in the command
|
||||
builder. If you launch by hand:
|
||||
|
||||
```bash
|
||||
export VLLM_USE_BREAKABLE_CUDAGRAPH=0
|
||||
export VLLM_ROCM_USE_AITER=1
|
||||
export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1
|
||||
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4
|
||||
|
||||
vllm serve MiniMaxAI/MiniMax-M3-MXFP8 \
|
||||
--tensor-parallel-size 8 \
|
||||
--block-size 128 \
|
||||
--attention-backend TRITON_ATTN \
|
||||
--tool-call-parser minimax_m3 \
|
||||
--reasoning-parser minimax_m3 \
|
||||
--enable-auto-tool-choice
|
||||
```
|
||||
|
||||
### MXFP8 high-concurrency options on MI355X
|
||||
|
||||
The command above is the default serving configuration and is tuned for the
|
||||
general case. **AITER page-16 sparse paged attention** helps specifically at **long context and
|
||||
high concurrency**. It maps MiniMax-M3's top-k 128-token MSA blocks onto AITER page-16 block tables
|
||||
and runs AITER Gluon paged attention over only the selected KV pages.
|
||||
|
||||
The high-concurrency configuration is:
|
||||
|
||||
```bash
|
||||
# Long context + high concurrency only (isl >= 8k, conc >= 64). Regresses
|
||||
# short-context / low-concurrency serving.
|
||||
export VLLM_USE_BREAKABLE_CUDAGRAPH=0
|
||||
export VLLM_ROCM_USE_AITER=1
|
||||
export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1
|
||||
export VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1
|
||||
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4
|
||||
export VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16=0
|
||||
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION_MIN_SIZE_KB=256
|
||||
|
||||
vllm serve MiniMaxAI/MiniMax-M3-MXFP8 \
|
||||
--tensor-parallel-size 4 \
|
||||
--block-size 128 \
|
||||
--language-model-only \
|
||||
--moe-backend aiter \
|
||||
--kv-cache-dtype fp8 \
|
||||
--max-num-batched-tokens 32768 \
|
||||
--attention-backend TRITON_ATTN \
|
||||
--linear-backend emulation \
|
||||
--tool-call-parser minimax_m3 \
|
||||
--reasoning-parser minimax_m3 \
|
||||
--enable-auto-tool-choice
|
||||
```
|
||||
|
||||
Note: Outside the long-context/high-batch
|
||||
regime the shuffled layout costs more than it saves; leave it at `0`:
|
||||
|
||||
```bash
|
||||
export VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=0
|
||||
```
|
||||
|
||||
## Quantized Variant (NVFP4 on Blackwell)
|
||||
|
||||
[`nvidia/MiniMax-M3-NVFP4`](https://huggingface.co/nvidia/MiniMax-M3-NVFP4)
|
||||
is an NVFP4 checkpoint quantized by NVIDIA with ModelOpt. It requires a vLLM
|
||||
build with MiniMax-M3 NVFP4 support (vllm-project/vllm PR #46380).
|
||||
|
||||
Select the **nvfp4** variant and **B200** hardware in the command builder, or
|
||||
launch the tested B200 baseline directly:
|
||||
|
||||
```bash
|
||||
VLLM_FLOAT32_MATMUL_PRECISION=high \
|
||||
vllm serve nvidia/MiniMax-M3-NVFP4 \
|
||||
--tensor-parallel-size 8 \
|
||||
--trust-remote-code \
|
||||
--block-size 128 \
|
||||
--tool-call-parser minimax_m3 \
|
||||
--reasoning-parser minimax_m3 \
|
||||
--enable-auto-tool-choice
|
||||
```
|
||||
|
||||
Pair it with EAGLE3 speculative decoding (the *Spec decoding* feature) for
|
||||
faster decode — the validated B200 benchmark runs the NVFP4 target against the
|
||||
`Inferact/MiniMax-M3-EAGLE3` draft head (3 speculative tokens):
|
||||
|
||||
```bash
|
||||
VLLM_FLOAT32_MATMUL_PRECISION=high \
|
||||
vllm serve nvidia/MiniMax-M3-NVFP4 \
|
||||
--tensor-parallel-size 8 \
|
||||
--trust-remote-code \
|
||||
--block-size 128 \
|
||||
--language-model-only \
|
||||
--speculative-config '{"method": "eagle3", "model": "Inferact/MiniMax-M3-EAGLE3", "num_speculative_tokens": 3, "attention_backend": "FLASH_ATTN"}'
|
||||
```
|
||||
|
||||
For long-context or high-batch workloads where draft KV memory is the binding
|
||||
constraint, swap in the **GQA** variant:
|
||||
|
||||
```bash
|
||||
VLLM_FLOAT32_MATMUL_PRECISION=high \
|
||||
vllm serve nvidia/MiniMax-M3-NVFP4 \
|
||||
--tensor-parallel-size 8 \
|
||||
--trust-remote-code \
|
||||
--block-size 128 \
|
||||
--language-model-only \
|
||||
--speculative-config '{"method": "eagle3", "model": "Inferact/MiniMax-M3-EAGLE3-GQA", "num_speculative_tokens": 3, "attention_backend": "FLASH_ATTN"}'
|
||||
```
|
||||
|
||||
You may also consider FlashInfer for draft attention if it performs better for
|
||||
your workload, and local argmax reduction to reduce communication traffic:
|
||||
|
||||
```bash
|
||||
--speculative-config '{"method":"eagle3","model":"Inferact/MiniMax-M3-EAGLE3-GQA","num_speculative_tokens":3,"attention_backend":"FLASHINFER","use_local_argmax_reduction":true}'
|
||||
```
|
||||
|
||||
## Quantized Variant (MXFP4 on MI355X)
|
||||
|
||||
[`amd/MiniMax-M3-MXFP4`](https://huggingface.co/amd/MiniMax-M3-MXFP4)
|
||||
uses static OCP MXFP4 weights and dynamic OCP MXFP4 activations produced with
|
||||
AMD Quark. It targets AMD Instinct MI355X (gfx950) and requires a ROCm
|
||||
nightly containing the Quark MXFP4 fix from vLLM PR #45794.
|
||||
|
||||
Select the **mxfp4** variant and **MI355X** hardware in the command builder,
|
||||
or launch the tested TP8 baseline directly:
|
||||
|
||||
```bash
|
||||
vllm serve amd/MiniMax-M3-MXFP4 \
|
||||
--tensor-parallel-size 8 \
|
||||
--trust-remote-code \
|
||||
--block-size 128 \
|
||||
--attention-backend TRITON_ATTN \
|
||||
--mm-encoder-tp-mode data \
|
||||
--mm-encoder-attn-backend ROCM_AITER_FA \
|
||||
--tool-call-parser minimax_m3 \
|
||||
--enable-auto-tool-choice \
|
||||
--reasoning-parser minimax_m3
|
||||
```
|
||||
|
||||
### MXFP4 benchmark reproduction (InferenceX MI355X sweep)
|
||||
|
||||
The command above is the default serving configuration. The
|
||||
[SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) MI355X
|
||||
agentic-coding lane for this checkpoint is a **separate benchmark
|
||||
configuration** — single-node, text-only, GPU-resident, and driven by the GQA
|
||||
EAGLE3 draft head. It is not the recipe default and not a general serving
|
||||
recommendation.
|
||||
|
||||
The lane is the `minimaxm3-fp4-mi355x-vllm-agentic-mtp` entry in
|
||||
`configs/amd-master.yaml`, launched by
|
||||
`benchmarks/single_node/agentic/minimaxm3_fp4_mi355x_mtp.sh` in
|
||||
[SemiAnalysisAI/InferenceX](https://github.com/SemiAnalysisAI/InferenceX):
|
||||
|
||||
| Setting | Benchmark lane |
|
||||
|---------|----------------|
|
||||
| Served checkpoint | `amd/MiniMax-M3-MXFP4` |
|
||||
| Hardware | one MI355X node |
|
||||
| Scenario | agentic-coding trace replay |
|
||||
| Parallelism | TP4 and TP2 — no expert parallelism, no KV offload |
|
||||
| TP4 concurrency sweep | 1, 4, 5, 8, 10, 12, 15, 20, 24, 32 |
|
||||
| TP2 concurrency sweep | 1, 2 |
|
||||
| Draft head | `Inferact/MiniMax-M3-EAGLE3-GQA`, 3 speculative tokens |
|
||||
| Attention | `ROCM_AITER_UNIFIED_ATTN` for both the target and the draft |
|
||||
| Indexer K cache | `fp8` (`--attention-config`); MiniMax-M3's default is bf16 |
|
||||
| Model path | text-only (`--language-model-only`), prefix caching enabled |
|
||||
|
||||
Differences from the default command above: TP4/TP2 instead of TP8, the
|
||||
text-only language-model path, `--kv-cache-dtype fp8`, the FP8 sparse-attention
|
||||
indexer K cache, the AITER MXFP4 MoE backend, unified AITER attention instead
|
||||
of `TRITON_ATTN`, the GQA EAGLE3 draft, and a hand-tuned decode
|
||||
`cudagraph_capture_sizes` list sized for the lane's `--max-num-seqs` range. The
|
||||
lane also pins thinking server-side with `--default-chat-template-kwargs`, and
|
||||
it passes no `--hf-overrides` — the pinned image supplies the sparse-PA
|
||||
index-cache and top-k refresh behavior itself, and re-adding the override
|
||||
would rewrite the served model config.
|
||||
|
||||
Run it on a current official `vllm/vllm-openai-rocm:nightly`. The lane pins one
|
||||
exact nightly digest rather than a release tag: the AITER shuffled KV-cache
|
||||
layout that MiniMax-M3 sparse paged attention relies on was changed by the
|
||||
KV-cache layout standardization in
|
||||
[vllm-project/vllm#51718](https://github.com/vllm-project/vllm/pull/51718),
|
||||
and the nightly restores it. The lane's pinned digest also carries the AITER
|
||||
indexer scoring and top-k kernels for the MiniMax-M3 sparse-attention path from
|
||||
[vllm-project/vllm#52664](https://github.com/vllm-project/vllm/pull/52664),
|
||||
which is what makes the FP8 indexer K cache and the unconditional shuffled
|
||||
layout below worthwhile. Don't hard-code a transient tag here — the digest
|
||||
validated for these numbers lives in the InferenceX config.
|
||||
|
||||
#### Environment
|
||||
|
||||
```bash
|
||||
# TP4 and TP2 are the only topologies this lane validates.
|
||||
TP_SIZE="${TP_SIZE:-4}"
|
||||
CONC="${CONC:-32}"
|
||||
if [ "$TP_SIZE" -ne 4 ] && [ "$TP_SIZE" -ne 2 ]; then
|
||||
echo "TP_SIZE must be 2 or 4 for this benchmark lane" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Shuffled KV-cache layout, now unconditional. Earlier digests needed the
|
||||
# page-16 sparse-attention fast path's one-KV-head-per-rank property, so TP2
|
||||
# (two of MiniMax-M3's four KV heads per rank) had to fall back to 0; the
|
||||
# pinned digest's AITER indexer kernels serve both widths from the shuffled
|
||||
# layout, and the lane runs TP4 and TP2 with it enabled.
|
||||
export VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1
|
||||
|
||||
# Kernel selection — required to match the benchmarked MoE path.
|
||||
# Without these (and `--moe-backend aiter` on the serve command), vLLM's MXFP4
|
||||
# MoE oracle selects the TRITON_UNFUSED fallback on gfx950 instead of the
|
||||
# AITER_MXFP4_MXFP4 kernel the InferenceX lane actually runs.
|
||||
export VLLM_ENGINE_READY_TIMEOUT_S=3600
|
||||
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800
|
||||
export VLLM_USE_BREAKABLE_CUDAGRAPH=0
|
||||
export VLLM_ROCM_USE_AITER=1
|
||||
export VLLM_ROCM_USE_AITER_MOE=1
|
||||
export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1
|
||||
|
||||
# Benchmark-only — accuracy-sensitive; NOT part of the default command.
|
||||
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4
|
||||
export VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16=0
|
||||
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION_MIN_SIZE_KB=256
|
||||
```
|
||||
|
||||
#### Serving and evaluation (real target verification)
|
||||
|
||||
This is the form to use for the lane's accuracy evaluation and for any normal
|
||||
serving. The draft's proposals are verified against the real target model:
|
||||
|
||||
```bash
|
||||
vllm serve amd/MiniMax-M3-MXFP4 \
|
||||
--port "${PORT:-8000}" \
|
||||
--tensor-parallel-size "$TP_SIZE" \
|
||||
--trust-remote-code \
|
||||
--block-size 128 \
|
||||
--gpu-memory-utilization 0.90 \
|
||||
--enable-chunked-prefill \
|
||||
--max-num-batched-tokens 32768 \
|
||||
--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY", "cudagraph_capture_sizes": [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,20,22,24,26,28,30,32,34,36,48,64,72,80,88,96,104,112,120,128]}' \
|
||||
--language-model-only \
|
||||
--enable-prefix-caching \
|
||||
--attention-backend ROCM_AITER_UNIFIED_ATTN \
|
||||
--moe-backend aiter \
|
||||
--kv-cache-dtype fp8 \
|
||||
--attention-config '{"indexer_kv_dtype": "fp8"}' \
|
||||
--tool-call-parser minimax_m3 \
|
||||
--reasoning-parser minimax_m3 \
|
||||
--enable-auto-tool-choice \
|
||||
--default-chat-template-kwargs '{"thinking_mode":"enabled"}' \
|
||||
--max-num-seqs "$((2 * CONC))" \
|
||||
--stream-interval 20 \
|
||||
--speculative-config '{"method": "eagle3", "model": "Inferact/MiniMax-M3-EAGLE3-GQA", "num_speculative_tokens": 3, "attention_backend": "ROCM_AITER_UNIFIED_ATTN"}'
|
||||
```
|
||||
|
||||
#### Throughput measurement only (synthetic acceptance)
|
||||
|
||||
The throughput arm of the sweep replaces only the `--speculative-config` with
|
||||
a synthetic rejection sampler so that every concurrency point is measured
|
||||
against the same acceptance rate:
|
||||
|
||||
```bash
|
||||
--speculative-config '{"method": "eagle3", "model": "Inferact/MiniMax-M3-EAGLE3-GQA", "num_speculative_tokens": 3, "attention_backend": "ROCM_AITER_UNIFIED_ATTN", "rejection_sample_method": "synthetic", "synthetic_acceptance_length": 2.78}'
|
||||
```
|
||||
|
||||
`2.78` is **not** a tuning knob and **not** a production shortcut. It is the
|
||||
committed golden EAGLE3-GQA acceptance length for MiniMax-M3 with thinking
|
||||
enabled at 3 speculative tokens, taken from
|
||||
[`golden_al_distribution/minimaxm3_eagle3_gqa.yaml`](https://github.com/SemiAnalysisAI/InferenceX/blob/main/golden_al_distribution/minimaxm3_eagle3_gqa.yaml).
|
||||
Synthetic rejection standardizes throughput comparisons; it bypasses real
|
||||
verification, so it must never be used for accuracy evaluation or for serving
|
||||
users. Use the real-verification form above for everything else.
|
||||
|
||||
Selecting the AITER MXFP4 MoE backend (`--moe-backend aiter`) is a
|
||||
kernel-selection choice (same MXFP4 math, faster kernel), not a precision
|
||||
reduction. The INT4 all-reduce, the fp8 KV cache, and the fp8 indexer K cache
|
||||
are throughput optimizations for high concurrency and are **kept out of the
|
||||
default command pending accuracy validation** — `--attention-config
|
||||
'{"indexer_kv_dtype": "fp8"}'` quantizes the sparse-attention indexer's K
|
||||
cache away from MiniMax-M3's bf16 default and depends on the indexer kernels
|
||||
in the pinned digest. The raised `--gpu-memory-utilization 0.90` and the
|
||||
explicit `cudagraph_capture_sizes` list are likewise lane-specific: the list
|
||||
is sized for this sweep's `--max-num-seqs` range (`2 × concurrency`, so at
|
||||
most 64 here) and capping graph capture at 128 would shrink CUDA-graph
|
||||
coverage on a general deployment that serves wider batches. Performance
|
||||
figures on the InferenceX dashboard are **reported externally by SemiAnalysis
|
||||
and have not been reproduced in this repository.**
|
||||
|
||||
#### References for this benchmark lane
|
||||
|
||||
- Current source PR: [SemiAnalysisAI/InferenceX#2959](https://github.com/SemiAnalysisAI/InferenceX/pull/2959)
|
||||
(image bump, fp8 indexer K cache, tuned decode graphs) — reviewed at head
|
||||
`43c23db28f400f643f79dc490699910a010d3360`
|
||||
- Full sweep run for that lane: [actions/runs/34609443618](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34609443618)
|
||||
— all TP4 and TP2 throughput and real-verification eval jobs passed on
|
||||
commit `c738b5030e2277cc701ab15e115cb6584e0cacd0`, whose lane script and
|
||||
config entry are byte-identical to the reviewed head
|
||||
- Original lane PR: [SemiAnalysisAI/InferenceX#2781](https://github.com/SemiAnalysisAI/InferenceX/pull/2781)
|
||||
([sweep](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/33239952571))
|
||||
- Golden GQA acceptance-length curve: [`minimaxm3_eagle3_gqa.yaml`](https://github.com/SemiAnalysisAI/InferenceX/blob/main/golden_al_distribution/minimaxm3_eagle3_gqa.yaml)
|
||||
- KV-cache layout change the image pin compensates for: [vllm-project/vllm#51718](https://github.com/vllm-project/vllm/pull/51718)
|
||||
- AITER indexer scoring and top-k kernels the pinned digest adds: [vllm-project/vllm#52664](https://github.com/vllm-project/vllm/pull/52664)
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **`--block-size` mismatch.** MSA's sparse block size is 128; the vLLM KV
|
||||
cache block size must match. Using the default (16) misaligns the sparse
|
||||
attention indexing (on AMD it crashes with `No common block size for 16`).
|
||||
- **Parsers.** `--tool-call-parser` and `--reasoning-parser` both use
|
||||
`minimax_m3` — distinct from `minimax_m2` used by earlier releases.
|
||||
- **Long context KV cache.** See *Context Length & GPU Memory* above — cap
|
||||
`--max-model-len` or scale to multi-node TP if you OOM.
|
||||
- **Vision encoder.** The encoder is small, so at high TP the *Encoder
|
||||
Parallel* option runs it data-parallel (`--mm-encoder-tp-mode data`) to
|
||||
avoid TP comm overhead; it also turns on the vision-encoder attention
|
||||
backend (FlashInfer on NVIDIA, `--mm-encoder-attn-backend FLASHINFER`;
|
||||
AITER FlashAttention on AMD, `ROCM_AITER_FA`) and the host-shared-memory
|
||||
multimodal processor cache (`--mm-processor-cache-type shm`). For **text-only** workloads enable *Text only*
|
||||
(`--language-model-only`) to skip loading the encoder and free VRAM — it is
|
||||
mutually exclusive with Encoder Parallel.
|
||||
|
||||
|
||||
## EAGLE3 Speculative Decoding
|
||||
|
||||
Two Inferact draft heads are available as selectable modes in the *Spec decoding*
|
||||
feature — both use `method: eagle3` with 3 speculative tokens and share
|
||||
MiniMax-M3's embedding table and LM head:
|
||||
|
||||
| Mode | Repo | KV heads | Draft KV cache |
|
||||
|------|------|----------|----------------|
|
||||
| **MHA** (default) | `Inferact/MiniMax-M3-EAGLE3` | 64 | baseline |
|
||||
| **GQA** | `Inferact/MiniMax-M3-EAGLE3-GQA` | 4 | **16× smaller** |
|
||||
|
||||
The GQA head uses Grouped Query Attention (4 KV heads grouped from 64 query
|
||||
heads) — the only architectural difference from the MHA head. The 16× reduction
|
||||
in draft KV state makes it preferable when draft memory is the binding constraint
|
||||
(long-context serving or large batch sizes).
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/MiniMaxAI/MiniMax-M3)
|
||||
- [MXFP8 variant](https://huggingface.co/MiniMaxAI/MiniMax-M3-MXFP8)
|
||||
- [NVFP4 variant](https://huggingface.co/nvidia/MiniMax-M3-NVFP4)
|
||||
- [AMD MXFP4 variant](https://huggingface.co/amd/MiniMax-M3-MXFP4)
|
||||
- [EAGLE3 draft head (MHA)](https://huggingface.co/Inferact/MiniMax-M3-EAGLE3)
|
||||
- [EAGLE3 draft head (GQA)](https://huggingface.co/Inferact/MiniMax-M3-EAGLE3-GQA)
|
||||
- [MiniMax](https://www.minimax.io/)
|
||||
- [MiniMax Agent](https://agent.minimax.io/)
|
||||
- [MiniMax Platform](https://platform.minimax.io/)
|
||||
- [Previous release: MiniMax-M2.7](https://huggingface.co/MiniMaxAI/MiniMax-M2.7)
|
||||
@@ -0,0 +1,143 @@
|
||||
meta:
|
||||
title: "InternVL3.5"
|
||||
slug: "internvl3.5"
|
||||
provider: "InternVL (OpenGVLab)"
|
||||
description: "InternVL 3.5 vision-language models from Shanghai AI Lab with thinking-mode prompting"
|
||||
date_added: 2025-09-20
|
||||
date_updated: 2026-04-17
|
||||
tasks:
|
||||
- multimodal
|
||||
related_recipes:
|
||||
- internlm/Intern-S1
|
||||
hardware:
|
||||
h100: verified
|
||||
mi300x: verified
|
||||
mi325x: verified
|
||||
mi355x: verified
|
||||
|
||||
model:
|
||||
model_id: "OpenGVLab/InternVL3_5-8B"
|
||||
min_vllm_version: "0.10.0"
|
||||
architecture: dense
|
||||
parameter_count: "8B"
|
||||
active_parameters: "8B"
|
||||
context_length: 40960
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
text_only:
|
||||
description: "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
|
||||
args:
|
||||
- "--language-model-only"
|
||||
encoder_parallel:
|
||||
description: "Run the vision encoder in data-parallel mode — avoids TP comm overhead on the small encoder. Mutually exclusive with text_only."
|
||||
args:
|
||||
- "--mm-encoder-tp-mode"
|
||||
- "data"
|
||||
|
||||
opt_in_features:
|
||||
- text_only
|
||||
- encoder_parallel
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 19
|
||||
description: "BF16 weights for the 8B variant"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
|
||||
kv_offload_support:
|
||||
offloading_cpu: verified
|
||||
offloading_fs: verified
|
||||
|
||||
hardware_overrides: {}
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[InternVL3.5](https://github.com/OpenGVLab/InternVL) is a vision-language model developed
|
||||
by Shanghai AI Laboratory. It supports single-image and multi-image prompts, plus an
|
||||
optional "thinking mode" via a custom system prompt.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Hardware: 1x GPU with >=20 GB VRAM (A100, L40S, H100, etc.)
|
||||
- vLLM >= 0.10.0
|
||||
|
||||
### Install vLLM (CUDA)
|
||||
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend auto
|
||||
```
|
||||
|
||||
### Install vLLM (AMD ROCm MI300X/MI325X/MI355X)
|
||||
|
||||
```bash
|
||||
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.14.1/rocm700
|
||||
```
|
||||
|
||||
## Launch command
|
||||
|
||||
```bash
|
||||
vllm serve OpenGVLab/InternVL3_5-8B --trust-remote-code
|
||||
```
|
||||
|
||||
On AMD:
|
||||
|
||||
```bash
|
||||
export VLLM_ROCM_USE_AITER=1
|
||||
vllm serve OpenGVLab/InternVL3_5-8B --trust-remote-code
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
Single image:
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
client = OpenAI(api_key="", base_url="http://0.0.0.0:8000/v1")
|
||||
model_name = client.models.list().data[0].id
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model=model_name,
|
||||
messages=[{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "Describe the image."},
|
||||
{"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg"}},
|
||||
],
|
||||
}],
|
||||
temperature=0.0,
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
## Thinking Mode
|
||||
|
||||
Set a thinking system prompt and use `temperature=0.6` to mitigate repetition:
|
||||
|
||||
```python
|
||||
THINKING_SYSTEM_PROMPT = """
|
||||
You are an AI assistant that rigorously follows this response protocol:
|
||||
|
||||
1. First, conduct a detailed analysis of the question. Consider different angles, potential
|
||||
solutions, and reason through the problem step-by-step. Enclose this entire thinking process
|
||||
within <think> and </think> tags.
|
||||
|
||||
2. After the thinking section, provide a clear, concise, and direct answer to the user's
|
||||
question. Separate the answer from the think section with a newline.
|
||||
""".strip()
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [InternVL3.5-8B](https://huggingface.co/OpenGVLab/InternVL3_5-8B)
|
||||
- [InternVL GitHub](https://github.com/OpenGVLab/InternVL)
|
||||
@@ -0,0 +1,179 @@
|
||||
meta:
|
||||
title: "MOSS-Transcribe-Diarize"
|
||||
slug: "moss-transcribe-diarize"
|
||||
provider: "OpenMOSS"
|
||||
description: "OpenMOSS's 0.9B end-to-end multi-speaker long-audio transcription model with timestamps and speaker labels, served through vLLM's OpenAI-compatible /v1/audio/transcriptions API."
|
||||
date_added: 2026-07-08
|
||||
date_updated: 2026-08-10
|
||||
tasks:
|
||||
- multimodal
|
||||
performance_headline: "0.9B end-to-end transcription with speaker labels and timestamps"
|
||||
related_recipes:
|
||||
- OpenMOSS-Team/MOSS-TTS
|
||||
|
||||
model:
|
||||
model_id: "OpenMOSS-Team/MOSS-Transcribe-Diarize"
|
||||
min_vllm_version: "0.27.0"
|
||||
architecture: dense
|
||||
parameter_count: "0.9B"
|
||||
active_parameters: "0.9B"
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
base_env: {}
|
||||
|
||||
dependencies:
|
||||
- note: "Audio extras are required for loading and preprocessing audio files."
|
||||
command: 'uv pip install -U "vllm[audio]" --torch-backend auto'
|
||||
|
||||
features: {}
|
||||
|
||||
opt_in_features: []
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 4
|
||||
description: "MOSS-Transcribe-Diarize-0.9B - single-GPU long-form multi-speaker transcription"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
|
||||
hardware_overrides: {}
|
||||
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[MOSS-Transcribe-Diarize](https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize)
|
||||
is a 0.9B end-to-end speech-to-text model for long-form multi-speaker
|
||||
transcription. It uses a Whisper-style audio encoder and a Qwen3-style causal
|
||||
decoder to generate structured transcripts with text, timestamps, and speaker
|
||||
labels in a single pass.
|
||||
|
||||
Example output:
|
||||
|
||||
```text
|
||||
[0.11][S01] Good morning! [1.03]
|
||||
[1.11][S02] Morning, let's start the meeting. [2.42]
|
||||
```
|
||||
|
||||
Key capabilities:
|
||||
|
||||
- End-to-end multi-speaker transcription without a separate ASR, diarization,
|
||||
or alignment pipeline.
|
||||
- Structured text output with speaker IDs and timestamps.
|
||||
- Long-audio transcription for meetings, interviews, podcasts, and customer
|
||||
support recordings.
|
||||
- Prompt-based hotwords for names, organizations, product names, project
|
||||
codes, and domain vocabulary.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Install vLLM with audio dependencies.
|
||||
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
|
||||
uv pip install -U "vllm[audio]" --torch-backend auto
|
||||
```
|
||||
|
||||
## Launching with vLLM
|
||||
|
||||
```bash
|
||||
vllm serve OpenMOSS-Team/MOSS-Transcribe-Diarize \
|
||||
--trust-remote-code \
|
||||
--served-model-name moss-mtd
|
||||
```
|
||||
|
||||
The model uses custom Hugging Face processor code, so `--trust-remote-code` is
|
||||
required.
|
||||
|
||||
## Client Usage
|
||||
|
||||
### OpenAI Transcription API
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||||
|
||||
with open("meeting.wav", "rb") as audio_file:
|
||||
transcription = client.audio.transcriptions.create(
|
||||
model="moss-mtd",
|
||||
file=audio_file,
|
||||
response_format="diarized_json",
|
||||
temperature=0,
|
||||
)
|
||||
|
||||
print(transcription.text)
|
||||
|
||||
for segment in transcription.segments:
|
||||
print(
|
||||
f"[{segment.start:.2f} - {segment.end:.2f}] "
|
||||
f"{segment.speaker}: {segment.text}"
|
||||
)
|
||||
```
|
||||
|
||||
### cURL
|
||||
|
||||
```bash
|
||||
curl http://localhost:8000/v1/audio/transcriptions \
|
||||
-F "model=moss-mtd" \
|
||||
-F "file=@meeting.wav" \
|
||||
-F "response_format=diarized_json" \
|
||||
-F "temperature=0"
|
||||
```
|
||||
|
||||
## Output Format
|
||||
|
||||
With `response_format=diarized_json`, vLLM returns an OpenAI-compatible
|
||||
response with structured speaker segments:
|
||||
|
||||
```json
|
||||
{
|
||||
"task": "transcribe",
|
||||
"duration": 5.1,
|
||||
"text": "Let's review the deployment plan. The vLLM server is already running.",
|
||||
"segments": [
|
||||
{
|
||||
"type": "transcript.text.segment",
|
||||
"id": "seg_0",
|
||||
"start": 0.41,
|
||||
"end": 2.8,
|
||||
"text": "Let's review the deployment plan.",
|
||||
"speaker": "S01"
|
||||
},
|
||||
{
|
||||
"type": "transcript.text.segment",
|
||||
"id": "seg_1",
|
||||
"start": 3.12,
|
||||
"end": 5.1,
|
||||
"text": "The vLLM server is already running.",
|
||||
"speaker": "S02"
|
||||
}
|
||||
],
|
||||
"usage": {
|
||||
"type": "duration",
|
||||
"seconds": 6
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Speaker IDs such as `S01`, `S02`, and `S03` are generated by the model and are
|
||||
intended to remain consistent within the audio.
|
||||
|
||||
Use `response_format=json` if you want the model's native timestamped
|
||||
transcript instead.
|
||||
|
||||
```text
|
||||
[0.41][S01] Let's review the deployment plan. [2.80]
|
||||
[3.12][S02] The vLLM server is already running. [5.10]
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize)
|
||||
- [GitHub repository](https://github.com/OpenMOSS/MOSS-Transcribe-Diarize)
|
||||
- [vLLM audio support](https://docs.vllm.ai/en/latest/features/multimodal_inputs.html)
|
||||
@@ -0,0 +1,204 @@
|
||||
meta:
|
||||
title: "PaddleOCR-VL-1.5"
|
||||
slug: "paddleocr-vl-1.5"
|
||||
provider: "PaddlePaddle"
|
||||
description: "PaddleOCR-VL-1.5 (0.9B) — next-gen compact VLM for document parsing; adds text spotting, seal recognition, and Tibetan/Bengali"
|
||||
date_added: 2026-05-11
|
||||
date_updated: 2026-05-11
|
||||
tasks:
|
||||
- multimodal
|
||||
related_recipes:
|
||||
- "PaddlePaddle/PaddleOCR-VL"
|
||||
|
||||
model:
|
||||
model_id: "PaddlePaddle/PaddleOCR-VL-1.5"
|
||||
min_vllm_version: "0.11.1"
|
||||
architecture: dense
|
||||
parameter_count: "0.9B"
|
||||
active_parameters: "0.9B"
|
||||
context_length: 131072
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
- "--max-num-batched-tokens"
|
||||
- "16384"
|
||||
- "--no-enable-prefix-caching"
|
||||
- "--mm-processor-cache-gb"
|
||||
- "0"
|
||||
base_env: {}
|
||||
|
||||
dependencies:
|
||||
- note: "PaddlePaddle runtime (install in a separate venv from vllm to avoid conflicts)"
|
||||
command: "uv pip install paddlepaddle-gpu==3.2.1 --extra-index-url https://www.paddlepaddle.org.cn/packages/stable/cu126/"
|
||||
- note: "PaddleOCR document-parsing helpers (1.5 ships under the same paddleocr[doc-parser] extra)"
|
||||
command: 'uv pip install -U "paddleocr[doc-parser]"'
|
||||
- note: "Safetensors loader used by the doc-parser path"
|
||||
command: "uv pip install safetensors"
|
||||
|
||||
features:
|
||||
text_only:
|
||||
description: "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
|
||||
args:
|
||||
- "--language-model-only"
|
||||
encoder_parallel:
|
||||
description: "Run the vision encoder in data-parallel mode — avoids TP comm overhead on the small encoder. Mutually exclusive with text_only."
|
||||
args:
|
||||
- "--mm-encoder-tp-mode"
|
||||
- "data"
|
||||
|
||||
opt_in_features:
|
||||
- text_only
|
||||
- encoder_parallel
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 2
|
||||
description: "BF16 weights — small footprint, runs on most GPUs"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
|
||||
kv_offload_support:
|
||||
offloading_cpu: verified
|
||||
offloading_fs: verified
|
||||
|
||||
hardware_overrides:
|
||||
amd:
|
||||
extra_args: []
|
||||
extra_env:
|
||||
SAFETENSORS_FAST_GPU: "1"
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[PaddleOCR-VL-1.5](https://huggingface.co/PaddlePaddle/PaddleOCR-VL-1.5) is the next
|
||||
generation of PaddleOCR-VL — same 0.9B architecture (NaViT-style dynamic-resolution
|
||||
vision encoder + ERNIE-4.5-0.3B LM), with substantial accuracy gains and new tasks:
|
||||
|
||||
- **94.5% on OmniDocBench v1.5** (SOTA)
|
||||
- **Text spotting** — line-level localization + recognition in one pass
|
||||
- **Seal recognition** — new task with SOTA results
|
||||
- **Irregular-shape localization** — polygonal detection under skew/warping
|
||||
- **Multilingual** — adds Tibetan and Bengali
|
||||
- Cross-page table merging and paragraph-heading recognition
|
||||
|
||||
Architecture is identical to PaddleOCR-VL, so the vLLM launch command and feature
|
||||
flags are the same.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Hardware: 1x GPU (small VRAM footprint)
|
||||
- vLLM >= 0.11.1 (nightly if not released yet)
|
||||
- AMD ROCm wheels require Python 3.12, ROCm 7.2.3, and glibc >= 2.35.
|
||||
|
||||
### Install vLLM (NVIDIA)
|
||||
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --pre \
|
||||
--extra-index-url https://wheels.vllm.ai/nightly \
|
||||
--extra-index-url https://download.pytorch.org/whl/cu129 \
|
||||
--index-strategy unsafe-best-match
|
||||
```
|
||||
|
||||
### Install vLLM (AMD ROCm)
|
||||
|
||||
```bash
|
||||
uv venv --python 3.12
|
||||
source .venv/bin/activate
|
||||
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm
|
||||
```
|
||||
|
||||
## Launch command
|
||||
|
||||
```bash
|
||||
vllm serve PaddlePaddle/PaddleOCR-VL-1.5 \
|
||||
--trust-remote-code \
|
||||
--max-num-batched-tokens 16384 \
|
||||
--no-enable-prefix-caching \
|
||||
--mm-processor-cache-gb 0
|
||||
```
|
||||
|
||||
AMD MI300X/MI325X/MI355X:
|
||||
|
||||
```bash
|
||||
export SAFETENSORS_FAST_GPU=1
|
||||
|
||||
vllm serve PaddlePaddle/PaddleOCR-VL-1.5 \
|
||||
--trust-remote-code \
|
||||
--max-num-batched-tokens 16384 \
|
||||
--mm-processor-cache-gb 0
|
||||
```
|
||||
|
||||
Tip: OCR workloads don't benefit much from prefix caching or image reuse, so disable
|
||||
those to avoid hashing/caching overhead.
|
||||
|
||||
## Client Usage
|
||||
|
||||
Task-specific prompts (note the two new tasks `spotting` and `seal`):
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600)
|
||||
|
||||
TASKS = {
|
||||
"ocr": "OCR:",
|
||||
"table": "Table Recognition:",
|
||||
"formula": "Formula Recognition:",
|
||||
"chart": "Chart Recognition:",
|
||||
"spotting": "Spotting:",
|
||||
"seal": "Seal Recognition:",
|
||||
}
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="PaddlePaddle/PaddleOCR-VL-1.5",
|
||||
messages=[{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "image_url", "image_url": {"url": "https://.../receipt.png"}},
|
||||
{"type": "text", "text": TASKS["spotting"]},
|
||||
],
|
||||
}],
|
||||
temperature=0.0,
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
## Offline Inference with PP-DocLayoutV2
|
||||
|
||||
Use separate venvs for `vllm` and `paddlepaddle` to avoid conflicts. If you see
|
||||
"The model PaddleOCR-VL-1.5-0.9B does not exist.", add
|
||||
`--served-model-name PaddleOCR-VL-1.5-0.9B`.
|
||||
|
||||
```bash
|
||||
uv pip install paddlepaddle-gpu==3.2.1 --extra-index-url https://www.paddlepaddle.org.cn/packages/stable/cu126/
|
||||
uv pip install -U "paddleocr[doc-parser]"
|
||||
uv pip install safetensors
|
||||
```
|
||||
|
||||
```python
|
||||
from paddleocr import PaddleOCRVL
|
||||
|
||||
pipeline = PaddleOCRVL(
|
||||
vl_rec_model_name="PaddleOCR-VL-1.5-0.9B",
|
||||
vl_rec_backend="vllm-server",
|
||||
vl_rec_server_url="http://localhost:8000/v1",
|
||||
layout_detection_model_name="PP-DocLayoutV2",
|
||||
layout_detection_model_dir="/path/to/your/PP-DocLayoutV2/",
|
||||
)
|
||||
|
||||
output = pipeline.predict("https://paddle-model-ecology.bj.bcebos.com/paddlex/imgs/demo_image/paddleocr_vl_demo.png")
|
||||
for i, res in enumerate(output):
|
||||
res.save_to_json(save_path=f"output_{i}.json")
|
||||
res.save_to_markdown(save_path=f"output_{i}.md")
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [PaddleOCR-VL-1.5 on Hugging Face](https://huggingface.co/PaddlePaddle/PaddleOCR-VL-1.5)
|
||||
- [PaddleOCR GitHub](https://github.com/PaddlePaddle/PaddleOCR)
|
||||
- [vLLM support tracking issue](https://github.com/vllm-project/vllm/issues/33554)
|
||||
@@ -0,0 +1,187 @@
|
||||
meta:
|
||||
title: "PaddleOCR-VL-1.6"
|
||||
slug: "paddleocr-vl-1.6"
|
||||
provider: "PaddlePaddle"
|
||||
description: "PaddleOCR-VL-1.6 (0.9B) — region-aware data optimization + progressive post-training; new SOTA 96.33% on OmniDocBench v1.6, drop-in replacement for 1.5"
|
||||
date_added: 2026-07-11
|
||||
date_updated: 2026-07-08
|
||||
tasks:
|
||||
- multimodal
|
||||
related_recipes:
|
||||
- "PaddlePaddle/PaddleOCR-VL-1.5"
|
||||
- "PaddlePaddle/PaddleOCR-VL"
|
||||
|
||||
model:
|
||||
model_id: "PaddlePaddle/PaddleOCR-VL-1.6"
|
||||
min_vllm_version: "0.11.1"
|
||||
architecture: dense
|
||||
parameter_count: "0.9B"
|
||||
active_parameters: "0.9B"
|
||||
context_length: 131072
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
- "--max-num-batched-tokens"
|
||||
- "16384"
|
||||
- "--no-enable-prefix-caching"
|
||||
- "--mm-processor-cache-gb"
|
||||
- "0"
|
||||
base_env: {}
|
||||
|
||||
dependencies:
|
||||
- note: "PaddlePaddle runtime (install in a separate venv from vllm to avoid conflicts)"
|
||||
command: "uv pip install paddlepaddle-gpu==3.2.1 --extra-index-url https://www.paddlepaddle.org.cn/packages/stable/cu126/"
|
||||
brand: NVIDIA
|
||||
- note: "PaddleOCR document-parsing helpers — 1.6 requires >= 3.6.0 (pipeline_version v1.6)"
|
||||
command: 'uv pip install -U "paddleocr[doc-parser]>=3.6.0"'
|
||||
- note: "Safetensors loader used by the doc-parser path"
|
||||
command: "uv pip install safetensors"
|
||||
|
||||
features:
|
||||
text_only:
|
||||
description: "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
|
||||
args:
|
||||
- "--language-model-only"
|
||||
encoder_parallel:
|
||||
description: "Run the vision encoder in data-parallel mode — avoids TP comm overhead on the small encoder. Mutually exclusive with text_only."
|
||||
args:
|
||||
- "--mm-encoder-tp-mode"
|
||||
- "data"
|
||||
|
||||
opt_in_features:
|
||||
- text_only
|
||||
- encoder_parallel
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 2
|
||||
description: "BF16 weights — small footprint, runs on most GPUs"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
|
||||
kv_offload_support:
|
||||
offloading_cpu: verified
|
||||
offloading_fs: verified
|
||||
|
||||
hardware_overrides:
|
||||
amd:
|
||||
extra_args: []
|
||||
extra_env:
|
||||
SAFETENSORS_FAST_GPU: "1"
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[PaddleOCR-VL-1.6](https://huggingface.co/PaddlePaddle/PaddleOCR-VL-1.6) upgrades
|
||||
PaddleOCR-VL-1.5 with a region-aware data optimization framework (targeted enhancement
|
||||
of weak regions found in the previous model) and a progressive post-training recipe
|
||||
based on curated data selection and reinforcement learning:
|
||||
|
||||
- **96.33% on OmniDocBench v1.6** (new SOTA), plus new records on OmniDocBench v1.5
|
||||
and Real5-OmniDocBench (scanning, warping, screen-photography, illumination, skew)
|
||||
- Major gains in table, Chinese ancient document, and Chinese rare character recognition
|
||||
- Improved seal/stamp recognition, text spotting, and chart recognition
|
||||
|
||||
The architecture is **fully compatible with PaddleOCR-VL-1.5** (same 0.9B NaViT-style
|
||||
dynamic-resolution vision encoder + ERNIE-4.5-0.3B LM), so migration is plug-and-play:
|
||||
the vLLM launch command and feature flags are unchanged.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Hardware: 1x GPU (small VRAM footprint)
|
||||
- vLLM >= 0.11.1
|
||||
|
||||
### Install vLLM
|
||||
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm
|
||||
```
|
||||
|
||||
## Launch command
|
||||
|
||||
```bash
|
||||
vllm serve PaddlePaddle/PaddleOCR-VL-1.6 \
|
||||
--trust-remote-code \
|
||||
--max-num-batched-tokens 16384 \
|
||||
--no-enable-prefix-caching \
|
||||
--mm-processor-cache-gb 0
|
||||
```
|
||||
|
||||
Tip: OCR workloads don't benefit much from prefix caching or image reuse, so disable
|
||||
those to avoid hashing/caching overhead.
|
||||
|
||||
## Client Usage
|
||||
|
||||
Task-specific prompts (same task set as 1.5, including `spotting` and `seal`):
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600)
|
||||
|
||||
TASKS = {
|
||||
"ocr": "OCR:",
|
||||
"table": "Table Recognition:",
|
||||
"formula": "Formula Recognition:",
|
||||
"chart": "Chart Recognition:",
|
||||
"spotting": "Spotting:",
|
||||
"seal": "Seal Recognition:",
|
||||
}
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="PaddlePaddle/PaddleOCR-VL-1.6",
|
||||
messages=[{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "image_url", "image_url": {"url": "https://.../receipt.png"}},
|
||||
{"type": "text", "text": TASKS["spotting"]},
|
||||
],
|
||||
}],
|
||||
temperature=0.0,
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
## Offline Inference with PP-DocLayoutV3
|
||||
|
||||
Use separate venvs for `vllm` and `paddlepaddle` to avoid conflicts. The 1.6 pipeline
|
||||
requires `paddleocr[doc-parser] >= 3.6.0` and `pipeline_version="v1.6"`, which
|
||||
automatically uses [PP-DocLayoutV3](https://huggingface.co/PaddlePaddle/PP-DocLayoutV3)
|
||||
for layout detection (PP-DocLayoutV2 belongs to the v1 pipeline). Pass
|
||||
`vl_rec_api_model_name="PaddlePaddle/PaddleOCR-VL-1.6"` so the client matches the
|
||||
model name the vLLM server exposes — otherwise it defaults to
|
||||
`PaddleOCR-VL-1.6-0.9B` and fails with "The model PaddleOCR-VL-1.6-0.9B does not exist."
|
||||
|
||||
```bash
|
||||
uv pip install paddlepaddle-gpu==3.2.1 --extra-index-url https://www.paddlepaddle.org.cn/packages/stable/cu126/
|
||||
uv pip install -U "paddleocr[doc-parser]>=3.6.0"
|
||||
uv pip install safetensors
|
||||
```
|
||||
|
||||
```python
|
||||
from paddleocr import PaddleOCRVL
|
||||
|
||||
pipeline = PaddleOCRVL(
|
||||
pipeline_version="v1.6",
|
||||
vl_rec_backend="vllm-server",
|
||||
vl_rec_server_url="http://localhost:8000/v1",
|
||||
vl_rec_api_model_name="PaddlePaddle/PaddleOCR-VL-1.6",
|
||||
)
|
||||
|
||||
output = pipeline.predict("https://paddle-model-ecology.bj.bcebos.com/paddlex/imgs/demo_image/paddleocr_vl_demo.png")
|
||||
for i, res in enumerate(output):
|
||||
res.save_to_json(save_path=f"output_{i}.json")
|
||||
res.save_to_markdown(save_path=f"output_{i}.md")
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [PaddleOCR-VL-1.6 on Hugging Face](https://huggingface.co/PaddlePaddle/PaddleOCR-VL-1.6)
|
||||
- [Technical report (arXiv)](https://arxiv.org/pdf/2606.03264)
|
||||
- [PaddleOCR GitHub](https://github.com/PaddlePaddle/PaddleOCR)
|
||||
- [vLLM support PR](https://github.com/vllm-project/vllm/pull/27758)
|
||||
@@ -0,0 +1,190 @@
|
||||
meta:
|
||||
title: "PaddleOCR-VL"
|
||||
slug: "paddleocr-vl"
|
||||
provider: "PaddlePaddle"
|
||||
description: "PaddleOCR-VL (0.9B) — compact vision-language model for document parsing, OCR, tables, formulas, charts"
|
||||
date_added: 2025-11-04
|
||||
date_updated: 2026-04-17
|
||||
tasks:
|
||||
- multimodal
|
||||
related_recipes: []
|
||||
|
||||
model:
|
||||
model_id: "PaddlePaddle/PaddleOCR-VL"
|
||||
min_vllm_version: "0.11.1"
|
||||
architecture: dense
|
||||
parameter_count: "0.9B"
|
||||
active_parameters: "0.9B"
|
||||
context_length: 131072
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
- "--max-num-batched-tokens"
|
||||
- "16384"
|
||||
- "--no-enable-prefix-caching"
|
||||
- "--mm-processor-cache-gb"
|
||||
- "0"
|
||||
base_env: {}
|
||||
|
||||
dependencies:
|
||||
- note: "PaddlePaddle runtime (install in a separate venv from vllm to avoid conflicts)"
|
||||
command: "uv pip install paddlepaddle-gpu==3.2.1 --extra-index-url https://www.paddlepaddle.org.cn/packages/stable/cu126/"
|
||||
- note: "PaddleOCR document-parsing helpers"
|
||||
command: 'uv pip install -U "paddleocr[doc-parser]"'
|
||||
- note: "Safetensors loader used by the doc-parser path"
|
||||
command: "uv pip install safetensors"
|
||||
|
||||
features:
|
||||
text_only:
|
||||
description: "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
|
||||
args:
|
||||
- "--language-model-only"
|
||||
encoder_parallel:
|
||||
description: "Run the vision encoder in data-parallel mode — avoids TP comm overhead on the small encoder. Mutually exclusive with text_only."
|
||||
args:
|
||||
- "--mm-encoder-tp-mode"
|
||||
- "data"
|
||||
|
||||
opt_in_features:
|
||||
- text_only
|
||||
- encoder_parallel
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 2
|
||||
description: "BF16 weights — small footprint, runs on most GPUs"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
|
||||
kv_offload_support:
|
||||
offloading_cpu: verified
|
||||
offloading_fs: verified
|
||||
|
||||
hardware_overrides:
|
||||
amd:
|
||||
extra_args: []
|
||||
extra_env:
|
||||
SAFETENSORS_FAST_GPU: "1"
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[PaddleOCR-VL](https://huggingface.co/PaddlePaddle/PaddleOCR-VL) is a SOTA resource-efficient
|
||||
model for document parsing. Its core (PaddleOCR-VL-0.9B) combines a NaViT-style dynamic
|
||||
resolution visual encoder with an ERNIE-4.5-0.3B language model, optimized for OCR,
|
||||
tables, formulas, and chart recognition.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Hardware: 1x GPU (small VRAM footprint)
|
||||
- vLLM >= 0.11.1 (nightly if not released yet)
|
||||
- AMD ROCm wheels require Python 3.12, ROCm 7.2.3, and glibc >= 2.35.
|
||||
|
||||
### Install vLLM (NVIDIA)
|
||||
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --pre \
|
||||
--extra-index-url https://wheels.vllm.ai/nightly \
|
||||
--extra-index-url https://download.pytorch.org/whl/cu129 \
|
||||
--index-strategy unsafe-best-match
|
||||
```
|
||||
|
||||
### Install vLLM (AMD ROCm)
|
||||
|
||||
```bash
|
||||
uv venv --python 3.12
|
||||
source .venv/bin/activate
|
||||
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm
|
||||
```
|
||||
|
||||
## Launch command
|
||||
|
||||
```bash
|
||||
vllm serve PaddlePaddle/PaddleOCR-VL \
|
||||
--trust-remote-code \
|
||||
--max-num-batched-tokens 16384 \
|
||||
--no-enable-prefix-caching \
|
||||
--mm-processor-cache-gb 0
|
||||
```
|
||||
|
||||
AMD MI300X/MI325X/MI355X:
|
||||
|
||||
```bash
|
||||
export SAFETENSORS_FAST_GPU=1
|
||||
|
||||
vllm serve PaddlePaddle/PaddleOCR-VL \
|
||||
--trust-remote-code \
|
||||
--max-num-batched-tokens 16384 \
|
||||
--no-enable-prefix-caching \
|
||||
--mm-processor-cache-gb 0
|
||||
```
|
||||
|
||||
Tip: OCR workloads don't benefit much from prefix caching or image reuse, so disable
|
||||
those to avoid hashing/caching overhead.
|
||||
|
||||
## Client Usage
|
||||
|
||||
Task-specific prompts:
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600)
|
||||
|
||||
TASKS = {
|
||||
"ocr": "OCR:",
|
||||
"table": "Table Recognition:",
|
||||
"formula": "Formula Recognition:",
|
||||
"chart": "Chart Recognition:",
|
||||
}
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="PaddlePaddle/PaddleOCR-VL",
|
||||
messages=[{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "image_url", "image_url": {"url": "https://.../receipt.png"}},
|
||||
{"type": "text", "text": TASKS["ocr"]},
|
||||
],
|
||||
}],
|
||||
temperature=0.0,
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
## Offline Inference with PP-DocLayoutV2
|
||||
|
||||
Use separate venvs for `vllm` and `paddlepaddle` to avoid conflicts. If you see
|
||||
"The model PaddleOCR-VL-0.9B does not exist.", add `--served-model-name PaddleOCR-VL-0.9B`.
|
||||
|
||||
```bash
|
||||
uv pip install paddlepaddle-gpu==3.2.1 --extra-index-url https://www.paddlepaddle.org.cn/packages/stable/cu126/
|
||||
uv pip install -U "paddleocr[doc-parser]"
|
||||
uv pip install safetensors
|
||||
```
|
||||
|
||||
```python
|
||||
from paddleocr import PaddleOCRVL
|
||||
|
||||
pipeline = PaddleOCRVL(
|
||||
vl_rec_backend="vllm-server",
|
||||
vl_rec_server_url="http://localhost:8000/v1",
|
||||
layout_detection_model_name="PP-DocLayoutV2",
|
||||
layout_detection_model_dir="/path/to/your/PP-DocLayoutV2/",
|
||||
)
|
||||
|
||||
output = pipeline.predict("https://paddle-model-ecology.bj.bcebos.com/paddlex/imgs/demo_image/paddleocr_vl_demo.png")
|
||||
for i, res in enumerate(output):
|
||||
res.save_to_json(save_path=f"output_{i}.json")
|
||||
res.save_to_markdown(save_path=f"output_{i}.md")
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [PaddleOCR-VL on Hugging Face](https://huggingface.co/PaddlePaddle/PaddleOCR-VL)
|
||||
- [PaddleOCR GitHub](https://github.com/PaddlePaddle/PaddleOCR)
|
||||
@@ -0,0 +1,161 @@
|
||||
meta:
|
||||
title: "QwQ-32B"
|
||||
slug: "qwq-32b"
|
||||
provider: "Qwen"
|
||||
description: "Qwen's 32B dense reasoning model, validated for Intel Xeon 6 CPU execution."
|
||||
date_added: 2026-08-17
|
||||
date_updated: 2026-09-12
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "Validated for Intel Xeon 6 CPU execution"
|
||||
related_recipes:
|
||||
- "Qwen/Qwen3-32B"
|
||||
hardware:
|
||||
xeon6: verified
|
||||
|
||||
model:
|
||||
model_id: "Qwen/QwQ-32B"
|
||||
min_vllm_version: "0.7.3"
|
||||
architecture: dense
|
||||
parameter_count: "32B"
|
||||
active_parameters: "32B"
|
||||
context_length: 40960
|
||||
base_args: []
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with the Hermes parser"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "hermes"
|
||||
reasoning:
|
||||
description: "Extract QwQ reasoning content with the DeepSeek-R1 reasoning parser"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "deepseek_r1"
|
||||
long_context:
|
||||
description: "YaRN RoPE scaling to the full 131,072-token context, which the model card requires for prompts over 8,192 tokens. Static YaRN holds the scaling factor constant regardless of input length, so leave this off unless you actually serve long prompts."
|
||||
args:
|
||||
- "--hf-overrides"
|
||||
- '{"max_position_embeddings":131072,"rope_parameters":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":32768}}'
|
||||
- "--max-model-len"
|
||||
- "131072"
|
||||
|
||||
opt_in_features:
|
||||
- long_context
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 77
|
||||
description: "Full precision BF16"
|
||||
supported_hardware:
|
||||
- xeon6
|
||||
w8a8_int8:
|
||||
model_id: "RedHatAI/QwQ-32B-quantized.w8a8"
|
||||
precision: int8
|
||||
vram_minimum_gb: 39
|
||||
description: "Red Hat AI INT8 W8A8 compressed-tensors checkpoint"
|
||||
supported_hardware:
|
||||
- xeon6
|
||||
awq:
|
||||
model_id: "Qwen/QwQ-32B-AWQ"
|
||||
precision: int4
|
||||
vram_minimum_gb: 20
|
||||
description: "Qwen official AWQ 4-bit checkpoint"
|
||||
supported_hardware:
|
||||
- xeon6
|
||||
extra_args:
|
||||
- "--quantization"
|
||||
- "awq"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
|
||||
kv_offload_support:
|
||||
offloading_cpu: verified
|
||||
offloading_fs: verified
|
||||
|
||||
hardware_overrides: {}
|
||||
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
QwQ-32B is Qwen's 32B dense reasoning model. The CPU model validation table
|
||||
marks it supported on Intel Xeon 6.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Hardware: Intel Xeon 6 CPUs
|
||||
- vLLM CPU >= 0.8.5 for this validated Xeon 6 configuration
|
||||
|
||||
### pip (Intel Xeon 6 CPUs)
|
||||
|
||||
For Intel and AMD x86 CPUs, follow the
|
||||
[CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels)
|
||||
installation instructions.
|
||||
|
||||
### Docker (Intel Xeon 6 CPUs)
|
||||
|
||||
```bash
|
||||
docker pull vllm/vllm-openai-cpu:latest-x86_64
|
||||
```
|
||||
|
||||
## Intel Xeon 6
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/QwQ-32B \
|
||||
--tensor-parallel-size 1 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser hermes \
|
||||
--reasoning-parser deepseek_r1
|
||||
```
|
||||
|
||||
Docker (the image entrypoint is `vllm serve`):
|
||||
|
||||
```bash
|
||||
docker run \
|
||||
--privileged --ipc=host -p 8000:8000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai-cpu:latest-x86_64 Qwen/QwQ-32B \
|
||||
--tensor-parallel-size 1 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser hermes \
|
||||
--reasoning-parser deepseek_r1
|
||||
```
|
||||
|
||||
W8A8 INT8:
|
||||
|
||||
```bash
|
||||
vllm serve RedHatAI/QwQ-32B-quantized.w8a8
|
||||
```
|
||||
|
||||
The example uses TP=1. Adjust tensor parallelism based on system topology;
|
||||
topology-specific CPU binding is intentionally not hard-coded.
|
||||
|
||||
## Runtime and Platform Tuning
|
||||
|
||||
The following settings are intentionally not prescribed as portable CPU model
|
||||
defaults because they depend on the workload, hardware, or runtime environment:
|
||||
|
||||
- `--max-num-batched-tokens`: scheduler/throughput tuning.
|
||||
- `--max-num-seqs`: concurrency and scheduler-capacity tuning.
|
||||
- `--gpu-memory-utilization`: platform memory-budget tuning.
|
||||
- `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning.
|
||||
- `VLLM_ENGINE_ITERATION_TIMEOUT_S`: operational runtime timeout.
|
||||
|
||||
These settings may still appear in validated hardware-specific overrides. Tune
|
||||
them at deployment time based on platform resources, workload shape, and
|
||||
latency/throughput goals.
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/Qwen/QwQ-32B)
|
||||
- [W8A8 variant](https://huggingface.co/RedHatAI/QwQ-32B-quantized.w8a8)
|
||||
- [vLLM reasoning outputs](https://docs.vllm.ai/en/latest/features/reasoning_outputs.html)
|
||||
- [vLLM tool calling](https://docs.vllm.ai/en/latest/features/tool_calling.html)
|
||||
@@ -0,0 +1,128 @@
|
||||
meta:
|
||||
title: "Qwen2.5-32B"
|
||||
slug: "qwen2.5-32b"
|
||||
provider: "Qwen"
|
||||
description: "Qwen2.5 32B dense base (pretrained) language model for text completion — verified on TPU v6e (Trillium)."
|
||||
date_added: 2026-06-05
|
||||
date_updated: 2026-06-04
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "Verified on TPU v6e (Trillium) with BF16, TP=4 on a 2x2 slice"
|
||||
related_recipes: []
|
||||
hardware:
|
||||
trillium: verified
|
||||
|
||||
model:
|
||||
model_id: "Qwen/Qwen2.5-32B"
|
||||
min_vllm_version: "0.6.2"
|
||||
architecture: dense
|
||||
parameter_count: "32B"
|
||||
active_parameters: "32B"
|
||||
context_length: 131072
|
||||
base_args: []
|
||||
base_env: {}
|
||||
|
||||
features: {}
|
||||
|
||||
opt_in_features: []
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 77
|
||||
description: "Full precision BF16 — TPU v6e 2x2 (4 chips, TP=4) or a single 80GB+ GPU"
|
||||
|
||||
fp8:
|
||||
model_id: "RedHatAI/Qwen2.5-32B-FP8-dynamic"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 39
|
||||
description: "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
|
||||
kv_offload_support:
|
||||
offloading_cpu: verified
|
||||
offloading_fs: verified
|
||||
|
||||
hardware_overrides: {}
|
||||
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
Qwen2.5-32B is the 32B dense **base** (pretrained, non-instruct) model in the Qwen2.5 series. It's a text-completion model — use the `/v1/completions` endpoint rather than chat, or pick `Qwen/Qwen2.5-32B-Instruct` if you need a chat-tuned variant. BF16 is the training precision and gives the best accuracy.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### NVIDIA
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend auto
|
||||
```
|
||||
|
||||
### TPU (Trillium / v6e)
|
||||
Use the official vLLM TPU image. The 32B model is served on a `v6e-4` slice (`--topology 2x2`, 4 chips, TP=4).
|
||||
```bash
|
||||
export HF_HOME=/dev/shm
|
||||
export HF_TOKEN=<your HF token>
|
||||
```
|
||||
|
||||
## Deployment Configurations
|
||||
|
||||
### TPU v6e (Trillium, 2x2 slice, TP=4)
|
||||
Verified end-to-end on a 4-chip v6e slice.
|
||||
```bash
|
||||
export HF_HOME=/dev/shm
|
||||
export HF_TOKEN=<your HF token>
|
||||
export TP=4
|
||||
export MAX_MODEL_LEN=4096
|
||||
vllm serve Qwen/Qwen2.5-32B \
|
||||
--seed 42 \
|
||||
--gpu-memory-utilization 0.98 \
|
||||
--max-num-batched-tokens 2048 \
|
||||
--max-num-seqs 128 \
|
||||
--tensor-parallel-size $TP \
|
||||
--max-model-len $MAX_MODEL_LEN
|
||||
```
|
||||
|
||||
### Single GPU / node (BF16)
|
||||
```bash
|
||||
vllm serve Qwen/Qwen2.5-32B \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000 \
|
||||
--tensor-parallel-size 2
|
||||
```
|
||||
|
||||
## Configuration Tips
|
||||
|
||||
- `--max-model-len 4096` keeps memory in check on a 4-chip v6e slice; native context is 128K — raise it if you have headroom.
|
||||
- vLLM has been V1-engine-only for a long time, so no `VLLM_USE_V1=1` export is needed even though the upstream TPU recipe still shows it.
|
||||
- vLLM uses 90% of device memory by default; the TPU recipe pushes `--gpu-memory-utilization 0.98` to maximize KV cache.
|
||||
- `--max-num-batched-tokens` and `--max-num-seqs` tune batch throughput vs latency.
|
||||
|
||||
## Benchmarking
|
||||
|
||||
Launch the server with `--no-enable-prefix-caching` for consistent measurements.
|
||||
|
||||
```bash
|
||||
vllm bench serve \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000 \
|
||||
--model Qwen/Qwen2.5-32B \
|
||||
--dataset-name random \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 128
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/Qwen/Qwen2.5-32B)
|
||||
- [Qwen2.5-32B on Trillium (v6e)](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Qwen2.5-32B)
|
||||
@@ -0,0 +1,205 @@
|
||||
meta:
|
||||
title: "Qwen2.5-VL-72B-Instruct"
|
||||
slug: "qwen2.5-vl-72b-instruct"
|
||||
provider: "Qwen"
|
||||
description: "Qwen2.5-VL dense vision-language model (72B) for high-quality image and video understanding."
|
||||
date_added: 2025-08-22
|
||||
date_updated: 2026-04-17
|
||||
tasks:
|
||||
- multimodal
|
||||
- text
|
||||
performance_headline: "Verified on 4x A100 and 4x MI300X/MI325X/MI355X with BF16"
|
||||
related_recipes:
|
||||
- "Qwen/Qwen2.5-VL-7B-Instruct"
|
||||
hardware:
|
||||
mi300x: verified
|
||||
mi325x: verified
|
||||
mi355x: verified
|
||||
|
||||
model:
|
||||
model_id: "Qwen/Qwen2.5-VL-72B-Instruct"
|
||||
min_vllm_version: "0.10.2"
|
||||
architecture: dense
|
||||
parameter_count: "72B"
|
||||
active_parameters: "72B"
|
||||
context_length: 128000
|
||||
base_args: []
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
text_only:
|
||||
description: "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
|
||||
args:
|
||||
- "--language-model-only"
|
||||
encoder_parallel:
|
||||
description: "Run the vision encoder in data-parallel mode — avoids TP comm overhead on the small encoder. Mutually exclusive with text_only."
|
||||
args:
|
||||
- "--mm-encoder-tp-mode"
|
||||
- "data"
|
||||
|
||||
opt_in_features:
|
||||
- text_only
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 173
|
||||
description: "Full precision BF16 — 4x A100 80GB or 4x MI300X/MI325X/MI355X"
|
||||
awq:
|
||||
model_id: "Qwen/Qwen2.5-VL-72B-Instruct-AWQ"
|
||||
precision: int4
|
||||
vram_minimum_gb: 43
|
||||
description: "AWQ 4-bit quantized weights"
|
||||
extra_args:
|
||||
- "--quantization"
|
||||
- "awq"
|
||||
|
||||
fp8:
|
||||
model_id: "RedHatAI/Qwen2.5-VL-72B-Instruct-FP8-dynamic"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 87
|
||||
description: "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
- multi_node_tp_pp
|
||||
|
||||
kv_offload_support:
|
||||
offloading_cpu: verified
|
||||
offloading_fs: verified
|
||||
|
||||
hardware_overrides:
|
||||
hopper:
|
||||
extra_args:
|
||||
- "--mm-encoder-tp-mode"
|
||||
- "data"
|
||||
extra_env: {}
|
||||
blackwell:
|
||||
extra_args:
|
||||
- "--mm-encoder-tp-mode"
|
||||
- "data"
|
||||
extra_env: {}
|
||||
amd:
|
||||
# PR #125 verified on MI300X only; TP=2 for throughput, TP=8 for latency.
|
||||
extra_args:
|
||||
- "--mm-encoder-tp-mode"
|
||||
- "data"
|
||||
- "--limit-mm-per-prompt"
|
||||
- '{"image":2,"video":0}'
|
||||
extra_env:
|
||||
VLLM_ROCM_USE_AITER: "1"
|
||||
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
This guide describes how to run Qwen2.5-VL series on the targeted accelerated stack. Since BF16 is the commonly used precision for Qwen2.5-VL training, using BF16 in inference ensures the best accuracy.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### NVIDIA
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend auto
|
||||
```
|
||||
|
||||
### AMD ROCm (MI300X, MI325X, MI355X)
|
||||
> Note: The vLLM wheel for ROCm requires Python 3.12, ROCm 7.0, and glibc >= 2.35. Use the Docker flow if your environment is incompatible.
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm
|
||||
```
|
||||
|
||||
### TPU Deployment
|
||||
TPU support has been verified for the smaller [Qwen2.5-VL-7B-Instruct](/Qwen/Qwen2.5-VL-7B-Instruct) on a single v6e chip — the 72B is not yet validated on Trillium. See the [upstream TPU recipe](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Qwen2.5-VL) (7B).
|
||||
|
||||
## Deployment Configurations
|
||||
|
||||
### 4xA100 (BF16, TP=4)
|
||||
```bash
|
||||
export CUDA_VISIBLE_DEVICES=0,1,2,3
|
||||
vllm serve Qwen/Qwen2.5-VL-72B-Instruct \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000 \
|
||||
--tensor-parallel-size 4 \
|
||||
--mm-encoder-tp-mode data \
|
||||
--limit-mm-per-prompt '{"image":2,"video":0}'
|
||||
```
|
||||
|
||||
### 4xMI300X/MI325X/MI355X (BF16, TP=4)
|
||||
```bash
|
||||
export CUDA_VISIBLE_DEVICES=0,1,2,3
|
||||
export VLLM_ROCM_USE_AITER=1
|
||||
vllm serve Qwen/Qwen2.5-VL-72B-Instruct \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000 \
|
||||
--tensor-parallel-size 4 \
|
||||
--mm-encoder-tp-mode data \
|
||||
--limit-mm-per-prompt '{"image":2,"video":0}'
|
||||
```
|
||||
|
||||
### Qwen2.5-VL-7B-Instruct (DP=4)
|
||||
For medium-size 7B model, data parallelism works better.
|
||||
|
||||
```bash
|
||||
export CUDA_VISIBLE_DEVICES=0,1,2,3
|
||||
vllm serve Qwen/Qwen2.5-VL-7B-Instruct \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000 \
|
||||
--data-parallel-size 4 \
|
||||
--limit-mm-per-prompt '{"image":2,"video":0}'
|
||||
```
|
||||
|
||||
## Configuration Tips
|
||||
|
||||
- `--max-model-len 65536` is usually good for most scenarios (native context is 128K).
|
||||
- For A100-80GB devices, TP must be >= 2 to avoid OOM.
|
||||
- `--limit-mm-per-prompt` caps incoming multimodal requests.
|
||||
- `--mm-encoder-tp-mode data` deploys the small ViT encoder in DP fashion (ViT ≈ 675M vs 72B LM).
|
||||
- vLLM uses 90% of GPU memory by default; set `--gpu-memory-utilization=0.95` to maximize KV cache.
|
||||
|
||||
## Benchmarking
|
||||
|
||||
Launch the server with `--no-enable-prefix-caching` to get consistent measurements.
|
||||
|
||||
### VisionArena-Chat
|
||||
```bash
|
||||
vllm bench serve \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000 \
|
||||
--backend openai-chat \
|
||||
--endpoint /v1/chat/completions \
|
||||
--model Qwen/Qwen2.5-VL-72B-Instruct \
|
||||
--dataset-name hf \
|
||||
--dataset-path lmarena-ai/VisionArena-Chat \
|
||||
--num-prompts 128
|
||||
```
|
||||
|
||||
### Random Synthetic
|
||||
```bash
|
||||
vllm bench serve \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000 \
|
||||
--model Qwen/Qwen2.5-VL-72B-Instruct \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 128
|
||||
```
|
||||
|
||||
Workload mixes:
|
||||
- Prompt-heavy: 8000 in / 1000 out
|
||||
- Decode-heavy: 1000 in / 8000 out
|
||||
- Balanced: 1000 in / 1000 out
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct)
|
||||
- [vLLM multimodal inputs guide](https://docs.vllm.ai/en/latest/features/multimodal_inputs.html)
|
||||
@@ -0,0 +1,235 @@
|
||||
meta:
|
||||
title: "Qwen2.5-VL-7B-Instruct"
|
||||
slug: "qwen2.5-vl-7b-instruct"
|
||||
provider: "Qwen"
|
||||
description: "Qwen2.5-VL dense vision-language model (7B) for image and video understanding — fits on a single TPU v6e chip, one GPU, or Intel Xeon 6 CPU."
|
||||
date_added: 2025-08-22
|
||||
date_updated: 2026-09-12
|
||||
tasks:
|
||||
- multimodal
|
||||
- text
|
||||
performance_headline: "Verified on TPU v6e (Trillium) and Intel Xeon 6 CPU with BF16"
|
||||
related_recipes:
|
||||
- "Qwen/Qwen2.5-VL-72B-Instruct"
|
||||
hardware:
|
||||
trillium: verified
|
||||
xeon6: verified
|
||||
|
||||
model:
|
||||
model_id: "Qwen/Qwen2.5-VL-7B-Instruct"
|
||||
min_vllm_version: "0.10.2"
|
||||
architecture: dense
|
||||
parameter_count: "7B"
|
||||
active_parameters: "7B"
|
||||
context_length: 128000
|
||||
base_args: []
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
text_only:
|
||||
description: "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
|
||||
args:
|
||||
- "--language-model-only"
|
||||
encoder_parallel:
|
||||
description: "Run the vision encoder in data-parallel mode — avoids TP comm overhead on the small encoder. Mutually exclusive with text_only."
|
||||
args:
|
||||
- "--mm-encoder-tp-mode"
|
||||
- "data"
|
||||
|
||||
opt_in_features:
|
||||
- text_only
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 17
|
||||
description: "Full precision BF16 — fits on a single TPU v6e chip or one 24GB+ GPU"
|
||||
supported_hardware: [trillium, xeon6]
|
||||
awq:
|
||||
model_id: "Qwen/Qwen2.5-VL-7B-Instruct-AWQ"
|
||||
precision: int4
|
||||
vram_minimum_gb: 5
|
||||
description: "AWQ 4-bit quantized weights"
|
||||
supported_hardware: [trillium]
|
||||
extra_args:
|
||||
- "--quantization"
|
||||
- "awq"
|
||||
|
||||
fp8:
|
||||
model_id: "RedHatAI/Qwen2.5-VL-7B-Instruct-FP8-Dynamic"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 9
|
||||
description: "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
|
||||
kv_offload_support:
|
||||
offloading_cpu: verified
|
||||
offloading_fs: verified
|
||||
|
||||
hardware_overrides: {}
|
||||
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
Qwen2.5-VL-7B-Instruct is the small dense vision-language model in the Qwen2.5-VL series. At 7B it fits comfortably on a single accelerator — one TPU v6e (Trillium) chip, or a single 24GB+ GPU — and is also validated for Intel Xeon 6 CPU serving. BF16 is the precision used in training, so BF16 inference gives the best accuracy.
|
||||
|
||||
For the 72B sibling, see [Qwen2.5-VL-72B-Instruct](/Qwen/Qwen2.5-VL-72B-Instruct).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### NVIDIA
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend auto
|
||||
```
|
||||
|
||||
### Intel Xeon 6 CPUs
|
||||
|
||||
- Hardware: Intel Xeon 6 CPUs
|
||||
- vLLM CPU >= 0.8.5 for this validated Xeon 6 configuration
|
||||
|
||||
For Intel and AMD x86 CPUs, follow the
|
||||
[CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels)
|
||||
installation instructions.
|
||||
|
||||
```bash
|
||||
docker pull vllm/vllm-openai-cpu:latest-x86_64
|
||||
```
|
||||
|
||||
### TPU (Trillium / v6e)
|
||||
Use the official vLLM TPU image. The 7B model fits on a single v6e chip (`v6e-1`, `--topology 1x1`).
|
||||
```bash
|
||||
export HF_HOME=/dev/shm
|
||||
export HF_TOKEN=<your HF token>
|
||||
docker run --rm --privileged --net=host \
|
||||
--shm-size=16G \
|
||||
-e HF_HOME -e HF_TOKEN \
|
||||
vllm/vllm-tpu:latest \
|
||||
vllm serve ...
|
||||
```
|
||||
|
||||
## Deployment Configurations
|
||||
|
||||
### Intel Xeon 6
|
||||
|
||||
Choose tensor parallelism based on the system topology; the recipe does not
|
||||
prescribe a fixed TP/DP layout or hard-code NUMA node IDs.
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen2.5-VL-7B-Instruct
|
||||
```
|
||||
|
||||
Docker:
|
||||
|
||||
```bash
|
||||
docker run \
|
||||
--privileged --ipc=host -p 8000:8000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai-cpu:latest-x86_64 Qwen/Qwen2.5-VL-7B-Instruct \
|
||||
--tensor-parallel-size 1 \
|
||||
--mm-encoder-tp-mode data
|
||||
```
|
||||
|
||||
#### Runtime and Platform Tuning
|
||||
|
||||
The following settings are intentionally not prescribed as portable CPU model
|
||||
defaults because they depend on the workload, hardware, or runtime environment:
|
||||
|
||||
- `--max-num-batched-tokens`: scheduler/throughput tuning.
|
||||
- `--max-num-seqs`: concurrency and scheduler-capacity tuning.
|
||||
- `--gpu-memory-utilization`: platform memory-budget tuning.
|
||||
- `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning.
|
||||
- `VLLM_ENGINE_ITERATION_TIMEOUT_S`: operational runtime timeout.
|
||||
|
||||
These settings may still appear in validated hardware-specific overrides. Tune
|
||||
them at deployment time based on platform resources, workload shape, and
|
||||
latency/throughput goals.
|
||||
|
||||
### TPU v6e (Trillium, single chip)
|
||||
Verified end-to-end on a single v6e chip.
|
||||
```bash
|
||||
export HF_HOME=/dev/shm
|
||||
export HF_TOKEN=<your HF token>
|
||||
vllm serve Qwen/Qwen2.5-VL-7B-Instruct \
|
||||
--tensor-parallel-size 1 \
|
||||
--pipeline-parallel-size 1 \
|
||||
--dtype bfloat16 \
|
||||
--gpu-memory-utilization 0.98 \
|
||||
--max-model-len 16384 \
|
||||
--limit-mm-per-prompt '{"image":10,"video":0}' \
|
||||
--mm-processor-kwargs '{"max_pixels":1003520}' \
|
||||
--guided-decoding-backend xgrammar \
|
||||
--disable-chunked-mm-input
|
||||
```
|
||||
|
||||
### Single GPU (BF16, TP=1)
|
||||
```bash
|
||||
export CUDA_VISIBLE_DEVICES=0
|
||||
vllm serve Qwen/Qwen2.5-VL-7B-Instruct \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000 \
|
||||
--limit-mm-per-prompt '{"image":2,"video":0}'
|
||||
```
|
||||
|
||||
### Multi-GPU (DP=4)
|
||||
For higher throughput across a node, data parallelism works better than TP for a model this size.
|
||||
```bash
|
||||
export CUDA_VISIBLE_DEVICES=0,1,2,3
|
||||
vllm serve Qwen/Qwen2.5-VL-7B-Instruct \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000 \
|
||||
--data-parallel-size 4 \
|
||||
--limit-mm-per-prompt '{"image":2,"video":0}'
|
||||
```
|
||||
|
||||
## Configuration Tips
|
||||
|
||||
- `--max-model-len 16384` is a safe default on a single v6e chip (native context is 128K); raise it if you have headroom.
|
||||
- `--limit-mm-per-prompt` caps incoming multimodal requests per prompt.
|
||||
- `--mm-processor-kwargs '{"max_pixels":1003520}'` bounds the per-image resolution to control encoder cost.
|
||||
- vLLM uses 90% of device memory by default; on TPU the recipe pushes `--gpu-memory-utilization 0.98` to maximize KV cache.
|
||||
- `--disable-chunked-mm-input` is recommended on TPU for stable multimodal batching.
|
||||
|
||||
## Benchmarking
|
||||
|
||||
Launch the server with `--no-enable-prefix-caching` to get consistent measurements.
|
||||
|
||||
### VisionArena-Chat
|
||||
```bash
|
||||
vllm bench serve \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000 \
|
||||
--backend openai-chat \
|
||||
--endpoint /v1/chat/completions \
|
||||
--model Qwen/Qwen2.5-VL-7B-Instruct \
|
||||
--dataset-name hf \
|
||||
--dataset-path lmarena-ai/VisionArena-Chat \
|
||||
--num-prompts 128
|
||||
```
|
||||
|
||||
### Random Synthetic
|
||||
```bash
|
||||
vllm bench serve \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000 \
|
||||
--model Qwen/Qwen2.5-VL-7B-Instruct \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--num-prompts 128
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct)
|
||||
- [Qwen2.5-VL on Trillium (v6e)](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Qwen2.5-VL)
|
||||
- [vLLM multimodal inputs guide](https://docs.vllm.ai/en/latest/features/multimodal_inputs.html)
|
||||
@@ -0,0 +1,129 @@
|
||||
meta:
|
||||
title: "Qwen3-1.7B"
|
||||
slug: "qwen3-1.7b"
|
||||
provider: "Qwen"
|
||||
description: "Qwen3 1.7B dense model with hybrid thinking/non-thinking modes, validated for BF16 serving on Intel Xeon 6."
|
||||
date_added: 2026-08-14
|
||||
date_updated: 2026-09-12
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "Validated on Intel Xeon 6 (GNR) with BF16 serving"
|
||||
related_recipes:
|
||||
- "Qwen/Qwen3-4B"
|
||||
hardware:
|
||||
xeon6: verified
|
||||
|
||||
model:
|
||||
model_id: "Qwen/Qwen3-1.7B"
|
||||
min_vllm_version: "0.8.5"
|
||||
architecture: dense
|
||||
parameter_count: "1.7B"
|
||||
active_parameters: "1.7B"
|
||||
context_length: 40960
|
||||
base_args: []
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with the Hermes parser"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "hermes"
|
||||
reasoning:
|
||||
description: "Enable Qwen3 thinking mode with the Qwen3 reasoning parser"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "qwen3"
|
||||
|
||||
opt_in_features: []
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 5
|
||||
description: "Full precision BF16"
|
||||
supported_hardware: [xeon6]
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
|
||||
hardware_overrides:
|
||||
cpu:
|
||||
extra_args:
|
||||
- "--max-num-batched-tokens"
|
||||
- "16384"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.8"
|
||||
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
Qwen3-1.7B is a compact dense Qwen3 model with hybrid thinking and non-thinking
|
||||
modes. This recipe includes an Intel Xeon 6 CPU configuration validated with
|
||||
BF16 serving.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Hardware: 1x Xeon 6 CPUs
|
||||
- vLLM >= 0.8.5
|
||||
|
||||
### pip (Intel Xeon 6 CPUs)
|
||||
|
||||
For Intel and AMD x86 CPUs, follow the
|
||||
[CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels)
|
||||
installation instructions.
|
||||
|
||||
### Docker (Intel Xeon 6 CPUs)
|
||||
|
||||
```bash
|
||||
docker pull vllm/vllm-openai-cpu:latest-x86_64
|
||||
```
|
||||
|
||||
## Intel Xeon 6
|
||||
|
||||
Choose the tensor parallel size based on the system topology and deployment
|
||||
requirements. Add `--tensor-parallel-size <N>` when needed; the recipe does
|
||||
not prescribe a Xeon 6 TP value or hard-code NUMA node IDs.
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-1.7B \
|
||||
--max-num-batched-tokens 16384 \
|
||||
--gpu-memory-utilization 0.8
|
||||
```
|
||||
|
||||
Docker:
|
||||
|
||||
```bash
|
||||
docker run \
|
||||
--privileged --ipc=host -p 8000:8000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai-cpu:latest-x86_64 Qwen/Qwen3-1.7B \
|
||||
--tensor-parallel-size 1 \
|
||||
--max-num-batched-tokens 16384 \
|
||||
--gpu-memory-utilization 0.8 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser hermes \
|
||||
--reasoning-parser qwen3
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
|
||||
response = client.chat.completions.create(
|
||||
model="Qwen/Qwen3-1.7B",
|
||||
messages=[{"role": "user", "content": "Explain tensor parallelism briefly."}],
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/Qwen/Qwen3-1.7B)
|
||||
- [vLLM CPU installation](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/)
|
||||
@@ -0,0 +1,152 @@
|
||||
meta:
|
||||
title: "Qwen3-14B"
|
||||
slug: "qwen3-14b"
|
||||
provider: "Qwen"
|
||||
description: "Qwen3 14B dense model with hybrid thinking/non-thinking modes, validated for Intel Xeon 6 CPU serving."
|
||||
date_added: 2026-08-17
|
||||
date_updated: 2026-09-12
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "Validated on Intel Xeon 6 CPU with BF16 serving"
|
||||
related_recipes:
|
||||
- "Qwen/Qwen3-4B"
|
||||
- "Qwen/Qwen3-32B"
|
||||
hardware:
|
||||
xeon6: verified
|
||||
|
||||
model:
|
||||
model_id: "Qwen/Qwen3-14B"
|
||||
min_vllm_version: "0.8.5"
|
||||
architecture: dense
|
||||
parameter_count: "14B"
|
||||
active_parameters: "14B"
|
||||
context_length: 40960
|
||||
base_args: []
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with the Hermes parser"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "hermes"
|
||||
reasoning:
|
||||
description: "Enable Qwen3 thinking mode with the Qwen3 reasoning parser"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "qwen3"
|
||||
long_context:
|
||||
description: "YaRN RoPE scaling to the model card's validated 131,072-token context. Static YaRN holds the scaling factor constant regardless of input length, so leave this off unless you actually serve long prompts — it can cost quality below 32K."
|
||||
args:
|
||||
- "--hf-overrides"
|
||||
- '{"max_position_embeddings":131072,"rope_parameters":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":32768}}'
|
||||
- "--max-model-len"
|
||||
- "131072"
|
||||
|
||||
opt_in_features:
|
||||
- reasoning
|
||||
- long_context
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 34
|
||||
description: "Full precision BF16"
|
||||
supported_hardware:
|
||||
- xeon6
|
||||
fp8:
|
||||
model_id: "Qwen/Qwen3-14B-FP8"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 17
|
||||
description: "Qwen official FP8 checkpoint"
|
||||
awq:
|
||||
model_id: "Qwen/Qwen3-14B-AWQ"
|
||||
precision: int4
|
||||
vram_minimum_gb: 10
|
||||
description: "Qwen official AWQ 4-bit checkpoint"
|
||||
supported_hardware:
|
||||
- xeon6
|
||||
extra_args:
|
||||
- "--quantization"
|
||||
- "awq"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
|
||||
kv_offload_support:
|
||||
offloading_cpu: verified
|
||||
offloading_fs: verified
|
||||
|
||||
hardware_overrides: {}
|
||||
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
Qwen3-14B is a Qwen3 dense model with hybrid thinking and non-thinking modes.
|
||||
Intel Xeon 6 CPU serving was validated with BF16.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Hardware: Intel Xeon 6 CPUs
|
||||
- vLLM >= 0.8.5
|
||||
|
||||
### pip (Intel Xeon 6 CPUs)
|
||||
|
||||
Follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels)
|
||||
installation instructions.
|
||||
|
||||
### Docker (Intel Xeon 6 CPUs)
|
||||
|
||||
```bash
|
||||
docker pull vllm/vllm-openai-cpu:latest-x86_64
|
||||
```
|
||||
|
||||
## Intel Xeon 6
|
||||
|
||||
Choose tensor parallelism based on the system topology; the recipe does not
|
||||
prescribe a fixed TP/DP layout or hard-code NUMA node IDs.
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-14B
|
||||
```
|
||||
|
||||
Docker:
|
||||
|
||||
```bash
|
||||
docker run \
|
||||
--privileged --ipc=host -p 8000:8000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai-cpu:latest-x86_64 Qwen/Qwen3-14B \
|
||||
--tensor-parallel-size 1 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser hermes
|
||||
```
|
||||
|
||||
## Configuration Notes
|
||||
|
||||
- TP/DP values and explicit CPU bindings are intentionally not prescribed.
|
||||
- The `qwen3` reasoning parser is available in vLLM 0.9.0 and later.
|
||||
|
||||
## Runtime and Platform Tuning
|
||||
|
||||
The following settings are intentionally not prescribed as portable CPU model
|
||||
defaults because they depend on the workload, hardware, or runtime environment:
|
||||
|
||||
- `--max-num-batched-tokens`: scheduler/throughput tuning.
|
||||
- `--max-num-seqs`: concurrency and scheduler-capacity tuning.
|
||||
- `--gpu-memory-utilization`: platform memory-budget tuning.
|
||||
- `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning.
|
||||
- `VLLM_ENGINE_ITERATION_TIMEOUT_S`: operational runtime timeout.
|
||||
|
||||
These settings may still appear in validated hardware-specific overrides. Tune
|
||||
them at deployment time based on platform resources, workload shape, and
|
||||
latency/throughput goals.
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/Qwen/Qwen3-14B)
|
||||
- [vLLM CPU installation](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/)
|
||||
@@ -0,0 +1,181 @@
|
||||
meta:
|
||||
title: "Qwen3-235B-A22B-Instruct"
|
||||
slug: "qwen3-235b-a22b-instruct-2507"
|
||||
provider: "Qwen"
|
||||
description: "Flagship Qwen3 MoE instruct model with 235B total and 22B active parameters, tuned for high-quality text generation."
|
||||
date_added: 2025-10-15
|
||||
date_updated: 2026-04-17
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "Verified on 4x/8x H200, MI300X/MI325X/MI355X nodes (BF16 and FP8)"
|
||||
related_recipes: []
|
||||
hardware:
|
||||
h200: verified
|
||||
mi300x: verified
|
||||
mi325x: verified
|
||||
mi355x: verified
|
||||
|
||||
model:
|
||||
model_id: "Qwen/Qwen3-235B-A22B-Instruct-2507"
|
||||
min_vllm_version: "0.10.0"
|
||||
architecture: moe
|
||||
parameter_count: "235B"
|
||||
active_parameters: "22B"
|
||||
context_length: 262144
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with Hermes-compatible parser"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "hermes"
|
||||
|
||||
spec_decoding:
|
||||
description: "Eagle3 speculative decoding with externally trained draft model; the served checkpoint is unchanged."
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"model": "RedHatAI/Qwen3-235B-A22B-Instruct-2507-speculator.eagle3", "num_speculative_tokens": 3, "method": "eagle3"}'
|
||||
|
||||
opt_in_features:
|
||||
- spec_decoding
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 564
|
||||
description: "Full precision BF16 — requires 4x H200 or 8x MI300X/MI325X/MI355X"
|
||||
fp8:
|
||||
model_id: "Qwen/Qwen3-235B-A22B-FP8"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 240
|
||||
# FP8 block_n=128 + moe_intermediate_size=1536 → TP must divide 12; cap at 4.
|
||||
tp: 4
|
||||
description: "Qwen official FP8 checkpoint for improved efficiency on SM90+"
|
||||
nvfp4:
|
||||
model_id: "nvidia/Qwen3-235B-A22B-Instruct-2507-NVFP4"
|
||||
precision: nvfp4
|
||||
vram_minimum_gb: 141
|
||||
description: "NVIDIA NVFP4 quantized weights for Blackwell GPUs"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
|
||||
redhat_nvfp4:
|
||||
model_id: "RedHatAI/Qwen3-235B-A22B-Instruct-2507-NVFP4"
|
||||
precision: nvfp4
|
||||
label: "NVFP4 (RedHat)"
|
||||
vram_minimum_gb: 152
|
||||
description: "RedHatAI NVFP4 quantization"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- single_node_tep
|
||||
- single_node_dep
|
||||
- multi_node_tp
|
||||
- multi_node_tp_pp
|
||||
- multi_node_dep
|
||||
- multi_node_tep
|
||||
- pd_cluster
|
||||
|
||||
kv_offload_support:
|
||||
offloading_cpu: verified
|
||||
offloading_fs: verified
|
||||
|
||||
hardware_overrides: {}
|
||||
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[Qwen3](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507) is the flagship instruct MoE model in the Qwen3 series, with 235B total parameters and 22B active parameters. This guide covers deploying the model efficiently with vLLM on NVIDIA and AMD GPUs.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### NVIDIA CUDA (pip)
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend=auto
|
||||
```
|
||||
|
||||
### AMD ROCm (pip)
|
||||
> Note: The vLLM wheel for ROCm requires Python 3.12, ROCm 7.0, and glibc >= 2.35. Use the Docker flow if your environment is incompatible.
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.14.1/rocm700
|
||||
```
|
||||
|
||||
## Deployment Configurations
|
||||
|
||||
### BF16 on MI300X/MI325X/MI355X (4 GPUs)
|
||||
|
||||
```bash
|
||||
HIP_VISIBLE_DEVICES="4,5,6,7" \
|
||||
VLLM_ROCM_USE_AITER=1 \
|
||||
VLLM_ROCM_USE_AITER_MHA=0 \
|
||||
SAFETENSORS_FAST_GPU=1 \
|
||||
vllm serve Qwen/Qwen3-235B-A22B \
|
||||
--trust-remote-code \
|
||||
-tp 4 \
|
||||
--swap-space 32 \
|
||||
--distributed-executor-backend mp \
|
||||
--max-num-batched-tokens 32768 \
|
||||
--max-model-len 32768 \
|
||||
--no-enable-prefix-caching \
|
||||
--gpu-memory-utilization 0.8
|
||||
```
|
||||
|
||||
### FP8 on MI300X/MI325X/MI355X (4 GPUs)
|
||||
|
||||
```bash
|
||||
HIP_VISIBLE_DEVICES="4,5,6,7" \
|
||||
VLLM_ROCM_USE_AITER=1 \
|
||||
VLLM_ROCM_USE_AITER_MHA=0 \
|
||||
SAFETENSORS_FAST_GPU=1 \
|
||||
vllm serve Qwen/Qwen3-235B-A22B-FP8 \
|
||||
--trust-remote-code \
|
||||
-tp 4 \
|
||||
--swap-space 16 \
|
||||
--distributed-executor-backend mp \
|
||||
--max-num-batched-tokens 32768 \
|
||||
--max-model-len 32768 \
|
||||
--no-enable-prefix-caching \
|
||||
--gpu-memory-utilization 0.8
|
||||
```
|
||||
|
||||
### TPU Deployment
|
||||
|
||||
- [Qwen3-32B on Trillium (v6e)](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Qwen3)
|
||||
- [Qwen3-4B on Trillium (v6e)](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Qwen3)
|
||||
|
||||
## Client Usage
|
||||
|
||||
```bash
|
||||
vllm bench serve \
|
||||
--model "Qwen/Qwen3-235B-A22B-FP8" \
|
||||
--dataset-name random \
|
||||
--random-input-len 8192 \
|
||||
--random-output-len 1024 \
|
||||
--request-rate 10000 \
|
||||
--num-prompts 16 \
|
||||
--ignore-eos \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- Use `--max-num-batched-tokens` and `--max-model-len` to fit memory constraints on smaller nodes.
|
||||
- If OOM at startup, lower `--gpu-memory-utilization` (e.g. 0.8) and disable prefix caching.
|
||||
- AMD builds require `VLLM_ROCM_USE_AITER=1` for best performance.
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507)
|
||||
- [FP8 checkpoint](https://huggingface.co/Qwen/Qwen3-235B-A22B-FP8)
|
||||
- [vLLM documentation](https://docs.vllm.ai/)
|
||||
@@ -0,0 +1,229 @@
|
||||
meta:
|
||||
title: "Qwen3-30B-A3B"
|
||||
slug: "qwen3-30b-a3b"
|
||||
provider: "Qwen"
|
||||
description: "Qwen3 MoE model with 30.5B total and 3.3B active parameters, validated for BF16 serving on Intel Xeon 6 and Intel Arc Pro B70 and W8A8 serving on Ascend A3."
|
||||
date_added: 2026-08-14
|
||||
date_updated: 2026-09-23
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "Validated on Intel Xeon 6 (GNR), Ascend A3 W8A8 and 4x Intel Arc Pro B70 with BF16 serving"
|
||||
related_recipes:
|
||||
- "Qwen/Qwen3-235B-A22B-Instruct-2507"
|
||||
hardware:
|
||||
xeon6: verified
|
||||
ascend_910c: verified
|
||||
arc_pro_b70: verified
|
||||
|
||||
model:
|
||||
model_id: "Qwen/Qwen3-30B-A3B"
|
||||
min_vllm_version: "0.8.5"
|
||||
architecture: moe
|
||||
parameter_count: "30.5B"
|
||||
active_parameters: "3.3B"
|
||||
context_length: 40960
|
||||
base_args: []
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with the Hermes parser"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "hermes"
|
||||
reasoning:
|
||||
description: "Enable Qwen3 thinking mode with the Qwen3 reasoning parser"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "qwen3"
|
||||
long_context:
|
||||
description: "YaRN RoPE scaling to the model card's validated 131,072-token context. Static YaRN holds the scaling factor constant regardless of input length, so leave this off unless you actually serve long prompts — it can cost quality below 32K."
|
||||
args:
|
||||
- "--hf-overrides"
|
||||
- '{"max_position_embeddings":131072,"rope_parameters":{"rope_type":"yarn","rope_theta":1000000,"factor":4.0,"original_max_position_embeddings":32768}}'
|
||||
- "--max-model-len"
|
||||
- "131072"
|
||||
|
||||
opt_in_features:
|
||||
- long_context
|
||||
|
||||
hardware_opt_in_features:
|
||||
ascend_910c:
|
||||
- tool_calling
|
||||
- reasoning
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 74
|
||||
description: "Full precision BF16"
|
||||
supported_hardware: [xeon6, arc_pro_b70]
|
||||
ascend_w8a8:
|
||||
model_id: "Eco-Tech/Qwen3-30B-A3B-w8a8"
|
||||
precision: int8
|
||||
label: "W8A8 (Ascend A3)"
|
||||
vram_minimum_gb: 37
|
||||
description: "ModelSlim INT8 W8A8 checkpoint for Ascend 910C, verified with TP4 + expert parallelism."
|
||||
supported_hardware: [ascend_910c]
|
||||
tp: 4
|
||||
min_vllm_version: "0.23.0"
|
||||
extra_args:
|
||||
- "--trust-remote-code"
|
||||
- "--max-num-seqs"
|
||||
- "100"
|
||||
- "--max-model-len"
|
||||
- "40960"
|
||||
- "--max-num-batched-tokens"
|
||||
- "16384"
|
||||
- "--enable-expert-parallel"
|
||||
- "--quantization"
|
||||
- "ascend"
|
||||
- "--distributed_executor_backend"
|
||||
- "mp"
|
||||
- "--no-enable-prefix-caching"
|
||||
- "--async-scheduling"
|
||||
- "--compilation-config"
|
||||
- '{"cudagraph_mode": "FULL_DECODE_ONLY"}'
|
||||
- "--additional-config"
|
||||
- '{"enable_flashcomm1": true, "weight_nz_mode": 2}'
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.95"
|
||||
extra_env:
|
||||
VLLM_USE_MODELSCOPE: "True"
|
||||
HCCL_OP_EXPANSION_MODE: "AIV"
|
||||
HCCL_BUFFSIZE: "1024"
|
||||
OMP_PROC_BIND: "false"
|
||||
OMP_NUM_THREADS: "1"
|
||||
TASK_QUEUE_ENABLE: "1"
|
||||
PYTORCH_NPU_ALLOC_CONF: "expandable_segments:True"
|
||||
VLLM_USE_RUST_FRONTEND: "1"
|
||||
hardware_overrides:
|
||||
ascend_910c:
|
||||
docker_image: "swr.cn-southwest-2.myhuaweicloud.com/base_image/ascend-ci/vllm-ascend:v0.23.0-a3"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
|
||||
hardware_overrides:
|
||||
cpu:
|
||||
extra_args:
|
||||
- "--max-num-batched-tokens"
|
||||
- "16384"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.8"
|
||||
xpu:
|
||||
extra_args:
|
||||
- "--block-size"
|
||||
- "64"
|
||||
- "--enforce-eager"
|
||||
- "--no-enable-prefix-caching"
|
||||
- "--disable-sliding-window"
|
||||
- "--max-model-len"
|
||||
- "9472"
|
||||
- "--max-num-batched-tokens"
|
||||
- "8192"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.9"
|
||||
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
Qwen3-30B-A3B is a mixture-of-experts Qwen3 model with 30.5B total parameters
|
||||
and 3.3B active parameters per token. This recipe includes an Intel Xeon 6 CPU
|
||||
configuration validated with BF16 serving and an Ascend A3 W8A8 variant.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Hardware: 4x Xeon 6 CPUs, or 4x Intel Arc Pro B70 (32 GB per card).
|
||||
- vLLM >= 0.8.5
|
||||
|
||||
### pip (Intel Xeon 6 CPUs)
|
||||
|
||||
For Intel and AMD x86 CPUs, follow the
|
||||
[CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels)
|
||||
installation instructions.
|
||||
|
||||
### Docker (Intel Xeon 6 CPUs)
|
||||
|
||||
```bash
|
||||
docker pull vllm/vllm-openai-cpu:latest-x86_64
|
||||
```
|
||||
|
||||
### Docker (Intel Arc Pro B70)
|
||||
|
||||
```bash
|
||||
docker pull vllm/vllm-openai-xpu:latest
|
||||
```
|
||||
|
||||
## Intel Xeon 6
|
||||
|
||||
Choose the tensor parallel size based on the system topology and deployment
|
||||
requirements. Add `--tensor-parallel-size <N>` when needed; the recipe does
|
||||
not prescribe a Xeon 6 TP value or hard-code NUMA node IDs.
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-30B-A3B \
|
||||
--max-num-batched-tokens 16384 \
|
||||
--gpu-memory-utilization 0.8
|
||||
```
|
||||
|
||||
Docker:
|
||||
|
||||
```bash
|
||||
docker run \
|
||||
--privileged --ipc=host -p 8000:8000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai-cpu:latest-x86_64 Qwen/Qwen3-30B-A3B \
|
||||
--tensor-parallel-size 1 \
|
||||
--max-num-batched-tokens 16384 \
|
||||
--gpu-memory-utilization 0.8 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser hermes \
|
||||
--reasoning-parser qwen3
|
||||
```
|
||||
|
||||
## Intel Arc Pro B70 (XPU)
|
||||
|
||||
Validated on 4x Intel Arc Pro B70 (32 GB per card) with the official vLLM XPU
|
||||
image `vllm/vllm-openai-xpu:latest`, BF16, TP=4.
|
||||
|
||||
```bash
|
||||
docker run --device /dev/dri \
|
||||
-v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
|
||||
--privileged --ipc=host -p 8000:8000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai-xpu:latest \
|
||||
Qwen/Qwen3-30B-A3B \
|
||||
--tensor-parallel-size 4 \
|
||||
--block-size 64 \
|
||||
--enforce-eager \
|
||||
--no-enable-prefix-caching \
|
||||
--disable-sliding-window \
|
||||
--max-model-len 9472 \
|
||||
--max-num-batched-tokens 8192 \
|
||||
--gpu-memory-utilization 0.9
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
|
||||
response = client.chat.completions.create(
|
||||
model="Qwen/Qwen3-30B-A3B",
|
||||
messages=[{"role": "user", "content": "Explain tensor parallelism briefly."}],
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/Qwen/Qwen3-30B-A3B)
|
||||
- [Ascend A3 W8A8 checkpoint](https://www.modelscope.cn/models/Eco-Tech/Qwen3-30B-A3B-w8a8)
|
||||
- [vLLM Ascend](https://github.com/vllm-project/vllm-ascend)
|
||||
- [vLLM CPU installation](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/)
|
||||
@@ -0,0 +1,249 @@
|
||||
meta:
|
||||
title: "Qwen3-32B"
|
||||
slug: "qwen3-32b"
|
||||
provider: "Qwen"
|
||||
description: "Qwen3 32B dense model with hybrid thinking/non-thinking modes — verified on TPU v6e (Trillium)."
|
||||
date_added: 2025-10-15
|
||||
date_updated: 2026-09-10
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "Verified on TPU v6e (Trillium) and v7 (Ironwood) with BF16"
|
||||
related_recipes:
|
||||
- "Qwen/Qwen3-4B"
|
||||
hardware:
|
||||
trillium: verified
|
||||
ironwood: verified
|
||||
arc_pro_b60: verified
|
||||
arc_pro_b70: verified
|
||||
|
||||
model:
|
||||
model_id: "Qwen/Qwen3-32B"
|
||||
min_vllm_version: "0.8.5"
|
||||
architecture: dense
|
||||
parameter_count: "32B"
|
||||
active_parameters: "32B"
|
||||
context_length: 40960
|
||||
base_args: []
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with the Hermes parser"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "hermes"
|
||||
reasoning:
|
||||
description: "Enable Qwen3 thinking mode (chain-of-thought) with the Qwen3 reasoning parser"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "qwen3"
|
||||
long_context:
|
||||
description: "YaRN RoPE scaling to the model card's validated 131,072-token context. Static YaRN holds the scaling factor constant regardless of input length, so leave this off unless you actually serve long prompts — it can cost quality below 32K."
|
||||
args:
|
||||
- "--hf-overrides"
|
||||
- '{"max_position_embeddings":131072,"rope_parameters":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":32768}}'
|
||||
- "--max-model-len"
|
||||
- "131072"
|
||||
|
||||
spec_decoding:
|
||||
description: "Eagle3 speculative decoding with externally trained draft model; the served checkpoint is unchanged."
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"model": "RedHatAI/Qwen3-32B-speculator.eagle3", "num_speculative_tokens": 3, "method": "eagle3"}'
|
||||
|
||||
opt_in_features:
|
||||
- spec_decoding
|
||||
- long_context
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 77
|
||||
description: "Full precision BF16 — TPU v6e 2x2 (4 chips, TP=4) or a single 80GB+ GPU"
|
||||
fp8:
|
||||
model_id: "Qwen/Qwen3-32B-FP8"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 39
|
||||
description: "Qwen official FP8 checkpoint — single 40 GB GPU"
|
||||
awq:
|
||||
model_id: "Qwen/Qwen3-32B-AWQ"
|
||||
precision: int4
|
||||
vram_minimum_gb: 20
|
||||
description: "AWQ 4-bit quantized weights — fits on a single 24 GB GPU"
|
||||
extra_args:
|
||||
- "--quantization"
|
||||
- "awq"
|
||||
|
||||
redhat_fp8:
|
||||
model_id: "RedHatAI/Qwen3-32B-FP8-dynamic"
|
||||
precision: fp8
|
||||
label: "FP8 Dynamic (RedHat)"
|
||||
vram_minimum_gb: 39
|
||||
description: "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
|
||||
redhat_fp8_block:
|
||||
model_id: "RedHatAI/Qwen3-32B-FP8-block"
|
||||
precision: fp8
|
||||
label: "FP8 Block (RedHat)"
|
||||
vram_minimum_gb: 39
|
||||
description: "RedHatAI FP8 block quantization"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
|
||||
nvfp4:
|
||||
model_id: "RedHatAI/Qwen3-32B-NVFP4"
|
||||
precision: nvfp4
|
||||
vram_minimum_gb: 22
|
||||
description: "RedHatAI NVFP4 quantization"
|
||||
|
||||
redhat_nvfp4:
|
||||
model_id: "RedHatAI/Qwen3-32B-NVFP4A16"
|
||||
precision: nvfp4
|
||||
label: "NVFP4 A16 (RedHat)"
|
||||
vram_minimum_gb: 22
|
||||
description: "RedHatAI NVFP4 with BF16 activations"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
|
||||
kv_offload_support:
|
||||
offloading_cpu: verified
|
||||
offloading_fs: verified
|
||||
|
||||
hardware_overrides:
|
||||
xpu:
|
||||
extra_args:
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.90"
|
||||
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
Qwen3-32B is the flagship dense model in the Qwen3 series, with hybrid thinking / non-thinking modes. BF16 is the training precision and gives the best accuracy.
|
||||
|
||||
Toggle thinking mode per request with `enable_thinking` in `chat_template_kwargs`, or set the **Reasoning** feature to add the Qwen3 reasoning parser so `<think>` content is split into a separate field.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### NVIDIA
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend auto
|
||||
```
|
||||
|
||||
### TPU (Trillium / v6e)
|
||||
Use the official vLLM TPU image. The 32B model is served on a `v6e-4` slice (`--topology 2x2`, 4 chips, TP=4).
|
||||
```bash
|
||||
export HF_HOME=/dev/shm
|
||||
export HF_TOKEN=<your HF token>
|
||||
```
|
||||
|
||||
### Intel XPU (B60 / B70)
|
||||
Use the official vLLM XPU image.
|
||||
```bash
|
||||
docker pull vllm/vllm-openai-xpu:latest
|
||||
```
|
||||
|
||||
## Deployment Configurations
|
||||
|
||||
### TPU v6e (Trillium, 2x2 slice, TP=4)
|
||||
Verified end-to-end on a 4-chip v6e slice.
|
||||
```bash
|
||||
export HF_HOME=/dev/shm
|
||||
export HF_TOKEN=<your HF token>
|
||||
export TP=4
|
||||
export MAX_MODEL_LEN=4096
|
||||
vllm serve Qwen/Qwen3-32B \
|
||||
--seed 42 \
|
||||
--gpu-memory-utilization 0.98 \
|
||||
--max-num-batched-tokens 2048 \
|
||||
--max-num-seqs 256 \
|
||||
--tensor-parallel-size $TP \
|
||||
--max-model-len $MAX_MODEL_LEN
|
||||
```
|
||||
|
||||
### TPU v7 (Ironwood, single host, TP=2)
|
||||
Verified end-to-end on a single Ironwood host (`tpu7x-standard-1t`, topology `1x1x1`). Served via the OpenAI API server entrypoint.
|
||||
```bash
|
||||
export HF_HOME=/data
|
||||
export HUGGING_FACE_HUB_TOKEN=<your HF token>
|
||||
export LAYOUT_Q_PROJ_AS_NDH=1
|
||||
export USE_BATCHED_RPA_KERNEL=1
|
||||
vllm serve Qwen/Qwen3-32B \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000 \
|
||||
--seed 42 \
|
||||
--tensor-parallel-size 2 \
|
||||
--data-parallel-size 1 \
|
||||
--max-model-len 9216 \
|
||||
--max-num-batched-tokens 4096 \
|
||||
--max-num-seqs 135 \
|
||||
--gpu-memory-utilization 0.98 \
|
||||
--block-size 256 \
|
||||
--kv-cache-dtype fp8 \
|
||||
--no-enable-prefix-caching \
|
||||
--async-scheduling
|
||||
```
|
||||
> Ironwood support ships in the nightly TPU image (e.g. `vllm/vllm-tpu:nightly-20260330-2f76400-8c0b626`).
|
||||
|
||||
### Single node (BF16, TP=2)
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-32B \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000 \
|
||||
--tensor-parallel-size 2
|
||||
```
|
||||
|
||||
### Intel XPU (B60, B70)
|
||||
|
||||
Validated on 4× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the
|
||||
official vLLM XPU image `vllm/vllm-openai-xpu:latest`.
|
||||
|
||||
```bash
|
||||
docker run --device /dev/dri \
|
||||
-v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
|
||||
--privileged --ipc=host -p 8000:8000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai-xpu:latest \
|
||||
Qwen/Qwen3-32B \
|
||||
--gpu-memory-utilization 0.90 \
|
||||
--tensor-parallel-size 4 \
|
||||
--max-model-len 8192
|
||||
```
|
||||
|
||||
## Configuration Tips
|
||||
|
||||
- `--max-model-len 4096` keeps memory in check on a 4-chip v6e slice; native context is 40K — raise it if you have headroom.
|
||||
- vLLM uses 90% of device memory by default; the TPU recipe pushes `--gpu-memory-utilization 0.98` to maximize KV cache.
|
||||
- `--max-num-batched-tokens` and `--max-num-seqs` tune batch throughput vs latency.
|
||||
- For thinking mode Qwen recommends `temperature=0.6, top_p=0.95, top_k=20`; for non-thinking, `temperature=0.7, top_p=0.8`.
|
||||
|
||||
## Benchmarking
|
||||
|
||||
Launch the server with `--no-enable-prefix-caching` for consistent measurements.
|
||||
|
||||
```bash
|
||||
vllm bench serve \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000 \
|
||||
--model Qwen/Qwen3-32B \
|
||||
--dataset-name random \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 128
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/Qwen/Qwen3-32B)
|
||||
- [Qwen3 on Trillium (v6e)](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Qwen3)
|
||||
- [Qwen3-32B on Ironwood (v7)](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/ironwood/vLLM/Qwen3-32B)
|
||||
@@ -0,0 +1,175 @@
|
||||
meta:
|
||||
title: "Qwen3-4B"
|
||||
slug: "qwen3-4b"
|
||||
provider: "Qwen"
|
||||
description: "Qwen3 4B dense model with hybrid thinking/non-thinking modes — fits on a single TPU v6e chip, one GPU or one Xeon 6 NUMA node."
|
||||
date_added: 2025-10-15
|
||||
date_updated: 2026-09-12
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "Verified on TPU v6e (Trillium) with BF16 on a single chip"
|
||||
related_recipes:
|
||||
- "Qwen/Qwen3-32B"
|
||||
hardware:
|
||||
trillium: verified
|
||||
xeon6: verified
|
||||
|
||||
model:
|
||||
model_id: "Qwen/Qwen3-4B"
|
||||
min_vllm_version: "0.8.5"
|
||||
architecture: dense
|
||||
parameter_count: "4B"
|
||||
active_parameters: "4B"
|
||||
context_length: 40960
|
||||
base_args: []
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with the Hermes parser"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "hermes"
|
||||
reasoning:
|
||||
description: "Enable Qwen3 thinking mode (chain-of-thought) with the Qwen3 reasoning parser"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "qwen3"
|
||||
long_context:
|
||||
description: "YaRN RoPE scaling to the model card's validated 131,072-token context. Static YaRN holds the scaling factor constant regardless of input length, so leave this off unless you actually serve long prompts — it can cost quality below 32K."
|
||||
args:
|
||||
- "--hf-overrides"
|
||||
- '{"max_position_embeddings":131072,"rope_parameters":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":32768}}'
|
||||
- "--max-model-len"
|
||||
- "131072"
|
||||
|
||||
opt_in_features:
|
||||
- long_context
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 10
|
||||
description: "Full precision BF16 — fits on a single TPU v6e chip or one 16GB+ GPU or one Xeon 6 NUMA node"
|
||||
supported_hardware: [trillium, xeon6]
|
||||
fp8:
|
||||
model_id: "Qwen/Qwen3-4B-FP8"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 5
|
||||
description: "Qwen official FP8 checkpoint"
|
||||
supported_hardware: [trillium]
|
||||
|
||||
redhat_fp8:
|
||||
model_id: "RedHatAI/Qwen3-4B-FP8-dynamic"
|
||||
precision: fp8
|
||||
label: "FP8 Dynamic (RedHat)"
|
||||
vram_minimum_gb: 5
|
||||
description: "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
|
||||
kv_offload_support:
|
||||
offloading_cpu: verified
|
||||
offloading_fs: verified
|
||||
|
||||
hardware_overrides: {}
|
||||
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
Qwen3-4B is the small dense model in the Qwen3 series, with hybrid thinking / non-thinking modes. At 4B it fits on a single accelerator — one TPU v6e (Trillium) chip, a single 16GB+ GPU or one Xeon 6 NUMA node — making it the cheapest entry point in the family. BF16 is the training precision and gives the best accuracy.
|
||||
|
||||
Toggle thinking mode per request with `enable_thinking` in `chat_template_kwargs`, or set the **Reasoning** feature to add the Qwen3 reasoning parser so `<think>` content is split into a separate field.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### NVIDIA
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend auto
|
||||
```
|
||||
|
||||
### pip (Intel Xeon 6 CPUs)
|
||||
For Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions.
|
||||
|
||||
### TPU (Trillium / v6e)
|
||||
Use the official vLLM TPU image. The 4B model fits on a single v6e chip (`v6e-1`, `--topology 1x1`).
|
||||
```bash
|
||||
export HF_HOME=/dev/shm
|
||||
export HF_TOKEN=<your HF token>
|
||||
```
|
||||
|
||||
## Deployment Configurations
|
||||
|
||||
### TPU v6e (Trillium, single chip, TP=1)
|
||||
Verified end-to-end on a single v6e chip.
|
||||
```bash
|
||||
export HF_HOME=/dev/shm
|
||||
export HF_TOKEN=<your HF token>
|
||||
export TP=1
|
||||
export MAX_MODEL_LEN=4096
|
||||
vllm serve Qwen/Qwen3-4B \
|
||||
--seed 42 \
|
||||
--gpu-memory-utilization 0.98 \
|
||||
--max-num-batched-tokens 1024 \
|
||||
--max-num-seqs 128 \
|
||||
--tensor-parallel-size $TP \
|
||||
--max-model-len $MAX_MODEL_LEN
|
||||
```
|
||||
|
||||
### Single GPU (BF16)
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-4B \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000
|
||||
```
|
||||
|
||||
### Intel Xeon 6 Deployment via Docker
|
||||
|
||||
Launch the x86 CPU vLLM Docker container for `Qwen/Qwen3-4B`:
|
||||
|
||||
```bash
|
||||
docker run \
|
||||
--privileged --ipc=host -p 8000:8000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai-cpu:latest-x86_64 Qwen/Qwen3-4B \
|
||||
--tensor-parallel-size 1 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser hermes \
|
||||
--reasoning-parser qwen3
|
||||
```
|
||||
|
||||
## Configuration Tips
|
||||
|
||||
- `--max-model-len 4096` is the TPU-recipe default; native context is 40K — raise it if you have headroom.
|
||||
- vLLM uses 90% of device memory by default; the TPU recipe pushes `--gpu-memory-utilization 0.98` to maximize KV cache.
|
||||
- For thinking mode Qwen recommends `temperature=0.6, top_p=0.95, top_k=20`; for non-thinking, `temperature=0.7, top_p=0.8`.
|
||||
|
||||
## Benchmarking
|
||||
|
||||
Launch the server with `--no-enable-prefix-caching` for consistent measurements.
|
||||
|
||||
```bash
|
||||
vllm bench serve \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000 \
|
||||
--model Qwen/Qwen3-4B \
|
||||
--dataset-name random \
|
||||
--random-input-len 1024 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 128
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/Qwen/Qwen3-4B)
|
||||
- [Qwen3 on Trillium (v6e)](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Qwen3)
|
||||
@@ -0,0 +1,138 @@
|
||||
meta:
|
||||
title: "Qwen3-8B"
|
||||
slug: "qwen3-8b"
|
||||
provider: "Qwen"
|
||||
description: "Qwen3 8.2B dense model with hybrid thinking/non-thinking modes, validated for BF16 serving on Intel Xeon 6."
|
||||
date_added: 2026-08-14
|
||||
date_updated: 2026-09-12
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "Validated on Intel Xeon 6 (GNR) with BF16 serving"
|
||||
related_recipes:
|
||||
- "Qwen/Qwen3-4B"
|
||||
- "Qwen/Qwen3-32B"
|
||||
hardware:
|
||||
xeon6: verified
|
||||
|
||||
model:
|
||||
model_id: "Qwen/Qwen3-8B"
|
||||
min_vllm_version: "0.8.5"
|
||||
architecture: dense
|
||||
parameter_count: "8.2B"
|
||||
active_parameters: "8.2B"
|
||||
context_length: 40960
|
||||
base_args: []
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with the Hermes parser"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "hermes"
|
||||
reasoning:
|
||||
description: "Enable Qwen3 thinking mode with the Qwen3 reasoning parser"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "qwen3"
|
||||
long_context:
|
||||
description: "YaRN RoPE scaling to the model card's validated 131,072-token context. Static YaRN holds the scaling factor constant regardless of input length, so leave this off unless you actually serve long prompts — it can cost quality below 32K."
|
||||
args:
|
||||
- "--hf-overrides"
|
||||
- '{"max_position_embeddings":131072,"rope_parameters":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":32768}}'
|
||||
- "--max-model-len"
|
||||
- "131072"
|
||||
|
||||
opt_in_features:
|
||||
- long_context
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 20
|
||||
description: "Full precision BF16"
|
||||
supported_hardware: [xeon6]
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
|
||||
hardware_overrides:
|
||||
cpu:
|
||||
extra_args:
|
||||
- "--max-num-batched-tokens"
|
||||
- "16384"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.8"
|
||||
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
Qwen3-8B is a dense Qwen3 model with hybrid thinking and non-thinking modes.
|
||||
This recipe includes an Intel Xeon 6 CPU configuration validated with BF16
|
||||
serving.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Hardware: 2x Xeon 6 CPUs
|
||||
- vLLM >= 0.8.5
|
||||
|
||||
### pip (Intel Xeon 6 CPUs)
|
||||
|
||||
For Intel and AMD x86 CPUs, follow the
|
||||
[CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels)
|
||||
installation instructions.
|
||||
|
||||
### Docker (Intel Xeon 6 CPUs)
|
||||
|
||||
```bash
|
||||
docker pull vllm/vllm-openai-cpu:latest-x86_64
|
||||
```
|
||||
|
||||
## Intel Xeon 6
|
||||
|
||||
Choose the tensor parallel size based on the system topology and deployment
|
||||
requirements. Add `--tensor-parallel-size <N>` when needed; the recipe does
|
||||
not prescribe a Xeon 6 TP value or hard-code NUMA node IDs.
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-8B \
|
||||
--max-num-batched-tokens 16384 \
|
||||
--gpu-memory-utilization 0.8
|
||||
```
|
||||
|
||||
Docker:
|
||||
|
||||
```bash
|
||||
docker run \
|
||||
--privileged --ipc=host -p 8000:8000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai-cpu:latest-x86_64 Qwen/Qwen3-8B \
|
||||
--tensor-parallel-size 1 \
|
||||
--max-num-batched-tokens 16384 \
|
||||
--gpu-memory-utilization 0.8 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser hermes \
|
||||
--reasoning-parser qwen3
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
|
||||
response = client.chat.completions.create(
|
||||
model="Qwen/Qwen3-8B",
|
||||
messages=[{"role": "user", "content": "Explain tensor parallelism briefly."}],
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/Qwen/Qwen3-8B)
|
||||
- [vLLM CPU installation](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/)
|
||||
@@ -0,0 +1,154 @@
|
||||
meta:
|
||||
title: "Qwen3-ASR-1.7B"
|
||||
slug: "qwen3-asr-1.7b"
|
||||
provider: "Qwen"
|
||||
description: "Speech-to-text model supporting 11 languages, multiple accents, and singing voice with customizable text-context prompting."
|
||||
date_added: 2026-01-29
|
||||
date_updated: 2026-04-17
|
||||
tasks:
|
||||
- multimodal
|
||||
performance_headline: "Accurate multilingual ASR, including singing voice; single-GPU serving"
|
||||
related_recipes: []
|
||||
|
||||
model:
|
||||
model_id: "Qwen/Qwen3-ASR-1.7B"
|
||||
min_vllm_version: "0.16.0"
|
||||
architecture: dense
|
||||
parameter_count: "2.3B"
|
||||
active_parameters: "2.3B"
|
||||
context_length: 65536
|
||||
base_args: []
|
||||
base_env: {}
|
||||
|
||||
dependencies:
|
||||
- note: "Audio extras — required for ASR input pre-processing (librosa, soundfile)"
|
||||
command: 'uv pip install -U "vllm[audio]"'
|
||||
|
||||
features: {}
|
||||
|
||||
opt_in_features: []
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 4
|
||||
description: "Full precision BF16 — fits on a single mid-range GPU"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
|
||||
kv_offload_support:
|
||||
offloading_cpu: verified
|
||||
offloading_fs: verified
|
||||
|
||||
hardware_overrides:
|
||||
amd:
|
||||
extra_args: []
|
||||
extra_env:
|
||||
VLLM_ROCM_USE_AITER: "1"
|
||||
SAFETENSORS_FAST_GPU: "1"
|
||||
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
Qwen3-ASR is a speech-to-text model that achieves accurate and robust recognition across 11 languages and multiple accents. It supports prompting the model with text context in any format to produce customized ASR results and performs well on singing-voice recognition. This guide demonstrates how to deploy Qwen3-ASR efficiently with vLLM.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Install vLLM with audio dependencies:
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --pre \
|
||||
--extra-index-url https://wheels.vllm.ai/nightly/cu129 \
|
||||
--extra-index-url https://download.pytorch.org/whl/cu129 \
|
||||
--index-strategy unsafe-best-match
|
||||
uv pip install "vllm[audio]"
|
||||
```
|
||||
|
||||
## Launching with vLLM
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-ASR-1.7B
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
### Chat Completions (OpenAI SDK)
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="Qwen/Qwen3-ASR-1.7B",
|
||||
messages=[{
|
||||
"role": "user",
|
||||
"content": [{
|
||||
"type": "audio_url",
|
||||
"audio_url": {"url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_en.wav"},
|
||||
}],
|
||||
}],
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Transcription API
|
||||
```python
|
||||
import httpx
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||||
audio_url = "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_en.wav"
|
||||
audio_file = httpx.get(audio_url).content
|
||||
|
||||
transcription = client.audio.transcriptions.create(
|
||||
model="Qwen/Qwen3-ASR-1.7B",
|
||||
file=audio_file,
|
||||
)
|
||||
print(transcription.text)
|
||||
```
|
||||
|
||||
### cURL
|
||||
```bash
|
||||
curl http://localhost:8000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"messages": [
|
||||
{"role": "user", "content": [
|
||||
{"type": "audio_url", "audio_url": {"url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_en.wav"}}
|
||||
]}
|
||||
]
|
||||
}'
|
||||
```
|
||||
|
||||
### Offline Inference
|
||||
```python
|
||||
from vllm import LLM, SamplingParams
|
||||
from vllm.assets.audio import AudioAsset
|
||||
|
||||
llm = LLM(model="Qwen/Qwen3-ASR-1.7B")
|
||||
audio_asset = AudioAsset("winning_call")
|
||||
|
||||
conversation = [{
|
||||
"role": "user",
|
||||
"content": [{"type": "audio_url", "audio_url": {"url": audio_asset.url}}],
|
||||
}]
|
||||
|
||||
sampling_params = SamplingParams(temperature=0.01, max_tokens=256)
|
||||
outputs = llm.chat(conversation, sampling_params=sampling_params)
|
||||
print(outputs[0].outputs[0].text)
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- Make sure `vllm[audio]` extras are installed or audio requests will fail.
|
||||
- Use the nightly wheel until Qwen3-ASR support lands in the stable release.
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/Qwen/Qwen3-ASR-1.7B)
|
||||
- [vLLM audio support](https://docs.vllm.ai/en/latest/features/multimodal_inputs.html)
|
||||
@@ -0,0 +1,206 @@
|
||||
meta:
|
||||
title: "Qwen3-Coder-480B-A35B-Instruct"
|
||||
slug: "qwen3-coder-480b-a35b-instruct"
|
||||
provider: "Qwen"
|
||||
description: "Large coder MoE with 480B total / 35B active parameters, strong tool-use and code generation capabilities."
|
||||
date_added: 2025-07-23
|
||||
date_updated: 2026-04-17
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "HumanEval 0.939, MBPP 0.918 (FP8). Recommended FP8 on 8x H200/H20 via DP=8"
|
||||
related_recipes: []
|
||||
hardware:
|
||||
h200: verified
|
||||
mi300x: verified
|
||||
mi325x: verified
|
||||
mi355x: verified
|
||||
ironwood: verified
|
||||
|
||||
model:
|
||||
model_id: "Qwen/Qwen3-Coder-480B-A35B-Instruct"
|
||||
min_vllm_version: "0.10.0"
|
||||
architecture: moe
|
||||
parameter_count: "480B"
|
||||
active_parameters: "35B"
|
||||
context_length: 262144
|
||||
base_args: []
|
||||
base_env: {}
|
||||
|
||||
dependencies:
|
||||
- note: "Optional: opt into DeepGEMM FP8 MoE kernels for extra throughput"
|
||||
command: "export VLLM_USE_DEEP_GEMM=1"
|
||||
optional: true
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with Qwen3 Coder parser"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "qwen3_coder"
|
||||
|
||||
opt_in_features: []
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 1152
|
||||
description: "Full BF16 — 8x H200/H20 (141GB × 8) recommended"
|
||||
fp8:
|
||||
model_id: "Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 576
|
||||
# FP8 block_n=128 + moe_intermediate_size=2560 → TP must divide 20; cap at 4.
|
||||
tp: 4
|
||||
description: "Qwen official FP8 checkpoint — required for DP=8 serving"
|
||||
nvfp4:
|
||||
model_id: "nvidia/Qwen3-Coder-480B-A35B-Instruct-NVFP4"
|
||||
precision: nvfp4
|
||||
vram_minimum_gb: 288
|
||||
description: "NVIDIA NVFP4 quantized weights for Blackwell GPUs"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
|
||||
redhat_fp8:
|
||||
model_id: "RedHatAI/Qwen3-Coder-480B-A35B-Instruct-FP8"
|
||||
precision: fp8
|
||||
label: "FP8 (RedHat)"
|
||||
vram_minimum_gb: 576
|
||||
description: "RedHatAI FP8 quantization"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- single_node_tep
|
||||
- single_node_dep
|
||||
- multi_node_tp
|
||||
- multi_node_tp_pp
|
||||
- multi_node_dep
|
||||
- multi_node_tep
|
||||
- pd_cluster
|
||||
|
||||
hardware_overrides:
|
||||
hopper:
|
||||
extra_args: []
|
||||
extra_env:
|
||||
VLLM_USE_DEEP_GEMM: "1"
|
||||
amd:
|
||||
# Verified on 8× MI300X / MI325X / MI355X (FP8 path uses --data-parallel-size 8
|
||||
# instead of TP8 — swap strategy to DP if deploying the FP8 checkpoint).
|
||||
extra_args: ["--attention-backend", "rocm_aiter_fa", "--moe-backend", "aiter"]
|
||||
extra_env:
|
||||
VLLM_ROCM_USE_AITER: "1"
|
||||
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[Qwen3-Coder](https://github.com/QwenLM/Qwen3-Coder) is an advanced large language model created by the Qwen team. `Qwen3-Coder-480B-A35B-Instruct` is the flagship coder MoE with 480B total / 35B active parameters. vLLM supports it including tool calling; the guide below covers BF16 and FP8 serving on NVIDIA and AMD GPUs.
|
||||
|
||||
TPU support is provided through [vLLM TPU](https://github.com/vllm-project/tpu-inference) with a recipe for [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/ironwood/vLLM/Qwen3-Coder-480B-A35B). The Ironwood docker command rendered by the hardware picker uses the `vllm/vllm-tpu` image; pin to the tag specified by the upstream recipe.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### CUDA
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend auto
|
||||
```
|
||||
|
||||
### ROCm (MI300X, MI325X, MI355X)
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
|
||||
```
|
||||
The ROCm wheel requires Python 3.12, ROCm 7.0, and glibc >= 2.35.
|
||||
|
||||
## Deployment Configurations
|
||||
|
||||
### 8xH200 / 8xH20 BF16
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-Coder-480B-A35B-Instruct \
|
||||
--max-model-len 32000 \
|
||||
--enable-expert-parallel \
|
||||
--tensor-parallel-size 8 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser qwen3_coder
|
||||
```
|
||||
|
||||
### 8xH200 / 8xH20 FP8 (DP=8, recommended)
|
||||
```bash
|
||||
VLLM_USE_DEEP_GEMM=1 vllm serve Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
|
||||
--max-model-len 131072 \
|
||||
--enable-expert-parallel \
|
||||
--data-parallel-size 8 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser qwen3_coder
|
||||
```
|
||||
|
||||
### 8xMI300X/MI325X/MI355X BF16
|
||||
```bash
|
||||
VLLM_ROCM_USE_AITER=1 vllm serve Qwen/Qwen3-Coder-480B-A35B-Instruct \
|
||||
--max-model-len 32000 \
|
||||
--attention-backend rocm_aiter_fa \
|
||||
--moe-backend aiter \
|
||||
--enable-expert-parallel \
|
||||
--tensor-parallel-size 8 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser qwen3_coder
|
||||
```
|
||||
|
||||
### 8xMI300X/MI325X/MI355X FP8
|
||||
```bash
|
||||
VLLM_ROCM_USE_AITER=1 vllm serve Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
|
||||
--trust-remote-code \
|
||||
--max-model-len 131072 \
|
||||
--attention-backend rocm_aiter_fa \
|
||||
--moe-backend aiter \
|
||||
--enable-expert-parallel \
|
||||
--data-parallel-size 8 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser qwen3_coder
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
```bash
|
||||
vllm bench serve \
|
||||
--backend vllm \
|
||||
--model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
|
||||
--endpoint /v1/completions \
|
||||
--dataset-name random \
|
||||
--random-input 2048 \
|
||||
--random-output 1024 \
|
||||
--max-concurrency 10 \
|
||||
--num-prompt 100
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **Context-length OOM**: A single H20 node cannot serve the native 262144 context. Reduce `--max-model-len` or raise `--gpu-memory-utilization`.
|
||||
- **TP=8 failure on FP8**: Expect `ValueError: The output_size of gate's and up's weight = 320 is not divisible by weight quantization block_n = 128.` on FP8 with TP=8. Switch to `--data-parallel-size 8` instead.
|
||||
- **DeepGEMM**: set `VLLM_USE_DEEP_GEMM=1` for faster FP8 matmul. Follow the [setup instructions](https://github.com/vllm-project/vllm/blob/main/benchmarks/kernels/deepgemm/README.md#setup) to install it.
|
||||
- **Tool calling**: add `--tool-call-parser qwen3_coder` as shown above.
|
||||
|
||||
## Evaluation
|
||||
|
||||
| Dataset | Test Type | Pass@1 Score |
|
||||
|------------|--------------------|--------------|
|
||||
| HumanEval | Base tests | 0.939 |
|
||||
| HumanEval+ | Base + extra tests | 0.902 |
|
||||
| MBPP | Base tests | 0.918 |
|
||||
| MBPP+ | Base + extra tests | 0.794 |
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/Qwen/Qwen3-Coder-480B-A35B-Instruct)
|
||||
- [FP8 checkpoint](https://huggingface.co/Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8)
|
||||
- [Qwen3-Coder GitHub](https://github.com/QwenLM/Qwen3-Coder)
|
||||
- [EvalPlus](https://github.com/evalplus/evalplus)
|
||||
- [TPU recipes: Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/ironwood/vLLM/Qwen3-Coder-480B-A35B)
|
||||
@@ -0,0 +1,232 @@
|
||||
meta:
|
||||
title: "Qwen3-Next-80B-A3B-Instruct"
|
||||
slug: "qwen3-next-80b-a3b-instruct"
|
||||
provider: "Qwen"
|
||||
description: "Advanced Qwen3-Next MoE model (80B total / 3B active) with hybrid attention, highly sparse experts, and multi-token prediction."
|
||||
date_added: 2025-09-11
|
||||
date_updated: 2026-09-10
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "Highly sparse MoE with MTP-accelerated decoding, runs on 4x H200/H20/A100/A800"
|
||||
related_recipes: []
|
||||
hardware:
|
||||
h200: verified
|
||||
mi300x: verified
|
||||
mi325x: verified
|
||||
mi355x: verified
|
||||
|
||||
model:
|
||||
model_id: "Qwen/Qwen3-Next-80B-A3B-Instruct"
|
||||
min_vllm_version: "0.17.0"
|
||||
architecture: moe
|
||||
parameter_count: "80B"
|
||||
active_parameters: "3B"
|
||||
context_length: 262144
|
||||
base_args: []
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with Hermes parser"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "hermes"
|
||||
spec_decoding:
|
||||
description: "Multi-token prediction speculative decoding for lower latency"
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"qwen3_next_mtp","num_speculative_tokens":2}'
|
||||
- "--no-enable-chunked-prefill"
|
||||
long_context:
|
||||
description: "YaRN RoPE scaling to the model card's validated 1M-token context (factor 4.0 over the native 262,144). Static YaRN holds the scaling factor constant regardless of input length, so leave this off unless you actually serve long prompts."
|
||||
args:
|
||||
- "--hf-overrides"
|
||||
- '{"rope_parameters":{"rope_type":"yarn","rope_theta":10000000,"partial_rotary_factor":0.25,"factor":4.0,"original_max_position_embeddings":262144}}'
|
||||
- "--max-model-len"
|
||||
- "1010000"
|
||||
env:
|
||||
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
|
||||
|
||||
opt_in_features:
|
||||
- spec_decoding
|
||||
- long_context
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 192
|
||||
description: "Full precision BF16 — fits on 4x H200/H20/A100/A800"
|
||||
fp8:
|
||||
model_id: "Qwen/Qwen3-Next-80B-A3B-Instruct-FP8"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 96
|
||||
description: "Qwen official FP8 checkpoint — recommended on SM90/SM100"
|
||||
nvfp4:
|
||||
model_id: "nvidia/Qwen3-Next-80B-A3B-Instruct-NVFP4"
|
||||
precision: nvfp4
|
||||
vram_minimum_gb: 48
|
||||
description: "NVIDIA NVFP4 quantized weights for Blackwell GPUs"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
|
||||
redhat_fp8:
|
||||
model_id: "RedHatAI/Qwen3-Next-80B-A3B-Instruct-FP8-dynamic"
|
||||
precision: fp8
|
||||
label: "FP8 Dynamic (RedHat)"
|
||||
vram_minimum_gb: 96
|
||||
description: "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
|
||||
redhat_fp8_plain:
|
||||
model_id: "RedHatAI/Qwen3-Next-80B-A3B-Instruct-FP8"
|
||||
precision: fp8
|
||||
label: "FP8 (RedHat)"
|
||||
vram_minimum_gb: 96
|
||||
description: "RedHatAI FP8 quantization"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
|
||||
redhat_fp8_block:
|
||||
model_id: "RedHatAI/Qwen3-Next-80B-A3B-Instruct-FP8-block"
|
||||
precision: fp8
|
||||
label: "FP8 Block (RedHat)"
|
||||
vram_minimum_gb: 96
|
||||
description: "RedHatAI FP8 block quantization"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
|
||||
redhat_nvfp4:
|
||||
model_id: "RedHatAI/Qwen3-Next-80B-A3B-Instruct-NVFP4"
|
||||
precision: nvfp4
|
||||
label: "NVFP4 (RedHat)"
|
||||
vram_minimum_gb: 48
|
||||
description: "RedHatAI NVFP4 quantization"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- single_node_tep
|
||||
- single_node_dep
|
||||
- multi_node_tp
|
||||
- multi_node_tp_pp
|
||||
- multi_node_dep
|
||||
- multi_node_tep
|
||||
|
||||
hardware_overrides:
|
||||
blackwell:
|
||||
extra_args:
|
||||
- "--moe-backend"
|
||||
- "flashinfer_trtllm"
|
||||
- "--attention-backend"
|
||||
- "FLASH_ATTN"
|
||||
extra_env:
|
||||
VLLM_USE_DEEP_GEMM: "0"
|
||||
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[Qwen3-Next](https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd&from=research.latest-advancements-list) is an advanced LLM from the Qwen team featuring:
|
||||
- A hybrid attention mechanism
|
||||
- A highly sparse Mixture-of-Experts (MoE) structure
|
||||
- Training-stability-friendly optimizations
|
||||
- A multi-token prediction mechanism for faster inference
|
||||
|
||||
## Prerequisites
|
||||
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend auto
|
||||
```
|
||||
|
||||
## Deployment Configurations
|
||||
|
||||
Launch on 4x H200/H20 or 4x A100/A800 GPUs.
|
||||
|
||||
### Basic Multi-GPU (BF16)
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-Next-80B-A3B-Instruct \
|
||||
--tensor-parallel-size 4 \
|
||||
--served-model-name qwen3-next \
|
||||
--enable-prefix-caching
|
||||
```
|
||||
|
||||
If you hit `torch.AcceleratorError: CUDA error: an illegal memory access was encountered`, add `--compilation_config.cudagraph_mode=PIECEWISE`.
|
||||
|
||||
### FP8 (SM90/SM100)
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-Next-80B-A3B-Instruct-FP8 \
|
||||
--tensor-parallel-size 4 \
|
||||
--enable-prefix-caching
|
||||
```
|
||||
|
||||
On SM100, accelerate with the FlashInfer TRTLLM MoE kernel:
|
||||
```bash
|
||||
VLLM_USE_DEEP_GEMM=0 \
|
||||
vllm serve Qwen/Qwen3-Next-80B-A3B-Instruct-FP8 \
|
||||
--tensor-parallel-size 4 \
|
||||
--moe-backend flashinfer_trtllm \
|
||||
--attention-backend FLASH_ATTN
|
||||
```
|
||||
|
||||
### MTP (Multi-Token Prediction)
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-Next-80B-A3B-Instruct \
|
||||
--tokenizer-mode auto --gpu-memory-utilization 0.8 \
|
||||
--speculative-config '{"method": "qwen3_next_mtp", "num_speculative_tokens": 2}' \
|
||||
--tensor-parallel-size 4 --no-enable-chunked-prefill
|
||||
```
|
||||
|
||||
### Tool / Function Calling
|
||||
```bash
|
||||
vllm serve ... --tool-call-parser hermes --enable-auto-tool-choice
|
||||
```
|
||||
|
||||
### AMD (MI300X/MI325X/MI355X)
|
||||
```bash
|
||||
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.14.1/rocm700
|
||||
```
|
||||
```bash
|
||||
SAFETENSORS_FAST_GPU=1 \
|
||||
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
|
||||
vllm serve Qwen/Qwen3-Next-80B-A3B-Instruct \
|
||||
--tensor-parallel-size 4 \
|
||||
--max-model-len 32768 \
|
||||
--no-enable-prefix-caching \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
Benchmark:
|
||||
```bash
|
||||
vllm bench serve \
|
||||
--backend vllm \
|
||||
--model Qwen/Qwen3-Next-80B-A3B-Instruct \
|
||||
--served-model-name qwen3-next \
|
||||
--endpoint /v1/completions \
|
||||
--dataset-name random \
|
||||
--random-input 2048 \
|
||||
--random-output 1024 \
|
||||
--max-concurrency 10 \
|
||||
--num-prompt 100
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **Sub-optimal MoE performance warning**: Tune the MoE Triton kernel with [benchmark_moe](https://github.com/vllm-project/vllm/blob/main/benchmarks/kernels/benchmark_moe.py), then set `VLLM_TUNED_CONFIG_FOLDER` to the directory containing the generated config.
|
||||
- **IMA error in DP mode**: add `--compilation_config.cudagraph_mode=PIECEWISE`.
|
||||
- For more parallel topologies, see the [Data Parallel Deployment docs](https://docs.vllm.ai/en/latest/serving/data_parallel_deployment.html).
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct)
|
||||
- [FP8 checkpoint](https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct-FP8)
|
||||
- [Qwen3-Next blog](https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd)
|
||||
@@ -0,0 +1,343 @@
|
||||
meta:
|
||||
title: "Qwen3-VL-235B-A22B-Instruct"
|
||||
slug: "qwen3-vl-235b-a22b-instruct"
|
||||
provider: "Qwen"
|
||||
description: "Qwen3-VL flagship MoE vision-language model with 235B total / 22B active parameters, supporting images, video, and long context."
|
||||
date_added: 2025-09-28
|
||||
date_updated: 2026-04-17
|
||||
tasks:
|
||||
- multimodal
|
||||
- text
|
||||
performance_headline: "Strong on images, video, and text — #1 open model on text on lmarena.ai at release"
|
||||
related_recipes: []
|
||||
hardware:
|
||||
h100: verified
|
||||
h200: verified
|
||||
b200: verified
|
||||
mi300x: verified
|
||||
mi325x: verified
|
||||
mi355x: verified
|
||||
|
||||
model:
|
||||
model_id: "Qwen/Qwen3-VL-235B-A22B-Instruct"
|
||||
min_vllm_version: "0.11.0"
|
||||
architecture: moe
|
||||
parameter_count: "235B"
|
||||
active_parameters: "22B"
|
||||
context_length: 262144
|
||||
base_args: []
|
||||
base_env: {}
|
||||
docker_image:
|
||||
amd: "vllm/vllm-openai-rocm:v0.28.0"
|
||||
|
||||
dependencies:
|
||||
- note: "Recommended for offline multimodal inference (image/video pre-processing helpers)"
|
||||
command: "uv pip install qwen-vl-utils==0.0.14"
|
||||
optional: true
|
||||
|
||||
features:
|
||||
text_only:
|
||||
description: "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
|
||||
args:
|
||||
- "--language-model-only"
|
||||
encoder_parallel:
|
||||
description: "Run the vision encoder in data-parallel mode — avoids TP comm overhead on the small encoder. Mutually exclusive with text_only."
|
||||
args:
|
||||
- "--mm-encoder-tp-mode"
|
||||
- "data"
|
||||
long_context:
|
||||
description: "Extend the 262K context to 1M tokens with YaRN. Static scaling costs quality on
|
||||
short prompts and the wider window inflates the KV budget, so it is opt-in."
|
||||
args:
|
||||
- "--hf-overrides"
|
||||
- '{"text_config":{"rope_parameters":{"rope_type":"yarn","factor":3.0,"original_max_position_embeddings":262144,"mrope_section":[24,20,20],"mrope_interleaved":true}}}'
|
||||
- "--max-model-len"
|
||||
- "1000000"
|
||||
env:
|
||||
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
|
||||
|
||||
opt_in_features:
|
||||
- text_only
|
||||
- long_context
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 564
|
||||
description: "Full BF16 — ideal on H200/B200 with 8 GPUs"
|
||||
fp8:
|
||||
model_id: "Qwen/Qwen3-VL-235B-A22B-Instruct-FP8"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 282
|
||||
# FP8 block_n=128 + moe_intermediate_size=1536 → TP must divide 12; cap at 4.
|
||||
tp: 4
|
||||
description: "Qwen official FP8 checkpoint for optimal H100 memory efficiency"
|
||||
nvfp4:
|
||||
model_id: "nvidia/Qwen3-VL-235B-A22B-Instruct-NVFP4"
|
||||
precision: nvfp4
|
||||
vram_minimum_gb: 141
|
||||
description: "NVIDIA NVFP4 quantized weights for Blackwell GPUs"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
|
||||
redhat_fp8:
|
||||
model_id: "RedHatAI/Qwen3-VL-235B-A22B-Instruct-FP8-dynamic"
|
||||
precision: fp8
|
||||
label: "FP8 Dynamic (RedHat)"
|
||||
vram_minimum_gb: 282
|
||||
description: "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
|
||||
redhat_fp8_block:
|
||||
model_id: "RedHatAI/Qwen3-VL-235B-A22B-Instruct-FP8-block"
|
||||
precision: fp8
|
||||
label: "FP8 Block (RedHat)"
|
||||
vram_minimum_gb: 282
|
||||
description: "RedHatAI FP8 block quantization"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
|
||||
redhat_nvfp4:
|
||||
model_id: "RedHatAI/Qwen3-VL-235B-A22B-Instruct-NVFP4"
|
||||
precision: nvfp4
|
||||
label: "NVFP4 (RedHat)"
|
||||
vram_minimum_gb: 150
|
||||
description: "RedHatAI NVFP4 quantization"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- single_node_tep
|
||||
- single_node_dep
|
||||
- multi_node_tp
|
||||
- multi_node_tp_pp
|
||||
- multi_node_dep
|
||||
- multi_node_tep
|
||||
- pd_cluster
|
||||
|
||||
kv_offload_support:
|
||||
offloading_cpu: verified
|
||||
offloading_fs: verified
|
||||
|
||||
hardware_overrides:
|
||||
hopper:
|
||||
extra_args:
|
||||
- "--mm-encoder-tp-mode"
|
||||
- "data"
|
||||
- "--async-scheduling"
|
||||
extra_env: {}
|
||||
blackwell:
|
||||
extra_args:
|
||||
- "--mm-encoder-tp-mode"
|
||||
- "data"
|
||||
- "--async-scheduling"
|
||||
extra_env: {}
|
||||
amd:
|
||||
extra_args:
|
||||
- "--mm-encoder-tp-mode"
|
||||
- "data"
|
||||
- "--async-scheduling"
|
||||
- "--enable-chunked-prefill"
|
||||
extra_env:
|
||||
VLLM_WORKER_MULTIPROC_METHOD: "spawn"
|
||||
HIP_FORCE_DEV_KERNARG: "1"
|
||||
VLLM_ROCM_USE_AITER: "1"
|
||||
VLLM_ROCM_USE_AITER_MHA: "1"
|
||||
VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT: "1"
|
||||
SAFETENSORS_FAST_GPU: "1"
|
||||
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[Qwen3-VL](https://github.com/QwenLM/Qwen3-VL) is the most powerful vision-language model in the Qwen series, delivering upgrades to text understanding & generation, visual perception & reasoning, extended context, spatial/video dynamics, and agent interaction. The flagship `Qwen3-VL-235B-A22B-Instruct` is a MoE model that requires at least 8 GPUs with ≥80 GB memory each (A100/H100/H200 class).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
|
||||
# Install vLLM >= 0.11.0
|
||||
uv pip install -U vllm
|
||||
|
||||
# Install Qwen-VL utility library (recommended for offline inference)
|
||||
uv pip install qwen-vl-utils==0.0.14
|
||||
```
|
||||
|
||||
## Deployment Configurations
|
||||
|
||||
### H100 (Image + Video, FP8)
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-VL-235B-A22B-Instruct-FP8 \
|
||||
--tensor-parallel-size 8 \
|
||||
--mm-encoder-tp-mode data \
|
||||
--enable-expert-parallel \
|
||||
--async-scheduling
|
||||
```
|
||||
|
||||
### H100 (Image-Only, FP8, TP4)
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-VL-235B-A22B-Instruct-FP8 \
|
||||
--tensor-parallel-size 4 \
|
||||
--limit-mm-per-prompt.video 0 \
|
||||
--async-scheduling \
|
||||
--gpu-memory-utilization 0.95 \
|
||||
--max-num-seqs 128
|
||||
```
|
||||
|
||||
### A100 & H100 (Image-Only, BF16)
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-VL-235B-A22B-Instruct \
|
||||
--tensor-parallel-size 8 \
|
||||
--limit-mm-per-prompt.video 0 \
|
||||
--async-scheduling
|
||||
```
|
||||
|
||||
### A100 & H100 (Image + Video, BF16)
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-VL-235B-A22B-Instruct \
|
||||
--tensor-parallel-size 8 \
|
||||
--max-model-len 128000 \
|
||||
--async-scheduling
|
||||
```
|
||||
|
||||
### H200 & B200
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-VL-235B-A22B-Instruct \
|
||||
--tensor-parallel-size 8 \
|
||||
--mm-encoder-tp-mode data \
|
||||
--async-scheduling
|
||||
```
|
||||
|
||||
### MI300X/MI325X/MI355X (BF16, TP8)
|
||||
```bash
|
||||
export SAFETENSORS_FAST_GPU="1"
|
||||
export HIP_FORCE_DEV_KERNARG="1"
|
||||
export HIP_VISIBLE_DEVICES="0,1,2,3,4,5,6,7"
|
||||
export VLLM_WORKER_MULTIPROC_METHOD="spawn"
|
||||
export VLLM_ROCM_USE_AITER="1"
|
||||
export VLLM_ROCM_USE_AITER_MHA="1"
|
||||
export VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT="1"
|
||||
vllm serve Qwen/Qwen3-VL-235B-A22B-Instruct \
|
||||
--tensor-parallel-size 8 \
|
||||
--mm-encoder-tp-mode data \
|
||||
--enable-expert-parallel \
|
||||
--async-scheduling \
|
||||
--gpu-memory-utilization 0.94 \
|
||||
--max-model-len 32768 \
|
||||
--max-num-seqs 10240 \
|
||||
--max-num-batched-tokens 32768 \
|
||||
--attention-backend ROCM_AITER_FA
|
||||
```
|
||||
|
||||
### MI300X/MI325X/MI355X (FP8, TP8)
|
||||
```bash
|
||||
export SAFETENSORS_FAST_GPU="1"
|
||||
export HIP_FORCE_DEV_KERNARG="1"
|
||||
export HIP_VISIBLE_DEVICES="0,1,2,3,4,5,6,7"
|
||||
export VLLM_WORKER_MULTIPROC_METHOD="spawn"
|
||||
export VLLM_ROCM_USE_AITER="1"
|
||||
export VLLM_ROCM_USE_AITER_MHA="1"
|
||||
export VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT="1"
|
||||
vllm serve Qwen/Qwen3-VL-235B-A22B-Instruct-FP8 \
|
||||
--tensor-parallel-size 8 \
|
||||
--mm-encoder-tp-mode data \
|
||||
--enable-expert-parallel \
|
||||
--async-scheduling \
|
||||
--enable-chunked-prefill \
|
||||
--limit-mm-per-prompt '{"image": 1}' \
|
||||
--gpu-memory-utilization 0.94 \
|
||||
--max-model-len 32768 \
|
||||
--max-num-seqs 10240 \
|
||||
--max-num-batched-tokens 32768 \
|
||||
--kv-cache-dtype fp8 \
|
||||
--compilation-config '{"mode": 3, "cudagraph_mode": "FULL_AND_PIECEWISE", "custom_ops": ["+rms_norm", "+quant_fp8"]}' \
|
||||
--attention-backend ROCM_AITER_FA
|
||||
```
|
||||
|
||||
> The above configuration runs efficiently at high concurrencies (up to 512) for context lengths up to 8k. If TPOT is not satisfactory, lower `--max-num-batched-tokens`. For longer contexts, `--max-num-seqs` and `--max-num-batched-tokens` may need retuning.
|
||||
|
||||
### MI300X / MI355X (FP8, EAGLE3 Speculative Decoding)
|
||||
|
||||
Adds EAGLE3 speculative decoding (`RedHatAI/Qwen3-VL-235B-A22B-Instruct-speculator.eagle3`) on top of the FP8 target. **`--attention-backend ROCM_AITER_FA` is mandatory** — the default `ROCM_ATTN` path makes SD slower than baseline on ROCm. Keep `VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=0` and omit `--async-scheduling` (incompatible with SD).
|
||||
|
||||
```bash
|
||||
export SAFETENSORS_FAST_GPU="1"
|
||||
export HIP_FORCE_DEV_KERNARG="1"
|
||||
export HIP_VISIBLE_DEVICES="0,1,2,3,4,5,6,7"
|
||||
export VLLM_WORKER_MULTIPROC_METHOD="spawn"
|
||||
export VLLM_ROCM_USE_AITER="1"
|
||||
export VLLM_ROCM_USE_AITER_MHA="1"
|
||||
export VLLM_ROCM_USE_AITER_RMSNORM="0"
|
||||
export VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT="0"
|
||||
|
||||
vllm serve Qwen/Qwen3-VL-235B-A22B-Instruct-FP8 \
|
||||
--tensor-parallel-size 8 \
|
||||
--gpu-memory-utilization 0.94 \
|
||||
--distributed-executor-backend mp \
|
||||
--enable-chunked-prefill \
|
||||
--max-model-len 16384 \
|
||||
--max-num-seqs 32 \
|
||||
--max-num-batched-tokens 8192 \
|
||||
--mm-encoder-tp-mode data \
|
||||
--enable-expert-parallel \
|
||||
--attention-backend ROCM_AITER_FA \
|
||||
--compilation-config '{"mode": 3, "cudagraph_mode": "FULL_AND_PIECEWISE", "custom_ops": ["-rms_norm"], "pass_config": {"fuse_norm_quant":false}}' \
|
||||
--speculative-config '{"model": "RedHatAI/Qwen3-VL-235B-A22B-Instruct-speculator.eagle3", "method": "eagle3", "num_speculative_tokens": 4}'
|
||||
```
|
||||
|
||||
## Configuration Tips
|
||||
|
||||
- Use `--limit-mm-per-prompt.video 0` if your server only serves image inputs to save memory.
|
||||
- `OMP_NUM_THREADS=1` reduces CPU contention during preprocessing.
|
||||
- The model's context length is 262K. Reduce `--max-model-len` (e.g. 128000) if you don't need the full range.
|
||||
- `--async-scheduling` overlaps scheduling with decoding for better throughput.
|
||||
- `--mm-encoder-tp-mode data` deploys the vision encoder in data-parallel fashion for better performance.
|
||||
- If your inputs are mostly unique, pass `--mm-processor-cache-gb 0` to skip caching overhead.
|
||||
- Extend context to 1M tokens with the **Long Context** toggle, which applies YaRN via
|
||||
`--hf-overrides` and raises `--max-model-len`. Static YaRN costs quality on short prompts.
|
||||
|
||||
Text-only mode: pass `--limit-mm-per-prompt.video 0 --limit-mm-per-prompt.image 0` to free memory for KV cache when serving text-only traffic.
|
||||
|
||||
## Client Usage
|
||||
|
||||
```python
|
||||
import time
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600)
|
||||
|
||||
messages = [{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "image_url", "image_url": {"url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"}},
|
||||
{"type": "text", "text": "Read all the text in the image."},
|
||||
],
|
||||
}]
|
||||
|
||||
start = time.time()
|
||||
response = client.chat.completions.create(
|
||||
model="Qwen/Qwen3-VL-235B-A22B-Instruct",
|
||||
messages=messages,
|
||||
max_tokens=2048,
|
||||
)
|
||||
print(f"Response costs: {time.time() - start:.2f}s")
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- OOM on A100 / H100 BF16: reduce `--max-model-len`, drop to image-only, or switch to the FP8 checkpoint.
|
||||
- If enabling `--mm-encoder-tp-mode data` raises memory pressure, lower `--gpu-memory-utilization`.
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/Qwen/Qwen3-VL-235B-A22B-Instruct)
|
||||
- [FP8 checkpoint](https://huggingface.co/Qwen/Qwen3-VL-235B-A22B-Instruct-FP8)
|
||||
- [Qwen3-VL GitHub](https://github.com/QwenLM/Qwen3-VL)
|
||||
- [vLLM multimodal inputs guide](https://docs.vllm.ai/en/latest/features/multimodal_inputs.html)
|
||||
@@ -0,0 +1,138 @@
|
||||
meta:
|
||||
title: "Qwen3-VL-30B-A3B-Instruct"
|
||||
slug: "qwen3-vl-30b-a3b-instruct"
|
||||
provider: "Qwen"
|
||||
description: "Qwen3-VL MoE vision-language model with 30B total / 3B active parameters, supporting image, video, and text workloads."
|
||||
date_added: 2026-08-17
|
||||
date_updated: 2026-09-12
|
||||
tasks:
|
||||
- multimodal
|
||||
- text
|
||||
performance_headline: "Validated on Intel Xeon 6 CPU with BF16 serving"
|
||||
related_recipes:
|
||||
- "Qwen/Qwen3-VL-235B-A22B-Instruct"
|
||||
hardware:
|
||||
xeon6: verified
|
||||
|
||||
model:
|
||||
model_id: "Qwen/Qwen3-VL-30B-A3B-Instruct"
|
||||
min_vllm_version: "0.11.0"
|
||||
architecture: moe
|
||||
parameter_count: "30B"
|
||||
active_parameters: "3B"
|
||||
context_length: 262144
|
||||
base_args: []
|
||||
base_env: {}
|
||||
|
||||
dependencies:
|
||||
- note: "Recommended for offline multimodal inference (image/video pre-processing helpers)"
|
||||
command: "uv pip install qwen-vl-utils==0.0.14"
|
||||
optional: true
|
||||
|
||||
features:
|
||||
text_only:
|
||||
description: "Skip loading the vision encoder for text-only workloads — frees memory for KV cache."
|
||||
args:
|
||||
- "--language-model-only"
|
||||
encoder_parallel:
|
||||
description: "Run the vision encoder in data-parallel mode."
|
||||
args:
|
||||
- "--mm-encoder-tp-mode"
|
||||
- "data"
|
||||
|
||||
opt_in_features:
|
||||
- text_only
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 72
|
||||
description: "Full precision BF16"
|
||||
supported_hardware:
|
||||
- xeon6
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- single_node_tep
|
||||
- single_node_dep
|
||||
- multi_node_tp
|
||||
- multi_node_tp_pp
|
||||
- multi_node_dep
|
||||
- multi_node_tep
|
||||
|
||||
kv_offload_support:
|
||||
offloading_cpu: verified
|
||||
offloading_fs: verified
|
||||
|
||||
hardware_overrides: {}
|
||||
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
Qwen3-VL-30B-A3B-Instruct is a multimodal Qwen3-VL MoE model. Intel Xeon 6
|
||||
CPU serving was validated with BF16.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Hardware: Intel Xeon 6 CPUs
|
||||
- vLLM >= 0.11.0
|
||||
|
||||
### pip (Intel Xeon 6 CPUs)
|
||||
|
||||
For Intel and AMD x86 CPUs, follow the
|
||||
[CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels)
|
||||
installation instructions.
|
||||
|
||||
### Docker (Intel Xeon 6 CPUs)
|
||||
|
||||
```bash
|
||||
docker pull vllm/vllm-openai-cpu:latest-x86_64
|
||||
```
|
||||
|
||||
## Intel Xeon 6
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct \
|
||||
--tensor-parallel-size 1 \
|
||||
--mm-encoder-tp-mode data
|
||||
```
|
||||
|
||||
Docker (the image entrypoint is `vllm serve`):
|
||||
|
||||
```bash
|
||||
docker run \
|
||||
--privileged --ipc=host -p 8000:8000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai-cpu:latest-x86_64 Qwen/Qwen3-VL-30B-A3B-Instruct \
|
||||
--tensor-parallel-size 1 \
|
||||
--mm-encoder-tp-mode data
|
||||
```
|
||||
|
||||
The example uses TP=1. Adjust TP/DP based on system topology; NUMA binding is
|
||||
intentionally not hard-coded in the recipe.
|
||||
|
||||
## Configuration Notes
|
||||
|
||||
- TP/DP values and topology-specific CPU binding remain deployment choices.
|
||||
|
||||
## Runtime and Platform Tuning
|
||||
|
||||
The following settings are intentionally not prescribed as portable CPU model
|
||||
defaults because they depend on the workload, hardware, or runtime environment:
|
||||
|
||||
- `--max-num-batched-tokens`: scheduler/throughput tuning.
|
||||
- `--max-num-seqs`: concurrency and scheduler-capacity tuning.
|
||||
- `--gpu-memory-utilization`: platform memory-budget tuning.
|
||||
- `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning.
|
||||
- `VLLM_ENGINE_ITERATION_TIMEOUT_S`: operational runtime timeout.
|
||||
|
||||
These settings may still appear in validated hardware-specific overrides. Tune
|
||||
them at deployment time based on platform resources, workload shape, and
|
||||
latency/throughput goals.
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/Qwen/Qwen3-VL-30B-A3B-Instruct)
|
||||
- [vLLM CPU installation](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/)
|
||||
@@ -0,0 +1,133 @@
|
||||
meta:
|
||||
title: "Qwen3.5-0.8B"
|
||||
slug: "qwen3.5-0.8b"
|
||||
provider: "Qwen"
|
||||
description: "Qwen3.5 tiny dense multimodal model (0.8B) — ultra-low-VRAM / edge serving with 262K context"
|
||||
date_added: 2026-04-22
|
||||
date_updated: 2026-09-14
|
||||
tasks:
|
||||
- multimodal
|
||||
- text
|
||||
performance_headline: "Tiny Qwen3.5 dense for edge / draft-model use"
|
||||
related_recipes:
|
||||
- "Qwen/Qwen3.5-2B"
|
||||
hardware:
|
||||
arc_pro_b60: verified
|
||||
arc_pro_b70: verified
|
||||
|
||||
model:
|
||||
model_id: "Qwen/Qwen3.5-0.8B"
|
||||
min_vllm_version: "0.17.0"
|
||||
default_frontend: rust
|
||||
architecture: dense
|
||||
parameter_count: "0.8B"
|
||||
active_parameters: "0.8B"
|
||||
context_length: 262144
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with Qwen3 Coder parser"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "qwen3_coder"
|
||||
reasoning:
|
||||
description: "Enable chain-of-thought reasoning with Qwen3 parser"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "qwen3"
|
||||
text_only:
|
||||
description: "Skip loading the vision encoder for text-only workloads."
|
||||
args:
|
||||
- "--language-model-only"
|
||||
|
||||
opt_in_features:
|
||||
- text_only
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 2
|
||||
description: "Full precision BF16 — runs on any modern GPU"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
|
||||
hardware_overrides:
|
||||
xpu:
|
||||
extra_args:
|
||||
- "--enforce-eager"
|
||||
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B) is the smallest
|
||||
member of the Qwen3.5 family — same hybrid gated delta networks architecture
|
||||
and 262K context, at a size suited to edge devices or as a draft model for
|
||||
speculative decoding with larger Qwen3.5 checkpoints.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **vLLM version:** >= 0.17.0
|
||||
- **Hardware:** any modern GPU (>=4 GB VRAM) or Intel Arc Pro B60/B70
|
||||
|
||||
### Install vLLM
|
||||
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend=auto
|
||||
```
|
||||
|
||||
### Docker
|
||||
```bash
|
||||
docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70)
|
||||
```
|
||||
|
||||
## Launching the Server
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3.5-0.8B \
|
||||
--max-model-len 262144 \
|
||||
--reasoning-parser qwen3
|
||||
```
|
||||
|
||||
### Docker (Intel XPU B60 / B70)
|
||||
|
||||
Validated on 1× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the
|
||||
official vLLM XPU image `vllm/vllm-openai-xpu:latest`.
|
||||
|
||||
```bash
|
||||
docker run --device /dev/dri \
|
||||
-v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
|
||||
--privileged --ipc=host -p 8000:8000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai-xpu:latest \
|
||||
Qwen/Qwen3.5-0.8B \
|
||||
--reasoning-parser qwen3 \
|
||||
--enforce-eager
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
|
||||
resp = client.chat.completions.create(
|
||||
model="Qwen/Qwen3.5-0.8B",
|
||||
messages=[{"role": "user", "content": "Hi!"}],
|
||||
max_tokens=64,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/Qwen/Qwen3.5-0.8B)
|
||||
- [Base checkpoint](https://huggingface.co/Qwen/Qwen3.5-0.8B-Base)
|
||||
@@ -0,0 +1,270 @@
|
||||
meta:
|
||||
title: "Qwen3.5-122B-A10B"
|
||||
slug: "qwen3.5-122b-a10b"
|
||||
provider: "Qwen"
|
||||
description: "Mid-size Qwen3.5 multimodal MoE (122B total / 10B active) with gated delta networks, 256 experts, and 262K context"
|
||||
date_added: 2026-04-22
|
||||
date_updated: 2026-10-07
|
||||
tasks:
|
||||
- multimodal
|
||||
- text
|
||||
performance_headline: "Qwen3.5 mid-tier MoE — fits on 4x H200 BF16 or 2x H200 FP8"
|
||||
related_recipes:
|
||||
- "Qwen/Qwen3.5-397B-A17B"
|
||||
- "Qwen/Qwen3.5-35B-A3B"
|
||||
hardware:
|
||||
mi325x: verified
|
||||
mi355x: verified
|
||||
|
||||
model:
|
||||
model_id: "Qwen/Qwen3.5-122B-A10B"
|
||||
min_vllm_version: "0.17.0"
|
||||
default_frontend: rust
|
||||
docker_image:
|
||||
# Stable release the AMD numbers in the guide were measured on (2026-10-07).
|
||||
amd: "vllm/vllm-openai-rocm:v0.31.0"
|
||||
architecture: moe
|
||||
parameter_count: "122B"
|
||||
active_parameters: "10B"
|
||||
context_length: 262144
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with Qwen3 Coder parser"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "qwen3_coder"
|
||||
reasoning:
|
||||
description: "Enable chain-of-thought reasoning with Qwen3 parser"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "qwen3"
|
||||
spec_decoding:
|
||||
description: "Multi-token prediction speculative decoding for lower latency"
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"mtp","num_speculative_tokens":1}'
|
||||
text_only:
|
||||
description: "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
|
||||
args:
|
||||
- "--language-model-only"
|
||||
encoder_parallel:
|
||||
description: "Run the vision encoder in data-parallel mode — avoids TP comm overhead on the small encoder. Mutually exclusive with text_only."
|
||||
args:
|
||||
- "--mm-encoder-tp-mode"
|
||||
- "data"
|
||||
long_context:
|
||||
description: "YaRN RoPE scaling past the native 262,144-token window to ~1,010,000 tokens. Static YaRN holds the scaling factor constant regardless of input length, so leave this off unless you actually serve long prompts; for ~524K, halve the factor to 2.0."
|
||||
args:
|
||||
- "--hf-overrides"
|
||||
- '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}'
|
||||
- "--max-model-len"
|
||||
- "1010000"
|
||||
env:
|
||||
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
|
||||
|
||||
opt_in_features:
|
||||
- spec_decoding
|
||||
- text_only
|
||||
- long_context
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 293
|
||||
description: "Full precision BF16 — requires 4x H200 or equivalent"
|
||||
fp8:
|
||||
model_id: "Qwen/Qwen3.5-122B-A10B-FP8"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 147
|
||||
description: "Qwen official FP8 checkpoint — fits on 2x H200"
|
||||
hardware_overrides:
|
||||
amd:
|
||||
# fp8 KV keeps MTP's gain flat across context on ROCm (with a bf16 KV cache
|
||||
# MTP decode slows down as the prompt grows), and it is what the AMD numbers
|
||||
# in the guide were measured with.
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
nvfp4:
|
||||
model_id: "RedHatAI/Qwen3.5-122B-A10B-NVFP4"
|
||||
precision: nvfp4
|
||||
vram_minimum_gb: 74
|
||||
description: "RedHatAI compressed-tensors NVFP4 quantization for Blackwell GPUs"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
gptq_int4:
|
||||
model_id: "Qwen/Qwen3.5-122B-A10B-GPTQ-Int4"
|
||||
precision: int4
|
||||
vram_minimum_gb: 74
|
||||
description: "GPTQ Int4 checkpoint — single-GPU serving on 80GB hardware"
|
||||
|
||||
redhat_fp8:
|
||||
model_id: "RedHatAI/Qwen3.5-122B-A10B-FP8-dynamic"
|
||||
precision: fp8
|
||||
label: "FP8 Dynamic (RedHat)"
|
||||
vram_minimum_gb: 147
|
||||
description: "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- single_node_tep
|
||||
- single_node_dep
|
||||
- multi_node_tp
|
||||
- multi_node_dep
|
||||
- multi_node_tep
|
||||
- pd_cluster
|
||||
|
||||
hardware_overrides:
|
||||
amd:
|
||||
# AITER is opt-in on ROCm and the image does not set it; without these two the
|
||||
# server runs ROCM_ATTN + the Triton FP8 MoE and single-stream decode of this
|
||||
# hybrid GDN model drops 4x from ~200 to ~12k prompt tokens. With AITER on, the
|
||||
# auto-selection still picks ROCM_ATTN, so name the backend. ROCM_AITER_FA rather
|
||||
# than ROCM_AITER_UNIFIED_ATTN: on gfx950 with v0.31.0 the unified backend runs
|
||||
# AITER's new Gluon kernel, ~20% slower and with MTP decode that slows down as
|
||||
# the prompt grows. Verified 2026-10-07 on one MI325X and one MI355X with the FP8
|
||||
# checkpoint (see the AMD section of the guide).
|
||||
extra_args:
|
||||
- "--attention-backend"
|
||||
- "ROCM_AITER_FA"
|
||||
extra_env:
|
||||
VLLM_ROCM_USE_AITER: "1"
|
||||
|
||||
strategy_overrides:
|
||||
pd_cluster:
|
||||
prefill:
|
||||
env:
|
||||
VLLM_SSM_CONV_STATE_LAYOUT: "DS"
|
||||
decode:
|
||||
env:
|
||||
VLLM_SSM_CONV_STATE_LAYOUT: "DS"
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[Qwen3.5-122B-A10B](https://huggingface.co/Qwen/Qwen3.5-122B-A10B) is a mid-tier
|
||||
member of the Qwen3.5 family, sharing the gated delta networks MoE architecture
|
||||
with 122B total parameters and 10B activated per token (256 experts). It is
|
||||
multimodal (vision + text) and natively supports 262K context.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **vLLM version:** >= 0.17.0
|
||||
- **Hardware (BF16):** 4x H200 or 8x H100
|
||||
- **Hardware (FP8):** 2x H200 or 4x H100
|
||||
- **Hardware (Int4):** single 80 GB GPU
|
||||
|
||||
### Install vLLM
|
||||
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend=auto
|
||||
```
|
||||
|
||||
## Launching the Server
|
||||
|
||||
### BF16 on 4xH200 (TP4)
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3.5-122B-A10B \
|
||||
--tensor-parallel-size 4 \
|
||||
--max-model-len 262144 \
|
||||
--reasoning-parser qwen3
|
||||
```
|
||||
|
||||
### FP8 on 2xH200 (TP2)
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3.5-122B-A10B-FP8 \
|
||||
--tensor-parallel-size 2 \
|
||||
--max-model-len 262144 \
|
||||
--reasoning-parser qwen3
|
||||
```
|
||||
|
||||
### Throughput-focused (text-only, EP)
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3.5-122B-A10B-FP8 \
|
||||
-dp 4 --enable-expert-parallel \
|
||||
--language-model-only \
|
||||
--reasoning-parser qwen3 \
|
||||
--enable-prefix-caching
|
||||
```
|
||||
|
||||
### MTP speculative decoding
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3.5-122B-A10B-FP8 \
|
||||
--tensor-parallel-size 2 \
|
||||
--speculative-config '{"method": "mtp", "num_speculative_tokens": 1}' \
|
||||
--reasoning-parser qwen3
|
||||
```
|
||||
|
||||
### AMD (MI325X / MI355X, single GPU, FP8)
|
||||
|
||||
The FP8 checkpoint fits one MI300X-class GPU. Three settings matter on ROCm and none is on by
|
||||
default: `VLLM_ROCM_USE_AITER=1` switches the MoE to the AITER FP8 kernels,
|
||||
`--attention-backend ROCM_AITER_FA` keeps decode flat across context (the auto-selected
|
||||
`ROCM_ATTN` decays with prompt length on this hybrid GDN model), and `--kv-cache-dtype fp8`
|
||||
keeps MTP flat across context too.
|
||||
|
||||
```bash
|
||||
VLLM_ROCM_USE_AITER=1 vllm serve Qwen/Qwen3.5-122B-A10B-FP8 \
|
||||
--attention-backend ROCM_AITER_FA \
|
||||
--kv-cache-dtype fp8 --max-num-seqs 64 \
|
||||
--reasoning-parser qwen3
|
||||
```
|
||||
|
||||
`--max-num-seqs 64` because the default exceeds the Mamba block budget on 192 GB. MTP
|
||||
(`--speculative-config '{"method":"mtp","num_speculative_tokens":1}'`) works with the same
|
||||
flags. Single-stream decode (tokens/s at ~200 / ~2k / ~12k / ~30k prompt tokens, 128 output
|
||||
tokens, prefix caching off), measured 2026-10-07 on `vllm/vllm-openai-rocm:v0.31.0`:
|
||||
|
||||
| | MI325X | MI355X |
|
||||
|---|---|---|
|
||||
| AITER + `ROCM_AITER_FA` | 128 / 127 / 127 / 127 | 143 / 143 / 143 / 143 |
|
||||
| + MTP | 184 / 169 / 172 / 164 | 204 / 193 / 195 / 190 |
|
||||
|
||||
This recipe used `ROCM_AITER_UNIFIED_ATTN` before v0.31.0. Avoid it on MI350X/MI355X with
|
||||
v0.31.0: there it runs AITER's new gfx950 Gluon kernel, which decodes at 110 tokens/s and,
|
||||
with MTP, falls from 200 to 72 tokens/s between ~200 and ~30k prompt tokens
|
||||
([vllm-project/vllm#60408](https://github.com/vllm-project/vllm/issues/60408)). MI325X is
|
||||
unaffected (126 tokens/s; 182 to 162 with MTP).
|
||||
|
||||
## Client Usage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
|
||||
resp = client.chat.completions.create(
|
||||
model="Qwen/Qwen3.5-122B-A10B",
|
||||
messages=[{"role": "user", "content": "Summarize the gated delta networks paper."}],
|
||||
max_tokens=512,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **CUDA graph / Mamba cache size error:** reduce `--max-cudagraph-capture-size`
|
||||
(default 512). See [vLLM PR #34571](https://github.com/vllm-project/vllm/pull/34571).
|
||||
- **Disable reasoning:** add `--default-chat-template-kwargs '{"enable_thinking": false}'`.
|
||||
- **Prefix Caching (Mamba):** currently experimental in "align" mode.
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/Qwen/Qwen3.5-122B-A10B)
|
||||
- [FP8 checkpoint](https://huggingface.co/Qwen/Qwen3.5-122B-A10B-FP8)
|
||||
- [GPTQ-Int4 checkpoint](https://huggingface.co/Qwen/Qwen3.5-122B-A10B-GPTQ-Int4)
|
||||
- [Qwen3.5-397B-A17B recipe](../Qwen3.5-397B-A17B)
|
||||
@@ -0,0 +1,209 @@
|
||||
meta:
|
||||
title: "Qwen3.5-27B"
|
||||
slug: "qwen3.5-27b"
|
||||
provider: "Qwen"
|
||||
description: "Qwen3.5 dense multimodal model (27B) with gated delta networks hybrid attention, MTP, and 262K context"
|
||||
date_added: 2026-04-22
|
||||
date_updated: 2026-09-14
|
||||
tasks:
|
||||
- multimodal
|
||||
- text
|
||||
performance_headline: "Qwen3.5 flagship dense — single-GPU FP8 or 2x GPU or 2-4x Intel Arc Pro B60/B70 BF16"
|
||||
related_recipes:
|
||||
- "Qwen/Qwen3.5-397B-A17B"
|
||||
- "Qwen/Qwen3.5-35B-A3B"
|
||||
- "Qwen/Qwen3.5-9B"
|
||||
hardware:
|
||||
arc_pro_b60: verified
|
||||
arc_pro_b70: verified
|
||||
|
||||
model:
|
||||
model_id: "Qwen/Qwen3.5-27B"
|
||||
min_vllm_version: "0.17.0"
|
||||
default_frontend: rust
|
||||
architecture: dense
|
||||
parameter_count: "27B"
|
||||
active_parameters: "27B"
|
||||
context_length: 262144
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with Qwen3 Coder parser"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "qwen3_coder"
|
||||
reasoning:
|
||||
description: "Enable chain-of-thought reasoning with Qwen3 parser"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "qwen3"
|
||||
spec_decoding:
|
||||
description: "Multi-token prediction speculative decoding for lower latency"
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"mtp","num_speculative_tokens":1}'
|
||||
text_only:
|
||||
description: "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
|
||||
args:
|
||||
- "--language-model-only"
|
||||
encoder_parallel:
|
||||
description: "Run the vision encoder in data-parallel mode — avoids TP comm overhead on the small encoder. Mutually exclusive with text_only."
|
||||
args:
|
||||
- "--mm-encoder-tp-mode"
|
||||
- "data"
|
||||
long_context:
|
||||
description: "YaRN RoPE scaling past the native 262,144-token window to ~1,010,000 tokens. Static YaRN holds the scaling factor constant regardless of input length, so leave this off unless you actually serve long prompts; for ~524K, halve the factor to 2.0."
|
||||
args:
|
||||
- "--hf-overrides"
|
||||
- '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}'
|
||||
- "--max-model-len"
|
||||
- "1010000"
|
||||
env:
|
||||
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
|
||||
|
||||
opt_in_features:
|
||||
- spec_decoding
|
||||
- text_only
|
||||
- long_context
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 65
|
||||
description: "Full precision BF16 — fits on 1x H200, 2x H100, 4x Intel Arc Pro B60, 2x Intel Arc Pro B70"
|
||||
fp8:
|
||||
model_id: "Qwen/Qwen3.5-27B-FP8"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 33
|
||||
description: "Qwen official FP8 checkpoint — single 40 GB GPU"
|
||||
gptq_int4:
|
||||
model_id: "Qwen/Qwen3.5-27B-GPTQ-Int4"
|
||||
precision: int4
|
||||
vram_minimum_gb: 17
|
||||
description: "GPTQ Int4 checkpoint — fits on a single 24 GB GPU"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
|
||||
hardware_overrides:
|
||||
xpu:
|
||||
extra_args:
|
||||
- "--enforce-eager"
|
||||
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[Qwen3.5-27B](https://huggingface.co/Qwen/Qwen3.5-27B) is the flagship dense
|
||||
model of the Qwen3.5 family. It uses the same gated delta networks hybrid
|
||||
attention as its MoE siblings, supports vision+text input, and natively serves
|
||||
262K context. MTP (multi-token prediction) is supported out of the box for
|
||||
low-latency decoding.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **vLLM version:** >= 0.17.0
|
||||
- **Hardware (BF16):** 1x H200, 2x H100, 4x Intel Arc Pro B60, 2x Intel Arc Pro B70
|
||||
- **Hardware (FP8):** single 40 GB GPU (H100/H200/L40S)
|
||||
- **Hardware (Int4):** single 24 GB GPU
|
||||
|
||||
### Install vLLM
|
||||
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend=auto
|
||||
```
|
||||
|
||||
### Docker
|
||||
```bash
|
||||
docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70)
|
||||
```
|
||||
|
||||
## Launching the Server
|
||||
|
||||
### Single-GPU FP8
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3.5-27B-FP8 \
|
||||
--max-model-len 262144 \
|
||||
--reasoning-parser qwen3
|
||||
```
|
||||
|
||||
### BF16 on 2xH100 (TP2)
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3.5-27B \
|
||||
--tensor-parallel-size 2 \
|
||||
--max-model-len 262144 \
|
||||
--reasoning-parser qwen3
|
||||
```
|
||||
|
||||
### MTP speculative decoding
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3.5-27B-FP8 \
|
||||
--speculative-config '{"method": "mtp", "num_speculative_tokens": 1}' \
|
||||
--reasoning-parser qwen3
|
||||
```
|
||||
|
||||
### Text-only (skip vision encoder)
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3.5-27B-FP8 \
|
||||
--language-model-only \
|
||||
--reasoning-parser qwen3 \
|
||||
--enable-prefix-caching
|
||||
```
|
||||
|
||||
### Docker (Intel XPU B60 / B70)
|
||||
|
||||
Validated on 4× Intel Arc Pro B60 or 2x Intel Arc Pro B70 (B60 24 GB, B70 32 GB per card) with the
|
||||
official vLLM XPU image `vllm/vllm-openai-xpu:latest`.
|
||||
|
||||
```bash
|
||||
docker run --device /dev/dri \
|
||||
-v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
|
||||
--privileged --ipc=host -p 8000:8000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai-xpu:latest \
|
||||
Qwen/Qwen3.5-27B \
|
||||
--reasoning-parser qwen3 \
|
||||
--tensor-parallel-size 4 \
|
||||
--max-model-len 8192 \
|
||||
--enforce-eager
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
|
||||
resp = client.chat.completions.create(
|
||||
model="Qwen/Qwen3.5-27B",
|
||||
messages=[{"role": "user", "content": "Write a haiku about gated delta networks."}],
|
||||
max_tokens=256,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **CUDA graph / Mamba cache size error:** reduce `--max-cudagraph-capture-size`
|
||||
(default 512). See [vLLM PR #34571](https://github.com/vllm-project/vllm/pull/34571).
|
||||
- **Disable reasoning:** add `--default-chat-template-kwargs '{"enable_thinking": false}'`.
|
||||
- **Prefix Caching (Mamba):** currently experimental in "align" mode.
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/Qwen/Qwen3.5-27B)
|
||||
- [FP8 checkpoint](https://huggingface.co/Qwen/Qwen3.5-27B-FP8)
|
||||
- [GPTQ-Int4 checkpoint](https://huggingface.co/Qwen/Qwen3.5-27B-GPTQ-Int4)
|
||||
- [Qwen3.5-397B-A17B recipe](../Qwen3.5-397B-A17B)
|
||||
@@ -0,0 +1,140 @@
|
||||
meta:
|
||||
title: "Qwen3.5-2B"
|
||||
slug: "qwen3.5-2b"
|
||||
provider: "Qwen"
|
||||
description: "Qwen3.5 mini dense multimodal model (2B) — edge / low-VRAM serving with 262K context"
|
||||
date_added: 2026-04-22
|
||||
date_updated: 2026-09-14
|
||||
tasks:
|
||||
- multimodal
|
||||
- text
|
||||
performance_headline: "Edge-scale Qwen3.5 dense — fits on 8 GB GPUs or 1x Intel Arc Pro B60/B70"
|
||||
related_recipes:
|
||||
- "Qwen/Qwen3.5-4B"
|
||||
- "Qwen/Qwen3.5-0.8B"
|
||||
hardware:
|
||||
arc_pro_b60: verified
|
||||
arc_pro_b70: verified
|
||||
|
||||
model:
|
||||
model_id: "Qwen/Qwen3.5-2B"
|
||||
min_vllm_version: "0.17.0"
|
||||
default_frontend: rust
|
||||
architecture: dense
|
||||
parameter_count: "2B"
|
||||
active_parameters: "2B"
|
||||
context_length: 262144
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with Qwen3 Coder parser"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "qwen3_coder"
|
||||
reasoning:
|
||||
description: "Enable chain-of-thought reasoning with Qwen3 parser"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "qwen3"
|
||||
spec_decoding:
|
||||
description: "Multi-token prediction speculative decoding for lower latency"
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"mtp","num_speculative_tokens":1}'
|
||||
text_only:
|
||||
description: "Skip loading the vision encoder for text-only workloads."
|
||||
args:
|
||||
- "--language-model-only"
|
||||
|
||||
opt_in_features:
|
||||
- spec_decoding
|
||||
- text_only
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 5
|
||||
description: "Full precision BF16 — fits on an 8 GB GPU or Intel Arc Pro B60/B70"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
|
||||
hardware_overrides:
|
||||
xpu:
|
||||
extra_args:
|
||||
- "--enforce-eager"
|
||||
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[Qwen3.5-2B](https://huggingface.co/Qwen/Qwen3.5-2B) is a miniature dense
|
||||
Qwen3.5 model — the full gated delta networks architecture, vision encoder,
|
||||
and 262K context, in a form small enough for 8 GB consumer GPUs or edge
|
||||
inference.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **vLLM version:** >= 0.17.0
|
||||
- **Hardware:** single 8 GB GPU or Intel Arc Pro B60/B70
|
||||
|
||||
### Install vLLM
|
||||
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend=auto
|
||||
```
|
||||
|
||||
### Docker
|
||||
```bash
|
||||
docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70)
|
||||
```
|
||||
|
||||
## Launching the Server
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3.5-2B \
|
||||
--max-model-len 262144 \
|
||||
--reasoning-parser qwen3
|
||||
```
|
||||
|
||||
### Docker (Intel XPU B60 / B70)
|
||||
|
||||
Validated on 1× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the
|
||||
official vLLM XPU image `vllm/vllm-openai-xpu:latest`.
|
||||
|
||||
```bash
|
||||
docker run --device /dev/dri \
|
||||
-v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
|
||||
--privileged --ipc=host -p 8000:8000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai-xpu:latest \
|
||||
Qwen/Qwen3.5-2B \
|
||||
--reasoning-parser qwen3 \
|
||||
--enforce-eager
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
|
||||
resp = client.chat.completions.create(
|
||||
model="Qwen/Qwen3.5-2B",
|
||||
messages=[{"role": "user", "content": "Hi!"}],
|
||||
max_tokens=64,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/Qwen/Qwen3.5-2B)
|
||||
- [Base checkpoint](https://huggingface.co/Qwen/Qwen3.5-2B-Base)
|
||||
@@ -0,0 +1,254 @@
|
||||
meta:
|
||||
title: "Qwen3.5-35B-A3B"
|
||||
slug: "qwen3.5-35b-a3b"
|
||||
provider: "Qwen"
|
||||
description: "Compact Qwen3.5 multimodal MoE (35B total / 3B active) with gated delta networks, 256 experts, and 262K context"
|
||||
date_added: 2026-04-22
|
||||
date_updated: 2026-09-30
|
||||
tasks:
|
||||
- multimodal
|
||||
- text
|
||||
performance_headline: "Compact Qwen3.5 MoE — single-GPU FP8, 2x GPU or 2x Xeon 6 NUMA nodes or 4x Intel Arc Pro B60/B70 BF16 serving"
|
||||
related_recipes:
|
||||
- "Qwen/Qwen3.5-397B-A17B"
|
||||
- "Qwen/Qwen3.5-122B-A10B"
|
||||
hardware:
|
||||
xeon6: verified
|
||||
arc_pro_b60: verified
|
||||
arc_pro_b70: verified
|
||||
|
||||
model:
|
||||
model_id: "Qwen/Qwen3.5-35B-A3B"
|
||||
min_vllm_version: "0.17.0"
|
||||
default_frontend: rust
|
||||
architecture: moe
|
||||
parameter_count: "35B"
|
||||
active_parameters: "3B"
|
||||
context_length: 262144
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with Qwen3 Coder parser"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "qwen3_coder"
|
||||
reasoning:
|
||||
description: "Enable chain-of-thought reasoning with Qwen3 parser"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "qwen3"
|
||||
spec_decoding:
|
||||
description: "Multi-token prediction speculative decoding for lower latency"
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"mtp","num_speculative_tokens":1}'
|
||||
text_only:
|
||||
description: "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
|
||||
args:
|
||||
- "--language-model-only"
|
||||
encoder_parallel:
|
||||
description: "Run the vision encoder in data-parallel mode — avoids TP comm overhead on the small encoder. Mutually exclusive with text_only."
|
||||
args:
|
||||
- "--mm-encoder-tp-mode"
|
||||
- "data"
|
||||
long_context:
|
||||
description: "YaRN RoPE scaling past the native 262,144-token window to ~1,010,000 tokens. Static YaRN holds the scaling factor constant regardless of input length, so leave this off unless you actually serve long prompts; for ~524K, halve the factor to 2.0."
|
||||
args:
|
||||
- "--hf-overrides"
|
||||
- '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}'
|
||||
- "--max-model-len"
|
||||
- "1010000"
|
||||
env:
|
||||
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
|
||||
|
||||
opt_in_features:
|
||||
- spec_decoding
|
||||
- text_only
|
||||
- long_context
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 84
|
||||
description: "Full precision BF16 — fits on 1x H200, 2x H100, 2x Xeon 6 NUMA nodes, or 4x Intel Arc Pro B60 / B70"
|
||||
supported_hardware: [xeon6, arc_pro_b60, arc_pro_b70]
|
||||
fp8:
|
||||
model_id: "Qwen/Qwen3.5-35B-A3B-FP8"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 42
|
||||
description: "Qwen official FP8 checkpoint — single-GPU serving"
|
||||
supported_hardware: [arc_pro_b60, arc_pro_b70]
|
||||
gptq_int4:
|
||||
model_id: "Qwen/Qwen3.5-35B-A3B-GPTQ-Int4"
|
||||
precision: int4
|
||||
vram_minimum_gb: 21
|
||||
description: "GPTQ Int4 checkpoint — fits on a single 24GB GPU"
|
||||
supported_hardware: [arc_pro_b60, arc_pro_b70]
|
||||
|
||||
redhat_fp8:
|
||||
model_id: "RedHatAI/Qwen3.5-35B-A3B-FP8-dynamic"
|
||||
precision: fp8
|
||||
label: "FP8 Dynamic (RedHat)"
|
||||
vram_minimum_gb: 42
|
||||
description: "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization"
|
||||
supported_hardware: [arc_pro_b60, arc_pro_b70]
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- single_node_tep
|
||||
- single_node_dep
|
||||
- multi_node_tp
|
||||
- multi_node_dep
|
||||
- multi_node_tep
|
||||
|
||||
hardware_overrides:
|
||||
cpu:
|
||||
extra_env:
|
||||
# Allow cold weight downloads and CPU compilation to finish before startup times out.
|
||||
VLLM_ENGINE_READY_TIMEOUT_S: "1800"
|
||||
xpu:
|
||||
extra_args:
|
||||
- "--enforce-eager"
|
||||
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[Qwen3.5-35B-A3B](https://huggingface.co/Qwen/Qwen3.5-35B-A3B) is the smallest
|
||||
MoE in the Qwen3.5 family, sharing the gated delta networks architecture with
|
||||
35B total parameters and 3B activated per token (256 experts). With FP8 weights
|
||||
it fits on a single 80 GB GPU and supports the full 262K context.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **vLLM version:** >= 0.17.0
|
||||
- **Hardware (BF16):** 1x H200, 2x H100, 2x Xeon 6 NUMA nodes, 4x Intel Arc Pro B60 / B70
|
||||
- **Hardware (FP8):** single H100/H200
|
||||
- **Hardware (Int4):** single 24 GB GPU
|
||||
|
||||
### Pip Install
|
||||
|
||||
#### NVIDIA
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend=auto
|
||||
```
|
||||
|
||||
#### CPU
|
||||
For Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions.
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
export VLLM_VERSION=$(curl -s https://api.github.com/repos/vllm-project/vllm/releases/latest | jq -r .tag_name | sed 's/^v//')
|
||||
uv pip install https://github.com/vllm-project/vllm/releases/download/v${VLLM_VERSION}/vllm-${VLLM_VERSION}+cpu-cp38-abi3-manylinux_2_35_x86_64.whl --torch-backend cpu
|
||||
```
|
||||
|
||||
### Docker
|
||||
```bash
|
||||
docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70)
|
||||
```
|
||||
|
||||
## Launching the Server
|
||||
|
||||
### Single-GPU FP8
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3.5-35B-A3B-FP8 \
|
||||
--max-model-len 262144 \
|
||||
--reasoning-parser qwen3
|
||||
```
|
||||
|
||||
### BF16 on 2xH200 (TP2)
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3.5-35B-A3B \
|
||||
--tensor-parallel-size 2 \
|
||||
--max-model-len 262144 \
|
||||
--reasoning-parser qwen3
|
||||
```
|
||||
|
||||
### Intel Xeon 6 Deployment via Docker
|
||||
|
||||
Launch the x86 CPU vLLM Docker container for `Qwen/Qwen3.5-35B-A3B`:
|
||||
|
||||
```bash
|
||||
docker run \
|
||||
--privileged --ipc=host -p 8000:8000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
-e VLLM_ENGINE_READY_TIMEOUT_S=1800 \
|
||||
vllm/vllm-openai-cpu:latest-x86_64 Qwen/Qwen3.5-35B-A3B \
|
||||
--trust-remote-code \
|
||||
--tensor-parallel-size 1 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser qwen3_coder \
|
||||
--reasoning-parser qwen3 \
|
||||
--mm-encoder-tp-mode data
|
||||
```
|
||||
|
||||
### MTP speculative decoding
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3.5-35B-A3B-FP8 \
|
||||
--speculative-config '{"method": "mtp", "num_speculative_tokens": 1}' \
|
||||
--reasoning-parser qwen3
|
||||
```
|
||||
|
||||
### Docker (Intel XPU B60 / B70)
|
||||
|
||||
Validated on 4× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the
|
||||
official vLLM XPU image `vllm/vllm-openai-xpu:latest`.
|
||||
|
||||
```bash
|
||||
docker run --device /dev/dri \
|
||||
-v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
|
||||
--privileged --ipc=host -p 8000:8000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai-xpu:latest \
|
||||
Qwen/Qwen3.5-35B-A3B \
|
||||
--reasoning-parser qwen3 \
|
||||
--tensor-parallel-size 4 \
|
||||
--max-model-len 8192 \
|
||||
--enforce-eager
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
|
||||
resp = client.chat.completions.create(
|
||||
model="Qwen/Qwen3.5-35B-A3B",
|
||||
messages=[{"role": "user", "content": "Explain gated delta networks in one paragraph."}],
|
||||
max_tokens=512,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **CPU startup timeout:** a cold weight download followed by CPU compilation
|
||||
can exceed the default 600-second engine readiness timeout. The CPU recipe
|
||||
sets `VLLM_ENGINE_READY_TIMEOUT_S=1800`; set any external health-check
|
||||
deadline higher (for example, 2100 seconds). Increasing only the external
|
||||
deadline does not prevent the frontend from timing out.
|
||||
- **CUDA graph / Mamba cache size error:** reduce `--max-cudagraph-capture-size`
|
||||
(default 512). See [vLLM PR #34571](https://github.com/vllm-project/vllm/pull/34571).
|
||||
- **Disable reasoning:** add `--default-chat-template-kwargs '{"enable_thinking": false}'`.
|
||||
- **Prefix Caching (Mamba):** currently experimental in "align" mode.
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/Qwen/Qwen3.5-35B-A3B)
|
||||
- [FP8 checkpoint](https://huggingface.co/Qwen/Qwen3.5-35B-A3B-FP8)
|
||||
- [GPTQ-Int4 checkpoint](https://huggingface.co/Qwen/Qwen3.5-35B-A3B-GPTQ-Int4)
|
||||
- [Qwen3.5-397B-A17B recipe](../Qwen3.5-397B-A17B)
|
||||
@@ -0,0 +1,503 @@
|
||||
meta:
|
||||
title: "Qwen3.5-397B"
|
||||
slug: "qwen3.5-397b"
|
||||
provider: "Qwen"
|
||||
description: "Multimodal MoE model with gated delta networks architecture, 397B total / 17B active parameters, up to 262K context"
|
||||
date_added: 2026-02-16
|
||||
date_updated: 2026-10-07
|
||||
tasks:
|
||||
- multimodal
|
||||
- text
|
||||
performance_headline: "Verified on 8x H200, 8x MI300X/MI355X, a single MI350X/MI355X (MXFP4), GB200 nodes, and Ascend 950DT"
|
||||
related_recipes:
|
||||
- "Qwen/Qwen3.6-35B-A3B"
|
||||
hardware:
|
||||
h200: verified
|
||||
gb200: verified
|
||||
mi300x: verified
|
||||
mi325x: verified
|
||||
mi350x: verified
|
||||
mi355x: verified
|
||||
ascend_950dt: verified
|
||||
|
||||
model:
|
||||
model_id: "Qwen/Qwen3.5-397B-A17B"
|
||||
min_vllm_version: "0.17.0"
|
||||
default_frontend: rust
|
||||
architecture: moe
|
||||
parameter_count: "397B"
|
||||
active_parameters: "17B"
|
||||
context_length: 262144
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
base_env:
|
||||
VLLM_DEEP_GEMM_WARMUP: "skip"
|
||||
VLLM_USE_DEEP_GEMM: "0"
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with Qwen3 Coder parser"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "qwen3_coder"
|
||||
reasoning:
|
||||
description: "Enable chain-of-thought reasoning with Qwen3 parser"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "qwen3"
|
||||
spec_decoding:
|
||||
description: "Multiple options for multi-token prediction speculative decoding for lower latency. Also has DFlash option. "
|
||||
default_mode: mtp
|
||||
modes:
|
||||
mtp:
|
||||
description: "Native MTP head, 3 draft tokens."
|
||||
variants: [default, nvfp4, fp8, gptq_int4, amd_mxfp4]
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"mtp","num_speculative_tokens":3}'
|
||||
dflash:
|
||||
label: "DFlash"
|
||||
description: "DFlash speculative decoding with an externally trained draft model; the served checkpoint is unchanged."
|
||||
variants:
|
||||
- default
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"model": "RedHatAI/Qwen3.5-397B-A17B-speculator.dflash", "num_speculative_tokens": 7, "method": "dflash"}'
|
||||
qwen3_5_mtp:
|
||||
label: "MTP (Ascend)"
|
||||
description: "MTP mode documented for vLLM Ascend."
|
||||
variants: [ascend_w8a8_mtp, ascend_w4a4_mxfp4]
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"qwen3_5_mtp","num_speculative_tokens":3}'
|
||||
text_only:
|
||||
description: "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
|
||||
args:
|
||||
- "--language-model-only"
|
||||
encoder_parallel:
|
||||
description: "Run the vision encoder in data-parallel mode — avoids TP comm overhead on the small encoder. Mutually exclusive with text_only."
|
||||
args:
|
||||
- "--mm-encoder-tp-mode"
|
||||
- "data"
|
||||
long_context:
|
||||
description: "YaRN RoPE scaling past the native 262,144-token window to ~1,010,000 tokens. Static YaRN holds the scaling factor constant regardless of input length, so leave this off unless you actually serve long prompts; for ~524K, halve the factor to 2.0."
|
||||
args:
|
||||
- "--hf-overrides"
|
||||
- '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}'
|
||||
- "--max-model-len"
|
||||
- "1010000"
|
||||
env:
|
||||
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
|
||||
|
||||
opt_in_features:
|
||||
- spec_decoding
|
||||
- text_only
|
||||
- long_context
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 953
|
||||
description: "Full precision BF16 — requires 8x H200 or equivalent"
|
||||
nvfp4:
|
||||
model_id: "nvidia/Qwen3.5-397B-A17B-NVFP4"
|
||||
precision: nvfp4
|
||||
vram_minimum_gb: 238
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
fp8:
|
||||
model_id: "RedHatAI/Qwen3.5-397B-A17B-FP8-dynamic"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 477
|
||||
description: "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
gptq_int4:
|
||||
model_id: "Qwen/Qwen3.5-397B-A17B-GPTQ-Int4"
|
||||
precision: int4
|
||||
vram_minimum_gb: 239
|
||||
description: "GPTQ Int4 checkpoint — halves VRAM vs FP8"
|
||||
amd_mxfp4:
|
||||
model_id: "amd/Qwen3.5-397B-A17B-MXFP4"
|
||||
precision: mxfp4
|
||||
label: "MXFP4 (W4A4)"
|
||||
# Mixed precision (AMD Quark): routed experts and full attention in MXFP4 with dynamic MXFP4
|
||||
# activations; GDN projections, shared experts, router, lm_head and the MTP layer stay BF16.
|
||||
# Sized from the checkpoint: ceil(239.1 GB x 1.2) = 287 GB, one 288 GB MI350X/MI355X.
|
||||
vram_minimum_gb: 287
|
||||
description: "AMD Quark MXFP4 checkpoint for MI350X/MI355X (gfx950) — fits one 288 GB GPU"
|
||||
supported_hardware: ["arch:gfx950"]
|
||||
tp: 1
|
||||
min_vllm_version: "0.31.0"
|
||||
# None of these is the default on ROCm: the AITER kernels; an attention backend whose decode stays
|
||||
# flat with context (the auto-selected ROCM_ATTN slows ~6x by 30k tokens here); and a sequence cap,
|
||||
# because the default max_num_seqs (1024) exceeds the ~296 Mamba cache blocks one GPU leaves.
|
||||
extra_env:
|
||||
VLLM_ROCM_USE_AITER: "1"
|
||||
extra_args:
|
||||
- "--attention-backend"
|
||||
- "ROCM_AITER_FA"
|
||||
- "--max-num-seqs"
|
||||
- "64"
|
||||
ascend_w8a8_mtp:
|
||||
model_id: "Eco-Tech/Qwen3.5-397B-A17B-w8a8-mtp"
|
||||
precision: int8
|
||||
label: "W8A8 (Ascend)"
|
||||
vram_minimum_gb: 477
|
||||
supported_hardware: [ascend_950dt]
|
||||
tp: 8
|
||||
min_vllm_version: "0.23.0"
|
||||
default_modes:
|
||||
spec_decoding: qwen3_5_mtp
|
||||
extra_args:
|
||||
- "--distributed-executor-backend"
|
||||
- "mp"
|
||||
- "--data-parallel-size"
|
||||
- "1"
|
||||
- "--quantization"
|
||||
- "ascend"
|
||||
- "--served-model-name"
|
||||
- "qwen3.5"
|
||||
- "--max-num-seqs"
|
||||
- "128"
|
||||
- "--max-model-len"
|
||||
- "133000"
|
||||
- "--max-num-batched-tokens"
|
||||
- "8192"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.95"
|
||||
- "--enable-prefix-caching"
|
||||
- "--async-scheduling"
|
||||
- "--compilation-config"
|
||||
- '{"cudagraph_mode":"FULL_DECODE_ONLY"}'
|
||||
- "--additional-config"
|
||||
- '{"enable_cpu_binding":true}'
|
||||
extra_env:
|
||||
VLLM_USE_MODELSCOPE: "True"
|
||||
PYTORCH_NPU_ALLOC_CONF: "expandable_segments:True"
|
||||
HCCL_OP_EXPANSION_MODE: "AIV"
|
||||
HCCL_BUFFSIZE: "400"
|
||||
OMP_PROC_BIND: "false"
|
||||
OMP_NUM_THREADS: "100"
|
||||
TASK_QUEUE_ENABLE: "1"
|
||||
VLLM_ASCEND_ENABLE_PREFETCH_MLP: "1"
|
||||
HCCL_INTRA_PCIE_ENABLE: "1"
|
||||
HCCL_INTRA_ROCE_ENABLE: "0"
|
||||
hardware_overrides:
|
||||
ascend_950dt:
|
||||
docker_image: "quay.io/ascend/vllm-ascend:v0.23.0-950DT"
|
||||
ascend_w4a4_mxfp4:
|
||||
model_id: "Eco-Tech/Qwen3.5-397B-A17B-w4a4-mxfp4"
|
||||
precision: mxfp4
|
||||
label: "W4A4 (Ascend)"
|
||||
vram_minimum_gb: 239
|
||||
supported_hardware: [ascend_950dt]
|
||||
tp: 8
|
||||
min_vllm_version: "0.23.0"
|
||||
default_modes:
|
||||
spec_decoding: qwen3_5_mtp
|
||||
extra_args:
|
||||
- "--distributed-executor-backend"
|
||||
- "mp"
|
||||
- "--data-parallel-size"
|
||||
- "1"
|
||||
- "--quantization"
|
||||
- "ascend"
|
||||
- "--served-model-name"
|
||||
- "qwen3.5"
|
||||
- "--max-num-seqs"
|
||||
- "128"
|
||||
- "--max-model-len"
|
||||
- "133000"
|
||||
- "--max-num-batched-tokens"
|
||||
- "8192"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.95"
|
||||
- "--enable-prefix-caching"
|
||||
- "--async-scheduling"
|
||||
- "--compilation-config"
|
||||
- '{"cudagraph_mode":"FULL_DECODE_ONLY"}'
|
||||
- "--additional-config"
|
||||
- '{"enable_cpu_binding":true}'
|
||||
extra_env:
|
||||
VLLM_USE_MODELSCOPE: "True"
|
||||
PYTORCH_NPU_ALLOC_CONF: "expandable_segments:True"
|
||||
HCCL_OP_EXPANSION_MODE: "AIV"
|
||||
HCCL_BUFFSIZE: "400"
|
||||
OMP_PROC_BIND: "false"
|
||||
OMP_NUM_THREADS: "100"
|
||||
TASK_QUEUE_ENABLE: "1"
|
||||
VLLM_ASCEND_ENABLE_PREFETCH_MLP: "1"
|
||||
HCCL_INTRA_PCIE_ENABLE: "1"
|
||||
HCCL_INTRA_ROCE_ENABLE: "0"
|
||||
hardware_overrides:
|
||||
ascend_950dt:
|
||||
docker_image: "quay.io/ascend/vllm-ascend:v0.23.0-950DT"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- single_node_tep
|
||||
- single_node_dep
|
||||
- multi_node_tp
|
||||
- multi_node_tp_pp
|
||||
- multi_node_dep
|
||||
- multi_node_tep
|
||||
- pd_cluster
|
||||
|
||||
hardware_overrides:
|
||||
# FlashInfer TRTLLM MoE kernels are Blackwell-only (Hopper fails with "kernel
|
||||
# does not support current device"); pin them per-generation so Hopper and
|
||||
# AMD keep auto-selection (AITER on ROCm).
|
||||
blackwell:
|
||||
extra_args:
|
||||
- "--moe-backend"
|
||||
- "flashinfer_trtllm"
|
||||
|
||||
strategy_overrides:
|
||||
pd_cluster:
|
||||
prefill:
|
||||
env:
|
||||
VLLM_SSM_CONV_STATE_LAYOUT: "DS"
|
||||
decode:
|
||||
env:
|
||||
VLLM_SSM_CONV_STATE_LAYOUT: "DS"
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[Qwen3.5](https://huggingface.co/Qwen/Qwen3.5-397B-A17B) is a multimodal mixture-of-experts model featuring a gated delta networks architecture with 397B total parameters and 17B active parameters. This guide covers how to efficiently deploy and serve the model across different hardware configurations and workload profiles using vLLM.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### Pip Install
|
||||
|
||||
#### NVIDIA
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend=auto
|
||||
```
|
||||
|
||||
#### AMD
|
||||
> Note: The vLLM wheel for ROCm requires Python 3.12, ROCm 7.0, and glibc >= 2.35. If your environment does not meet these requirements, please use the Docker-based setup. Supported GPUs: MI300X, MI325X, MI355X.
|
||||
```bash
|
||||
uv venv --python 3.12
|
||||
source .venv/bin/activate
|
||||
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm
|
||||
```
|
||||
|
||||
### Docker
|
||||
|
||||
#### NVIDIA
|
||||
```bash
|
||||
docker run --gpus all \
|
||||
-p 8000:8000 \
|
||||
--ipc=host \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai Qwen/Qwen3.5-397B-A17B \
|
||||
--tensor-parallel-size 8 \
|
||||
--reasoning-parser qwen3 \
|
||||
--enable-prefix-caching
|
||||
```
|
||||
For Blackwell GPUs, use `vllm/vllm-openai:cu130-nightly`.
|
||||
|
||||
#### AMD
|
||||
```bash
|
||||
docker run --device=/dev/kfd --device=/dev/dri \
|
||||
--security-opt seccomp=unconfined \
|
||||
--group-add video \
|
||||
--ipc=host \
|
||||
-p 8000:8000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai-rocm:latest \
|
||||
Qwen/Qwen3.5-397B-A17B-FP8 \
|
||||
--tensor-parallel-size 8 \
|
||||
--reasoning-parser qwen3 \
|
||||
--enable-prefix-caching
|
||||
```
|
||||
|
||||
## Deployment Configurations
|
||||
|
||||
The configurations below have been verified on 8x H200 GPUs and 8x MI300X/MI355X GPUs. We recommend using the official FP8 checkpoint [Qwen/Qwen3.5-397B-A17B-FP8](https://huggingface.co/Qwen/Qwen3.5-397B-A17B-FP8) for optimal serving efficiency.
|
||||
|
||||
### Throughput-Focused (Text-Only)
|
||||
|
||||
For maximum text throughput under high concurrency, use `--language-model-only` to skip loading the vision encoder and free up memory for KV cache, and enable Expert Parallelism.
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3.5-397B-A17B-FP8 \
|
||||
-dp 8 \
|
||||
--enable-expert-parallel \
|
||||
--language-model-only \
|
||||
--reasoning-parser qwen3 \
|
||||
--enable-prefix-caching
|
||||
```
|
||||
|
||||
### Throughput-Focused (Multimodal)
|
||||
|
||||
For multimodal workloads, use `--mm-encoder-tp-mode data` for data-parallel vision encoding and `--mm-processor-cache-type shm` for shared-memory caching of preprocessed multimodal inputs.
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3.5-397B-A17B-FP8 \
|
||||
-dp 8 \
|
||||
--enable-expert-parallel \
|
||||
--mm-encoder-tp-mode data \
|
||||
--mm-processor-cache-type shm \
|
||||
--reasoning-parser qwen3 \
|
||||
--enable-prefix-caching
|
||||
```
|
||||
|
||||
To enable tool calling, add `--enable-auto-tool-choice --tool-call-parser qwen3_coder` to the serve command.
|
||||
|
||||
### Latency-Focused
|
||||
|
||||
For latency-sensitive workloads at low concurrency, enable MTP-1 speculative decoding and disable prefix caching. MTP-1 reduces time-per-output-token (TPOT) with a high acceptance rate, at the cost of lower throughput under load.
|
||||
|
||||
> Note (AMD): MTP runs on ROCm with vLLM 0.31.0 (verified with the single-GPU MXFP4 setup below). On ROCm,
|
||||
> pair it with `--kv-cache-dtype fp8`: with the default KV cache its speedup fades as the prompt grows.
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3.5-397B-A17B-FP8 \
|
||||
--tensor-parallel-size 8 \
|
||||
--speculative-config '{"method": "mtp", "num_speculative_tokens": 1}' \
|
||||
--reasoning-parser qwen3
|
||||
```
|
||||
|
||||
### GB200 Deployment
|
||||
|
||||
We recommend using the NVFP4 checkpoint [nvidia/Qwen3.5-397B-A17B-NVFP4](https://huggingface.co/nvidia/Qwen3.5-397B-A17B-NVFP4) for optimal serving efficiency on GB200 nodes.
|
||||
|
||||
```bash
|
||||
vllm serve nvidia/Qwen3.5-397B-A17B-NVFP4 \
|
||||
-dp 4 \
|
||||
--enable-expert-parallel \
|
||||
--language-model-only \
|
||||
--reasoning-parser qwen3 \
|
||||
--enable-prefix-caching
|
||||
```
|
||||
|
||||
### MI355X Deployment
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3.5-397B-A17B-FP8 \
|
||||
-tp 2 \
|
||||
--enable-expert-parallel \
|
||||
--language-model-only \
|
||||
--reasoning-parser qwen3 \
|
||||
--enable-prefix-caching
|
||||
```
|
||||
|
||||
### Single MI350X / MI355X (MXFP4)
|
||||
|
||||
AMD's MXFP4 checkpoint [amd/Qwen3.5-397B-A17B-MXFP4](https://huggingface.co/amd/Qwen3.5-397B-A17B-MXFP4) (239 GB)
|
||||
fits one 288 GB MI350X or MI355X: pick the **MXFP4 (W4A4)** variant above. It sets three things vLLM does not
|
||||
pick on its own: `VLLM_ROCM_USE_AITER=1` turns on the AITER kernels; `--attention-backend ROCM_AITER_FA` replaces
|
||||
the auto-selected `ROCM_ATTN`, whose decode slows down as the context grows on this hybrid model; and
|
||||
`--max-num-seqs 64`, because the default (1024) is more than the ~296 Mamba cache blocks a single GPU leaves,
|
||||
and the server would refuse to start.
|
||||
|
||||
```bash
|
||||
VLLM_ROCM_USE_AITER=1 vllm serve amd/Qwen3.5-397B-A17B-MXFP4 \
|
||||
--tensor-parallel-size 1 \
|
||||
--attention-backend ROCM_AITER_FA \
|
||||
--max-num-seqs 64 \
|
||||
--trust-remote-code \
|
||||
--reasoning-parser qwen3
|
||||
```
|
||||
|
||||
Single-stream decode in tokens/s at ~200 / ~2k / ~12k / ~30k prompt tokens (128 output tokens), one GPU,
|
||||
`vllm/vllm-openai-rocm:v0.31.0`:
|
||||
|
||||
| | ~200 | ~2k | ~12k | ~30k |
|
||||
|---|---|---|---|---|
|
||||
| MXFP4, this variant (MI355X) | 119 | 119 | 119 | 117 |
|
||||
| + MTP (3 tokens) + `--kv-cache-dtype fp8` (MI355X) | 219 | 201 | 209 | 188 |
|
||||
| + MTP (3 tokens), default KV cache (MI350X) | 198 | 164 | 125 | 80 |
|
||||
| auto-selected `ROCM_ATTN` instead of `ROCM_AITER_FA` (MI355X) | 116 | 89 | 39 | 19 |
|
||||
| GPTQ-Int4, AITER + `ROCM_AITER_FA` (MI350X) | 75 | 74 | 74 | 74 |
|
||||
|
||||
With MTP, add `--kv-cache-dtype fp8`: on ROCm, MTP's gain otherwise fades as the prompt grows. The MXFP4
|
||||
checkpoint quantizes activations as well as weights (W4A4): on the same wikitext-2 tokens vLLM scored PPL 3.43
|
||||
with it and 3.39 with GPTQ-Int4 (W4A16), so GPTQ-Int4 is the higher-fidelity option at about two thirds of the speed.
|
||||
|
||||
## Processing Ultra-Long Texts
|
||||
|
||||
Qwen3.5-397B-A17B natively supports `262,144` tokens. To serve longer inputs, turn on the
|
||||
**Long Context** feature above: it applies the model card's YaRN RoPE
|
||||
configuration via `--hf-overrides` and raises `--max-model-len` to ~1,010,000 tokens.
|
||||
Static YaRN holds the scaling factor constant regardless of input length, so
|
||||
leave it off unless you actually serve long prompts; for a ~524K workload,
|
||||
halve the card's `factor` to `2.0`.
|
||||
|
||||
See the [model card](https://huggingface.co/Qwen/Qwen3.5-397B-A17B#processing-ultra-long-texts) for the full parameter reference.
|
||||
|
||||
## Client Usage
|
||||
|
||||
```python
|
||||
import time
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="EMPTY",
|
||||
base_url="http://localhost:8000/v1",
|
||||
timeout=3600
|
||||
)
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{
|
||||
"type": "image_url",
|
||||
"image_url": {
|
||||
"url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"
|
||||
}
|
||||
},
|
||||
{
|
||||
"type": "text",
|
||||
"text": "Read all the text in the image."
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
|
||||
start = time.time()
|
||||
response = client.chat.completions.create(
|
||||
model="Qwen/Qwen3.5-397B-A17B",
|
||||
messages=messages,
|
||||
max_tokens=2048
|
||||
)
|
||||
print(f"Response costs: {time.time() - start:.2f}s")
|
||||
print(f"Generated text: {response.choices[0].message.content}")
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**CUDA graph / Mamba cache size error**
|
||||
|
||||
You may encounter:
|
||||
```
|
||||
assert num_cache_lines >= batch
|
||||
```
|
||||
This occurs because the CUDA graph capture size is larger than the Mamba cache size. Reduce `--max-cudagraph-capture-size` (default is 512). See https://github.com/vllm-project/vllm/pull/34571 for details.
|
||||
|
||||
**Configuration tips**
|
||||
|
||||
- **Disable Reasoning**: Add `--reasoning-parser qwen3 --default-chat-template-kwargs '{"enable_thinking": false}'` to disable reasoning mode via command-line parameters.
|
||||
- **Prefix Caching**: Prefix caching for Mamba cache "align" mode is currently experimental.
|
||||
- **Multi-token Prediction**: MTP-1 reduces per-token latency but degrades throughput under high concurrency because speculative tokens consume KV cache capacity. Adjust `num_speculative_tokens` (1-5) based on your use case.
|
||||
- **Encoder Data Parallelism**: `--mm-encoder-tp-mode data` deploys the vision encoder in a data-parallel fashion. This consumes additional memory and may require adjustment of `--gpu-memory-utilization`.
|
||||
## References
|
||||
|
||||
- Model card: https://huggingface.co/Qwen/Qwen3.5-397B-A17B
|
||||
- FP8 checkpoint: https://huggingface.co/Qwen/Qwen3.5-397B-A17B-FP8
|
||||
- NVFP4 checkpoint: https://huggingface.co/nvidia/Qwen3.5-397B-A17B-NVFP4
|
||||
- GPTQ-Int4 checkpoint: https://huggingface.co/Qwen/Qwen3.5-397B-A17B-GPTQ-Int4
|
||||
- AMD MXFP4 checkpoint: https://huggingface.co/amd/Qwen3.5-397B-A17B-MXFP4
|
||||
- Ascend W8A8 checkpoint: https://www.modelscope.cn/models/Eco-Tech/Qwen3.5-397B-A17B-w8a8-mtp
|
||||
- Ascend W4A4 checkpoint: https://www.modelscope.cn/models/Eco-Tech/Qwen3.5-397B-A17B-w4a4-mxfp4
|
||||
- vLLM Ascend tutorial: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/Qwen3.5-397B-A17B.html
|
||||
- vLLM documentation: https://docs.vllm.ai/
|
||||
@@ -0,0 +1,195 @@
|
||||
meta:
|
||||
title: "Qwen3.5-4B"
|
||||
slug: "qwen3.5-4b"
|
||||
provider: "Qwen"
|
||||
description: "Qwen3.5 compact dense multimodal model (4B) — fits on 16 GB consumer GPUs with full 262K context or one Xeon 6 NUMA node or Intel Arc Pro B60/B70"
|
||||
date_added: 2026-04-22
|
||||
date_updated: 2026-09-14
|
||||
tasks:
|
||||
- multimodal
|
||||
- text
|
||||
performance_headline: "Consumer-GPU-friendly Qwen3.5 dense with MTP support"
|
||||
related_recipes:
|
||||
- "Qwen/Qwen3.5-9B"
|
||||
- "Qwen/Qwen3.5-2B"
|
||||
hardware:
|
||||
xeon6: verified
|
||||
arc_pro_b60: verified
|
||||
arc_pro_b70: verified
|
||||
|
||||
model:
|
||||
model_id: "Qwen/Qwen3.5-4B"
|
||||
min_vllm_version: "0.17.0"
|
||||
default_frontend: rust
|
||||
architecture: dense
|
||||
parameter_count: "4B"
|
||||
active_parameters: "4B"
|
||||
context_length: 262144
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with Qwen3 Coder parser"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "qwen3_coder"
|
||||
reasoning:
|
||||
description: "Enable chain-of-thought reasoning with Qwen3 parser"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "qwen3"
|
||||
spec_decoding:
|
||||
description: "Multi-token prediction speculative decoding for lower latency"
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"mtp","num_speculative_tokens":1}'
|
||||
text_only:
|
||||
description: "Skip loading the vision encoder for text-only workloads."
|
||||
args:
|
||||
- "--language-model-only"
|
||||
long_context:
|
||||
description: "YaRN RoPE scaling past the native 262,144-token window to ~1,010,000 tokens. Static YaRN holds the scaling factor constant regardless of input length, so leave this off unless you actually serve long prompts; for ~524K, halve the factor to 2.0."
|
||||
args:
|
||||
- "--hf-overrides"
|
||||
- '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}'
|
||||
- "--max-model-len"
|
||||
- "1010000"
|
||||
env:
|
||||
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
|
||||
|
||||
opt_in_features:
|
||||
- spec_decoding
|
||||
- text_only
|
||||
- long_context
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 10
|
||||
description: "Full precision BF16 — fits on a single 16 GB GPU or one Xeon 6 NUMA node or one Intel Arc Pro B60/B70"
|
||||
supported_hardware: [xeon6, arc_pro_b60, arc_pro_b70]
|
||||
|
||||
fp8:
|
||||
model_id: "RedHatAI/Qwen3.5-4B-FP8-dynamic"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 5
|
||||
description: "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization"
|
||||
supported_hardware: [arc_pro_b60, arc_pro_b70]
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
|
||||
hardware_overrides:
|
||||
xpu:
|
||||
extra_args:
|
||||
- "--enforce-eager"
|
||||
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[Qwen3.5-4B](https://huggingface.co/Qwen/Qwen3.5-4B) is the compact dense
|
||||
entry in the Qwen3.5 family — same gated delta networks architecture, vision
|
||||
encoder, 262K context, and MTP decoding as the larger siblings, sized for
|
||||
16 GB consumer GPUs.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **vLLM version:** >= 0.17.0
|
||||
- **Hardware:** single 16 GB GPU (RTX 4080 / L4 / A10 / T4-24GB) or one Xeon 6 NUMA node or one Intel Arc Pro B60/B70
|
||||
|
||||
### Install vLLM
|
||||
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend=auto
|
||||
```
|
||||
|
||||
### pip (Intel Xeon 6 CPUs)
|
||||
For Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions.
|
||||
|
||||
### Docker
|
||||
```bash
|
||||
docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70)
|
||||
```
|
||||
|
||||
## Launching the Server
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3.5-4B \
|
||||
--max-model-len 262144 \
|
||||
--reasoning-parser qwen3
|
||||
```
|
||||
|
||||
### Intel Xeon 6 Deployment via Docker
|
||||
|
||||
Launch the x86 CPU vLLM Docker container for `Qwen/Qwen3.5-4B`:
|
||||
|
||||
```bash
|
||||
docker run \
|
||||
--privileged --ipc=host -p 8000:8000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai-cpu:latest-x86_64 Qwen/Qwen3.5-4B \
|
||||
--trust-remote-code \
|
||||
--tensor-parallel-size 1 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser qwen3_coder \
|
||||
--reasoning-parser qwen3
|
||||
```
|
||||
|
||||
### Docker (Intel XPU B60 / B70)
|
||||
|
||||
Validated on 1× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the
|
||||
official vLLM XPU image `vllm/vllm-openai-xpu:latest`.
|
||||
|
||||
```bash
|
||||
docker run --device /dev/dri \
|
||||
-v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
|
||||
--privileged --ipc=host -p 8000:8000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai-xpu:latest \
|
||||
Qwen/Qwen3.5-4B \
|
||||
--reasoning-parser qwen3 \
|
||||
--enforce-eager
|
||||
```
|
||||
|
||||
|
||||
### MTP speculative decoding
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3.5-4B \
|
||||
--speculative-config '{"method": "mtp", "num_speculative_tokens": 1}' \
|
||||
--reasoning-parser qwen3
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
|
||||
resp = client.chat.completions.create(
|
||||
model="Qwen/Qwen3.5-4B",
|
||||
messages=[{"role": "user", "content": "Hello!"}],
|
||||
max_tokens=128,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **CUDA graph / Mamba cache size error:** reduce `--max-cudagraph-capture-size`.
|
||||
- **Disable reasoning:** add `--default-chat-template-kwargs '{"enable_thinking": false}'`.
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/Qwen/Qwen3.5-4B)
|
||||
- [Base checkpoint](https://huggingface.co/Qwen/Qwen3.5-4B-Base)
|
||||
@@ -0,0 +1,177 @@
|
||||
meta:
|
||||
title: "Qwen3.5-9B"
|
||||
slug: "qwen3.5-9b"
|
||||
provider: "Qwen"
|
||||
description: "Qwen3.5 dense multimodal model (9B) with gated delta networks hybrid attention, MTP, and 262K context"
|
||||
date_added: 2026-04-22
|
||||
date_updated: 2026-09-14
|
||||
tasks:
|
||||
- multimodal
|
||||
- text
|
||||
performance_headline: "Single-GPU Qwen3.5 dense with MTP-accelerated decoding"
|
||||
related_recipes:
|
||||
- "Qwen/Qwen3.5-27B"
|
||||
- "Qwen/Qwen3.5-4B"
|
||||
hardware:
|
||||
arc_pro_b60: verified
|
||||
arc_pro_b70: verified
|
||||
|
||||
model:
|
||||
model_id: "Qwen/Qwen3.5-9B"
|
||||
min_vllm_version: "0.17.0"
|
||||
default_frontend: rust
|
||||
architecture: dense
|
||||
parameter_count: "9B"
|
||||
active_parameters: "9B"
|
||||
context_length: 262144
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with Qwen3 Coder parser"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "qwen3_coder"
|
||||
reasoning:
|
||||
description: "Enable chain-of-thought reasoning with Qwen3 parser"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "qwen3"
|
||||
spec_decoding:
|
||||
description: "Multi-token prediction speculative decoding for lower latency"
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"mtp","num_speculative_tokens":1}'
|
||||
text_only:
|
||||
description: "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache."
|
||||
args:
|
||||
- "--language-model-only"
|
||||
long_context:
|
||||
description: "YaRN RoPE scaling past the native 262,144-token window to ~1,010,000 tokens. Static YaRN holds the scaling factor constant regardless of input length, so leave this off unless you actually serve long prompts; for ~524K, halve the factor to 2.0."
|
||||
args:
|
||||
- "--hf-overrides"
|
||||
- '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}'
|
||||
- "--max-model-len"
|
||||
- "1010000"
|
||||
env:
|
||||
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
|
||||
|
||||
opt_in_features:
|
||||
- spec_decoding
|
||||
- text_only
|
||||
- long_context
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 22
|
||||
description: "Full precision BF16 — single 24 GB GPU or Intel Arc Pro B60/B70"
|
||||
|
||||
fp8:
|
||||
model_id: "RedHatAI/Qwen3.5-9B-FP8-dynamic"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 11
|
||||
description: "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
|
||||
hardware_overrides:
|
||||
xpu:
|
||||
extra_args:
|
||||
- "--enforce-eager"
|
||||
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[Qwen3.5-9B](https://huggingface.co/Qwen/Qwen3.5-9B) is a dense multimodal
|
||||
model from the Qwen3.5 family — same gated delta networks hybrid attention,
|
||||
vision encoder, 262K context, and MTP support as its larger siblings, but
|
||||
sized to fit comfortably on a single 24 GB GPU.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **vLLM version:** >= 0.17.0
|
||||
- **Hardware:** single 24 GB GPU (RTX 4090 / L4 / A10G / H100) or Intel Arc Pro B60/B70
|
||||
|
||||
### Install vLLM
|
||||
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend=auto
|
||||
```
|
||||
|
||||
### Docker
|
||||
```bash
|
||||
docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70)
|
||||
```
|
||||
|
||||
## Launching the Server
|
||||
|
||||
### Single-GPU BF16
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3.5-9B \
|
||||
--max-model-len 262144 \
|
||||
--reasoning-parser qwen3
|
||||
```
|
||||
|
||||
### MTP speculative decoding
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3.5-9B \
|
||||
--speculative-config '{"method": "mtp", "num_speculative_tokens": 1}' \
|
||||
--reasoning-parser qwen3
|
||||
```
|
||||
|
||||
### Docker (Intel XPU B60 / B70)
|
||||
|
||||
Validated on 1× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the
|
||||
official vLLM XPU image `vllm/vllm-openai-xpu:latest`.
|
||||
|
||||
```bash
|
||||
docker run --device /dev/dri \
|
||||
-v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
|
||||
--privileged --ipc=host -p 8000:8000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai-xpu:latest \
|
||||
Qwen/Qwen3.5-9B \
|
||||
--reasoning-parser qwen3 \
|
||||
--max-model-len 8192 \
|
||||
--enforce-eager
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
|
||||
resp = client.chat.completions.create(
|
||||
model="Qwen/Qwen3.5-9B",
|
||||
messages=[{"role": "user", "content": "Hello!"}],
|
||||
max_tokens=128,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **CUDA graph / Mamba cache size error:** reduce `--max-cudagraph-capture-size`
|
||||
(default 512). See [vLLM PR #34571](https://github.com/vllm-project/vllm/pull/34571).
|
||||
- **Disable reasoning:** add `--default-chat-template-kwargs '{"enable_thinking": false}'`.
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/Qwen/Qwen3.5-9B)
|
||||
- [Base checkpoint](https://huggingface.co/Qwen/Qwen3.5-9B-Base)
|
||||
@@ -0,0 +1,637 @@
|
||||
meta:
|
||||
title: "Qwen3.6-27B"
|
||||
slug: "Qwen3.6-27b"
|
||||
provider: "Qwen"
|
||||
description: "Qwen3.6 dense multimodal model (27B) with gated delta networks hybrid attention, MTP, and 262K context"
|
||||
date_added: 2026-04-22
|
||||
date_updated: 2026-10-12
|
||||
tasks:
|
||||
- multimodal
|
||||
- text
|
||||
performance_headline: "Qwen3.6 flagship dense — single-GPU FP8, 2x GPU BF16, 2-4x Intel Arc Pro B60/B70 BF16, Ascend 910C TP2+DP8, or Ascend 950DT TP1+DP8"
|
||||
related_recipes:
|
||||
- "Qwen/Qwen3.6-35B-A3B"
|
||||
hardware:
|
||||
dgx_spark_gb10: verified
|
||||
rtx_pro_6000: verified
|
||||
dgx_station_gb300: verified
|
||||
rtx_5090: verified
|
||||
ascend_950dt: verified
|
||||
ascend_910c: verified
|
||||
arc_pro_b60: verified
|
||||
arc_pro_b70: verified
|
||||
mi300x: verified
|
||||
|
||||
model:
|
||||
model_id: "Qwen/Qwen3.6-27B"
|
||||
min_vllm_version: "0.17.0"
|
||||
default_frontend: rust
|
||||
architecture: dense
|
||||
parameter_count: "27B"
|
||||
active_parameters: "27B"
|
||||
context_length: 262144
|
||||
docker_image:
|
||||
# The AMD numbers in the guide were measured on this tag (2026-10-12).
|
||||
amd: "vllm/vllm-openai-rocm:v0.31.0"
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with Qwen3 Coder parser"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "qwen3_coder"
|
||||
reasoning:
|
||||
description: "Enable chain-of-thought reasoning with Qwen3 parser"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "qwen3"
|
||||
spec_decoding:
|
||||
description: "Speculative decoding — pick native MTP or the Ascend MTP mode."
|
||||
default_mode: mtp
|
||||
modes:
|
||||
mtp:
|
||||
label: "MTP"
|
||||
description: "Native MTP head, 3 draft tokens."
|
||||
variants: [default, fp8, nvfp4, redhat_fp8]
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"mtp","num_speculative_tokens":3}'
|
||||
hardware_overrides:
|
||||
# GB10 is decode-bound at low concurrency, so a deeper draft pays off.
|
||||
dgx_spark_gb10:
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"mtp","num_speculative_tokens":5}'
|
||||
qwen3_5_mtp:
|
||||
label: "MTP (Ascend)"
|
||||
description: "MTP mode documented for vLLM Ascend."
|
||||
variants: [ascend_w8a8_mxfp8, ascend_w8a8]
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"qwen3_5_mtp","num_speculative_tokens":3,"enforce_eager":true}'
|
||||
text_only:
|
||||
description: "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
|
||||
args:
|
||||
- "--language-model-only"
|
||||
encoder_parallel:
|
||||
description: "Run the vision encoder in data-parallel mode — avoids TP comm overhead on the small encoder. Mutually exclusive with text_only."
|
||||
args:
|
||||
- "--mm-encoder-tp-mode"
|
||||
- "data"
|
||||
long_context:
|
||||
description: "YaRN RoPE scaling past the native 262,144-token window to ~1,010,000 tokens. Static YaRN holds the scaling factor constant regardless of input length, so leave this off unless you actually serve long prompts; for ~524K, halve the factor to 2.0."
|
||||
args:
|
||||
- "--hf-overrides"
|
||||
- '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}'
|
||||
- "--max-model-len"
|
||||
- "1010000"
|
||||
env:
|
||||
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
|
||||
|
||||
opt_in_features:
|
||||
- spec_decoding
|
||||
- text_only
|
||||
- long_context
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 65
|
||||
description: "Full precision BF16 — fits on 1x H200, 2x H100, 4x Intel Arc Pro B60, 2x Intel Arc Pro B70, or an Ascend A3 node (TP2+DP8)"
|
||||
tp:
|
||||
ascend_910c: 2
|
||||
hardware_overrides:
|
||||
rtx_pro_6000:
|
||||
extra_args:
|
||||
- "--max-num-seqs"
|
||||
- "8"
|
||||
ascend_910c:
|
||||
docker_image: "swr.cn-southwest-2.myhuaweicloud.com/base_image/ascend-ci/vllm-ascend:v0.23.0-a3"
|
||||
extra_args:
|
||||
- "--data-parallel-size"
|
||||
- "8"
|
||||
- "--served-model-name"
|
||||
- "qwen3.6"
|
||||
- "--max-num-seqs"
|
||||
- "32"
|
||||
- "--max-model-len"
|
||||
- "133000"
|
||||
- "--max-num-batched-tokens"
|
||||
- "16384"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.90"
|
||||
- "--no-enable-prefix-caching"
|
||||
- "--compilation-config"
|
||||
- '{"cudagraph_mode":"FULL_DECODE_ONLY"}'
|
||||
- "--async-scheduling"
|
||||
extra_env:
|
||||
VLLM_USE_MODELSCOPE: "True"
|
||||
PYTORCH_NPU_ALLOC_CONF: "expandable_segments:True"
|
||||
HCCL_BUFFSIZE: "512"
|
||||
OMP_PROC_BIND: "false"
|
||||
OMP_NUM_THREADS: "1"
|
||||
TASK_QUEUE_ENABLE: "1"
|
||||
fp8:
|
||||
model_id: "Qwen/Qwen3.6-27B-FP8"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 33
|
||||
description: "Qwen official FP8 checkpoint — single 40 GB GPU"
|
||||
extra_args:
|
||||
- "--max-num-seqs"
|
||||
- "512"
|
||||
hardware_overrides:
|
||||
dgx_spark_gb10:
|
||||
extra_env:
|
||||
VLLM_USE_DEEP_GEMM: "0"
|
||||
rtx_pro_6000:
|
||||
extra_env:
|
||||
VLLM_USE_DEEP_GEMM: "0"
|
||||
nvfp4:
|
||||
model_id: "nvidia/Qwen3.6-27B-NVFP4"
|
||||
precision: nvfp4
|
||||
vram_minimum_gb: 27
|
||||
tp: 1
|
||||
# SM120 GPUs only reach FlashInfer's XQA decode kernel from 0.28.0.
|
||||
min_vllm_version: "0.28.0"
|
||||
nightly_required: true
|
||||
description: "NVIDIA ModelOpt NVFP4 checkpoint — MLP linears quantized to NVFP4, attention linears and KV cache in FP8. Single Blackwell GPU, including DGX Spark."
|
||||
hardware_overrides:
|
||||
dgx_station_gb300:
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
- "--attention-backend"
|
||||
- "flashinfer"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.92"
|
||||
- "--max-model-len"
|
||||
- "262144"
|
||||
- "--max-num-batched-tokens"
|
||||
- "8192"
|
||||
- "--enable-chunked-prefill"
|
||||
- "--enable-prefix-caching"
|
||||
- "--load-format"
|
||||
- "fastsafetensors"
|
||||
- "--async-scheduling"
|
||||
dgx_spark_gb10:
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
- "--attention-backend"
|
||||
- "flashinfer"
|
||||
# GB10 shares 128 GB with the CPU, so leave half the pool to the host.
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.5"
|
||||
- "--max-model-len"
|
||||
- "262144"
|
||||
- "--max-num-seqs"
|
||||
- "8"
|
||||
- "--max-num-batched-tokens"
|
||||
- "8192"
|
||||
- "--enable-chunked-prefill"
|
||||
- "--enable-prefix-caching"
|
||||
- "--load-format"
|
||||
- "fastsafetensors"
|
||||
- "--async-scheduling"
|
||||
rtx_pro_6000:
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
- "--attention-backend"
|
||||
- "flashinfer"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.9"
|
||||
- "--max-model-len"
|
||||
- "262144"
|
||||
- "--max-num-seqs"
|
||||
- "8"
|
||||
- "--max-num-batched-tokens"
|
||||
- "8192"
|
||||
- "--enable-chunked-prefill"
|
||||
- "--enable-prefix-caching"
|
||||
- "--load-format"
|
||||
- "fastsafetensors"
|
||||
- "--async-scheduling"
|
||||
rtx_5090:
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
- "--attention-backend"
|
||||
- "FLASHINFER"
|
||||
- "--attention-config"
|
||||
- '{"use_trtllm_attention": true}'
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.93"
|
||||
# 32 GB leaves ~9 GB for KV cache after the NVFP4 weights.
|
||||
- "--max-model-len"
|
||||
- "36608"
|
||||
- "--max-num-seqs"
|
||||
- "8"
|
||||
- "--max-num-batched-tokens"
|
||||
- "4096"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.95"
|
||||
- "--enable-chunked-prefill"
|
||||
- "--enable-prefix-caching"
|
||||
- "--async-scheduling"
|
||||
- "--skip-mm-profiling"
|
||||
extra_env:
|
||||
VLLM_HAS_FLASHINFER_CUBIN: "1"
|
||||
redhat_fp8:
|
||||
model_id: "RedHatAI/Qwen3.6-27B-FP8"
|
||||
precision: fp8
|
||||
label: "FP8 (RedHat)"
|
||||
vram_minimum_gb: 33
|
||||
description: "RedHatAI FP8 dynamic quantization"
|
||||
ascend_w8a8_mxfp8:
|
||||
model_id: "Eco-Tech/Qwen3.6-27B-w8a8-mxfp8"
|
||||
precision: mxfp8
|
||||
label: "W8A8 (Ascend)"
|
||||
vram_minimum_gb: 34
|
||||
supported_hardware: [ascend_950dt]
|
||||
tp: 1
|
||||
min_vllm_version: "0.23.0"
|
||||
default_modes:
|
||||
spec_decoding: qwen3_5_mtp
|
||||
extra_args:
|
||||
- "--distributed-executor-backend"
|
||||
- "mp"
|
||||
- "--data-parallel-size"
|
||||
- "8"
|
||||
- "--quantization"
|
||||
- "ascend"
|
||||
- "--served-model-name"
|
||||
- "qwen3.6"
|
||||
- "--max-num-seqs"
|
||||
- "32"
|
||||
- "--max-model-len"
|
||||
- "262144"
|
||||
- "--max-num-batched-tokens"
|
||||
- "16384"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.90"
|
||||
- "--no-enable-prefix-caching"
|
||||
- "--compilation-config"
|
||||
- '{"cudagraph_mode":"FULL_DECODE_ONLY"}'
|
||||
- "--additional-config"
|
||||
- '{"enable_cpu_binding":true}'
|
||||
extra_env:
|
||||
VLLM_USE_MODELSCOPE: "True"
|
||||
PYTORCH_NPU_ALLOC_CONF: "expandable_segments:True"
|
||||
HCCL_BUFFSIZE: "512"
|
||||
OMP_PROC_BIND: "false"
|
||||
OMP_NUM_THREADS: "1"
|
||||
TASK_QUEUE_ENABLE: "1"
|
||||
hardware_overrides:
|
||||
ascend_950dt:
|
||||
docker_image: "quay.io/ascend/vllm-ascend:v0.23.0-a5"
|
||||
ascend_w8a8:
|
||||
model_id: "Eco-Tech/Qwen3.6-27B-w8a8"
|
||||
precision: int8
|
||||
label: "W8A8 (Ascend A3)"
|
||||
vram_minimum_gb: 33
|
||||
supported_hardware: [ascend_910c]
|
||||
tp: 2
|
||||
min_vllm_version: "0.23.0"
|
||||
default_modes:
|
||||
spec_decoding: qwen3_5_mtp
|
||||
extra_args:
|
||||
- "--data-parallel-size"
|
||||
- "8"
|
||||
- "--quantization"
|
||||
- "ascend"
|
||||
- "--served-model-name"
|
||||
- "qwen3.6"
|
||||
- "--max-num-seqs"
|
||||
- "32"
|
||||
- "--max-model-len"
|
||||
- "133000"
|
||||
- "--max-num-batched-tokens"
|
||||
- "16384"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.90"
|
||||
- "--no-enable-prefix-caching"
|
||||
- "--compilation-config"
|
||||
- '{"cudagraph_mode":"FULL_DECODE_ONLY"}'
|
||||
- "--additional-config"
|
||||
- '{"enable_cpu_binding":true}'
|
||||
- "--async-scheduling"
|
||||
extra_env:
|
||||
VLLM_USE_MODELSCOPE: "True"
|
||||
PYTORCH_NPU_ALLOC_CONF: "expandable_segments:True"
|
||||
HCCL_BUFFSIZE: "512"
|
||||
OMP_PROC_BIND: "false"
|
||||
OMP_NUM_THREADS: "1"
|
||||
TASK_QUEUE_ENABLE: "1"
|
||||
hardware_overrides:
|
||||
ascend_910c:
|
||||
docker_image: "swr.cn-southwest-2.myhuaweicloud.com/base_image/ascend-ci/vllm-ascend:v0.23.0-a3"
|
||||
description: "ModelSlim INT8 W8A8 checkpoint for one Atlas 800 A3 node (16x Ascend 910C), using TP2+DP8. Requires --quantization ascend."
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
|
||||
hardware_overrides:
|
||||
amd:
|
||||
# AITER is opt-in on ROCm and the image does not set it. Without these two the
|
||||
# server auto-selects ROCM_ATTN, which cannot use its native kernel for this
|
||||
# model (head size 256, 784-token hybrid blocks) and falls back to a Triton
|
||||
# path whose single-stream decode drops ~5x from ~200 to ~32k prompt tokens.
|
||||
# ROCM_AITER_FA stays flat across context on MI300X and is also the backend
|
||||
# #1082 moved to for gfx950 on v0.31.0. Verified 2026-10-12 on one MI300X
|
||||
# (see the AMD section of the guide).
|
||||
extra_args:
|
||||
- "--attention-backend"
|
||||
- "ROCM_AITER_FA"
|
||||
extra_env:
|
||||
VLLM_ROCM_USE_AITER: "1"
|
||||
xpu:
|
||||
extra_args:
|
||||
- "--enforce-eager"
|
||||
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[Qwen3.6-27B](https://huggingface.co/Qwen/Qwen3.6-27B) is the flagship dense
|
||||
model of the Qwen3.6 family. It uses the same gated delta networks hybrid
|
||||
attention as its MoE siblings, supports vision+text input, and natively serves
|
||||
262K context. MTP (multi-token prediction) is supported out of the box for
|
||||
low-latency decoding.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **vLLM version:** >= 0.17.0
|
||||
- **NVFP4 vLLM version:** >= 0.28.0 (currently nightly). 0.24.0 loads the
|
||||
checkpoint, but FlashInfer only selects its XQA decode kernel on SM120
|
||||
GPUs from 0.28.0.
|
||||
- **Hardware (BF16):** 1x H200, 2x H100, 4x Intel Arc Pro B60, 2x Intel Arc Pro B70
|
||||
- **Hardware (FP8):** single 40 GB GPU (H100/H200/L40S)
|
||||
- **Hardware (Int4):** single 24 GB GPU
|
||||
- **Hardware (NVFP4):** single NVIDIA Blackwell GPU (B200/B300/GB10 DGX Spark)
|
||||
- **Hardware (AMD):** 1x MI300X (BF16 fits one 192 GB GPU)
|
||||
|
||||
### Install vLLM
|
||||
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend=auto
|
||||
```
|
||||
|
||||
### Docker
|
||||
```bash
|
||||
docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70)
|
||||
```
|
||||
|
||||
## Launching the Server
|
||||
|
||||
### Single-GPU FP8
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3.6-27B-FP8 \
|
||||
--max-model-len 262144 \
|
||||
--reasoning-parser qwen3
|
||||
```
|
||||
|
||||
### BF16 on 2xH100 (TP2)
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3.6-27B \
|
||||
--tensor-parallel-size 2 \
|
||||
--max-model-len 262144 \
|
||||
--reasoning-parser qwen3
|
||||
```
|
||||
|
||||
### MTP speculative decoding
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3.6-27B-FP8 \
|
||||
--speculative-config '{"method": "mtp", "num_speculative_tokens": 1}' \
|
||||
--reasoning-parser qwen3
|
||||
```
|
||||
|
||||
### Text-only (skip vision encoder)
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3.6-27B-FP8 \
|
||||
--language-model-only \
|
||||
--reasoning-parser qwen3 \
|
||||
--enable-prefix-caching
|
||||
```
|
||||
|
||||
### NVFP4 on Blackwell
|
||||
|
||||
The [`nvidia/Qwen3.6-27B-NVFP4`](https://huggingface.co/nvidia/Qwen3.6-27B-NVFP4)
|
||||
checkpoint is NVIDIA's ModelOpt re-quantization: MLP linears drop to NVFP4
|
||||
(W4A16) while the attention linears and KV cache stay FP8, so the ~22 GB
|
||||
weights fit a single Blackwell GPU. vLLM auto-detects the ModelOpt
|
||||
quantization from the checkpoint, so no `--quantization` flag is needed, but
|
||||
it does need vLLM 0.28.0+ (currently nightly) for FlashInfer's XQA decode path
|
||||
on SM120. NVIDIA reports near-lossless accuracy versus the FP8 baseline.
|
||||
|
||||
Select the NVFP4 variant in the command builder above and switch the hardware
|
||||
pill for the validated single-node launch on DGX Spark (GB10), DGX Station
|
||||
(GB300), RTX PRO 6000 or RTX 5090. Memory utilization, context length, batch
|
||||
size, attention backend and MTP depth all differ per box and are encoded per
|
||||
hardware, so the generated command is the tested one.
|
||||
|
||||
#### **2× DGX Spark (TP2)**
|
||||
|
||||
Two Sparks joined over ConnectX is the one layout the builder can't render, so
|
||||
it is spelled out here. Download the checkpoint on all nodes. Start the worker
|
||||
node first and then the head node — the head coordinates the others, so they
|
||||
need to be ready to connect when it comes up.
|
||||
|
||||
```bash
|
||||
# COMMON variables example values — same on all nodes
|
||||
export HEAD_IP="<node1-connectx-ip>" # The ConnectX IP of Node1 (Head Node)
|
||||
export ETH_IF="enp1s0f1np1" # CX7 ethernet interface which is UP
|
||||
export IB_IF="rocep1s0f1,roceP2p1s0f1" # BOTH matching RoCE interfaces
|
||||
export MASTER_PORT=29501 # Free port on the Head Node
|
||||
export CONTAINER_NAME=vllm_node
|
||||
export IMAGE="vllm/vllm-openai:nightly"
|
||||
```
|
||||
|
||||
```bash
|
||||
# Head Node
|
||||
docker run --privileged --ulimit nofile=1048576:1048576 --ipc=host \
|
||||
--gpus all --rm --network host --name "$CONTAINER_NAME" --entrypoint="" \
|
||||
-e MN_IF_NAME="$ETH_IF" \
|
||||
-e UCX_NET_DEVICES="$ETH_IF" \
|
||||
-e NCCL_SOCKET_IFNAME="$ETH_IF" \
|
||||
-e NCCL_IB_HCA="$IB_IF" \
|
||||
-e NCCL_IB_DISABLE=0 \
|
||||
-e OMPI_MCA_btl_tcp_if_include="$ETH_IF" \
|
||||
-e GLOO_SOCKET_IFNAME="$ETH_IF" \
|
||||
-e TP_SOCKET_IFNAME="$ETH_IF" \
|
||||
-e NCCL_IGNORE_CPU_AFFINITY=1 \
|
||||
-e VLLM_FLOAT32_MATMUL_PRECISION=high \
|
||||
-e HEAD_IP="$HEAD_IP" \
|
||||
-e MASTER_PORT="$MASTER_PORT" \
|
||||
-v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
|
||||
"$IMAGE" vllm serve nvidia/Qwen3.6-27B-NVFP4 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000 \
|
||||
--trust-remote-code \
|
||||
--kv-cache-dtype fp8 \
|
||||
--attention-backend flashinfer \
|
||||
--gpu-memory-utilization 0.5 \
|
||||
--max-model-len 262144 \
|
||||
--max-num-seqs 8 \
|
||||
--max-num-batched-tokens 8192 \
|
||||
--enable-chunked-prefill \
|
||||
--async-scheduling \
|
||||
--enable-prefix-caching \
|
||||
--speculative-config '{"method":"mtp","num_speculative_tokens":5,"draft_tensor_parallel_size":2}' \
|
||||
--load-format fastsafetensors \
|
||||
--reasoning-parser qwen3 \
|
||||
--tool-call-parser qwen3_coder \
|
||||
--enable-auto-tool-choice \
|
||||
--tensor-parallel-size 2 \
|
||||
--nnodes 2 \
|
||||
--node-rank 0 \
|
||||
--master-addr "$HEAD_IP" \
|
||||
--master-port "$MASTER_PORT"
|
||||
|
||||
# Worker node
|
||||
docker run --privileged --ulimit nofile=1048576:1048576 --ipc=host \
|
||||
--gpus all --rm --network host --name "$CONTAINER_NAME" --entrypoint="" \
|
||||
-e MN_IF_NAME="$ETH_IF" \
|
||||
-e UCX_NET_DEVICES="$ETH_IF" \
|
||||
-e NCCL_SOCKET_IFNAME="$ETH_IF" \
|
||||
-e NCCL_IB_HCA="$IB_IF" \
|
||||
-e NCCL_IB_DISABLE=0 \
|
||||
-e OMPI_MCA_btl_tcp_if_include="$ETH_IF" \
|
||||
-e GLOO_SOCKET_IFNAME="$ETH_IF" \
|
||||
-e TP_SOCKET_IFNAME="$ETH_IF" \
|
||||
-e NCCL_IGNORE_CPU_AFFINITY=1 \
|
||||
-e VLLM_FLOAT32_MATMUL_PRECISION=high \
|
||||
-e HEAD_IP="$HEAD_IP" \
|
||||
-e MASTER_PORT="$MASTER_PORT" \
|
||||
-v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
|
||||
"$IMAGE" vllm serve nvidia/Qwen3.6-27B-NVFP4 \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000 \
|
||||
--trust-remote-code \
|
||||
--kv-cache-dtype fp8 \
|
||||
--attention-backend flashinfer \
|
||||
--gpu-memory-utilization 0.5 \
|
||||
--max-model-len 262144 \
|
||||
--max-num-seqs 8 \
|
||||
--max-num-batched-tokens 8192 \
|
||||
--enable-chunked-prefill \
|
||||
--async-scheduling \
|
||||
--enable-prefix-caching \
|
||||
--speculative-config '{"method":"mtp","num_speculative_tokens":5,"draft_tensor_parallel_size":2}' \
|
||||
--load-format fastsafetensors \
|
||||
--reasoning-parser qwen3 \
|
||||
--tool-call-parser qwen3_coder \
|
||||
--enable-auto-tool-choice \
|
||||
--tensor-parallel-size 2 \
|
||||
--nnodes 2 \
|
||||
--node-rank 1 \
|
||||
--master-addr "$HEAD_IP" \
|
||||
--master-port "$MASTER_PORT" \
|
||||
--headless
|
||||
```
|
||||
|
||||
|
||||
### AMD (MI300X, single GPU, BF16)
|
||||
|
||||
BF16 fits one MI300X. Two settings matter on ROCm and neither is on by default:
|
||||
`VLLM_ROCM_USE_AITER=1` and an explicit attention backend. The auto-selected
|
||||
`ROCM_ATTN` cannot use its native kernel for this model and its decode slows
|
||||
sharply as the prompt grows; `ROCM_AITER_FA` stays flat.
|
||||
|
||||
```bash
|
||||
VLLM_ROCM_USE_AITER=1 vllm serve Qwen/Qwen3.6-27B \
|
||||
--attention-backend ROCM_AITER_FA \
|
||||
--enable-auto-tool-choice --tool-call-parser qwen3_coder \
|
||||
--reasoning-parser qwen3
|
||||
```
|
||||
|
||||
MTP works with the same flags (add the **Speculative decoding** feature above).
|
||||
Single-stream decode (tokens/s at ~200 / ~2k / ~12k / ~32k prompt tokens, 512
|
||||
output tokens, prefix caching on, which is the default), measured 2026-10-12 on
|
||||
one MI300X with `vllm/vllm-openai-rocm:v0.31.0`:
|
||||
|
||||
| | MI300X |
|
||||
|---|---|
|
||||
| image defaults (`ROCM_ATTN`) | 66 / 56 / 27 / 13 |
|
||||
| AITER + `ROCM_AITER_UNIFIED_ATTN` | 66 / 66 / 60 / 49 |
|
||||
| AITER + `ROCM_AITER_FA` | 67 / 67 / 65 / 62 |
|
||||
| + MTP (`num_speculative_tokens: 3`) | 139 / 176 / 168 / 157 |
|
||||
|
||||
The synthetic prompts are repetitive, which flatters MTP; on real coding-agent
|
||||
requests (10-25k-token contexts) decode averaged 27 tok/s with image defaults,
|
||||
68 with `ROCM_AITER_FA` and 161 with MTP added. GSM8K (first 250 test
|
||||
questions, greedy, thinking off) is unchanged: 0.952 image defaults, 0.952
|
||||
`ROCM_AITER_FA`, 0.964 `ROCM_AITER_UNIFIED_ATTN`, 0.964 `ROCM_AITER_FA` + MTP.
|
||||
|
||||
### Docker (Intel XPU B60 / B70)
|
||||
|
||||
Validated on 4× Intel Arc Pro B60 or 2x Intel Arc Pro B70 (B60 24 GB, B70 32 GB per card) with the
|
||||
official vLLM XPU image `vllm/vllm-openai-xpu:latest`.
|
||||
|
||||
```bash
|
||||
docker run --device /dev/dri \
|
||||
-v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
|
||||
--privileged --ipc=host -p 8000:8000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai-xpu:latest \
|
||||
Qwen/Qwen3.6-27B \
|
||||
--reasoning-parser qwen3 \
|
||||
--tensor-parallel-size 4 \
|
||||
--max-model-len 8192 \
|
||||
--enforce-eager
|
||||
```
|
||||
|
||||
## Processing Ultra-Long Texts
|
||||
|
||||
Qwen3.6-27B natively supports `262,144` tokens. To serve longer inputs, turn on the
|
||||
**Long Context** feature above: it applies the model card's YaRN RoPE
|
||||
configuration via `--hf-overrides` and raises `--max-model-len` to ~1,010,000 tokens.
|
||||
Static YaRN holds the scaling factor constant regardless of input length, so
|
||||
leave it off unless you actually serve long prompts; for a ~524K workload,
|
||||
halve the card's `factor` to `2.0`.
|
||||
|
||||
See the [model card](https://huggingface.co/Qwen/Qwen3.6-27B#processing-ultra-long-texts) for the full parameter reference.
|
||||
|
||||
## Client Usage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
|
||||
resp = client.chat.completions.create(
|
||||
model="Qwen/Qwen3.6-27B",
|
||||
messages=[{"role": "user", "content": "Write a haiku about gated delta networks."}],
|
||||
max_tokens=256,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **CUDA graph / Mamba cache size error:** reduce `--max-cudagraph-capture-size`
|
||||
(default 512). See [vLLM PR #34571](https://github.com/vllm-project/vllm/pull/34571).
|
||||
- **Disable reasoning:** add `--default-chat-template-kwargs '{"enable_thinking": false}'`.
|
||||
- **Prefix Caching (Mamba):** currently experimental in "align" mode.
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/Qwen/Qwen3.6-27B)
|
||||
- [FP8 checkpoint](https://huggingface.co/Qwen/Qwen3.6-27B-FP8)
|
||||
- [GPTQ-Int4 checkpoint](https://huggingface.co/Qwen/Qwen3.6-27B-GPTQ-Int4)
|
||||
- [NVIDIA NVFP4 checkpoint](https://huggingface.co/nvidia/Qwen3.6-27B-NVFP4)
|
||||
- [Ascend W8A8 checkpoint](https://www.modelscope.cn/models/Eco-Tech/Qwen3.6-27B-w8a8-mxfp8)
|
||||
- [Ascend A3 W8A8 checkpoint](https://www.modelscope.cn/models/Eco-Tech/Qwen3.6-27B-w8a8)
|
||||
- [vLLM Ascend tutorial](https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/Qwen3.5-27B-Qwen3.6-27B.html)
|
||||
- [Qwen3.6-35B-A3B recipe](../Qwen3.6-35B-A3B)
|
||||
@@ -0,0 +1,381 @@
|
||||
meta:
|
||||
title: "Qwen3.6-35B-A3B"
|
||||
slug: "qwen3.6-35b-a3b"
|
||||
provider: "Qwen"
|
||||
description: "Smaller Qwen3.6 multimodal MoE model (35B total / 3B active) with BF16, FP8, and NVIDIA NVFP4 variants"
|
||||
date_added: 2026-04-16
|
||||
date_updated: 2026-09-14
|
||||
tasks:
|
||||
- multimodal
|
||||
- text
|
||||
performance_headline: "Compact Qwen3.6 MoE with 3B active parameters — single-GPU FP8 or 2-4 GPU or 4x Intel Arc Pro B60/B70 BF16 serving"
|
||||
related_recipes:
|
||||
- "Qwen/Qwen3.5-397B-A17B"
|
||||
hardware:
|
||||
dgx_spark_gb10: verified
|
||||
h100: verified
|
||||
h200: verified
|
||||
mi300x: verified
|
||||
mi325x: verified
|
||||
mi355x: verified
|
||||
rtx_pro_6000: verified
|
||||
dgx_station_gb300: verified
|
||||
rtx_5090: verified
|
||||
arc_pro_b60: verified
|
||||
arc_pro_b70: verified
|
||||
|
||||
model:
|
||||
model_id: "Qwen/Qwen3.6-35B-A3B"
|
||||
min_vllm_version: "0.17.0"
|
||||
default_frontend: rust
|
||||
architecture: moe
|
||||
parameter_count: "35B"
|
||||
active_parameters: "3B"
|
||||
context_length: 262144
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with Qwen3 Coder parser"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "qwen3_coder"
|
||||
reasoning:
|
||||
description: "Enable chain-of-thought reasoning with Qwen3 parser"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "qwen3"
|
||||
spec_decoding:
|
||||
description: "Multi-token prediction speculative decoding for lower latency"
|
||||
default_mode: mtp
|
||||
modes:
|
||||
mtp:
|
||||
label: "MTP"
|
||||
description: "Built-in Multi-Token Prediction, 3 draft tokens."
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"mtp","num_speculative_tokens":3,"moe_backend":"triton"}'
|
||||
hardware_overrides:
|
||||
rtx_pro_6000:
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"mtp","num_speculative_tokens":4,"moe_backend":"flashinfer_cutlass"}'
|
||||
rtx_5090:
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"mtp","num_speculative_tokens":4,"moe_backend":"flashinfer_cutlass"}'
|
||||
dspark:
|
||||
label: "DSpark"
|
||||
description: "DSpark drafting with an externally trained draft model; the served checkpoint is unchanged."
|
||||
variants:
|
||||
- default
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"model": "RedHatAI/Qwen3.6-35B-A3B-speculator.dspark", "num_speculative_tokens": 7, "method": "dspark"}'
|
||||
text_only:
|
||||
description: "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
|
||||
args:
|
||||
- "--language-model-only"
|
||||
encoder_parallel:
|
||||
description: "Run the vision encoder in data-parallel mode — avoids TP comm overhead on the small encoder. Mutually exclusive with text_only."
|
||||
args:
|
||||
- "--mm-encoder-tp-mode"
|
||||
- "data"
|
||||
long_context:
|
||||
description: "YaRN RoPE scaling past the native 262,144-token window to ~1,010,000 tokens. Static YaRN holds the scaling factor constant regardless of input length, so leave this off unless you actually serve long prompts; for ~524K, halve the factor to 2.0."
|
||||
args:
|
||||
- "--hf-overrides"
|
||||
- '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}'
|
||||
- "--max-model-len"
|
||||
- "1010000"
|
||||
env:
|
||||
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
|
||||
|
||||
opt_in_features:
|
||||
- spec_decoding
|
||||
- text_only
|
||||
- long_context
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 84
|
||||
description: "Full precision BF16 — fits on 1x H200, 2x H100, 4x Intel Arc Pro B60 / B70"
|
||||
hardware_overrides:
|
||||
rtx_pro_6000:
|
||||
extra_args:
|
||||
- "--max-num-seqs"
|
||||
- "8"
|
||||
fp8:
|
||||
model_id: "Qwen/Qwen3.6-35B-A3B-FP8"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 42
|
||||
description: "Qwen official FP8 checkpoint — single-GPU serving"
|
||||
hardware_overrides:
|
||||
rtx_pro_6000:
|
||||
extra_env:
|
||||
VLLM_USE_DEEP_GEMM: "0"
|
||||
nvfp4:
|
||||
model_id: "nvidia/Qwen3.6-35B-A3B-NVFP4"
|
||||
precision: nvfp4
|
||||
vram_minimum_gb: 21
|
||||
tp: 1
|
||||
min_vllm_version: "0.28.0"
|
||||
docker_image: "vllm/vllm-openai:v0.28.0"
|
||||
description: "NVIDIA ModelOpt NVFP4 checkpoint for Blackwell GPUs, including DGX Spark"
|
||||
hardware_overrides:
|
||||
dgx_spark_gb10:
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
- "--attention-backend"
|
||||
- "flashinfer"
|
||||
- "--moe-backend"
|
||||
- "marlin"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.5"
|
||||
- "--max-model-len"
|
||||
- "262144"
|
||||
- "--max-num-seqs"
|
||||
- "8"
|
||||
- "--max-num-batched-tokens"
|
||||
- "8192"
|
||||
- "--enable-chunked-prefill"
|
||||
- "--async-scheduling"
|
||||
- "--enable-prefix-caching"
|
||||
- "--load-format"
|
||||
- "fastsafetensors"
|
||||
dgx_station_gb300:
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
- "--attention-backend"
|
||||
- "flashinfer"
|
||||
- "--moe-backend"
|
||||
- "marlin"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.92"
|
||||
- "--max-model-len"
|
||||
- "262144"
|
||||
- "--max-num-batched-tokens"
|
||||
- "8192"
|
||||
- "--enable-chunked-prefill"
|
||||
- "--async-scheduling"
|
||||
- "--enable-prefix-caching"
|
||||
- "--load-format"
|
||||
- "fastsafetensors"
|
||||
rtx_pro_6000:
|
||||
extra_args:
|
||||
- "--quantization"
|
||||
- "modelopt_fp4"
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
- "--block-size"
|
||||
- "128"
|
||||
- "--moe-backend"
|
||||
- "marlin"
|
||||
- "--attention-backend"
|
||||
- "flashinfer"
|
||||
- "--attention-config"
|
||||
- '{"use_trtllm_attention":true}'
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.9"
|
||||
- "--max-num-seqs"
|
||||
- "8"
|
||||
- "--max-num-batched-tokens"
|
||||
- "8192"
|
||||
- "--async-scheduling"
|
||||
- "--enable-chunked-prefill"
|
||||
- "--enable-prefix-caching"
|
||||
- "--skip-mm-profiling"
|
||||
- "--load-format"
|
||||
- "fastsafetensors"
|
||||
extra_env:
|
||||
VLLM_HAS_FLASHINFER_CUBIN: "1"
|
||||
rtx_5090:
|
||||
extra_args:
|
||||
- "--quantization"
|
||||
- "modelopt_fp4"
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
- "--block-size"
|
||||
- "128"
|
||||
- "--moe-backend"
|
||||
- "marlin"
|
||||
- "--attention-backend"
|
||||
- "flashinfer"
|
||||
- "--attention-config"
|
||||
- '{"use_trtllm_attention":true}'
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.85"
|
||||
- "--max-model-len"
|
||||
- "65536"
|
||||
- "--max-num-seqs"
|
||||
- "8"
|
||||
- "--max-num-batched-tokens"
|
||||
- "8192"
|
||||
- "--enable-chunked-prefill"
|
||||
- "--enable-prefix-caching"
|
||||
- "--async-scheduling"
|
||||
- "--skip-mm-profiling"
|
||||
extra_env:
|
||||
VLLM_HAS_FLASHINFER_CUBIN: "1"
|
||||
|
||||
redhat_nvfp4:
|
||||
model_id: "RedHatAI/Qwen3.6-35B-A3B-NVFP4"
|
||||
precision: nvfp4
|
||||
label: "NVFP4 (RedHat)"
|
||||
vram_minimum_gb: 21
|
||||
description: "RedHatAI compressed-tensors NVFP4 quantization for Blackwell GPUs"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
|
||||
redhat_fp8:
|
||||
model_id: "RedHatAI/Qwen3.6-35B-A3B-FP8-dynamic"
|
||||
precision: fp8
|
||||
label: "FP8 Dynamic (RedHat)"
|
||||
vram_minimum_gb: 42
|
||||
description: "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
|
||||
redhat_fp8_plain:
|
||||
model_id: "RedHatAI/Qwen3.6-35B-A3B-FP8"
|
||||
precision: fp8
|
||||
label: "FP8 (RedHat)"
|
||||
vram_minimum_gb: 42
|
||||
description: "RedHatAI FP8 quantization"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- single_node_tep
|
||||
- single_node_dep
|
||||
- multi_node_tp
|
||||
- multi_node_tp_pp
|
||||
- multi_node_dep
|
||||
- multi_node_tep
|
||||
|
||||
hardware_overrides:
|
||||
amd:
|
||||
extra_args: []
|
||||
extra_env:
|
||||
VLLM_ROCM_USE_AITER: "1"
|
||||
xpu:
|
||||
extra_args:
|
||||
- "--enforce-eager"
|
||||
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[Qwen3.6-35B-A3B](https://huggingface.co/Qwen/Qwen3.6-35B-A3B) is the smaller sibling
|
||||
of Qwen3.5, sharing the same gated-delta-networks MoE architecture but with 35B total
|
||||
parameters and 3B activated (256 experts, 8 routed + 1 shared). The recipe covers the
|
||||
BF16 base model, Qwen's official FP8 checkpoint, and NVIDIA's ModelOpt NVFP4
|
||||
checkpoint.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **vLLM version:** >= 0.17.0
|
||||
- **NVFP4 vLLM version:** >= 0.28.0. 0.24.0 loads the checkpoint, but
|
||||
FlashInfer only selects its XQA decode kernel on SM120 GPUs from 0.28.0.
|
||||
- **Hardware (BF16):** 1x H200, 2x H100, 4x Intel Arc Pro B60 / B70
|
||||
- **Hardware (FP8):** single H100/H200 or 1x MI300X/MI325X/MI355X
|
||||
- **Hardware (NVFP4):** NVIDIA Blackwell GPUs, including DGX Spark (GB10)
|
||||
|
||||
## NVFP4 on Blackwell
|
||||
|
||||
All NVFP4 tuning lives in `variants.nvfp4.hardware_overrides`, keyed by GPU
|
||||
profile, so the exact flag set for a box is readable straight from the recipe
|
||||
YAML (or from `by_hardware` in the JSON API) without going through the
|
||||
command builder.
|
||||
|
||||
Four profiles are tuned and verified. DGX Spark (GB10) and DGX Station
|
||||
(GB300) run the checkpoint on the default loader with FlashInfer attention,
|
||||
`--moe-backend marlin` and a 3-token MTP draft on Triton; Spark shares
|
||||
unified memory with the host, so `--gpu-memory-utilization` is capped at 0.5.
|
||||
The SM120 parts (RTX PRO 6000, RTX 5090) load through `--quantization
|
||||
modelopt_fp4` with FlashInfer's TRT-LLM attention kernels, which need
|
||||
`--block-size 128` and `VLLM_HAS_FLASHINFER_CUBIN=1`, and sustain a deeper
|
||||
4-token MTP draft on `flashinfer_cutlass`. The RTX 5090's 32 GB caps context
|
||||
at 64K.
|
||||
|
||||
Datacenter Blackwell (B200/GB200/B300/GB300) intentionally carries no
|
||||
overrides — it runs vLLM's own NVFP4 defaults, which select the native FP4
|
||||
path rather than the Marlin W4A16 kernel the workstation parts need.
|
||||
|
||||
### Measured on DGX Spark (GB10)
|
||||
|
||||
The GB10 profile above was re-measured on 2026-08-31 with aiperf 0.12.0 on
|
||||
vLLM 0.28.0 (Ubuntu 24.04.4, driver 580.159.03, CUDA 13.0), one GB10 at TP1,
|
||||
serving from `vllm/vllm-openai:v0.28.0`. On SPEED-Bench at 8K input / 256
|
||||
output tokens and concurrency 1, it sustained 6,265 prefill tok/s and 97.7
|
||||
decode tok/s at a 3.95 s mean end-to-end latency. The flag set is unchanged —
|
||||
the sweep confirms the tuning rather than revising it, so the command the
|
||||
builder emits for GB10 is the one that produced these numbers.
|
||||
|
||||
### Docker (Intel XPU B60 / B70)
|
||||
|
||||
Validated on 4× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the
|
||||
official vLLM XPU image `vllm/vllm-openai-xpu:latest`.
|
||||
|
||||
```bash
|
||||
docker run --device /dev/dri \
|
||||
-v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
|
||||
--privileged --ipc=host -p 8000:8000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai-xpu:latest \
|
||||
Qwen/Qwen3.6-35B-A3B \
|
||||
--reasoning-parser qwen3 \
|
||||
--tensor-parallel-size 4 \
|
||||
--max-model-len 8192 \
|
||||
--enforce-eager
|
||||
```
|
||||
|
||||
## Processing Ultra-Long Texts
|
||||
|
||||
Qwen3.6-35B-A3B natively supports `262,144` tokens. To serve longer inputs, turn on the
|
||||
**Long Context** feature above: it applies the model card's YaRN RoPE
|
||||
configuration via `--hf-overrides` and raises `--max-model-len` to ~1,010,000 tokens.
|
||||
Static YaRN holds the scaling factor constant regardless of input length, so
|
||||
leave it off unless you actually serve long prompts; for a ~524K workload,
|
||||
halve the card's `factor` to `2.0`.
|
||||
|
||||
See the [model card](https://huggingface.co/Qwen/Qwen3.6-35B-A3B#processing-ultra-long-texts) for the full parameter reference.
|
||||
|
||||
## Client Usage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
|
||||
resp = client.chat.completions.create(
|
||||
model="Qwen/Qwen3.6-35B-A3B",
|
||||
messages=[{"role": "user", "content": "Explain gated delta networks in one paragraph."}],
|
||||
max_tokens=512,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **CUDA graph / Mamba cache size error:** reduce `--max-cudagraph-capture-size`
|
||||
(default 512). See [vLLM PR #34571](https://github.com/vllm-project/vllm/pull/34571).
|
||||
- **Reasoning disable:** add `--default-chat-template-kwargs '{"enable_thinking": false}'`.
|
||||
- **Prefix Caching (Mamba):** currently experimental in "align" mode.
|
||||
|
||||
## References
|
||||
|
||||
- [Qwen3.6-35B-A3B on Hugging Face](https://huggingface.co/Qwen/Qwen3.6-35B-A3B)
|
||||
- [FP8 checkpoint](https://huggingface.co/Qwen/Qwen3.6-35B-A3B-FP8)
|
||||
- [NVIDIA NVFP4 checkpoint](https://huggingface.co/nvidia/Qwen3.6-35B-A3B-NVFP4)
|
||||
- [Qwen3.5 recipe (sibling 397B-A17B flagship)](../Qwen3.5-397B-A17B)
|
||||
@@ -0,0 +1,436 @@
|
||||
meta:
|
||||
title: "Qwen3.8-2.4T-A95B"
|
||||
slug: "qwen3.8-2.4t-a95b"
|
||||
provider: "Qwen"
|
||||
description: "2.4T-parameter hybrid-attention MoE (~95B active) with linear attention on 69 of 92 layers, 512 routed experts, a built-in MTP draft head, 262K native context window and extensible to 1M context"
|
||||
date_added: 2026-08-10
|
||||
date_updated: 2026-09-14
|
||||
tasks:
|
||||
- text
|
||||
# Measured, not a capacity claim. The previous headline said "down to a single node",
|
||||
# which is an 8-GPU statement and false on the NVL4 trays (GB200/GB300), where a node is
|
||||
# 4 GPUs — NVFP4 is 1348 GiB against 1110 GiB per tray. Count GPUs, and lead with the
|
||||
# two numbers people actually choose between: per-user latency and per-GPU throughput.
|
||||
performance_headline: "NVFP4 W4A4 on 8 Blackwell GPUs: 304 output tok/s per user with MTP-3, up to 4300 total tok/s per GPU at DEP16"
|
||||
related_recipes:
|
||||
- "Qwen/Qwen3.5-397B-A17B"
|
||||
# B300 is the profile the headline numbers above were taken against, so the page and
|
||||
# the JSON API open there rather than on the heuristic's H200. A visitor who has picked
|
||||
# their own GPU still wins — the stored preference outranks this.
|
||||
default_hardware: b300
|
||||
# `verified` = run end-to-end on that GPU. gb300 earned it on 2026-08-13: NVFP4 W4A4
|
||||
# served at TP8 across two trays on vLLM 0.1.dev19754+g3a0914114, correct generations,
|
||||
# full CUDA-graph capture, and a concurrency sweep (see the Performance section).
|
||||
# Everything else stays absent — the honest silent default. Nothing is `unsupported`:
|
||||
# no configuration has been observed to fail.
|
||||
hardware:
|
||||
b300: verified
|
||||
gb300: verified
|
||||
|
||||
model:
|
||||
# Upstream renamed this repo on 2026-08-10: it was `Qwen/Qwen3.8-2.4T-A95B-0807`, and
|
||||
# that id now redirects here. A DIFFERENT, older checkpoint used to hold this name and
|
||||
# was moved to `Qwen/Qwen3.8-2.4T-A95B-Deprecated` — 260 files, a per-expert tensor
|
||||
# layout, and `Qwen3_5MoeForConditionalGeneration` with a vision tower. Do not point at
|
||||
# the Deprecated repo; it is a different model, not an older commit of this one.
|
||||
model_id: "Qwen/Qwen3.8-2.4T-A95B"
|
||||
# Verified against upstream, not taken from the model card (the README is a stub):
|
||||
# `Qwen3_5MoeForCausalLM` — the architecture this checkpoint declares — is present in
|
||||
# vllm/model_executor/models/registry.py on `main` but ABSENT at v0.26.0, the latest
|
||||
# stable release, whose docs/models/supported_models.md lists only the
|
||||
# `*ForConditionalGeneration` members of the Qwen3.5 family. Nightly is a real
|
||||
# requirement here. Re-check at the next stable tag and pin it.
|
||||
min_vllm_version: "nightly"
|
||||
default_frontend: rust
|
||||
nightly_required: true
|
||||
docker_image:
|
||||
nvidia: "vllm/vllm-openai:qwen38"
|
||||
amd: "vllm/vllm-openai-rocm:qwen38"
|
||||
architecture: moe
|
||||
parameter_count: "2.4T"
|
||||
active_parameters: "95B"
|
||||
# 262144 is what config.json (max_position_embeddings) and tokenizer_config.json both
|
||||
# declare; 1010000 is the extended ceiling upstream ships this recipe with, forced via
|
||||
# base_args below. The extension has NOT been verified against the checkpoint here — the
|
||||
# staged copy tops out at 262144 on both fields.
|
||||
context_length: 262144
|
||||
# No --trust-remote-code: config.json carries no `auto_map`, so there is no remote code
|
||||
# to trust — vLLM's own handler loads this architecture.
|
||||
base_args: []
|
||||
base_env:
|
||||
# Weight load runs from 1.32 TiB (NVFP4) to 4.45 TiB (BF16). The default readiness
|
||||
# window expires long before the first token on a cold shared filesystem.
|
||||
VLLM_ENGINE_READY_TIMEOUT_S: "3600"
|
||||
|
||||
dependencies:
|
||||
- note: "config.json declares transformers_version 5.4.0; older releases do not know this model_type."
|
||||
command: 'uv pip install -U "transformers>=5.4.0"'
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice. The chat template emits tool calls as <tool_call><function=…><parameter=…> XML, which is the qwen3_coder format — not JSON."
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "qwen3_coder"
|
||||
reasoning:
|
||||
description: "Parse <think> blocks separately from the final answer. The template opens every assistant turn with <think>, so without this the reasoning lands in message.content."
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "qwen3"
|
||||
spec_decoding:
|
||||
description: "Multi-token prediction using the draft head shipped inside the checkpoint — no separate speculator repo needed."
|
||||
# config.json sets mtp_num_hidden_layers: 1 and the weight index carries `mtp.*`
|
||||
# tensors, so the draft head is present. Depth is 3, NOT 1: the single draft layer is
|
||||
# run autoregressively, and depth is a throughput/latency knob rather than a property
|
||||
# of the checkpoint. Measured, MTP-1 is not worth enabling — 64.8% acceptance but only
|
||||
# +3.4% at concurrency 1 and a LOSS above it (-9% at 128, -23% at 256), because the
|
||||
# draft pass displaces real work once the batch is compute-bound. Depth 3 is where it
|
||||
# pays: ~2.3x per-user output rate (130 -> 307 tok/s on FP8 TP16, 133 -> 304 on NVFP4
|
||||
# TP8). Re-tune against your own acceptance rate if your traffic differs.
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"mtp","num_speculative_tokens":3}'
|
||||
|
||||
opt_in_features:
|
||||
- spec_decoding
|
||||
|
||||
# vram_minimum_gb is ceil(real safetensors total_size in GB x 1.2) for every variant, not
|
||||
# params x bytes_per_param: at 2.4T the rounded parameter count is worth ~50 GB of error,
|
||||
# and the quantized builds are not uniformly 4-bit.
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
# index total_size 4,892,388,705,238 B = 4892.4 GB
|
||||
vram_minimum_gb: 5871
|
||||
description: "Full-precision BF16 — 4.45 TiB of weights: 24 GPUs, i.e. 3x 8xB300 nodes or 6 GB300 NVL4 trays."
|
||||
fp8:
|
||||
model_id: "Qwen/Qwen3.8-2.4T-A95B-FP8"
|
||||
precision: fp8
|
||||
# index total_size 2,496,154,358,768 B = 2496.2 GB. Block-scaled FP8
|
||||
# (quantization_config.weight_block_size [128, 128]).
|
||||
vram_minimum_gb: 2996
|
||||
description: "Official block-scaled FP8 checkpoint — 2.27 TiB: 16 GPUs, i.e. 2x 8xB300 nodes or 4 GB300 NVL4 trays (3 trays hold it, but TP12 is invalid — 64 attention heads are not divisible by 12)."
|
||||
mxfp4:
|
||||
model_id: "Inferact/Qwen3.8-2.4T-A95B-MXFP4"
|
||||
precision: mxfp4
|
||||
label: "MXFP4 (AMD)"
|
||||
# Sized from the staged copy of this build: config.json quantization_config
|
||||
# quant_method `mxfp4`, index total_size 1,597,051,793,152 B = 1597.1 GB.
|
||||
# It declares the same Qwen3_5MoeForCausalLM / 92 layers / 512 experts as the base.
|
||||
vram_minimum_gb: 1917
|
||||
# AMD-targeted build. MXFP4 is not brand-restricted by default in this catalog, so the
|
||||
# allowlist is explicit rather than implied by the precision.
|
||||
supported_hardware: [mi300x, mi325x, mi355x]
|
||||
description: "MXFP4 build for AMD Instinct — 1.45 TiB: 8 GPUs, a single 8xMI355X or 8xMI325X node."
|
||||
nvfp4:
|
||||
model_id: "Inferact/Qwen3.8-2.4T-A95B-NVFP4"
|
||||
precision: nvfp4
|
||||
label: "NVFP4 (W4A4)"
|
||||
# Sized from the staged copy: quant_method `modelopt`, quant_algo `NVFP4`, index
|
||||
# total_size 1,447,265,951,488 B = 1447.3 GB. Same base architecture as above.
|
||||
vram_minimum_gb: 1737
|
||||
# NVFP4 already implies NVIDIA Blackwell in this catalog; listed explicitly because
|
||||
# W4A4 quantizes activations too and has not been exercised on every profile here.
|
||||
supported_hardware: [b200, b300, gb200, gb300]
|
||||
# FlashInfer CuTe-DSL GEMM kernels. vLLM's own description of this value is
|
||||
# "(NVFP4, MXFP8)", so it belongs to this variant specifically — not to the
|
||||
# block-scaled FP8 checkpoint, not to BF16 (no quantized linears at all), and not to
|
||||
# the MXFP4 build, whose ROCm path is the `aiter` backend instead. CUDA-only.
|
||||
extra_args:
|
||||
- "--linear-backend"
|
||||
- "flashinfer_cutedsl"
|
||||
description: "NVFP4 build with 4-bit activations for NVIDIA Blackwell — 1.32 TiB, the smallest footprint: 8 GPUs, i.e. one 8xB300 node or 2 GB300 NVL4 trays."
|
||||
nvfp4_reap25:
|
||||
model_id: "RedHatAI/Qwen3.8-2.4T-A95B-NVFP4-REAP-25"
|
||||
precision: nvfp4
|
||||
label: "NVFP4-REAP-25 (RedHatAI)"
|
||||
# (Router-weighted Expert Activation Pruning, arxiv.org/abs/2510.13999) drops 128 of
|
||||
# the 512 routed experts before quantization (config.json num_experts: 384),
|
||||
# calibrated on 1024 samples from mlabonne/open-perfectblend.
|
||||
vram_minimum_gb: 1373
|
||||
supported_hardware: [b200, b300, gb200, gb300]
|
||||
description: "NVFP4 quantization + 25% uniform expert pruning via REAP — 4.25x compression with ~99% accuracy recovery"
|
||||
|
||||
# Full MoE strategy set. Note the 3.4x spread between variants: NVFP4/MXFP4 fit 8 GPUs
|
||||
# while BF16 needs 24, so no per-STRATEGY floor is set — one sized for BF16 would block
|
||||
# the 8-GPU layout the quantized builds exist to enable.
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- single_node_tep
|
||||
- single_node_dep
|
||||
- multi_node_tp
|
||||
- multi_node_tep
|
||||
- multi_node_dep
|
||||
- multi_node_tp_pp
|
||||
- multi_node_tp_dp
|
||||
- pd_cluster
|
||||
|
||||
# A gb300 "node" is a 4-GPU NVL4 tray (288 GB/GPU, 1152 GB/tray). The SMALLEST variant
|
||||
# (NVFP4, 1737 GB sized) does not fit one, so no single-node strategy is reachable on this
|
||||
# GPU for ANY variant — measured, not inferred: TP8 across two trays is the minimum that
|
||||
# loads. 8 is therefore the floor every variant clears.
|
||||
#
|
||||
# It is deliberately the NVFP4 number rather than a per-variant one. `strategy_min_gpus`
|
||||
# has no per-variant keyspace, and sizing this key for BF16 (24) would block the 8-GPU
|
||||
# layout the quantized builds exist to enable. The bigger variants need more — FP8 16,
|
||||
# BF16 24 — and say so in their `description`, which is what the Variant row shows.
|
||||
strategy_min_gpus:
|
||||
gb300: 8
|
||||
|
||||
hardware_overrides: {}
|
||||
|
||||
# `--all2all-backend` only governs MoE expert-parallel communication, so it is set on the
|
||||
# EP strategies and nowhere else: under plain TP the 512 routed experts are TP-sharded and
|
||||
# no all2all happens, so the flag would be inert noise on single_node_tp / multi_node_tp /
|
||||
# multi_node_tp_pp.
|
||||
#
|
||||
# `allgather_reducescatter` is vLLM's CURRENT DEFAULT (vllm/config/parallel.py) and stays
|
||||
# the portable baseline. The previous revision said to swap it "only with a measurement to
|
||||
# justify it" — that measurement now exists, so Blackwell moves to the one-sided NVLink
|
||||
# all2all: DEP16 on NVFP4 reaches ~4300 total TPS/GPU and TP4·DP4 + EP16 on FP8 reaches
|
||||
# ~3200, against ~2170 measured on plain TP8 with the default backend.
|
||||
#
|
||||
# It is gated to `blackwell` rather than applied outright because
|
||||
# `flashinfer_nvlink_one_sided` needs an NVLink domain spanning the EP group. That holds
|
||||
# on NVL4/NVL72 Blackwell; it does not hold for Hopper multi-node (no cross-node NVLink)
|
||||
# and has no ROCm build, so both keep the allgather path.
|
||||
strategy_overrides:
|
||||
single_node_tep:
|
||||
extra_args:
|
||||
- "--all2all-backend"
|
||||
- "allgather_reducescatter"
|
||||
hardware_overrides:
|
||||
blackwell:
|
||||
extra_args:
|
||||
- "--all2all-backend"
|
||||
- "flashinfer_nvlink_one_sided"
|
||||
single_node_dep:
|
||||
extra_args:
|
||||
- "--all2all-backend"
|
||||
- "allgather_reducescatter"
|
||||
hardware_overrides:
|
||||
blackwell:
|
||||
extra_args:
|
||||
- "--all2all-backend"
|
||||
- "flashinfer_nvlink_one_sided"
|
||||
multi_node_tep:
|
||||
extra_args:
|
||||
- "--all2all-backend"
|
||||
- "allgather_reducescatter"
|
||||
hardware_overrides:
|
||||
blackwell:
|
||||
extra_args:
|
||||
- "--all2all-backend"
|
||||
- "flashinfer_nvlink_one_sided"
|
||||
multi_node_dep:
|
||||
extra_args:
|
||||
- "--all2all-backend"
|
||||
- "allgather_reducescatter"
|
||||
hardware_overrides:
|
||||
blackwell:
|
||||
extra_args:
|
||||
- "--all2all-backend"
|
||||
- "flashinfer_nvlink_one_sided"
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
Qwen3.8-2.4T-A95B is a 2.4-trillion-parameter Mixture-of-Experts model with roughly 95B
|
||||
parameters active per token — 512 routed experts with 10 active, plus one shared expert,
|
||||
over a 92-layer hybrid-attention backbone.
|
||||
|
||||
**The layer mix is the interesting part.** Only 23 of the 92 layers run full attention
|
||||
(`full_attention_interval: 4`); the other 69 run linear attention with a constant
|
||||
recurrent state. This is a text-only causal LM: the architecture is
|
||||
`Qwen3_5MoeForCausalLM`, `config.json` has no `vision_config`, and the weight index
|
||||
contains zero vision tensors.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **vLLM: nightly.** for the best performance
|
||||
- **transformers >= 5.4.0**, per the version recorded in `config.json`.
|
||||
|
||||
```bash
|
||||
uv venv && source .venv/bin/activate
|
||||
uv pip install -U vllm --extra-index-url https://wheels.vllm.ai/nightly
|
||||
uv pip install -U "transformers>=5.4.0"
|
||||
```
|
||||
|
||||
## Choosing a variant
|
||||
|
||||
Counted in **GPUs, not nodes** — a "node" is 8 GPUs on B300/MI355X/H200 but only 4 on
|
||||
the GB200/GB300 NVL4 trays, and that difference decides whether a variant fits.
|
||||
|
||||
| Variant | Weights | Sized | B300 (268 GB) | MI355X (288 GB) | H200 (141 GB) | GB300 tray (4 GPU) |
|
||||
|---|---:|---:|---:|---:|---:|---:|
|
||||
| BF16 | 4.45 TiB | 5871 GB | 24 GPUs | 24 GPUs | 48 GPUs | 6 trays |
|
||||
| FP8 | 2.27 TiB | 2996 GB | 16 GPUs | 16 GPUs | 32 GPUs | **4 trays (TP16)** |
|
||||
| MXFP4 (AMD) | 1.45 TiB | 1917 GB | — | **8 GPUs** | 16 GPUs | — |
|
||||
| NVFP4 W4A4 (NVIDIA) | 1.32 TiB | 1737 GB | **8 GPUs** | — | 16 GPUs | **2 trays (TP8)** |
|
||||
| NVFP4-REAP-25 (RedHatAI) | 1.04 TiB | 1373 GB | **8 GPUs** | — | 16 GPUs | **2 trays (TP8)** |
|
||||
|
||||
**TP must divide the 64 attention heads**, so only 1/2/4/8/16/32 are legal. This rounds
|
||||
some layouts up: FP8 needs 2325 GiB, which is 3 GB300 trays by capacity, but TP12 is
|
||||
invalid — so FP8 is a **16-GPU / 4-tray** deployment on GB300, not 12.
|
||||
|
||||
## Launch commands
|
||||
|
||||
### Low latency
|
||||
|
||||
NVFP4 W4A4, TP8 (one 8xB300 node, or two GB300 trays):
|
||||
|
||||
```bash
|
||||
vllm serve Inferact/Qwen3.8-2.4T-A95B-NVFP4 \
|
||||
--tensor-parallel-size 8 \
|
||||
--max-model-len 262144 \
|
||||
--kv-cache-dtype fp8 \
|
||||
--reasoning-parser qwen3 \
|
||||
--enable-auto-tool-choice --tool-call-parser qwen3_coder
|
||||
```
|
||||
|
||||
FP8, TP16 across two 8-GPU nodes. Same command on both, changing only `--node-rank`;
|
||||
rank > 0 additionally takes `--headless` (it must not start an API server).
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3.8-2.4T-A95B-FP8 \
|
||||
--tensor-parallel-size 16 \
|
||||
--nnodes 2 --node-rank 0 --master-addr $HEAD_ADDR \
|
||||
--max-model-len 262144 \
|
||||
--kv-cache-dtype fp8 \
|
||||
--reasoning-parser qwen3
|
||||
```
|
||||
|
||||
Add `--speculative-config '{"method":"mtp","num_speculative_tokens":3}'` for the
|
||||
latency numbers below — MTP-3 is worth roughly 2.3x on per-user output rate.
|
||||
|
||||
MXFP4 on 8xMI355X — swap the model id and drop `--kv-cache-dtype fp8` unless you have
|
||||
confirmed the ROCm path supports it in your build.
|
||||
|
||||
### High throughput
|
||||
|
||||
Expert parallelism, not plain TP, and a one-sided NVLink all2all. The recipe sets
|
||||
`--all2all-backend flashinfer_nvlink_one_sided` automatically on the EP strategies when
|
||||
the hardware is Blackwell. The MoE kernel differs by precision, so set it explicitly:
|
||||
|
||||
- **FP8, TP4·DP4 + EP16** (`multi_node_tp_dp`, 4 GB300 trays): add
|
||||
`--moe-backend flashinfer_trtllm --enable-expert-parallel`. See
|
||||
<https://github.com/vllm-project/vllm/pull/51924>.
|
||||
- **NVFP4, DEP16** (`multi_node_dep`, 4 trays): add `--moe-backend flashinfer_cutedsl`.
|
||||
|
||||
These MoE backends are deliberately NOT pinned in the variant args: they are measured
|
||||
for the EP layouts above, and the plain-TP path has not been measured with them. vLLM
|
||||
auto-selects `FLASHINFER_TRTLLM` for NVFP4 under TP, which is what the TP8 numbers below
|
||||
were taken with.
|
||||
|
||||
### Pipeline parallelism
|
||||
|
||||
`multi_node_tp_pp` **works** on this architecture — verified 2026-08-13 on GB300 with
|
||||
TP4 x PP3 across 3 trays on the FP8 checkpoint: it loads, captures full CUDA graphs
|
||||
(51 piecewise + 35 full-decode), and generates correctly. Earlier revisions of this
|
||||
recipe warned that hybrid linear-attention models had historically lagged on PP support;
|
||||
that caveat is resolved. It remains a fallback rather than a default — PP exists for GPU
|
||||
counts that pure TP cannot express (12 GPUs is 3 trays but TP12 is illegal), and TP or
|
||||
EP is faster whenever the GPU count is a power of two.
|
||||
|
||||
## Performance
|
||||
|
||||
Two operating points, because they want different layouts. Latency wants TP and
|
||||
speculative decoding; throughput wants expert parallelism.
|
||||
|
||||
**Low latency** — output tokens/s per user:
|
||||
|
||||
| Variant | Layout | no MTP | MTP-3 | with kernel opts |
|
||||
|---|---|---:|---:|---:|
|
||||
| FP8 | TP16 | 130 | **307** | 150 / 360 (target) |
|
||||
| NVFP4 | TP8 | 133 | **304** | — |
|
||||
|
||||
**High throughput** — total (input+output) tokens/s per GPU:
|
||||
|
||||
| Variant | Layout | Backends | total TPS/GPU |
|
||||
|---|---|---|---:|
|
||||
| FP8 | TP4·DP4 + EP16 | one-sided all2all + trtllm-gen MoE + activation opt | up to **3200** |
|
||||
| NVFP4 | DEP16 | one-sided all2all + cute-dsl MoE | up to **4300** |
|
||||
|
||||
The pending kernel work behind the "target" column is gdn decode fusion, split-k GEMM,
|
||||
and all-reduce fusion.
|
||||
|
||||
**Verification run** (what earned `gb300: verified`) — NVFP4 W4A4, TP8 across 2 trays,
|
||||
8k in / 1k out, vLLM `0.1.dev19754+g3a0914114`, no EP, default MoE backend:
|
||||
|
||||
| Concurrency | TPS/user | total TPS/GPU | TPOT |
|
||||
|---:|---:|---:|---:|
|
||||
| 1 | 101 | 108 | 9.9 ms |
|
||||
| 32 | 41.9 | 1338 | 23.9 ms |
|
||||
| 64 | 26.9 | 1715 | 37.2 ms |
|
||||
| 128 | 18.3 | 2170 | 54.7 ms |
|
||||
|
||||
That 2170 against 4300 is the cost of plain TP with the default all2all — it is the
|
||||
measurement the EP + one-sided-NVLink defaults above are based on.
|
||||
|
||||
Three things that moved the numbers materially:
|
||||
|
||||
- **Use MTP depth 3, not 1.** MTP-1 was measured at 64.8% acceptance and is not worth
|
||||
it: +3.4% at concurrency 1 and a *loss* above it (−9% at 128, −23% at 256), because
|
||||
the draft pass displaces real work once the batch is compute-bound. Depth 3 is where
|
||||
speculation pays (~2.3x per-user).
|
||||
- **`--load-format fastsafetensors --safetensors-load-strategy lazy`** cut weight load
|
||||
from 545 s to 306 s (−44%) on a 1.32 TiB checkpoint over shared storage.
|
||||
- **Size `--max-model-len` to the workload.** At 262144 the engine reserved KV for 25
|
||||
concurrent requests; at 9240 (8k in + 1k out) the same 70 GiB of KV serves 506.
|
||||
|
||||
## Client usage
|
||||
|
||||
`generation_config.json` ships `temperature: 1.0`, `top_p: 0.95`, `top_k: 20`.
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600)
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="Qwen/Qwen3.8-2.4T-A95B",
|
||||
messages=[{"role": "user", "content": "Give me three primes above 100."}],
|
||||
temperature=1.0, top_p=0.95, max_tokens=2048,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
## Processing Ultra-Long Texts
|
||||
The model has 262k native context length and it can be extended to 1M with --max-model-len
|
||||
flag. The best value can be picked based on the use case and GPU ram usage tradeoff.
|
||||
|
||||
For example, to enable full 1M context length on this model:
|
||||
|
||||
```bash
|
||||
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 vllm serve Qwen/Qwen3.8-2.4T-A95B \
|
||||
--max-model-len 1010000 \
|
||||
--hf-overrides '{"max_position_embeddings": 1010000}' \
|
||||
...
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**CUDA graph / recurrent-state cache size error.** On hybrid models an
|
||||
`assert num_cache_lines >= batch` failure means the CUDA-graph capture size exceeds the
|
||||
state cache. Reduce `--max-cudagraph-capture-size` (default 512), or fall back to
|
||||
`--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}'` if full capture
|
||||
misbehaves on this architecture. See <https://github.com/vllm-project/vllm/pull/34571>.
|
||||
|
||||
**Engine start times out.** Raise `VLLM_ENGINE_READY_TIMEOUT_S` (set to 3600 by this
|
||||
recipe) and prefer a real `/v1/chat/completions` probe over `/health` for readiness —
|
||||
loading terabytes plus kernel JIT takes minutes.
|
||||
|
||||
The linear backend feature "--linear-backend flashinfer_cutedsl" is Docker only until
|
||||
FlashInfer version upgrade in the vllm build. The NVFP4 variant sets that flag, so on a
|
||||
pip install either use the Docker image pinned above or drop the flag.
|
||||
|
||||
## References
|
||||
|
||||
- Model card: https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B
|
||||
- FP8 checkpoint: https://huggingface.co/Qwen/Qwen3.8-2.4T-A95B-FP8
|
||||
- MXFP4 build (AMD): https://huggingface.co/Inferact/Qwen3.8-2.4T-A95B-MXFP4
|
||||
- NVFP4 W4A4 build (NVIDIA): https://huggingface.co/Inferact/Qwen3.8-2.4T-A95B-NVFP4
|
||||
- NVFP4-REAP-25 build (RedHatAI): https://huggingface.co/RedHatAI/Qwen3.8-2.4T-A95B-NVFP4-REAP-25
|
||||
- REAP: Router-weighted Expert Activation Pruning: https://arxiv.org/abs/2510.13999
|
||||
- vLLM documentation: https://docs.vllm.ai/
|
||||
@@ -0,0 +1,612 @@
|
||||
meta:
|
||||
title: "Qwen3.8-27B"
|
||||
slug: "qwen3.8-27b"
|
||||
provider: "Qwen"
|
||||
description: "27B-parameter dense hybrid-attention model with linear attention on 48 of 64 layers, a vision tower, a built-in MTP draft head, 262K native context window and extensible to 1M context"
|
||||
date_added: 2026-08-14
|
||||
date_updated: 2026-10-15
|
||||
tasks:
|
||||
- multimodal
|
||||
- text
|
||||
performance_headline: "Fits one Blackwell GPU in every precision: NVFP4 in 24.6 GiB, 6.6M KV tokens at 1M context"
|
||||
related_recipes:
|
||||
- "Qwen/Qwen3.8-2.4T-A95B"
|
||||
# `verified` = run end-to-end on that GPU. gb300 earned it on 2026-08-14: BF16 and FP8 at
|
||||
# TP4, NVFP4 at TP1 on vLLM 0.1.dev19754+g3a0914114, correct generations at 1M context,
|
||||
# and working tool calls. Everything else stays absent — the honest silent default.
|
||||
hardware:
|
||||
gb300: verified
|
||||
# Blackwell B200, verified 2026-09-20: the RedHatAI MXFP4 variant (true W4A4, TP1) booted,
|
||||
# served and answered on 1x B200 — the same hardware used for the checkpoint's own eval.
|
||||
b200: verified
|
||||
# Consumer Blackwell (sm120), verified 2026-08-15 on vLLM
|
||||
# 0.26.1rc1.dev608+g99a10304d. Each entry was booted, served and answered a
|
||||
# verifiable prompt, with MTP acceptance read from /metrics rather than
|
||||
# inferred from throughput. rtx_5090 is NVFP4 on one card and requires
|
||||
# --enforce-eager; rtx_5090_2x covers FP8 and both NVFP4 builds at TP2.
|
||||
rtx_5090: verified
|
||||
rtx_5090_2x: verified
|
||||
rtx_pro_6000: verified
|
||||
dgx_spark_gb10: verified
|
||||
# Huawei Ascend NPU, verified 2026-08-18 on vLLM Ascend 0.23.0 (image
|
||||
# quay.io/ascend/vllm-ascend:qwen3.8-a5). The `ascend_w8a8` variant booted on a
|
||||
# single 950PR at TP1 with the in-checkpoint MTP head and FULL_DECODE_ONLY
|
||||
# ACLGraph and answered a long-form prompt correctly.
|
||||
ascend_950pr: verified
|
||||
arc_pro_b60: verified
|
||||
arc_pro_b70: verified
|
||||
|
||||
model:
|
||||
model_id: "Qwen/Qwen3.8-27B"
|
||||
# A NUMBER, not the string "nightly" the 2.4T sibling uses. `maxVersion`
|
||||
# (CommandBuilder.jsx:3146-3150) maps each dotted field through `parseInt(n,10) || 0`, so
|
||||
# "nightly" scores 0.0.0 and LOSES to any real version: pick a Mooncake KV-offload option
|
||||
# and the Install header would take the kv_store YAMLs' "0.21.0" instead, understating the
|
||||
# requirement.
|
||||
#
|
||||
# 0.17.0 is where `Qwen3_5ForConditionalGeneration` is already registered. The runs behind
|
||||
# this recipe all used the Docker image pinned below (a fork build in the 0.27 range), so
|
||||
# older releases are unverified rather than known-good — and `spec_decoding` in particular
|
||||
# wants the gated-delta-net speculative fix (vllm-project/vllm#51812 / #51674), which no
|
||||
# released tag carries yet.
|
||||
min_vllm_version: "0.17.0"
|
||||
default_frontend: rust
|
||||
# NVIDIA only. The rocm tag with the same name is an AMD build of the 2.4T *text* model
|
||||
# (no `ai.vllm.build.commit` label) and has never been exercised on the 27B; naming it
|
||||
# here would imply a tested AMD path that does not exist. Non-NVIDIA hardware falls back
|
||||
# to the generic nightly image.
|
||||
docker_image: "vllm/vllm-openai:qwen38"
|
||||
architecture: dense
|
||||
parameter_count: "27B"
|
||||
active_parameters: "27B"
|
||||
# 262144 is what config.json (text_config.max_position_embeddings) and
|
||||
# tokenizer_config.json both declare. 1M is reachable — see Processing Ultra-Long Texts.
|
||||
context_length: 262144
|
||||
# No --trust-remote-code: config.json carries no `auto_map`, so there is no remote code
|
||||
# to trust — vLLM's own handler loads this architecture.
|
||||
base_args: []
|
||||
base_env: {}
|
||||
|
||||
# vLLM registers its OWN Qwen3_5Config into AutoConfig, so config parsing does not depend
|
||||
# on the transformers version at all — and vLLM already floors transformers at 5.5.3 in
|
||||
# requirements/common.txt. This pin covers the Qwen3-VL *processor* classes this checkpoint
|
||||
# needs, matching the 5.8.0 its config.json was written by. Do not restate it as "older
|
||||
# releases do not know this model_type": transformers has shipped a qwen3_5 module since
|
||||
# 5.2.0, so that claim is false.
|
||||
dependencies:
|
||||
- note: "Matches the transformers version config.json was written by (5.8.0), for the Qwen3-VL processor classes. vLLM parses the config with its own Qwen3_5Config."
|
||||
command: 'uv pip install -U "transformers>=5.8.0"'
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice. The default parser is qwen3_coder."
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "qwen3_coder"
|
||||
reasoning:
|
||||
description: "Parse <think> blocks separately from the final answer. The template opens every assistant turn with <think>, so without this the reasoning lands in message.content."
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "qwen3"
|
||||
spec_decoding:
|
||||
description: "Speculative decoding — pick MTP (in-checkpoint), DFlash2, or DSpark (external drafts)."
|
||||
default_mode: mtp
|
||||
modes:
|
||||
mtp:
|
||||
label: "MTP"
|
||||
# text_config sets mtp_num_hidden_layers: 1 and the weight index carries `mtp.*`
|
||||
# tensors, so the draft head is present. Depth 3 matches the 2.4T sibling. Measured
|
||||
# acceptance on short prompts: 92.2% (BF16), 84.8% (FP8), 81.5-84.8% (NVFP4).
|
||||
# num_speculative_tokens is spelled out deliberately. vLLM reads `mtp_num_hidden_layers`
|
||||
# off the TOP-level config (config/speculative.py), but this checkpoint carries it only
|
||||
# under `text_config`, so the depth cannot be inferred here — omit it and startup fails.
|
||||
description: "Native MTP head, 3 draft tokens."
|
||||
variants: [default, fp8, nvfp4, nvfp4_nvidia, int4, mxfp4]
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"mtp","num_speculative_tokens":3}'
|
||||
hardware_overrides:
|
||||
dgx_spark_gb10:
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"mtp","num_speculative_tokens":5}'
|
||||
rtx_pro_6000:
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"mtp","num_speculative_tokens":5}'
|
||||
qwen3_5_mtp:
|
||||
label: "MTP (Ascend)"
|
||||
# vLLM Ascend documents `qwen3_5_mtp` for this head (it aliases to `mtp` and logs a
|
||||
# deprecation warning). `enforce_eager` applies to the DRAFT model only: the target
|
||||
# model still replays its FULL_DECODE_ONLY graph.
|
||||
description: "Same in-checkpoint MTP head, spelled as vLLM Ascend documents it, with the draft head run eager."
|
||||
variants: [ascend_w8a8, fp8]
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"qwen3_5_mtp","num_speculative_tokens":3,"enforce_eager":true}'
|
||||
dflash:
|
||||
label: "DFlash2"
|
||||
description: "External DFlash2 drafter (incoai/Qwen3.8-27B-DFlash2). method is dflash; DFlash2 is selected by the draft architecture. Needs vLLM >=0.28.0, which shipped vllm-project/vllm#52816. num_speculative_tokens must be 7 (block_size 8 minus one)."
|
||||
variants: [default]
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"dflash","model":"incoai/Qwen3.8-27B-DFlash2","num_speculative_tokens":7}'
|
||||
dspark:
|
||||
label: "DSpark"
|
||||
description: "External DSpark drafter (RedHatAI/Qwen3.8-27B-speculator.dspark), with a Markov head and confidence head. The model card specifies 8 draft tokens."
|
||||
variants: [default]
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"dspark","model":"RedHatAI/Qwen3.8-27B-speculator.dspark","num_speculative_tokens":8}'
|
||||
text_only:
|
||||
description: "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
|
||||
args:
|
||||
- "--language-model-only"
|
||||
encoder_parallel:
|
||||
description: "Run the vision encoder in data-parallel mode — avoids TP comm overhead on the small encoder. Mutually exclusive with text_only."
|
||||
args:
|
||||
- "--mm-encoder-tp-mode"
|
||||
- "data"
|
||||
long_context:
|
||||
description: "YaRN RoPE scaling past the native 262,144-token window to 1,000,000 tokens. Static YaRN holds the scaling factor constant regardless of input length, so leave this off unless you actually serve long prompts; for ~524K, halve the factor to 2.0."
|
||||
args:
|
||||
- "--hf-overrides"
|
||||
- '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}'
|
||||
- "--max-model-len"
|
||||
- "1000000"
|
||||
env:
|
||||
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
|
||||
|
||||
opt_in_features:
|
||||
- spec_decoding
|
||||
- text_only
|
||||
- long_context
|
||||
|
||||
# vram_minimum_gb is ceil(real safetensors bytes in GB x 1.2) for every variant, not
|
||||
# params x bytes_per_param: the quantized builds are not uniformly 4-bit.
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
# 55,563,006,776 B on disk = 55.6 GB
|
||||
vram_minimum_gb: 67
|
||||
description: "Full-precision BF16 — 51.7 GiB of weights: 1 GPU."
|
||||
hardware_overrides:
|
||||
rtx_pro_6000:
|
||||
extra_args:
|
||||
- "--max-num-seqs"
|
||||
- "8"
|
||||
dgx_spark_gb10:
|
||||
extra_args:
|
||||
- "--max-num-seqs"
|
||||
- "8"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.8"
|
||||
fp8:
|
||||
model_id: "Qwen/Qwen3.8-27B-FP8"
|
||||
precision: fp8
|
||||
# 30,866,866,928 B = 30.9 GB. Block-scaled FP8
|
||||
# (quantization_config.weight_block_size [128, 128]).
|
||||
vram_minimum_gb: 38
|
||||
description: "Official block-scaled FP8 checkpoint — 28.7 GiB: 1 GPU."
|
||||
# ascend_950pr added 2026-08-26: vllm-project/vllm-ascend#14852 dispatches
|
||||
# `quant_method: fp8` + `weight_block_size` in AscendFp8Config.get_quant_method and
|
||||
# re-quantizes the block-scaled tiles to MXFP8 at load time on 950. No
|
||||
# `--quantization ascend` — the scheme is picked up from the checkpoint config.
|
||||
supported_hardware: [h100, h200, b200, b300, gb200, gb300, mi300x, mi325x, mi355x, rtx_5090, rtx_5090_2x, ascend_950pr]
|
||||
hardware_overrides:
|
||||
ascend_950pr:
|
||||
docker_image: "quay.io/ascend/vllm-ascend:nightly-main-a5"
|
||||
nvfp4:
|
||||
model_id: "Inferact/Qwen3.8-27B-NVFP4"
|
||||
precision: nvfp4
|
||||
# Without a label the pill renders precision.toUpperCase() = "NVFP4", so a bare "NVFP4"
|
||||
# label would be a no-op. (W4A4) is the fact worth surfacing: activations are 4-bit too.
|
||||
label: "NVFP4 (W4A4)"
|
||||
# 26,381,249,312 B = 26.4 GB. quant_method `modelopt`, quant_algo `NVFP4`.
|
||||
vram_minimum_gb: 32
|
||||
supported_hardware: [b200, b300, gb200, gb300, rtx_5090, rtx_5090_2x]
|
||||
description: "NVFP4 build for NVIDIA Blackwell — 24.6 GiB, the smallest footprint: 1 GPU."
|
||||
# The verified 1x RTX 5090 launch from the guide: CUDA graph capture OOMs on 31.4 GiB
|
||||
# usable, so --enforce-eager is required, and the KV pool holds a 32K context.
|
||||
hardware_overrides:
|
||||
rtx_5090:
|
||||
extra_args:
|
||||
- "--max-model-len"
|
||||
- "32768"
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
- "--enforce-eager"
|
||||
nvfp4_nvidia:
|
||||
model_id: "nvidia/Qwen3.8-27B-NVFP4"
|
||||
precision: nvfp4
|
||||
# Without a label the pill renders precision.toUpperCase() = "NVFP4", so a bare "NVFP4"
|
||||
# label would be a no-op.
|
||||
label: "NVFP4 (NVIDIA)"
|
||||
# 21,904,998,400 B = 21.9 GB. quant_method `modelopt`, quant_algo `NVFP4-FP8 mixed-precision`.
|
||||
vram_minimum_gb: 32
|
||||
supported_hardware: [rtx_pro_6000, dgx_spark_gb10]
|
||||
description: "NVFP4 build for NVIDIA Blackwell — 21.9 GiB, the smallest footprint: 1 GPU."
|
||||
hardware_overrides:
|
||||
dgx_spark_gb10:
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.7"
|
||||
- "--max-model-len"
|
||||
- "262144"
|
||||
- "--max-num-seqs"
|
||||
- "8"
|
||||
- "--max-num-batched-tokens"
|
||||
- "8192"
|
||||
- "--enable-chunked-prefill"
|
||||
- "--async-scheduling"
|
||||
- "--enable-prefix-caching"
|
||||
- "--load-format"
|
||||
- "fastsafetensors"
|
||||
rtx_pro_6000:
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
- "--max-model-len"
|
||||
- "262144"
|
||||
- "--max-num-seqs"
|
||||
- "8"
|
||||
- "--max-num-batched-tokens"
|
||||
- "8192"
|
||||
- "--enable-chunked-prefill"
|
||||
- "--async-scheduling"
|
||||
- "--enable-prefix-caching"
|
||||
int4:
|
||||
model_id: "RedHatAI/Qwen3.8-27B-INT4"
|
||||
precision: int4
|
||||
label: "INT4 (W4A16)"
|
||||
vram_minimum_gb: 24
|
||||
supported_hardware: [h100, h200, b200, b300, gb200, gb300, rtx_5090, rtx_5090_2x]
|
||||
description: "Red Hat AI compressed-tensors W4A16 INT4 build — 19.5 GB: 1 GPU, runs on Hopper too."
|
||||
mxfp4:
|
||||
model_id: "RedHatAI/Qwen3.8-27B-MXFP4"
|
||||
precision: mxfp4
|
||||
label: "MXFP4 (W4A4)"
|
||||
# 19.83 GB on disk (19,832,841,464 B). Transformer linear weights+activations are FP4
|
||||
# (OCP Microscaling, group-32, E8M0 scales); the vision tower, embeddings and the
|
||||
# linear-attention in_proj_a/b stay BF16.
|
||||
vram_minimum_gb: 24
|
||||
supported_hardware: [h100, h200, b200, b300, gb200, gb300, rtx_5090, rtx_5090_2x]
|
||||
description: "Red Hat AI compressed-tensors MXFP4 build — 19.8 GB: 1 GPU. True W4A4 on Blackwell (SM100+); W4A16 via Marlin on Hopper and elsewhere."
|
||||
|
||||
ascend_w8a8:
|
||||
model_id: "Eco-Tech/Qwen3.8-27B-w8a8"
|
||||
precision: int8
|
||||
label: "W8A8 (Ascend)"
|
||||
# 32,128,768,176 B across 10 shards = 32.1 GB. ModelSlim keeps the vision tower in BF16
|
||||
# and stores FP32 per-channel scales, so this is not a uniform 8-bit build.
|
||||
vram_minimum_gb: 39
|
||||
supported_hardware: [ascend_950pr]
|
||||
default_modes:
|
||||
spec_decoding: qwen3_5_mtp
|
||||
# `--quantization ascend` is what selects the ModelSlim scheme; without it the loader
|
||||
# treats the checkpoint as float. ACLGraph replays decode only. CPU binding cuts
|
||||
# cross-core scheduling overhead on the Kunpeng host.
|
||||
extra_args:
|
||||
- "--quantization"
|
||||
- "ascend"
|
||||
- "--compilation-config"
|
||||
- '{"cudagraph_mode":"FULL_DECODE_ONLY"}'
|
||||
- "--additional-config"
|
||||
- '{"enable_cpu_binding":true}'
|
||||
- "--max-num-batched-tokens"
|
||||
- "16384"
|
||||
extra_env:
|
||||
PYTORCH_NPU_ALLOC_CONF: "expandable_segments:True"
|
||||
HCCL_BUFFSIZE: "512"
|
||||
OMP_PROC_BIND: "false"
|
||||
OMP_NUM_THREADS: "1"
|
||||
# `computeDockerMeta` has no Huawei brand, so a bare string `docker_image` would only pin
|
||||
# NVIDIA. The exact variant+hardware override lands regardless of brand.
|
||||
hardware_overrides:
|
||||
ascend_950pr:
|
||||
docker_image: "quay.io/ascend/vllm-ascend:qwen3.8-a5"
|
||||
description: "ModelSlim INT8 W8A8 for Ascend NPU — 32.1 GB: 1 NPU. Requires --quantization ascend."
|
||||
|
||||
# Dense model: no routed experts, so none of the expert-parallel or PD strategies apply.
|
||||
#
|
||||
# `multi_node_tp` is deliberately ABSENT, unlike the 2.4T sibling. Multi-node TP sets
|
||||
# TP = gpus_per_node x nodes, i.e. 16 on the smallest 2-node layout — and 24 attention heads
|
||||
# are not divisible by 16, so vLLM raises in VllmConfig.__post_init__ before a single weight
|
||||
# loads. There is no schema mechanism to gate a strategy by node count, so offering it at
|
||||
# all would ship a command that cannot start. It costs nothing: 51.7 GiB of BF16 weights fit
|
||||
# one GPU, so nothing here needs a second node.
|
||||
kv_offload_support:
|
||||
offloading_cpu: verified
|
||||
offloading_fs: verified
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
|
||||
hardware_overrides:
|
||||
xpu:
|
||||
extra_args:
|
||||
- "--enforce-eager"
|
||||
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
Qwen3.8-27B is the 27-billion-parameter dense member of the Qwen3.8 family, on the same
|
||||
hybrid-attention backbone as the 2.4T MoE flagship.
|
||||
|
||||
**The layer mix is the interesting part.** Only 16 of the 64 layers run full attention
|
||||
(`full_attention_interval: 4`); the other 48 run linear attention with a constant
|
||||
recurrent state. Unlike the 2.4T this is a multimodal model: the architecture is
|
||||
`Qwen3_5ForConditionalGeneration` and `config.json` carries a `vision_config`. Text
|
||||
serving is what this recipe covers and what has been verified. Quantized builds (NVFP4, FP8,
|
||||
INT4, and Red Hat AI's MXFP4) are offered as variants alongside the base checkpoint.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **transformers >= 5.8.0**, matching the version `config.json` was written by. vLLM parses
|
||||
the config with its own `Qwen3_5Config`, so this is really about the Qwen3-VL processor.
|
||||
|
||||
## Launch commands
|
||||
|
||||
### Low latency
|
||||
|
||||
NVFP4, TP1:
|
||||
|
||||
```bash
|
||||
vllm serve Inferact/Qwen3.8-27B-NVFP4 \
|
||||
--tensor-parallel-size 1 \
|
||||
--max-model-len 262144 \
|
||||
--kv-cache-dtype fp8 \
|
||||
--reasoning-parser qwen3 \
|
||||
--enable-auto-tool-choice --tool-call-parser qwen3_coder
|
||||
```
|
||||
|
||||
FP8, TP4 (one GB300 tray) for the largest KV cache:
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3.8-27B-FP8 \
|
||||
--tensor-parallel-size 4 \
|
||||
--max-model-len 262144 \
|
||||
--kv-cache-dtype fp8 \
|
||||
--reasoning-parser qwen3
|
||||
```
|
||||
|
||||
Add `--speculative-config '{"method":"mtp","num_speculative_tokens":3}'` for MTP.
|
||||
|
||||
### MXFP4 (W4A4) — Blackwell
|
||||
|
||||
```bash
|
||||
vllm serve RedHatAI/Qwen3.8-27B-MXFP4 \
|
||||
--tensor-parallel-size 1 \
|
||||
--reasoning-parser qwen3 \
|
||||
--enable-auto-tool-choice --tool-call-parser qwen3_coder
|
||||
```
|
||||
|
||||
True W4A4 on Blackwell. The in-checkpoint MTP head, long-context YaRN and tool calling all
|
||||
carry over from the base recipe.
|
||||
|
||||
### 2x RTX 5090 (consumer Blackwell, sm120)
|
||||
|
||||
Verified on vLLM `0.26.1rc1.dev608+g99a10304d`, TP2 across two cards.
|
||||
|
||||
**NVFP4 uses the real kernel here.** vLLM selects
|
||||
`FlashInferCutlassNvFp4LinearKernel for NVFP4 GEMM` on sm120 — a cutlass path, not an
|
||||
emulation fallback — for both the `Inferact` build above and `unsloth/Qwen3.8-27B-NVFP4`.
|
||||
|
||||
**The block-scaled FP8 checkpoint needs no workaround.** vLLM auto-disables DeepGemm for
|
||||
`model_type=qwen3_5_text` on Blackwell and falls back to CUTLASS, so it loads unaided.
|
||||
Verified by running *without* `VLLM_USE_DEEP_GEMM=0`: identical 377,456-token KV pool.
|
||||
|
||||
```bash
|
||||
vllm serve unsloth/Qwen3.8-27B-NVFP4 \
|
||||
--tensor-parallel-size 2 \
|
||||
--max-model-len 262144 \
|
||||
--kv-cache-dtype fp8 \
|
||||
--reasoning-parser qwen3 \
|
||||
--enable-auto-tool-choice --tool-call-parser qwen3_coder
|
||||
```
|
||||
|
||||
At 262,144 context, GPU KV cache size as reported at startup:
|
||||
|
||||
| precision | KV tokens | weights/GPU | MTP acceptance |
|
||||
|---|---|---|---|
|
||||
| FP8 | 377,456 | 14.28 GiB | 0.771 |
|
||||
| NVFP4 (Inferact) | 445,875 | 12.02 GiB | 0.897 |
|
||||
| NVFP4 (unsloth) | 920,517 | 10.64 GiB | 0.788 |
|
||||
|
||||
The two NVFP4 builds are not interchangeable on 32 GB cards: `unsloth` is
|
||||
mixed-precision (FP8 channel-wise alongside the 4-bit groups) and leaves room for roughly
|
||||
twice the KV cache, while the uniform-W4A4 `Inferact` build drafts better. Both serve
|
||||
correctly.
|
||||
|
||||
The in-checkpoint MTP head works in every precision above. Acceptance is measured from
|
||||
`vllm:spec_decode_num_{accepted,draft}_tokens_total`, since throughput alone cannot
|
||||
distinguish a working drafter from one that loaded and was ignored.
|
||||
|
||||
### 1x RTX 5090 — NVFP4 needs `--enforce-eager`
|
||||
|
||||
One card has **31.4 GiB usable**, not 32, and NVFP4 fits only with CUDA graphs off:
|
||||
|
||||
```bash
|
||||
vllm serve Inferact/Qwen3.8-27B-NVFP4 \
|
||||
--tensor-parallel-size 1 \
|
||||
--max-model-len 32768 \
|
||||
--kv-cache-dtype fp8 \
|
||||
--enforce-eager \
|
||||
--reasoning-parser qwen3
|
||||
```
|
||||
|
||||
Without `--enforce-eager`, startup dies in CUDA graph capture with
|
||||
`torch.OutOfMemoryError: Tried to allocate 784.00 MiB`. **Raising or lowering
|
||||
`--gpu-memory-utilization` does not help** — 0.80 and 0.93 both leave the same 47.06 MiB
|
||||
free, because that budget covers weights and KV while graph capture allocates outside it.
|
||||
|
||||
KV pool at 32K context: 91,022 tokens with `--enforce-eager` alone; 135,926 adding
|
||||
`--language-model-only`; 152,917 also capping `--max-num-seqs 8`; 76,458 with bf16 KV
|
||||
instead of fp8 — so fp8 KV is a choice here, not a requirement. The MTP head still fits
|
||||
(90,112 tokens, 0.754 acceptance). Those two flags are levers for a bigger pool, not fixes
|
||||
for the OOM: only `--enforce-eager` decides whether the server starts.
|
||||
|
||||
### Intel XPU B60 / B70
|
||||
|
||||
Validated on 4x Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the
|
||||
official vLLM XPU image `vllm/vllm-openai-xpu:latest`. `--enforce-eager` is required on this
|
||||
backend. Select Arc Pro B60/B70 in the hardware picker above for the exact command.
|
||||
|
||||
### Huawei Ascend 950PR-w8a8 (vLLM Ascend)
|
||||
|
||||
Verified on a single 950PR, TP1, vLLM Ascend 0.23.0 from the
|
||||
`quay.io/ascend/vllm-ascend:qwen3.8-a5` image.
|
||||
|
||||
The ModelSlim INT8 build is the checkpoint to use here. `--quantization ascend` is what
|
||||
selects the ModelSlim scheme — without it the loader treats the weights as float. The MTP
|
||||
head inside the checkpoint works and is spelled `qwen3_5_mtp` on this backend.
|
||||
|
||||
Both the weight and the image come from ModelScope, not HuggingFace: set
|
||||
`VLLM_USE_MODELSCOPE=True` or pass a local path.
|
||||
|
||||
```bash
|
||||
export VLLM_USE_MODELSCOPE=True
|
||||
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
|
||||
export HCCL_BUFFSIZE=512
|
||||
export OMP_PROC_BIND=false
|
||||
export OMP_NUM_THREADS=1
|
||||
|
||||
vllm serve Eco-Tech/Qwen3.8-27B-w8a8 \
|
||||
--tensor-parallel-size 1 \
|
||||
--quantization ascend \
|
||||
--max-num-batched-tokens 16384 \
|
||||
--enable-prefix-caching \
|
||||
--reasoning-parser qwen3 \
|
||||
--tool-call-parser qwen3_coder \
|
||||
--mm-encoder-tp-mode data \
|
||||
--speculative-config '{"method":"qwen3_5_mtp","num_speculative_tokens":3,"enforce_eager":true}' \
|
||||
--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \
|
||||
--additional-config '{"enable_cpu_binding":true}'
|
||||
```
|
||||
|
||||
`--reasoning-parser qwen3` is not optional in practice: the chat template opens every
|
||||
assistant turn with `<think>`, so without it the entire reasoning block lands in
|
||||
`message.content` and a 2048-token budget can be spent before the answer starts.
|
||||
|
||||
Measured single-stream decode: 64 tok/s (2048-token completion, one request, MTP on).
|
||||
Read MTP acceptance from `vllm:spec_decode_num_{accepted,draft}_tokens_total` before
|
||||
tuning `num_speculative_tokens` — throughput alone cannot tell a working drafter from
|
||||
one that loaded and was ignored.
|
||||
|
||||
### Huawei Ascend 950PR-Native block-scaled FP8 (vLLM Ascend)
|
||||
|
||||
Verified on a single 950PR, TP1, vLLM Ascend 0.23.0 from the
|
||||
`quay.io/ascend/vllm-ascend:nightly-main-a5` image.
|
||||
|
||||
The official `Qwen/Qwen3.8-27B-FP8` checkpoint serves directly on a 950PR — no offline
|
||||
re-quantization and **no `--quantization ascend`**. vLLM Ascend reads `quant_method: fp8`
|
||||
and `weight_block_size` straight from the checkpoint, expands the `weight_scale_inv`
|
||||
tiles at load time and re-quantizes to MXFP8 on 950.
|
||||
|
||||
Requires a vLLM Ascend build that includes
|
||||
[vllm-ascend#14852](https://github.com/vllm-project/vllm-ascend/pull/14852) (merged
|
||||
2026-08-26); no tagged release carries it yet. The W8A8 path above is unaffected.
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3.8-27B-FP8 \
|
||||
--tensor-parallel-size 1 \
|
||||
--max-model-len 128000 \
|
||||
--max-num-seqs 16 \
|
||||
--gpu-memory-utilization 0.85 \
|
||||
--trust-remote-code \
|
||||
--enable-prefix-caching \
|
||||
--reasoning-parser qwen3 \
|
||||
--tool-call-parser qwen3_coder \
|
||||
--mm-encoder-tp-mode data \
|
||||
--speculative-config '{"method": "qwen3_5_mtp", "num_speculative_tokens": 3, "enforce_eager": true}'
|
||||
```
|
||||
|
||||
### DFlash2 (opt-in)
|
||||
|
||||
DFlash2 is not a standalone model. Serve `Qwen/Qwen3.8-27B` and attach
|
||||
[incoai/Qwen3.8-27B-DFlash2](https://huggingface.co/incoai/Qwen3.8-27B-DFlash2)
|
||||
as the drafter. Requires vLLM >=0.28.0, which shipped
|
||||
[vllm#52816](https://github.com/vllm-project/vllm/pull/52816).
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3.8-27B \
|
||||
--speculative-config '{"method":"dflash","model":"incoai/Qwen3.8-27B-DFlash2","num_speculative_tokens":7}'
|
||||
```
|
||||
|
||||
### DSpark (opt-in)
|
||||
|
||||
DSpark is an external confidence-scheduled drafter. Serve `Qwen/Qwen3.8-27B` and attach
|
||||
[RedHatAI/Qwen3.8-27B-speculator.dspark](https://huggingface.co/RedHatAI/Qwen3.8-27B-speculator.dspark)
|
||||
with eight speculative tokens:
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3.8-27B \
|
||||
--speculative-config '{"method":"dspark","model":"RedHatAI/Qwen3.8-27B-speculator.dspark","num_speculative_tokens":8}'
|
||||
```
|
||||
|
||||
## Client usage
|
||||
|
||||
`generation_config.json` ships `temperature: 1.0`, `top_p: 0.95`, `top_k: 20`.
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600)
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="Qwen/Qwen3.8-27B",
|
||||
messages=[{"role": "user", "content": "Give me three primes above 100."}],
|
||||
temperature=1.0, top_p=0.95, max_tokens=2048,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
### Thinking modes
|
||||
|
||||
The model supports no-think and adaptive thinking through `chat_template_kwargs`, per
|
||||
request or server-wide via `--default-chat-template-kwargs`:
|
||||
|
||||
- `{"enable_thinking": false}` — no thinking, the model answers directly.
|
||||
- `{"reasoning_effort": "low"}` — adaptive thinking. `xhigh` (default), `medium`, `low`.
|
||||
|
||||
## Processing Ultra-Long Texts
|
||||
|
||||
The model is native at 262k. To extend it to 1M, turn on the **Long Context**
|
||||
feature above: it applies the model card's YaRN RoPE configuration via
|
||||
`--hf-overrides` and raises `--max-model-len` to 1,000,000. Static YaRN holds
|
||||
the scaling factor constant regardless of input length, so leave it off unless
|
||||
you actually serve long prompts; for a ~524K workload, halve the card's
|
||||
`factor` to `2.0`.
|
||||
|
||||
Note the override is nested under `text_config` here, where the 2.4T takes it flat.
|
||||
|
||||
See the [model card](https://huggingface.co/Qwen/Qwen3.8-27B#processing-ultra-long-texts) for the full parameter reference.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**Two different "MXFP4" paths.** The *native* `Mxfp4Config` path (gpt-oss-style weight-only
|
||||
checkpoints) has no linear-layer method and falls back to `UnquantizedLinearMethod`, so it does
|
||||
not run. This recipe's **RedHatAI MXFP4** variant instead uses the *compressed-tensors*
|
||||
`mxfp4-pack-quantized` format, which maps to `CompressedTensorsW4A4Mxfp4` — a real W4A4 method
|
||||
on Blackwell (SM100+) and W4A16-via-Marlin elsewhere. That path loads and serves correctly on
|
||||
Nvidia; pick the **MXFP4** variant (or NVFP4) when you want the 4-bit build.
|
||||
|
||||
## References
|
||||
|
||||
- Model card: https://huggingface.co/Qwen/Qwen3.8-27B
|
||||
- FP8 checkpoint: https://huggingface.co/Qwen/Qwen3.8-27B-FP8
|
||||
- NVFP4 build (NVIDIA): https://huggingface.co/nvidia/Qwen3.8-27B-NVFP4
|
||||
- NVFP4 build (Inferact): https://huggingface.co/Inferact/Qwen3.8-27B-NVFP4
|
||||
- MXFP4 build (RedHatAI): https://huggingface.co/RedHatAI/Qwen3.8-27B-MXFP4
|
||||
- Ascend INT8 checkpoint (ModelScope): https://www.modelscope.cn/models/Eco-Tech/Qwen3.8-27B-w8a8
|
||||
- vLLM documentation: https://docs.vllm.ai/
|
||||
- vLLM Ascend tutorial: https://docs.vllm.ai/projects/ascend/en/latest/tutorials/models/Qwen3.8-27B.html
|
||||
- DFlash2 draft: https://huggingface.co/incoai/Qwen3.8-27B-DFlash2
|
||||
- DSpark draft: https://huggingface.co/RedHatAI/Qwen3.8-27B-speculator.dspark
|
||||
- DFlash2 blog: https://inco.ai/blog/dflash2/
|
||||
- vLLM DFlash2 PR: https://github.com/vllm-project/vllm/pull/52816
|
||||
- Native FP8 support on Ascend: https://github.com/vllm-project/vllm-ascend/pull/14852
|
||||
@@ -0,0 +1,430 @@
|
||||
meta:
|
||||
title: "Qwen3.8-Flash-Next"
|
||||
slug: "qwen3.8-flash-next"
|
||||
provider: "Qwen"
|
||||
description: "Qwen4 architecture preview with a 125B-parameter main model, supplemented by an additional 51B N-gram embeddings, with 6B parameters activated per token."
|
||||
date_updated: 2026-10-13
|
||||
tasks:
|
||||
- multimodal
|
||||
- text
|
||||
performance_headline: "Qwen4 architecture preview with 6B active parameters and efficient 262K context"
|
||||
related_recipes:
|
||||
- "Qwen/Qwen3-Next-80B-A3B-Instruct"
|
||||
hardware:
|
||||
h200: verified
|
||||
h100: verified
|
||||
gb200: verified
|
||||
gb300: verified
|
||||
mi355x: verified
|
||||
rtx_pro_6000_4x: verified
|
||||
dgx_station_gb300: verified
|
||||
|
||||
model:
|
||||
model_id: "Qwen/Qwen3.8-Flash-Next"
|
||||
min_vllm_version: "0.30.0"
|
||||
default_frontend: rust
|
||||
docker_image:
|
||||
nvidia: "vllm/vllm-openai:nightly"
|
||||
amd: "vllm/vllm-openai-rocm:nightly"
|
||||
nightly_required: true
|
||||
install:
|
||||
pip:
|
||||
note: "Nightly wheels are recommended and verified by this recipe; stable vLLM ≥0.30.0 also runs the model."
|
||||
docker:
|
||||
note: "Use the vLLM nightly image."
|
||||
architecture: moe
|
||||
parameter_count: "176B"
|
||||
active_parameters: "6B"
|
||||
context_length: 262144
|
||||
base_args:
|
||||
- "--max-num-seqs"
|
||||
- "256"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.90"
|
||||
- "--enable-prefix-caching"
|
||||
base_env: {}
|
||||
|
||||
dependencies: []
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable automatic tool choice with the Qwen3 Coder parser"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "qwen3_coder"
|
||||
reasoning:
|
||||
description: "Extract thinking output with the Qwen3 reasoning parser"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "qwen3"
|
||||
spec_decoding:
|
||||
description: "Use the checkpoint's Multi-Token Prediction module for faster decoding"
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"mtp","num_speculative_tokens":3}'
|
||||
text_only:
|
||||
description: "Skip the vision encoder for text-only workloads and leave more memory for KV cache"
|
||||
args:
|
||||
- "--language-model-only"
|
||||
long_context:
|
||||
description: "YaRN RoPE scaling past the native 262,144-token window to 1,000,000 tokens. Static YaRN holds the scaling factor constant regardless of input length, so leave this off unless you actually serve long prompts; for ~524K, halve the factor to 2.0."
|
||||
args:
|
||||
- "--hf-overrides"
|
||||
- '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}'
|
||||
- "--max-model-len"
|
||||
- "1000000"
|
||||
env:
|
||||
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
|
||||
|
||||
opt_in_features:
|
||||
- text_only
|
||||
- long_context
|
||||
|
||||
hardware_opt_in_features:
|
||||
rtx_pro_6000_4x:
|
||||
- spec_decoding
|
||||
|
||||
variants:
|
||||
default:
|
||||
model_id: "Inferact/Qwen3.8-Flash-Next-NVFP4"
|
||||
precision: nvfp4
|
||||
vram_minimum_gb: 130
|
||||
description: "Inferact quantized NVFP4 weights with per-layer embeddings in nvfp4 too"
|
||||
hardware_overrides:
|
||||
dgx_station_gb300:
|
||||
extra_args:
|
||||
- "--max-num-seqs"
|
||||
- "32"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.95"
|
||||
- "--max-num-batched-tokens"
|
||||
- "8192"
|
||||
- "--enable-prefix-caching"
|
||||
- "--distributed-executor-backend"
|
||||
- "mp"
|
||||
- "-cc.mode"
|
||||
- "none"
|
||||
- "-cc.cudagraph_mode"
|
||||
- "full_decode_only"
|
||||
rtx_pro_6000_4x:
|
||||
extra_args:
|
||||
- "--max-num-seqs"
|
||||
- "16"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.95"
|
||||
- "--max-num-batched-tokens"
|
||||
- "8192"
|
||||
- "--enable-prefix-caching"
|
||||
- "--moe-backend"
|
||||
- "marlin"
|
||||
fp8:
|
||||
model_id: "Qwen/Qwen3.8-Flash-Next-FP8"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 250
|
||||
description: "Official dynamic FP8 checkpoint with blockwise weight quantization"
|
||||
hardware_overrides:
|
||||
dgx_station_gb300:
|
||||
extra_args:
|
||||
- "--max-num-seqs"
|
||||
- "32"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.95"
|
||||
- "--max-num-batched-tokens"
|
||||
- "8192"
|
||||
- "--enable-prefix-caching"
|
||||
- "--distributed-executor-backend"
|
||||
- "mp"
|
||||
- "-cc.mode"
|
||||
- "none"
|
||||
- "-cc.cudagraph_mode"
|
||||
- "full_decode_only"
|
||||
rtx_pro_6000_4x:
|
||||
extra_args:
|
||||
- "--max-num-seqs"
|
||||
- "16"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.95"
|
||||
- "--max-num-batched-tokens"
|
||||
- "8192"
|
||||
- "--enable-prefix-caching"
|
||||
bf16:
|
||||
model_id: "Qwen/Qwen3.8-Flash-Next"
|
||||
precision: bf16
|
||||
vram_minimum_gb: 423
|
||||
description: "Official BF16 checkpoint"
|
||||
|
||||
# TP+EP is the default deployment; it benchmarks consistently faster than
|
||||
# plain TP for this model, which stays offered as an alternative.
|
||||
default_strategy: single_node_tep
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- single_node_tep
|
||||
- single_node_dep
|
||||
|
||||
kv_offload_support:
|
||||
offloading_cpu: verified
|
||||
offloading_fs: verified
|
||||
|
||||
hardware_overrides:
|
||||
hopper:
|
||||
extra_args:
|
||||
- "--moe-backend"
|
||||
- "triton"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.85"
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
- "--attention-config.indexer_kv_dtype"
|
||||
- "fp8"
|
||||
extra_env: {}
|
||||
blackwell:
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
- "--attention-config.indexer_kv_dtype"
|
||||
- "fp8"
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
tp: 4
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[Qwen3.8-Flash-Next](https://huggingface.co/Qwen/Qwen3.8-Flash-Next) is a
|
||||
multimodal, ultra-sparse Mixture-of-Experts model. It has 125B parameters,
|
||||
including an additional 51B N-gram embedding table, while activating 6B
|
||||
parameters per token.
|
||||
|
||||
The architecture combines four main ideas:
|
||||
|
||||
- **GDN + QSA:** three of every four layers use Gated DeltaNet to compress history;
|
||||
the fourth uses Qwen Sparse Attention for precise long-range retrieval.
|
||||
- **Gated Residual:** four residual branches dynamically control cross-layer reads
|
||||
and writes.
|
||||
- **N-gram Embedding:** a 51B lookup memory adds capacity with little per-token
|
||||
compute and can be asynchronously offloaded to host memory. (note that offload
|
||||
currently only runs on Nvidia devices)
|
||||
- **MTP:** the built-in Multi-Token Prediction module supports speculative decoding.
|
||||
|
||||
The checkpoint natively supports 262,144 tokens. Qwen reports that QSA reaches up
|
||||
to 10.2x prefill and 6.6x decode attention-kernel speedups at one million tokens.
|
||||
On GB300, TEP4 is the recommended full-tray configuration. On an 8x H200 node,
|
||||
use TEP8 for the official FP8 checkpoint; plain TP8 is incompatible with its
|
||||
128-wide quantization blocks.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Runtime:** use the nightly image, `vllm/vllm-openai:nightly` on NVIDIA or
|
||||
`vllm/vllm-openai-rocm:nightly` on AMD, or the nightly PyPI wheels. Stable
|
||||
≥0.30.0 runs the model, but this recipe is tuned and verified on nightly.
|
||||
- **Hardware:** NVIDIA CUDA and AMD ROCm instructions are provided below.
|
||||
- **FP8 memory:** the checkpoint is 172.78 GiB; it fits on a single GB300 GPU
|
||||
(288 GB HBM3e).
|
||||
- **BF16 memory:** the checkpoint is 335.28 GiB. The validated TP2 configuration
|
||||
used about 190 GiB per GB300 GPU.
|
||||
- **Host memory for N-gram Embedding offload:** at least 51 GB plus
|
||||
runtime headroom.
|
||||
|
||||
The Install block above generates the nightly-wheel and Docker launch flows.
|
||||
|
||||
## Serving
|
||||
|
||||
### Recommended FP8 configuration on 4x GB300
|
||||
|
||||
Use the official
|
||||
[`Qwen/Qwen3.8-Flash-Next-FP8`](https://huggingface.co/Qwen/Qwen3.8-Flash-Next-FP8)
|
||||
checkpoint tested by this recipe.
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3.8-Flash-Next-FP8 \
|
||||
--tensor-parallel-size 4 \
|
||||
--enable-expert-parallel \
|
||||
--gpu-memory-utilization 0.90 \
|
||||
--max-num-seqs 256 \
|
||||
--enable-prefix-caching \
|
||||
--kv-cache-dtype fp8 \
|
||||
--attention-config.indexer_kv_dtype fp8 \
|
||||
--speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser qwen3_coder \
|
||||
--reasoning-parser qwen3
|
||||
```
|
||||
|
||||
TEP4 is the recommended full-tray configuration — we recommend it over plain
|
||||
TP4 because expert parallelism is consistently faster for this model. TP4 and
|
||||
TEP4 are both validated full-tray configurations, including with MTP3.
|
||||
|
||||
To serve BF16, use the `Qwen/Qwen3.8-Flash-Next` checkpoint with the same
|
||||
arguments.
|
||||
|
||||
### Single-GPU deployment on GB200/GB300
|
||||
|
||||
The model fits on a single GPU: the FP8 checkpoint on one GB300 (288 GB
|
||||
HBM3e), and the NVFP4 variant on GB200 as well. Single-GPU TP1 suits
|
||||
evaluation and development; multi-GPU TEP remains the recommended deployment
|
||||
for serving throughput.
|
||||
|
||||
### Recommended FP8 configuration on 8x H200
|
||||
|
||||
Use TEP8 with the Triton MoE backend on Hopper:
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3.8-Flash-Next-FP8 \
|
||||
--tensor-parallel-size 8 \
|
||||
--enable-expert-parallel \
|
||||
--moe-backend triton \
|
||||
--gpu-memory-utilization 0.85 \
|
||||
--max-num-seqs 256 \
|
||||
--enable-prefix-caching \
|
||||
--kv-cache-dtype fp8 \
|
||||
--attention-config.indexer_kv_dtype fp8 \
|
||||
--speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser qwen3_coder \
|
||||
--reasoning-parser qwen3
|
||||
```
|
||||
|
||||
Plain TP8 is incompatible with the FP8 checkpoint; use TEP8.
|
||||
|
||||
### Recommended FP8 configuration on 8x H100
|
||||
|
||||
H100's 80GB/GPU is not enough headroom for the 51B N-gram/PLE embedding table
|
||||
— it relies on PLE CPU offload, which vLLM enables by default. Use TEP8 with
|
||||
the Triton MoE backend; as on other hardware, expert parallelism is faster
|
||||
than plain TP, and plain TP8 is incompatible with the FP8 checkpoint's
|
||||
128-wide quantization blocks.
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3.8-Flash-Next-FP8 \
|
||||
--tensor-parallel-size 8 \
|
||||
--enable-expert-parallel \
|
||||
--moe-backend triton \
|
||||
--gpu-memory-utilization 0.85 \
|
||||
--max-num-seqs 256 \
|
||||
--enable-prefix-caching \
|
||||
--kv-cache-dtype fp8 \
|
||||
--attention-config.indexer_kv_dtype fp8 \
|
||||
--speculative-config '{"method":"mtp","num_speculative_tokens":3}' \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser qwen3_coder \
|
||||
--reasoning-parser qwen3
|
||||
```
|
||||
|
||||
Benchmark MTP with SPEED-Bench (`--dataset-name speed_bench`), not `random`.
|
||||
|
||||
### FP8 configuration on a 4x MI355X GPU
|
||||
```bash
|
||||
export VLLM_ROCM_USE_AITER=1
|
||||
export VLLM_ROCM_USE_AITER_MOE=0
|
||||
|
||||
vllm serve Qwen/Qwen3.8-Flash-Next-FP8 \
|
||||
--tensor-parallel-size 4 \
|
||||
--max-model-len auto \
|
||||
--gpu-memory-utilization 0.9 \
|
||||
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
|
||||
```
|
||||
|
||||
### Full native context
|
||||
|
||||
The checkpoint advertises a native 262,144-token context, which vLLM uses when
|
||||
`--max-model-len` is omitted. Startup and bounded evaluation were validated with
|
||||
this limit configured, but a single 262K-token request was not tested.
|
||||
|
||||
### MTP speculative decoding
|
||||
|
||||
Add the following option:
|
||||
|
||||
```bash
|
||||
--speculative-config '{"method":"mtp","num_speculative_tokens":3}'
|
||||
```
|
||||
|
||||
### Offload the 51B N-gram embedding table
|
||||
|
||||
Builds containing N-gram Embedding offload can keep the N-gram lookup
|
||||
memory in host RAM and asynchronously prefetch the required rows. The initial
|
||||
implementation supports both ModelRunner V1 and V2. PLE CPU offload is required
|
||||
for DEP, and also required for TP/TEP on 80GB-class GPUs (H100) where the
|
||||
embedding table alone exceeds available headroom — optional only on GPUs with
|
||||
more VRAM per rank (H200 and above). vLLM enables it by default; set it
|
||||
explicitly with `--engram-config '{"cpu_offload":true}'`.
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3.8-Flash-Next-FP8 \
|
||||
--tensor-parallel-size 4 \
|
||||
--enable-expert-parallel \
|
||||
--max-model-len 262144 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser qwen3_coder \
|
||||
--reasoning-parser qwen3
|
||||
```
|
||||
|
||||
### Extend to one million tokens with YaRN
|
||||
|
||||
The checkpoint is native at 262K. For a one-million-token workload, turn on the
|
||||
**Long Context** feature above: it applies the model card's YaRN RoPE
|
||||
configuration under `text_config` (where this checkpoint carries
|
||||
`rope_parameters`) and raises `--max-model-len` to 1,000,000. Evaluate
|
||||
shorter-context quality before making it the default — static YaRN holds the
|
||||
scaling factor constant regardless of input length.
|
||||
|
||||
## Verifying
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
|
||||
response = client.chat.completions.create(
|
||||
model="Qwen/Qwen3.8-Flash-Next-FP8",
|
||||
messages=[{
|
||||
"role": "user",
|
||||
"content": "Explain how Gated DeltaNet and Qwen Sparse Attention complement each other.",
|
||||
}],
|
||||
max_tokens=8196,
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
A correct response should explain that GDN maintains a compact recurrent summary,
|
||||
while QSA selectively retrieves important regions from the full token history.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **FP8 on eight GPUs:** use TEP8; plain TP8 is incompatible with this
|
||||
checkpoint. TP4 is also supported if GPU memory capacity allows.
|
||||
- **Mamba-cache capacity error at startup:** keep `--max-num-seqs 256`.
|
||||
- **Runtime OOM on large multimodal batches:** keep `--gpu-memory-utilization 0.90`.
|
||||
- **Out of memory while loading:** enable N-gram Embedding CPU offload, increase
|
||||
TP size, or reduce `--max-model-len` to reserve less KV cache. On 4x H100
|
||||
80GB, plain TP4 OOMs at engine startup without PLE CPU offload — keep the
|
||||
default `--engram-config '{"cpu_offload":true}'`.
|
||||
- **XPU/TPU startup error:** these platforms are not supported by the initial
|
||||
implementation.
|
||||
- **Pipeline-parallel startup error:** N-gram Embedding does not initially support
|
||||
pipeline parallelism; use single-node TP or TEP instead.
|
||||
- **DEP startup failure:** DEP requires PLE CPU offload (on by default); DEP
|
||||
without it is not supported.
|
||||
- **MTP memory pressure:** reduce `num_speculative_tokens` below 3.
|
||||
- **PLE:** a network layer that injects N-gram Embeddings into the main model.
|
||||
|
||||
## KV cache offloading
|
||||
|
||||
Offloading is available upstream since vLLM v0.30.0 ([vllm-project/vllm#57145](https://github.com/vllm-project/vllm/pull/57145)).
|
||||
Validation used [Qwen/Qwen3.8-Flash-Next-FP8](https://huggingface.co/Qwen/Qwen3.8-Flash-Next-FP8) on 4×H200 (TP4),
|
||||
vLLM commit `d1b4028d7e`, the Python frontend, `--enforce-eager`,
|
||||
`--max-model-len 16384`, `--max-num-seqs 16` and `--max-num-batched-tokens 8192`.
|
||||
Qwen validation used the FP8 variant with `--limit-mm-per-prompt '{"image":0,"video":0}'`.
|
||||
Validation used `TieringOffloadingSpec`: 8 GiB total CPU tier, `blocks_per_chunk: 1`,
|
||||
a persistent filesystem `root_dir`, eight read/write threads and `PYTHONHASHSEED=0`.
|
||||
Three CPU-tier reloads and three filesystem reloads after restarting the server passed.
|
||||
Reloaded responses matched the cold output, including the reasoning field.
|
||||
The five-shot GSM8K diagnostic scored 31/32; this is a small sample, not the full benchmark.
|
||||
These results cover this text-only configuration, not the builder's larger tier/chunk defaults or other variants.
|
||||
|
||||
## References
|
||||
|
||||
- [Qwen3.8-Flash-Next model card](https://huggingface.co/Qwen/Qwen3.8-Flash-Next)
|
||||
- [Qwen3.8-Flash-Next FP8 checkpoint](https://huggingface.co/Qwen/Qwen3.8-Flash-Next-FP8)
|
||||
- [Qwen3.8-Flash-Next: A New Architecture, Towards Ultimate Cost-Efficiency](https://qwen.ai/blog?id=qwen3.8-flash-next)
|
||||
- [Qwen3-Next recipe](../Qwen3-Next-80B-A3B-Instruct)
|
||||
@@ -0,0 +1,137 @@
|
||||
meta:
|
||||
title: "Qwen3Guard-Gen-8B"
|
||||
slug: "qwen3guard-gen-8b"
|
||||
provider: "Qwen"
|
||||
description: "Lightweight text-only guardrail/safety classifier model in the Qwen3Guard family."
|
||||
date_added: 2025-11-28
|
||||
date_updated: 2026-04-30
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "Runs on a single GPU; serves safety classifications over OpenAI-compatible API"
|
||||
related_recipes: []
|
||||
hardware:
|
||||
mi300x: verified
|
||||
|
||||
model:
|
||||
model_id: "Qwen/Qwen3Guard-Gen-8B"
|
||||
min_vllm_version: "0.10.0"
|
||||
architecture: dense
|
||||
parameter_count: "8B"
|
||||
active_parameters: "8B"
|
||||
context_length: 32768
|
||||
base_args: []
|
||||
base_env: {}
|
||||
|
||||
features: {}
|
||||
|
||||
opt_in_features: []
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 19
|
||||
description: "Full precision BF16 — single GPU with >=20 GB VRAM"
|
||||
small_4b:
|
||||
model_id: "Qwen/Qwen3Guard-Gen-4B"
|
||||
precision: bf16
|
||||
vram_minimum_gb: 10
|
||||
description: "4B variant for more constrained deployments"
|
||||
tiny_0_6b:
|
||||
model_id: "Qwen/Qwen3Guard-Gen-0.6B"
|
||||
precision: bf16
|
||||
vram_minimum_gb: 4
|
||||
description: "0.6B variant for edge / ultra-low-cost serving"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
|
||||
kv_offload_support:
|
||||
offloading_cpu: verified
|
||||
offloading_fs: verified
|
||||
|
||||
hardware_overrides: {}
|
||||
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
**Qwen3Guard-Gen** is a lightweight text-only guardrail model. This guide describes how to run the 8B variant — as well as the 4B and 0.6B variants — on GPU using vLLM.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
### CUDA
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend auto
|
||||
```
|
||||
|
||||
### ROCm
|
||||
> Note: The vLLM wheel for ROCm requires Python 3.12, ROCm 7.0, and glibc >= 2.35.
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
|
||||
```
|
||||
|
||||
## Deployment Configurations
|
||||
|
||||
### Single GPU (CUDA)
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3Guard-Gen-8B \
|
||||
--host 0.0.0.0 \
|
||||
--max-model-len 32768
|
||||
```
|
||||
|
||||
### Single GPU (ROCm)
|
||||
```bash
|
||||
export VLLM_ROCM_USE_AITER=1
|
||||
vllm serve Qwen/Qwen3Guard-Gen-8B \
|
||||
--host 0.0.0.0 \
|
||||
--max-model-len 32768
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600)
|
||||
|
||||
messages = [{"role": "user", "content": "Tell me how to make a bomb."}]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model="Qwen/Qwen3Guard-Gen-8B",
|
||||
messages=messages,
|
||||
temperature=0.0,
|
||||
)
|
||||
print("Generated text:", response.choices[0].message.content)
|
||||
# Safety: Unsafe
|
||||
# Categories: Violent
|
||||
```
|
||||
|
||||
## Benchmarking
|
||||
|
||||
```bash
|
||||
vllm bench serve \
|
||||
--model Qwen/Qwen3Guard-Gen-8B \
|
||||
--dataset-name random \
|
||||
--random-input-len 2000 \
|
||||
--random-output-len 512 \
|
||||
--num-prompts 100
|
||||
```
|
||||
|
||||
## Available Variants
|
||||
|
||||
The Qwen3Guard-Gen series includes multiple model sizes, all compatible with the same vLLM serving commands:
|
||||
- **Qwen/Qwen3Guard-Gen-8B**
|
||||
- **Qwen/Qwen3Guard-Gen-4B**
|
||||
- **Qwen/Qwen3Guard-Gen-0.6B**
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/Qwen/Qwen3Guard-Gen-8B)
|
||||
- [Qwen3Guard-Gen-4B](https://huggingface.co/Qwen/Qwen3Guard-Gen-4B)
|
||||
- [Qwen3Guard-Gen-0.6B](https://huggingface.co/Qwen/Qwen3Guard-Gen-0.6B)
|
||||
@@ -0,0 +1,179 @@
|
||||
meta:
|
||||
title: "MiMo-V2-Flash"
|
||||
slug: "mimo-v2-flash"
|
||||
provider: "MiMo (Xiaomi)"
|
||||
description: "Xiaomi's MoE reasoning model (309B total / 15B active) with hybrid attention and MTP for fast agentic workflows"
|
||||
date_added: 2025-12-20
|
||||
date_updated: 2026-10-08
|
||||
tasks:
|
||||
- text
|
||||
related_recipes: []
|
||||
hardware:
|
||||
h200: verified
|
||||
mi300x: verified
|
||||
mi325x: verified
|
||||
mi355x: verified
|
||||
|
||||
model:
|
||||
model_id: "XiaomiMiMo/MiMo-V2-Flash"
|
||||
min_vllm_version: "0.20.0"
|
||||
architecture: moe
|
||||
parameter_count: "309B"
|
||||
active_parameters: "15B"
|
||||
context_length: 262144
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
- "--generation-config"
|
||||
- "vllm"
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "MiMo tool-call parser"
|
||||
args:
|
||||
- "--tool-call-parser"
|
||||
- "mimo"
|
||||
reasoning:
|
||||
description: "Qwen3 reasoning parser"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "qwen3"
|
||||
|
||||
opt_in_features: []
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: fp8
|
||||
vram_minimum_gb: 371
|
||||
description: "Native FP8 weights; 4x H200 recommended with TP4"
|
||||
extra_args:
|
||||
- "--tensor-parallel-size"
|
||||
- "4"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.9"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- single_node_tep
|
||||
- multi_node_tp
|
||||
- multi_node_tp_pp
|
||||
- multi_node_tep
|
||||
- multi_node_dep
|
||||
- pd_cluster
|
||||
|
||||
hardware_overrides:
|
||||
amd:
|
||||
# AITER is disabled for MiMo-V2-Flash (regression when enabled).
|
||||
extra_args: []
|
||||
extra_env:
|
||||
VLLM_ROCM_USE_AITER: "0"
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[MiMo-V2-Flash](https://huggingface.co/XiaomiMiMo/MiMo-V2-Flash) is a MoE language model
|
||||
with 309B total parameters and 15B active. Designed for high-speed reasoning and agentic
|
||||
workflows, it features hybrid attention and Multi-Token Prediction (MTP) to reduce
|
||||
inference cost.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Hardware: 4x H200 (TP4) or equivalent aggregate VRAM (~320 GB with FP8)
|
||||
- vLLM >= 0.20.0
|
||||
|
||||
### Install vLLM (NVIDIA)
|
||||
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install vllm --torch-backend auto
|
||||
```
|
||||
|
||||
### Install vLLM (AMD ROCm MI300X/MI325X/MI355X)
|
||||
|
||||
```bash
|
||||
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
|
||||
```
|
||||
|
||||
## Launch commands
|
||||
|
||||
Basic TP4:
|
||||
|
||||
```bash
|
||||
vllm serve XiaomiMiMo/MiMo-V2-Flash \
|
||||
--host 0.0.0.0 --port 9001 --seed 1024 \
|
||||
--served-model-name mimo_v2_flash \
|
||||
--tensor-parallel-size 4 \
|
||||
--trust-remote-code \
|
||||
--gpu-memory-utilization 0.9 \
|
||||
--generation-config vllm
|
||||
```
|
||||
|
||||
With tool calling + reasoning:
|
||||
|
||||
```bash
|
||||
vllm serve XiaomiMiMo/MiMo-V2-Flash \
|
||||
--tensor-parallel-size 4 --trust-remote-code --gpu-memory-utilization 0.9 \
|
||||
--tool-call-parser mimo \
|
||||
--reasoning-parser qwen3 \
|
||||
--generation-config vllm \
|
||||
--served-model-name mimo_v2_flash
|
||||
```
|
||||
|
||||
DP + TP + EP:
|
||||
|
||||
```bash
|
||||
vllm serve XiaomiMiMo/MiMo-V2-Flash \
|
||||
--data-parallel-size 2 \
|
||||
--tensor-parallel-size 4 \
|
||||
--enable-expert-parallel \
|
||||
--trust-remote-code \
|
||||
--gpu-memory-utilization 0.9 \
|
||||
--generation-config vllm \
|
||||
--served-model-name mimo_v2_flash
|
||||
```
|
||||
|
||||
AMD:
|
||||
|
||||
```bash
|
||||
export VLLM_ROCM_USE_AITER=0
|
||||
vllm serve XiaomiMiMo/MiMo-V2-Flash --tensor-parallel-size 4 \
|
||||
--trust-remote-code --gpu-memory-utilization 0.9 --generation-config vllm
|
||||
```
|
||||
|
||||
Tunable flags:
|
||||
- `--max-model-len=65536` works well; max is 128K.
|
||||
- `--max-num-batched-tokens=32768` for prompt-heavy; 16K/8K for lower latency.
|
||||
- `--gpu-memory-utilization=0.95` to maximize KV cache.
|
||||
|
||||
## Client Usage
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:9001/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "mimo_v2_flash",
|
||||
"messages": [{"role": "user", "content": "Hello MiMo!"}],
|
||||
"chat_template_kwargs": {"enable_thinking": true}
|
||||
}'
|
||||
```
|
||||
|
||||
Set `"enable_thinking": false` (or omit the kwargs) to disable thinking mode.
|
||||
|
||||
## Benchmarking
|
||||
|
||||
```bash
|
||||
vllm bench serve \
|
||||
--model XiaomiMiMo/MiMo-V2-Flash \
|
||||
--dataset-name random --random-input-len 8000 --random-output-len 1000 \
|
||||
--request-rate 3 --num-prompts 1800 --ignore-eos
|
||||
```
|
||||
|
||||
## Accuracy (GSM8K)
|
||||
|
||||
Reported 5-shot `exact_match`: flexible 0.9128, strict 0.9075.
|
||||
|
||||
## References
|
||||
|
||||
- [MiMo-V2-Flash on Hugging Face](https://huggingface.co/XiaomiMiMo/MiMo-V2-Flash)
|
||||
@@ -0,0 +1,137 @@
|
||||
meta:
|
||||
title: "MiMo-V2.5-Pro"
|
||||
slug: "mimo-v2-5-pro"
|
||||
provider: "MiMo (Xiaomi)"
|
||||
description: "Xiaomi's flagship MoE reasoning model (1.02T total / 42B active) with hybrid attention, native FP8 weights, and Multi-Token Prediction"
|
||||
date_added: 2026-04-27
|
||||
date_updated: 2026-04-27
|
||||
tasks:
|
||||
- text
|
||||
related_recipes:
|
||||
- "XiaomiMiMo/MiMo-V2.5"
|
||||
hardware:
|
||||
h200: verified
|
||||
|
||||
model:
|
||||
model_id: "XiaomiMiMo/MiMo-V2.5-Pro"
|
||||
min_vllm_version: "0.21.0"
|
||||
architecture: moe
|
||||
parameter_count: "1T"
|
||||
active_parameters: "42B"
|
||||
context_length: 1048576
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
- "--max-model-len auto"
|
||||
- "--generation-config"
|
||||
- "vllm"
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "MiMo tool-call parser"
|
||||
args:
|
||||
- "--tool-call-parser"
|
||||
- "mimo"
|
||||
- "--enable-auto-tool-choice"
|
||||
reasoning:
|
||||
description: "MiMo reasoning parser"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "mimo"
|
||||
spec_decoding:
|
||||
description: "Multi-Token Prediction"
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"mtp","num_speculative_tokens":1}'
|
||||
|
||||
opt_in_features:
|
||||
- spec_decoding
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: fp8
|
||||
vram_minimum_gb: 1224
|
||||
description: "Native FP8 weights (block-wise e4m3 128x128); 8x H200 with TP8"
|
||||
extra_args:
|
||||
- "--tensor-parallel-size"
|
||||
- "8"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.95"
|
||||
- "--max-model-len"
|
||||
- "auto"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- single_node_tep
|
||||
- pd_cluster
|
||||
|
||||
hardware_overrides: {}
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
MiMo-V2.5-Pro is Xiaomi's flagship MoE reasoning model with 1.02T total parameters
|
||||
and 42B active per token. It uses 384 routed experts (top-8) with hybrid
|
||||
attention (full-attention + sliding-window 128 at 6:1 ratio) over 70 layers
|
||||
(1 dense + 69 MoE) and ships with native FP8 (block-wise e4m3) weights.
|
||||
A 3-layer Multi-Token Prediction (MTP) head enables speculative decoding for
|
||||
~3x output speed.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Hardware: 8x H200 (TP8)
|
||||
|
||||
### Pull the vLLM docker image
|
||||
|
||||
Stable vLLM does not yet support MiMo V2.5. Use the pre-built image:
|
||||
|
||||
```bash
|
||||
docker pull vllm/vllm-openai:mimov25-cu129
|
||||
```
|
||||
|
||||
## Launch commands
|
||||
|
||||
Single-node TP8 (H200):
|
||||
|
||||
```bash
|
||||
vllm serve XiaomiMiMo/MiMo-V2.5-Pro \
|
||||
--tensor-parallel-size 8 \
|
||||
--trust-remote-code \
|
||||
--gpu-memory-utilization 0.95 \
|
||||
--max-model-len auto \
|
||||
--generation-config vllm
|
||||
```
|
||||
|
||||
With tool calling + reasoning:
|
||||
|
||||
```bash
|
||||
vllm serve XiaomiMiMo/MiMo-V2.5-Pro \
|
||||
--tensor-parallel-size 8 \
|
||||
--trust-remote-code \
|
||||
--gpu-memory-utilization 0.95 \
|
||||
--max-model-len auto \
|
||||
--reasoning-parser mimo \
|
||||
--tool-call-parser mimo \
|
||||
--enable-auto-tool-choice \
|
||||
--generation-config vllm
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "XiaomiMiMo/MiMo-V2.5-Pro",
|
||||
"messages": [{"role": "user", "content": "Hello MiMo!"}],
|
||||
"chat_template_kwargs": {"enable_thinking": true}
|
||||
}'
|
||||
```
|
||||
|
||||
Set `"enable_thinking": false` (or omit the kwargs) to disable thinking mode.
|
||||
|
||||
|
||||
## References
|
||||
|
||||
- [MiMo-V2.5-Pro on Hugging Face](https://huggingface.co/XiaomiMiMo/MiMo-V2.5-Pro)
|
||||
@@ -0,0 +1,172 @@
|
||||
meta:
|
||||
title: "MiMo-V2.5"
|
||||
slug: "mimo-v2-5"
|
||||
provider: "MiMo (Xiaomi)"
|
||||
description: "MiMo-V2.5 is a native omnimodal model with strong agentic capabilities, supporting text, image, video, and audio understanding within a unified architecture"
|
||||
date_added: 2026-04-27
|
||||
date_updated: 2026-04-27
|
||||
tasks:
|
||||
- multimodal
|
||||
- text
|
||||
related_recipes:
|
||||
- "XiaomiMiMo/MiMo-V2-Flash"
|
||||
hardware:
|
||||
h200: verified
|
||||
|
||||
model:
|
||||
model_id: "XiaomiMiMo/MiMo-V2.5"
|
||||
min_vllm_version: "0.21.0"
|
||||
architecture: moe
|
||||
parameter_count: "311B"
|
||||
active_parameters: "15B"
|
||||
context_length: 1048576
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
- "--generation-config"
|
||||
- "vllm"
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "MiMo tool-call parser"
|
||||
args:
|
||||
- "--tool-call-parser"
|
||||
- "mimo"
|
||||
- "--enable-auto-tool-choice"
|
||||
reasoning:
|
||||
description: "MiMo reasoning parser"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "mimo"
|
||||
spec_decoding:
|
||||
description: "Multi-Token Prediction"
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"mtp","num_speculative_tokens":1}'
|
||||
|
||||
opt_in_features:
|
||||
- spec_decoding
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: fp8
|
||||
vram_minimum_gb: 374
|
||||
# FP8 block-wise e4m3 128x128 — TP=8 hits an attention-projection shape
|
||||
# mismatch (param padded to a 128-aligned shard the checkpoint can't fill).
|
||||
# Author-verified config is TP=4. EP doesn't help (mismatch is on attention,
|
||||
# not MoE), so single_node_tep at 8 GPUs is also broken — use a 4-GPU box
|
||||
# or accept 4 idle GPUs on an 8-GPU box.
|
||||
tp: 4
|
||||
description: "Native FP8 weights (block-wise e4m3 128x128); 4x H200 with TP4"
|
||||
extra_args:
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.95"
|
||||
- "--max-model-len"
|
||||
- "auto"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- single_node_tep
|
||||
- multi_node_tp
|
||||
- multi_node_tep
|
||||
- multi_node_dep
|
||||
- pd_cluster
|
||||
|
||||
hardware_overrides: {}
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
MiMo-V2.5 is Xiaomi's native omnimodal MoE model with 310B total parameters
|
||||
and 15B active per token, supporting text, image, video, and audio understanding
|
||||
within a unified architecture. Built on the MiMo-V2-Flash backbone with dedicated
|
||||
vision (729M) and audio (261M) encoders, it uses 256 routed experts (top-8) with
|
||||
hybrid attention (SWA-128 + full-attention at 5:1 ratio) over 48 layers
|
||||
(1 dense + 47 MoE) and ships with native FP8 (block-wise e4m3) weights.
|
||||
A 3-layer Multi-Token Prediction (MTP) head is included for speculative decoding.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Hardware: 4x H200 (TP4)
|
||||
|
||||
### Pull the vLLM docker image
|
||||
|
||||
Stable vLLM does not yet support MiMo V2.5. Use the pre-built image:
|
||||
|
||||
```bash
|
||||
docker pull vllm/vllm-openai:mimov25-cu129
|
||||
```
|
||||
|
||||
## Launch commands
|
||||
|
||||
Single-node TP4 (H200):
|
||||
|
||||
```bash
|
||||
vllm serve XiaomiMiMo/MiMo-V2.5 \
|
||||
--tensor-parallel-size 4 \
|
||||
--trust-remote-code \
|
||||
--gpu-memory-utilization 0.95 \
|
||||
--max-model-len auto \
|
||||
--generation-config vllm
|
||||
```
|
||||
|
||||
With tool calling + reasoning:
|
||||
|
||||
```bash
|
||||
vllm serve XiaomiMiMo/MiMo-V2.5 \
|
||||
--tensor-parallel-size 4 \
|
||||
--trust-remote-code \
|
||||
--gpu-memory-utilization 0.95 \
|
||||
--max-model-len auto \
|
||||
--reasoning-parser mimo \
|
||||
--tool-call-parser mimo \
|
||||
--enable-auto-tool-choice \
|
||||
--generation-config vllm
|
||||
```
|
||||
|
||||
Tunable flags:
|
||||
- `--max-model-len` — full context is 1,048,576; use `auto` to size to KV budget.
|
||||
- `--max-num-batched-tokens=32768` for prompt-heavy workloads; lower for latency.
|
||||
- `--gpu-memory-utilization=0.95` to maximize KV cache.
|
||||
|
||||
## Client Usage
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "XiaomiMiMo/MiMo-V2.5",
|
||||
"messages": [{"role": "user", "content": "Hello MiMo!"}],
|
||||
"chat_template_kwargs": {"enable_thinking": true}
|
||||
}'
|
||||
```
|
||||
|
||||
Set `"enable_thinking": false` (or omit the kwargs) to disable thinking mode.
|
||||
|
||||
## Benchmarking
|
||||
|
||||
Launch the server with `--no-enable-prefix-caching` to get consistent measurements.
|
||||
|
||||
### VisionArena-Chat
|
||||
```
|
||||
vllm bench serve \
|
||||
--model XiaomiMiMo/MiMo-V2.5 \
|
||||
--backend openai-chat \
|
||||
--endpoint /v1/chat/completions \
|
||||
--dataset-name hf \
|
||||
--dataset-path lmarena-ai/VisionArena-Chat \
|
||||
--num-prompts 128
|
||||
```
|
||||
|
||||
### Random Synthetic
|
||||
```bash
|
||||
vllm bench serve \
|
||||
--model XiaomiMiMo/MiMo-V2.5 \
|
||||
--dataset-name random --random-input-len 8000 --random-output-len 1000 \
|
||||
--request-rate 3 --num-prompts 1800 --ignore-eos
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [MiMo-V2.5 on Hugging Face](https://huggingface.co/XiaomiMiMo/MiMo-V2.5)
|
||||
@@ -0,0 +1,318 @@
|
||||
meta:
|
||||
title: "MiMo-V2.6-Flash-RL"
|
||||
slug: "mimo-v2-6-flash-rl"
|
||||
provider: "MiMo (Xiaomi)"
|
||||
description: "Xiaomi's efficiency-balanced omnimodal MoE (309B total / 15B active) with hybrid attention, FP8-compute/mxfp4-stored weights, 1M context, and a DFlash speculative decoder — the default MOPD checkpoint fixes the RL release's tool-call repetition"
|
||||
date_added: 2026-09-22
|
||||
date_updated: 2026-10-07
|
||||
tasks:
|
||||
- text
|
||||
- multimodal
|
||||
related_recipes:
|
||||
- "XiaomiMiMo/MiMo-V2.6-Pro-RL"
|
||||
- "XiaomiMiMo/MiMo-V2-Flash"
|
||||
hardware:
|
||||
h200: verified
|
||||
gb300: verified
|
||||
mi300x: verified
|
||||
mi325x: verified
|
||||
mi355x: verified
|
||||
|
||||
model:
|
||||
model_id: "XiaomiMiMo/MiMo-V2.6-Flash-RL"
|
||||
min_vllm_version: "0.31.0"
|
||||
docker_image:
|
||||
nvidia: "vllm/vllm-openai:v0.31.0"
|
||||
# AMD stays on the ROCm nightly: the gfx942 MXFP4 routing gate (vllm#58262)
|
||||
# merged after the v0.31.0 branch cut, so MI300X/MI325X need it.
|
||||
amd: "vllm/vllm-openai-rocm:nightly"
|
||||
architecture: moe
|
||||
parameter_count: "309B"
|
||||
active_parameters: "15B"
|
||||
context_length: 1048576
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
- "--max-model-len"
|
||||
- "auto"
|
||||
- "--generation-config"
|
||||
- "vllm"
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "MiMo tool-call parser"
|
||||
args:
|
||||
- "--tool-call-parser"
|
||||
- "mimo"
|
||||
- "--enable-auto-tool-choice"
|
||||
reasoning:
|
||||
description: "MiMo reasoning parser"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "mimo"
|
||||
spec_decoding:
|
||||
description: "DFlash 5-layer MTP drafter (predicts 7 tokens); ships in the dflash/ subfolder of every checkpoint. Set DFLASH_DIR to that directory first (see the guide)"
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"dflash","model":"$DFLASH_DIR","num_speculative_tokens":7}'
|
||||
|
||||
opt_in_features:
|
||||
- spec_decoding
|
||||
|
||||
variants:
|
||||
# MOPD supersedes the RL release as the default served checkpoint (same
|
||||
# architecture and weight format; the RL checkpoint stays selectable below),
|
||||
# mirroring how the DeepSeek-V4-Pro recipe rolled its 0813 release to default.
|
||||
default:
|
||||
model_id: "XiaomiMiMo/MiMo-V2.6-Flash-MOPD"
|
||||
label: "MOPD"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 208
|
||||
description: "FP8 (e4m3) compute with mxfp4-stored MoE weights; 173 GB checkpoint — 4x H200 with TP4. MOPD2-distilled upgrade that fixes tool-call repetition; recommended."
|
||||
tp:
|
||||
mi300x: 2
|
||||
extra_args:
|
||||
- "--tensor-parallel-size"
|
||||
- "4"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.95"
|
||||
hardware_overrides:
|
||||
mi300x:
|
||||
extra_args:
|
||||
- "--moe-backend"
|
||||
- "aiter_triton_mxfp4_bf16"
|
||||
mi325x:
|
||||
extra_args:
|
||||
- "--moe-backend"
|
||||
- "aiter_triton_mxfp4_bf16"
|
||||
rl:
|
||||
# The original 2026-09-22 RL checkpoint. No `model_id` override: it serves
|
||||
# the recipe's own repo, like DeepSeek-V4-Pro's "FP8 (Preview)" variant.
|
||||
label: "RL (original)"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 208
|
||||
description: "Same shape as the MOPD variant (173 GB, FP8-compute/mxfp4-stored) — the original RL release, kept for reproducing earlier results. Suffers from tool-call repetition (the more affected of the two sizes); prefer MOPD for agentic workloads."
|
||||
tp:
|
||||
mi300x: 2
|
||||
extra_args:
|
||||
- "--tensor-parallel-size"
|
||||
- "4"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.95"
|
||||
hardware_overrides:
|
||||
mi300x:
|
||||
extra_args:
|
||||
- "--moe-backend"
|
||||
- "aiter_triton_mxfp4_bf16"
|
||||
mi325x:
|
||||
extra_args:
|
||||
- "--moe-backend"
|
||||
- "aiter_triton_mxfp4_bf16"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- single_node_tep
|
||||
- multi_node_tp
|
||||
- multi_node_tep
|
||||
- multi_node_dep
|
||||
- pd_cluster
|
||||
|
||||
hardware_overrides:
|
||||
amd:
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
extra_env:
|
||||
VLLM_ROCM_USE_AITER: "1"
|
||||
VLLM_ROCM_QUICK_REDUCE_QUANTIZATION: "INT4"
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
This recipe serves the MiMo-V2.6-Flash family from two checkpoints on the
|
||||
Variant row:
|
||||
|
||||
- **MOPD** (default) — [XiaomiMiMo/MiMo-V2.6-Flash-MOPD](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-MOPD),
|
||||
the MOPD2-distilled upgrade and the recommended starting point for new
|
||||
deployments. It fixes the RL release's most visible agentic failure mode —
|
||||
tool-call repetition, where the model keeps re-issuing the same or highly
|
||||
similar tool calls while making no progress (the blog measures the Flash
|
||||
line as the more affected of the two) — with benchmark quality unchanged,
|
||||
folded into a regular MOPD2 pass via a short specialized-teacher run
|
||||
([diagnosis blog](https://mimo.xiaomi.com/blog/mimo-v2-6-tool-call-repetition)).
|
||||
- **RL (original)** — [XiaomiMiMo/MiMo-V2.6-Flash-RL](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL),
|
||||
the 2026-09-22 RL release, kept for reproducing earlier results. It can get
|
||||
stuck re-issuing the same tool calls in agentic workloads; prefer **MOPD**
|
||||
unless you are reproducing a specific earlier run.
|
||||
|
||||
MOPD2 (Technical Report §5.6) is multi-teacher on-policy distillation:
|
||||
domain-specialized mixRL and SFT teachers supervise the student on three
|
||||
on-policy streams (standard MOPD, teacher-prefix OPD, and SFT-prefix OPD).
|
||||
|
||||
The architecture is shared by both checkpoints: a sparse MoE model with
|
||||
309B total parameters and 15B active per token — 48 layers (1 dense + 47 MoE)
|
||||
with 256 routed experts (top-8), hybrid attention (sliding-window 128 and
|
||||
global attention), and a 5-layer DFlash-style MTP drafter that predicts 7
|
||||
tokens per pass. The model is natively omnimodal (text, image, video, audio
|
||||
via a 681M-param MiMo ViT and audio encoders) and supports up to 1M tokens of
|
||||
context.
|
||||
|
||||
Weights are stored as mxfp4 and computed as FP8 (block-wise e4m3, 128x128),
|
||||
so the checkpoint is only 173 GB on disk. Loading this mixed storage format
|
||||
requires vLLM with the MiMo V2 mxfp4/bf16-router support
|
||||
([vllm-project/vllm#57784](https://github.com/vllm-project/vllm/pull/57784)),
|
||||
first shipped in the v0.31.0 stable release.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Hardware: 4x H200 (TP4), 4x MI355X/MI325X/MI300X (TP4), or equivalent aggregate VRAM (>= 208 GB)
|
||||
|
||||
### Pull the vLLM docker image
|
||||
|
||||
vLLM 0.31.0 is the first stable release that can load the mxfp4-stored
|
||||
weights ([vllm-project/vllm#57784](https://github.com/vllm-project/vllm/pull/57784));
|
||||
earlier stable releases cannot:
|
||||
|
||||
```bash
|
||||
docker pull vllm/vllm-openai:v0.31.0
|
||||
```
|
||||
|
||||
MI300X / MI325X (gfx942) additionally need the MiMo MXFP4 routing gate
|
||||
([vllm-project/vllm#58262](https://github.com/vllm-project/vllm/pull/58262)),
|
||||
which merged after the v0.31.0 branch cut — on those GPUs use a nightly wheel
|
||||
built after 2026-09-30 (`uv pip install -U vllm --extra-index-url
|
||||
https://wheels.vllm.ai/nightly/cu130`) or the ROCm nightly image.
|
||||
|
||||
This release also registers the `mimo` tool-call and reasoning parsers with
|
||||
the Rust frontend
|
||||
([vllm-project/vllm#57933](https://github.com/vllm-project/vllm/pull/57933)).
|
||||
|
||||
The Hugging Face model cards point at `vllm/vllm-openai:mimov25-cu129`, the
|
||||
V2.5-era image; that tag predates the mxfp4-stored-weight support linked above,
|
||||
so use the v0.31.0 image recommended above.
|
||||
|
||||
## Launch command
|
||||
|
||||
Single-node TP4 (H200). Selecting the **RL (original)** variant in the builder
|
||||
swaps only the checkpoint id (`XiaomiMiMo/MiMo-V2.6-Flash-RL`); everything
|
||||
else is identical:
|
||||
|
||||
```bash
|
||||
vllm serve XiaomiMiMo/MiMo-V2.6-Flash-MOPD \
|
||||
--tensor-parallel-size 4 \
|
||||
--trust-remote-code \
|
||||
--gpu-memory-utilization 0.95 \
|
||||
--max-model-len auto \
|
||||
--reasoning-parser mimo \
|
||||
--tool-call-parser mimo \
|
||||
--enable-auto-tool-choice \
|
||||
--generation-config vllm
|
||||
```
|
||||
|
||||
### AMD (MI355X, gfx950)
|
||||
|
||||
```bash
|
||||
VLLM_ROCM_USE_AITER=1 \
|
||||
VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 \
|
||||
vllm serve XiaomiMiMo/MiMo-V2.6-Flash-MOPD \
|
||||
--tensor-parallel-size 4 \
|
||||
--trust-remote-code \
|
||||
--kv-cache-dtype fp8 \
|
||||
--max-model-len auto \
|
||||
--reasoning-parser mimo \
|
||||
--tool-call-parser mimo \
|
||||
--enable-auto-tool-choice \
|
||||
--generation-config vllm
|
||||
```
|
||||
|
||||
MI355X also runs TP2 just fine.
|
||||
|
||||
### AMD (MI300X / MI325X, gfx942)
|
||||
|
||||
gfx942 has no native MXFP4 MoE path: without an explicit backend vLLM's
|
||||
oracle falls back to OCP emulation — correct, but about 6.5x lower output
|
||||
throughput. Select AITER's Triton W4A16 kernel instead with
|
||||
`--moe-backend aiter_triton_mxfp4_bf16`; supported since
|
||||
[vllm-project/vllm#58262](https://github.com/vllm-project/vllm/pull/58262)
|
||||
(merged 2026-09-30, after the v0.31.0 branch cut — not in any stable release
|
||||
yet, so this flag needs a nightly wheel; it also needs aiter >= 0.1.20, which
|
||||
the ROCm nightly image ships). Verified end-to-end on MI300X in that PR — see
|
||||
it for the gsm8k and `vllm bench serve` numbers against the emulation
|
||||
baseline:
|
||||
|
||||
```bash
|
||||
VLLM_ROCM_USE_AITER=1 \
|
||||
VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 \
|
||||
vllm serve XiaomiMiMo/MiMo-V2.6-Flash-MOPD \
|
||||
--tensor-parallel-size 2 \
|
||||
--trust-remote-code \
|
||||
--max-model-len auto \
|
||||
--reasoning-parser mimo \
|
||||
--tool-call-parser mimo \
|
||||
--enable-auto-tool-choice \
|
||||
--moe-backend aiter_triton_mxfp4_bf16 \
|
||||
--generation-config vllm
|
||||
```
|
||||
With DFlash speculative decoding (drafter ships inside the checkpoint).
|
||||
vLLM does not resolve a `<repo>/dflash` Hub subfolder as the draft model —
|
||||
`model` must be a concrete local path to the `dflash/` directory inside the
|
||||
downloaded snapshot. Resolve it per checkpoint (run `hf download
|
||||
<checkpoint>` first if it is not cached yet):
|
||||
|
||||
```bash
|
||||
# MOPD variant
|
||||
DFLASH_DIR=$(ls -d ~/.cache/huggingface/hub/models--XiaomiMiMo--MiMo-V2.6-Flash-MOPD/snapshots/*/dflash)
|
||||
# RL (original) variant
|
||||
# DFLASH_DIR=$(ls -d ~/.cache/huggingface/hub/models--XiaomiMiMo--MiMo-V2.6-Flash-RL/snapshots/*/dflash)
|
||||
echo "$DFLASH_DIR"
|
||||
```
|
||||
|
||||
```bash
|
||||
vllm serve XiaomiMiMo/MiMo-V2.6-Flash-MOPD \
|
||||
--tensor-parallel-size 4 \
|
||||
--trust-remote-code \
|
||||
--gpu-memory-utilization 0.95 \
|
||||
--max-model-len auto \
|
||||
--speculative-config "{\"method\":\"dflash\",\"model\":\"$DFLASH_DIR\",\"num_speculative_tokens\":7}" \
|
||||
--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \
|
||||
--reasoning-parser mimo \
|
||||
--tool-call-parser mimo \
|
||||
--enable-auto-tool-choice \
|
||||
--generation-config vllm
|
||||
```
|
||||
|
||||
The drafter inherits the target's tensor-parallel size by default. To pin it
|
||||
explicitly, add `"draft_tensor_parallel_size": 4` to the speculative config
|
||||
(useful if you want the draft on fewer GPUs than the target, or to silence
|
||||
mis-detection on asymmetric topologies).
|
||||
|
||||
## Client Usage
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "XiaomiMiMo/MiMo-V2.6-Flash-MOPD",
|
||||
"messages": [{"role": "user", "content": "Hello MiMo!"}],
|
||||
"temperature": 1.0,
|
||||
"top_p": 0.95,
|
||||
"chat_template_kwargs": {"enable_thinking": true}
|
||||
}'
|
||||
```
|
||||
|
||||
Recommended sampling: `temperature=1.0`, `top_p=0.95`.
|
||||
Set `"enable_thinking": false` (or omit the kwargs) to disable thinking mode.
|
||||
|
||||
## References
|
||||
|
||||
- [MiMo-V2.6-Flash-MOPD on Hugging Face](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-MOPD)
|
||||
- [MiMo-V2.6-Flash-RL on Hugging Face](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Flash-RL)
|
||||
- [MiMo-V2.6 technical report](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL/blob/main/MiMo_V2_6_technical_report.pdf) (§5.6 covers MOPD2)
|
||||
- [Diagnosing and Mitigating Tool-Call Repetition in MiMo-V2.6](https://mimo.xiaomi.com/blog/mimo-v2-6-tool-call-repetition)
|
||||
- [MiMo-V2.6 blog post](https://mimo.xiaomi.com/mimo-v2-6)
|
||||
- [vLLM PR #57784 — mxfp4 MoE / bf16 router / DFlash for MiMo V2](https://github.com/vllm-project/vllm/pull/57784)
|
||||
- [vLLM PR #57933 — MiMo tool-call/reasoning parsers for the Rust frontend](https://github.com/vllm-project/vllm/pull/57933)
|
||||
- [vLLM PR #58262 — MiMo-V2.6 MXFP4 on gfx942 (`--moe-backend aiter_triton_mxfp4_bf16`)](https://github.com/vllm-project/vllm/pull/58262)
|
||||
- [SGLang MiMo cookbook](https://docs.sglang.io/cookbook/autoregressive/Xiaomi/MiMo-V2.5)
|
||||
- [vLLM ROCm GPU install docs](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/?device=rocm)
|
||||
@@ -0,0 +1,284 @@
|
||||
meta:
|
||||
title: "MiMo-V2.6-Pro-RL"
|
||||
slug: "mimo-v2-6-pro-rl"
|
||||
provider: "MiMo (Xiaomi)"
|
||||
description: "Xiaomi's flagship omnimodal MoE (1.02T total / 42B active) with hybrid attention, FP8-compute/mxfp4-stored weights, 1M context, and a DFlash speculative decoder — the default MOPD checkpoint fixes the RL release's tool-call repetition"
|
||||
date_added: 2026-09-22
|
||||
date_updated: 2026-10-07
|
||||
tasks:
|
||||
- text
|
||||
- multimodal
|
||||
related_recipes:
|
||||
- "XiaomiMiMo/MiMo-V2.6-Flash-RL"
|
||||
- "XiaomiMiMo/MiMo-V2.5-Pro"
|
||||
hardware:
|
||||
h200: verified
|
||||
gb300: verified
|
||||
mi355x: verified
|
||||
|
||||
model:
|
||||
model_id: "XiaomiMiMo/MiMo-V2.6-Pro-RL"
|
||||
min_vllm_version: "0.31.0"
|
||||
docker_image:
|
||||
nvidia: "vllm/vllm-openai:v0.31.0"
|
||||
# AMD stays on the ROCm nightly the MI355X verification ran against.
|
||||
amd: "vllm/vllm-openai-rocm:nightly"
|
||||
architecture: moe
|
||||
parameter_count: "1T"
|
||||
active_parameters: "42B"
|
||||
context_length: 1048576
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
- "--max-model-len"
|
||||
- "auto"
|
||||
- "--generation-config"
|
||||
- "vllm"
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "MiMo tool-call parser"
|
||||
args:
|
||||
- "--tool-call-parser"
|
||||
- "mimo"
|
||||
- "--enable-auto-tool-choice"
|
||||
reasoning:
|
||||
description: "MiMo reasoning parser"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "mimo"
|
||||
spec_decoding:
|
||||
description: "DFlash 5-layer MTP drafter (predicts 7 tokens); ships in the dflash/ subfolder of every checkpoint. Set DFLASH_DIR to that directory first (see the guide)"
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"dflash","model":"$DFLASH_DIR","num_speculative_tokens":7}'
|
||||
|
||||
opt_in_features:
|
||||
- spec_decoding
|
||||
|
||||
variants:
|
||||
# MOPD supersedes the RL release as the default served checkpoint (same
|
||||
# architecture and weight format; the RL checkpoint stays selectable below),
|
||||
# mirroring how the DeepSeek-V4-Pro recipe rolled its 0813 release to default.
|
||||
default:
|
||||
model_id: "XiaomiMiMo/MiMo-V2.6-Pro-MOPD"
|
||||
label: "MOPD"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 680
|
||||
description: "FP8 (e4m3) compute with mxfp4-stored MoE weights; 566 GB checkpoint — 8x H200 with TP8, or 4x MI355X with TP4. MOPD2-distilled upgrade that fixes tool-call repetition; recommended."
|
||||
tp:
|
||||
mi355x: 4
|
||||
extra_args:
|
||||
- "--tensor-parallel-size"
|
||||
- "8"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.95"
|
||||
rl:
|
||||
# The original 2026-09-22 RL checkpoint. No `model_id` override: it serves
|
||||
# the recipe's own repo, like DeepSeek-V4-Pro's "FP8 (Preview)" variant.
|
||||
label: "RL (original)"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 680
|
||||
description: "Same shape as the MOPD variant (566 GB, FP8-compute/mxfp4-stored) — the original RL release, kept for reproducing earlier results. Suffers from tool-call repetition; prefer MOPD for agentic workloads."
|
||||
tp:
|
||||
mi355x: 4
|
||||
extra_args:
|
||||
- "--tensor-parallel-size"
|
||||
- "8"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.95"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- single_node_tep
|
||||
- pd_cluster
|
||||
|
||||
hardware_overrides:
|
||||
amd:
|
||||
# AITER is opt-in on ROCm and the image does not set it; without it the MoE
|
||||
# falls back off the native CDNA 4 MXFP4 path. INT4 quick-reduce quantizes the
|
||||
# allreduce, which is a real cost at this TP. Verified 2026-09-23 on an
|
||||
# 8x MI355X node, vllm/vllm-openai-rocm:nightly @ 0961bbae2.
|
||||
extra_args:
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.93"
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
extra_env:
|
||||
VLLM_ROCM_USE_AITER: "1"
|
||||
VLLM_ROCM_QUICK_REDUCE_QUANTIZATION: "INT4"
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
This recipe serves the MiMo-V2.6-Pro family from two checkpoints on the
|
||||
Variant row:
|
||||
|
||||
- **MOPD** (default) — [XiaomiMiMo/MiMo-V2.6-Pro-MOPD](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-MOPD),
|
||||
the MOPD2-distilled upgrade and the recommended starting point for new
|
||||
deployments. It fixes the RL release's most visible agentic failure mode —
|
||||
tool-call repetition, where the model keeps re-issuing the same or highly
|
||||
similar tool calls while making no progress — with benchmark quality
|
||||
unchanged, folded into a regular MOPD2 pass via a short specialized-teacher
|
||||
run ([diagnosis blog](https://mimo.xiaomi.com/blog/mimo-v2-6-tool-call-repetition)).
|
||||
- **RL (original)** — [XiaomiMiMo/MiMo-V2.6-Pro-RL](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL),
|
||||
the 2026-09-22 RL release, kept for reproducing earlier results. It can get
|
||||
stuck re-issuing the same tool calls in agentic workloads; prefer **MOPD**
|
||||
unless you are reproducing a specific earlier run.
|
||||
|
||||
MOPD2 (Technical Report §5.6) is multi-teacher on-policy distillation:
|
||||
domain-specialized mixRL and SFT teachers supervise the student on three
|
||||
on-policy streams (standard MOPD, teacher-prefix OPD, and SFT-prefix OPD).
|
||||
|
||||
The architecture is shared by both checkpoints: a sparse MoE model with
|
||||
1.02T total parameters and 42B active per token — 70 layers (1 dense + 69 MoE)
|
||||
with 384 routed experts (top-8), hybrid attention (sliding-window 128 and
|
||||
global attention at a 6:1 ratio), and a 5-layer DFlash-style MTP drafter that
|
||||
predicts 7 tokens per pass. The model is natively omnimodal (text, image,
|
||||
video, audio via a 681M-param MiMo ViT and audio encoders) and supports up to
|
||||
1M tokens of context.
|
||||
|
||||
Weights are stored as mxfp4 and computed as FP8 (block-wise e4m3, 128x128),
|
||||
so the checkpoint is 566 GB on disk. Loading this mixed storage format
|
||||
requires vLLM with the MiMo V2 mxfp4/bf16-router support
|
||||
([vllm-project/vllm#57784](https://github.com/vllm-project/vllm/pull/57784)),
|
||||
first shipped in the v0.31.0 stable release.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Hardware: 8x H200 (TP8), 4x MI355X (TP4), or equivalent aggregate VRAM (>= 680 GB)
|
||||
|
||||
### Pull the vLLM docker image
|
||||
|
||||
vLLM 0.31.0 is the first stable release that can load the mxfp4-stored
|
||||
weights ([vllm-project/vllm#57784](https://github.com/vllm-project/vllm/pull/57784));
|
||||
earlier stable releases cannot:
|
||||
|
||||
```bash
|
||||
docker pull vllm/vllm-openai:v0.31.0
|
||||
```
|
||||
|
||||
A vLLM nightly wheel built after 2026-09-20 also works (`uv pip install -U
|
||||
vllm --extra-index-url https://wheels.vllm.ai/nightly/cu130`).
|
||||
|
||||
The Hugging Face model cards point at `vllm/vllm-openai:mimov25-cu129`, the
|
||||
V2.5-era image; that tag predates the mxfp4-stored-weight support linked above,
|
||||
so use the v0.31.0 image recommended above.
|
||||
|
||||
This release also registers the `mimo` tool-call and reasoning parsers with
|
||||
the Rust frontend
|
||||
([vllm-project/vllm#57933](https://github.com/vllm-project/vllm/pull/57933)).
|
||||
|
||||
## Launch command
|
||||
|
||||
Single-node TP8 (H200). Selecting the **RL (original)** variant in the builder
|
||||
swaps only the checkpoint id (`XiaomiMiMo/MiMo-V2.6-Pro-RL`); everything else
|
||||
is identical:
|
||||
|
||||
```bash
|
||||
vllm serve XiaomiMiMo/MiMo-V2.6-Pro-MOPD \
|
||||
--tensor-parallel-size 8 \
|
||||
--trust-remote-code \
|
||||
--gpu-memory-utilization 0.95 \
|
||||
--max-model-len auto \
|
||||
--reasoning-parser mimo \
|
||||
--tool-call-parser mimo \
|
||||
--enable-auto-tool-choice \
|
||||
--generation-config vllm
|
||||
```
|
||||
|
||||
### AMD (MI355X, TP4)
|
||||
|
||||
```bash
|
||||
VLLM_ROCM_USE_AITER=1 \
|
||||
VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4 \
|
||||
vllm serve XiaomiMiMo/MiMo-V2.6-Pro-MOPD \
|
||||
--tensor-parallel-size 4 \
|
||||
--trust-remote-code \
|
||||
--gpu-memory-utilization 0.93 \
|
||||
--kv-cache-dtype fp8 \
|
||||
--max-model-len auto \
|
||||
--reasoning-parser mimo \
|
||||
--tool-call-parser mimo \
|
||||
--enable-auto-tool-choice \
|
||||
--generation-config vllm
|
||||
```
|
||||
|
||||
MXFP4 MoE weights run natively on CDNA 4, so no load-time requantization is
|
||||
needed. `VLLM_ROCM_USE_AITER=1` is off by default and is required to reach
|
||||
the AITER MXFP4 MoE and FP8 block-scaled GEMM kernels. Attention resolves to
|
||||
`TRITON_ATTN_DIFFKV`, the only backend covering this model's asymmetric head
|
||||
dims (QK 192 / V 128) together with attention sinks and sliding-window
|
||||
attention.
|
||||
|
||||
On MI355X, **Tensor Parallel** defaults to **TP4**: 4x288 GiB clears the 680 GiB
|
||||
requirement, so TP4 halves the GPUs per server and doubles the servers per node.
|
||||
For TP8, pass `--tensor-parallel-size 8`.
|
||||
|
||||
The checkpoint pre-shards its fused QKV into `num_key_value_heads` (8) chunks,
|
||||
each carrying its own FP8 scales. TP8 consumes them directly; TP4 merges two
|
||||
chunks per rank, which vLLM handles by dequantizing, reordering and
|
||||
re-quantizing at load time. TP4 costs no accuracy.
|
||||
|
||||
With DFlash speculative decoding (drafter ships inside the checkpoint).
|
||||
vLLM does not resolve a `<repo>/dflash` Hub subfolder as the draft model —
|
||||
`model` must be a concrete local path to the `dflash/` directory inside the
|
||||
downloaded snapshot. Resolve it per checkpoint (run `hf download
|
||||
<checkpoint>` first if it is not cached yet):
|
||||
|
||||
```bash
|
||||
# MOPD variant
|
||||
DFLASH_DIR=$(ls -d ~/.cache/huggingface/hub/models--XiaomiMiMo--MiMo-V2.6-Pro-MOPD/snapshots/*/dflash)
|
||||
# RL (original) variant
|
||||
# DFLASH_DIR=$(ls -d ~/.cache/huggingface/hub/models--XiaomiMiMo--MiMo-V2.6-Pro-RL/snapshots/*/dflash)
|
||||
echo "$DFLASH_DIR"
|
||||
```
|
||||
|
||||
```bash
|
||||
vllm serve XiaomiMiMo/MiMo-V2.6-Pro-MOPD \
|
||||
--tensor-parallel-size 8 \
|
||||
--trust-remote-code \
|
||||
--gpu-memory-utilization 0.95 \
|
||||
--max-model-len auto \
|
||||
--speculative-config "{\"method\":\"dflash\",\"model\":\"$DFLASH_DIR\",\"num_speculative_tokens\":7}" \
|
||||
--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}' \
|
||||
--reasoning-parser mimo \
|
||||
--tool-call-parser mimo \
|
||||
--enable-auto-tool-choice \
|
||||
--generation-config vllm
|
||||
```
|
||||
|
||||
The drafter inherits the target's tensor-parallel size by default. To pin it
|
||||
explicitly, add `"draft_tensor_parallel_size": 8` to the speculative config
|
||||
(useful if you want the draft on fewer GPUs than the target, or to silence
|
||||
mis-detection on asymmetric topologies).
|
||||
|
||||
## Client Usage
|
||||
|
||||
```bash
|
||||
curl -X POST http://localhost:8000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "XiaomiMiMo/MiMo-V2.6-Pro-MOPD",
|
||||
"messages": [{"role": "user", "content": "Hello MiMo!"}],
|
||||
"temperature": 1.0,
|
||||
"top_p": 0.95,
|
||||
"chat_template_kwargs": {"enable_thinking": true}
|
||||
}'
|
||||
```
|
||||
|
||||
Recommended sampling: `temperature=1.0`, `top_p=0.95`.
|
||||
Set `"enable_thinking": false` (or omit the kwargs) to disable thinking mode.
|
||||
|
||||
## References
|
||||
|
||||
- [MiMo-V2.6-Pro-MOPD on Hugging Face](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-MOPD)
|
||||
- [MiMo-V2.6-Pro-RL on Hugging Face](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL)
|
||||
- [MiMo-V2.6 technical report](https://huggingface.co/XiaomiMiMo/MiMo-V2.6-Pro-RL/blob/main/MiMo_V2_6_technical_report.pdf) (§5.6 covers MOPD2)
|
||||
- [Diagnosing and Mitigating Tool-Call Repetition in MiMo-V2.6](https://mimo.xiaomi.com/blog/mimo-v2-6-tool-call-repetition)
|
||||
- [MiMo-V2.6 blog post](https://mimo.xiaomi.com/mimo-v2-6)
|
||||
- [vLLM PR #57784 — mxfp4 MoE / bf16 router / DFlash for MiMo V2](https://github.com/vllm-project/vllm/pull/57784)
|
||||
- [vLLM PR #57933 — MiMo tool-call/reasoning parsers for the Rust frontend](https://github.com/vllm-project/vllm/pull/57933)
|
||||
- [SGLang MiMo cookbook](https://docs.sglang.io/cookbook/autoregressive/Xiaomi/MiMo-V2.5)
|
||||
- [vLLM ROCm GPU install docs](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/?device=rocm)
|
||||
@@ -0,0 +1,181 @@
|
||||
meta:
|
||||
title: "Trinity-Large-Thinking"
|
||||
slug: "trinity-large-thinking"
|
||||
provider: "Arcee AI"
|
||||
description: "Arcee AI's reasoning-focused sparse MoE (AfmoeForCausalLM) with structured <think> traces and agentic tool use"
|
||||
date_added: 2026-04-10
|
||||
date_updated: 2026-04-17
|
||||
tasks:
|
||||
- text
|
||||
related_recipes: []
|
||||
|
||||
model:
|
||||
model_id: "arcee-ai/Trinity-Large-Thinking"
|
||||
min_vllm_version: "0.11.1"
|
||||
architecture: moe
|
||||
parameter_count: "398B"
|
||||
active_parameters: "13B"
|
||||
context_length: 262144
|
||||
base_args:
|
||||
- "--dtype"
|
||||
- "bfloat16"
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Qwen3 Coder tool-call parser with automatic tool choice"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "qwen3_coder"
|
||||
reasoning:
|
||||
description: "DeepSeek-R1 reasoning parser extracts <think>...</think> into message.reasoning"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "deepseek_r1"
|
||||
|
||||
opt_in_features: []
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 955
|
||||
description: "Full precision BF16 on multi-GPU (sparse MoE — multi-GPU recommended)"
|
||||
nvfp4:
|
||||
model_id: "arcee-ai/Trinity-Large-Thinking-NVFP4"
|
||||
precision: nvfp4
|
||||
vram_minimum_gb: 239
|
||||
description: "NVIDIA NVFP4 quantized weights for Blackwell GPUs"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
|
||||
redhat_nvfp4:
|
||||
model_id: "RedHatAI/Trinity-Large-Thinking-NVFP4"
|
||||
precision: nvfp4
|
||||
label: "NVFP4 (RedHat)"
|
||||
vram_minimum_gb: 254
|
||||
description: "RedHatAI NVFP4 quantization"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
- multi_node_tp_pp
|
||||
- multi_node_tep
|
||||
- multi_node_dep
|
||||
|
||||
hardware_overrides: {}
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[Trinity-Large-Thinking](https://huggingface.co/arcee-ai/Trinity-Large-Thinking) is
|
||||
Arcee AI's reasoning-focused Trinity Large checkpoint — a sparse MoE model designed for
|
||||
long-horizon planning, tool use, and multi-step agent workflows. It uses the
|
||||
`AfmoeForCausalLM` architecture and emits explicit reasoning traces inside `<think>...</think>`
|
||||
blocks.
|
||||
|
||||
For multi-turn chat and agentic loops, reasoning tokens should be preserved across
|
||||
turns as part of the working state.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- vLLM >= 0.11.1
|
||||
- Hardware: multi-GPU recommended for production deployments
|
||||
|
||||
### Install vLLM
|
||||
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm openai --torch-backend auto
|
||||
```
|
||||
|
||||
## Launch command
|
||||
|
||||
```bash
|
||||
vllm serve arcee-ai/Trinity-Large-Thinking \
|
||||
--dtype bfloat16 \
|
||||
--reasoning-parser deepseek_r1 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser qwen3_coder
|
||||
```
|
||||
|
||||
Why these flags:
|
||||
- `--reasoning-parser deepseek_r1` extracts `<think>...</think>` into `message.reasoning`.
|
||||
- `--enable-auto-tool-choice` lets the model decide when to call tools.
|
||||
- `--tool-call-parser qwen3_coder` converts tool calls into OpenAI-style `tool_calls`.
|
||||
- `--dtype bfloat16` matches the recommended serving dtype.
|
||||
|
||||
Add parallelism flags (`--tensor-parallel-size`, `--data-parallel-size`, or
|
||||
`--enable-expert-parallel`) for your hardware. Lower `--max-model-len` if you don't
|
||||
need the full long-context config.
|
||||
|
||||
## Validation Request
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
|
||||
model = client.models.list().data[0].id
|
||||
|
||||
tools = [{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a location.",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {"location": {"type": "string"}},
|
||||
"required": ["location"],
|
||||
},
|
||||
},
|
||||
}]
|
||||
|
||||
response = client.chat.completions.create(
|
||||
model=model,
|
||||
messages=[{"role": "user", "content": "What is the weather in Paris right now?"}],
|
||||
tools=tools, tool_choice="auto",
|
||||
)
|
||||
|
||||
msg = response.choices[0].message
|
||||
reasoning = msg.reasoning
|
||||
print("reasoning:", reasoning)
|
||||
print("content:", msg.content)
|
||||
print("tool_calls:", msg.tool_calls)
|
||||
```
|
||||
|
||||
## Preserving Reasoning Across Turns
|
||||
|
||||
Pass reasoning back as `reasoning` on assistant messages:
|
||||
|
||||
```python
|
||||
assistant_msg = {"role": "assistant", "content": msg.content or ""}
|
||||
if reasoning:
|
||||
assistant_msg["reasoning"] = reasoning
|
||||
if msg.tool_calls:
|
||||
assistant_msg["tool_calls"] = [
|
||||
{"id": tc.id, "type": "function",
|
||||
"function": {"name": tc.function.name, "arguments": tc.function.arguments}}
|
||||
for tc in msg.tool_calls
|
||||
]
|
||||
messages.append(assistant_msg)
|
||||
```
|
||||
|
||||
Rules:
|
||||
- Pass reasoning back as `reasoning`.
|
||||
- Keep `content` as an empty string (not `null`) on tool-only turns.
|
||||
- Append the assistant message before tool-result messages.
|
||||
- Use `/v1/chat/completions` for structured reasoning output.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **No reasoning:** start server with `--reasoning-parser deepseek_r1`; use `/v1/chat/completions`.
|
||||
- **Tool calls as plain text:** enable `--enable-auto-tool-choice` and `--tool-call-parser qwen3_coder`.
|
||||
- **Loses coherence after tool turns:** preserve `reasoning` on each assistant turn; don't set content to `null`.
|
||||
- **OOM:** lower `--max-model-len`; scale parallelism; use a local checkpoint path.
|
||||
|
||||
## References
|
||||
|
||||
- [Trinity-Large-Thinking on Hugging Face](https://huggingface.co/arcee-ai/Trinity-Large-Thinking)
|
||||
@@ -0,0 +1,148 @@
|
||||
meta:
|
||||
title: "ERNIE-4.5"
|
||||
slug: "ernie-4.5"
|
||||
provider: "Ernie (Baidu)"
|
||||
description: "Baidu ERNIE 4.5 MoE text models (21B-A3B, 300B-A47B) with BF16 and FP8 support plus ERNIE-MTP speculative decoding"
|
||||
date_added: 2025-09-03
|
||||
date_updated: 2026-04-17
|
||||
tasks:
|
||||
- text
|
||||
related_recipes:
|
||||
- baidu/ERNIE-4.5-VL-28B-A3B-PT
|
||||
|
||||
model:
|
||||
model_id: "baidu/ERNIE-4.5-21B-A3B-PT"
|
||||
min_vllm_version: "0.10.1"
|
||||
architecture: moe
|
||||
parameter_count: "21B"
|
||||
active_parameters: "3B"
|
||||
context_length: 131072
|
||||
base_args: []
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
spec_decoding:
|
||||
description: "ERNIE-MTP (multi-token prediction) speculative decoding"
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"ernie_mtp","model":"baidu/ERNIE-4.5-21B-A3B-PT","num_speculative_tokens":1}'
|
||||
|
||||
opt_in_features:
|
||||
- spec_decoding
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 106
|
||||
description: "BF16 weights; fits on 1x80GB GPU (21B variant)"
|
||||
|
||||
"300b":
|
||||
model_id: "baidu/ERNIE-4.5-300B-A47B-PT"
|
||||
precision: bf16
|
||||
vram_minimum_gb: 640
|
||||
description: "300B total / 47B active; 8x80GB with FP8 online, 16x80GB for BF16"
|
||||
extra_args:
|
||||
- "--tensor-parallel-size"
|
||||
- "8"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.95"
|
||||
- "--quantization"
|
||||
- "fp8"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
- multi_node_tep
|
||||
- multi_node_dep
|
||||
|
||||
kv_offload_support:
|
||||
offloading_cpu: verified
|
||||
offloading_fs: verified
|
||||
|
||||
hardware_overrides:
|
||||
amd:
|
||||
extra_args:
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.9"
|
||||
extra_env:
|
||||
VLLM_ROCM_USE_AITER: "1"
|
||||
SAFETENSORS_FAST_GPU: "1"
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
ERNIE 4.5 is Baidu's MoE language model family. This recipe covers the text-only
|
||||
variants:
|
||||
|
||||
- `baidu/ERNIE-4.5-21B-A3B-PT` — 21B total / 3B active (fits on 1x80GB)
|
||||
- `baidu/ERNIE-4.5-300B-A47B-PT` — 300B total / 47B active (8x80GB FP8 or 16x80GB BF16)
|
||||
|
||||
Both support ERNIE-MTP speculative decoding via `--speculative-config`.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- transformers >= 4.54.0
|
||||
- vLLM >= 0.10.1
|
||||
- Hardware depends on variant (see above)
|
||||
|
||||
### Install vLLM
|
||||
|
||||
```bash
|
||||
uv venv --python 3.12 --seed
|
||||
source .venv/bin/activate
|
||||
uv pip install vllm --torch-backend=auto
|
||||
```
|
||||
|
||||
## Launch commands
|
||||
|
||||
21B on 1x80GB GPU:
|
||||
|
||||
```bash
|
||||
vllm serve baidu/ERNIE-4.5-21B-A3B-PT
|
||||
```
|
||||
|
||||
300B on 8x80GB with vLLM FP8 online quantization:
|
||||
|
||||
```bash
|
||||
vllm serve baidu/ERNIE-4.5-300B-A47B-PT \
|
||||
--tensor-parallel-size 8 \
|
||||
--gpu-memory-utilization 0.95 \
|
||||
--quantization fp8
|
||||
```
|
||||
|
||||
300B on 16x80GB native BF16 (multi-node via Ray):
|
||||
|
||||
```bash
|
||||
vllm serve baidu/ERNIE-4.5-300B-A47B-PT --tensor-parallel-size 16
|
||||
```
|
||||
|
||||
ERNIE-MTP speculative decoding (21B example):
|
||||
|
||||
```bash
|
||||
vllm serve baidu/ERNIE-4.5-21B-A3B-PT \
|
||||
--speculative-config '{"method":"ernie_mtp","model":"baidu/ERNIE-4.5-21B-A3B-PT","num_speculative_tokens":1}'
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
Standard OpenAI-compatible API; model ID is the HF repo.
|
||||
|
||||
## Benchmarking
|
||||
|
||||
```bash
|
||||
vllm bench serve \
|
||||
--model baidu/ERNIE-4.5-21B-A3B-PT \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 --random-output-len 1000 \
|
||||
--request-rate 10 --num-prompts 16 --ignore-eos
|
||||
```
|
||||
|
||||
Test configurations: prompt-heavy (8k/1k), decode-heavy (1k/8k), balanced (1k/1k).
|
||||
Vary `--num-prompts` across 1, 16, 32, 64, 128, 256, 512.
|
||||
|
||||
## References
|
||||
|
||||
- [ERNIE-4.5-21B-A3B-PT](https://huggingface.co/baidu/ERNIE-4.5-21B-A3B-PT)
|
||||
- [ERNIE-4.5-300B-A47B-PT](https://huggingface.co/baidu/ERNIE-4.5-300B-A47B-PT)
|
||||
- [vLLM multi-node deployment](https://docs.vllm.ai/en/latest/serving/parallelism_scaling.html)
|
||||
@@ -0,0 +1,163 @@
|
||||
meta:
|
||||
title: "ERNIE-4.5-VL"
|
||||
slug: "ernie-4.5-vl"
|
||||
provider: "Ernie (Baidu)"
|
||||
description: "Baidu ERNIE 4.5 VL MoE vision-language models (28B-A3B, 424B-A47B) with heterogeneous text/vision experts"
|
||||
date_added: 2025-09-03
|
||||
date_updated: 2026-04-17
|
||||
tasks:
|
||||
- multimodal
|
||||
related_recipes:
|
||||
- baidu/ERNIE-4.5-21B-A3B-PT
|
||||
hardware:
|
||||
mi300x: verified
|
||||
mi325x: verified
|
||||
mi355x: verified
|
||||
|
||||
model:
|
||||
model_id: "baidu/ERNIE-4.5-VL-28B-A3B-PT"
|
||||
min_vllm_version: "0.11.0"
|
||||
architecture: moe
|
||||
parameter_count: "28B"
|
||||
active_parameters: "3B"
|
||||
context_length: 131072
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
text_only:
|
||||
description: "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
|
||||
args:
|
||||
- "--language-model-only"
|
||||
encoder_parallel:
|
||||
description: "Run the vision encoder in data-parallel mode — avoids TP comm overhead on the small encoder. Mutually exclusive with text_only."
|
||||
args:
|
||||
- "--mm-encoder-tp-mode"
|
||||
- "data"
|
||||
|
||||
opt_in_features:
|
||||
- text_only
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 67
|
||||
description: "BF16 weights; fits on 1x80GB GPU (28B VL variant)"
|
||||
|
||||
"424b":
|
||||
model_id: "baidu/ERNIE-4.5-VL-424B-A47B-PT"
|
||||
precision: bf16
|
||||
vram_minimum_gb: 1120
|
||||
description: "424B total / 47B active; 8x140GB BF16 or 16x80GB BF16"
|
||||
extra_args:
|
||||
- "--trust-remote-code"
|
||||
- "--tensor-parallel-size"
|
||||
- "8"
|
||||
|
||||
"424b_fp8":
|
||||
model_id: "baidu/ERNIE-4.5-VL-424B-A47B-PT"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 640
|
||||
description: "424B with FP8 online quantization + CPU offload for 8x80GB testing"
|
||||
extra_args:
|
||||
- "--trust-remote-code"
|
||||
- "--tensor-parallel-size"
|
||||
- "8"
|
||||
- "--quantization"
|
||||
- "fp8"
|
||||
- "--cpu-offload-gb"
|
||||
- "50"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
- multi_node_tep
|
||||
- multi_node_dep
|
||||
|
||||
kv_offload_support:
|
||||
offloading_cpu: verified
|
||||
offloading_fs: verified
|
||||
|
||||
hardware_overrides: {}
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
ERNIE 4.5 VL is Baidu's multimodal MoE model with heterogeneous experts (separate text
|
||||
and vision experts). Because of the heterogeneous architecture, **torch.compile and CUDA
|
||||
graphs are not supported**.
|
||||
|
||||
- `baidu/ERNIE-4.5-VL-28B-A3B-PT` — 28B total / 3B active (1x80GB)
|
||||
- `baidu/ERNIE-4.5-VL-424B-A47B-PT` — 424B total / 47B active (8x140GB BF16, 8x80GB FP8+offload, or 16x80GB BF16)
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- vLLM: support added to main branch recently; install latest
|
||||
- Hardware depends on variant
|
||||
|
||||
### Install vLLM (CUDA)
|
||||
|
||||
```bash
|
||||
uv venv --python 3.12 --seed
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend auto
|
||||
```
|
||||
|
||||
### Install vLLM (AMD ROCm MI300X/MI325X/MI355X)
|
||||
|
||||
```bash
|
||||
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
|
||||
```
|
||||
|
||||
## Launch commands
|
||||
|
||||
28B on 1x80GB:
|
||||
|
||||
```bash
|
||||
vllm serve baidu/ERNIE-4.5-VL-28B-A3B-PT --trust-remote-code
|
||||
```
|
||||
|
||||
424B BF16 on 8x140GB:
|
||||
|
||||
```bash
|
||||
vllm serve baidu/ERNIE-4.5-VL-424B-A47B-PT \
|
||||
--trust-remote-code \
|
||||
--tensor-parallel-size 8
|
||||
```
|
||||
|
||||
424B with FP8 + CPU offload on 8x80GB (testing only):
|
||||
|
||||
```bash
|
||||
vllm serve baidu/ERNIE-4.5-VL-424B-A47B-PT \
|
||||
--trust-remote-code \
|
||||
--tensor-parallel-size 8 \
|
||||
--quantization fp8 \
|
||||
--cpu-offload-gb 50
|
||||
```
|
||||
|
||||
28B on AMD MI300X+:
|
||||
|
||||
```bash
|
||||
VLLM_ROCM_USE_AITER=1 SAFETENSORS_FAST_GPU=1 \
|
||||
vllm serve baidu/ERNIE-4.5-VL-28B-A3B-PT \
|
||||
--tensor-parallel-size 4 \
|
||||
--gpu-memory-utilization 0.9 \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
## Benchmarking
|
||||
|
||||
```bash
|
||||
vllm bench serve \
|
||||
--model baidu/ERNIE-4.5-VL-28B-A3B-PT \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 --random-output-len 1000 \
|
||||
--request-rate 10 --num-prompts 16 --ignore-eos --trust-remote-code
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [ERNIE-4.5-VL-28B-A3B-PT](https://huggingface.co/baidu/ERNIE-4.5-VL-28B-A3B-PT)
|
||||
- [ERNIE-4.5-VL-424B-A47B-PT](https://huggingface.co/baidu/ERNIE-4.5-VL-424B-A47B-PT)
|
||||
@@ -0,0 +1,147 @@
|
||||
meta:
|
||||
title: "Unlimited-OCR"
|
||||
slug: "unlimited-ocr"
|
||||
provider: "Ernie (Baidu)"
|
||||
description: "Baidu's state-of-the-art document-parsing model with Reference Sliding Window Attention (R-SWA), optimized for full-page OCR and markdown generation."
|
||||
date_added: 2026-06-28
|
||||
date_updated: 2026-06-28
|
||||
tasks:
|
||||
- multimodal
|
||||
performance_headline: "One-shot long-horizon document parsing to markdown"
|
||||
related_recipes:
|
||||
- "deepseek-ai/DeepSeek-OCR"
|
||||
|
||||
model:
|
||||
model_id: "baidu/Unlimited-OCR"
|
||||
min_vllm_version: "0.25.0"
|
||||
docker_image: "vllm/vllm-openai:unlimited-ocr"
|
||||
install:
|
||||
pip:
|
||||
note: "Unlimited-OCR (UnlimitedOCRForCausalLM) is supported since vLLM v0.25.0."
|
||||
architecture: dense
|
||||
parameter_count: "3B"
|
||||
active_parameters: "3B"
|
||||
context_length: 32768
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
- "--logits_processors"
|
||||
- "vllm.model_executor.models.unlimited_ocr:NGramPerReqLogitsProcessor"
|
||||
- "--no-enable-prefix-caching"
|
||||
- "--mm-processor-cache-gb"
|
||||
- "0"
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
encoder_parallel:
|
||||
description: "Run the vision encoder in data-parallel mode — avoids TP comm overhead on the small encoder."
|
||||
args:
|
||||
- "--mm-encoder-tp-mode"
|
||||
- "data"
|
||||
|
||||
opt_in_features:
|
||||
- encoder_parallel
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 8
|
||||
description: "Full precision BF16 (~3B params)"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
|
||||
hardware_overrides: {}
|
||||
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
[Unlimited-OCR](https://huggingface.co/baidu/Unlimited-OCR) is Baidu's
|
||||
DeepSeek-OCR-lineage document-parsing model. It shares the DeepSeek-OCR
|
||||
*gundam* vision stack (SAM-ViT-B + CLIP-L DeepEncoder, `base_size=1024` /
|
||||
`image_size=640` / crop) and ships with the same n-gram logits processor for
|
||||
high-quality, repetition-free document → markdown generation.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Hardware**: a single GPU with >=8 GB VRAM is enough for BF16 inference.
|
||||
- **vLLM**: served from the dedicated release image (the architecture is not yet
|
||||
in a stable pip wheel). Pull the image shown in the Install → Docker tab.
|
||||
|
||||
## Serving recipe (required)
|
||||
|
||||
The model has **no chat template** and is trained for a specific prompt/decode
|
||||
recipe — without it the model returns empty output. The required pieces:
|
||||
|
||||
- register the no-repeat-ngram logits processor on the server:
|
||||
`--logits_processors vllm.model_executor.models.unlimited_ocr:NGramPerReqLogitsProcessor`
|
||||
- prompt text must begin with a literal `<image>` (e.g. `<image>document parsing.`)
|
||||
- `skip_special_tokens=False`
|
||||
- pass the processor args per request: `ngram_size=35`, `window_size=128`
|
||||
(use `window_size=1024` for multi-page / PDF input)
|
||||
|
||||
## Client Usage
|
||||
|
||||
### Online OCR serving
|
||||
|
||||
```bash
|
||||
docker run --rm --gpus all --network host --ipc host \
|
||||
vllm/vllm-openai:unlimited-ocr \
|
||||
baidu/Unlimited-OCR \
|
||||
--trust-remote-code \
|
||||
--logits_processors vllm.model_executor.models.unlimited_ocr:NGramPerReqLogitsProcessor \
|
||||
--no-enable-prefix-caching \
|
||||
--mm-processor-cache-gb 0
|
||||
```
|
||||
|
||||
```python
|
||||
import time
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600)
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "text", "text": "<image>document parsing."},
|
||||
{"type": "image_url", "image_url": {"url": "https://huggingface.co/baidu/Unlimited-OCR/resolve/main/assets/baidu.png"}},
|
||||
],
|
||||
}
|
||||
]
|
||||
|
||||
start = time.time()
|
||||
response = client.chat.completions.create(
|
||||
model="baidu/Unlimited-OCR",
|
||||
messages=messages,
|
||||
max_tokens=8192,
|
||||
temperature=0.0,
|
||||
extra_body={
|
||||
"skip_special_tokens": False,
|
||||
"vllm_xargs": {"ngram_size": 35, "window_size": 128},
|
||||
},
|
||||
)
|
||||
print(f"Response costs: {time.time() - start:.2f}s")
|
||||
print(f"Generated text: {response.choices[0].message.content}")
|
||||
```
|
||||
|
||||
The raw output carries `<|ref|>…<|/ref|>` / `<|det|>…<|/det|>` grounding tokens;
|
||||
unwrap the `<|ref|>` text and drop the `<|det|>` coordinate boxes to get clean markdown.
|
||||
|
||||
## Troubleshooting / Configuration Tips
|
||||
|
||||
- **The custom logits processor is required** — without it long documents loop on
|
||||
`<|det|>` coordinate tokens.
|
||||
- Empty output almost always means the prompt is missing the literal `<image>`
|
||||
prefix or `skip_special_tokens` was left `True`.
|
||||
- OCR tasks don't benefit from prefix caching / image reuse, so prefix caching and
|
||||
the mm-processor cache are disabled in the recipe.
|
||||
- Single-image input uses gundam (crop) mode; multi-image requests automatically
|
||||
fall back to non-crop (base) mode — use `window_size=1024` for those.
|
||||
|
||||
## References
|
||||
|
||||
- [Unlimited-OCR on Hugging Face](https://huggingface.co/baidu/Unlimited-OCR)
|
||||
- [vLLM multimodal inputs guide](https://docs.vllm.ai/en/latest/features/multimodal_inputs.html#offline-inference)
|
||||
@@ -0,0 +1,235 @@
|
||||
meta:
|
||||
title: "Param-2-17B-A2.4B-Thinking"
|
||||
slug: "param2-17b-a2-4b-thinking"
|
||||
provider: "BharatGen"
|
||||
description: "BharatGen's hybrid-MoE reasoning model for English, Hindi and 21 Indian languages, with thinking traces and Hermes-style tool calling"
|
||||
date_added: 2026-09-20
|
||||
date_updated: 2026-09-20
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "17B total / 2.4B active parameters; the full BF16 checkpoint serves at TP1 on a single H100 or a single 48 GB L40S"
|
||||
related_recipes: []
|
||||
default_hardware: h100
|
||||
hardware:
|
||||
h100: verified
|
||||
l40s: verified
|
||||
|
||||
model:
|
||||
model_id: "bharatgenai/Param2-17B-A2.4B-Thinking"
|
||||
min_vllm_version: "0.20.0"
|
||||
architecture: moe
|
||||
parameter_count: "17B"
|
||||
active_parameters: "2.4B"
|
||||
context_length: 32768
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
- "--async-scheduling"
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Parse the model's <tool_call> JSON blocks into OpenAI tool calls with the Hermes parser"
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "hermes"
|
||||
reasoning:
|
||||
description: "Split the model's <think> traces into the reasoning field with the Qwen3 parser"
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "qwen3"
|
||||
|
||||
opt_in_features: []
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 41
|
||||
tp: 1
|
||||
description: "Official BF16 checkpoint; approximately 34.3 GB of repository weights, served with TP1"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
|
||||
hardware_overrides: {}
|
||||
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
tp: 1
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
`bharatgenai/Param2-17B-A2.4B-Thinking` is BharatGen's early post-training
|
||||
checkpoint of Param-2-17B, a hybrid Mixture-of-Experts model with **17B total
|
||||
parameters** and **2.4B activated per token**. Each sparse block routes to 6
|
||||
of 64 experts and additionally keeps **2 shared experts** always active; the
|
||||
first layer is dense (`first_k_dense_replace: 1`) before the MoE stack begins.
|
||||
The shared-expert design is aimed at stable cross-lingual representations for
|
||||
Indian languages.
|
||||
|
||||
Beyond English and Hindi it covers 21 scheduled Indian languages — Assamese,
|
||||
Bengali, Bodo, Dogri, Gujarati, Kannada, Konkani, Kashmiri, Maithili,
|
||||
Malayalam, Manipuri, Marathi, Nepali, Oriya, Punjabi, Sanskrit, Santali,
|
||||
Sindhi, Tamil, Telugu and Urdu — and was pretrained on roughly 22.6T tokens.
|
||||
|
||||
The checkpoint is a **thinking** model: its chat template always opens an
|
||||
assistant turn with a `<think>` block, and tool calls are emitted as
|
||||
Hermes-style `<tool_call>{"name": ..., "arguments": ...}</tool_call>` JSON.
|
||||
That is why the two Features above map onto the `qwen3` reasoning parser and
|
||||
the `hermes` tool-call parser.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **vLLM:** 0.20.0 or newer. `Param2MoEForCausalLM` was registered upstream in
|
||||
[PR #38000](https://github.com/vllm-project/vllm/pull/38000) and the
|
||||
tensor-parallel attention-head fix landed in
|
||||
[PR #39707](https://github.com/vllm-project/vllm/pull/39707); both ship in
|
||||
v0.20.0. Nothing extra needs to be installed.
|
||||
- **`--trust-remote-code` is required.** vLLM implements the model natively
|
||||
(`vllm/model_executor/models/param2moe.py`), but `model_type: param2moe` is
|
||||
not registered with Transformers, so the config still loads from the repo's
|
||||
`configuration_param2moe.py`.
|
||||
- **Hardware:** one GPU is enough. The BF16 weights are ~34.3 GB, so an H100
|
||||
80GB leaves ample room for the 32,768-token KV cache. A single 48 GB
|
||||
**L40S** also runs the same configuration: at the default
|
||||
`--gpu-memory-utilization 0.9` roughly 8–9 GB remains for KV, and this
|
||||
model's KV cache is small (21 layers × 8 KV heads × 64 head dim ≈ 42 KB per
|
||||
token), so about 6 concurrent sequences fit at the full 32,768-token
|
||||
window — more if you cap `--max-model-len`.
|
||||
- **License:** the checkpoint is released under the BharatGen
|
||||
**Non-Commercial** license — review it before deploying.
|
||||
|
||||
## Launching the Server
|
||||
|
||||
This is the exact configuration verified on both H100 and L40S:
|
||||
|
||||
```bash
|
||||
vllm serve bharatgenai/Param2-17B-A2.4B-Thinking \
|
||||
--trust-remote-code \
|
||||
--async-scheduling \
|
||||
--tensor-parallel-size 1 \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser hermes \
|
||||
--reasoning-parser qwen3
|
||||
```
|
||||
|
||||
To pin the server to one specific GPU on a multi-GPU node, prefix the command
|
||||
with `CUDA_VISIBLE_DEVICES=<id>`; add `--port <port>` if 8000 is taken.
|
||||
|
||||
`--async-scheduling` overlaps scheduling with the model forward pass and cuts
|
||||
host overhead between decode steps. It is not required — drop it if you hit an
|
||||
incompatibility with another feature you enable.
|
||||
|
||||
TP1 is the recommended layout: the model fits on one GPU and a 2.4B
|
||||
active-parameter forward pass gains little from sharding. To raise aggregate
|
||||
throughput, run several independent TP1 replicas behind a load balancer rather
|
||||
than increasing TP — especially on L40S, which has no NVLink, so cross-GPU TP
|
||||
would fall back to PCIe.
|
||||
|
||||
## Client Usage
|
||||
|
||||
Reasoning traces arrive in `message.reasoning` once `--reasoning-parser qwen3`
|
||||
is active:
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||||
response = client.chat.completions.create(
|
||||
model="bharatgenai/Param2-17B-A2.4B-Thinking",
|
||||
messages=[
|
||||
{"role": "user", "content": "एक कक्षा में 30 छात्र हैं। पूरी कक्षा का औसत 70 है। शीर्ष 10 का औसत 85 और निचले 10 का औसत 50 है। मध्य 10 छात्रों का औसत क्या है?"}
|
||||
],
|
||||
temperature=0,
|
||||
)
|
||||
msg = response.choices[0].message
|
||||
print(msg.reasoning)
|
||||
print(msg.content)
|
||||
```
|
||||
|
||||
The model card recommends **deterministic decoding** (`temperature=0`) for
|
||||
evaluations, structured tasks and production use; sampling defaults of
|
||||
`temperature=0.7`, `top_p=0.9`, `top_k=50` are suggested for open-ended
|
||||
generation. The card also notes the model is strong at structured output
|
||||
(Markdown, tables, JSON), and that clear task/format/constraint instructions
|
||||
in the prompt matter noticeably.
|
||||
|
||||
### Skipping the thinking block
|
||||
|
||||
The chat template accepts `enable_thinking`. Setting it to `false` prefills an
|
||||
empty `<think></think>` block so the model answers directly:
|
||||
|
||||
```python
|
||||
response = client.chat.completions.create(
|
||||
model="bharatgenai/Param2-17B-A2.4B-Thinking",
|
||||
messages=[{"role": "user", "content": "Name the capital of Maharashtra."}],
|
||||
extra_body={"chat_template_kwargs": {"enable_thinking": False}},
|
||||
)
|
||||
```
|
||||
|
||||
### Tool calling
|
||||
|
||||
With the Tool Calling feature on, pass `tools` as usual and read
|
||||
`message.tool_calls`:
|
||||
|
||||
```python
|
||||
response = client.chat.completions.create(
|
||||
model="bharatgenai/Param2-17B-A2.4B-Thinking",
|
||||
messages=[{"role": "user", "content": "What is the weather in Pune?"}],
|
||||
tools=[{
|
||||
"type": "function",
|
||||
"function": {
|
||||
"name": "get_weather",
|
||||
"description": "Get the current weather for a city",
|
||||
"parameters": {
|
||||
"type": "object",
|
||||
"properties": {"city": {"type": "string"}},
|
||||
"required": ["city"],
|
||||
},
|
||||
},
|
||||
}],
|
||||
tool_choice="auto",
|
||||
)
|
||||
print(response.choices[0].message.tool_calls)
|
||||
```
|
||||
|
||||
## Benchmarking
|
||||
|
||||
```bash
|
||||
vllm bench serve \
|
||||
--model bharatgenai/Param2-17B-A2.4B-Thinking \
|
||||
--dataset-name random \
|
||||
--random-input-len 2048 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 200
|
||||
```
|
||||
|
||||
Because thinking traces count toward the output budget, keep
|
||||
`--random-output-len` generous when benchmarking with reasoning enabled, or
|
||||
benchmark with `enable_thinking: false` to measure the non-reasoning path.
|
||||
|
||||
## Notes and Limitations
|
||||
|
||||
- **Context:** the native window is 32,768 tokens. The model card states a
|
||||
128k-context variant is planned but not yet released, so do not raise
|
||||
`--max-model-len` past 32,768 on this checkpoint.
|
||||
- **No MTP head:** `num_nextn_predict_layers: 0`, so there is no
|
||||
speculative-decoding draft shipped with this checkpoint.
|
||||
- **Early post-training checkpoint:** BharatGen describes the release as an
|
||||
early post-training snapshot intended for downstream use and further
|
||||
fine-tuning.
|
||||
- **Verification scope:** the `verified` H100 and L40S badges cover the BF16
|
||||
checkpoint at TP1 with the launch command above. Other GPUs are untested
|
||||
rather than known-bad.
|
||||
- **L40S headroom:** at 48 GB the weights alone take ~34.3 GB, so KV cache is
|
||||
the binding constraint rather than weight capacity. Lower
|
||||
`--max-model-len` if you need more concurrency than the full context window
|
||||
allows.
|
||||
|
||||
## References
|
||||
|
||||
- [Param2-17B-A2.4B-Thinking on Hugging Face](https://huggingface.co/bharatgenai/Param2-17B-A2.4B-Thinking)
|
||||
- [BharatGen on Hugging Face](https://huggingface.co/bharatgenai)
|
||||
- [vLLM support PR #38000 — Add support for BharatGen's Param2MoE model](https://github.com/vllm-project/vllm/pull/38000)
|
||||
- [vLLM PR #39707 — Fix attention-head mismatch under tensor parallelism for param2moe](https://github.com/vllm-project/vllm/pull/39707)
|
||||
- [vLLM supported models — `Param2MoEForCausalLM`](https://docs.vllm.ai/en/latest/models/supported_models.html)
|
||||
@@ -0,0 +1,178 @@
|
||||
meta:
|
||||
title: "DeepSeek-OCR-2"
|
||||
slug: "deepseek-ocr-2"
|
||||
provider: "DeepSeek"
|
||||
description: "Next-generation DeepSeek OCR model with improved document-to-markdown grounding and optical context compression."
|
||||
date_added: 2026-03-28
|
||||
date_updated: 2026-04-17
|
||||
tasks:
|
||||
- multimodal
|
||||
performance_headline: "Improved grounding and markdown conversion over DeepSeek-OCR"
|
||||
related_recipes:
|
||||
- "deepseek-ai/DeepSeek-OCR"
|
||||
|
||||
model:
|
||||
model_id: "deepseek-ai/DeepSeek-OCR-2"
|
||||
min_vllm_version: "0.16.0"
|
||||
architecture: dense
|
||||
parameter_count: "3B"
|
||||
active_parameters: "3B"
|
||||
context_length: 8192
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
- "--logits_processors"
|
||||
- "vllm.model_executor.models.deepseek_ocr:NGramPerReqLogitsProcessor"
|
||||
- "--no-enable-prefix-caching"
|
||||
- "--mm-processor-cache-gb"
|
||||
- "0"
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
text_only:
|
||||
description: "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
|
||||
args:
|
||||
- "--language-model-only"
|
||||
encoder_parallel:
|
||||
description: "Run the vision encoder in data-parallel mode — avoids TP comm overhead on the small encoder. Mutually exclusive with text_only."
|
||||
args:
|
||||
- "--mm-encoder-tp-mode"
|
||||
- "data"
|
||||
|
||||
opt_in_features:
|
||||
- text_only
|
||||
- encoder_parallel
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 7
|
||||
description: "Full precision BF16 (~3.4B params)"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
|
||||
hardware_overrides: {}
|
||||
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
DeepSeek-OCR-2 is a frontier OCR model exploring optical context compression for LLMs.
|
||||
It iterates on DeepSeek-OCR with better grounding and markdown conversion, and supports
|
||||
prompts like `<image>\n<|grounding|>Convert the document to markdown.` for richer
|
||||
document parsing.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Hardware**: Single GPU with >=8 GB VRAM is typically sufficient for BF16 inference.
|
||||
- **vLLM**: Current stable release (tested with `uv pip install -U vllm --torch-backend auto`).
|
||||
- **Python**: 3.10+
|
||||
|
||||
Install vLLM:
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend auto
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
### Offline OCR (Python)
|
||||
|
||||
```python
|
||||
from vllm import LLM, SamplingParams
|
||||
from vllm.model_executor.models.deepseek_ocr import NGramPerReqLogitsProcessor
|
||||
from PIL import Image
|
||||
|
||||
llm = LLM(
|
||||
model="deepseek-ai/DeepSeek-OCR-2",
|
||||
enable_prefix_caching=False,
|
||||
mm_processor_cache_gb=0,
|
||||
logits_processors=[NGramPerReqLogitsProcessor],
|
||||
)
|
||||
|
||||
image_1 = Image.open("path/to/your/image_1.png").convert("RGB")
|
||||
image_2 = Image.open("path/to/your/image_2.png").convert("RGB")
|
||||
# prompt = "<image>\nFree OCR. "
|
||||
prompt = "<image>\n<|grounding|>Convert the document to markdown. "
|
||||
|
||||
model_input = [
|
||||
{"prompt": prompt, "multi_modal_data": {"image": image_1}},
|
||||
{"prompt": prompt, "multi_modal_data": {"image": image_2}},
|
||||
]
|
||||
|
||||
sampling_param = SamplingParams(
|
||||
temperature=0.0,
|
||||
max_tokens=8192,
|
||||
extra_args=dict(
|
||||
ngram_size=30,
|
||||
window_size=90,
|
||||
whitelist_token_ids={128821, 128822}, # <td>, </td>
|
||||
),
|
||||
skip_special_tokens=False,
|
||||
)
|
||||
|
||||
for output in llm.generate(model_input, sampling_param):
|
||||
print(output.outputs[0].text)
|
||||
```
|
||||
|
||||
### Online OCR serving
|
||||
|
||||
```bash
|
||||
vllm serve deepseek-ai/DeepSeek-OCR-2 \
|
||||
--logits_processors vllm.model_executor.models.deepseek_ocr:NGramPerReqLogitsProcessor \
|
||||
--no-enable-prefix-caching \
|
||||
--mm-processor-cache-gb 0
|
||||
```
|
||||
|
||||
```python
|
||||
import time
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600)
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "image_url", "image_url": {"url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"}},
|
||||
{"type": "text", "text": "Free OCR."},
|
||||
],
|
||||
}
|
||||
]
|
||||
|
||||
start = time.time()
|
||||
response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-OCR-2",
|
||||
messages=messages,
|
||||
max_tokens=2048,
|
||||
temperature=0.0,
|
||||
extra_body={
|
||||
"skip_special_tokens": False,
|
||||
"vllm_xargs": {
|
||||
"ngram_size": 30,
|
||||
"window_size": 90,
|
||||
"whitelist_token_ids": [128821, 128822],
|
||||
},
|
||||
},
|
||||
)
|
||||
print(f"Response costs: {time.time() - start:.2f}s")
|
||||
print(f"Generated text: {response.choices[0].message.content}")
|
||||
```
|
||||
|
||||
## Troubleshooting / Configuration Tips
|
||||
|
||||
- **Use the custom logits processor** along with the model for optimal OCR and markdown
|
||||
generation performance.
|
||||
- Unlike multi-turn chat, OCR tasks do not typically benefit from prefix caching or image
|
||||
reuse, so disable these features to avoid unnecessary hashing and caching overhead.
|
||||
- DeepSeek-OCR-2 works better with plain prompts than instruction formats. See the
|
||||
[official main prompts](https://huggingface.co/deepseek-ai/DeepSeek-OCR-2#main-prompts).
|
||||
- Depending on your hardware, adjust `max_num_batched_tokens` for better throughput.
|
||||
|
||||
## References
|
||||
|
||||
- [DeepSeek-OCR-2 on Hugging Face](https://huggingface.co/deepseek-ai/DeepSeek-OCR-2)
|
||||
- [vLLM multimodal inputs guide](https://docs.vllm.ai/en/latest/features/multimodal_inputs.html#offline-inference)
|
||||
@@ -0,0 +1,226 @@
|
||||
meta:
|
||||
title: "DeepSeek-OCR"
|
||||
slug: "deepseek-ocr"
|
||||
provider: "DeepSeek"
|
||||
description: "Frontier OCR model exploring optical context compression for LLMs, optimized for document parsing and markdown generation."
|
||||
date_added: 2025-10-22
|
||||
date_updated: 2026-04-17
|
||||
tasks:
|
||||
- multimodal
|
||||
performance_headline: "Optical context compression for efficient OCR and document understanding"
|
||||
related_recipes:
|
||||
- "deepseek-ai/DeepSeek-OCR-2"
|
||||
hardware:
|
||||
mi300x: verified
|
||||
mi325x: verified
|
||||
mi355x: verified
|
||||
arc_pro_b60: verified
|
||||
arc_pro_b70: verified
|
||||
|
||||
model:
|
||||
model_id: "deepseek-ai/DeepSeek-OCR"
|
||||
min_vllm_version: "0.12.0"
|
||||
architecture: dense
|
||||
parameter_count: "3B"
|
||||
active_parameters: "3B"
|
||||
context_length: 8192
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
- "--logits-processors"
|
||||
- "vllm.model_executor.models.deepseek_ocr:NGramPerReqLogitsProcessor"
|
||||
- "--mm-processor-cache-gb"
|
||||
- "0"
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
text_only:
|
||||
description: "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
|
||||
args:
|
||||
- "--language-model-only"
|
||||
encoder_parallel:
|
||||
description: "Run the vision encoder in data-parallel mode — avoids TP comm overhead on the small encoder. Mutually exclusive with text_only."
|
||||
args:
|
||||
- "--mm-encoder-tp-mode"
|
||||
- "data"
|
||||
|
||||
opt_in_features:
|
||||
- text_only
|
||||
- encoder_parallel
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: bf16
|
||||
vram_minimum_gb: 7
|
||||
description: "Full precision BF16 (~3.3B params)"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- multi_node_tp
|
||||
|
||||
hardware_overrides:
|
||||
amd:
|
||||
extra_args: []
|
||||
extra_env:
|
||||
VLLM_ROCM_USE_AITER: "1"
|
||||
SAFETENSORS_FAST_GPU: "1"
|
||||
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
DeepSeek-OCR is a frontier OCR model exploring optical context compression for LLMs.
|
||||
It is optimized for document parsing, free-form OCR, and markdown generation from images,
|
||||
and ships with a custom n-gram logits processor for optimal quality.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Hardware**: Single GPU with >=8 GB VRAM (CUDA, MI300X / MI325X / MI355X, or Intel XPU B60 / B70)
|
||||
- **vLLM**: Current stable release
|
||||
- **Python**: 3.10+ (CUDA) or 3.12 (ROCm)
|
||||
|
||||
### CUDA
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend auto
|
||||
```
|
||||
|
||||
### ROCm (MI300X, MI325X, MI355X)
|
||||
Requires Python 3.12, ROCm 7.2.1, and glibc >= 2.35. Use the Docker flow from the
|
||||
[vLLM installation guide](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/#pre-built-images)
|
||||
if your host environment does not meet those requirements.
|
||||
```bash
|
||||
uv venv --python 3.12
|
||||
source .venv/bin/activate
|
||||
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
|
||||
```
|
||||
|
||||
### Intel XPU (B60 / B70)
|
||||
```bash
|
||||
docker pull vllm/vllm-openai-xpu:latest
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
### Offline OCR (Python)
|
||||
|
||||
```python
|
||||
from vllm import LLM, SamplingParams
|
||||
from vllm.model_executor.models.deepseek_ocr import NGramPerReqLogitsProcessor
|
||||
from PIL import Image
|
||||
|
||||
llm = LLM(
|
||||
model="deepseek-ai/DeepSeek-OCR",
|
||||
enable_prefix_caching=False,
|
||||
mm_processor_cache_gb=0,
|
||||
logits_processors=[NGramPerReqLogitsProcessor],
|
||||
)
|
||||
|
||||
image_1 = Image.open("path/to/your/image_1.png").convert("RGB")
|
||||
image_2 = Image.open("path/to/your/image_2.png").convert("RGB")
|
||||
prompt = "<image>\nFree OCR."
|
||||
|
||||
model_input = [
|
||||
{"prompt": prompt, "multi_modal_data": {"image": image_1}},
|
||||
{"prompt": prompt, "multi_modal_data": {"image": image_2}},
|
||||
]
|
||||
|
||||
sampling_param = SamplingParams(
|
||||
temperature=0.0,
|
||||
max_tokens=8192,
|
||||
extra_args=dict(
|
||||
ngram_size=30,
|
||||
window_size=90,
|
||||
whitelist_token_ids={128821, 128822}, # <td>, </td>
|
||||
),
|
||||
skip_special_tokens=False,
|
||||
)
|
||||
|
||||
for output in llm.generate(model_input, sampling_param):
|
||||
print(output.outputs[0].text)
|
||||
```
|
||||
|
||||
### Online OCR serving
|
||||
|
||||
**CUDA**
|
||||
```bash
|
||||
vllm serve deepseek-ai/DeepSeek-OCR \
|
||||
--logits-processors vllm.model_executor.models.deepseek_ocr:NGramPerReqLogitsProcessor \
|
||||
--mm-processor-cache-gb 0
|
||||
```
|
||||
|
||||
**ROCm (MI300X / MI325X / MI355X)**
|
||||
```bash
|
||||
export SAFETENSORS_FAST_GPU=1
|
||||
export VLLM_ROCM_USE_AITER=1
|
||||
|
||||
vllm serve deepseek-ai/DeepSeek-OCR \
|
||||
--logits-processors vllm.model_executor.models.deepseek_ocr:NGramPerReqLogitsProcessor \
|
||||
--mm-processor-cache-gb 0
|
||||
```
|
||||
|
||||
**Intel XPU (B60, B70)**
|
||||
|
||||
```bash
|
||||
docker run --device /dev/dri \
|
||||
-v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
|
||||
--privileged --ipc=host -p 8000:8000 \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
vllm/vllm-openai-xpu:latest \
|
||||
deepseek-ai/DeepSeek-OCR \
|
||||
--logits-processors vllm.model_executor.models.deepseek_ocr:NGramPerReqLogitsProcessor \
|
||||
--mm-processor-cache-gb 0
|
||||
```
|
||||
|
||||
Query the running server with the OpenAI client (same for CUDA / ROCm / XPU):
|
||||
|
||||
```python
|
||||
import time
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600)
|
||||
|
||||
messages = [
|
||||
{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "image_url", "image_url": {"url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"}},
|
||||
{"type": "text", "text": "Free OCR."},
|
||||
],
|
||||
}
|
||||
]
|
||||
|
||||
start = time.time()
|
||||
response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-OCR",
|
||||
messages=messages,
|
||||
max_tokens=2048,
|
||||
temperature=0.0,
|
||||
extra_body={
|
||||
"skip_special_tokens": False,
|
||||
"vllm_xargs": {
|
||||
"ngram_size": 30,
|
||||
"window_size": 90,
|
||||
"whitelist_token_ids": [128821, 128822],
|
||||
},
|
||||
},
|
||||
)
|
||||
print(f"Response costs: {time.time() - start:.2f}s")
|
||||
print(f"Generated text: {response.choices[0].message.content}")
|
||||
```
|
||||
|
||||
## Troubleshooting / Configuration Tips
|
||||
|
||||
- **Use the custom logits processor** along with the model for optimal OCR and markdown
|
||||
generation performance.
|
||||
- Unlike multi-turn chat, OCR tasks do not typically benefit from prefix caching or image
|
||||
reuse, so disable these features to avoid unnecessary hashing and caching overhead.
|
||||
- DeepSeek-OCR works better with plain prompts than instruction formats. See the
|
||||
[official example prompts](https://github.com/deepseek-ai/DeepSeek-OCR/blob/2ac6d64a00656693b79c4f759a5e62c1b78bbeb1/DeepSeek-OCR-master/DeepSeek-OCR-vllm/config.py#L27-L37).
|
||||
- Depending on your hardware, adjust `max_num_batched_tokens` for better throughput.
|
||||
|
||||
## References
|
||||
|
||||
- [DeepSeek-OCR on Hugging Face](https://huggingface.co/deepseek-ai/DeepSeek-OCR)
|
||||
- [vLLM multimodal inputs guide](https://docs.vllm.ai/en/latest/features/multimodal_inputs.html#offline-inference)
|
||||
@@ -0,0 +1,342 @@
|
||||
meta:
|
||||
title: "DeepSeek-R1"
|
||||
slug: "deepseek-r1"
|
||||
provider: "DeepSeek"
|
||||
description: "DeepSeek-R1 is a 671B-parameter MoE reasoning model built on the DeepSeek-V3 architecture, trained with large-scale reinforcement learning for strong chain-of-thought capabilities."
|
||||
date_added: 2025-08-01
|
||||
date_updated: 2026-10-02
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "Open-weights RL-trained reasoning model with native FP8 / FP4 variants"
|
||||
related_recipes:
|
||||
- "deepseek-ai/DeepSeek-V3"
|
||||
- "deepseek-ai/DeepSeek-V3.1"
|
||||
hardware:
|
||||
h200: verified
|
||||
b200: verified
|
||||
gb200: verified
|
||||
mi300x: verified
|
||||
mi325x: verified
|
||||
mi350x: verified
|
||||
mi355x: verified
|
||||
|
||||
model:
|
||||
model_id: "deepseek-ai/DeepSeek-R1"
|
||||
min_vllm_version: "0.12.0"
|
||||
default_frontend: rust
|
||||
architecture: moe
|
||||
parameter_count: "671B"
|
||||
active_parameters: "37B"
|
||||
context_length: 163840
|
||||
supports_dcp: true
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable DeepSeek-R1 tool calling with the deepseek_v3 tool-call parser."
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "deepseek_v3"
|
||||
- "--chat-template"
|
||||
- "examples/tool_chat_template_deepseekr1.jinja"
|
||||
reasoning:
|
||||
description: "Enable reasoning/thinking mode with the DeepSeek R1 reasoning parser."
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "deepseek_r1"
|
||||
|
||||
opt_in_features: []
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: fp8
|
||||
vram_minimum_gb: 805
|
||||
description: "Native FP8 weights on 8xH200 (recommended)"
|
||||
r1_0528:
|
||||
model_id: "deepseek-ai/DeepSeek-R1-0528"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 805
|
||||
description: "May 2025 DeepSeek-R1 refresh (DeepSeek-R1-0528)"
|
||||
nvfp4:
|
||||
model_id: "nvidia/DeepSeek-R1-0528-NVFP4-v2"
|
||||
precision: nvfp4
|
||||
vram_minimum_gb: 403
|
||||
description: "NVIDIA NVFP4 quantized weights for Blackwell GPUs"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
mxfp4:
|
||||
model_id: "amd/DeepSeek-R1-0528-MXFP4-MTP-MoEFP4"
|
||||
precision: mxfp4
|
||||
vram_minimum_gb: 403
|
||||
description: "AMD MXFP4 quantized weights for MI350X / MI355X (gfx950)"
|
||||
supported_hardware:
|
||||
- mi350x
|
||||
- mi355x
|
||||
tp: 8
|
||||
extra_args:
|
||||
- "--no-enable-expert-parallel"
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
extra_env:
|
||||
VLLM_ROCM_USE_SKINNY_GEMM: "0"
|
||||
|
||||
redhat_nvfp4:
|
||||
model_id: "RedHatAI/DeepSeek-R1-NVFP4-FP8-BLOCK"
|
||||
precision: nvfp4
|
||||
label: "NVFP4+FP8 Block (RedHat)"
|
||||
vram_minimum_gb: 444
|
||||
description: "RedHatAI NVFP4+FP8 block mixed-precision checkpoint"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- single_node_tep
|
||||
- single_node_dep
|
||||
- single_node_dpa_tp
|
||||
- multi_node_tp
|
||||
- multi_node_tp_pp
|
||||
- multi_node_dep
|
||||
- multi_node_tep
|
||||
- pd_cluster
|
||||
|
||||
hardware_overrides:
|
||||
nvidia:
|
||||
extra_args:
|
||||
- "--enable-expert-parallel"
|
||||
extra_env: {}
|
||||
amd:
|
||||
extra_args: []
|
||||
extra_env:
|
||||
VLLM_ROCM_USE_AITER: "1"
|
||||
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
hardware_overrides:
|
||||
mi355x:
|
||||
extra_env:
|
||||
VLLM_ROCM_QUICK_REDUCE_QUANTIZATION: "INT4"
|
||||
VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS: "1"
|
||||
single_node_dpa_tp:
|
||||
extra_env:
|
||||
VLLM_ENGINE_READY_TIMEOUT_S: "1800"
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
DeepSeek-R1 is a 671B-parameter Mixture-of-Experts reasoning model (37B activated per
|
||||
token) that shares its architecture with DeepSeek-V3, so the same launch recipes apply
|
||||
to both. DeepSeek publishes a refreshed checkpoint as `DeepSeek-R1-0528`, NVIDIA
|
||||
publishes an FP4 quantized variant (`nvidia/DeepSeek-R1-FP4`) that runs on Blackwell
|
||||
GPUs with fewer devices, and AMD publishes an MXFP4 quantized variant
|
||||
(`amd/DeepSeek-R1-0528-MXFP4-MTP-MoEFP4`) that runs on Instinct GPUs.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Hardware (FP8)**: 8x H200 (CUDA) or 8x MI300X / MI325X / MI355X (ROCm)
|
||||
- **Hardware (NVFP4)**: 4x B200 GPUs
|
||||
- **Hardware (MXFP4)**: 8x MI350X / MI355X (ROCm)
|
||||
- **vLLM**: Current stable release
|
||||
|
||||
### CUDA
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend auto
|
||||
```
|
||||
|
||||
### ROCm (MI300X, MI325X, MI355X)
|
||||
Requires Python 3.12, ROCm 7.2.1, and glibc >= 2.35.
|
||||
```bash
|
||||
uv venv --python 3.12
|
||||
source .venv/bin/activate
|
||||
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
### 8xH200 / 8xMI300X (FP8)
|
||||
|
||||
Tensor Parallel + Expert Parallel (TP8+EP) — CUDA:
|
||||
```bash
|
||||
vllm serve deepseek-ai/DeepSeek-R1-0528 \
|
||||
--trust-remote-code \
|
||||
--tensor-parallel-size 8 \
|
||||
--enable-expert-parallel
|
||||
```
|
||||
|
||||
Tensor Parallel + Expert Parallel (TP8+EP) — ROCm:
|
||||
```bash
|
||||
export VLLM_ROCM_USE_AITER=1
|
||||
|
||||
vllm serve deepseek-ai/DeepSeek-R1 \
|
||||
--trust-remote-code \
|
||||
--tensor-parallel-size 8 \
|
||||
--enable-expert-parallel
|
||||
```
|
||||
|
||||
Data Parallel + Expert Parallel (DP8+EP) — CUDA:
|
||||
```bash
|
||||
vllm serve deepseek-ai/DeepSeek-R1-0528 \
|
||||
--trust-remote-code \
|
||||
--data-parallel-size 8 \
|
||||
--enable-expert-parallel
|
||||
```
|
||||
|
||||
### 8xMI355X (FP8)
|
||||
|
||||
Tensor Parallel (TP8) — ROCm. Validated on MI355X only; INT4 quick reduce and
|
||||
fused shared experts are not enabled on other AMD GPUs or other strategies:
|
||||
```bash
|
||||
export VLLM_ROCM_USE_AITER=1
|
||||
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4
|
||||
export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1
|
||||
|
||||
vllm serve deepseek-ai/DeepSeek-R1 \
|
||||
--trust-remote-code \
|
||||
--tensor-parallel-size 8
|
||||
```
|
||||
|
||||
### 8xMI350X / 8xMI355X (DPA+TP)
|
||||
|
||||
Each GPU runs its own attention replica with a private KV cache, while the MoE
|
||||
experts stay tensor-sharded across all eight GPUs. Aggregate KV-cache
|
||||
capacity is roughly 8x that of a single TP8 replica, useful for workloads
|
||||
benefiting from larger KV capacities.
|
||||
|
||||
Engine (all eight ranks in one process, listening on 8100):
|
||||
```bash
|
||||
export VLLM_ROCM_USE_AITER=1
|
||||
export VLLM_ENGINE_READY_TIMEOUT_S=1800
|
||||
|
||||
vllm serve deepseek-ai/DeepSeek-R1 \
|
||||
--trust-remote-code \
|
||||
--port 8100 \
|
||||
--data-parallel-size 8 \
|
||||
--tensor-parallel-size 1
|
||||
```
|
||||
|
||||
Router (client-facing on 8000, spreads requests across the eight DP ranks):
|
||||
```bash
|
||||
uv pip install vllm-router
|
||||
|
||||
vllm-router \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000 \
|
||||
--policy consistent_hash \
|
||||
--intra-node-data-parallel-size 8 \
|
||||
--worker-urls http://localhost:8100 \
|
||||
--worker-startup-timeout-secs 1800
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- `--intra-node-data-parallel-size` must match the engine's
|
||||
`--data-parallel-size`. A mismatch silently under-uses ranks.
|
||||
- `consistent_hash` is the policy that makes this worthwhile for chat: it pins
|
||||
a conversation to the rank that already holds its prefix, so the larger
|
||||
aggregate cache actually gets hit. `round_robin` scatters turns of the same
|
||||
conversation across ranks and throws the prefix away.
|
||||
- Start the engine first, then the router; `--worker-startup-timeout-secs 1800`
|
||||
covers the engine's weight load either way.
|
||||
- Send client traffic to the router on port 8000, never to 8100 directly.
|
||||
|
||||
### 4xB200 (FP4)
|
||||
|
||||
On vLLM v0.28.0 and later, the FlashInfer MoE kernels are selected automatically
|
||||
for both FP4 and FP8 on Blackwell — no environment variables are needed. To pin
|
||||
the backend explicitly, pass `--moe-backend flashinfer_trtllm`.
|
||||
|
||||
Tensor Parallel + Expert Parallel (TP4+EP):
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=0,1,2,3 vllm serve nvidia/DeepSeek-R1-FP4 \
|
||||
--trust-remote-code \
|
||||
--tensor-parallel-size 4 \
|
||||
--enable-expert-parallel
|
||||
```
|
||||
|
||||
Data Parallel + Expert Parallel (DP4+EP):
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=0,1,2,3 vllm serve nvidia/DeepSeek-R1-FP4 \
|
||||
--trust-remote-code \
|
||||
--data-parallel-size 4 \
|
||||
--enable-expert-parallel
|
||||
```
|
||||
|
||||
### 8xMI350X / 8xMI355X (MXFP4)
|
||||
|
||||
Tensor Parallel + FP8 KV cache (TP8):
|
||||
```bash
|
||||
export VLLM_ROCM_USE_AITER=1
|
||||
export VLLM_ROCM_USE_SKINNY_GEMM=0
|
||||
|
||||
vllm serve amd/DeepSeek-R1-0528-MXFP4-MTP-MoEFP4 \
|
||||
--trust-remote-code \
|
||||
--tensor-parallel-size 8 \
|
||||
--kv-cache-dtype fp8
|
||||
```
|
||||
|
||||
## Benchmarking
|
||||
|
||||
For benchmarking, prefix caching is disabled by default in vLLM — no extra server
|
||||
flag is needed.
|
||||
|
||||
```bash
|
||||
# FP8 benchmark
|
||||
vllm bench serve \
|
||||
--model deepseek-ai/DeepSeek-R1-0528 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--request-rate 10000 \
|
||||
--num-prompts 16 \
|
||||
--ignore-eos
|
||||
```
|
||||
|
||||
```bash
|
||||
# NVFP4 benchmark
|
||||
vllm bench serve \
|
||||
--model nvidia/DeepSeek-R1-FP4 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--request-rate 10000 \
|
||||
--num-prompts 16 \
|
||||
--ignore-eos
|
||||
```
|
||||
|
||||
```bash
|
||||
# MXFP4 benchmark (MI350X / MI355X)
|
||||
vllm bench serve \
|
||||
--model amd/DeepSeek-R1-0528-MXFP4-MTP-MoEFP4 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--request-rate 10000 \
|
||||
--num-prompts 16 \
|
||||
--ignore-eos
|
||||
```
|
||||
|
||||
Test different workloads by adjusting input/output lengths:
|
||||
- Prompt-heavy: 8000 input / 1000 output
|
||||
- Decode-heavy: 1000 input / 8000 output
|
||||
- Balanced: 1000 input / 1000 output
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **Disaggregated Serving with Wide EP (Experimental GB200)**: See
|
||||
[vLLM issue #33583](https://github.com/vllm-project/vllm/issues/33583),
|
||||
[the vLLM blog post](https://blog.vllm.ai/2026/02/03/dsr1-gb200-part1.html), and
|
||||
[this reference fork](https://github.com/minosfuture/vllm/tree/pd_gb200_0114/runs/DS-R1/fp4)
|
||||
for GB200 disaggregated serving recipes.
|
||||
|
||||
## References
|
||||
|
||||
- [DeepSeek-R1 on Hugging Face](https://huggingface.co/deepseek-ai/DeepSeek-R1)
|
||||
- [DeepSeek-R1-0528 on Hugging Face](https://huggingface.co/deepseek-ai/DeepSeek-R1-0528)
|
||||
- [NVIDIA DeepSeek-R1-FP4](https://huggingface.co/nvidia/DeepSeek-R1-FP4)
|
||||
- [AMD DeepSeek-R1-0528-MXFP4-MTP-MoEFP4](https://huggingface.co/amd/DeepSeek-R1-0528-MXFP4-MTP-MoEFP4)
|
||||
- [vLLM Expert Parallelism docs](https://docs.vllm.ai/en/latest/serving/expert_parallel_deployment.html)
|
||||
- [vLLM Data Parallel deployment docs](https://docs.vllm.ai/en/latest/serving/data_parallel_deployment.html)
|
||||
@@ -0,0 +1,196 @@
|
||||
meta:
|
||||
title: "DeepSeek-V3.1"
|
||||
slug: "deepseek-v3.1"
|
||||
provider: "DeepSeek"
|
||||
description: "DeepSeek-V3.1 is a hybrid MoE model that supports dynamic switching between thinking and non-thinking modes, with tool calling and function execution."
|
||||
date_added: 2025-08-21
|
||||
date_updated: 2026-09-14
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "Hybrid thinking / non-thinking MoE with native FP8 and tool calling"
|
||||
related_recipes:
|
||||
- "deepseek-ai/DeepSeek-V3"
|
||||
- "deepseek-ai/DeepSeek-V3.2-Exp"
|
||||
hardware:
|
||||
h200: verified
|
||||
mi300x: verified
|
||||
mi325x: verified
|
||||
mi355x: verified
|
||||
|
||||
model:
|
||||
model_id: "deepseek-ai/DeepSeek-V3.1"
|
||||
min_vllm_version: "0.12.0"
|
||||
default_frontend: rust
|
||||
architecture: moe
|
||||
parameter_count: "671B"
|
||||
active_parameters: "37B"
|
||||
context_length: 163840
|
||||
supports_dcp: true
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
- "--enable-expert-parallel"
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable DeepSeek-V3.1 tool calling with the deepseek_v31 tool-call parser."
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "deepseek_v31"
|
||||
- "--chat-template"
|
||||
- "examples/tool_chat_template_deepseekv31.jinja"
|
||||
reasoning:
|
||||
description: "Dynamic thinking mode via chat_template_kwargs={'thinking': true|false}. No separate parser flag is required; the chat template emits <think>...</think> content."
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "deepseek_v3"
|
||||
|
||||
opt_in_features: []
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: fp8
|
||||
vram_minimum_gb: 805
|
||||
description: "Native FP8 weights on 8xH200 (or H20) with 141GB per GPU"
|
||||
nvfp4:
|
||||
model_id: "nvidia/DeepSeek-V3.1-NVFP4"
|
||||
precision: nvfp4
|
||||
vram_minimum_gb: 403
|
||||
description: "NVIDIA NVFP4 quantized weights for Blackwell GPUs"
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- single_node_tep
|
||||
- single_node_dep
|
||||
- multi_node_tp
|
||||
- multi_node_tp_pp
|
||||
- multi_node_dep
|
||||
- multi_node_tep
|
||||
- pd_cluster
|
||||
|
||||
hardware_overrides:
|
||||
amd:
|
||||
extra_args: []
|
||||
extra_env:
|
||||
VLLM_ROCM_USE_AITER: "1"
|
||||
VLLM_ROCM_USE_AITER_MOE: "1"
|
||||
SAFETENSORS_FAST_GPU: "1"
|
||||
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
DeepSeek-V3.1 is a hybrid MoE model that supports both thinking and non-thinking modes.
|
||||
You can dynamically switch between the two modes from the client by passing
|
||||
`extra_body={"chat_template_kwargs": {"thinking": True|False}}`.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Hardware**: 8x H200/H20 (CUDA) or 8x MI300X / MI325X / MI355X (ROCm)
|
||||
- **vLLM**: Current stable release
|
||||
|
||||
### CUDA
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend auto
|
||||
```
|
||||
|
||||
### ROCm (MI300X, MI325X, MI355X)
|
||||
Requires Python 3.12, ROCm 7.2.1, and glibc >= 2.35.
|
||||
```bash
|
||||
uv venv --python 3.12
|
||||
source .venv/bin/activate
|
||||
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
|
||||
```
|
||||
|
||||
## Launching DeepSeek-V3.1
|
||||
|
||||
### 8xH200 / 8xH20 (CUDA)
|
||||
|
||||
```bash
|
||||
vllm serve deepseek-ai/DeepSeek-V3.1 \
|
||||
--enable-expert-parallel \
|
||||
--tensor-parallel-size 8 \
|
||||
--served-model-name ds31
|
||||
```
|
||||
|
||||
### 8xMI300X / MI325X / MI355X (ROCm)
|
||||
|
||||
```bash
|
||||
export SAFETENSORS_FAST_GPU=1
|
||||
export VLLM_ROCM_USE_AITER=1
|
||||
export VLLM_ROCM_USE_AITER_MOE=1
|
||||
|
||||
vllm serve deepseek-ai/DeepSeek-V3.1 \
|
||||
--tensor-parallel-size 8 \
|
||||
--enable-expert-parallel
|
||||
```
|
||||
|
||||
### Function calling
|
||||
|
||||
vLLM supports user-defined tool calling for DeepSeek-V3.1. Add these flags when launching
|
||||
the server. The example chat template ships in the official container and can also be
|
||||
downloaded from the vLLM repo:
|
||||
[`tool_chat_template_deepseekv31.jinja`](https://github.com/vllm-project/vllm/blob/main/examples/tool_chat_template_deepseekv31.jinja).
|
||||
|
||||
```bash
|
||||
vllm serve ... \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser deepseek_v31 \
|
||||
--chat-template examples/tool_chat_template_deepseekv31.jinja
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
### OpenAI Python SDK
|
||||
|
||||
Control thinking mode via `extra_body={"chat_template_kwargs": {"thinking": False}}`
|
||||
(or `True` to enable thinking).
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
|
||||
model = client.models.list().data[0].id
|
||||
|
||||
messages = [
|
||||
{"role": "system", "content": "You are a helpful assistant"},
|
||||
{"role": "user", "content": "Who are you?"},
|
||||
{"role": "assistant", "content": "<think>Hmm</think>I am DeepSeek"},
|
||||
{"role": "user", "content": "9.11 and 9.8, which is greater?"},
|
||||
]
|
||||
response = client.chat.completions.create(
|
||||
model=model,
|
||||
messages=messages,
|
||||
extra_body={"chat_template_kwargs": {"thinking": False}},
|
||||
)
|
||||
print(response.choices[0].message.content)
|
||||
```
|
||||
|
||||
When `thinking=True`, output includes a `</think>` segment delimiting chain-of-thought;
|
||||
when `thinking=False`, the model produces a direct answer without the thinking segment.
|
||||
|
||||
### curl
|
||||
|
||||
```bash
|
||||
curl http://localhost:8000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "ds31",
|
||||
"messages": [
|
||||
{"role": "user", "content": "9.11 and 9.8, which is greater?"}
|
||||
],
|
||||
"chat_template_kwargs": {"thinking": true}
|
||||
}'
|
||||
```
|
||||
|
||||
## References
|
||||
|
||||
- [DeepSeek-V3.1 on Hugging Face](https://huggingface.co/deepseek-ai/DeepSeek-V3.1)
|
||||
- [vLLM tool chat template for DeepSeek-V3.1](https://github.com/vllm-project/vllm/blob/main/examples/tool_chat_template_deepseekv31.jinja)
|
||||
@@ -0,0 +1,183 @@
|
||||
meta:
|
||||
title: "DeepSeek-V3.2-Exp"
|
||||
slug: "deepseek-v3.2-exp"
|
||||
provider: "DeepSeek"
|
||||
description: "Experimental DeepSeek-V3.2 preview with sparse attention (MQA-like logits) and FP8 KV cache; architecture matches DeepSeek-V3.1 except for the sparse attention mechanism."
|
||||
date_added: 2025-09-29
|
||||
date_updated: 2026-09-14
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "Sparse attention MoE with FP8 KV cache and strong GSM8K score (~0.96)"
|
||||
related_recipes:
|
||||
- "deepseek-ai/DeepSeek-V3.1"
|
||||
- "deepseek-ai/DeepSeek-V3.2"
|
||||
hardware:
|
||||
h200: verified
|
||||
mi300x: verified
|
||||
mi325x: verified
|
||||
mi355x: verified
|
||||
|
||||
model:
|
||||
model_id: "deepseek-ai/DeepSeek-V3.2-Exp"
|
||||
min_vllm_version: "0.12.0"
|
||||
default_frontend: rust
|
||||
architecture: moe
|
||||
parameter_count: "671B"
|
||||
active_parameters: "37B"
|
||||
context_length: 163840
|
||||
supports_dcp: true
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
base_env: {}
|
||||
|
||||
dependencies:
|
||||
- note: "DeepGEMM pinned build for MQA logits (FP8 MoE kernels)"
|
||||
command: "uv pip install git+https://github.com/deepseek-ai/DeepGEMM.git@v2.1.1.post3 --no-build-isolation"
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable tool calling with DeepSeek V3.2 chat template support."
|
||||
args:
|
||||
- "--tokenizer-mode"
|
||||
- "deepseek_v32"
|
||||
- "--tool-call-parser"
|
||||
- "deepseek_v32"
|
||||
- "--enable-auto-tool-choice"
|
||||
reasoning:
|
||||
description: "Dynamic thinking mode via chat_template_kwargs, same as DeepSeek-V3.1."
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "deepseek_v3"
|
||||
|
||||
opt_in_features: []
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: fp8
|
||||
vram_minimum_gb: 805
|
||||
description: "Native FP8 weights on 8xH200 (or H20, or 8xB200) with FP8 KV cache default"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- single_node_tep
|
||||
- single_node_dep
|
||||
- multi_node_tp
|
||||
- multi_node_tp_pp
|
||||
- multi_node_dep
|
||||
- multi_node_tep
|
||||
|
||||
hardware_overrides:
|
||||
amd:
|
||||
# V3.2-Exp uses sparse attention → --block-size 1 + bfloat16 KV cache.
|
||||
extra_args:
|
||||
- "--block-size"
|
||||
- "1"
|
||||
- "--kv-cache-dtype"
|
||||
- "bfloat16"
|
||||
- "--max-num-batched-tokens"
|
||||
- "32768"
|
||||
extra_env:
|
||||
VLLM_ROCM_USE_AITER: "1"
|
||||
VLLM_ROCM_USE_AITER_MOE: "1"
|
||||
SAFETENSORS_FAST_GPU: "1"
|
||||
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
DeepSeek-V3.2-Exp is a sparse-attention MoE preview. Its main architecture is similar to
|
||||
DeepSeek-V3.1, with a sparse attention mechanism. This guide covers deployment on NVIDIA
|
||||
Hopper/Blackwell GPUs and AMD MI300X / MI325X / MI355X GPUs.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Hardware**: 8x H200/H20/B200 (CUDA) or 8x MI300X / MI325X / MI355X (ROCm)
|
||||
- **vLLM**: Current stable release
|
||||
- **DeepGEMM** (CUDA only): Required for MQA logits computation (and optionally for MoE)
|
||||
|
||||
### CUDA
|
||||
```bash
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend auto
|
||||
uv pip install git+https://github.com/deepseek-ai/DeepGEMM.git@v2.1.1.post3 --no-build-isolation
|
||||
```
|
||||
|
||||
### ROCm (MI300X, MI325X, MI355X)
|
||||
Requires Python 3.12, ROCm 7.2.1, and glibc >= 2.35. If your environment does not
|
||||
meet these requirements, use the Docker-based setup described in the
|
||||
[documentation](https://docs.vllm.ai/en/latest/getting_started/installation/gpu/#pre-built-images).
|
||||
```bash
|
||||
uv venv --python 3.12
|
||||
source .venv/bin/activate
|
||||
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
|
||||
```
|
||||
|
||||
Note: DeepGEMM is used both for MoE and for MQA logits computation. It is required for
|
||||
MQA logits on CUDA. To disable the MoE path only, set `VLLM_USE_DEEP_GEMM=0`. Some users
|
||||
report better performance with `VLLM_USE_DEEP_GEMM=0` (e.g. on H20), and this also skips
|
||||
the long warmup.
|
||||
|
||||
## Launching DeepSeek-V3.2-Exp
|
||||
|
||||
### 8xH200 / 8xH20 (CUDA)
|
||||
|
||||
Using the recommended EP/DP mode:
|
||||
```bash
|
||||
vllm serve deepseek-ai/DeepSeek-V3.2-Exp -dp 8 --enable-expert-parallel
|
||||
```
|
||||
|
||||
Using tensor parallel:
|
||||
```bash
|
||||
vllm serve deepseek-ai/DeepSeek-V3.2-Exp -tp 8
|
||||
```
|
||||
|
||||
### 8xB200 (CUDA)
|
||||
|
||||
Same as 8xH200 / 8xH20.
|
||||
|
||||
### 8xMI300X / MI325X / MI355X (ROCm)
|
||||
|
||||
```bash
|
||||
export SAFETENSORS_FAST_GPU=1
|
||||
export VLLM_ROCM_USE_AITER=1
|
||||
export VLLM_ROCM_USE_AITER_MOE=1
|
||||
|
||||
vllm serve deepseek-ai/DeepSeek-V3.2-Exp \
|
||||
--tensor-parallel-size 8 \
|
||||
--max-num-batched-tokens 32768 \
|
||||
--trust-remote-code \
|
||||
--kv-cache-dtype bfloat16 \
|
||||
--block-size 1
|
||||
```
|
||||
|
||||
## Accuracy Benchmarking
|
||||
|
||||
```bash
|
||||
lm-eval --model local-completions --tasks gsm8k \
|
||||
--model_args model=deepseek-ai/DeepSeek-V3.2-Exp,base_url=http://127.0.0.1:8000/v1/completions,num_concurrent=100,max_retries=3,tokenized_requests=False
|
||||
```
|
||||
|
||||
Reported GSM8K score: `0.9591` (5-shot) and `0.9538` (20-shot).
|
||||
|
||||
## Performance Tips
|
||||
|
||||
1. The kernels are mainly optimized for TP=1, so it is recommended to run this model
|
||||
under EP/DP mode (e.g. DP=8, EP=8, TP=1). If you hit any errors or hangs, try tensor
|
||||
parallel instead. Simple TP works and is more robust, but the performance is not optimal.
|
||||
2. The default config uses a custom `fp8` KV cache. You can also use `bfloat16` KV cache
|
||||
by specifying `kv_cache_dtype=bfloat16`. FP8 allows more tokens to be cached but incurs
|
||||
quantization/dequantization overhead. Use `bfloat16` for short requests and `fp8` for
|
||||
long requests.
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **`CUDA error (flashmla-src/csrc/smxx/mla_combine.cu:201): invalid configuration argument`**:
|
||||
This may be caused by too large a batch size. Try `--max-num-seqs 256` or smaller
|
||||
(default is 1024).
|
||||
- For thinking-mode toggling, refer to the DeepSeek-V3.1 recipe (`deepseek-ai/DeepSeek-V3.1`).
|
||||
|
||||
## References
|
||||
|
||||
- [DeepSeek-V3.2-Exp on Hugging Face](https://huggingface.co/deepseek-ai/DeepSeek-V3.2-Exp)
|
||||
- [End-to-end tutorial (Jupyter Notebook)](https://github.com/vllm-project/recipes/blob/main/DeepSeek/DeepSeek_v3_2_vLLM_getting_started_guide.ipynb)
|
||||
@@ -0,0 +1,177 @@
|
||||
meta:
|
||||
title: "DeepSeek-V3.2"
|
||||
slug: "deepseek-v3.2"
|
||||
provider: "DeepSeek"
|
||||
description: "DeepSeek V3.2 MoE model with MLA attention, sparse attention, and scalable RL for strong reasoning and agent capabilities."
|
||||
date_added: 2025-12-04
|
||||
date_updated: 2026-09-14
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "GPT-5-level reasoning with efficient MoE inference"
|
||||
related_recipes: []
|
||||
hardware:
|
||||
h100: verified
|
||||
h200: verified
|
||||
|
||||
model:
|
||||
model_id: "deepseek-ai/DeepSeek-V3.2"
|
||||
min_vllm_version: "0.18.0"
|
||||
default_frontend: rust
|
||||
architecture: moe
|
||||
parameter_count: "671B"
|
||||
active_parameters: "37B"
|
||||
context_length: 163840
|
||||
supports_dcp: true
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
- "--kernel-config.enable_flashinfer_autotune=False"
|
||||
base_env: {}
|
||||
|
||||
dependencies:
|
||||
- note: "DeepGEMM pinned build for MQA logits (FP8 MoE kernels)"
|
||||
command: "uv pip install git+https://github.com/deepseek-ai/DeepGEMM.git@v2.1.1.post3 --no-build-isolation"
|
||||
- note: "Set VLLM_USE_DEEP_GEMM=0 to skip DeepGEMM for the MoE path (recommended on H20)"
|
||||
command: "export VLLM_USE_DEEP_GEMM=0"
|
||||
optional: true
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable tool calling with DeepSeek V3.2 chat template support."
|
||||
args:
|
||||
- "--tokenizer-mode"
|
||||
- "deepseek_v32"
|
||||
- "--tool-call-parser"
|
||||
- "deepseek_v32"
|
||||
- "--enable-auto-tool-choice"
|
||||
reasoning:
|
||||
description: "Enable reasoning/thinking mode with the DeepSeek V3 reasoning parser."
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "deepseek_v3"
|
||||
spec_decoding:
|
||||
description: "Multi-Token Prediction speculative decoding with 3 speculative tokens."
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"mtp","num_speculative_tokens":3}'
|
||||
|
||||
opt_in_features:
|
||||
- spec_decoding
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: fp8
|
||||
vram_minimum_gb: 805
|
||||
description: "Native FP8 checkpoint (F8_E4M3)"
|
||||
nvfp4:
|
||||
model_id: "nvidia/DeepSeek-V3.2-NVFP4"
|
||||
precision: nvfp4
|
||||
vram_minimum_gb: 403
|
||||
description: "NVIDIA FP4 quantized variant with FP8 KV cache for reduced VRAM usage."
|
||||
extra_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
|
||||
redhat_nvfp4:
|
||||
model_id: "RedHatAI/DeepSeek-V3.2-NVFP4-FP8-BLOCK"
|
||||
precision: nvfp4
|
||||
label: "NVFP4+FP8 Block (RedHat)"
|
||||
vram_minimum_gb: 430
|
||||
description: "RedHatAI NVFP4+FP8 block mixed-precision checkpoint"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- single_node_tep
|
||||
- single_node_dep
|
||||
- multi_node_tp
|
||||
- multi_node_tp_pp
|
||||
- multi_node_dep
|
||||
- multi_node_tep
|
||||
- pd_cluster
|
||||
|
||||
hardware_overrides: {}
|
||||
|
||||
strategy_overrides: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
DeepSeek-V3.2 is a Mixture-of-Experts model that balances computational efficiency with
|
||||
strong reasoning and agent capabilities through three technical innovations: DeepSeek Sparse
|
||||
Attention (DSA) for efficient long-context processing, a scalable reinforcement learning
|
||||
framework achieving GPT-5-level performance, and a large-scale agentic task synthesis pipeline
|
||||
for robust tool-use generalization.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Hardware**: Minimum 8x H100/H200 80GB GPUs (BF16) or 3x H200 (NVFP4 variant).
|
||||
- **vLLM**: Version 0.18.0 or later (nightly recommended).
|
||||
- **Python**: 3.10+
|
||||
- **CUDA**: 12.x or later (CUDA 13.x may require extra env vars; see Troubleshooting).
|
||||
- **Disk**: ~1.3 TB for BF16 weights; ~350 GB for NVFP4 variant.
|
||||
- **DeepGEMM** (recommended):
|
||||
```bash
|
||||
uv pip install git+https://github.com/deepseek-ai/DeepGEMM.git@v2.1.1.post3 --no-build-isolation
|
||||
```
|
||||
Note: Set `VLLM_USE_DEEP_GEMM=0` to disable MoE DeepGEMM if you experience issues
|
||||
(e.g., on H20 GPUs) or want to skip the long warmup.
|
||||
|
||||
## Client Usage
|
||||
|
||||
Launch the server:
|
||||
```bash
|
||||
vllm serve deepseek-ai/DeepSeek-V3.2 \
|
||||
--tensor-parallel-size 8 \
|
||||
--trust-remote-code \
|
||||
--kernel-config.enable_flashinfer_autotune=False \
|
||||
--tokenizer-mode deepseek_v32 \
|
||||
--tool-call-parser deepseek_v32 \
|
||||
--enable-auto-tool-choice \
|
||||
--reasoning-parser deepseek_v3
|
||||
```
|
||||
|
||||
Use the OpenAI Python SDK to interact with the server:
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(
|
||||
api_key="your-api-key",
|
||||
base_url="http://localhost:8000/v1",
|
||||
)
|
||||
|
||||
# Standard chat
|
||||
response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-V3.2",
|
||||
messages=[{"role": "user", "content": "Hello!"}],
|
||||
)
|
||||
|
||||
# Thinking / reasoning mode
|
||||
response = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-V3.2",
|
||||
messages=[{"role": "user", "content": "Solve this step by step..."}],
|
||||
extra_body={"chat_template_kwargs": {"thinking": True}},
|
||||
)
|
||||
```
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
**`ptxas fatal: Value 'sm_110a' is not defined for option 'gpu-name'`**
|
||||
This can occur on CUDA 13.x. Fix by exporting:
|
||||
```bash
|
||||
export TRITON_PTXAS_PATH=/usr/local/cuda/bin/ptxas
|
||||
export PATH=/usr/local/cuda/bin:$PATH
|
||||
export LD_LIBRARY_PATH=/usr/local/cuda/lib64:${LD_LIBRARY_PATH:-}
|
||||
```
|
||||
|
||||
**TP=8 performance on Hopper/Blackwell**
|
||||
Avoid `-tp 8` with FlashMLA-Sparse. Due to kernel restrictions, TP=8 yields only 16 heads
|
||||
per rank but is padded to 64, causing overhead. Prefer TP=2 (Hopper) or TP=1 (Blackwell)
|
||||
with DP/EP mode: `vllm serve deepseek-ai/DeepSeek-V3.2 -dp 8 --enable-expert-parallel`.
|
||||
|
||||
**DeepGEMM warmup too slow**
|
||||
Set `VLLM_USE_DEEP_GEMM=0` to disable MoE DeepGEMM and skip the long warmup.
|
||||
|
||||
## References
|
||||
|
||||
- [DeepSeek-V3.2 on Hugging Face](https://huggingface.co/deepseek-ai/DeepSeek-V3.2)
|
||||
- [DeepSeek API Documentation](https://api-docs.deepseek.com/)
|
||||
- [vLLM Data Parallel Deployment Guide](https://docs.vllm.ai/en/latest/serving/data_parallel_deployment.html)
|
||||
@@ -0,0 +1,253 @@
|
||||
meta:
|
||||
title: "DeepSeek-V3"
|
||||
slug: "deepseek-v3"
|
||||
provider: "DeepSeek"
|
||||
description: "DeepSeek-V3 is a 671B-parameter Mixture-of-Experts model with native FP8 weights and strong reasoning, coding, and math capabilities."
|
||||
date_added: 2025-08-01
|
||||
date_updated: 2026-09-10
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "Frontier open-weights MoE with native FP8 and FP4 variants"
|
||||
related_recipes:
|
||||
- "deepseek-ai/DeepSeek-R1"
|
||||
- "deepseek-ai/DeepSeek-V3.1"
|
||||
hardware:
|
||||
h200: verified
|
||||
b200: verified
|
||||
gb200: verified
|
||||
mi300x: verified
|
||||
mi325x: verified
|
||||
mi350x: verified
|
||||
mi355x: verified
|
||||
|
||||
model:
|
||||
model_id: "deepseek-ai/DeepSeek-V3"
|
||||
min_vllm_version: "0.12.0"
|
||||
architecture: moe
|
||||
parameter_count: "671B"
|
||||
active_parameters: "37B"
|
||||
context_length: 163840
|
||||
supports_dcp: true
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
- "--enable-expert-parallel"
|
||||
base_env: {}
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable DeepSeek-V3 tool calling with the deepseek_v3 tool-call parser."
|
||||
args:
|
||||
- "--enable-auto-tool-choice"
|
||||
- "--tool-call-parser"
|
||||
- "deepseek_v3"
|
||||
- "--chat-template"
|
||||
- "examples/tool_chat_template_deepseekv3.jinja"
|
||||
reasoning:
|
||||
description: "Enable reasoning/thinking mode with the DeepSeek R1 reasoning parser."
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "deepseek_v3"
|
||||
|
||||
opt_in_features: []
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: fp8
|
||||
vram_minimum_gb: 805
|
||||
description: "Native FP8 weights on 8xH200 (recommended)"
|
||||
v3_0324:
|
||||
model_id: "deepseek-ai/DeepSeek-V3-0324"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 805
|
||||
description: "March 2025 DeepSeek-V3 refresh (DeepSeek-V3-0324)"
|
||||
fp4:
|
||||
model_id: "nvidia/DeepSeek-V3-FP4"
|
||||
precision: fp4
|
||||
vram_minimum_gb: 403
|
||||
description: "NVIDIA FP4 quantized weights for Blackwell (e.g. 4xB200)"
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- single_node_tep
|
||||
- single_node_dep
|
||||
- single_node_dpa_tp
|
||||
- multi_node_tp
|
||||
- multi_node_tp_pp
|
||||
- multi_node_dep
|
||||
- multi_node_tep
|
||||
- pd_cluster
|
||||
|
||||
hardware_overrides:
|
||||
amd:
|
||||
extra_args: []
|
||||
extra_env:
|
||||
VLLM_ROCM_USE_AITER: "1"
|
||||
|
||||
strategy_overrides:
|
||||
single_node_dpa_tp:
|
||||
extra_env:
|
||||
VLLM_ENGINE_READY_TIMEOUT_S: "1800"
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
DeepSeek-V3 is a 671B-parameter Mixture-of-Experts model (37B activated per token)
|
||||
shipped with native FP8 weights. It shares its architecture with DeepSeek-R1, so the
|
||||
same launch recipes apply to both models. DeepSeek publishes a refreshed checkpoint as
|
||||
[`DeepSeek-V3-0324`](https://huggingface.co/deepseek-ai/DeepSeek-V3-0324), available
|
||||
here as the **V3 0324** variant. For Blackwell GPUs, NVIDIA publishes an FP4 quantized
|
||||
variant (`nvidia/DeepSeek-V3-FP4` / `nvidia/DeepSeek-R1-FP4`) that runs on fewer GPUs.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- **Hardware (FP8)**: 8x H200 (CUDA) or 8x MI300X / MI325X / MI355X (ROCm)
|
||||
- **Hardware (FP4)**: 4x B200 GPUs
|
||||
- **vLLM**: Current stable release
|
||||
|
||||
### CUDA
|
||||
```bash
|
||||
uv venv
|
||||
source .venv/bin/activate
|
||||
uv pip install -U vllm --torch-backend auto
|
||||
```
|
||||
|
||||
### ROCm (MI300X, MI325X, MI355X)
|
||||
Requires Python 3.12, ROCm 7.2.1, and glibc >= 2.35.
|
||||
```bash
|
||||
uv venv --python 3.12
|
||||
source .venv/bin/activate
|
||||
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
|
||||
```
|
||||
|
||||
## Client Usage
|
||||
|
||||
### 8xH200 / 8xMI300X (FP8)
|
||||
|
||||
Tensor Parallel + Expert Parallel (TP8+EP) — CUDA:
|
||||
```bash
|
||||
vllm serve deepseek-ai/DeepSeek-V3 \
|
||||
--trust-remote-code \
|
||||
--tensor-parallel-size 8 \
|
||||
--enable-expert-parallel
|
||||
```
|
||||
|
||||
Tensor Parallel + Expert Parallel (TP8+EP) — ROCm:
|
||||
```bash
|
||||
export VLLM_ROCM_USE_AITER=1
|
||||
|
||||
vllm serve deepseek-ai/DeepSeek-V3 \
|
||||
--trust-remote-code \
|
||||
--tensor-parallel-size 8 \
|
||||
--enable-expert-parallel
|
||||
```
|
||||
|
||||
Data Parallel + Expert Parallel (DP8+EP) — CUDA:
|
||||
```bash
|
||||
vllm serve deepseek-ai/DeepSeek-V3 \
|
||||
--trust-remote-code \
|
||||
--data-parallel-size 8 \
|
||||
--enable-expert-parallel
|
||||
```
|
||||
|
||||
### 8xMI350X / 8xMI355X (DPA+TP)
|
||||
|
||||
Each GPU runs its own attention replica with a private KV cache, while the MoE
|
||||
experts stay tensor-sharded across all eight GPUs. Aggregate KV-cache
|
||||
capacity is roughly 8x that of a single TP8 replica, useful for workloads
|
||||
benefiting from larger KV capacities.
|
||||
|
||||
Engine (all eight ranks in one process, listening on 8100):
|
||||
```bash
|
||||
export VLLM_ROCM_USE_AITER=1
|
||||
export VLLM_ENGINE_READY_TIMEOUT_S=1800
|
||||
|
||||
vllm serve deepseek-ai/DeepSeek-V3-0324 \
|
||||
--trust-remote-code \
|
||||
--port 8100 \
|
||||
--data-parallel-size 8 \
|
||||
--tensor-parallel-size 1
|
||||
```
|
||||
|
||||
Router (client-facing on 8000, spreads requests across the eight DP ranks):
|
||||
```bash
|
||||
uv pip install vllm-router
|
||||
|
||||
vllm-router \
|
||||
--host 0.0.0.0 \
|
||||
--port 8000 \
|
||||
--policy consistent_hash \
|
||||
--intra-node-data-parallel-size 8 \
|
||||
--worker-urls http://localhost:8100 \
|
||||
--worker-startup-timeout-secs 1800
|
||||
```
|
||||
|
||||
Notes:
|
||||
|
||||
- `--intra-node-data-parallel-size` must match the engine's
|
||||
`--data-parallel-size`. A mismatch silently under-uses ranks.
|
||||
- `consistent_hash` is the policy that makes this worthwhile for chat: it pins
|
||||
a conversation to the rank that already holds its prefix, so the larger
|
||||
aggregate cache actually gets hit. `round_robin` scatters turns of the same
|
||||
conversation across ranks and throws the prefix away.
|
||||
- Start the engine first, then the router; `--worker-startup-timeout-secs 1800`
|
||||
covers the engine's weight load either way.
|
||||
- Send client traffic to the router on port 8000, never to 8100 directly.
|
||||
|
||||
### 4xB200 (FP4)
|
||||
|
||||
On vLLM v0.28.0 and later, the FlashInfer MoE kernels are selected automatically
|
||||
for both FP4 and FP8 on Blackwell — no environment variables are needed. To pin
|
||||
the backend explicitly, pass `--moe-backend flashinfer_trtllm`.
|
||||
|
||||
Tensor Parallel + Expert Parallel (TP4+EP):
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=0,1,2,3 vllm serve nvidia/DeepSeek-V3-FP4 \
|
||||
--trust-remote-code \
|
||||
--tensor-parallel-size 4 \
|
||||
--enable-expert-parallel
|
||||
```
|
||||
|
||||
Data Parallel + Expert Parallel (DP4+EP):
|
||||
```bash
|
||||
CUDA_VISIBLE_DEVICES=0,1,2,3 vllm serve nvidia/DeepSeek-V3-FP4 \
|
||||
--trust-remote-code \
|
||||
--data-parallel-size 4 \
|
||||
--enable-expert-parallel
|
||||
```
|
||||
|
||||
## Benchmarking
|
||||
|
||||
For benchmarking, prefix caching is disabled by default in vLLM — no extra server
|
||||
flag is needed.
|
||||
|
||||
```bash
|
||||
# Prompt-heavy benchmark (8k/1k)
|
||||
vllm bench serve \
|
||||
--model deepseek-ai/DeepSeek-V3 \
|
||||
--dataset-name random \
|
||||
--random-input-len 8000 \
|
||||
--random-output-len 1000 \
|
||||
--request-rate 10000 \
|
||||
--num-prompts 16 \
|
||||
--ignore-eos
|
||||
```
|
||||
|
||||
Test different workloads by adjusting input/output lengths:
|
||||
- Prompt-heavy: 8000 input / 1000 output
|
||||
- Decode-heavy: 1000 input / 8000 output
|
||||
- Balanced: 1000 input / 1000 output
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
- **Disaggregated Serving with Wide EP (Experimental GB200)**: See
|
||||
[vLLM issue #33583](https://github.com/vllm-project/vllm/issues/33583) and the
|
||||
[vLLM blog post](https://blog.vllm.ai/2026/02/03/dsr1-gb200-part1.html) for
|
||||
GB200 disaggregated serving recipes.
|
||||
|
||||
## References
|
||||
|
||||
- [DeepSeek-V3 on Hugging Face](https://huggingface.co/deepseek-ai/DeepSeek-V3)
|
||||
- [DeepSeek-V3-0324 on Hugging Face](https://huggingface.co/deepseek-ai/DeepSeek-V3-0324)
|
||||
- [NVIDIA DeepSeek-V3-FP4](https://huggingface.co/nvidia/DeepSeek-V3-FP4)
|
||||
- [vLLM Expert Parallelism docs](https://docs.vllm.ai/en/latest/serving/expert_parallel_deployment.html)
|
||||
- [vLLM Data Parallel deployment docs](https://docs.vllm.ai/en/latest/serving/data_parallel_deployment.html)
|
||||
@@ -0,0 +1,464 @@
|
||||
meta:
|
||||
title: "DeepSeek-V4-Flash-Vision-Exp"
|
||||
slug: "deepseek-v4-flash-vision-exp"
|
||||
provider: "DeepSeek"
|
||||
description: "DeepSeek's first experimental multimodal V4 model — the V4-Flash MoE backbone plus a 32-layer vision tower, 1M context, and a fused DSpark draft module."
|
||||
date_added: 2026-09-01
|
||||
date_updated: 2026-09-14
|
||||
tasks:
|
||||
- text
|
||||
- multimodal
|
||||
performance_headline: "First multimodal V4 — 285B/13B MoE scoring 83.5% on OCRBench from a single GB200 NVL4 tray"
|
||||
related_recipes:
|
||||
- "deepseek-ai/DeepSeek-V4-Flash"
|
||||
- "deepseek-ai/DeepSeek-V4-Pro"
|
||||
# The vision path is unreleased — it lives in the open support PR
|
||||
# vllm-project/vllm#54566 (see model.install below). GB200 is the only
|
||||
# NVIDIA hardware with a published end-to-end run on that branch: TP4 + EP
|
||||
# on one NVL4 tray, FP8 KV cache, block size 256, DSpark k=3, full
|
||||
# 1000-sample OCRBench at 83.5% with zero request errors.
|
||||
# Atlas A5 8-card mixed serving of the native FP4+FP8 Vision-Exp checkpoint
|
||||
# (DP4+EP, TP=1, no speculative decoding). 950PR verified 2026-09-10 on
|
||||
# vLLM Ascend; 950DT uses the same command (4 of 8 NPUs). See the hardware
|
||||
# note for the required hash-router bias fix.
|
||||
# AMD MI350X (gfx950): ROCm vision merged in vllm-project/vllm#55107 (2026-09-11)
|
||||
# on the #54566 base. Verified text+vision on stock ROCm nightly at single-node
|
||||
# TP2 and TP4, with DSpark k=3 (needs VLLM_USE_V2_MODEL_RUNNER=1). See
|
||||
# "Launching on AMD" in the guide.
|
||||
default_hardware: gb200
|
||||
hardware:
|
||||
gb200: verified
|
||||
mi350x: verified
|
||||
ascend_950dt_8x: verified
|
||||
ascend_950pr_8x: verified
|
||||
|
||||
model:
|
||||
model_id: "deepseek-ai/DeepSeek-V4-Flash-Vision-Exp"
|
||||
# Support targets the next release; nothing has shipped yet, so the pinned
|
||||
# image below is the only runnable path.
|
||||
min_vllm_version: "0.30.0"
|
||||
nightly_required: true
|
||||
docker_image:
|
||||
nvidia: "vllm/vllm-openai:deepseekv4-flash-vision"
|
||||
amd: "vllm/vllm-openai-rocm:nightly"
|
||||
install:
|
||||
# Docker first so it is the default tab. The official wheel can't run this
|
||||
# checkpoint: released vLLM prior to v0.30.0 routes it to the text-only
|
||||
# DeepseekV4ForCausalLM, which then fails on the vision/aligner tensors.
|
||||
# v0.30.0 already resolves it to DeepseekV4ForConditionalGeneration (verified on the ROCm image, 2026-10-02).
|
||||
docker:
|
||||
note: "NVIDIA: use the dedicated DeepSeek-V4-Flash-Vision image until support lands in the standard vLLM image. AMD: vllm/vllm-openai-rocm:v0.30.0 (ROCm 7.2.3) or a nightly >= 2026-09-11 (includes #55107); on v0.30.0 also pass --compilation-config '{\"cudagraph_mode\":\"FULL_DECODE_ONLY\"}' (see the AMD section). Ascend 950PR/950DT: pin quay.io/ascend/vllm-ascend:v0.23.0-a5 and overlay vllm-project/vllm-ascend#16259."
|
||||
pip: false
|
||||
architecture: moe
|
||||
parameter_count: "285B"
|
||||
active_parameters: "13B"
|
||||
context_length: 1048576
|
||||
flashinfer_autotune: true
|
||||
base_args:
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
- "--block-size"
|
||||
- "256"
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "DeepSeek V4 tool-call parsing with automatic tool choice."
|
||||
args:
|
||||
- "--tool-call-parser"
|
||||
- "deepseek_v4"
|
||||
- "--enable-auto-tool-choice"
|
||||
reasoning:
|
||||
description: "Enable reasoning/thinking mode with the DeepSeek V4 reasoning parser."
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "deepseek_v4"
|
||||
- "--reasoning-config"
|
||||
- '{"reasoning_parser":"deepseek_v4","reasoning_start_str":"","reasoning_end_str":""}'
|
||||
# Empty start/end strings make the Ascend chat path emit content=null
|
||||
# for a full completion. Keep the parser, drop the override.
|
||||
hardware_overrides:
|
||||
npu:
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "deepseek_v4"
|
||||
# Single method, so the boolean form (not `modes`): the checkpoint's fused
|
||||
# DSpark module is the only drafter with a published run on this branch. Left
|
||||
# OUT of opt_in_features on purpose — the verified GB200 configuration had
|
||||
# DSpark on, and DeepSeek's own model card enables it too, so the default
|
||||
# builder state reproduces the tested command. The checkpoint also carries
|
||||
# 3 MTP layers (`num_nextn_predict_layers: 3`); that path is untested here.
|
||||
spec_decoding:
|
||||
description: "DSpark self-drafting from the checkpoint's fused draft module — 3 draft tokens, probabilistic sampling, adaptive verification. Mean acceptance length 2.99 tokens/forward on mixed text+image traffic."
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"dspark","model":"deepseek-ai/DeepSeek-V4-Flash-Vision-Exp","num_speculative_tokens":3,"draft_sample_method":"probabilistic","enable_adaptive_verification":true}'
|
||||
# 950 has no working MTP or DSpark path yet. Empty args so a toggle-on
|
||||
# still emits no --speculative-config. Default-off via
|
||||
# hardware_opt_in_features below — GB200 keeps DSpark on.
|
||||
hardware_overrides:
|
||||
# ROCm: the DeepseekV4 indexer backend rejects adaptive verification, so it
|
||||
# is forced false (DSpark still drafts k=3). VLLM_USE_V2_MODEL_RUNNER=1
|
||||
# (hardware_overrides.amd.extra_env) is also required.
|
||||
amd:
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"dspark","model":"deepseek-ai/DeepSeek-V4-Flash-Vision-Exp","num_speculative_tokens":3,"draft_sample_method":"probabilistic","enable_adaptive_verification":false}'
|
||||
npu:
|
||||
args: []
|
||||
|
||||
# Spec decoding stays default-on everywhere except the A5 8-card pills.
|
||||
hardware_opt_in_features:
|
||||
ascend_950pr_8x:
|
||||
- spec_decoding
|
||||
ascend_950dt_8x:
|
||||
- spec_decoding
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: fp8
|
||||
# Mixed FP4 experts + FP8 remainder, so the bytes-per-param table
|
||||
# underestimates: sized from the real checkpoint (167.8 GB x 1.2).
|
||||
vram_minimum_gb: 202
|
||||
description: "Native FP4+FP8 mixed checkpoint (MoE experts FP4, attention/norm/router FP8) with a BF16 vision tower, aligner, and the fused DSpark draft module."
|
||||
hardware_overrides:
|
||||
# Exact-GPU pin: computeDockerMeta has no Huawei brand, so the model-level
|
||||
# docker_image string only affects NVIDIA. Hide pip — official wheels
|
||||
# have no NPU kernels.
|
||||
ascend_950pr_8x:
|
||||
docker_image: "quay.io/ascend/vllm-ascend:v0.23.0-a5"
|
||||
install:
|
||||
pip: false
|
||||
ascend_950dt_8x:
|
||||
docker_image: "quay.io/ascend/vllm-ascend:v0.23.0-a5"
|
||||
install:
|
||||
pip: false
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- single_node_tep
|
||||
- single_node_dep
|
||||
- multi_node_tep
|
||||
- multi_node_dep
|
||||
|
||||
# TP + EP is the verified layout (TP4 + EP on one GB200 NVL4 tray).
|
||||
default_strategy: single_node_tep
|
||||
|
||||
# Atlas A5 mixed serving is DP+EP only. The native checkpoint has 64
|
||||
# attention heads; 950's FP8 DSA operator accepts local num_heads_q of
|
||||
# 64 or 128, so TP must stay 1. TEP/TP and intra-node PD are not offered.
|
||||
strategy_hardware:
|
||||
single_node_tp:
|
||||
npu: unsupported
|
||||
single_node_tep:
|
||||
npu: unsupported
|
||||
multi_node_tep:
|
||||
npu: unsupported
|
||||
multi_node_dep:
|
||||
npu: unsupported
|
||||
|
||||
# Shared by 950DT 8x (8×96 GB) and 950PR 8x (8×128 GB). Mixed serving uses
|
||||
# 4 of the 8 NPUs (DP4+EP, TP=1). HCCL_BUFFSIZE=1024 is required for MC2
|
||||
# dispatch. last-wins shadows the DEP builder's --data-parallel-size 8
|
||||
# and the recipe-level --block-size 256 / --kv-cache-dtype fp8.
|
||||
hardware_overrides:
|
||||
amd:
|
||||
extra_args:
|
||||
- "--tensor-parallel-size"
|
||||
- "2"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.9"
|
||||
- "--max-model-len"
|
||||
- "32768"
|
||||
extra_env:
|
||||
VLLM_ROCM_USE_AITER: "1"
|
||||
VLLM_USE_BREAKABLE_CUDAGRAPH: "1"
|
||||
VLLM_USE_V2_MODEL_RUNNER: "1"
|
||||
npu:
|
||||
extra_args:
|
||||
- "--data-parallel-size"
|
||||
- "4"
|
||||
- "--block-size"
|
||||
- "128"
|
||||
- "--kv-cache-dtype"
|
||||
- "auto"
|
||||
- "--tokenizer-mode"
|
||||
- "deepseek_v4"
|
||||
- "--safetensors-load-strategy"
|
||||
- "prefetch"
|
||||
- "--async-scheduling"
|
||||
- "--max-model-len"
|
||||
- "8192"
|
||||
- "--max-num-seqs"
|
||||
- "16"
|
||||
- "--max-num-batched-tokens"
|
||||
- "2048"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.9"
|
||||
- "--api-server-count"
|
||||
- "1"
|
||||
- "--compilation-config"
|
||||
- '{"cudagraph_mode":"FULL_DECODE_ONLY"}'
|
||||
- "--additional-config"
|
||||
- '{"enable_cpu_binding": true, "multistream_overlap_shared_expert": true, "enable_shared_expert_dp": true}'
|
||||
extra_env:
|
||||
PYTORCH_NPU_ALLOC_CONF: "expandable_segments:True"
|
||||
OMP_PROC_BIND: "false"
|
||||
OMP_NUM_THREADS: "10"
|
||||
HCCL_BUFFSIZE: "1024"
|
||||
HCCL_OP_EXPANSION_MODE: "AIV"
|
||||
TASK_QUEUE_ENABLE: "1"
|
||||
ASCEND_RT_VISIBLE_DEVICES: "0,1,2,3"
|
||||
|
||||
hardware_notes:
|
||||
ascend_950dt_8x:
|
||||
title: "vLLM Ascend A5 image required"
|
||||
body: "Mixed serving on one 8-card 950DT node uses 4 of 8 NPUs (DP4+EP, TP=1). Same command as 950PR. Pin quay.io/ascend/vllm-ascend:v0.23.0-a5 — the NVIDIA pip wheel cannot run here. Set HCCL_BUFFSIZE=1024 before launch. Native Flash-Vision checkpoints need the hash-router bias skip in vllm-project/vllm-ascend#16259 — without it load_weights aborts on model.layers.0.mlp.gate.e_score_correction_bias."
|
||||
ascend_950pr_8x:
|
||||
title: "vLLM Ascend A5 image required"
|
||||
body: "Mixed serving on one 8-card 950PR node uses 4 of 8 NPUs (DP4+EP, TP=1). Same command as 950DT. Pin quay.io/ascend/vllm-ascend:v0.23.0-a5 and set HCCL_BUFFSIZE=1024 before launch. Native Flash-Vision checkpoints need the hash-router bias skip in vllm-project/vllm-ascend#16259 — without it load_weights aborts on model.layers.0.mlp.gate.e_score_correction_bias."
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
DeepSeek-V4-Flash-Vision-Exp is DeepSeek's first multimodal model in the V4 family.
|
||||
It keeps the DeepSeek-V4-Flash language backbone — 43 layers, 256 routed experts with
|
||||
6 active per token, Compressed Sparse Attention plus manifold-constrained
|
||||
hyper-connections, a 1,048,576-token context window — and adds a 32-layer / 1024-dim
|
||||
ViT with a two-layer aligner (~0.5B parameters on top of the 284B backbone). Weights
|
||||
are **FP4+FP8 mixed**: MoE experts in FP4, the remaining attention / norm / router
|
||||
params in FP8, vision tower in BF16. The checkpoint is 48 shards / ~168 GB and carries
|
||||
the fused **DSpark** draft module.
|
||||
|
||||
DeepSeek reports it beats DeepSeek-V4-Flash-0731 on multimodal agent benchmarks
|
||||
(ApexBench 36.5 vs 26.2, Agents' Last Exam 27.3 vs 25.2) while holding text-agent
|
||||
parity (Terminal Bench 2.1 at 83.9, DeepSWE at 59.3).
|
||||
|
||||
## Prerequisites
|
||||
|
||||
Vision support is not in a stable vLLM release yet — it lives in
|
||||
[vllm-project/vllm#54566](https://github.com/vllm-project/vllm/pull/54566). Until that
|
||||
lands, use the pinned `vllm/vllm-openai:deepseekv4-flash-vision` image; the official
|
||||
wheel routes this checkpoint to the text-only class and fails on the vision tensors.
|
||||
|
||||
- **Weights:** ~168 GB on disk (48 shards). Budget ~202 GB of VRAM before KV cache.
|
||||
- **Hardware:** NVIDIA (verified on GB200) and AMD MI350X (gfx950);
|
||||
XPU still has no vision implementation.
|
||||
- **Tokenizer / chat template:** nothing to pass. The repo ships no Jinja template
|
||||
(prompt encoding lives in `encoding/`); vLLM resolves `--tokenizer-mode` to
|
||||
`deepseek_v4` automatically, which is what turns OpenAI content blocks into
|
||||
`<｜deepseek_image｜>` placeholders.
|
||||
|
||||
## Launching the server
|
||||
|
||||
### Verified configuration — one GB200 NVL4 tray (TP4 + EP)
|
||||
|
||||
```bash
|
||||
export VLLM_FLASHINFER_AUTOTUNE_SKIP_OPS="trtllm_fp4_block_scale_moe,flashinfer::trtllm_fp4_block_scale_moe"
|
||||
|
||||
vllm serve deepseek-ai/DeepSeek-V4-Flash-Vision-Exp \
|
||||
--tensor-parallel-size 4 \
|
||||
--enable-expert-parallel \
|
||||
--kv-cache-dtype fp8 \
|
||||
--block-size 256 \
|
||||
--max-model-len 32768 \
|
||||
--speculative-config '{"method":"dspark","model":"deepseek-ai/DeepSeek-V4-Flash-Vision-Exp","num_speculative_tokens":3,"draft_sample_method":"probabilistic","enable_adaptive_verification":true}' \
|
||||
--allowed-local-media-path /mnt/lustre
|
||||
```
|
||||
|
||||
Three things there are environment-specific, which is why the builder above leaves
|
||||
them out:
|
||||
|
||||
- `VLLM_FLASHINFER_AUTOTUNE_SKIP_OPS` excludes the FP4 block-scale MoE op from
|
||||
FlashInfer's startup autotuning. Export it if autotuning that op fails or costs more
|
||||
startup time than it earns back on your FlashInfer build; it is a tuning knob, not a
|
||||
requirement. The Advanced row's `--no-enable-flashinfer-autotune` turns autotuning
|
||||
off wholesale instead.
|
||||
- `--max-model-len 32768` bounds KV cache for the benchmark. The checkpoint advertises
|
||||
1M tokens; that is **not** what was measured, and 1M-token KV needs far more memory
|
||||
than the weights do. Raise it deliberately, or use the Advanced row's
|
||||
`--max-model-len auto`.
|
||||
- `--allowed-local-media-path` is only needed for `file://` image URLs. Drop it if
|
||||
clients send `http(s)://` or base64 data URLs.
|
||||
|
||||
### 8-GPU nodes (H200 / B200 / B300)
|
||||
|
||||
Same **Tensor + Expert Parallel** strategy; the builder emits
|
||||
`--tensor-parallel-size 8 --enable-expert-parallel`. The language backbone is the one
|
||||
the [DeepSeek-V4-Flash](/deepseek-ai/DeepSeek-V4-Flash) recipe already runs there, but
|
||||
the vision path has no published run on those GPUs — expect to tune
|
||||
`--gpu-memory-utilization` and `--max-num-seqs`. The Blackwell
|
||||
`--attention_config.indexer_kv_dtype mxfp4` / `--moe-backend deep_gemm_mega_moe` tweaks
|
||||
the text recipe applies are deliberately not set here: the verified vision run did not
|
||||
use them.
|
||||
|
||||
### AMD MI350X (gfx950)
|
||||
|
||||
ROCm vision support is merged (#55107 on the #54566 base); a
|
||||
`vllm/vllm-openai-rocm:nightly` built on/after 2026-09-11 runs the checkpoint
|
||||
natively. Three env vars are required:
|
||||
|
||||
- `VLLM_ROCM_USE_AITER=1` — the sparse-attention indexer's ROCm path runs only on
|
||||
AITER; without it the engine aborts at KV-cache profiling.
|
||||
- `VLLM_USE_BREAKABLE_CUDAGRAPH=1` — the multimodal wrapper is not torch-compiled, so
|
||||
the default `FULL_AND_PIECEWISE` capture cannot attach without breakable graphs.
|
||||
- `VLLM_USE_V2_MODEL_RUNNER=1` — required for DSpark (the V1 ROCm runner refuses it);
|
||||
harmless for plain serving.
|
||||
|
||||
Verified single-node launch:
|
||||
|
||||
```bash
|
||||
docker run -d --name dsv4-vision \
|
||||
--ipc host --network host --privileged --shm-size 128g \
|
||||
--device /dev/kfd --device /dev/dri --group-add video \
|
||||
-v ~/.cache/huggingface:/root/.cache/huggingface \
|
||||
-e VLLM_ROCM_USE_AITER=1 \
|
||||
-e VLLM_USE_BREAKABLE_CUDAGRAPH=1 \
|
||||
-e VLLM_USE_V2_MODEL_RUNNER=1 \
|
||||
vllm/vllm-openai-rocm:nightly \
|
||||
deepseek-ai/DeepSeek-V4-Flash-Vision-Exp \
|
||||
--tensor-parallel-size 2 \
|
||||
--kv-cache-dtype fp8 --block-size 256 \
|
||||
--gpu-memory-utilization 0.9 \
|
||||
--max-model-len 32768
|
||||
```
|
||||
|
||||
Observed backends: AITER Flash-Attention for the ViT, DeepSeek `fp8_ds_mla` KV cache,
|
||||
FP8 Lightning-Indexer cache, `DEEPSEEK_SPARSE_SWA` (block 256), and the
|
||||
`ROCM_FLASHMLA_SPARSE_DSV4` MLA backend. TP2 loads 75.2 GiB/card (1.76M-token KV pool
|
||||
at 131K context); TP4 loads 38.8 GiB/card.
|
||||
|
||||
**Stable v0.30.0 and single-card TP1** (verified 2026-10-02 on one MI350X). The stable
|
||||
`vllm/vllm-openai-rocm:v0.30.0` image (ROCm 7.2.3) already contains #54566 and #55107 and
|
||||
runs the checkpoint natively — text, vision and DSpark — with the same three env vars. On
|
||||
v0.30.0 only, also pass `--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}'`:
|
||||
v0.30.0's ROCm platform still defaults DeepSeek V4 on gfx950 + ModelRunner V2 to eager
|
||||
(the #52644 workaround, removed by #57229 after v0.30.0 was branched), so without it decode
|
||||
runs without CUDA graphs. Batch 1 at TP1: 12 tok/s eager vs 94 tok/s with
|
||||
`FULL_DECODE_ONLY`, and 137–164 tok/s with DSpark (k=3, adaptive verification off per the
|
||||
AMD override). Against eager in the same run, a 25-item known-answer check scored 25/25 and
|
||||
decode-vs-prefill logprobs agreed within eager's range. Nightlies after 2026-09-17 and
|
||||
v0.31+ don't need the override. TP1 fits one 288 GB card (147.9 GiB of weights).
|
||||
|
||||
|
||||
## Reasoning modes
|
||||
|
||||
Same three-tier control as the rest of the V4 family, driven from
|
||||
`chat_template_kwargs` rather than a flag — `encoding/encoding_dsv4.py` names the
|
||||
effort levels `low` / `high` / `max`. DeepSeek evaluated this model at
|
||||
`reasoning_effort: "max"` with `temperature = 1.0, top_p = 0.95`.
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||||
model = "deepseek-ai/DeepSeek-V4-Flash-Vision-Exp"
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model=model,
|
||||
messages=[{"role": "user", "content": "What is 17*19? Return only the final integer."}],
|
||||
temperature=1.0,
|
||||
top_p=0.95,
|
||||
extra_body={
|
||||
"chat_template_kwargs": {"thinking": True, "reasoning_effort": "max"},
|
||||
},
|
||||
)
|
||||
print(resp.choices[0].message.reasoning)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
## Client usage — image input
|
||||
|
||||
Standard OpenAI multi-part content. Multiple images per request are allowed with no
|
||||
cap, so the practical limit is `--max-model-len`.
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||||
|
||||
resp = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-V4-Flash-Vision-Exp",
|
||||
messages=[{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "image_url", "image_url": {"url": "https://example.com/chart.png"}},
|
||||
{"type": "text", "text": "Read the y-axis label and the highest bar's value."},
|
||||
],
|
||||
}],
|
||||
max_tokens=512,
|
||||
)
|
||||
print(resp.choices[0].message.content)
|
||||
```
|
||||
|
||||
```bash
|
||||
curl http://localhost:8000/v1/chat/completions \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{
|
||||
"model": "deepseek-ai/DeepSeek-V4-Flash-Vision-Exp",
|
||||
"messages": [{
|
||||
"role": "user",
|
||||
"content": [
|
||||
{"type": "image_url", "image_url": {"url": "https://example.com/chart.png"}},
|
||||
{"type": "text", "text": "Describe the image."}
|
||||
]
|
||||
}],
|
||||
"max_tokens": 512
|
||||
}'
|
||||
```
|
||||
|
||||
Image preprocessing is fixed by the checkpoint config (the processor takes no kwargs):
|
||||
each image costs at most 387 prompt tokens (`vision_max_n_token: 384` plus the
|
||||
compressor-alignment pad), images below `vision_min_pixels: 147456` are upscaled, and
|
||||
aspect ratios beyond 8:1 are cropped. An image's exact token count depends on where it
|
||||
sits in the prompt, since the compressor pads each image to a 4-token block boundary.
|
||||
|
||||
## Speculative decoding
|
||||
|
||||
DSpark drafts from the module fused into the checkpoint, so `--speculative-config`
|
||||
names the target repo as its own `model` — there is no separate draft repo to download.
|
||||
Measured over the full OCRBench run on GB200 (mixed text+image traffic): **2.99**
|
||||
tokens/forward mean acceptance length, **66.3%** overall acceptance
|
||||
(83.9% / 66.5% / 50.7% by draft position). Image tokens in the prompt do not degrade
|
||||
acceptance — the drafter reads image content through the target's hidden states.
|
||||
|
||||
Stay at `num_speculative_tokens: 3` unless you verify acceptance yourself; the
|
||||
checkpoint's trained DSpark block width is 5, but 3 is the depth with published
|
||||
numbers.
|
||||
|
||||
## Benchmarking
|
||||
|
||||
OCRBench, full 1000 samples, on the verified GB200 configuration: **835/1000
|
||||
(83.5%)**, 0 request errors, 312.6 s (~3.2 samples/s). Strongest categories are Key
|
||||
Information Extraction (92.5%), Doc-oriented VQA (90.0%) and Scene Text VQA (89.5%);
|
||||
weakest are handwritten math (49.0%) and handwriting recognition (58.0%).
|
||||
|
||||
For throughput:
|
||||
|
||||
```bash
|
||||
vllm bench serve \
|
||||
--model deepseek-ai/DeepSeek-V4-Flash-Vision-Exp \
|
||||
--dataset-name random \
|
||||
--random-input-len 4096 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts 200
|
||||
```
|
||||
|
||||
## Limitations
|
||||
|
||||
- **Pre-release.** Flags and defaults can change before #54566 lands; re-check before
|
||||
deploying, and keep the image pinned.
|
||||
- **XPU has no vision path** — it raises `NotImplementedError` at model construction.
|
||||
- **GB200 (NVIDIA) and MI350X (AMD) are the hardware with a verified vision run.**
|
||||
Every other pill is the repo's fail-open default, not a tested claim.
|
||||
- **PD disaggregation is not offered**, and the CPU/filesystem KV-offload pills stay
|
||||
off — neither has a verified run to record. The Mooncake KV-store pills follow the
|
||||
repo's fail-open default and are offered, but untested here.
|
||||
- **1M context is advertised, not measured.** The reference run used 32K.
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/deepseek-ai/DeepSeek-V4-Flash-Vision-Exp)
|
||||
- [vLLM support PR #54566](https://github.com/vllm-project/vllm/pull/54566)
|
||||
- [vLLM ROCm vision PR #55107](https://github.com/vllm-project/vllm/pull/55107)
|
||||
- [DeepSeek-V4-Flash recipe](/deepseek-ai/DeepSeek-V4-Flash) — the text-only sibling
|
||||
- [DeepSpec (DSpark)](https://github.com/deepseek-ai/DeepSpec)
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,972 @@
|
||||
meta:
|
||||
title: "DeepSeek-V4-Pro"
|
||||
slug: "deepseek-v4-pro"
|
||||
provider: "DeepSeek"
|
||||
description: "DeepSeek V4 flagship MoE (1.6T total / 49B active) with hybrid CSA+HCA attention, manifold-constrained hyper-connections, Muon-trained on 32T+ tokens, and three-tier reasoning."
|
||||
date_added: 2026-04-24
|
||||
date_updated: 2026-10-13
|
||||
tasks:
|
||||
- text
|
||||
performance_headline: "Frontier 1.6T/49B reasoning MoE with native FP4+FP8 weights, MTP and DSpark speculative decoding, and 1M-token context"
|
||||
related_recipes: []
|
||||
hardware:
|
||||
h200: verified
|
||||
b200: verified
|
||||
gb200: verified
|
||||
b300: verified
|
||||
gb300: verified
|
||||
mi300x: unsupported
|
||||
mi325x: unsupported
|
||||
mi355x: verified
|
||||
|
||||
model:
|
||||
model_id: "deepseek-ai/DeepSeek-V4-Pro"
|
||||
min_vllm_version: "0.20.0"
|
||||
default_frontend: rust
|
||||
docker_image:
|
||||
amd: "vllm/vllm-openai-rocm:nightly"
|
||||
architecture: moe
|
||||
parameter_count: "1600B"
|
||||
active_parameters: "49B"
|
||||
context_length: 1048576
|
||||
flashinfer_autotune: true
|
||||
base_args:
|
||||
- "--trust-remote-code"
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
- "--block-size"
|
||||
- "256"
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable tool calling with DeepSeek V4 chat template support."
|
||||
args:
|
||||
- "--tokenizer-mode"
|
||||
- "deepseek_v4"
|
||||
- "--tool-call-parser"
|
||||
- "deepseek_v4"
|
||||
- "--enable-auto-tool-choice"
|
||||
reasoning:
|
||||
description: "Enable reasoning/thinking mode with the DeepSeek V4 reasoning parser."
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "deepseek_v4"
|
||||
spec_decoding:
|
||||
description: "Speculative decoding — pick a drafting method."
|
||||
# Mode = the --speculative-config method only (args); the served checkpoint
|
||||
# is owned by the Variant row. Default is MTP; the 0813 (default) and DSpark
|
||||
# checkpoints flip this to dspark via their `default_modes`.
|
||||
default_mode: mtp
|
||||
modes:
|
||||
mtp:
|
||||
label: "MTP"
|
||||
description: "Built-in Multi-Token Prediction, 2 draft tokens."
|
||||
# The preview FP4+FP8 and NVFP4 checkpoints ship an MTP head only — their
|
||||
# config.json carries no dspark_* block, so dspark can't be drafted there.
|
||||
variants:
|
||||
- fp8
|
||||
- nvfp4
|
||||
- redhat_nvfp4
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"mtp","num_speculative_tokens":2}'
|
||||
dspark:
|
||||
label: "DSpark"
|
||||
description: "DSpark drafting, 7 draft tokens, probabilistic. Requires the 0813 or DSpark checkpoint (Variant row)."
|
||||
# Only the fused checkpoints carry the DSpark draft — the 0813 release
|
||||
# (default) and the DSpark re-pack of the preview weights.
|
||||
variants:
|
||||
- default
|
||||
- dspark
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'
|
||||
|
||||
opt_in_features:
|
||||
- spec_decoding
|
||||
|
||||
variants:
|
||||
default:
|
||||
# Official 0813 release: supersedes the preview, same architecture with the
|
||||
# fused DSpark draft module attached (config.json gains the dspark_* block —
|
||||
# byte-identical to the DSpark re-pack below).
|
||||
model_id: "deepseek-ai/DeepSeek-V4-Pro-0813"
|
||||
label: "FP8 (0813)"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 960
|
||||
description: "Official DeepSeek-V4-Pro release (2026-08-13) superseding the preview — substantially stronger agentic performance, with the DSpark draft module attached."
|
||||
# DSpark drafting landed in 0.25.0, above this recipe's 0.20.0 baseline.
|
||||
min_vllm_version: "0.25.0"
|
||||
default_modes:
|
||||
spec_decoding: dspark
|
||||
hardware_overrides:
|
||||
blackwell:
|
||||
extra_args:
|
||||
- "--moe-backend"
|
||||
- "deep_gemm_mega_moe"
|
||||
fp8:
|
||||
# Marked "(Preview)" rather than left bare: with the 0813 release on the same
|
||||
# row, an unqualified FP8 pill reads as "the plain one" instead of "the
|
||||
# superseded one". Matches DeepSeek's own naming — their benchmark tables
|
||||
# say `DeepSeek-V4-Pro-0813` vs `DeepSeek-V4-Pro (Preview)`.
|
||||
label: "FP8 (Preview)"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 960
|
||||
description: "Native FP4+FP8 mixed checkpoint (MoE experts FP4, remaining params FP8)"
|
||||
# deep_gemm_mega_moe is FP8-only; NVFP4 selects its backend separately.
|
||||
# Layered on the recipe-level Blackwell FP4 indexer cache.
|
||||
hardware_overrides:
|
||||
blackwell:
|
||||
extra_args:
|
||||
- "--moe-backend"
|
||||
- "deep_gemm_mega_moe"
|
||||
nvfp4:
|
||||
model_id: "nvidia/DeepSeek-V4-Pro-NVFP4"
|
||||
precision: nvfp4
|
||||
vram_minimum_gb: 960
|
||||
description: "NVIDIA modelopt NVFP4 checkpoint (MoE experts NVFP4; attention, shared experts, router head, and MTP stay FP8) for Blackwell GPUs."
|
||||
hardware_overrides:
|
||||
blackwell:
|
||||
extra_args:
|
||||
- "--moe-backend"
|
||||
- "flashinfer_cutedsl"
|
||||
dspark:
|
||||
# Fused DSpark checkpoint — the preview weights with the DSpark draft module
|
||||
# attached. A distinct served repo, so it lives on the Variant axis;
|
||||
# selecting it defaults the spec-decoding method to dspark (below).
|
||||
model_id: "deepseek-ai/DeepSeek-V4-Pro-DSpark"
|
||||
label: "DSpark"
|
||||
precision: fp8
|
||||
vram_minimum_gb: 960
|
||||
description: "Preview FP4+FP8 weights with the DSpark draft module baked in — same base checkpoint as the fp8 variant plus a fused speculative decoder."
|
||||
# DSpark support landed after the recipe baseline (0.20.0), in vLLM 0.25.0.
|
||||
min_vllm_version: "0.25.0"
|
||||
default_modes:
|
||||
spec_decoding: dspark
|
||||
hardware_overrides:
|
||||
blackwell:
|
||||
extra_args:
|
||||
- "--moe-backend"
|
||||
- "deep_gemm_mega_moe"
|
||||
|
||||
redhat_nvfp4:
|
||||
model_id: "RedHatAI/DeepSeek-V4-Pro-NVFP4-FP8"
|
||||
precision: nvfp4
|
||||
label: "NVFP4+FP8 (RedHat)"
|
||||
vram_minimum_gb: 1000
|
||||
description: "RedHatAI NVFP4+FP8 mixed-precision checkpoint"
|
||||
kv_offload_support:
|
||||
offloading_cpu: verified
|
||||
offloading_fs: verified
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- single_node_tep
|
||||
- single_node_dep
|
||||
- multi_node_tep
|
||||
- multi_node_dep
|
||||
- pd_cluster
|
||||
|
||||
# TP+EP is the default deployment; single_node_tp is offered only as a
|
||||
# latency-oriented alternative.
|
||||
default_strategy: single_node_tep
|
||||
|
||||
hardware_overrides:
|
||||
hopper:
|
||||
# H200 (141 GB × 8 = 1128 GB) leaves KV cache tight at 1M context with
|
||||
# ~960 GB mixed-precision weights replicated across DP ranks — cap to 800K.
|
||||
# H20 (no profile here) needs `--max-model-len 200000 --max-num-seqs 16` (#617).
|
||||
extra_args:
|
||||
- "--max-model-len"
|
||||
- "800000"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.95"
|
||||
- "--max-num-seqs"
|
||||
- "512"
|
||||
- "--no-enable-flashinfer-autotune"
|
||||
- "--compilation-config"
|
||||
- '{"mode": 0, "cudagraph_mode": "FULL_DECODE_ONLY"}'
|
||||
blackwell:
|
||||
# Common to every variant on Blackwell. The FP8-only deep_gemm_mega_moe MoE
|
||||
# kernel is added per-variant (FP8 checkpoints), not here.
|
||||
extra_args:
|
||||
- "--attention_config.indexer_kv_dtype=mxfp4"
|
||||
amd:
|
||||
extra_args:
|
||||
- "--distributed-executor-backend"
|
||||
- "mp"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.9"
|
||||
- "--max-num-seqs"
|
||||
- "512"
|
||||
- "--max-num-batched-tokens"
|
||||
- "8192"
|
||||
- "--compilation-config"
|
||||
- '{"mode": 3, "cudagraph_mode": "FULL_DECODE_ONLY"}'
|
||||
extra_env:
|
||||
VLLM_ROCM_USE_AITER: "1"
|
||||
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
# Latency-oriented TP-only deployment: no expert parallelism (single_node_tp
|
||||
# strategy doesn't add --enable-expert-parallel) and no MoE mega-kernel
|
||||
# backend on Blackwell. Autotune is disabled to minimize startup time.
|
||||
hardware_overrides:
|
||||
hopper:
|
||||
extra_args:
|
||||
- "--no-enable-flashinfer-autotune"
|
||||
blackwell:
|
||||
# Replaces recipe-level blackwell override: keeps the FP4 indexer cache
|
||||
# but drops any variant-level --moe-backend selection (TP-only uses
|
||||
# the default MoE backend).
|
||||
extra_args:
|
||||
- "--attention_config.indexer_kv_dtype=mxfp4"
|
||||
- "--no-enable-flashinfer-autotune"
|
||||
single_node_tep:
|
||||
extra_args:
|
||||
- "--compilation-config"
|
||||
- '{"cudagraph_mode":"FULL_AND_PIECEWISE", "custom_ops":["all"]}'
|
||||
multi_node_tep:
|
||||
extra_args:
|
||||
- "--compilation-config"
|
||||
- '{"cudagraph_mode":"FULL_AND_PIECEWISE", "custom_ops":["all"]}'
|
||||
single_node_dep:
|
||||
extra_args:
|
||||
- "--compilation-config"
|
||||
- '{"cudagraph_mode":"FULL_AND_PIECEWISE", "custom_ops":["all"]}'
|
||||
multi_node_dep:
|
||||
extra_args:
|
||||
- "--compilation-config"
|
||||
- '{"cudagraph_mode":"FULL_AND_PIECEWISE", "custom_ops":["all"]}'
|
||||
pd_cluster:
|
||||
env:
|
||||
VLLM_USE_NCCL_SYMM_MEM: "1"
|
||||
NCCL_CUMEM_ENABLE: "1"
|
||||
NCCL_MNNVL_ENABLE: "1"
|
||||
NCCL_NVLS_ENABLE: "1"
|
||||
prefill:
|
||||
nodes:
|
||||
default: 2 # 2 GB200 NVL4 trays = 8 GPUs (~960 GB weights don't fit on 1 tray)
|
||||
gb300: 1 # 1 GB300 NVL4 tray = 4 GPUs (1072 GB/tray fits weights)
|
||||
parallelism: dep # DP + expert-parallel pool
|
||||
vllm_args:
|
||||
- "--max-num-seqs"
|
||||
- "16"
|
||||
- "--max-num-batched-tokens"
|
||||
- "16384"
|
||||
- "--no-disable-hybrid-kv-cache-manager"
|
||||
- "--enable-sleep-mode"
|
||||
env: {}
|
||||
decode:
|
||||
nodes:
|
||||
default: 2 # 2 GB200 NVL4 trays = 8 GPUs
|
||||
gb300: 1 # 1 GB300 NVL4 tray = 4 GPUs
|
||||
parallelism: dep
|
||||
vllm_args:
|
||||
- "--max-num-seqs"
|
||||
- "1024"
|
||||
- "--max-num-batched-tokens"
|
||||
- "1024"
|
||||
- "--max-cudagraph-capture-size"
|
||||
- "1024"
|
||||
- "--compilation-config"
|
||||
- '{"cudagraph_mode":"FULL_DECODE_ONLY"}'
|
||||
- "--no-disable-hybrid-kv-cache-manager"
|
||||
- "--enable-sleep-mode"
|
||||
env: {}
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
DeepSeek-V4-Pro is the flagship of the V4 preview family: a 1.6T-total / 49B-active
|
||||
Mixture-of-Experts model. It pairs a **hybrid attention stack** — Compressed Sparse
|
||||
Attention (CSA) + Heavily Compressed Attention (HCA) — with **Manifold-Constrained
|
||||
Hyper-Connections (mHC)** to reach 27% of V3.2's per-token inference FLOPs and 10% of
|
||||
V3.2's KV cache at 1M context. Pre-trained on 32T+ tokens with the **Muon optimizer**
|
||||
for faster convergence; post-training is a two-stage pipeline (domain-specific expert
|
||||
cultivation + unified consolidation via on-policy distillation).
|
||||
|
||||
Checkpoint is **FP4+FP8 mixed**: MoE expert weights are stored in FP4 while the
|
||||
remaining (attention / norm / router) params stay in FP8.
|
||||
|
||||
An **NVFP4 variant** (`nvidia/DeepSeek-V4-Pro-NVFP4`) is also available — NVIDIA
|
||||
modelopt re-quantizes the MoE experts to standard NVFP4 while attention, shared
|
||||
experts, router head, and MTP stay FP8. Pick it from the **Variant** row; it runs on
|
||||
Blackwell GPUs with the FP4 indexer cache.
|
||||
|
||||
## Checkpoints
|
||||
|
||||
Four checkpoints are on the **Variant** row. They differ in *which weights you serve*
|
||||
and *which draft module ships with them* — the speculative method itself is picked on
|
||||
the **Spec Decoding** row.
|
||||
|
||||
| Variant | Repo | Draft | Notes |
|
||||
| :-- | :-- | :-- | :-- |
|
||||
| **FP8 (0813)** (default) | `deepseek-ai/DeepSeek-V4-Pro-0813` | DSpark | Official release, preview structure + DSpark |
|
||||
| **FP8 (Preview)** | `deepseek-ai/DeepSeek-V4-Pro` | MTP | Preview FP4+FP8 mixed weights |
|
||||
| **NVFP4** | `nvidia/DeepSeek-V4-Pro-NVFP4` | MTP | modelopt re-quant, Blackwell |
|
||||
| **DSpark** | `deepseek-ai/DeepSeek-V4-Pro-DSpark` | DSpark | Preview weights + fused DSpark module |
|
||||
|
||||
**0813 is the official DeepSeek-V4-Pro release** and the default here, superseding
|
||||
the preview with substantially stronger agentic capability — 87.9 on Terminal Bench
|
||||
2.1, 62.7 on DeepSWE and 42.7 on HLE (60.0 with tools), versus 72.1, 12.8 and 37.7
|
||||
for the preview. It is built on the preview model structure with a DSpark speculative
|
||||
decoding module attached. The preview weights remain available as the
|
||||
**FP8 (Preview)** variant for reproducing earlier results.
|
||||
|
||||
**DSpark** is *not* new weights — it is the preview checkpoint with a speculative
|
||||
decoding module attached (see [DeepSpec](https://github.com/deepseek-ai/DeepSpec)).
|
||||
Both fused checkpoints add a `dspark_*` block to `config.json`; the preview and NVFP4
|
||||
checkpoints have no such block, which is why the DSpark method is offered only on the
|
||||
two variants that carry the draft. Selecting either one auto-enables **Spec Decoding**
|
||||
with `method: dspark`, emitting:
|
||||
|
||||
```
|
||||
--speculative-config '{"method":"dspark","num_speculative_tokens":7,"draft_sample_method":"probabilistic"}'
|
||||
```
|
||||
|
||||
The fused checkpoints are ~893 GB on disk versus ~865 GB for the preview — the draft
|
||||
module is the difference. Both require **vLLM 0.25.0**, above this recipe's 0.20.0
|
||||
baseline; the Install block bumps automatically when either is selected.
|
||||
|
||||
## Reasoning modes
|
||||
|
||||
The chat template exposes three reasoning-effort modes:
|
||||
|
||||
- **Non-think** — fast, intuitive responses.
|
||||
- **Think High** — explicit chain-of-thought for complex problem-solving and planning.
|
||||
- **Think Max** — maximum reasoning effort; requires `--max-model-len >= 393216`
|
||||
(384K tokens) to avoid truncation.
|
||||
|
||||
Recommended sampling: `temperature = 1.0`, `top_p = 1.0`.
|
||||
|
||||
On the **0813** variant the effort levels are named `low` / `high` / `max`, and
|
||||
DeepSeek recommends `top_p = 0.95` for agentic scenarios (`1.0` otherwise) with
|
||||
`temperature = 1.0`. Allow up to **384K output tokens** at the `high` and `max`
|
||||
levels.
|
||||
|
||||
Note that the 0813 release ships no Jinja chat template — the repo provides an
|
||||
`encoding/` folder with `encode_messages` / `parse_message_from_completion_text`
|
||||
helpers instead. Serving through vLLM with `--tokenizer-mode deepseek_v4` (the
|
||||
**Tool Calling** pill) applies the built-in DeepSeek-V4 encoding, so the OpenAI-
|
||||
compatible endpoint works without the helper scripts.
|
||||
|
||||
### OpenAI Client Example
|
||||
|
||||
For DeepSeek-V4, keep reasoning controls in `chat_template_kwargs`, as it exposes a
|
||||
custom **Think Max** mode via `"reasoning_effort": "max"`.
|
||||
|
||||
```python
|
||||
from openai import OpenAI
|
||||
|
||||
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
|
||||
model = "deepseek-ai/DeepSeek-V4-Pro-0813"
|
||||
messages = [{"role": "user", "content": "What is 17*19? Return only the final integer."}]
|
||||
|
||||
# Non-think
|
||||
resp = client.chat.completions.create(
|
||||
model=model,
|
||||
messages=messages,
|
||||
)
|
||||
|
||||
# Think High
|
||||
resp = client.chat.completions.create(
|
||||
model=model,
|
||||
messages=messages,
|
||||
extra_body={
|
||||
"chat_template_kwargs": {
|
||||
"thinking": True,
|
||||
"reasoning_effort": "high",
|
||||
},
|
||||
},
|
||||
)
|
||||
|
||||
# Think Max
|
||||
resp = client.chat.completions.create(
|
||||
model=model,
|
||||
messages=messages,
|
||||
extra_body={
|
||||
"chat_template_kwargs": {
|
||||
"thinking": True,
|
||||
"reasoning_effort": "max",
|
||||
},
|
||||
},
|
||||
)
|
||||
```
|
||||
|
||||
## Recommended deployments
|
||||
|
||||
- **B300 (8× GPU)**: single-node DP + EP with `--data-parallel-size 8`.
|
||||
- **H200 (8× GPU)**: DP + EP with `--data-parallel-size 8`. Context is capped at
|
||||
800K tokens (`--max-model-len 800000`) to leave KV headroom with dense params
|
||||
replicated across ranks — applies to both single-node and multi-node H200.
|
||||
- **MI355X (8× GPU)**: validated with ROCm + AITER (`VLLM_ROCM_USE_AITER=1`),
|
||||
`--gpu-memory-utilization 0.9`, `--max-num-seqs 128`,
|
||||
`--max-num-batched-tokens 8192`, and `--distributed-executor-backend mp`.
|
||||
- **GB200 NVL4 (4× GPU per tray)**: the ~960 GB mixed-precision checkpoint does not
|
||||
fit on one tray; run multi-node DP + EP across **2 trays** (8 GPUs total) with
|
||||
`--data-parallel-size 8`. Pick the "Multi-Node" tab and set nodes to 2.
|
||||
|
||||
### MI355X (8×288GB)
|
||||
|
||||
```bash
|
||||
export VLLM_ROCM_USE_AITER=1
|
||||
|
||||
vllm serve deepseek-ai/DeepSeek-V4-Pro \
|
||||
--host localhost \
|
||||
--port 8001 \
|
||||
--dtype auto \
|
||||
--kv-cache-dtype fp8 \
|
||||
--tensor-parallel-size 8 \
|
||||
--max-num-seqs 512 \
|
||||
--max-num-batched-tokens 8192 \
|
||||
--distributed-executor-backend mp \
|
||||
--trust-remote-code \
|
||||
--gpu-memory-utilization 0.9 \
|
||||
--tokenizer-mode deepseek_v4 \
|
||||
--reasoning-parser deepseek_v4 \
|
||||
--tool-call-parser deepseek_v4 \
|
||||
--enable-auto-tool-choice \
|
||||
--compilation-config '{"mode": 3, "cudagraph_mode": "FULL_DECODE_ONLY"}'
|
||||
```
|
||||
|
||||
#### MI355X is validated on GSM8K dataset:
|
||||
|
||||
##### Launch command
|
||||
|
||||
```bash
|
||||
MODEL=deepseek-ai/DeepSeek-V4-Pro
|
||||
lm_eval --model local-completions \
|
||||
--model_args model=$MODEL,base_url=http://0.0.0.0:8001/v1/completions,num_concurrent=128,max_retries=10,max_gen_toks=2048,timeout=60000 \
|
||||
--batch_size auto \
|
||||
--tasks gsm8k \
|
||||
--num_fewshot 8 \
|
||||
--output_path . 2>&1 | tee -a eval.log
|
||||
```
|
||||
|
||||
##### Reported result
|
||||
|
||||
```text
|
||||
local-completions ({'model': 'deepseek-ai/DeepSeek-V4-Pro', 'base_url': 'http://0.0.0.0:8001/v1/completions', 'num_concurrent': 128, 'max_retries': 10, 'max_gen_toks': 2048, 'timeout': 60000}), gen_kwargs: ({}), limit: None, num_fewshot: 8, batch_size: auto
|
||||
|Tasks|Version| Filter |n-shot| Metric | |Value | |Stderr|
|
||||
|-----|------:|----------------|-----:|-----------|---|-----:|---|-----:|
|
||||
|gsm8k| 3|flexible-extract| 8|exact_match|↑ |0.9538|± |0.0058|
|
||||
| | |strict-match | 8|exact_match|↑ |0.9545|± |0.0057|
|
||||
```
|
||||
|
||||
## KV Cache Offloading
|
||||
|
||||
Agentic and multi-turn workloads reuse long prefixes whose KV state can exceed on-GPU
|
||||
capacity. The **KV Offload** row attaches a host-DRAM KV tier to any serving strategy —
|
||||
pick one of three connectors:
|
||||
|
||||
- **Simple** — `SimpleCPUOffloadConnector`: spills KV blocks to a per-rank region of CPU
|
||||
DRAM on each node. The simplest way to extend effective KV capacity for a single instance.
|
||||
- **Mooncake** — `MooncakeStoreConnector`: pools CPU DRAM into a distributed shared KV
|
||||
store — either embedded (each rank's vLLM worker runs a Mooncake client that donates a DRAM segment)
|
||||
or standalone (a per-node `mooncake_store_service` owns the node's DRAM and contributes
|
||||
it to the pool, decoupling cache lifetime from the engine). A cluster-wide
|
||||
`mooncake_master` coordinates the pool; across 2+ instances, GPU workers share it for
|
||||
cross-instance prefix reuse. Install from the extra-install block above; see the [Mooncake docs](https://kvcache-ai.github.io/Mooncake/).
|
||||
- **LMCache** — `LMCacheMPConnector`: a node-local KV pool served by a
|
||||
companion `lmcache server` process, launched before `vllm serve`.
|
||||
Single-node strategies only. Install from the extra-install block above; see the [LMCache docs](https://docs.lmcache.ai/).
|
||||
|
||||
## Agentic serving on GB200/GB300 (DSpark)
|
||||
|
||||
For agentic serving, use the following common settings with
|
||||
[DeepSeek-V4-Pro-0813](https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro-0813/tree/72e1d3230f6c080a530b0a1d46f8eb4602340597)
|
||||
and its native DSpark draft. These are separate from the command-builder defaults.
|
||||
|
||||
### Common settings
|
||||
|
||||
Set `VLLM_USE_V2_MODEL_RUNNER=1` and use these engine arguments:
|
||||
|
||||
```text
|
||||
--tokenizer-mode deepseek_v4 --trust-remote-code
|
||||
--kv-cache-dtype fp8 --block-size 256
|
||||
--no-disable-hybrid-kv-cache-manager --no-enable-flashinfer-autotune
|
||||
--attention-config '{"backend":"FLASHINFER_MLA_SPARSE_DSV4","use_prefill_query_quantization":true,"indexer_kv_dtype":"mxfp4"}'
|
||||
--speculative-config '{"method":"dspark","num_speculative_tokens":6,"draft_sample_method":"probabilistic","rejection_sample_method":"block"}'
|
||||
```
|
||||
|
||||
The example uses `K=6`, where `K` is `num_speculative_tokens`. Tune K and the
|
||||
prefill/decode (PD) ratio for each target concurrency to balance throughput,
|
||||
per-user latency and time to first token. Use the same K on both PD roles.
|
||||
|
||||
For aggregate TP serving, use
|
||||
`--disable-custom-all-reduce`, `VLLM_ALLREDUCE_USE_FLASHINFER=1`,
|
||||
`VLLM_FLASHINFER_ALLREDUCE_BACKEND=auto`; do not select an EP MoE backend.
|
||||
For DEP (TP1 with data and expert parallelism), both pools instead use
|
||||
`--enable-expert-parallel --enable-ep-weight-filter --enable-cumem-allocator`,
|
||||
`--moe-backend deep_gemm_amxf4_mega_moe` and `VLLM_DSV4_MEGA_FP8_COMBINE=1`.
|
||||
This MoE configuration supports the native mixed FP4/FP8 checkpoint, unlike
|
||||
the FP8-only `deep_gemm_mega_moe` setting above.
|
||||
|
||||
For aggregate and decode, set `cudagraph_mode=FULL_DECODE_ONLY` and `mode=0`
|
||||
in `--compilation-config`, with explicit capture sizes `n*(K+1)` for `n=1..S`.
|
||||
`S` is `max-num-seqs` per DP rank, or per engine for pure TP. Set the maximum
|
||||
capture size to `S*(K+1)`; aggregate serving can also include `n*K` sizes.
|
||||
Recompute the capture sizes when K or S changes. Size `max-num-batched-tokens`
|
||||
to cover verification tokens plus any prefill work scheduled on that engine.
|
||||
|
||||
### Scheduling, NUMA binding, and streaming
|
||||
|
||||
**DEP decode scheduling:** the decode example below uses:
|
||||
|
||||
```text
|
||||
--prefill-schedule-interval 16 --long-prefill-token-threshold 512
|
||||
```
|
||||
|
||||
The interval controls how often DP ranks admit new prefill requests. Increasing
|
||||
it can balance work across ranks but delays admission. The threshold limits
|
||||
long-prefill chunk size when requests share the token budget.
|
||||
Compare these overrides with leaving both flags unset, measuring throughput,
|
||||
time to first token, and inter-token latency at your target load. Retune when
|
||||
changing the workload, pool sizes, hardware, or engine version.
|
||||
|
||||
The prefill example uses `--long-prefill-token-threshold 1024` and leaves
|
||||
`--prefill-schedule-interval` unset.
|
||||
|
||||
**DEP NUMA binding:** on every prefill and decode node, enable binding and
|
||||
provide one NUMA node ID per visible GPU. For the four-GPU layout where GPUs
|
||||
0/1 are local to NUMA node 0 and GPUs 2/3 to NUMA node 1, use:
|
||||
|
||||
```text
|
||||
--numa-bind --numa-bind-nodes 0 0 1 1
|
||||
```
|
||||
|
||||
Match the list to the local GPU order and topology on each node, including
|
||||
GB300 deployments. See the [vLLM NUMA binding guide](https://docs.vllm.ai/en/latest/configuration/optimization/#numa-binding-for-multi-socket-gpu-nodes)
|
||||
for automatic detection and explicit mappings.
|
||||
|
||||
**Aggregate streaming:** add `--stream-interval 10` to the aggregate TP8
|
||||
worker at concurrency 1, 4, or 8. This batches streamed output into larger
|
||||
updates to reduce host overhead; use a smaller interval when smoother token
|
||||
delivery matters more. The DEP configurations leave this override unset.
|
||||
|
||||
### Running with Dynamo
|
||||
|
||||
The example below runs a DEP8 prefill pool and a DEP8 decode pool, each across
|
||||
two four-GPU GB200 nodes, plus a host for shared services and the frontend.
|
||||
Run the Python commands in `vllm/vllm-openai:nightly-dev-arm64-cu13.0.1-20ebb28`
|
||||
with the model, recipe files, GPUs, and RDMA devices accessible;
|
||||
use host networking so the advertised addresses are reachable across nodes.
|
||||
Install the matching Dynamo packages in each frontend/worker container:
|
||||
|
||||
```bash
|
||||
python3 -m pip install --extra-index-url https://pypi.nvidia.com \
|
||||
ai-dynamo==1.3.0.dev20260720 ai-dynamo-runtime==1.3.0.dev20260720
|
||||
```
|
||||
|
||||
Save the following as `dynamo-env.sh` and source it in each shell. Replace the
|
||||
example addresses with routable addresses on your cluster. `MODEL_PATH` must
|
||||
contain the pinned 0813 checkpoint on every worker.
|
||||
|
||||
```bash
|
||||
export INFRA_IP=10.0.0.10
|
||||
export ETCD_ENDPOINTS="http://${INFRA_IP}:2379"
|
||||
export NATS_SERVER="nats://${INFRA_IP}:4222"
|
||||
export DYN_NAMESPACE=dspark
|
||||
export MODEL_PATH=/models/DeepSeek-V4-Pro-0813
|
||||
export SERVED_MODEL=deepseek-ai/DeepSeek-V4-Pro-0813
|
||||
export PREFILL_HEAD_IP=10.0.0.11
|
||||
export DECODE_HEAD_IP=10.0.0.21
|
||||
```
|
||||
|
||||
**Shared services.** Provide `etcd`, `nats-server`, and `mooncake_master` on the
|
||||
service host's `PATH`. See the official [Dynamo service setup](https://github.com/ai-dynamo/dynamo/blob/59dd504105db498b11e5715854a2512d57b7a1e0/dev/docker-compose.yml)
|
||||
and [Mooncake installation](https://kvcache-ai.github.io/Mooncake/) for installation
|
||||
options. Start these services once on the private cluster network, before workers:
|
||||
|
||||
```bash
|
||||
source ./dynamo-env.sh
|
||||
etcd --data-dir ./etcd-data --listen-client-urls http://0.0.0.0:2379 \
|
||||
--advertise-client-urls "$ETCD_ENDPOINTS" >etcd.log 2>&1 &
|
||||
cat >nats.conf <<'EOF'
|
||||
port: 4222
|
||||
max_payload: 33554432
|
||||
jetstream { store_dir: "./nats-data" }
|
||||
EOF
|
||||
nats-server -c ./nats.conf >nats.log 2>&1 &
|
||||
mooncake_master --port=8700 --enable_http_metadata_server=true \
|
||||
--http_metadata_server_port=8701 --eviction_high_watermark_ratio=0.9 \
|
||||
--default_kv_lease_ttl=10000 --rpc_thread_num=16 \
|
||||
--enable_metric_reporting=true --metrics_port=8702 >mooncake.log 2>&1 &
|
||||
```
|
||||
|
||||
Wait for the services to listen before continuing. All hosts must reach the
|
||||
shared endpoints; workers must also reach each other's DP rendezvous and NIXL
|
||||
addresses. Existing managed services can supply the same endpoints.
|
||||
|
||||
**Frontend.** In a separate shell, start the HTTP endpoint for the disaggregated
|
||||
deployment:
|
||||
|
||||
```bash
|
||||
source ./dynamo-env.sh
|
||||
export DYN_ROUTER_ACTIVE_REQUEST_EXPIRY_SECS=3600
|
||||
export DYN_TCP_CHANNEL_BUFFER=128
|
||||
export DYN_TCP_REQUEST_TIMEOUT=60
|
||||
python3 -m dynamo.frontend --http-port 8000 --router-mode random \
|
||||
--router-session-affinity-ttl-secs 900
|
||||
```
|
||||
|
||||
- Aggregate: `--router-mode kv`, `DYN_ROUTER_TEMPERATURE=0` and
|
||||
`DYN_TOKENIZER=fastokens`.
|
||||
- Disaggregated: `--router-mode random` with session affinity, or `kv` with
|
||||
the same tokenizer settings. Send a stable conversation
|
||||
ID as `x-dynamo-session-id` and set `--router-session-affinity-ttl-secs` to
|
||||
cover the conversation lifetime.
|
||||
- Both DEP roles use `MultiConnector` with `NixlConnector` for PD transfer and
|
||||
`MooncakeStoreConnector` for shared prefix storage. Enable store lookup only
|
||||
on prefill, and set `VLLM_CONNECTOR_PREFETCH_DEPTH=0` there. Role-specific
|
||||
`--kv-transfer-config` values:
|
||||
|
||||
Prefill — save as `prefill-kv.json` in the worker's working directory:
|
||||
|
||||
```json
|
||||
{"kv_connector":"MultiConnector","kv_role":"kv_both","kv_connector_extra_config":{"connectors":[{"kv_connector":"NixlConnector","kv_role":"kv_both","kv_load_failure_policy":"fail","kv_buffer_device":"cuda","kv_connector_extra_config":{"enforce_handshake_compat":false,"enable_cross_layers_blocks":false}},{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_both","kv_connector_extra_config":{"load_async":true,"lookup_async":true,"enable_cross_layers_blocks":false,"enable_offload":false}}]}}
|
||||
```
|
||||
|
||||
Decode — save as `decode-kv.json` in the worker's working directory:
|
||||
|
||||
```json
|
||||
{"kv_connector":"MultiConnector","kv_role":"kv_consumer","kv_connector_extra_config":{"connectors":[{"kv_connector":"NixlConnector","kv_role":"kv_consumer","kv_load_failure_policy":"fail","kv_buffer_device":"cuda","kv_connector_extra_config":{"enforce_handshake_compat":false,"enable_cross_layers_blocks":false}},{"kv_connector":"MooncakeStoreConnector","kv_role":"kv_consumer","kv_connector_extra_config":{"load_async":true,"lookup_async":false,"enable_lookup":false,"enable_cross_layers_blocks":false,"enable_offload":false}}]}}
|
||||
```
|
||||
|
||||
**Worker environment.** Run the following in Bash on each worker node. Set
|
||||
`NODE_IP` to that node's routable address and `NODE_RANK` to 0 or 1 within its
|
||||
own pool. For example, prefill nodes `10.0.0.11` / `10.0.0.12` use ranks 0 / 1;
|
||||
decode nodes `10.0.0.21` / `10.0.0.22` also use ranks 0 / 1. All four GPUs are
|
||||
visible to one launch process per node. Adjust the NUMA and NIC mappings to
|
||||
match the hardware. The embedded Mooncake store uses 140 GB per local rank
|
||||
in this example; ensure enough host memory or resize it for your deployment.
|
||||
|
||||
```bash
|
||||
source ./dynamo-env.sh
|
||||
export NODE_IP=10.0.0.11 # Change on each worker node.
|
||||
NODE_RANK=0 # Use 1 on the second node of either pool.
|
||||
export CUDA_VISIBLE_DEVICES=0,1,2,3
|
||||
export VLLM_HOST_IP="$NODE_IP"
|
||||
export VLLM_NIXL_SIDE_CHANNEL_HOST="$NODE_IP"
|
||||
export VLLM_NIXL_SIDE_CHANNEL_PORT=20096
|
||||
export DYN_SYSTEM_PORT=8081
|
||||
export VLLM_USE_V2_MODEL_RUNNER=1 VLLM_DSV4_MEGA_FP8_COMBINE=1
|
||||
export PYTHONHASHSEED=0
|
||||
export MOONCAKE_MASTER="${INFRA_IP}:8700"
|
||||
export MOONCAKE_TE_META_DATA_SERVER="http://${INFRA_IP}:8701/metadata"
|
||||
export MOONCAKE_LOCAL_HOSTNAME="$NODE_IP"
|
||||
export MOONCAKE_CONFIG_PATH="$PWD/mooncake.json"
|
||||
cat >"$MOONCAKE_CONFIG_PATH" <<EOF
|
||||
{"metadata_server":"P2PHANDSHAKE",
|
||||
"master_server_address":"${MOONCAKE_MASTER}","global_segment_size":"140GB",
|
||||
"local_buffer_size":"4GB","protocol":"rdma",
|
||||
"device_name":"mlx5_0,mlx5_1,mlx5_2,mlx5_3","mode":"embedded","enable_offload":false}
|
||||
EOF
|
||||
|
||||
COMMON_ARGS=(
|
||||
--model "$MODEL_PATH" --served-model-name "$SERVED_MODEL"
|
||||
--tensor-parallel-size 1 --pipeline-parallel-size 1
|
||||
--data-parallel-size 8 --data-parallel-size-local 4
|
||||
--data-parallel-start-rank "$((4 * NODE_RANK))"
|
||||
--data-parallel-rpc-port 13345 --data-parallel-hybrid-lb
|
||||
--enable-expert-parallel --enable-ep-weight-filter --enable-cumem-allocator
|
||||
--moe-backend deep_gemm_amxf4_mega_moe
|
||||
--numa-bind --numa-bind-nodes 0 0 1 1
|
||||
--tokenizer-mode deepseek_v4 --trust-remote-code
|
||||
--max-model-len 1048576 --kv-cache-dtype fp8 --block-size 256
|
||||
--no-disable-hybrid-kv-cache-manager --no-enable-flashinfer-autotune
|
||||
--attention-config '{"backend":"FLASHINFER_MLA_SPARSE_DSV4","use_prefill_query_quantization":true,"indexer_kv_dtype":"mxfp4"}'
|
||||
--speculative-config '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block"}'
|
||||
)
|
||||
```
|
||||
|
||||
**Prefill workers.** Run once on each prefill node after the worker environment
|
||||
block, keeping `PREFILL_HEAD_IP` identical across the pool:
|
||||
|
||||
```bash
|
||||
export VLLM_CONNECTOR_PREFETCH_DEPTH=0
|
||||
python3 -m dynamo.vllm "${COMMON_ARGS[@]}" --disaggregation-mode prefill \
|
||||
--data-parallel-address "$PREFILL_HEAD_IP" \
|
||||
--kv-transfer-config "$(cat prefill-kv.json)" \
|
||||
--max-num-seqs 64 --max-num-batched-tokens 8192 \
|
||||
--long-prefill-token-threshold 1024 --gpu-memory-utilization 0.85 \
|
||||
--max-cudagraph-capture-size 384
|
||||
```
|
||||
|
||||
**Decode workers.** Run once on each decode node after the worker environment
|
||||
block. This example uses K=5 and S=8, so graph capture sizes are 6 through 48:
|
||||
|
||||
```bash
|
||||
python3 -m dynamo.vllm "${COMMON_ARGS[@]}" --disaggregation-mode decode \
|
||||
--data-parallel-address "$DECODE_HEAD_IP" \
|
||||
--kv-transfer-config "$(cat decode-kv.json)" \
|
||||
--max-num-seqs 8 --max-num-batched-tokens 8240 \
|
||||
--prefill-schedule-interval 16 --long-prefill-token-threshold 512 \
|
||||
--gpu-memory-utilization 0.9 --kv-cache-memory-bytes 37580963840 \
|
||||
--max-cudagraph-capture-size 48 \
|
||||
--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY","mode":0,"cudagraph_capture_sizes":[6,12,18,24,30,36,42,48]}'
|
||||
```
|
||||
|
||||
For DEP12/DEP16, set `--data-parallel-size` to 12/16 in the corresponding
|
||||
pool and run on three/four four-GPU nodes. Each pool keeps its own head address;
|
||||
start ranks are `4 * NODE_RANK` (0, 4, 8, 12). Tune scheduling for the target
|
||||
workload and recompute graph sizes when changing K or S. The same port numbers can be
|
||||
reused on separate hosts; co-located workers need distinct port allocations.
|
||||
The [official Dynamo vLLM examples](https://github.com/ai-dynamo/dynamo/tree/59dd504105db498b11e5715854a2512d57b7a1e0/examples/backends/vllm/launch)
|
||||
provide the aggregate and KV-routing launch patterns; apply this section's
|
||||
model and aggregate settings when using those patterns.
|
||||
|
||||
Once all workers register, send requests to the frontend, retaining the same
|
||||
session ID for turns in one conversation:
|
||||
|
||||
```bash
|
||||
curl "http://${INFRA_IP}:8000/v1/chat/completions" \
|
||||
-H 'Content-Type: application/json' -H 'x-dynamo-session-id: example-session' \
|
||||
-d '{"model":"deepseek-ai/DeepSeek-V4-Pro-0813","messages":[{"role":"user","content":"Hello"}],"max_tokens":32}'
|
||||
```
|
||||
|
||||
## Agentic benchmark reproduction (InferenceX MI355X sweep)
|
||||
|
||||
The command above is the default serving configuration. The SemiAnalysis InferenceX
|
||||
MI355X benchmark lane for this checkpoint is a separate high-concurrency benchmark
|
||||
configuration — not the recipe default. To reproduce that sweep, start from the
|
||||
default command above and apply these deltas:
|
||||
|
||||
**Source configuration:**
|
||||
|
||||
```yaml
|
||||
image: vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657
|
||||
```
|
||||
|
||||
**Applied to both arms:**
|
||||
|
||||
- Enable `VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4` (previously commented out).
|
||||
- Disable `VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=0` for this nightly.
|
||||
DSv4-Pro is a mixed checkpoint (MXFP4 routed experts, FP8 shared expert), and
|
||||
this image fails while loading incompatible scales and shapes when the fused
|
||||
shared-expert path is enabled.
|
||||
- Raise `--gpu-memory-utilization` from `0.8` to `0.86`.
|
||||
- Pin `--max-num-batched-tokens 8192` instead of the nightly default of `16384`.
|
||||
On the TP8 initialization check this raised GPU KV-cache capacity from `4,730,981`
|
||||
to `8,524,228` tokens and reduced peak activation memory from `11.44 GiB` to
|
||||
`8.9 GiB`.
|
||||
|
||||
**Applied to the DP-attention arm only:**
|
||||
|
||||
- Cap `--max-num-seqs` at `CONC` rather than `2*CONC`. The limit is per scheduler
|
||||
and DP-attention runs one scheduler per rank. The pure TP8 arm keeps the existing
|
||||
`2*CONC` headroom for AgentX subagent fan-out.
|
||||
- Add `--prefill-schedule-interval 8` and `--long-prefill-token-threshold 16384`.
|
||||
|
||||
With `--max-num-batched-tokens 8192` now pinned for both arms,
|
||||
`--long-prefill-token-threshold 16384` sits above the token budget and therefore
|
||||
never binds. It is inert rather than harmful, and the DP-attention point was
|
||||
measured with it present.
|
||||
|
||||
`--compilation-config '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE"}'` is
|
||||
unchanged; it was already in the checked-in recipe.
|
||||
|
||||
Export the AITER and INT4 quantized all-reduce environment variables before launch:
|
||||
|
||||
```bash
|
||||
export VLLM_ROCM_USE_AITER=1
|
||||
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4
|
||||
export VLLM_ROCM_USE_AITER_MOE=1
|
||||
export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=0
|
||||
export VLLM_ENGINE_READY_TIMEOUT_S=10800
|
||||
|
||||
vllm serve "$MODEL_PATH" --served-model-name "$MODEL" \
|
||||
--host 0.0.0.0 \
|
||||
--port "$VLLM_BACKEND_PORT" \
|
||||
--trust-remote-code \
|
||||
--async-scheduling \
|
||||
--distributed-executor-backend mp \
|
||||
--kv-cache-dtype fp8 \
|
||||
--max-num-batched-tokens 8192 \
|
||||
--gpu-memory-utilization 0.86 \
|
||||
--moe-backend aiter \
|
||||
--compilation-config '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE"}' \
|
||||
--speculative-config '{"method": "mtp", "num_speculative_tokens": 3, "rejection_sample_method": "synthetic", "synthetic_acceptance_length": 2.49}' \
|
||||
--tokenizer-mode deepseek_v4 \
|
||||
--tool-call-parser deepseek_v4 \
|
||||
--reasoning-parser deepseek_v4 \
|
||||
--enable-auto-tool-choice \
|
||||
--enable-prefix-caching \
|
||||
--prefix-cache-retention-interval 32768 \
|
||||
--no-disable-hybrid-kv-cache-manager \
|
||||
--max-num-seqs "8"
|
||||
```
|
||||
|
||||
`--prefix-cache-retention-interval` replaces the `VLLM_PREFIX_CACHE_RETENTION_INTERVAL`
|
||||
env var, removed in vLLM 0.30.0
|
||||
([vllm-project/vllm#55353](https://github.com/vllm-project/vllm/pull/55353)).
|
||||
|
||||
## Fixed-sequence benchmark reproduction (InferenceX MI355X 8K/1K)
|
||||
|
||||
This is a **second, separate InferenceX lane** — not a variation of the Agentic
|
||||
sweep above. It is a fixed-sequence measurement at input length 8192 and output
|
||||
length 1024 on one 8× MI355X node, run as two arms: **STP** (no speculative
|
||||
decoding) and **real MTP with two draft tokens**. Neither arm is the general
|
||||
recipe default, and neither is a recommendation for other AMD workloads.
|
||||
|
||||
The checkpoint is `deepseek-ai/DeepSeek-V4-Pro` — the **FP8 (Preview)** pill in
|
||||
the Variant row, whose mixed FP4+FP8 weights InferenceX labels `fp4`. It is not
|
||||
the `0813` default variant and not the DSpark checkpoint.
|
||||
|
||||
### Source configuration
|
||||
|
||||
| Item | Value |
|
||||
|---|---|
|
||||
| InferenceX PR | [#2792](https://github.com/SemiAnalysisAI/InferenceX/pull/2792) |
|
||||
| Config keys | `dsv4-fp4-mi355x-vllm` (STP), `dsv4-fp4-mi355x-vllm-mtp` (MTP) |
|
||||
| Hardware | one node, 8× MI355X |
|
||||
| Image | `vllm/vllm-openai-rocm:nightly-7c5dc571cbd1064ecc8a9b1045637ff647aa22cb` |
|
||||
| Image digest | `sha256:f0bdaf5217a09949842b45c1ea1f12260d3205ec81f143b320dfc2eb3ec95e55` |
|
||||
| Parallelism | TP 8, DP 1, EP 1 — no `--enable-expert-parallel` |
|
||||
| Workload | ISL 8192, OSL 1024; concurrency 4, 8, 16, 32, 64, 128, 256, 512 |
|
||||
| Requests per point | 10 × concurrency |
|
||||
|
||||
The image is pinned to an immutable nightly tag rather than a floating
|
||||
`:nightly`, because the measurements depend on the ROCm DeepSeek-V4 kernel state
|
||||
in that specific build. The Install block's AMD default stays on `:nightly`
|
||||
deliberately — pin the tag above only when reproducing these numbers.
|
||||
|
||||
### STP launch (no speculative decoding)
|
||||
|
||||
```bash
|
||||
export VLLM_ROCM_USE_AITER=1
|
||||
export VLLM_ROCM_USE_AITER_MOE=1
|
||||
# Fuses the FP8 shared expert into the MXFP4 routed-expert AITER kernel.
|
||||
# See the shared-expert note below for the build and eligibility requirements.
|
||||
export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1
|
||||
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4
|
||||
|
||||
vllm serve deepseek-ai/DeepSeek-V4-Pro \
|
||||
--port "${PORT:-8000}" \
|
||||
--tensor-parallel-size 8 \
|
||||
--data-parallel-size 1 \
|
||||
--async-scheduling \
|
||||
--no-enable-prefix-caching \
|
||||
--distributed-executor-backend mp \
|
||||
--gpu-memory-utilization 0.8 \
|
||||
--kv-cache-dtype fp8 \
|
||||
--trust-remote-code \
|
||||
--moe-backend aiter \
|
||||
--tokenizer-mode deepseek_v4 \
|
||||
--reasoning-parser deepseek_v4 \
|
||||
--compilation-config '{"mode":3,"cudagraph_mode":"FULL_AND_PIECEWISE"}'
|
||||
```
|
||||
|
||||
### MTP arm (real speculation, 2 draft tokens)
|
||||
|
||||
The MTP arm is the same launch plus one flag. Everything else — environment,
|
||||
parallelism, memory, cache, executor, MoE backend, parsers, compilation — is
|
||||
identical:
|
||||
|
||||
```bash
|
||||
--speculative-config '{"method":"mtp","num_speculative_tokens":2}'
|
||||
```
|
||||
|
||||
This is **real** MTP: draft tokens are verified against the target model. ROCm
|
||||
support for the DeepSeek-V4 MTP head landed in
|
||||
[vllm-project/vllm#43385](https://github.com/vllm-project/vllm/pull/43385). The
|
||||
same two-token config is available from the command builder — enable *Spec
|
||||
decoding* and pick the **MTP** mode with the FP8 (Preview) variant selected.
|
||||
|
||||
### Left at engine defaults
|
||||
|
||||
This lane deliberately passes no `--max-model-len`, `--max-num-seqs`,
|
||||
`--max-num-batched-tokens`, or `--block-size`, and no tool-calling flags. That
|
||||
differs from this recipe's general AMD configuration, which does set several of
|
||||
them — one reason this reproduction is documented separately rather than folded
|
||||
into the default command.
|
||||
|
||||
### Benchmarking the MTP arm needs chat formatting
|
||||
|
||||
MTP-style speculation is trained against chat-formatted input. Benchmarking it
|
||||
with raw completion prompts silently depresses the acceptance rate, so measure
|
||||
it against the chat endpoint, which applies the DeepSeek-V4 chat template that
|
||||
`--tokenizer-mode deepseek_v4` installs:
|
||||
|
||||
```bash
|
||||
CONC=32
|
||||
|
||||
vllm bench serve \
|
||||
--model deepseek-ai/DeepSeek-V4-Pro \
|
||||
--host localhost \
|
||||
--port "${PORT:-8000}" \
|
||||
--backend openai-chat \
|
||||
--endpoint /v1/chat/completions \
|
||||
--dataset-name random \
|
||||
--random-input-len 8192 \
|
||||
--random-output-len 1024 \
|
||||
--num-prompts "$((CONC * 10))" \
|
||||
--max-concurrency "$CONC" \
|
||||
--ignore-eos \
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
InferenceX drives its own benchmark client with a private DeepSeek-V4 encoder
|
||||
rather than the tokenizer's built-in template, so the numbers on its dashboard
|
||||
are not byte-identical to what the command above produces. Use the chat endpoint
|
||||
for any MTP acceptance measurement regardless.
|
||||
|
||||
### Shared-expert fusion requires a recent build
|
||||
|
||||
`VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1` is a request, not a guarantee:
|
||||
vLLM gates the fused shared-expert path on its own eligibility check.
|
||||
[vllm-project/vllm#53161](https://github.com/vllm-project/vllm/pull/53161),
|
||||
merged as `de69e821`, added the heterogeneous path that fuses the native-FP8
|
||||
shared expert into the MXFP4 routed-expert kernel. Current `:nightly` images
|
||||
include it, so the flag takes effect. On any build older than `de69e821` the
|
||||
check rejects this mixed FP4+FP8 checkpoint outright and the fusion
|
||||
self-disables at startup — including the immutable tag pinned in the
|
||||
reproduction table above, which predates the fix.
|
||||
|
||||
On a build that includes it, the fusion additionally requires gfx950 (MI355X),
|
||||
tensor parallelism 8, data parallelism 1, no expert parallelism and no EPLB,
|
||||
`--moe-backend aiter`, BF16 model dtype, and the DeepSeek-V4 block-128 FP8
|
||||
shared-expert quantization config. Both arms here satisfy this. Data-parallel
|
||||
attention does not, and keeps the unfused path.
|
||||
|
||||
Eligibility is also re-checked per invocation against AITER's tuned-config
|
||||
coverage for the number of MoE input rows. Decode-sized batches are covered and
|
||||
take the fused kernel; large prefill chunks fall back to the unfused path, so an
|
||||
8K-input workload runs fused during decode and unfused during prefill.
|
||||
|
||||
Failures are silent and logged at debug level, so check the server log before
|
||||
assuming the fused path executed rather than inferring it from the flag. The
|
||||
line to look for is `Using AITER DeepSeek V4 heterogeneous MoE for CSV-covered
|
||||
MoE input rows`.
|
||||
|
||||
### How this differs from the Agentic sweep above
|
||||
|
||||
| Setting | Fixed-sequence 8K/1K | Agentic sweep |
|
||||
|---|---|---|
|
||||
| Workload | ISL 8192 / OSL 1024, conc 4–512 | agentic-coding traces |
|
||||
| `--gpu-memory-utilization` | `0.8` | `0.86` |
|
||||
| Prefix caching | `--no-enable-prefix-caching` | `--enable-prefix-caching` |
|
||||
| Speculation | none (STP) or real MTP, 2 tokens | synthetic MTP, 3 tokens |
|
||||
| `--max-num-seqs` | not passed | `8` |
|
||||
| `--max-num-batched-tokens` | not passed | `8192` |
|
||||
|
||||
The Agentic lane's synthetic acceptance length is a benchmark-comparability
|
||||
device. This lane never uses it: the STP arm has no draft at all, and the MTP
|
||||
arm verifies against the real target.
|
||||
|
||||
### Provenance
|
||||
|
||||
Validation for this configuration comes from InferenceX, not from this
|
||||
repository — no MI355X benchmark was rerun here.
|
||||
|
||||
- InferenceX PR: [#2792](https://github.com/SemiAnalysisAI/InferenceX/pull/2792)
|
||||
- STP script: [`dsv4_fp4_mi355x_vllm.sh`](https://github.com/SemiAnalysisAI/InferenceX/blob/7393456f925f4962fc9bb6438567e73f7ca12bf8/benchmarks/single_node/fixed_seq_len/dsv4_fp4_mi355x_vllm.sh)
|
||||
- MTP script: [`dsv4_fp4_mi355x_vllm_mtp.sh`](https://github.com/SemiAnalysisAI/InferenceX/blob/7393456f925f4962fc9bb6438567e73f7ca12bf8/benchmarks/single_node/fixed_seq_len/dsv4_fp4_mi355x_vllm_mtp.sh)
|
||||
- Config matrix: [`configs/amd-master.yaml`](https://github.com/SemiAnalysisAI/InferenceX/blob/7393456f925f4962fc9bb6438567e73f7ca12bf8/configs/amd-master.yaml)
|
||||
- Successful sweep: [run 33538769698](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/33538769698),
|
||||
which ran at commit `7a209b121932696b7efd437ac50d19c53efc7a94`. All eight
|
||||
concurrency points passed on both arms, plus `lm-eval` GSM8K eval-only jobs at
|
||||
concurrency 128 and 512. Commits after that point on the PR branch do not touch
|
||||
either fixed-sequence script or these two config entries.
|
||||
- ROCm DeepSeek-V4 MTP support: [vllm-project/vllm#43385](https://github.com/vllm-project/vllm/pull/43385)
|
||||
@@ -0,0 +1,846 @@
|
||||
# DeepSeek-V4.1-Flash — serves /deepseek-ai/DeepSeek-V4.1-Flash. Validate: node scripts/build-recipes-api.mjs
|
||||
meta:
|
||||
title: "DeepSeek-V4.1-Flash"
|
||||
slug: "deepseek-v4.1-flash"
|
||||
provider: "DeepSeek"
|
||||
description: "DeepSeek V4.1 Flash vision-language MoE (552B backbone; 8B active per prompt token, 16B per output token) combining sliding-window plus compressed sparse attention with a two-level indexer, engram n-gram memory, hyper-connections, and a DSpark multi-token draft head."
|
||||
date_added: 2026-09-09
|
||||
date_updated: 2026-10-13
|
||||
tasks:
|
||||
- text
|
||||
- multimodal
|
||||
performance_headline: "1M context at 8-16B active parameters, with engram n-gram memory"
|
||||
related_recipes:
|
||||
- "deepseek-ai/DeepSeek-V4-Flash"
|
||||
- "deepseek-ai/DeepSeek-V4-Pro"
|
||||
default_hardware: gb200
|
||||
hardware:
|
||||
h100: verified
|
||||
h200: verified
|
||||
b200: verified
|
||||
gb200: verified
|
||||
gb300: verified
|
||||
b300: verified
|
||||
mi300x: verified
|
||||
mi325x: verified
|
||||
mi350x: verified
|
||||
mi355x: verified
|
||||
dgx_station_gb300: verified
|
||||
|
||||
model:
|
||||
model_id: "deepseek-ai/DeepSeek-V4.1-Flash"
|
||||
# Opt in to SMG gRPC PD; parsing runs on the router, not vLLM.
|
||||
smg_grpc:
|
||||
reasoning_parser: deepseek_v41
|
||||
tool_call_parser: deepseek_v41
|
||||
min_vllm_version: "0.30.0"
|
||||
# 0.30.0 has not shipped; the architecture lives on main (vllm-project/vllm#56228),
|
||||
# so NVIDIA renders `vllm/vllm-openai:nightly` instead of the launch-day
|
||||
# `deepseekv41-flash-0909` tag, which is now behind main.
|
||||
nightly_required: true
|
||||
default_frontend: rust
|
||||
docker_image:
|
||||
# ROCm 10.0 nightly channel; includes the AITER mHC delayed-pre path from
|
||||
# vllm-project/vllm#56503.
|
||||
amd: "vllm/vllm-openai-rocm:nightly-rocm100"
|
||||
architecture: moe
|
||||
parameter_count: "552B"
|
||||
active_parameters: "8-16B"
|
||||
context_length: 1048576
|
||||
base_args:
|
||||
- "--tokenizer-mode"
|
||||
- "deepseek_v41"
|
||||
base_env:
|
||||
VLLM_ENGINE_READY_TIMEOUT_S: "3600"
|
||||
install:
|
||||
docker:
|
||||
note: "Serve from the image for your GPU vendor — no pip wheel carries the DeepSeek-V4.1 architecture. Both vendors track the nightly image; the launch-day 0909 tag is superseded (see Prerequisites)."
|
||||
pip: false
|
||||
|
||||
dependencies: []
|
||||
|
||||
features:
|
||||
tool_calling:
|
||||
description: "Enable tool calling. V4.1 emits DSML tag blocks rather than JSON fences."
|
||||
args:
|
||||
- "--tool-call-parser"
|
||||
- "deepseek_v41"
|
||||
- "--enable-auto-tool-choice"
|
||||
reasoning:
|
||||
description: "Enable thinking mode with the DeepSeek V4.1 reasoning parser."
|
||||
args:
|
||||
- "--reasoning-parser"
|
||||
- "deepseek_v41"
|
||||
spec_decoding:
|
||||
description: "DSpark multi-token drafting, 5 tokens per round (the trained block size). On NVIDIA, adaptive verification scores each (request, position) slot and only high-scoring slots are verified, up to a startup-profiled compute budget. On AMD, vLLM currently refuses that flag, so the generated command sets enable_adaptive_verification:false and verifies the full 5-token block. Remove the AMD override only after a serve with the true flag actually starts on ROCm."
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block","enable_adaptive_verification":true}'
|
||||
hardware_overrides:
|
||||
# Adaptive verification writes query_start_loc on device and captures
|
||||
# varlen decode CUDA graphs. maybe_create_adaptive_verification_manager
|
||||
# therefore requires both supports_device_cpu_query_lens_mismatch()
|
||||
# and AttentionCGSupport.ALWAYS on every target attention builder.
|
||||
# On ROCm, DeepseekV41IndexerBackend fails the first check because
|
||||
# the helpers it inherits are CUDA plus DeepGEMM only. Forcing that
|
||||
# helper True still dies: DeepseekV41ROCMAiterSparseSWABackend reports
|
||||
# UNIFORM_BATCH. Features emit last in resolveCommand, so this
|
||||
# replacement must live here rather than in
|
||||
# hardware_overrides.amd.extra_args. Remove this override only when
|
||||
# a vLLM serve with enable_adaptive_verification:true starts on AMD.
|
||||
amd:
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block","enable_adaptive_verification":false}'
|
||||
# Offloaded experts make deep drafts expensive at high batch: k=5 to 4 seqs, k=1 above.
|
||||
dgx_station_gb300:
|
||||
args:
|
||||
- "--speculative-config"
|
||||
- '{"method":"dspark","num_speculative_tokens":5,"num_speculative_tokens_per_batch_size":[[1,4,5],[5,24,1]]}'
|
||||
text_only:
|
||||
description: "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
|
||||
args:
|
||||
- "--language-model-only"
|
||||
encoder_parallel:
|
||||
description: "Run the vision encoder in data-parallel mode — avoids TP comm overhead on the small encoder. Mutually exclusive with text_only."
|
||||
args:
|
||||
- "--mm-encoder-tp-mode"
|
||||
- "data"
|
||||
|
||||
opt_in_features:
|
||||
- spec_decoding
|
||||
- text_only
|
||||
|
||||
variants:
|
||||
default:
|
||||
precision: fp8
|
||||
vram_minimum_gb: 614
|
||||
description: "Native mixed checkpoint: MXFP4 MoE experts, MXFP8 block-quantized dense weights, UE8M0 scales throughout"
|
||||
hardware_overrides:
|
||||
# 8x80 GB is the smallest NVIDIA node that serves this checkpoint, and
|
||||
# only with the memory levers the InferenceX H100 arm settled on
|
||||
# (SemiAnalysisAI/InferenceX#2974, validated on the 2026-09-15 nightly in
|
||||
# SemiAnalysisAI/InferenceX#3135). Engram tables (183 GiB FP8) move to
|
||||
# pinned host DRAM behind UVA, leaving ~36 GiB/GPU of resident weights at
|
||||
# TP8. The sparse-attention indexer then allocates a
|
||||
# [max-num-batched-tokens, max-model-len] logits buffer at startup:
|
||||
# 8192 x 1M x 2 B is exactly 16 GiB and does not fit next to the weights,
|
||||
# so batched tokens are capped at 4096 (8 GiB) rather than the context
|
||||
# length, which would push the 256k-plus agentic traces off the model.
|
||||
# The V2 model runner and expandable segments keep the KV pool from
|
||||
# fragmenting around that buffer.
|
||||
h100:
|
||||
extra_args:
|
||||
- "--engram-config"
|
||||
- '{"cpu_offload":true}'
|
||||
- "--max-num-batched-tokens"
|
||||
- "4096"
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.92"
|
||||
extra_env:
|
||||
VLLM_USE_V2_MODEL_RUNNER: "1"
|
||||
PYTORCH_CUDA_ALLOC_CONF: "expandable_segments:True"
|
||||
# Single GB300 + Grace: Engram tables and 60 GiB of experts in host memory via UVA.
|
||||
# vram_minimum_gb is the GPU-resident fit budget (0.97 x 252); taxonomy vram_gb stays physical HBM.
|
||||
# 60 GiB is the lowest verified offload at 1M context with DSpark on; 55 fails the minimum-KV check.
|
||||
dgx_station_gb300:
|
||||
vram_minimum_gb: 244
|
||||
extra_args:
|
||||
- "--offload-backend"
|
||||
- "uva"
|
||||
- "--cpu-offload-gb"
|
||||
- "60"
|
||||
- "--cpu-offload-params"
|
||||
- "routed_experts.w13_weight"
|
||||
- "routed_experts.w2_weight"
|
||||
- "--engram-config"
|
||||
- '{"cpu_offload":true}'
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.97"
|
||||
- "--max-model-len"
|
||||
- "1048576"
|
||||
- "--max-num-batched-tokens"
|
||||
- "8192"
|
||||
- "--long-prefill-token-threshold"
|
||||
- "6144"
|
||||
- "--compilation-config"
|
||||
- '{"cudagraph_capture_sizes":[1,2,4,6,8,12,16,18,24,32,40,48,64,96,128]}'
|
||||
|
||||
compatible_strategies:
|
||||
- single_node_tp
|
||||
- single_node_tep
|
||||
- single_node_dep
|
||||
- multi_node_tp
|
||||
- multi_node_tep
|
||||
- multi_node_dep
|
||||
- pd_cluster
|
||||
|
||||
kv_offload_support:
|
||||
offloading_cpu: verified
|
||||
offloading_fs: verified
|
||||
|
||||
strategy_hardware:
|
||||
# Single GPU; only TP1 is measured.
|
||||
single_node_tep:
|
||||
dgx_station_gb300: unsupported
|
||||
single_node_dep:
|
||||
dgx_station_gb300: unsupported
|
||||
|
||||
hardware_overrides:
|
||||
amd:
|
||||
extra_args:
|
||||
- "--gpu-memory-utilization"
|
||||
- "0.9"
|
||||
# Allow CK a8w4 expert selection instead of pinning Triton W4A16.
|
||||
- "--moe-backend"
|
||||
- "aiter"
|
||||
extra_env:
|
||||
VLLM_ROCM_USE_AITER: "1"
|
||||
VLLM_ROCM_USE_AITER_MOE: "1"
|
||||
# Suppress repeated Gluon-unavailable warnings on gfx950.
|
||||
AITER_TRITON_LOG_LEVEL: "ERROR"
|
||||
# DeepseekV41ForCausalLM does not support torch.compile. The ROCm sparse
|
||||
# SWA backend only reports UNIFORM_BATCH, so default FULL_AND_PIECEWISE
|
||||
# cannot start unless breakable CUDA graphs are on.
|
||||
VLLM_USE_BREAKABLE_CUDAGRAPH: "1"
|
||||
|
||||
strategy_overrides:
|
||||
single_node_tp:
|
||||
# TP4 baseline (one GB200 NVL4 tray; 4 GPUs on MI350X or MI355X). H100
|
||||
# spans the whole 8x80 GB node: 4 cards hold 320 GB against a 614 GB
|
||||
# minimum, and even TP8 needs the H100 override above.
|
||||
tp:
|
||||
default: 4
|
||||
h100: 8
|
||||
b200: 2
|
||||
b300: 2
|
||||
gb200: 2
|
||||
gb300: 2
|
||||
mi355x: 2
|
||||
dgx_station_gb300: 1
|
||||
hardware_overrides:
|
||||
# 288 GiB per card. The Engram tables cost 47.2 GiB of device memory per
|
||||
# rank at TP4 and 94.4 GiB at TP2, which does not fit beside half the
|
||||
# checkpoint, so TP2 offloads them to pinned host memory. TP4 leaves them
|
||||
# resident. vllm-project/vllm#57491 widened two is_cuda() gates to
|
||||
# is_cuda_alike(), which is what makes the offload resolve on gfx950; on
|
||||
# builds carrying it cpu_offload also defaults on via VLLM_PLE_CPU_OFFLOAD,
|
||||
# so set it explicitly rather than inheriting the default.
|
||||
# 256 GB per card (gfx942). MI325X takes the TP4 default rather than the
|
||||
# MI355X TP2: at TP2 the resident weights are ~145 GiB per rank, which
|
||||
# leaves the smallest KV pool of any arm here, while TP4 lands at ~81 GiB
|
||||
# per rank with room to spare. The Engram tables are offloaded to pinned
|
||||
# host memory at TP4 as well -- gfx942 has no FP4 MFMA, so the MXFP4
|
||||
# experts run through a Triton kernel and the device memory is better
|
||||
# spent on KV. Same vllm-project/vllm#57491 gate as MI355X applies.
|
||||
#
|
||||
# gfx942 also differs from gfx950 on graph capture: every worker
|
||||
# segfaults during piecewise capture, so breakable CUDA graphs are off
|
||||
# and only full decode graphs are captured. Prefill runs eagerly.
|
||||
# 192 GB per card, the smallest gfx942 part here. Same TP4 default and the
|
||||
# same two gfx942 workarounds as MI325X below. TP2 is NOT offered on this
|
||||
# SKU: at TP2 the offloaded weights are ~145 GiB per rank, and a measured
|
||||
# run reported "Available KV cache memory: -13.51 GiB" -- the engine could
|
||||
# not start even with the sparse-attention indexer buffer halved. TP4
|
||||
# lands at ~81 GiB per rank and leaves a healthy KV pool.
|
||||
mi300x:
|
||||
extra_args:
|
||||
- "--engram-config"
|
||||
- '{"cpu_offload":true}'
|
||||
- "--compilation-config"
|
||||
- '{"cudagraph_mode":"FULL_DECODE_ONLY"}'
|
||||
extra_env:
|
||||
VLLM_USE_BREAKABLE_CUDAGRAPH: "0"
|
||||
mi325x:
|
||||
extra_args:
|
||||
- "--engram-config"
|
||||
- '{"cpu_offload":true}'
|
||||
- "--compilation-config"
|
||||
- '{"cudagraph_mode":"FULL_DECODE_ONLY"}'
|
||||
extra_env:
|
||||
VLLM_USE_BREAKABLE_CUDAGRAPH: "0"
|
||||
mi355x:
|
||||
extra_args:
|
||||
- "--engram-config"
|
||||
- '{"cpu_offload":true}'
|
||||
# ROCm paged MXFP4 sparse indexer (vllm-project/vllm#58671).
|
||||
- "--attention-config"
|
||||
- '{"indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}'
|
||||
# The fp4 indexer prefers block-size 128; auto-resolution
|
||||
# otherwise falls back to 64.
|
||||
- "--block-size"
|
||||
- "128"
|
||||
# Mono decode kernels (vllm-project/vllm#60397, #61082): two persistent
|
||||
# launches a layer on all 40 backbone layers for decode steps of up to
|
||||
# 48 rows, i.e. up to concurrency 8 with five DSpark drafts. Larger
|
||||
# steps take the regular path, so the gain fades as concurrency rises.
|
||||
# Replaces the VLLM_ROCM_MONO_DECODE environment variable.
|
||||
- "--monokernel"
|
||||
extra_env:
|
||||
# Aiter's Gluon sparse-MLA kernel, gfx950 only
|
||||
# (vllm-project/vllm#53492).
|
||||
VLLM_ROCM_USE_AITER_TRITON_SPARSE_MLA: "1"
|
||||
# INT4 quantization for faster all-reduce
|
||||
VLLM_ROCM_QUICK_REDUCE_QUANTIZATION: "INT4"
|
||||
blackwell:
|
||||
extra_args:
|
||||
- "--attention-config"
|
||||
- '{"backend":"FLASHINFER_MLA_SPARSE_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}'
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
- "--max-num-seqs"
|
||||
- "128"
|
||||
# TP2 halves the replica GPU count, so each rank holds the full
|
||||
# non-expert weights plus half the experts. Moving the Engram tables to
|
||||
# pinned host DRAM keeps that resident footprint inside HBM on these
|
||||
# SKUs; the lookups read the tables through UVA and the output is
|
||||
# unchanged.
|
||||
# B200 TP2 serves the low-concurrency (c1) agentic setting: THP-backed
|
||||
# Engram tables, 4096 batched tokens, and graphs captured up to 4092 tokens.
|
||||
b200:
|
||||
extra_args:
|
||||
- "--engram-config"
|
||||
- '{"cpu_offload":true,"use_thp":true}'
|
||||
- "--max-num-seqs"
|
||||
- "8"
|
||||
- "--max-num-batched-tokens"
|
||||
- "4096"
|
||||
- "--compilation-config"
|
||||
- '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092]}'
|
||||
- "--max-cudagraph-capture-size"
|
||||
- "4092"
|
||||
gb200:
|
||||
extra_args:
|
||||
- "--engram-config"
|
||||
- '{"cpu_offload":true}'
|
||||
gb300:
|
||||
extra_args:
|
||||
- "--engram-config"
|
||||
- '{"cpu_offload":true}'
|
||||
# Exact-GPU override, after the shared Blackwell block so it wins last-wins dedupe.
|
||||
dgx_station_gb300:
|
||||
extra_args:
|
||||
- "--attention-config"
|
||||
- '{"backend":"FLASHMLA_MEGA_ATTN_DSV41"}'
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8_ds_mla"
|
||||
- "--max-num-seqs"
|
||||
- "24"
|
||||
b300:
|
||||
extra_args:
|
||||
- "--engram-config"
|
||||
- '{"cpu_offload":true}'
|
||||
- "--compilation-config"
|
||||
- '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,768,1020,1536,2046,3072,4092,6144,8190]}'
|
||||
- "--max-cudagraph-capture-size"
|
||||
- "8190"
|
||||
- "--max-num-batched-tokens"
|
||||
- "8192"
|
||||
- "--max-num-seqs"
|
||||
- "256"
|
||||
# H200 takes the TP4 default on 8x141 GB. Moving the Engram tables to
|
||||
# pinned host DRAM (23.6 GiB per rank, 94 GiB across the replica) drops
|
||||
# resident weights to 81.2 GiB per GPU and leaves 38.5 GiB of KV per GPU,
|
||||
# about 20M tokens, measured on the InferenceX AgentX sweep. Without the
|
||||
# offload the tables stay row-sharded in HBM and the KV pool collapses.
|
||||
h200:
|
||||
extra_args:
|
||||
- "--engram-config"
|
||||
- '{"cpu_offload":true}'
|
||||
single_node_dep:
|
||||
hardware_overrides:
|
||||
blackwell:
|
||||
extra_args:
|
||||
- "--kernel-config"
|
||||
- '{"moe_backend":"deep_gemm_mega_moe"}'
|
||||
- "--attention-config"
|
||||
- '{"backend":"FLASHMLA_MEGA_ATTN_DSV41","indexer_kv_dtype":"mxfp4","indexer_sparse_logits":true}'
|
||||
- "--kv-cache-dtype"
|
||||
- "fp8"
|
||||
- "--max-num-seqs"
|
||||
- "128"
|
||||
- "--engram-config"
|
||||
- '{"embedding_across_dp":true}'
|
||||
# B200 DEP2 at c32: piecewise graphs extend to 4096 tokens so prefill-bearing
|
||||
# steps replay graphs, and new prefills are admitted every 64 engine steps,
|
||||
# aligned across DP ranks. The vLLM default of 1 admits on every step.
|
||||
b200:
|
||||
extra_args:
|
||||
- "--max-num-seqs"
|
||||
- "32"
|
||||
- "--max-num-batched-tokens"
|
||||
- "4096"
|
||||
- "--prefill-schedule-interval"
|
||||
- "64"
|
||||
- "--compilation-config"
|
||||
- '{"mode":"VLLM_COMPILE","cudagraph_mode":"FULL_AND_PIECEWISE","cudagraph_capture_sizes":[6,12,18,24,30,36,48,60,72,96,120,144,192,240,288,384,480,576,640,768,896,1024,1280,1536,1792,2048,2304,2560,2816,3072,3328,3584,3840,4096]}'
|
||||
- "--max-cudagraph-capture-size"
|
||||
- "4096"
|
||||
pd_cluster:
|
||||
# Verified 1P1D on GB200 NVL4: one tray per role, TP4 each, NIXL transfer.
|
||||
env:
|
||||
VLLM_DEEP_GEMM_WARMUP: "skip"
|
||||
prefill:
|
||||
nodes: 1
|
||||
parallelism: tp
|
||||
vllm_args:
|
||||
- "--max-num-seqs"
|
||||
- "32"
|
||||
decode:
|
||||
nodes: 1
|
||||
parallelism: tp
|
||||
vllm_args:
|
||||
- "--max-num-seqs"
|
||||
- "32"
|
||||
|
||||
hardware_notes:
|
||||
dgx_station_gb300:
|
||||
title: "Weights in Grace memory (UVA offload)"
|
||||
body: "Single GB300 with both Engram tables and 60 GiB of routed experts in pinned Grace memory (UVA offload). Nightly image only. The first boot on a new flag set runs a 75-90 min FlashInfer autotune; keep /root/.cache/vllm on a persistent volume."
|
||||
|
||||
guide: |
|
||||
## Overview
|
||||
|
||||
DeepSeek-V4.1 is a vision-language Mixture-of-Experts model: **552B backbone parameters
|
||||
plus 196B of Engram memory, 8B active per prompt token and 16B per output token**, 40
|
||||
transformer layers at hidden size 5120, with a 32-layer ViT and aligner in front of the
|
||||
text stack. Six things distinguish it from V4:
|
||||
|
||||
- **Causal encoder-decoder.** The 40 layers split into a 20-layer encoder and a
|
||||
20-layer decoder, and the decoder's global KV is projected from the encoder's final
|
||||
hidden state instead of being computed layer by layer. Prefill therefore only runs
|
||||
the encoder half, which is where the 8B-prefill / 16B-decode split comes from.
|
||||
- **Two-tier sparse attention.** Every layer attends over a 128-token sliding window.
|
||||
Layers that carry a compression ratio add compressed KV latents reaching further
|
||||
back, pooled by a learned softmax gate. A small side attention — the *indexer*,
|
||||
inherited from V3.2-Exp — scores those latents and keeps the best 512 per query,
|
||||
pre-filtered by a candidate stage that selects 2048 blocks of 8. Only four layers
|
||||
(2, 8, 14, 20) actually compress their own KV; the rest read that cache. V4.1 uses
|
||||
only compression ratios 1 and 2, where V4 used 4 and 128.
|
||||
- **Engram n-gram memory.** Layers 1 and 14 each own a hash table of ~384M rows x 256
|
||||
dims, looked up by 2-, 3- and 4-gram hashes of the input and written into the residual
|
||||
stream through a learned gate. These two tables alone are **196.6B parameters (~183 GiB)** —
|
||||
plan capacity for them, they dominate everything except the experts.
|
||||
- **Hyper-Connections.** The residual stream is carried as 4 parallel copies; each
|
||||
sublayer derives its own pre/post/combine coefficients from the stream, with the
|
||||
combine matrix made doubly stochastic by 20 Sinkhorn iterations.
|
||||
- **DSpark draft head.** Three stages (128 routed experts each, 3 activated) draft a
|
||||
block of 5 tokens, reading the attention input of layers 37-39, with a Markov bias
|
||||
head and a confidence head on the last stage.
|
||||
- **Mixed MXFP4/MXFP8 checkpoint.** Routed expert weights are MXFP4; everything else is
|
||||
MXFP8 block-quantized, UE8M0 scales throughout. Embedding and LM head are BF16.
|
||||
|
||||
Routing is 6 of 384 experts per token plus one shared expert, scored with
|
||||
`sqrtsoftplus` and a `noaux_tc` bias — with a **separate routing bias for tokens
|
||||
inside an image span**, so vision and text tokens do not compete for the same experts.
|
||||
|
||||
## Context length
|
||||
|
||||
1,048,576 tokens, reached by YaRN with factor 16 over a 65,536-token training window.
|
||||
Compressed KV rotates at its own RoPE theta (160,000) because one latent stands for
|
||||
several tokens, so its positions are further apart than the raw stream's.
|
||||
|
||||
## Images
|
||||
|
||||
Images enter the prompt as `<|deepseek_image|>` spans; every position in the span
|
||||
carries the image token id, and the per-position role (start / newline / end / image)
|
||||
comes from the processor. The vision tower is a 32-layer ViT at hidden size 1024, patch
|
||||
14, with a 3x downsampling aligner and a cap of 1024 tokens per image (minimum 295,936
|
||||
pixels). There is **no limit on images per prompt**. Merged embeddings enter the text
|
||||
model as `inputs_embeds`, before the hyper-connection stream expansion, while raw token
|
||||
ids still flow through so the router can apply the image routing bias.
|
||||
|
||||
Tick **Encoder parallel** to run the ViT data-parallel (`--mm-encoder-tp-mode data`)
|
||||
instead of tensor-parallel: at 32 layers / hidden 1024 the encoder is small enough
|
||||
that TP communication costs more than it saves, which can significantly reduce TTFT
|
||||
for multi-image requests. It is mutually exclusive with **Text only**.
|
||||
|
||||
## Reasoning and tool calling
|
||||
|
||||
Two thinking modes and a **numeric reasoning budget** rather than discrete tiers.
|
||||
Send them as `chat_template_kwargs`:
|
||||
|
||||
| key | values |
|
||||
|---|---|
|
||||
| `thinking` / `enable_thinking` | boolean; if you send both they must agree |
|
||||
| `reasoning_effort` | `low` (25), `high` (50), `xhigh` (75), `max` (100), **or an integer 1-100** |
|
||||
|
||||
The top-level OpenAI `reasoning_effort` field also works, where `"none"` turns
|
||||
thinking off. `"minimal"` and `"medium"` are rejected — they are not part of this
|
||||
model's set.
|
||||
|
||||
Note that the DeepSeek API maps its `low` / `high` / `max` tiers to 50 / 75 / 100,
|
||||
while the open-source prompt encoder (and therefore vLLM) maps `low` / `high` /
|
||||
`xhigh` / `max` to 25 / 50 / 75 / 100. A request tuned at `high` on the API runs at
|
||||
effort 50 here; pass an integer if you need parity.
|
||||
|
||||
> **With both keys unset, thinking is ON at effort 50.** The do-nothing config is
|
||||
> the most verbose one, so a request with a small `max_tokens` spends its budget on
|
||||
> the trace and returns empty `content` with `finish_reason=length`. That reads as a
|
||||
> broken model and is not. Either pin `thinking: false` or give the budget room.
|
||||
|
||||
In thinking mode the budget is rendered into the prompt as a `Reasoning Effort: N`
|
||||
prefix on the first turn only.
|
||||
|
||||
```python
|
||||
resp = client.chat.completions.create(
|
||||
model="deepseek-ai/DeepSeek-V4.1-Flash",
|
||||
messages=[{"role": "user", "content": "What is 17*19?"}],
|
||||
extra_body={"chat_template_kwargs": {"thinking": True, "reasoning_effort": 25}},
|
||||
)
|
||||
```
|
||||
|
||||
DeepSeek's published numbers use `temperature=1.0`, `top_p=0.95` and effort 100 with
|
||||
`max_tokens` of at least 256K. The tech report notes that efforts of 60-80 recover most
|
||||
of that accuracy at well under half the tokens, so reserve `max` for hard tasks.
|
||||
|
||||
Tool calls are wrapped in DSML tag blocks rather than JSON fences, and tool output
|
||||
comes back in `<tool_result>` tags.
|
||||
|
||||
## Speculative decoding
|
||||
|
||||
DSpark is the only speculative method for this checkpoint. V4.1 dropped the MTP module
|
||||
that V3 and V4 trained alongside the backbone. **Speculative decoding** enables a 5-token
|
||||
block. On NVIDIA it also turns on adaptive verification (see the feature description).
|
||||
On AMD, vLLM currently refuses adaptive verification. The generated command sets
|
||||
`enable_adaptive_verification:false` and verifies the full block. Two checks in
|
||||
`maybe_create_adaptive_verification_manager` fail on ROCm: the indexer helper
|
||||
`supports_device_cpu_query_lens_mismatch()` is False, and
|
||||
`DeepseekV41ROCMAiterSparseSWABackend` reports `AttentionCGSupport.UNIFORM_BATCH`
|
||||
rather than `ALWAYS`. Remove the AMD override only after a serve with the true flag
|
||||
starts. Acceptance depends on the workload, so measure it on your own traffic before
|
||||
sizing a deployment around it.
|
||||
The drafter's experts add roughly 14B parameters to the load.
|
||||
|
||||
## Serving text-only
|
||||
|
||||
Tick **Text only** to add `--language-model-only`, which skips the vision encoder
|
||||
entirely. Worth it whenever the workload is text: it drops the ViT and aligner from
|
||||
the load and frees that VRAM for KV cache. It is mutually exclusive with
|
||||
`encoder_parallel`. The verified GB200 runs (TP4 and 1P1D) were text-only.
|
||||
|
||||
## Prefill/Decode disaggregation
|
||||
|
||||
The **Prefill/Decode Disaggregation** strategy is the verified 1P1D layout on GB200
|
||||
NVL4: one tray (4 GPUs) per role, TP4 in each pool, KV handed over through NIXL,
|
||||
fronted by `vllm-router --vllm-pd-disaggregation`. Both pools disable FlashInfer
|
||||
autotune plus JIT and CuTeDSL warmup via `--kernel-config`, skip the DeepGEMM warmup
|
||||
(`VLLM_DEEP_GEMM_WARMUP=skip`), and cap `--max-num-seqs` at 32. On 8-GPU nodes the
|
||||
same layout becomes TP8 per role.
|
||||
|
||||
With **Speculative decoding** on, DSpark runs in both pools so the transferred KV
|
||||
stays compatible.
|
||||
|
||||
## Memory
|
||||
|
||||
The checkpoint is roughly **511 GB on disk** (476 GiB), which breaks down as:
|
||||
|
||||
| Component | Entries | Stored |
|
||||
|---|---:|---:|
|
||||
| Routed + DSpark experts (MXFP4) | 557.2B | 259.5 GiB |
|
||||
| Engram tables (FP8) | 196.6B | 183.1 GiB |
|
||||
| Attention, dense projections, routers (FP8) | 7.4B | 6.9 GiB |
|
||||
| Embedding + LM head (BF16), norms (FP32) | 2.0B | 3.9 GiB |
|
||||
| UE8M0 block scales | 23.6B | 21.9 GiB |
|
||||
|
||||
DeepSeek's 552B backbone figure covers the routed experts, attention and embeddings;
|
||||
the Engram tables, the ~14B DSpark drafter and the block scales sit outside it.
|
||||
|
||||
`vram_minimum_gb: 614` is that total times the schema's 1.2 headroom factor. It fits
|
||||
one GB200 NVL4 tray (768 GB) at TP4, or one 8-GPU H200 node (1128 GB) with room for
|
||||
KV cache.
|
||||
|
||||
KV cache is a small share of that budget. The compressed latents are shared across
|
||||
layers and trained to be stored in FP4, and DeepSeek puts the resulting global KV at
|
||||
**890 bytes per token**, about a quarter of V4-Flash; by that accounting a full 1M-token
|
||||
prompt holds under 1 GB of global KV, plus a fixed 128-token sliding window per layer.
|
||||
Weights and batch size, not cache, set the capacity limit — still, measure before
|
||||
assuming a given context/batch fits.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- An image (vLLM 0.30.0+); no pip wheel serves this architecture, so the Install block
|
||||
only offers Docker. On NVIDIA that is `vllm/vllm-openai:nightly`: the architecture
|
||||
landed on `main` in [vllm-project/vllm#56228](https://github.com/vllm-project/vllm/pull/56228),
|
||||
so any nightly from 2026-09-10 on serves it, and the launch-day
|
||||
`deepseekv41-flash-0909` tag is no longer needed. On AMD it is
|
||||
`vllm/vllm-openai-rocm:nightly-rocm100`, the ROCm 10.0 nightly, which also carries
|
||||
[vllm-project/vllm#56503](https://github.com/vllm-project/vllm/pull/56503) — the change
|
||||
that moves the mHC delayed pre block off the eager Torch reference and onto AITER
|
||||
and postdates the 0909 tag.
|
||||
- The Rust OpenAI frontend is selected by default in the command builder. Switch
|
||||
to Python if you encounter unsupported features or compatibility issues.
|
||||
- Expect a long first load: `VLLM_ENGINE_READY_TIMEOUT_S=3600` is set for that reason.
|
||||
- On AMD, the generated command sets `VLLM_USE_BREAKABLE_CUDAGRAPH=1`. DeepSeek-V4.1-Flash does not support `torch.compile`, and the ROCm sparse SWA backend only supports uniform-batch CUDA graphs. Without breakable CUDA graphs, default `FULL_AND_PIECEWISE` dies at capture.
|
||||
- On AMD, the generated command passes `--moe-backend aiter` rather than naming a kernel. Naming `aiter_triton_mxfp4_bf16` pins the Triton W4A16 `_moe_gemm_a16w4` kernel; the plain name lets vLLM select the Composable Kernel a8w4 experts, used in the MI355X benchmark below. It also sets `AITER_TRITON_LOG_LEVEL=ERROR` to drop AITER's per-call "Gluon unavailable" warning, which is constant on gfx950 and dominated the server log.
|
||||
- On MI355X, **Tensor Parallel** inherits vLLM's scheduler limits. Tune `--max-num-seqs` for your traffic and memory budget if needed. The benchmark configuration below records an explicit limit for reproducibility.
|
||||
- On AMD, ticking **Speculative decoding** also sets `enable_adaptive_verification:false`. vLLM currently refuses the true flag: `DeepseekV41IndexerBackend.supports_device_cpu_query_lens_mismatch()` is False, and `DeepseekV41ROCMAiterSparseSWABackend` reports `UNIFORM_BATCH` rather than `ALWAYS`. DSpark still drafts 5 tokens per round.
|
||||
|
||||
## Verifying
|
||||
|
||||
Serve, then send one text request and one image request — the vision tower is a
|
||||
separate path and a text-only smoke test will not exercise it:
|
||||
|
||||
```bash
|
||||
curl http://localhost:8000/v1/chat/completions \
|
||||
-H 'Content-Type: application/json' \
|
||||
-d '{"model":"deepseek-ai/DeepSeek-V4.1-Flash",
|
||||
"messages":[{"role":"user","content":"What is 17*19? Return only the integer."}]}'
|
||||
```
|
||||
|
||||
A correct answer is `323`.
|
||||
|
||||
## Performance tuning
|
||||
|
||||
Start with vLLM's defaults and tune one parameter at a time for your workload:
|
||||
|
||||
- `--max-num-seqs`: raise the running-sequence limit if it causes queueing and
|
||||
memory permits; lower it if larger batches hurt latency or memory use.
|
||||
- `--max-num-batched-tokens`: balance prefill throughput against decode latency
|
||||
and memory use by adjusting the token budget per scheduling iteration.
|
||||
- `--max-cudagraph-capture-size`: cover expected decode batches, including DSpark
|
||||
draft tokens, while accounting for graph memory and startup cost. The benchmark
|
||||
below rounds `128 * (1 + 5) = 768` up to 1024.
|
||||
- `--max-model-len`: size the context limit for your prompt plus output lengths.
|
||||
|
||||
### MI355X benchmark server
|
||||
|
||||
This runnable command matches the server arguments and execution environment in
|
||||
[InferenceX #3058](https://github.com/SemiAnalysisAI/InferenceX/pull/3058) at
|
||||
[commit `559ef756`](https://github.com/SemiAnalysisAI/InferenceX/blob/559ef7560a74d0fb8da64fdfc0c357ad2fc33c4f/benchmarks/single_node/agentic/dsv41flash_fp4_mi355x_vllm_mtp.sh).
|
||||
It uses four MI355X GPUs and the pinned image. Set `ROCR_VISIBLE_DEVICES` and `PORT`
|
||||
for your host, and `HF_HOME` to an existing writable Hugging Face cache if needed.
|
||||
Throughput mode uses synthetic acceptance for benchmark measurement only. Run
|
||||
`export EVAL_ONLY=true` before this command for real block rejection, as used by
|
||||
the InferenceX accuracy eval; use that mode for ordinary serving too.
|
||||
|
||||
```bash
|
||||
docker run --rm -i --network host --ipc host \
|
||||
--device /dev/kfd --device /dev/dri --group-add video \
|
||||
--security-opt seccomp=unconfined --workdir /tmp \
|
||||
-v "${HF_HOME:-$HOME/.cache/huggingface}:/root/.cache/huggingface" \
|
||||
-e ROCR_VISIBLE_DEVICES="${ROCR_VISIBLE_DEVICES:-0,1,2,3}" \
|
||||
-e PORT="${PORT:-8000}" -e EVAL_ONLY="${EVAL_ONLY:-false}" \
|
||||
--entrypoint /bin/bash \
|
||||
vllm/vllm-openai-rocm:nightly-eed1f3d0c6043bd494424a22443ee198dd56f657 -s <<'BASH'
|
||||
set -eo pipefail
|
||||
export HIP_VISIBLE_DEVICES="$ROCR_VISIBLE_DEVICES"
|
||||
export VLLM_ROCM_USE_AITER=1 VLLM_ROCM_USE_AITER_MOE=1
|
||||
export AITER_TRITON_LOG_LEVEL=ERROR VLLM_USE_BREAKABLE_CUDAGRAPH=1
|
||||
export OMP_NUM_THREADS=1 VLLM_ENGINE_READY_TIMEOUT_S=3600
|
||||
export VLLM_USE_RUST_FRONTEND=1 PYTHONUNBUFFERED=1 GPU_COUNT=4
|
||||
MODEL=deepseek-ai/DeepSeek-V4.1-Flash
|
||||
hf download "$MODEL"
|
||||
if [[ "$EVAL_ONLY" == true ]]; then
|
||||
SPEC_CONFIG='{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"block","enable_adaptive_verification":false}'
|
||||
else
|
||||
SPEC_CONFIG='{"method":"dspark","num_speculative_tokens":5,"draft_sample_method":"probabilistic","rejection_sample_method":"synthetic","synthetic_acceptance_length":3.51,"enable_adaptive_verification":false}'
|
||||
fi
|
||||
exec vllm serve "$MODEL" --served-model-name "$MODEL" \
|
||||
--host 0.0.0.0 --port "$PORT" --tensor-parallel-size 4 \
|
||||
--language-model-only \
|
||||
--tokenizer-mode deepseek_v41 \
|
||||
--tool-call-parser deepseek_v41 --enable-auto-tool-choice \
|
||||
--reasoning-parser deepseek_v41 \
|
||||
--moe-backend aiter \
|
||||
--gpu-memory-utilization 0.9 \
|
||||
--speculative-config "$SPEC_CONFIG" \
|
||||
--max-model-len 1048576 \
|
||||
--max-num-seqs 128 \
|
||||
--max-cudagraph-capture-size 1024 \
|
||||
--max-num-batched-tokens 16384 \
|
||||
--disable-uvicorn-access-log
|
||||
BASH
|
||||
```
|
||||
|
||||
The AgentX sweep uses task concurrencies 1, 2, 4, 8, 16 and 32, GPU-resident KV,
|
||||
and `WEKA_LOADER_OVERRIDE=semianalysis_cc_traces_weka_062126`. Five-token DSpark
|
||||
uses golden acceptance length 3.51. For replay setup and results, see
|
||||
[InferenceX #3058](https://github.com/SemiAnalysisAI/InferenceX/pull/3058) and its
|
||||
[pinned master config](https://github.com/SemiAnalysisAI/InferenceX/blob/559ef7560a74d0fb8da64fdfc0c357ad2fc33c4f/configs/amd-master.yaml#L1761).
|
||||
The command above starts the server; the InferenceX harness supplies the replay
|
||||
client. Validation evidence applies to that pinned benchmark configuration, not
|
||||
every future `:nightly` image or the general command's inherited scheduler limits.
|
||||
|
||||
## MI355X Tensor Parallel
|
||||
|
||||
On MI355X, **Tensor Parallel** defaults to **TP2** with Engram CPU offload
|
||||
(`--engram-config '{"cpu_offload":true}'`). The Engram tables cost 47.2 GiB of
|
||||
device memory per rank at TP4 and 94.4 GiB at TP2; on a 288 GiB card the TP2
|
||||
figure does not fit beside half the checkpoint, so the tables move to pinned
|
||||
host memory. TP2 halves the GPUs per server and so doubles the servers per
|
||||
node. For TP4, pass `--tensor-parallel-size 4` and
|
||||
`--engram-config '{"cpu_offload":false}'` to keep the tables resident.
|
||||
|
||||
The offload resolves on gfx950 only on builds carrying
|
||||
[vllm-project/vllm#57491](https://github.com/vllm-project/vllm/pull/57491),
|
||||
which widened two `is_cuda()` gates to `is_cuda_alike()`. Those builds also
|
||||
default `cpu_offload` on through `VLLM_PLE_CPU_OFFLOAD`, so set it explicitly
|
||||
rather than inheriting the default.
|
||||
|
||||
`--monokernel` selects the mono decode layer from
|
||||
[vllm-project/vllm#60397](https://github.com/vllm-project/vllm/pull/60397),
|
||||
which [vllm-project/vllm#61082](https://github.com/vllm-project/vllm/pull/61082)
|
||||
extended to all 40 backbone layers and moved from the
|
||||
`VLLM_ROCM_MONO_DECODE` environment variable to this flag. Two persistent
|
||||
launches cover a layer for decode steps of up to 48 rows, which with five
|
||||
DSpark drafts is concurrency 8; larger steps fall back to the regular path, so
|
||||
the gain is largest at low concurrency and fades as concurrency rises.
|
||||
|
||||
Decoder SWA bounded replay
|
||||
([vllm-project/vllm#59039](https://github.com/vllm-project/vllm/pull/59039))
|
||||
is on by default on ROCm, so the recipe no longer passes
|
||||
`--no-swa-bounded-replay`. Use a nightly from 2026-10-13 or later; earlier
|
||||
ROCm builds lack the replay window clamp and fault at the first prefix hit.
|
||||
|
||||
## MI300X Tensor Parallel
|
||||
|
||||
On MI300X, **Tensor Parallel** takes the recipe default of **TP4** with the
|
||||
Engram tables offloaded to pinned host memory
|
||||
(`--engram-config '{"cpu_offload":true}'`). At TP4 the resident weights are
|
||||
roughly 81 GiB per rank, which leaves a healthy KV pool on a 192 GB card.
|
||||
|
||||
**TP2 does not fit on this SKU.** At TP2 the offloaded weights rise to roughly
|
||||
145 GiB per rank, and a measured run reported
|
||||
`Available KV cache memory: -13.51 GiB`, so the engine refused to start with
|
||||
`No available memory for the cache blocks`. That was already with the
|
||||
sparse-attention indexer buffer halved to 4096 batched tokens. Raising
|
||||
`--gpu-memory-utilization` and halving the buffer again recover roughly 13 GiB
|
||||
between them, which still leaves no usable KV for a 1M-context model, so TP2
|
||||
is not offered here. MI325X runs the same TP2 configuration successfully on
|
||||
its larger 256 GB card.
|
||||
|
||||
MI300X is gfx942 and needs the same two workarounds as MI325X: no FP4 MFMA, so
|
||||
the MXFP4 experts run through a Triton kernel and device memory is better spent
|
||||
on KV than on resident Engram tables; and piecewise graph capture segfaults
|
||||
every worker, so this recipe sets `VLLM_USE_BREAKABLE_CUDAGRAPH=0` and captures
|
||||
only full decode graphs
|
||||
(`--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}'`), with prefill
|
||||
eager.
|
||||
|
||||
The offload resolves on gfx942 only on builds carrying
|
||||
[vllm-project/vllm#57491](https://github.com/vllm-project/vllm/pull/57491),
|
||||
the same gate MI325X and MI355X need. Those builds also default `cpu_offload`
|
||||
on through `VLLM_PLE_CPU_OFFLOAD`, so set it explicitly rather than inheriting
|
||||
the default.
|
||||
|
||||
## MI325X Tensor Parallel
|
||||
|
||||
On MI325X, **Tensor Parallel** takes the recipe default of **TP4**, with the
|
||||
Engram tables offloaded to pinned host memory
|
||||
(`--engram-config '{"cpu_offload":true}'`). A 256 GB card leaves roughly
|
||||
81 GiB per rank of resident weights at TP4 and roughly 145 GiB at TP2; TP2
|
||||
fits but leaves the smallest KV pool of any arm in this recipe, so TP4 is the
|
||||
default here rather than the TP2 that MI355X uses on its larger 288 GiB card.
|
||||
For TP2, pass `--tensor-parallel-size 2`.
|
||||
|
||||
MI325X is gfx942, which differs from gfx950 in two ways that matter. It has no
|
||||
FP4 MFMA, so the MXFP4 experts run through a Triton kernel and device memory is
|
||||
better spent on KV than on resident Engram tables. It also segfaults in every
|
||||
worker during piecewise graph capture, so this recipe sets
|
||||
`VLLM_USE_BREAKABLE_CUDAGRAPH=0` and captures only full decode graphs
|
||||
(`--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY"}'`); prefill runs
|
||||
eagerly.
|
||||
|
||||
The offload resolves on gfx942 only on builds carrying
|
||||
[vllm-project/vllm#57491](https://github.com/vllm-project/vllm/pull/57491),
|
||||
the same gate MI355X needs. Those builds also default `cpu_offload` on through
|
||||
`VLLM_PLE_CPU_OFFLOAD`, so set it explicitly rather than inheriting the
|
||||
default.
|
||||
|
||||
## H200 Tensor Parallel
|
||||
|
||||
On H200, **Tensor Parallel** uses the TP4 default with Engram CPU offload
|
||||
(`--engram-config '{"cpu_offload":true}'`). The Engram tables move to pinned
|
||||
host DRAM and are read through UVA, so the output is unchanged. On 141 GB
|
||||
cards that offload is what makes TP4 practical: it moves 23.6 GiB per rank off
|
||||
the GPU, leaving 81.2 GiB of resident weights and 38.5 GiB of KV per GPU
|
||||
(about 20M tokens). No scheduler caps are needed at this replica size.
|
||||
|
||||
## Blackwell TP and DEP
|
||||
|
||||
On B200, B300, GB200 and GB300, **Tensor Parallel** uses **TP2** with
|
||||
`FLASHINFER_MLA_SPARSE_DSV41` and Engram CPU offload
|
||||
(`--engram-config '{"cpu_offload":true}'`), which moves the Engram tables to
|
||||
pinned host DRAM and reads them through UVA; the output is unchanged.
|
||||
**Data + Expert Parallel** uses the hardware's
|
||||
GPU count: DEP4 on GB200/GB300 trays, DEP8 on B200/B300 nodes. It enables
|
||||
`--enable-expert-parallel`, `FLASHMLA_MEGA_ATTN_DSV41`,
|
||||
`--kernel-config '{"moe_backend":"deep_gemm_mega_moe"}'`, and
|
||||
`--engram-config '{"embedding_across_dp":true}'`.
|
||||
|
||||
Both configurations set `indexer_kv_dtype` to `mxfp4`, `indexer_sparse_logits`
|
||||
to `true`, `--kv-cache-dtype fp8`, and `--max-num-seqs 128`. Reasoning uses
|
||||
`--reasoning-parser deepseek_v41`.
|
||||
|
||||
Each Blackwell SKU carries an exact-GPU override that takes precedence over
|
||||
the shared Blackwell defaults. B300 keeps its existing CUDA Graph capture
|
||||
configuration (`--max-cudagraph-capture-size 8190`),
|
||||
`--max-num-batched-tokens 8192` and `--max-num-seqs 256`; B200, GB200 and
|
||||
GB300 add Engram CPU offload and otherwise inherit the shared Blackwell
|
||||
settings. Scheduler limits are workload-dependent at TP2, so size
|
||||
`--max-num-batched-tokens` and `--max-cudagraph-capture-size` for your own
|
||||
serving profile. For high interactivity on any of them, use
|
||||
`--tensor-parallel-size 4`.
|
||||
|
||||
## B200 AgentX tuning
|
||||
|
||||
On B200, TP2 at concurrency 1 uses `--engram-config '{"cpu_offload":true,"use_thp":true}'`,
|
||||
`--max-num-seqs 8` and `--max-num-batched-tokens 4096`. DEP2 (TP1 x DP2 with expert
|
||||
parallelism) serves concurrency 8 to 32; at c8 and c16 graph capture stops at 576 tokens and
|
||||
`--prefill-schedule-interval` stays at the vLLM default of 1, while c32 extends piecewise
|
||||
graphs to 4096 tokens and sets `--prefill-schedule-interval 64`. DEP4 sets
|
||||
`--prefill-schedule-interval 48` at c128 and keeps the default of 1 at c64.
|
||||
|
||||
For DEP2 at c128, add vLLM's native CPU KV offloading with
|
||||
`--kv-offloading-backend native --kv-offloading-size 32` (GiB per DP rank, 64 GiB per
|
||||
server), `--gpu-memory-utilization 0.955` and `--max-num-seqs 48`.
|
||||
|
||||
## DGX Station (GB300) Tensor Parallel
|
||||
|
||||
One GB300 exposes 252 GB of HBM next to 496 GB of Grace LPDDR5X, and the
|
||||
476 GiB checkpoint does not fit the HBM. The Station row serves TP1 with
|
||||
both Engram tables (189 GiB) and 60 GiB of routed experts in pinned host
|
||||
memory, read through UVA. At `--gpu-memory-utilization 0.97` the KV pool
|
||||
holds about 3.09M tokens at full 1M context. 60 GiB is the lowest
|
||||
verified offload at 1M context with speculative decoding on (55 fails the
|
||||
minimum-KV check). For more concurrent 1M contexts, raise
|
||||
`--cpu-offload-gb` to 70 and accept ~10% lower single-stream decode.
|
||||
|
||||
Measured on `vllm/vllm-openai:nightly-dee37d89`:
|
||||
|
||||
| | 1 stream | 2 | 4 | 8 | 12 | 16 |
|
||||
|---|--:|--:|--:|--:|--:|--:|
|
||||
| aggregate tok/s, prose | 90.5 | 124 | 168 | 302 | 378 | 416–423 |
|
||||
|
||||
- **Autotune.** The first boot on a new flag set or image runs a 75–90
|
||||
minute FlashInfer autotune. Mount `/root/.cache/vllm` on a persistent
|
||||
volume; later boots start in about 7 minutes.
|
||||
- **Host memory.** With the server loaded, ~430 of the 494 GiB Grace pool
|
||||
is in use (pinned weights plus loader shared memory).
|
||||
`VLLM_WEIGHT_OFFLOADING_DISABLE_PIN_MEMORY=1` frees ~66 GB at about 7%
|
||||
lower single-stream decode.
|
||||
|
||||
Launch scripts and full measurements:
|
||||
[J-M-Recipes/recipes](https://github.com/J-M-Recipes/recipes/tree/main/recipes/dgx-station-gb300/deepseek-v4.1-flash-vllm-uva-dspark).
|
||||
|
||||
## KV cache offloading
|
||||
|
||||
Offloading requires a build containing [vllm-project/vllm#57145](https://github.com/vllm-project/vllm/pull/57145).
|
||||
Validation used [deepseek-ai/DeepSeek-V4.1-Flash](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash) on 4×H200 (TP4),
|
||||
vLLM commit `d1b4028d7e`, the Python frontend, `--enforce-eager`,
|
||||
`--max-model-len 16384`, `--max-num-seqs 16` and `--max-num-batched-tokens 8192`.
|
||||
DeepSeek validation used `--gpu-memory-utilization 0.95`.
|
||||
Validation used `TieringOffloadingSpec`: 8 GiB total CPU tier, `blocks_per_chunk: 1`,
|
||||
a persistent filesystem `root_dir`, eight read/write threads and `PYTHONHASHSEED=0`.
|
||||
Three CPU-tier reloads and three filesystem reloads after restarting the server passed.
|
||||
Reloaded responses matched the cold output, including the reasoning field.
|
||||
The five-shot GSM8K diagnostic scored 31/32; this is a small sample, not the full benchmark.
|
||||
These results cover this text-only configuration, not the builder's larger tier/chunk defaults or other variants.
|
||||
|
||||
## References
|
||||
|
||||
- [Model card](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash)
|
||||
- [Technical report](https://huggingface.co/deepseek-ai/DeepSeek-V4.1-Flash/blob/main/DeepSeek_V41_Tech_Report.pdf)
|
||||
- [DeepSeek-V4-Flash recipe](/deepseek-ai/DeepSeek-V4-Flash) — the V4 sibling this diverges from
|
||||
- [DeepSeek-V3.2-Exp recipe](/deepseek-ai/DeepSeek-V3.2-Exp) — where the indexer first appeared
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user