iHore/vLLM
8
0
Fork 0
Code Issues Pull Requests Actions 1 Packages Projects Releases Wiki Activity

Merge branch 'main' into patches

Browse Source
This commit is contained in:
MOVZX
2026-10-18 16:04:36 +07:00
parent e0fe5df731 1b2a666391
commit c5ab42bc0f
299 changed files with 51483 additions and 2195 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.agents/skills/model-microbenchmark/SKILL.md
+40
View File
@@ -51,6 +51,46 @@ Key flags (plus all `EngineArgs`):
Read `median_us` (each replay is already the max over ranks).
## Kernel breakdown
Add `--profile-dir DIR` to see where the time goes. After timing, each case is
replayed 3 more times under `torch.profiler`; every rank writes
`DIR/<case>.rank<r>.kernels.json` (GPU time per kernel per forward, sorted)
and `DIR/<case>.rank<r>.trace.json.gz` (open in Perfetto). Rank 0's top
kernels are printed and included in `--output-json`.
```bash
python -m vllm.benchmarks.forward.profile DIR/decode.rank0.kernels.json
# Before/after a change, same case: per-kernel deltas, largest first.
python -m vllm.benchmarks.forward.profile base/decode.rank0.kernels.json new/decode.rank0.kernels.json
```
- Kernels are grouped by name without template or function arguments; the
JSON keeps one full name per group in `example`.
- Only the timed forward is included: the in-graph KV restore and rank
rendezvous are excluded (`pre_forward_us`). `span_us` (first to last
forward kernel) should be close to `median_us`, a bit higher from profiler
overhead; `busy_us` can exceed it when streams overlap.
- Compare the same rank across runs: under EP, ranks differ (e.g. collective
wait time shows up inside communication kernels).
For Nsight Systems (NVTX, NCCL, CPU side, GPU metrics), skip `--profile-dir`:
the two can't share CUPTI. Each case's 3 profiled replays run between
`cudaProfilerStart`/`Stop` in an NVTX range `forward/<case>`, so a capture range
records only them, one report per case (and rank):
```bash
nsys profile --capture-range=cudaProfilerApi --capture-range-end=repeat \
--cuda-graph-trace=node -o prof vllm bench forward ... # prof.1, prof.2, ...
torchrun --nproc-per-node=4 --no-python nsys profile ... -o prof_rank%q{RANK} \
python -m vllm.entrypoints.cli.main bench forward ...
nsys stats --report cuda_gpu_kern_sum --format csv -o kern prof.1.nsys-rep
```
Keep `--capture-range-end=repeat`: nsys's default stops after the first case
and leaves the bench hanging. `nsys stats` also counts the KV restore copies
that the JSON summary excludes.
## Caveats
- **Dummy weights lie for data-dependent kernels.** Dummy gates route every
.agents/skills/model-recipes/SKILL.md
+11 -5
View File
@@ -5,9 +5,12 @@ description: Find vLLM deployment recipes and configs for LLMs and Hugging Face
# Model Recipes
Model serving recipes are maintained in the
[models/ directory of vllm-project/recipes](https://github.com/vllm-project/recipes/tree/main/models).
The published catalog is at
Model serving recipes are maintained in this repository's
[recipes/models/ directory](../../../recipes/models). Omni (image, video,
audio) recipes stay in vllm-project/recipes for now (moving to vllm-omni),
and Ascend NPU content is inline in these recipes (moving to vllm-ascend
overlays). The published catalog, rendered by
[vllm-project/recipes](https://github.com/vllm-project/recipes), is at
[recipes.vllm.ai](https://recipes.vllm.ai).
## Usage notes
@@ -146,6 +149,9 @@ uploads, untested hardware claims, or non-standard flags that only work for one
setup. Before opening a pull request, confirm that the user ran the
configuration and can share the results.
Follow the recipes repository's
Edit the YAML under [recipes/models/](../../../recipes/models) in this
repository, following the recipes repository's
[add-recipe skill](https://github.com/vllm-project/recipes/blob/main/.agents/skills/add-recipe/SKILL.md)
in a checkout of `vllm-project/recipes` or the user's intended fork.
for the schema. Validate and preview it from a checkout of
`vllm-project/recipes` pointed at this directory, as
[recipes/README.md](../../../recipes/README.md) describes.
.buildkite/release-pipeline.yaml
+208 -473
View File
@@ -703,385 +703,6 @@ steps:
DOCKER_BUILDKIT: "1"
DOCKERHUB_USERNAME: "vllmbot"
# =============================================================================
# ROCm Release Pipeline (x86_64 only)
# =============================================================================
#
# vLLM version is determined by the Buildkite checkout (like CUDA pipeline).
# To build a specific version, trigger the build from that branch/tag.
#
# Environment variables for ROCm builds (set via Buildkite UI or schedule):
#
# Note: ROCm version is determined by BASE_IMAGE in docker/Dockerfile.rocm_base
#
# =============================================================================
- group: "Build ROCm Wheel / Image "
key: "build-rocm-wheel-image"
depends_on: ~
steps:
# ROCm Job 1: Build ROCm Base Wheels (with S3 caching)
- label: ":rocm: Build ROCm Base Image & Wheels"
id: build-rocm-base-wheels
depends_on: ~
agents:
queue: cpu_queue_release
commands:
- |
set -euo pipefail
# Generate cache key
CACHE_KEY=$$(.buildkite/scripts/cache-rocm-base-wheels.sh key)
ECR_CACHE_TAG="public.ecr.aws/q9t5s3a7/vllm-release-repo:$${CACHE_KEY}-rocm-base"
echo "========================================"
echo "ROCm Base Build Configuration"
echo "========================================"
echo " CACHE_KEY: $${CACHE_KEY}"
echo " ECR_CACHE_TAG: $${ECR_CACHE_TAG}"
echo "========================================"
# Login to ECR
aws ecr-public get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
IMAGE_EXISTS=false
WHEELS_EXIST=false
# Check ECR for Docker image
if docker manifest inspect "$${ECR_CACHE_TAG}" > /dev/null 2>&1; then
IMAGE_EXISTS=true
echo "ECR image cache HIT"
fi
# Check S3 for wheels
WHEEL_CACHE_STATUS=$(.buildkite/scripts/cache-rocm-base-wheels.sh check)
if [ "$${WHEEL_CACHE_STATUS}" = "hit" ]; then
WHEELS_EXIST=true
echo "S3 wheels cache HIT"
fi
# Scenario 1: Both cached (best case)
if [ "$${IMAGE_EXISTS}" = "true" ] && [ "$${WHEELS_EXIST}" = "true" ]; then
echo ""
echo "FULL CACHE HIT - Reusing both image and wheels"
echo ""
# Download wheels
.buildkite/scripts/cache-rocm-base-wheels.sh download
# Save ECR tag for downstream jobs
buildkite-agent meta-data set "rocm-base-image-tag" "$${ECR_CACHE_TAG}"
# Scenario 2: Full rebuild needed
else
echo ""
echo " CACHE MISS - Building from scratch..."
echo ""
# Build full base image and push to ECR
DOCKER_BUILDKIT=1 docker buildx build \
--file docker/Dockerfile.rocm_base \
--tag "$${ECR_CACHE_TAG}" \
--build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 \
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
--push \
.
# Build wheel extraction stage
DOCKER_BUILDKIT=1 docker buildx build \
--file docker/Dockerfile.rocm_base \
--tag rocm-base-debs:$${BUILDKITE_BUILD_NUMBER} \
--target debs_wheel_release \
--build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 \
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
--load \
.
# Extract and upload wheels
mkdir -p artifacts/rocm-base-wheels
cid=$(docker create rocm-base-debs:$${BUILDKITE_BUILD_NUMBER})
docker cp $${cid}:/app/debs/. artifacts/rocm-base-wheels/
docker rm $${cid}
.buildkite/scripts/cache-rocm-base-wheels.sh upload
# Cache base docker image to ECR
docker push "$${ECR_CACHE_TAG}"
buildkite-agent meta-data set "rocm-base-image-tag" "$${ECR_CACHE_TAG}"
echo ""
echo " Build complete - Image and wheels cached"
fi
artifact_paths:
- "artifacts/rocm-base-wheels/*.whl"
env:
DOCKER_BUILDKIT: "1"
S3_BUCKET: "vllm-wheels"
# ROCm Job 2: Build vLLM ROCm Wheel
- label: ":python: Build vLLM ROCm Wheel - x86_64"
id: build-rocm-vllm-wheel
depends_on:
- step: build-rocm-base-wheels
allow_failure: false
agents:
queue: cpu_queue_release
timeout_in_minutes: 180
commands:
# Download artifacts and prepare Docker image
- |
set -euo pipefail
# Ensure git tags are up-to-date (Buildkite's default fetch doesn't update tags)
# This fixes version detection when tags are moved/force-pushed
echo "Fetching latest tags from origin..."
git fetch --tags --force origin
# Log tag information for debugging version detection
echo "========================================"
echo "Git Tag Verification"
echo "========================================"
echo "Current HEAD: $(git rev-parse HEAD)"
echo "git describe --tags: $(git describe --tags 2>/dev/null || echo 'No tags found')"
echo ""
echo "Recent tags (pointing to commits near HEAD):"
git tag -l --sort=-creatordate | head -5
echo "setuptools_scm version detection:"
pip install -q setuptools_scm 2>/dev/null || true
python3 -c "import setuptools_scm; print(' Detected version:', setuptools_scm.get_version())" 2>/dev/null || echo " (setuptools_scm not available in this environment)"
echo "========================================"
# Download wheel artifacts from current build
echo "Downloading wheel artifacts from current build"
buildkite-agent artifact download "artifacts/rocm-base-wheels/*.whl" .
# Get ECR image tag from metadata (set by build-rocm-base-wheels)
ECR_IMAGE_TAG="$$(buildkite-agent meta-data get rocm-base-image-tag 2>/dev/null || echo '')"
if [ -z "$${ECR_IMAGE_TAG}" ]; then
echo "ERROR: rocm-base-image-tag metadata not found"
echo "This should have been set by the build-rocm-base-wheels job"
exit 1
fi
echo "Pulling base Docker image from ECR: $${ECR_IMAGE_TAG}"
# Login to ECR
aws ecr-public get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
# Pull base Docker image from ECR
docker pull "$${ECR_IMAGE_TAG}"
echo "Loaded base image: $${ECR_IMAGE_TAG}"
# Prepare base wheels for Docker build context
mkdir -p docker/context/base-wheels
touch docker/context/base-wheels/.keep
cp artifacts/rocm-base-wheels/*.whl docker/context/base-wheels/
echo "Base wheels for vLLM build:"
ls -lh docker/context/base-wheels/
echo "========================================"
echo "Building vLLM wheel with:"
echo " BUILDKITE_COMMIT: $${BUILDKITE_COMMIT}"
echo " BUILDKITE_BRANCH: $${BUILDKITE_BRANCH}"
echo " BASE_IMAGE: $${ECR_IMAGE_TAG}"
echo "========================================"
# Build vLLM wheel using local checkout (REMOTE_VLLM=0)
DOCKER_BUILDKIT=1 docker build \
--file docker/Dockerfile.rocm \
--target export_vllm_wheel_release \
--output type=local,dest=rocm-dist \
--build-arg BASE_IMAGE="$${ECR_IMAGE_TAG}" \
--build-arg REMOTE_VLLM=0 \
--build-arg GIT_REPO_CHECK=1 \
--build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 \
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
.
echo "Built vLLM wheel:"
ls -lh rocm-dist/*.whl
# Copy wheel to artifacts directory
mkdir -p artifacts/rocm-vllm-wheel
cp rocm-dist/*.whl artifacts/rocm-vllm-wheel/
echo "Final vLLM wheel:"
ls -lh artifacts/rocm-vllm-wheel/
artifact_paths:
- "artifacts/rocm-vllm-wheel/*.whl"
env:
DOCKER_BUILDKIT: "1"
S3_BUCKET: "vllm-wheels"
# ROCm Job 3: Upload Wheels to S3
- label: ":s3: Upload ROCm Wheels to S3"
id: upload-rocm-wheels
depends_on:
- step: build-rocm-vllm-wheel
allow_failure: false
agents:
queue: cpu_queue_release
timeout_in_minutes: 60
commands:
# Download all wheel artifacts and run upload
- |
set -euo pipefail
# Download artifacts from current build
echo "Downloading artifacts from current build"
buildkite-agent artifact download "artifacts/rocm-base-wheels/*.whl" .
buildkite-agent artifact download "artifacts/rocm-vllm-wheel/*.whl" .
# # Run upload script
bash .buildkite/scripts/upload-rocm-wheels.sh
env:
DOCKER_BUILDKIT: "1"
S3_BUCKET: "vllm-wheels"
# ROCm Job 4: Annotate ROCm Wheel Release
- label: ":memo: Annotate ROCm wheel release"
id: annotate-rocm-release
depends_on:
- upload-rocm-wheels
agents:
queue: cpu_queue_release
commands:
- "bash .buildkite/scripts/annotate-rocm-release.sh"
env:
S3_BUCKET: "vllm-wheels"
# ROCm Job 5: Generate Root Index for ROCm Wheels (for release only)
# This is the job to create https://wheels.vllm.ai/rocm/ index allowing
# users to install with `uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/`
- block: "Generate Root Index for ROCm Wheels for Release"
key: block-generate-root-index-rocm-wheels
depends_on: upload-rocm-wheels
- label: ":package: Generate Root Index for ROCm Wheels for Release"
depends_on: block-generate-root-index-rocm-wheels
id: generate-root-index-rocm-wheels
agents:
queue: cpu_queue_release
commands:
- "bash tools/vllm-rocm/generate-rocm-wheels-root-index.sh"
env:
S3_BUCKET: "vllm-wheels"
VARIANT: "rocm723"
# ROCm Job 6: Build ROCm Release Docker Image
- label: ":docker: Build release image - x86_64 - ROCm"
id: build-rocm-release-image
depends_on:
- step: block-build-release-images
allow_failure: true
- step: build-rocm-base-wheels
allow_failure: false
agents:
queue: cpu_queue_release
timeout_in_minutes: 60
commands:
- |
set -euo pipefail
# Login to ECR
aws ecr-public get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
# Get ECR image tag from metadata (set by build-rocm-base-wheels)
ECR_IMAGE_TAG="$$(buildkite-agent meta-data get rocm-base-image-tag 2>/dev/null || echo '')"
if [ -z "$${ECR_IMAGE_TAG}" ]; then
echo "ERROR: rocm-base-image-tag metadata not found"
echo "This should have been set by the build-rocm-base-wheels job"
exit 1
fi
echo "Pulling base Docker image from ECR: $${ECR_IMAGE_TAG}"
# Pull base Docker image from ECR
docker pull "$${ECR_IMAGE_TAG}"
echo "Loaded base image: $${ECR_IMAGE_TAG}"
# Pass the base image ECR tag to downstream steps (nightly publish)
buildkite-agent meta-data set "rocm-base-ecr-tag" "$${ECR_IMAGE_TAG}"
echo "========================================"
echo "Building vLLM ROCm release image with:"
echo " BASE_IMAGE: $${ECR_IMAGE_TAG}"
echo " BUILDKITE_COMMIT: $${BUILDKITE_COMMIT}"
echo "========================================"
# Build vLLM ROCm release image using cached base
DOCKER_BUILDKIT=1 docker build \
--build-arg max_jobs=16 \
--build-arg BASE_IMAGE="$${ECR_IMAGE_TAG}" \
--build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 \
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
--build-arg INSTALL_LMCACHE=true \
--tag public.ecr.aws/q9t5s3a7/vllm-release-repo:$${BUILDKITE_COMMIT}-rocm \
--target vllm-openai \
--progress plain \
-f docker/Dockerfile.rocm .
# Push to ECR
docker push public.ecr.aws/q9t5s3a7/vllm-release-repo:$${BUILDKITE_COMMIT}-rocm
echo ""
echo " Successfully built and pushed ROCm release image"
echo " Image: public.ecr.aws/q9t5s3a7/vllm-release-repo:$${BUILDKITE_COMMIT}-rocm"
echo ""
env:
DOCKER_BUILDKIT: "1"
S3_BUCKET: "vllm-wheels"
- label: "Publish nightly XPU image to DockerHub"
depends_on:
- create-manifest-xpu
if: build.env("NIGHTLY") == "1"
agents:
queue: small_cpu_queue_release
commands:
- "bash .buildkite/scripts/xpu/push-nightly-builds-xpu.sh"
- "bash .buildkite/scripts/cleanup-nightly-builds.sh nightly- vllm/vllm-openai-xpu"
plugins:
- docker-login#v3.0.0:
username: vllmbot
password-env: DOCKERHUB_TOKEN
env:
DOCKER_BUILDKIT: "1"
DOCKERHUB_USERNAME: "vllmbot"
- label: "Publish nightly ROCm image to DockerHub"
depends_on:
- build-rocm-release-image
if: build.env("NIGHTLY") == "1"
agents:
queue: small_cpu_queue_release
commands:
- "bash .buildkite/scripts/push-nightly-builds-rocm.sh"
# Clean up old nightly builds (keep only last 14)
- "bash .buildkite/scripts/cleanup-nightly-builds.sh nightly- vllm/vllm-openai-rocm"
- "bash .buildkite/scripts/cleanup-nightly-builds.sh base-nightly- vllm/vllm-openai-rocm"
plugins:
- docker-login#v3.0.0:
username: vllmbot
password-env: DOCKERHUB_TOKEN
env:
DOCKER_BUILDKIT: "1"
DOCKERHUB_USERNAME: "vllmbot"
- trigger: "perf-eval"
label: "Trigger perf-eval (CUDA)"
key: trigger-perf-eval-cuda
@@ -1103,7 +724,7 @@ steps:
key: trigger-perf-eval-rocm
if: build.env("NIGHTLY") == "1" || build.env("RUN_PERF_EVAL") == "1"
depends_on:
- build-rocm-release-image
- build-rocm100-release-image
async: true
soft_fail: true
build:
@@ -1112,138 +733,250 @@ steps:
env:
NIGHTLY: 1 # run all perf-eval workloads
VLLM_COMMIT: "$BUILDKITE_COMMIT"
VLLM_IMAGE_ROCM: "public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-rocm"
VLLM_IMAGE_ROCM: "public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-rocm100"
# =============================================================================
# ROCk Release Pipeline (x86_64 only)
# ROCm Release Pipelines (x86_64 only)
# =============================================================================
#
# Same shape as the ROCm pipeline above (Jobs 1 and 6 only), but built from
# docker/Dockerfile.rock_base and docker/Dockerfile.rock, which install the
# ROCm SDK from TheRock wheels instead of the ROCm apt repos.
# One group per ROCm stack, identified by its base Dockerfile (ROCM_BASE_DOCKERFILE).
# The scripts derive the ROCm version, wheel variant, cache keys and image tags from
# it (.buildkite/scripts/rocm/stack.sh), so both groups share one set of step
# definitions, anchored on the ROCm 10.0 group.
#
# Note: ROCm SDK version is determined by ROCM_SDK_VERSION in
# docker/Dockerfile.rock_base
# - ROCm 10.0 (TheRock, docker/Dockerfile.rocm{,_base}) is the default: it owns the
# :nightly / :latest images and the root wheel index.
# - ROCm 7.2 (docker/Dockerfile.rocm_72{,_base}) is kept for a transition period as
# rocm72 images and wheels.
#
# vLLM version is determined by the Buildkite checkout (like the CUDA pipeline).
# =============================================================================
- group: "Build ROCk Image"
key: "build-rock-image"
- group: "Build ROCm 10.0 Wheel / Image"
key: "build-rocm100"
depends_on: ~
steps:
# ROCk Job 1: Build ROCk Base Image (with ECR caching)
- label: ":rocm: Build ROCk Base Image"
id: build-rock-base-image
- label: ":rocm: Build ROCm 10.0 Base Image & Wheels"
id: build-rocm100-base
depends_on: ~
agents:
agents: &rocm_build_agents
queue: cpu_queue_release
timeout_in_minutes: 360
commands:
env: &rocm100_env
ROCM_BASE_DOCKERFILE: "docker/Dockerfile.rocm_base"
ROCM_DEFAULT_ARG: "--default"
DOCKER_BUILDKIT: "1"
S3_BUCKET: "vllm-wheels"
DOCKERHUB_USERNAME: "vllmbot"
commands: &rocm_base_commands
- |
set -euo pipefail
# The SDK version, torch wheels and component branches all live in
# the Dockerfile, so its hash is the cache key.
CACHE_KEY=$$(sha256sum docker/Dockerfile.rock_base | cut -c1-16)
ECR_CACHE_TAG="public.ecr.aws/q9t5s3a7/vllm-release-repo:$${CACHE_KEY}-rock-base"
echo "========================================"
echo "ROCk Base Build Configuration"
echo "========================================"
echo " CACHE_KEY: $${CACHE_KEY}"
echo " ECR_CACHE_TAG: $${ECR_CACHE_TAG}"
echo "========================================"
# Login to ECR
source .buildkite/scripts/rocm/stack.sh "$${ROCM_BASE_DOCKERFILE}"
SCCACHE_ARGS=(--build-arg USE_SCCACHE=1 --build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache
--build-arg SCCACHE_REGION_NAME=us-west-2 --build-arg SCCACHE_S3_NO_CREDENTIALS=0)
aws ecr-public get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
if docker manifest inspect "$${ECR_CACHE_TAG}" > /dev/null 2>&1; then
echo "ECR image cache HIT"
# Base image, cached in ECR by the base Dockerfile's hash
if docker manifest inspect "$${ROCM_STACK_ECR_BASE}" > /dev/null 2>&1; then
echo "ECR image cache HIT: $${ROCM_STACK_ECR_BASE}"
else
echo "CACHE MISS - Building from scratch..."
DOCKER_BUILDKIT=1 docker buildx build \
--file docker/Dockerfile.rock_base \
--tag "$${ECR_CACHE_TAG}" \
--build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 \
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
--push \
.
docker buildx build --file "$${ROCM_BASE_DOCKERFILE}" --tag "$${ROCM_STACK_ECR_BASE}" \
"$${SCCACHE_ARGS[@]}" --push .
fi
# Save ECR tag for downstream jobs
buildkite-agent meta-data set "rock-base-image-tag" "$${ECR_CACHE_TAG}"
env:
DOCKER_BUILDKIT: "1"
# Base wheels, cached in S3 (see cache-rocm-base-wheels.sh)
CACHE=.buildkite/scripts/cache-rocm-base-wheels.sh
if [ "$$($${CACHE} "$${ROCM_BASE_DOCKERFILE}" check)" = "hit" ]; then
$${CACHE} "$${ROCM_BASE_DOCKERFILE}" download
else
DEBS="rocm-base-debs:$${BUILDKITE_BUILD_NUMBER}-$${ROCM_STACK_VARIANT}"
docker buildx build --file "$${ROCM_BASE_DOCKERFILE}" --tag "$${DEBS}" \
--target debs_wheel_release "$${SCCACHE_ARGS[@]}" --load .
mkdir -p "$${ROCM_STACK_BASE_WHEELS_DIR}"
cid=$$(docker create "$${DEBS}")
docker cp "$${cid}":/app/debs/. "$${ROCM_STACK_BASE_WHEELS_DIR}/"
docker rm "$${cid}"
$${CACHE} "$${ROCM_BASE_DOCKERFILE}" upload
fi
buildkite-agent artifact upload "$${ROCM_STACK_BASE_WHEELS_DIR}/*.whl"
# ROCk Job 2: Build ROCk Release Docker Image
- label: ":docker: Build release image - x86_64 - ROCk"
id: build-rock-release-image
- label: ":python: Build vLLM ROCm 10.0 Wheel - x86_64"
id: build-rocm100-vllm-wheel
depends_on: build-rocm100-base
agents: *rocm_build_agents
timeout_in_minutes: 180
env: *rocm100_env
commands: &rocm_vllm_wheel_commands
- |
set -euo pipefail
source .buildkite/scripts/rocm/stack.sh "$${ROCM_BASE_DOCKERFILE}"
# Buildkite's default fetch doesn't update moved tags; version detection needs them
git fetch --tags --force origin
buildkite-agent artifact download "$${ROCM_STACK_BASE_WHEELS_DIR}/*.whl" .
aws ecr-public get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
docker pull "$${ROCM_STACK_ECR_BASE}"
mkdir -p docker/context/base-wheels
touch docker/context/base-wheels/.keep
cp "$${ROCM_STACK_BASE_WHEELS_DIR}"/*.whl docker/context/base-wheels/
docker build --file "$${ROCM_STACK_DOCKERFILE}" --target export_vllm_wheel_release \
--output type=local,dest=rocm-dist \
--build-arg BASE_IMAGE="$${ROCM_STACK_ECR_BASE}" --build-arg REMOTE_VLLM=0 \
--build-arg GIT_REPO_CHECK=1 --build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 --build-arg SCCACHE_S3_NO_CREDENTIALS=0 .
mkdir -p "$${ROCM_STACK_VLLM_WHEEL_DIR}"
cp rocm-dist/*.whl "$${ROCM_STACK_VLLM_WHEEL_DIR}/"
buildkite-agent artifact upload "$${ROCM_STACK_VLLM_WHEEL_DIR}/*.whl"
- label: ":s3: Upload ROCm 10.0 Wheels to S3"
id: upload-rocm100-wheels
depends_on: build-rocm100-vllm-wheel
agents: *rocm_build_agents
timeout_in_minutes: 60
env: *rocm100_env
commands: &rocm_upload_commands
- |
set -euo pipefail
source .buildkite/scripts/rocm/stack.sh "$${ROCM_BASE_DOCKERFILE}"
buildkite-agent artifact download "$${ROCM_STACK_BASE_WHEELS_DIR}/*.whl" .
buildkite-agent artifact download "$${ROCM_STACK_VLLM_WHEEL_DIR}/*.whl" .
bash .buildkite/scripts/upload-rocm-wheels.sh "$${ROCM_BASE_DOCKERFILE}"
- label: ":memo: Annotate ROCm 10.0 wheel release"
id: annotate-rocm100-release
depends_on: upload-rocm100-wheels
agents: *rocm_build_agents
env: *rocm100_env
commands: &rocm_annotate_commands
- bash .buildkite/scripts/annotate-rocm-release.sh "$${ROCM_BASE_DOCKERFILE}" $${ROCM_DEFAULT_ARG}
# Generate https://wheels.vllm.ai/rocm/ (the default stack's variant), so users can
# `uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/` (release only)
- block: "Generate Root Index for ROCm Wheels for Release"
key: block-generate-root-index-rocm-wheels
depends_on:
- upload-rocm100-wheels
- upload-rocm72-wheels
- label: ":package: Generate Root Index for ROCm Wheels for Release"
depends_on: block-generate-root-index-rocm-wheels
id: generate-root-index-rocm-wheels
agents: *rocm_build_agents
env: *rocm100_env
commands:
- "bash tools/vllm-rocm/generate-rocm-wheels-root-index.sh"
- label: ":docker: Build release image - x86_64 - ROCm 10.0"
id: build-rocm100-release-image
depends_on:
- step: block-build-release-images
allow_failure: true
- step: build-rock-base-image
- step: build-rocm100-base
allow_failure: false
agents:
queue: cpu_queue_release
agents: *rocm_build_agents
timeout_in_minutes: 180
commands:
env: *rocm100_env
commands: &rocm_release_image_commands
- |
set -euo pipefail
# Login to ECR
source .buildkite/scripts/rocm/stack.sh "$${ROCM_BASE_DOCKERFILE}"
aws ecr-public get-login-password --region us-east-1 | \
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
ECR_IMAGE_TAG="$$(buildkite-agent meta-data get rock-base-image-tag 2>/dev/null || echo '')"
if [ -z "$${ECR_IMAGE_TAG}" ]; then
echo "ERROR: rock-base-image-tag metadata not found"
echo "This should have been set by the build-rock-base-image job"
exit 1
fi
docker pull "$${ECR_IMAGE_TAG}"
# Pass the base image ECR tag to downstream steps (nightly publish)
buildkite-agent meta-data set "rock-base-ecr-tag" "$${ECR_IMAGE_TAG}"
echo "========================================"
echo "Building vLLM ROCk release image with:"
echo " BASE_IMAGE: $${ECR_IMAGE_TAG}"
echo " BUILDKITE_COMMIT: $${BUILDKITE_COMMIT}"
echo "========================================"
DOCKER_BUILDKIT=1 docker build \
--build-arg max_jobs=16 \
--build-arg BASE_IMAGE="$${ECR_IMAGE_TAG}" \
--build-arg USE_SCCACHE=1 \
docker pull "$${ROCM_STACK_ECR_BASE}"
docker build --file "$${ROCM_STACK_DOCKERFILE}" --target vllm-openai \
--tag "$${ROCM_STACK_ECR_IMAGE}" --progress plain \
--build-arg max_jobs=16 --build-arg BASE_IMAGE="$${ROCM_STACK_ECR_BASE}" \
--build-arg INSTALL_LMCACHE=true --build-arg USE_SCCACHE=1 \
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
--build-arg SCCACHE_REGION_NAME=us-west-2 \
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
--build-arg INSTALL_LMCACHE=true \
--tag public.ecr.aws/q9t5s3a7/vllm-release-repo:$${BUILDKITE_COMMIT}-rock \
--target vllm-openai \
--progress plain \
-f docker/Dockerfile.rock .
--build-arg SCCACHE_REGION_NAME=us-west-2 --build-arg SCCACHE_S3_NO_CREDENTIALS=0 .
docker push "$${ROCM_STACK_ECR_IMAGE}"
docker push public.ecr.aws/q9t5s3a7/vllm-release-repo:$${BUILDKITE_COMMIT}-rock
env:
- label: "Publish nightly ROCm 10.0 image to DockerHub"
id: publish-rocm100-nightly
depends_on: build-rocm100-release-image
if: build.env("NIGHTLY") == "1"
agents: &rocm_publish_agents
queue: small_cpu_queue_release
env: *rocm100_env
commands: &rocm_publish_nightly_commands
- bash .buildkite/scripts/push-nightly-builds-rocm.sh "$${ROCM_BASE_DOCKERFILE}" $${ROCM_DEFAULT_ARG}
plugins: &rocm_dockerhub_login
- docker-login#v3.0.0:
username: vllmbot
password-env: DOCKERHUB_TOKEN
- group: "Build ROCm 7.2 Wheel / Image"
key: "build-rocm72"
depends_on: ~
steps:
- label: ":rocm: Build ROCm 7.2 Base Image & Wheels"
id: build-rocm72-base
depends_on: ~
agents: *rocm_build_agents
timeout_in_minutes: 360
env: &rocm72_env
ROCM_BASE_DOCKERFILE: "docker/Dockerfile.rocm_72_base"
ROCM_DEFAULT_ARG: ""
DOCKER_BUILDKIT: "1"
S3_BUCKET: "vllm-wheels"
DOCKERHUB_USERNAME: "vllmbot"
commands: *rocm_base_commands
- label: "Publish nightly ROCk image to DockerHub"
- label: ":python: Build vLLM ROCm 7.2 Wheel - x86_64"
id: build-rocm72-vllm-wheel
depends_on: build-rocm72-base
agents: *rocm_build_agents
timeout_in_minutes: 180
env: *rocm72_env
commands: *rocm_vllm_wheel_commands
- label: ":s3: Upload ROCm 7.2 Wheels to S3"
id: upload-rocm72-wheels
depends_on: build-rocm72-vllm-wheel
agents: *rocm_build_agents
timeout_in_minutes: 60
env: *rocm72_env
commands: *rocm_upload_commands
- label: ":memo: Annotate ROCm 7.2 wheel release"
id: annotate-rocm72-release
depends_on: upload-rocm72-wheels
agents: *rocm_build_agents
env: *rocm72_env
commands: *rocm_annotate_commands
- label: ":docker: Build release image - x86_64 - ROCm 7.2"
id: build-rocm72-release-image
depends_on:
- build-rock-release-image
- step: block-build-release-images
allow_failure: true
- step: build-rocm72-base
allow_failure: false
agents: *rocm_build_agents
timeout_in_minutes: 180
env: *rocm72_env
commands: *rocm_release_image_commands
- label: "Publish nightly ROCm 7.2 image to DockerHub"
id: publish-rocm72-nightly
depends_on: build-rocm72-release-image
if: build.env("NIGHTLY") == "1"
agents: *rocm_publish_agents
env: *rocm72_env
commands: *rocm_publish_nightly_commands
plugins: *rocm_dockerhub_login
- label: "Publish nightly XPU image to DockerHub"
depends_on:
- create-manifest-xpu
if: build.env("NIGHTLY") == "1"
agents:
queue: small_cpu_queue_release
commands:
- "bash .buildkite/scripts/push-nightly-builds-rocm.sh rock rocm100"
# Clean up old nightly builds (keep only last 14)
- "bash .buildkite/scripts/cleanup-nightly-builds.sh nightly-rocm100- vllm/vllm-openai-rocm"
- "bash .buildkite/scripts/cleanup-nightly-builds.sh base-nightly-rocm100- vllm/vllm-openai-rocm"
- "bash .buildkite/scripts/xpu/push-nightly-builds-xpu.sh"
- "bash .buildkite/scripts/cleanup-nightly-builds.sh nightly- vllm/vllm-openai-xpu"
plugins:
- docker-login#v3.0.0:
username: vllmbot
@@ -1264,7 +997,8 @@ steps:
- create-multi-arch-manifest-ubuntu2404
- create-multi-arch-manifest-cuda-12-9-ubuntu2404
- create-manifest-xpu
- build-rocm-release-image
- build-rocm100-release-image
- build-rocm72-release-image
- input-release-version
# Wait for CPU builds if their block steps were unblocked, so publish
# doesn't race the in-progress CPU build. allow_failure lets publish
@@ -1289,6 +1023,7 @@ steps:
- "cuda-13-0-ubuntu-24-04"
- "cuda-12-9-ubuntu-24-04"
- "rocm"
- "rocm72"
- "xpu"
- "cpu"
plugins:
.buildkite/scripts/annotate-rocm-release.sh
+40 -66
View File
@@ -2,15 +2,23 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
#
# Generate Buildkite annotation for ROCm wheel release
# Generate Buildkite annotation for a ROCm stack's wheel release.
#
# Usage: annotate-rocm-release.sh <base-dockerfile> [--default]
# base-dockerfile e.g. docker/Dockerfile.rocm_base or docker/Dockerfile.rocm_72_base
# --default the stack also owns the plain :latest / :v<ver> Docker Hub tags
set -ex
# Extract build configuration from Dockerfile.rocm_base (single source of truth)
# Extract ROCm version dynamically from Dockerfile.rocm_base
# BASE_IMAGE format: rocm/dev-ubuntu-22.04:7.0-complete -> extracts "7.0"
ROCM_VERSION=$(grep -E '^ARG BASE_IMAGE=' docker/Dockerfile.rocm_base | sed -E 's/.*:([0-9]+\.[0-9]+).*/\1/' || echo "unknown")
PYTHON_VERSION=$(grep '^ARG PYTHON_VERSION=' docker/Dockerfile.rocm_base | sed 's/^ARG PYTHON_VERSION=//')
PYTORCH_ROCM_ARCH=$(grep '^ARG PYTORCH_ROCM_ARCH=' docker/Dockerfile.rocm_base | sed 's/^ARG PYTORCH_ROCM_ARCH=//')
# shellcheck source=.buildkite/scripts/rocm/stack.sh
source "$(dirname "${BASH_SOURCE[0]}")/rocm/stack.sh" "${1:?Usage: $0 <base-dockerfile> [--default]}"
TAG_VARIANTS=("-${ROCM_STACK_VARIANT}")
if [[ "${2:-}" == "--default" ]]; then
TAG_VARIANTS=("" "-${ROCM_STACK_VARIANT}")
fi
ROCM_VERSION="$ROCM_STACK_VERSION"
VARIANT="$ROCM_STACK_VARIANT"
PYTHON_VERSION=$(sed -nE 's/^ARG PYTHON_VERSION="?([^" ]+)"?.*/\1/p' "$ROCM_STACK_BASE_DOCKERFILE")
PYTORCH_ROCM_ARCH=$(sed -nE 's/^ARG PYTORCH_ROCM_ARCH="?([^"]+)"?.*/\1/p' "$ROCM_STACK_BASE_DOCKERFILE")
# Get release version, default to 1.0.0.dev for nightly/per-commit builds
RELEASE_VERSION=$(buildkite-agent meta-data get release-version 2>/dev/null || echo "")
@@ -18,22 +26,32 @@ if [ -z "${RELEASE_VERSION}" ]; then
RELEASE_VERSION="1.0.0.dev"
fi
ROCM_BASE_CACHE_KEY=$(.buildkite/scripts/cache-rocm-base-wheels.sh key)
# S3 URLs
S3_BUCKET="${S3_BUCKET:-vllm-wheels}"
S3_REGION="${AWS_DEFAULT_REGION:-us-west-2}"
S3_URL="http://${S3_BUCKET}.s3-website-${S3_REGION}.amazonaws.com"
S3_HOST="${S3_BUCKET}.s3-website-${S3_REGION}.amazonaws.com"
S3_URL="http://${S3_HOST}"
WHEEL_PATH="rocm/${BUILDKITE_COMMIT}/${VARIANT}-wheels/"
# Format ROCm version for path (e.g., "7.1" -> "rocm710")
ROCM_VERSION_PATH="rocm$(echo "${ROCM_VERSION}" | tr -d '.')"
ROCM_PATH="rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}"
buildkite-agent annotate --style 'success' --context 'rocm-release-workflow' << EOF
## ROCm Wheel and Docker Image Releases
DOCKER_CMDS="docker pull ${ROCM_STACK_ECR_BASE}
docker pull ${ROCM_STACK_ECR_IMAGE}"
for suffix in "${TAG_VARIANTS[@]}"; do
for tag in "latest${suffix}" "v${RELEASE_VERSION}${suffix}"; do
DOCKER_CMDS+="
docker tag ${ROCM_STACK_ECR_BASE} vllm/vllm-openai-rocm:${tag}-base
docker tag ${ROCM_STACK_ECR_IMAGE} vllm/vllm-openai-rocm:${tag}
docker push vllm/vllm-openai-rocm:${tag}-base
docker push vllm/vllm-openai-rocm:${tag}"
done
done
buildkite-agent annotate --style 'success' --context "rocm-release-workflow-${VARIANT}" << EOF
## ROCm ${ROCM_VERSION} Wheel and Docker Image Releases
### Build Configuration
| Setting | Value |
|---------|-------|
| **ROCm Version** | ${ROCM_VERSION} |
| **Index Variant** | ${VARIANT} |
| **Python Version** | ${PYTHON_VERSION} |
| **GPU Architectures** | ${PYTORCH_ROCM_ARCH} |
| **Branch** | \`${BUILDKITE_BRANCH}\` |
@@ -41,49 +59,20 @@ buildkite-agent annotate --style 'success' --context 'rocm-release-workflow' <<
### :package: Installation
**Install from this build (by commit):**
\`\`\`bash
pip install vllm --extra-index-url ${S3_URL}/${ROCM_PATH}/ --trusted-host ${S3_BUCKET}.s3-website-${S3_REGION}.amazonaws.com
# Example for ROCm ${ROCM_VERSION}:
pip install vllm --extra-index-url ${S3_URL}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/ --trusted-host ${S3_BUCKET}.s3-website-${S3_REGION}.amazonaws.com
\`\`\`
**Install from nightly (if published):**
\`\`\`bash
pip install vllm --extra-index-url ${S3_URL}/rocm/nightly/ --trusted-host ${S3_BUCKET}.s3-website-${S3_REGION}.amazonaws.com
# This build (by commit)
pip install vllm --extra-index-url ${S3_URL}/rocm/${BUILDKITE_COMMIT}/${VARIANT}/ --trusted-host ${S3_HOST}
# Nightly (if published)
pip install vllm --extra-index-url ${S3_URL}/rocm/nightly/${VARIANT}/ --trusted-host ${S3_HOST}
\`\`\`
### :floppy_disk: Download Wheels Directly
\`\`\`bash
# List all ROCm wheels
aws s3 ls s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/
# Download specific wheels
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/vllm-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/torch-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/triton-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/triton-kernels-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/torchvision-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/torchaudio-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/amdsmi-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/amd_aiter-*.whl .
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/flash-attn-*.whl .
aws s3 ls s3://${S3_BUCKET}/${WHEEL_PATH}
aws s3 cp --recursive --exclude "*" --include "*.whl" s3://${S3_BUCKET}/${WHEEL_PATH} .
\`\`\`
### :gear: Included Packages
- **vllm**: vLLM with ROCm support
- **torch**: PyTorch built for ROCm ${ROCM_VERSION}
- **triton**: Triton
- **triton-kernels**: Triton kernels
- **torchvision**: TorchVision for ROCm PyTorch
- **torchaudio**: Torchaudio for ROCm PyTorch
- **amdsmi**: AMD SMI Python bindings
- **amd_aiter**: Aiter for ROCm
- **flash-attn**: Flash Attention for ROCm
### :warning: Notes
- These wheels are built for **ROCm ${ROCM_VERSION}** and will NOT work with CUDA GPUs
- Supported GPU architectures: ${PYTORCH_ROCM_ARCH}
@@ -91,23 +80,8 @@ aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/flash-
### :package: Docker Image Release
To download and upload the image:
\`\`\`
docker pull public.ecr.aws/q9t5s3a7/vllm-release-repo:${BUILDKITE_COMMIT}-rocm-base
docker pull public.ecr.aws/q9t5s3a7/vllm-release-repo:${BUILDKITE_COMMIT}-rocm
docker tag public.ecr.aws/q9t5s3a7/vllm-release-repo:${ROCM_BASE_CACHE_KEY}-rocm-base vllm/vllm-openai-rocm:${BUILDKITE_COMMIT}-base
docker tag vllm/vllm-openai-rocm:${BUILDKITE_COMMIT}-base vllm/vllm-openai-rocm:latest-base
docker tag vllm/vllm-openai-rocm:${BUILDKITE_COMMIT}-base vllm/vllm-openai-rocm:v${RELEASE_VERSION}-base
docker push vllm/vllm-openai-rocm:latest-base
docker push vllm/vllm-openai-rocm:v${RELEASE_VERSION}-base
docker tag public.ecr.aws/q9t5s3a7/vllm-release-repo:${BUILDKITE_COMMIT}-rocm vllm/vllm-openai-rocm:${BUILDKITE_COMMIT}
docker tag vllm/vllm-openai-rocm:${BUILDKITE_COMMIT} vllm/vllm-openai-rocm:latest
docker tag vllm/vllm-openai-rocm:${BUILDKITE_COMMIT} vllm/vllm-openai-rocm:v${RELEASE_VERSION}
docker push vllm/vllm-openai-rocm:latest
docker push vllm/vllm-openai-rocm:v${RELEASE_VERSION}
${DOCKER_CMDS}
\`\`\`
EOF
.buildkite/scripts/cache-rocm-base-wheels.sh
+28 -34
View File
@@ -5,39 +5,33 @@
# Cache helper for ROCm base wheels
#
# This script manages caching of pre-built ROCm base wheels (torch, triton, etc.)
# to avoid rebuilding them when Dockerfile.rocm_base hasn't changed.
# to avoid rebuilding them when the stack's base Dockerfile hasn't changed.
#
# Usage:
# cache-rocm-base-wheels.sh check - Check if cache exists, outputs "hit" or "miss"
# cache-rocm-base-wheels.sh upload - Upload wheels to cache
# cache-rocm-base-wheels.sh download - Download wheels from cache
# cache-rocm-base-wheels.sh key - Output the cache key
# Usage: cache-rocm-base-wheels.sh <base-dockerfile> <command>
# base-dockerfile e.g. docker/Dockerfile.rocm_base or docker/Dockerfile.rocm_72_base
# Commands:
# check - Check if cache exists, outputs "hit" or "miss"
# upload - Upload wheels to cache
# download - Download wheels from cache
# key - Output the cache key
# path - Output the full S3 cache path
#
# Environment variables:
# S3_BUCKET - S3 bucket name (default: vllm-wheels)
#
# Note: ROCm version is determined by BASE_IMAGE in Dockerfile.rocm_base,
# so changes to ROCm version are captured by the Dockerfile hash.
# The cache key hashes the base Dockerfile and the repo files it COPYs in, which
# capture the ROCm version and every component pin (see rocm/stack.sh).
set -euo pipefail
BUCKET="${S3_BUCKET:-vllm-wheels}"
DOCKERFILE="docker/Dockerfile.rocm_base"
CACHE_PREFIX="rocm/cache"
# Generate hash from Dockerfile content + build args
generate_cache_key() {
# Include Dockerfile content
if [[ ! -f "$DOCKERFILE" ]]; then
echo "ERROR: Dockerfile not found: $DOCKERFILE" >&2
exit 1
fi
local dockerfile_hash=$(sha256sum "$DOCKERFILE" | cut -c1-16)
echo "${dockerfile_hash}"
}
CACHE_KEY=$(generate_cache_key)
# shellcheck source=.buildkite/scripts/rocm/stack.sh
source "$(dirname "${BASH_SOURCE[0]}")/rocm/stack.sh" "${1:?Usage: $0 <base-dockerfile> <command>}"
shift
CACHE_KEY="$ROCM_STACK_WHEEL_KEY"
WHEELS_DIR="$ROCM_STACK_BASE_WHEELS_DIR"
CACHE_PATH="s3://${BUCKET}/${CACHE_PREFIX}/${CACHE_KEY}/"
case "${1:-}" in
@@ -67,19 +61,19 @@ case "${1:-}" in
echo "Cache path: ${CACHE_PATH}"
echo ""
if [[ ! -d "artifacts/rocm-base-wheels" ]]; then
echo "ERROR: artifacts/rocm-base-wheels directory not found" >&2
if [[ ! -d "${WHEELS_DIR}" ]]; then
echo "ERROR: ${WHEELS_DIR} directory not found" >&2
exit 1
fi
WHEEL_COUNT=$(find artifacts/rocm-base-wheels -maxdepth 1 -name '*.whl' 2>/dev/null | wc -l)
WHEEL_COUNT=$(find "${WHEELS_DIR}" -maxdepth 1 -name '*.whl' 2>/dev/null | wc -l)
if [[ "$WHEEL_COUNT" -eq 0 ]]; then
echo "ERROR: No wheels found in artifacts/rocm-base-wheels/" >&2
echo "ERROR: No wheels found in ${WHEELS_DIR}/" >&2
exit 1
fi
echo "Uploading $WHEEL_COUNT wheels..."
aws s3 cp --recursive artifacts/rocm-base-wheels/ "${CACHE_PATH}"
aws s3 cp --recursive "${WHEELS_DIR}/" "${CACHE_PATH}"
echo ""
echo "Cache upload complete!"
@@ -93,17 +87,17 @@ case "${1:-}" in
echo "Cache key: ${CACHE_KEY}"
echo "Cache path: ${CACHE_PATH}"
echo ""
mkdir -p artifacts/rocm-base-wheels
mkdir -p "${WHEELS_DIR}"
# Use sync with include/exclude to only download .whl files
aws s3 sync "${CACHE_PATH}" artifacts/rocm-base-wheels/ \
aws s3 sync "${CACHE_PATH}" "${WHEELS_DIR}/" \
--exclude "*" \
--include "*.whl"
echo ""
echo "Downloaded wheels:"
find artifacts/rocm-base-wheels -maxdepth 1 -name '*.whl' -exec ls -lh {} \;
WHEEL_COUNT=$(find artifacts/rocm-base-wheels -maxdepth 1 -name '*.whl' 2>/dev/null | wc -l)
find "${WHEELS_DIR}" -maxdepth 1 -name '*.whl' -exec ls -lh {} \;
WHEEL_COUNT=$(find "${WHEELS_DIR}" -maxdepth 1 -name '*.whl' 2>/dev/null | wc -l)
echo ""
echo "Total: $WHEEL_COUNT wheels"
echo "========================================"
@@ -118,12 +112,12 @@ case "${1:-}" in
;;
*)
echo "Usage: $0 {check|upload|download|key|path}" >&2
echo "Usage: $0 <base-dockerfile> {check|upload|download|key|path}" >&2
echo "" >&2
echo "Commands:" >&2
echo " check - Check if cache exists, outputs 'hit' or 'miss'" >&2
echo " upload - Upload wheels from artifacts/rocm-base-wheels/ to cache" >&2
echo " download - Download wheels from cache to artifacts/rocm-base-wheels/" >&2
echo " upload - Upload wheels from ${WHEELS_DIR}/ to cache" >&2
echo " download - Download wheels from cache to ${WHEELS_DIR}/" >&2
echo " key - Output the cache key" >&2
echo " path - Output the full S3 cache path" >&2
exit 1
.buildkite/scripts/ci-bake-rocm.sh
+7 -54
View File
@@ -17,7 +17,7 @@ DEFAULT_REPO_SLUG="vllm-project/vllm"
DEFAULT_CI_HCL_SOURCE="docker/ci-rocm.hcl"
DEFAULT_CI_BASE_CONTENT_FILES=".dockerignore requirements/common.txt requirements/rocm.txt requirements/test/rocm.txt tools/install_torchcodec_rocm.sh rust-toolchain.toml tests/vllm_test_utils"
DEFAULT_CI_BASE_DOCKERFILE="docker/Dockerfile.rocm"
DEFAULT_CI_BASE_DOCKERFILE_STAGES="base rust_toolchain_input_0 rust-toolchain-input rust-toolchain build_nixl lmcache_source build_lmcache build_rocshmem build_deepep mori_base ci_base"
DEFAULT_CI_BASE_DOCKERFILE_STAGES="base rust_toolchain_input_0 rust-toolchain-input rust-toolchain build_nixl lmcache_source build_lmcache build_deepep mori_base ci_base"
DEFAULT_CI_BASE_METADATA_VERSION="3"
# ROCm CI forces REMOTE_VLLM=0, so content identity covers only the selected
# local-source stages rather than unreachable remote-fetch alternatives.
@@ -555,9 +555,6 @@ configure_custom_rocm_stages() {
CI_BASE_DOCKERFILE_STAGES="${CI_BASE_DOCKERFILE_STAGES:-${stages}}"
ROCM_CSRC_DOCKERFILE_STAGES="${ROCM_CSRC_DOCKERFILE_STAGES:-${stages}}"
ROCM_RUST_DOCKERFILE_STAGES="${ROCM_RUST_DOCKERFILE_STAGES:-${stages}}"
if [[ " ${stages} " != *" build_rocshmem "* ]]; then
unset ROCSHMEM_BRANCH ROCSHMEM_CACHE_KEY DEEPEP_CACHE_KEY
fi
}
compose_dependency_cache_key() {
@@ -1724,14 +1721,11 @@ ci_base_metadata_pairs() {
metadata_pair "vllm.rocm.nixl_commit" "${NIXL_BRANCH:-$(resolve_dockerfile_arg_value "${dockerfile}" "NIXL_BRANCH")}"
metadata_pair "vllm.rocm.ucx_repo" "$(resolve_dockerfile_arg_value "${dockerfile}" "UCX_REPO")"
metadata_pair "vllm.rocm.ucx_commit" "${UCX_BRANCH:-$(resolve_dockerfile_arg_value "${dockerfile}" "UCX_BRANCH")}"
metadata_pair "vllm.rocm.rocshmem_repo" "$(resolve_dockerfile_arg_value "${dockerfile}" "ROCSHMEM_REPO")"
metadata_pair "vllm.rocm.rocshmem_commit" "${ROCSHMEM_BRANCH:-$(resolve_dockerfile_arg_value "${dockerfile}" "ROCSHMEM_BRANCH")}"
metadata_pair "vllm.rocm.deepep_repo" "$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_REPO")"
metadata_pair "vllm.rocm.deepep_commit" "${DEEPEP_BRANCH:-$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_BRANCH")}"
metadata_pair "vllm.rocm.deepep_nic" "$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_NIC")"
metadata_pair "vllm.rocm.deepep_rocm_arch" "$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_ROCM_ARCH")"
metadata_pair "vllm.rocm.nixl_cache_key" "${NIXL_CACHE_KEY:-}"
metadata_pair "vllm.rocm.rocshmem_cache_key" "${ROCSHMEM_CACHE_KEY:-}"
metadata_pair "vllm.rocm.deepep_cache_key" "${DEEPEP_CACHE_KEY:-}"
}
@@ -2314,7 +2308,7 @@ extract_dependency_pins() {
return 0
fi
for var in NIXL_BRANCH UCX_BRANCH ROCSHMEM_BRANCH DEEPEP_BRANCH; do
for var in NIXL_BRANCH UCX_BRANCH DEEPEP_BRANCH; do
if [[ -n "${!var:-}" ]]; then
echo "Using provided ${var}: ${!var}"
continue
@@ -2336,17 +2330,14 @@ compute_dependency_cache_keys() {
local dockerfile_rocm=""
local nixl_branch=""
local ucx_branch=""
local rocshmem_branch=""
local deepep_branch=""
local nixl_material=""
local rocshmem_material=""
local deepep_material=""
bake_dir=$(dirname "${VLLM_BAKE_FILE}")
dockerfile_rocm="${CI_BASE_DOCKERFILE:-${bake_dir}/Dockerfile.rocm}"
nixl_branch=$(resolve_dockerfile_arg_value "${dockerfile_rocm}" "NIXL_BRANCH")
ucx_branch=$(resolve_dockerfile_arg_value "${dockerfile_rocm}" "UCX_BRANCH")
rocshmem_branch=$(resolve_dockerfile_arg_value "${dockerfile_rocm}" "ROCSHMEM_BRANCH")
deepep_branch=$(resolve_dockerfile_arg_value "${dockerfile_rocm}" "DEEPEP_BRANCH")
if [[ -n "${nixl_branch}" && -n "${ucx_branch}" ]]; then
@@ -2360,22 +2351,11 @@ compute_dependency_cache_keys() {
echo "NIXL dependency cache key: ${NIXL_CACHE_KEY}"
fi
if [[ -n "${rocshmem_branch}" ]]; then
rocshmem_material=$(compose_stage_cache_material "${dockerfile_rocm}" "base build_rocshmem")
ROCSHMEM_CACHE_KEY=$(
compose_dependency_cache_key \
"${rocshmem_branch}" \
"${rocshmem_material}"
)
export ROCSHMEM_CACHE_KEY
echo "ROCShmem dependency cache key: ${ROCSHMEM_CACHE_KEY}"
fi
if [[ -n "${deepep_branch}" && -n "${rocshmem_branch}" ]]; then
deepep_material=$(compose_stage_cache_material "${dockerfile_rocm}" "base build_rocshmem build_deepep")
if [[ -n "${deepep_branch}" ]]; then
deepep_material=$(compose_stage_cache_material "${dockerfile_rocm}" "base build_deepep")
DEEPEP_CACHE_KEY=$(
compose_dependency_cache_key \
"${deepep_branch}-rocshmem-${rocshmem_branch}" \
"${deepep_branch}" \
"${deepep_material}"
)
export DEEPEP_CACHE_KEY
@@ -2415,18 +2395,11 @@ dependency_cache_ref_for_target() {
printf '%s\n' "${cache_repo}:nixl-rocm-${NIXL_BRANCH}-ucx-${UCX_BRANCH:-}"
fi
;;
rocshmem-rocm-ci)
if [[ -n "${ROCSHMEM_CACHE_KEY:-}" ]]; then
printf '%s\n' "${cache_repo}:rocshmem-rocm-${ROCSHMEM_CACHE_KEY}"
elif [[ -n "${ROCSHMEM_BRANCH:-}" ]]; then
printf '%s\n' "${cache_repo}:rocshmem-rocm-${ROCSHMEM_BRANCH}"
fi
;;
deepep-rocm-ci)
if [[ -n "${DEEPEP_CACHE_KEY:-}" ]]; then
printf '%s\n' "${cache_repo}:deepep-rocm-${DEEPEP_CACHE_KEY}"
elif [[ -n "${DEEPEP_BRANCH:-}" ]]; then
printf '%s\n' "${cache_repo}:deepep-rocm-${DEEPEP_BRANCH}-rocshmem-${ROCSHMEM_BRANCH:-}"
printf '%s\n' "${cache_repo}:deepep-rocm-${DEEPEP_BRANCH}"
fi
;;
esac
@@ -2444,7 +2417,6 @@ add_dependency_cache_target() {
resolve_ci_base_dependency_targets() {
local mode="${ROCM_DEP_CACHE_EXPORT_MODE:-missing}"
local nixl_ref=""
local rocshmem_ref=""
local deepep_ref=""
[[ "${TARGET}" == "ci-base-rocm-ci-with-deps" ]] || return 0
@@ -2452,7 +2424,7 @@ resolve_ci_base_dependency_targets() {
case "${mode}" in
always)
echo "ROCM_DEP_CACHE_EXPORT_MODE=always; exporting all dependency caches serially"
for target in nixl-rocm-ci rocshmem-rocm-ci deepep-rocm-ci; do
for target in nixl-rocm-ci deepep-rocm-ci; do
if [[ -n "$(dependency_cache_ref_for_target "${target}")" ]]; then
add_dependency_cache_target "${target}"
fi
@@ -2482,16 +2454,6 @@ resolve_ci_base_dependency_targets() {
fi
fi
if [[ "${mode}" != "always" && -n "${ROCSHMEM_CACHE_KEY:-}" ]]; then
rocshmem_ref=$(dependency_cache_ref_for_target "rocshmem-rocm-ci")
if dependency_cache_ref_exists "${rocshmem_ref}"; then
echo "ROCShmem dependency cache exists: ${rocshmem_ref}"
else
echo "ROCShmem dependency cache missing; will seed: ${rocshmem_ref}"
add_dependency_cache_target "rocshmem-rocm-ci"
fi
fi
if [[ "${mode}" != "always" && -n "${DEEPEP_CACHE_KEY:-}" ]]; then
deepep_ref=$(dependency_cache_ref_for_target "deepep-rocm-ci")
if dependency_cache_ref_exists "${deepep_ref}"; then
@@ -2502,15 +2464,6 @@ resolve_ci_base_dependency_targets() {
fi
fi
# DeepEP inherits from ROCShmem. If ROCShmem is being seeded, seed DeepEP too
# so the pair stays consistent for future ci_base rebuilds.
if printf '%s\n' "${DEPENDENCY_CACHE_TARGETS[@]}" | grep -qx "rocshmem-rocm-ci" \
&& ! printf '%s\n' "${DEPENDENCY_CACHE_TARGETS[@]}" | grep -qx "deepep-rocm-ci" \
&& [[ -n "${DEEPEP_BRANCH:-}" ]]; then
echo "ROCShmem cache is missing; also seeding DeepEP cache"
add_dependency_cache_target "deepep-rocm-ci"
fi
BAKE_TARGETS=("ci-base-rocm-ci")
if [[ ${#DEPENDENCY_CACHE_TARGETS[@]} -eq 0 ]]; then
echo "All dependency caches exist; building ci_base without dependency cache exports"
.buildkite/scripts/cleanup-nightly-builds.sh
+5 -2
View File
@@ -5,14 +5,17 @@ set -ex
# Clean up old nightly builds from DockerHub, keeping only the last 14 builds
# This script uses DockerHub API to list and delete old tags with specified prefix
# Tags starting with "nightly-dev" are always excluded from deletion
# Usage: cleanup-nightly-builds.sh [TAG_PREFIX] [REPO]
# Usage: cleanup-nightly-builds.sh [TAG_PREFIX] [REPO] [EXCLUDE_PREFIX]
# Example: cleanup-nightly-builds.sh "nightly-"
# Example: cleanup-nightly-builds.sh "cu130-nightly-"
# Example: cleanup-nightly-builds.sh "nightly-" "vllm/vllm-openai-rocm"
# Example: cleanup-nightly-builds.sh "nightly-" "vllm/vllm-openai-rocm" "nightly-rocm"
# (EXCLUDE_PREFIX skips tags owned by another variant sharing TAG_PREFIX)
# Get tag prefix and repo from arguments
TAG_PREFIX="${1:-nightly-}"
REPO="${2:-vllm/vllm-openai}"
EXCLUDE_PREFIX="${3:-}"
echo "Cleaning up tags with prefix: $TAG_PREFIX in repository: $REPO"
@@ -57,7 +60,7 @@ get_all_tags() {
# Get both last_updated timestamp and tag name, separated by |
# Exclude mutable architecture tags and nightly-dev tags from cleanup
local tags=$(echo "$response" | jq -r --arg prefix "$TAG_PREFIX" '.results[] | select(.name | startswith($prefix)) | select(.name != ($prefix + "x86_64") and .name != ($prefix + "aarch64")) | select(.name | startswith("nightly-dev") | not) | "\(.last_updated)|\(.name)"')
local tags=$(echo "$response" | jq -r --arg prefix "$TAG_PREFIX" --arg exclude "$EXCLUDE_PREFIX" '.results[] | select(.name | startswith($prefix)) | select($exclude == "" or (.name | startswith($exclude) | not)) | select(.name != ($prefix + "x86_64") and .name != ($prefix + "aarch64")) | select(.name | startswith("nightly-dev") | not) | "\(.last_updated)|\(.name)"')
if [ -z "$tags" ]; then
break
.buildkite/scripts/generate-nightly-index.py
+63 -9
View File
@@ -12,7 +12,7 @@ from dataclasses import asdict, dataclass
from datetime import datetime
from pathlib import Path
from typing import Any
from urllib.parse import quote
from urllib.parse import quote, unquote, urlparse
import regex as re
@@ -51,6 +51,8 @@ class WheelFileInfo:
platform_tag: str
variant: str | None
filename: str
# Absolute URL for files hosted elsewhere (see --external-links)
url: str | None = None
def parse_from_filename(file: str) -> WheelFileInfo:
@@ -110,6 +112,29 @@ def parse_from_filename(file: str) -> WheelFileInfo:
)
def parse_external_url(url: str) -> WheelFileInfo:
"""Parse an absolute wheel or sdist URL hosted on another index."""
filename = unquote(urlparse(url).path.rsplit("/", 1)[-1])
if filename.endswith(".whl"):
info = parse_from_filename(filename)
else:
match = re.match(r"^(?P<name>.+)-(?P<version>[^-]+)\.(tar\.gz|zip)$", filename)
if not match:
raise ValueError(f"Unsupported external file: {filename}")
info = WheelFileInfo(
package_name=match.group("name"),
version=match.group("version"),
build_tag=None,
python_tag="source",
abi_tag="none",
platform_tag="any",
variant=None,
filename=filename,
)
info.url = url
return info
def generate_project_list(subdir_names: list[str], comment: str = "") -> str:
"""Generate project list HTML content linking to each project & variant subdirectory."""
href_tags = []
@@ -129,14 +154,19 @@ def generate_package_index_and_metadata(
href_tags = []
metadata = []
for file in sorted(wheel_files, key=lambda x: x.filename):
relative_path = (
wheel_base_dir.relative_to(index_base_dir, walk_up=True) / file.filename
)
# handle with '+' in URL, and avoid double-encoding '/' and already-encoded '%2B'
# NOTE: this is AWS S3 specific behavior!
file_path_quoted = quote(relative_path.as_posix(), safe=":%/")
if file.url:
file_path_quoted = file.url
else:
relative_path = (
wheel_base_dir.relative_to(index_base_dir, walk_up=True) / file.filename
)
# handle with '+' in URL, and avoid double-encoding '/' and already-encoded '%2B'
# NOTE: this is AWS S3 specific behavior!
file_path_quoted = quote(relative_path.as_posix(), safe=":%/")
href_tags.append(f' <a href="{file_path_quoted}">{file.filename}</a><br/>')
file_meta = asdict(file)
if file_meta["url"] is None:
del file_meta["url"]
file_meta["path"] = file_path_quoted
metadata.append(file_meta)
index_str = INDEX_HTML_TEMPLATE.format(items="\n".join(href_tags), comment=comment)
@@ -151,6 +181,7 @@ def generate_index_and_metadata(
default_variant: str | None = None,
alias_to_default: str | None = None,
comment: str = "",
external_urls: list[str] | None = None,
):
"""Generate index for all wheel files.
@@ -160,6 +191,8 @@ def generate_index_and_metadata(
index_base_dir (Path): Base directory to store index files.
default_variant (str | None): The default variant name, if any.
alias_to_default (str | None): Alias variant name for the default variant, if any.
external_urls (list[str] | None): Absolute URLs of files hosted on another index,
listed in the generated indices without being copied.
comment (str | None): Optional comment to include in the generated HTML files.
First, parse all wheel files to extract metadata.
@@ -214,6 +247,7 @@ def generate_index_and_metadata(
"""
parsed_files = [parse_from_filename(f) for f in whl_files]
parsed_files += [parse_external_url(u) for u in external_urls or []]
if not parsed_files:
print("No wheel files found, skipping index generation.")
@@ -232,10 +266,14 @@ def generate_index_and_metadata(
print(f"Detected ROCm variant from vllm: {rocm_variant}")
break
# Apply ROCm variant to all wheels without a variant
# Apply ROCm variant to all wheels without a variant. Dependencies with their
# own ROCm local version (e.g. TheRock's torch-2.12.0+rocm10.0.0) belong to
# the same variant rather than one of their own.
if rocm_variant:
for file in parsed_files:
if file.variant is None:
if file.variant is None or (
file.variant.startswith("rocm") and file.variant != rocm_variant
):
file.variant = rocm_variant
print(f"Inherited variant '{rocm_variant}' for {file.filename}")
@@ -366,6 +404,13 @@ if __name__ == "__main__":
default=None,
help="Alias variant name for the default variant",
)
parser.add_argument(
"--external-links",
type=str,
default=None,
help="Optional file with one absolute URL per line for packages hosted on "
"another index (listed alongside, but not copied into, this index)",
)
parser.add_argument(
"--comment",
type=str,
@@ -463,5 +508,14 @@ if __name__ == "__main__":
default_variant=None,
alias_to_default=args.alias_to_default,
comment=args.comment.strip(),
external_urls=(
[
line.strip()
for line in Path(args.external_links).read_text().splitlines()
if line.strip() and not line.startswith("#")
]
if args.external_links
else None
),
)
print(f"Successfully generated index and metadata in {output_dir}")
.buildkite/scripts/publish-release-images.sh
+28 -16
View File
@@ -11,9 +11,9 @@ set -euo pipefail
TARGET="${1:-all}"
case "${TARGET}" in
cuda-13-0 | cuda-12-9 | cuda-13-0-ubuntu-24-04 | \
cuda-12-9-ubuntu-24-04 | rocm | xpu | cpu | all) ;;
cuda-12-9-ubuntu-24-04 | rocm | rocm72 | xpu | cpu | all) ;;
*)
echo "Usage: $0 {cuda-13-0|cuda-12-9|cuda-13-0-ubuntu-24-04|cuda-12-9-ubuntu-24-04|rocm|xpu|cpu|all}"
echo "Usage: $0 {cuda-13-0|cuda-12-9|cuda-13-0-ubuntu-24-04|cuda-12-9-ubuntu-24-04|rocm|rocm72|xpu|cpu|all}"
exit 2
;;
esac
@@ -143,23 +143,35 @@ if target_enabled cuda-12-9-ubuntu-24-04; then
fi
# ---- ROCm ----
# Each ROCm stack is identified by its base Dockerfile (see rocm/stack.sh). The
# default stack also owns the plain :latest / :v<ver> tags.
publish_rocm_stack() {
local base_dockerfile="$1" default="$2" suffix
(
# shellcheck source=.buildkite/scripts/rocm/stack.sh
source .buildkite/scripts/rocm/stack.sh "$base_dockerfile"
local suffixes=("-${ROCM_STACK_VARIANT}")
[[ "$default" == "1" ]] && suffixes=("" "-${ROCM_STACK_VARIANT}")
docker pull "$ROCM_STACK_ECR_IMAGE"
docker pull "$ROCM_STACK_ECR_BASE"
for suffix in "${suffixes[@]}"; do
for tag in "latest${suffix}" "v${RELEASE_VERSION}${suffix}"; do
docker tag "$ROCM_STACK_ECR_IMAGE" "vllm/vllm-openai-rocm:${tag}"
docker tag "$ROCM_STACK_ECR_BASE" "vllm/vllm-openai-rocm:${tag}-base"
docker push "vllm/vllm-openai-rocm:${tag}"
docker push "vllm/vllm-openai-rocm:${tag}-base"
done
done
)
}
if target_enabled rocm; then
ROCM_BASE_CACHE_KEY=$(.buildkite/scripts/cache-rocm-base-wheels.sh key)
echo "ROCm base cache key: ${ROCM_BASE_CACHE_KEY}"
publish_rocm_stack docker/Dockerfile.rocm_base 1
fi
docker pull "public.ecr.aws/q9t5s3a7/vllm-release-repo:${COMMIT}-rocm"
docker pull "public.ecr.aws/q9t5s3a7/vllm-release-repo:${ROCM_BASE_CACHE_KEY}-rocm-base"
docker tag "public.ecr.aws/q9t5s3a7/vllm-release-repo:${COMMIT}-rocm" vllm/vllm-openai-rocm:latest
docker tag "public.ecr.aws/q9t5s3a7/vllm-release-repo:${COMMIT}-rocm" "vllm/vllm-openai-rocm:v${RELEASE_VERSION}"
docker push vllm/vllm-openai-rocm:latest
docker push "vllm/vllm-openai-rocm:v${RELEASE_VERSION}"
docker tag "public.ecr.aws/q9t5s3a7/vllm-release-repo:${ROCM_BASE_CACHE_KEY}-rocm-base" vllm/vllm-openai-rocm:latest-base
docker tag "public.ecr.aws/q9t5s3a7/vllm-release-repo:${ROCM_BASE_CACHE_KEY}-rocm-base" "vllm/vllm-openai-rocm:v${RELEASE_VERSION}-base"
docker push vllm/vllm-openai-rocm:latest-base
docker push "vllm/vllm-openai-rocm:v${RELEASE_VERSION}-base"
if target_enabled rocm72; then
publish_rocm_stack docker/Dockerfile.rocm_72_base 0
fi
# ---- XPU ----
.buildkite/scripts/push-nightly-builds-rocm.sh
+42 -65
View File
@@ -2,42 +2,36 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
#
# Push a ROCm-family nightly base image and nightly image from ECR to Docker Hub
# under vllm/vllm-openai-rocm, as base-nightly[-<variant>], nightly[-<variant>]
# and their -<commit> forms.
# Run when NIGHTLY=1 after the matching build-*-release-image step has pushed to ECR.
# Push a ROCm stack's nightly base image and nightly image from ECR to Docker Hub
# under vllm/vllm-openai-rocm, as base-nightly-<variant>, nightly-<variant> and their
# -<commit> forms, then prune old nightly tags. With --default, also push the plain
# base-nightly and nightly tags for the same images.
# Run when NIGHTLY=1 after the stack's release image has been pushed to ECR.
#
# Usage: push-nightly-builds-rocm.sh [ECR_TAG_SUFFIX TAG_VARIANT]
# Pass both arguments or neither. With none, the default ROCm apt build is
# published as :nightly. The TheRock build passes "rock" (the source ECR tag
# suffix) and "rocm714" (the Docker Hub tag flavor), giving :nightly-rocm714.
# Usage: push-nightly-builds-rocm.sh <base-dockerfile> [--default]
# e.g. "docker/Dockerfile.rocm_base --default" (ROCm 10.0, also :nightly)
# and "docker/Dockerfile.rocm_72_base" (ROCm 7.2, :nightly-rocm72).
#
# Local testing (no push to Docker Hub):
# BASE_ECR_IMAGE=<full-base-image-reference> \
# BUILDKITE_COMMIT=<commit-with-image-in-ecr> DRY_RUN=1 \
# bash .buildkite/scripts/push-nightly-builds-rocm.sh
# BUILDKITE_COMMIT=<commit-with-image-in-ecr> DRY_RUN=1 \
# bash .buildkite/scripts/push-nightly-builds-rocm.sh docker/Dockerfile.rocm_base --default
# Requires: AWS CLI configured (for ECR public login), Docker. For full run: Docker Hub login.
set -euxo pipefail
usage() {
echo "Usage: $0 [ECR_TAG_SUFFIX TAG_VARIANT]" >&2
echo "Usage: $0 <base-dockerfile> [--default]" >&2
}
DOCKERHUB_REPO="vllm/vllm-openai-rocm"
PUSH_DEFAULT=0
case "$#" in
0)
ECR_TAG_SUFFIX="rocm"
TAG_VARIANT=""
;;
1) ;;
2)
if [[ -z "$1" || -z "$2" ]]; then
if [[ "$2" != "--default" ]]; then
usage
exit 2
fi
ECR_TAG_SUFFIX="$1"
TAG_VARIANT="$2"
PUSH_DEFAULT=1
;;
*)
usage
@@ -49,61 +43,44 @@ esac
BUILDKITE_COMMIT="${BUILDKITE_COMMIT:?Set BUILDKITE_COMMIT to the commit SHA that has the image in ECR (e.g. from a previous release pipeline run)}"
DRY_RUN="${DRY_RUN:-0}"
ECR_REPO="public.ecr.aws/q9t5s3a7/vllm-release-repo"
# shellcheck source=.buildkite/scripts/rocm/stack.sh
source "$(dirname "${BASH_SOURCE[0]}")/rocm/stack.sh" "$1"
DOCKERHUB_REPO="vllm/vllm-openai-rocm"
VARIANT="$ROCM_STACK_VARIANT"
# Get the base image ECR tag (set by the build-*-release-image pipeline step)
BASE_METADATA_KEY="${ECR_TAG_SUFFIX}-base-ecr-tag"
if [[ -n "${BASE_ECR_IMAGE:-}" ]]; then
if [[ "$DRY_RUN" != "1" ]]; then
echo "ERROR: BASE_ECR_IMAGE may only be used with DRY_RUN=1" >&2
exit 1
fi
BASE_ORIG_TAG="$BASE_ECR_IMAGE"
elif ! BASE_ORIG_TAG="$(buildkite-agent meta-data get "$BASE_METADATA_KEY")"; then
echo "ERROR: Failed to read required Buildkite metadata '$BASE_METADATA_KEY'" >&2
exit 1
fi
if [[ -z "$BASE_ORIG_TAG" ]]; then
echo "ERROR: Required Buildkite metadata '$BASE_METADATA_KEY' is empty" >&2
echo "Set BASE_ECR_IMAGE to the full ECR base image reference for local testing" >&2
exit 1
BASE_TAGS=("base-nightly-${VARIANT}" "base-nightly-${VARIANT}-${BUILDKITE_COMMIT}")
TAGS=("nightly-${VARIANT}" "nightly-${VARIANT}-${BUILDKITE_COMMIT}")
if [[ "$PUSH_DEFAULT" == "1" ]]; then
BASE_TAGS+=("base-nightly" "base-nightly-${BUILDKITE_COMMIT}")
TAGS+=("nightly" "nightly-${BUILDKITE_COMMIT}")
fi
ORIG_TAG="${BUILDKITE_COMMIT}-${ECR_TAG_SUFFIX}"
VARIANT_SUFFIX=""
[[ -n "$TAG_VARIANT" ]] && VARIANT_SUFFIX="-$TAG_VARIANT"
BASE_TAG_NAME="base-nightly${VARIANT_SUFFIX}"
TAG_NAME="nightly${VARIANT_SUFFIX}"
BASE_TAG_NAME_COMMIT="base-nightly${VARIANT_SUFFIX}-${BUILDKITE_COMMIT}"
TAG_NAME_COMMIT="nightly${VARIANT_SUFFIX}-${BUILDKITE_COMMIT}"
echo "Pushing base image $ROCM_STACK_ECR_BASE as: ${BASE_TAGS[*]}"
echo "Pushing release image $ROCM_STACK_ECR_IMAGE as: ${TAGS[*]}"
echo "Pushing base image from ECR: $BASE_ORIG_TAG"
echo "Pushing release image from ECR tag: $ORIG_TAG to Docker Hub as $DOCKERHUB_REPO:$TAG_NAME and $DOCKERHUB_REPO:$TAG_NAME_COMMIT"
[[ "$DRY_RUN" == "1" ]] && echo "[DRY_RUN] Skipping push to Docker Hub"
# Login to ECR and pull the images built by build-*-release-image
# Login to ECR and pull the stack's images
aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
docker pull "$BASE_ORIG_TAG"
docker pull "${ECR_REPO}:${ORIG_TAG}"
docker pull "$ROCM_STACK_ECR_BASE"
docker pull "$ROCM_STACK_ECR_IMAGE"
# Tag for Docker Hub (base-nightly and base-nightly-<commit>, nightly and nightly-<commit>)
docker tag "$BASE_ORIG_TAG" "$DOCKERHUB_REPO":"$BASE_TAG_NAME"
docker tag "$BASE_ORIG_TAG" "$DOCKERHUB_REPO":"$BASE_TAG_NAME_COMMIT"
docker tag "${ECR_REPO}:${ORIG_TAG}" "$DOCKERHUB_REPO":"$TAG_NAME"
docker tag "${ECR_REPO}:${ORIG_TAG}" "$DOCKERHUB_REPO":"$TAG_NAME_COMMIT"
for tag in "${BASE_TAGS[@]}"; do docker tag "$ROCM_STACK_ECR_BASE" "$DOCKERHUB_REPO:$tag"; done
for tag in "${TAGS[@]}"; do docker tag "$ROCM_STACK_ECR_IMAGE" "$DOCKERHUB_REPO:$tag"; done
if [[ "$DRY_RUN" == "1" ]]; then
echo "[DRY_RUN] Would push $DOCKERHUB_REPO:$BASE_TAG_NAME and $DOCKERHUB_REPO:$BASE_TAG_NAME_COMMIT"
echo "[DRY_RUN] Would push $DOCKERHUB_REPO:$TAG_NAME and $DOCKERHUB_REPO:$TAG_NAME_COMMIT"
echo "[DRY_RUN] Local tags created. Exiting without push."
exit 0
fi
# Push to Docker Hub (docker-login plugin runs before this step in CI)
docker push "$DOCKERHUB_REPO":"$BASE_TAG_NAME"
docker push "$DOCKERHUB_REPO":"$BASE_TAG_NAME_COMMIT"
docker push "$DOCKERHUB_REPO":"$TAG_NAME"
docker push "$DOCKERHUB_REPO":"$TAG_NAME_COMMIT"
for tag in "${BASE_TAGS[@]}" "${TAGS[@]}"; do docker push "$DOCKERHUB_REPO:$tag"; done
echo "Pushed $DOCKERHUB_REPO: ${BASE_TAGS[*]} ${TAGS[*]}"
echo "Pushed $DOCKERHUB_REPO:$BASE_TAG_NAME and $DOCKERHUB_REPO:$BASE_TAG_NAME_COMMIT"
echo "Pushed $DOCKERHUB_REPO:$TAG_NAME and $DOCKERHUB_REPO:$TAG_NAME_COMMIT"
# Keep only the last 14 builds of each tag family. The plain nightly- families
# exclude the variant ones sharing their prefix; commit SHAs never start with "rocm".
CLEANUP=.buildkite/scripts/cleanup-nightly-builds.sh
bash "$CLEANUP" "nightly-${VARIANT}-" "$DOCKERHUB_REPO"
bash "$CLEANUP" "base-nightly-${VARIANT}-" "$DOCKERHUB_REPO"
if [[ "$PUSH_DEFAULT" == "1" ]]; then
bash "$CLEANUP" "nightly-" "$DOCKERHUB_REPO" "nightly-rocm"
bash "$CLEANUP" "base-nightly-" "$DOCKERHUB_REPO" "base-nightly-rocm"
fi
.buildkite/scripts/rocm/build-config.sh
+2 -2
View File
@@ -1,8 +1,8 @@
#!/usr/bin/env bash
# Shared Dockerfile selection for the normal AMD CI build chain.
# CI_ROCM_DOCKERFILE_BASE and CI_ROCM_DOCKERFILE accept repository-relative paths.
# For Rock, set CI_ROCM_DOCKERFILE_BASE=docker/Dockerfile.rock_base and
# CI_ROCM_DOCKERFILE=docker/Dockerfile.rock.
# For the legacy ROCm 7.2 stack, set CI_ROCM_DOCKERFILE_BASE=docker/Dockerfile.rocm_72_base
# and CI_ROCM_DOCKERFILE=docker/Dockerfile.rocm_72.
# Custom recipes must preserve the CI targets and artifact layout. Additional
# COPY inputs also need the corresponding *_CONTENT_FILES overrides.
.buildkite/scripts/rocm/stack.sh
+78
View File
@@ -0,0 +1,78 @@
#!/usr/bin/env bash
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
#
# Derive everything the release pipeline needs about a ROCm stack from its base
# Dockerfile, so the stack's image is the single input to every ROCm script.
#
# Usage: source .buildkite/scripts/rocm/stack.sh <base-dockerfile>
# e.g. docker/Dockerfile.rocm_base (ROCm 10.0) or docker/Dockerfile.rocm_72_base
#
# Exports:
# ROCM_STACK_BASE_DOCKERFILE the base Dockerfile
# ROCM_STACK_DOCKERFILE the matching vLLM Dockerfile (base name minus _base)
# ROCM_STACK_THEROCK 1 if the ROCm SDK comes from TheRock wheels
# ROCM_STACK_VERSION ROCm version, e.g. 10.0.0 or 7.2.3
# ROCM_STACK_VARIANT wheel variant and Docker Hub tag flavor from major.minor,
# same rule as setup.py (rocm100, rocm72)
# ROCM_STACK_IMAGE_KEY hash of the base Dockerfile (base image cache key)
# ROCM_STACK_WHEEL_KEY hash of the base Dockerfile plus files it COPYs from
# the repo (base wheel cache key)
# ROCM_STACK_ECR_BASE ECR tag of the cached base image
# ROCM_STACK_ECR_IMAGE ECR tag of this commit's release image
# ROCM_STACK_BASE_WHEELS_DIR local dir for base wheels (also the artifact path)
# ROCM_STACK_VLLM_WHEEL_DIR local dir for the vLLM wheel (also the artifact path)
rocm_stack_init() {
local base="$1"
if [[ ! -f "$base" ]]; then
echo "ROCm base Dockerfile not found: $base" >&2
return 2
fi
ROCM_STACK_BASE_DOCKERFILE="$base"
ROCM_STACK_DOCKERFILE="${base%_base}"
local sdk
sdk=$(sed -nE 's/^ARG ROCM_SDK_VERSION=([0-9.]+).*/\1/p' "$base" | head -1)
if [[ -n "$sdk" ]]; then
ROCM_STACK_THEROCK=1
ROCM_STACK_VERSION="$sdk"
else
ROCM_STACK_THEROCK=0
# BASE_IMAGE format: rocm/dev-ubuntu-22.04:7.2.3-complete -> 7.2.3
ROCM_STACK_VERSION=$(sed -nE 's/^ARG BASE_IMAGE=.*:([0-9]+(\.[0-9]+)+).*/\1/p' "$base" | head -1)
fi
if [[ -z "$ROCM_STACK_VERSION" ]]; then
echo "Could not determine the ROCm version from $base" >&2
return 2
fi
local major minor
IFS=. read -r major minor _ <<< "$ROCM_STACK_VERSION"
ROCM_STACK_VARIANT="rocm${major}${minor}"
ROCM_STACK_IMAGE_KEY=$(sha256sum "$base" | cut -c1-16)
local inputs
inputs=$(grep -E '^COPY ' "$base" | grep -v -- '--from' | awk '{print $2}' || true)
if [[ -z "$inputs" ]]; then
ROCM_STACK_WHEEL_KEY="$ROCM_STACK_IMAGE_KEY"
else
# shellcheck disable=SC2086
ROCM_STACK_WHEEL_KEY=$(sha256sum "$base" $inputs | sha256sum | cut -c1-16)
fi
local ecr="public.ecr.aws/q9t5s3a7/vllm-release-repo"
ROCM_STACK_ECR_BASE="${ecr}:${ROCM_STACK_IMAGE_KEY}-${ROCM_STACK_VARIANT}-base"
ROCM_STACK_ECR_IMAGE="${ecr}:${BUILDKITE_COMMIT:-unknown}-${ROCM_STACK_VARIANT}"
ROCM_STACK_BASE_WHEELS_DIR="artifacts/${ROCM_STACK_VARIANT}-base-wheels"
ROCM_STACK_VLLM_WHEEL_DIR="artifacts/${ROCM_STACK_VARIANT}-vllm-wheel"
export ROCM_STACK_BASE_DOCKERFILE ROCM_STACK_DOCKERFILE ROCM_STACK_THEROCK \
ROCM_STACK_VERSION ROCM_STACK_VARIANT ROCM_STACK_IMAGE_KEY \
ROCM_STACK_WHEEL_KEY ROCM_STACK_ECR_BASE ROCM_STACK_ECR_IMAGE \
ROCM_STACK_BASE_WHEELS_DIR ROCM_STACK_VLLM_WHEEL_DIR
}
if [[ $# -gt 0 ]]; then
rocm_stack_init "$1"
fi
.buildkite/scripts/upload-rocm-wheels.sh
+66 -20
View File
@@ -4,21 +4,31 @@
#
# Upload ROCm wheels to S3 with proper index generation
#
# Usage: upload-rocm-wheels.sh <base-dockerfile>
# base-dockerfile the ROCm stack, e.g. docker/Dockerfile.rocm_base (see rocm/stack.sh)
#
# Required environment variables:
# AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY (or IAM role)
# S3_BUCKET (default: vllm-wheels)
#
# S3 path structure:
# s3://vllm-wheels/rocm/{commit}/ - All wheels for this commit
# s3://vllm-wheels/rocm/nightly/ - Index pointing to latest nightly
# s3://vllm-wheels/rocm/{version}/ - Index for release versions
# s3://vllm-wheels/rocm/{commit}/<variant>-wheels/ - This stack's wheels for this commit
# s3://vllm-wheels/rocm/{commit}/<variant>/ - Index per ROCm variant (e.g. rocm100)
# s3://vllm-wheels/rocm/nightly/<variant>/ - Index pointing to latest nightly
# s3://vllm-wheels/rocm/{version}/<variant>/ - Index for release versions
# The top-level index.html at each prefix lists every variant present, since
# several ROCm stacks publish into the same prefixes.
set -ex
# ======== Configuration ========
BUCKET="${S3_BUCKET:-vllm-wheels}"
# shellcheck source=.buildkite/scripts/rocm/stack.sh
source "$(dirname "${BASH_SOURCE[0]}")/rocm/stack.sh" "${1:?Usage: $0 <base-dockerfile>}"
ROCM_SUBPATH="rocm/${BUILDKITE_COMMIT}"
S3_COMMIT_PREFIX="s3://$BUCKET/$ROCM_SUBPATH/"
WHEEL_DIRS="$ROCM_STACK_BASE_WHEELS_DIR $ROCM_STACK_VLLM_WHEEL_DIR"
WHEEL_SUBPATH="$ROCM_SUBPATH/${ROCM_STACK_VARIANT}-wheels"
S3_WHEEL_PREFIX="s3://$BUCKET/$WHEEL_SUBPATH/"
INDICES_OUTPUT_DIR="rocm-indices"
echo "========================================"
@@ -49,9 +59,11 @@ source .buildkite/scripts/lib/manylinux.sh
# ======== Part 1: Collect and prepare wheels ========
# Collect all wheels
rm -rf all-rocm-wheels "$INDICES_OUTPUT_DIR"
mkdir -p all-rocm-wheels
cp artifacts/rocm-base-wheels/*.whl all-rocm-wheels/ 2>/dev/null || true
cp artifacts/rocm-vllm-wheel/*.whl all-rocm-wheels/ 2>/dev/null || true
for dir in $WHEEL_DIRS; do
cp "$dir"/*.whl all-rocm-wheels/ 2>/dev/null || true
done
WHEEL_COUNT=$(find all-rocm-wheels -maxdepth 1 -name '*.whl' 2>/dev/null | wc -l)
echo "Total wheels to upload: $WHEEL_COUNT"
@@ -84,9 +96,9 @@ ls -lh all-rocm-wheels/
# ======== Part 2: Upload wheels to S3 ========
echo ""
echo "Uploading wheels to $S3_COMMIT_PREFIX"
echo "Uploading wheels to $S3_WHEEL_PREFIX"
for wheel in all-rocm-wheels/*.whl; do
aws s3 cp "$wheel" "$S3_COMMIT_PREFIX"
aws s3 cp "$wheel" "$S3_WHEEL_PREFIX"
done
# ======== Part 3: Generate and upload indices ========
@@ -95,28 +107,63 @@ done
echo ""
echo "Generating indices..."
obj_json="rocm-objects.json"
aws s3api list-objects-v2 --bucket "$BUCKET" --prefix "$ROCM_SUBPATH/" --delimiter / --output json > "$obj_json"
aws s3api list-objects-v2 --bucket "$BUCKET" --prefix "$WHEEL_SUBPATH/" --delimiter / --output json > "$obj_json"
mkdir -p "$INDICES_OUTPUT_DIR"
# Use the existing generate-nightly-index.py
# HACK: Replace regex module with stdlib re (same as CUDA script)
sed -i 's/import regex as re/import re/g' .buildkite/scripts/generate-nightly-index.py
sed -i 's/import regex as re/import re/g' tools/vllm-rocm/therock_wheels.py
sed -i 's/import pybase64 as base64/import base64/g' tools/vllm-rocm/therock_wheels.py
INDEX_ARGS=(--version "$ROCM_SUBPATH" --wheel-dir "${WHEEL_SUBPATH#rocm/}")
# TheRock's SDK, device kernels and torchvision/torchaudio are linked, not hosted
if [[ "$ROCM_STACK_THEROCK" == "1" ]]; then
$PYTHON tools/vllm-rocm/therock_wheels.py external-links \
--dockerfile "$ROCM_STACK_BASE_DOCKERFILE" \
--torch-wheel "$(ls "$ROCM_STACK_BASE_WHEELS_DIR"/torch-*.whl)" > therock-external-links.txt
cat therock-external-links.txt
INDEX_ARGS+=(--external-links therock-external-links.txt)
fi
$PYTHON .buildkite/scripts/generate-nightly-index.py \
--version "$ROCM_SUBPATH" \
"${INDEX_ARGS[@]}" \
--current-objects "$obj_json" \
--output-dir "$INDICES_OUTPUT_DIR" \
--comment "ROCm commit $BUILDKITE_COMMIT"
# Upload indices to commit directory
echo "Uploading indices to $S3_COMMIT_PREFIX"
aws s3 cp --recursive "$INDICES_OUTPUT_DIR/" "$S3_COMMIT_PREFIX"
# Upload this build's variant indices under an S3 prefix (e.g. rocm/nightly), then
# rebuild that prefix's top-level project list from every variant now present.
publish_indices() {
local prefix="$1"
local listing variants merged
echo "Uploading indices to s3://$BUCKET/$prefix/"
aws s3 cp --recursive "$INDICES_OUTPUT_DIR/" "s3://$BUCKET/$prefix/" --exclude "index.html"
listing=$(aws s3api list-objects-v2 --bucket "$BUCKET" --prefix "$prefix/" \
--delimiter / --query 'CommonPrefixes[].Prefix' --output text)
variants=$(tr '\t' '\n' <<< "$listing" | sed -n "s|^$prefix/\(rocm[0-9]*\)/$|\1|p" | sort -u)
merged=$(mktemp)
{
echo '<!DOCTYPE html>'
echo '<html>'
echo ' <meta name="pypi:repository-version" content="1.0">'
echo ' <body>'
for variant in $variants; do
echo " <a href=\"$variant/\">$variant/</a><br/>"
done
echo ' </body>'
echo '</html>'
} > "$merged"
echo "Variants under $prefix/: $(echo "$variants" | tr '\n' ' ')"
aws s3 cp "$merged" "s3://$BUCKET/$prefix/index.html"
rm -f "$merged"
}
publish_indices "$ROCM_SUBPATH"
# Only scheduled nightly builds should update the moving nightly index.
if [[ "${NIGHTLY:-0}" == "1" ]]; then
echo "Updating rocm/nightly/ index..."
aws s3 cp --recursive "$INDICES_OUTPUT_DIR/" "s3://$BUCKET/rocm/nightly/"
publish_indices "rocm/nightly"
fi
# Extract version from vLLM wheel and update version-specific index
@@ -129,8 +176,7 @@ if [ -n "$VLLM_WHEEL" ]; then
echo "Pure version: $PURE_VERSION"
if [[ "$VERSION" != *"dev"* ]]; then
echo "Updating rocm/$PURE_VERSION/ index..."
aws s3 cp --recursive "$INDICES_OUTPUT_DIR/" "s3://$BUCKET/rocm/$PURE_VERSION/"
publish_indices "rocm/$PURE_VERSION"
fi
fi
@@ -142,14 +188,14 @@ echo "ROCm Wheel Upload Complete!"
echo "========================================"
echo ""
echo "Wheels available at:"
echo " s3://$BUCKET/$ROCM_SUBPATH/"
echo " s3://$BUCKET/$WHEEL_SUBPATH/"
echo ""
echo "Install command (by commit):"
echo " pip install vllm --extra-index-url https://${BUCKET}.s3.amazonaws.com/$ROCM_SUBPATH/"
echo " pip install vllm --extra-index-url https://${BUCKET}.s3.amazonaws.com/$ROCM_SUBPATH/$ROCM_STACK_VARIANT/"
echo ""
if [[ "${NIGHTLY:-0}" == "1" ]]; then
echo "Install command (nightly):"
echo " pip install vllm --extra-index-url https://${BUCKET}.s3.amazonaws.com/rocm/nightly/"
echo " pip install vllm --extra-index-url https://${BUCKET}.s3.amazonaws.com/rocm/nightly/$ROCM_STACK_VARIANT/"
fi
echo ""
echo "Wheel count: $WHEEL_COUNT"
.buildkite/test_areas/models_basic.yaml
+2 -1
View File
@@ -72,8 +72,9 @@ steps:
- tests/models/test_deepseek_v41_pcp.py
- tests/models/transformers/test_layer_registry_e2e.py
- tests/models/test_deepseek_v41_replay_start.py
- tests/models/test_deepseek_v4_vl_stream_weights.py
commands:
- pytest -v -s models/test_terratorch.py models/transformers/test_backend.py models/test_registry.py models/test_deepseek_v4_vl_rocm.py models/transformers/test_layer_registry_e2e.py models/test_deepseek_v41_replay_start.py models/test_deepseek_v41_decoder_replay_layers.py models/test_deepseek_v41_replay_batch.py models/test_deepseek_v41_pcp.py
- pytest -v -s models/test_terratorch.py models/transformers/test_backend.py models/test_registry.py models/test_deepseek_v4_vl_rocm.py models/transformers/test_layer_registry_e2e.py models/test_deepseek_v41_replay_start.py models/test_deepseek_v41_decoder_replay_layers.py models/test_deepseek_v41_replay_batch.py models/test_deepseek_v41_pcp.py models/test_deepseek_v4_vl_stream_weights.py
mirror:
amd:
label: ":amd: (MI355) Basic Models (Other)"
.github/CODEOWNERS
+3
View File
@@ -165,6 +165,9 @@
.readthedocs.yaml @hmellor
mkdocs.yaml @hmellor
# Model recipes
/recipes @esmeetu
# Linting
/tools/pre_commit/.markdownlint.yaml @hmellor
.pre-commit-config.yaml @hmellor
.github/workflows/recipes.yml
+28
View File
@@ -0,0 +1,28 @@
name: Recipes
# Validate recipe changes against the renderer that publishes them
# (recipes.vllm.ai). The check logic lives in vllm-project/recipes and tracks
# its main branch, so a PR is held to the same validation that runs when
# vllm-project/recipes picks up this directory after merge.
on:
pull_request:
paths:
- 'recipes/**'
- '.github/workflows/recipes.yml'
concurrency:
group: ${{ github.workflow }}-${{ github.ref }}
cancel-in-progress: true
permissions:
contents: read
jobs:
validate:
name: Recipes
if: github.repository_owner == 'vllm-project'
uses: vllm-project/recipes/.github/workflows/validate-recipes.yml@main
with:
source: vllm
path: recipes
.github/workflows/scripts/run_ci_command.py
+12 -2
View File
@@ -621,6 +621,16 @@ def blocks_new_run(command: str, build: Mapping[str, Any]) -> bool:
return is_active_build(build)
def list_commit_builds(
buildkite: BuildkiteClient, command: str, commit: str
) -> list[dict[str, Any]]:
builds = buildkite.list_builds(commit)
if command not in AMD_CI_COMMANDS:
return builds
# bootstrap-amd.sh exits before uploading any test steps for PR webhook builds.
return [build for build in builds if build.get("source") != "webhook"]
def select_latest_build(
builds: Sequence[dict[str, Any]],
pr_number: int,
@@ -968,7 +978,7 @@ def handle_run_ci(
f"{ci_name} was already requested by this comment: {duplicate['web_url']}"
)
current_builds = buildkite.list_builds(pr["head"]["sha"])
current_builds = list_commit_builds(buildkite, command, pr["head"]["sha"])
active_build = next(
(
build
@@ -1023,7 +1033,7 @@ def handle_retry_failed(
ci_name = ci_name_for_command(command)
run_command = run_command_for_command(command)
retry_command = retry_command_for_command(command)
builds = buildkite.list_builds(pr["head"]["sha"])
builds = list_commit_builds(buildkite, command, pr["head"]["sha"])
build = select_latest_build(builds, pr["number"])
if build:
metadata = build.get("meta_data") or {}
.github/workflows/scripts/test_run_ci_command.py
+59
View File
@@ -883,6 +883,42 @@ class RunCiCommandTest(unittest.TestCase):
self.assertEqual(buildkite.created_builds, [])
self.assertIn("AMD CI is already running", github.comments[0])
def test_amd_run_ignores_finished_webhook_build_for_author(self) -> None:
webhook_build = {
"created_at": "2026-08-18T01:00:00Z",
"meta_data": {},
"number": 122,
"pull_request": {"id": 42},
"source": "webhook",
"state": "passed",
"web_url": "https://buildkite.example/amd-ci/builds/122",
}
ui_build = {**webhook_build, "source": "ui"}
command_build = {
**webhook_build,
"meta_data": {"github-comment-id": "98"},
"source": "api",
}
for build, allowed in (
(webhook_build, True),
(ui_build, False),
(command_build, False),
):
with self.subTest(source=build["source"]):
github = FakeGitHub(
permission="read",
pr=make_pr(labels=[{"name": "ready"}]),
)
buildkite = FakeBuildkite([[], [build]])
run(make_event(COMMAND_RUN_AMD_CI, "author"), github, buildkite)
self.assertEqual(bool(buildkite.created_builds), allowed)
if not allowed:
self.assertIn(
"AMD CI already ran on this commit", github.comments[0]
)
def test_run_all_sets_buildkite_environment(self) -> None:
github = FakeGitHub()
buildkite = FakeBuildkite([[], []])
@@ -1229,6 +1265,29 @@ class RunCiCommandTest(unittest.TestCase):
)
self.assertIn("Use `/amd-ci run`", github.comments[0])
def test_amd_ci_retry_ignores_webhook_build_for_the_current_head(self) -> None:
github = FakeGitHub(
permission="read",
pr=make_pr(labels=[{"name": "ready"}]),
)
webhook_build = {
"created_at": "2026-08-18T01:00:00Z",
"meta_data": {},
"number": 122,
"pull_request": {"id": 42},
"source": "webhook",
"state": "passed",
"web_url": "https://buildkite.example/amd-ci/builds/122",
}
buildkite = FakeBuildkite([[webhook_build]])
run(make_event(COMMAND_RETRY_AMD_FAILED, "author"), github, buildkite)
self.assertEqual(buildkite.retry_calls, [])
self.assertIn(
"No AMD CI build exists for the current PR head", github.comments[0]
)
def test_ci_retry_creates_filtered_build_for_new_head(self) -> None:
github = FakeGitHub(
behind=100,
.pre-commit-config.yaml
+6
View File
@@ -277,6 +277,12 @@ repos:
files: ^docker/(Dockerfile|versions\.json)$
pass_filenames: false
additional_dependencies: [dockerfile-parse]
- id: check-dockerignore-tracked
name: Check .dockerignore does not exclude tracked files
entry: python tools/pre_commit/check_dockerignore_tracked.py
language: python
always_run: true
pass_filenames: false
- id: check-boolean-context-manager
name: Check for boolean ops in with-statements
entry: python tools/pre_commit/check_boolean_context_manager.py
CMakeLists.txt
+12
View File
@@ -415,6 +415,7 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
"csrc/cuda_view.cu"
"csrc/cuda_utils_kernels.cu"
"csrc/activation_kernels.cu"
"csrc/muse_spark_mul_tanh_soft_cap_kernel.cu"
"csrc/ngram_embedding_kernels.cu"
"csrc/vocab_parallel_embedding_kernels.cu"
"csrc/quantization/activation_kernels.cu"
@@ -423,6 +424,7 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
"csrc/quantization/w8a8/fp8/per_token_group_quant.cu"
"csrc/quantization/gptq/q_gemm.cu"
"csrc/pos_encoding_kernels.cu"
"csrc/fused_muse_spark_qknorm_rope_kv_insert_kernel.cu"
"csrc/fused_qknorm_rope_kernel.cu"
"csrc/fused_minimax_m3_qknorm_rope_kv_insert_kernel.cu"
"csrc/layernorm_kernels.cu"
@@ -441,6 +443,11 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
"csrc/fused_deepseek_v4_qnorm_rope_kv_insert_kernel.cu"
"csrc/fused_kimi_k3_mla_key_concat_kv_cache_kernel.cu")
if(VLLM_GPU_LANG STREQUAL "CUDA")
list(APPEND VLLM_STABLE_EXT_SRC
"csrc/fused_qknorm_rope_cache_kernel.cu")
endif()
if(VLLM_GPU_LANG STREQUAL "CUDA" AND
DEFINED CMAKE_CUDA_COMPILER_VERSION AND
CMAKE_CUDA_COMPILER_VERSION VERSION_GREATER_EQUAL 12.0)
@@ -1112,6 +1119,11 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
"9.0a;10.0f;12.0f" "${CUDA_ARCHS}")
cuda_archs_loose_intersection(FUSED_GDN_DECODE_ARCHS
"8.0;8.6;8.9;9.0a;10.0f;11.0f;12.0f" "${CUDA_ARCHS}")
else()
# The fused GDN decode kernel needs only cp.async and BF16 math. Only the
# family-specific targets (10.0f, 11.0f, 12.0f) need CUDA >= 13.0.
cuda_archs_loose_intersection(FUSED_GDN_DECODE_ARCHS
"8.0;8.6;8.9;9.0a" "${CUDA_ARCHS}")
endif()
if(FUSED_KDA_DECODE_ARCHS)
set(FUSED_KDA_DECODE_SRC
benchmarks/kernels/benchmark_muse_spark_qknorm_rope_kv_insert.py
+326
View File
@@ -0,0 +1,326 @@
# SPDX-License-Identifier: Apache-2.0
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
"""Benchmark the fused Muse-Spark QK-norm + RoPE + KV-insert kernel vs unfused.
``tuned_us`` forces the cp.async schedule ``fused_qk_norm_rope`` only auto-selects
on SM 9.0; ``unfused_us`` is what production runs.
"""
import argparse
import itertools
import torch
import vllm._custom_ops as ops
HEAD_DIM = 64
DEFAULT_TOKENS = [1, 8, 64, 256, 1024, 4096, 8192, 16384, 32768, 65536]
def make_cos_sin_cache(max_pos, rotary_dim, base, device):
inv_freq = 1.0 / (
base
** (
torch.arange(0, rotary_dim, 2, dtype=torch.float32, device=device)
/ rotary_dim
)
)
t = torch.arange(max_pos, dtype=torch.float32, device=device)
freqs = torch.einsum("i,j->ij", t, inv_freq)
return torch.cat((freqs.cos(), freqs.sin()), dim=-1)
class Case:
"""Everything a single (num_tokens, config) measurement needs."""
def __init__(self, num_tokens, nq, nkv, dtype, kv_cache_dtype, device):
self.num_tokens = num_tokens
self.nq, self.nkv = nq, nkv
self.dtype = dtype
self.kv_cache_dtype = kv_cache_dtype
self.eps = 1e-5
self.block_size = 128
self.rotary_dim = HEAD_DIM
self.qsz, self.kvsz = nq * HEAD_DIM, nkv * HEAD_DIM
self.row = self.qsz + 2 * self.kvsz + self.qsz # [q | k | v | o_gate]
self.qkv = torch.randn(num_tokens, self.row, dtype=dtype, device=device)
self.pristine = self.qkv.clone()
max_pos = 32768
self.cos_sin = make_cos_sin_cache(max_pos, self.rotary_dim, 500_000.0, device)
self.positions = torch.randint(
0, max_pos, (num_tokens,), dtype=torch.int64, device=device
)
num_blocks = (num_tokens + self.block_size - 1) // self.block_size + 1
storage = torch.uint8 if kv_cache_dtype != "auto" else dtype
self.kv_cache = torch.zeros(
num_blocks,
nkv,
self.block_size,
2 * HEAD_DIM,
dtype=storage,
device=device,
)
self.k_cache, self.v_cache = self.kv_cache.transpose(1, 2).split(
HEAD_DIM, dim=-1
)
self.slot_mapping = torch.randperm(
num_blocks * self.block_size, dtype=torch.int64, device=device
)[:num_tokens]
self.k_scale = torch.tensor(1.0, dtype=torch.float32, device=device)
self.v_scale = torch.tensor(1.0, dtype=torch.float32, device=device)
self.ones = torch.ones(HEAD_DIM, dtype=dtype, device=device)
self.q_fp8 = torch.empty(
num_tokens, self.qsz, dtype=torch.float8_e4m3fn, device=device
)
self.q_scale = torch.tensor(0.37, dtype=torch.float32, device=device)
def views(self):
q, k, v, _ = self.qkv.split([self.qsz, self.kvsz, self.kvsz, self.qsz], dim=-1)
return q, k, v
def fused(self, use_rope=True, quant_query=False):
ops.fused_muse_spark_qknorm_rope_kv_insert(
self.qkv,
self.positions,
self.nq,
self.nkv,
HEAD_DIM,
self.eps,
cos_sin_cache=self.cos_sin if use_rope else None,
rotary_dim=self.rotary_dim if use_rope else 0,
slot_mapping=self.slot_mapping,
kv_cache=self.kv_cache,
block_size=self.block_size,
kv_cache_dtype=self.kv_cache_dtype,
k_scale=self.k_scale if self.kv_cache_dtype != "auto" else None,
v_scale=self.v_scale if self.kv_cache_dtype != "auto" else None,
q_fp8_out=self.q_fp8 if quant_query else None,
q_scale=self.q_scale if quant_query else None,
)
def separate_query_quant(self, use_rope=True):
self.fused(use_rope)
ops.scaled_fp8_quant(self.qkv[:, : self.qsz], scale=self.q_scale)
def _insert(self):
_, k, v = self.views()
ops.reshape_and_cache_flash(
k.view(-1, self.nkv, HEAD_DIM),
v.view(-1, self.nkv, HEAD_DIM),
self.k_cache,
self.v_cache,
self.slot_mapping,
self.kv_cache_dtype,
self.k_scale,
self.v_scale,
)
def unfused_rope(self, token_heads_per_warp: int = -1):
ops.fused_qk_norm_rope(
self.qkv,
self.nq,
self.nkv,
self.nkv + self.nq, # v heads plus the o-gate
HEAD_DIM,
self.eps,
self.ones,
self.ones,
self.cos_sin,
False,
self.positions,
token_heads_per_warp,
)
self._insert()
def fused_bytes(self):
el = self.qkv.element_size()
cache_el = self.kv_cache.element_size()
read = (self.nq + 2 * self.nkv) * HEAD_DIM * el
write = (self.nq + self.nkv) * HEAD_DIM * el
cache = 2 * self.nkv * HEAD_DIM * cache_el
return self.num_tokens * (read + write + cache)
def _l2_bytes(device) -> int:
return torch.cuda.get_device_properties(device).L2_cache_size
def _peak_bandwidth(device) -> float:
"""Measured large-copy bandwidth, in GB/s -- the practical ceiling."""
n = 1 << 28 # 256 Mi elements = 512 MiB each way at 2 bytes
a = torch.empty(n, dtype=torch.bfloat16, device=device)
b = torch.empty_like(a)
for _ in range(3):
b.copy_(a)
torch.accelerator.synchronize()
start, end = torch.cuda.Event(True), torch.cuda.Event(True)
start.record()
iters = 20
for _ in range(iters):
b.copy_(a)
end.record()
torch.accelerator.synchronize()
ms = start.elapsed_time(end) / iters
return (2 * a.numel() * a.element_size()) / (ms * 1e-3) / 1e9
def time_eager(fn, warmup=20, iters=100) -> float:
for _ in range(warmup):
fn()
torch.accelerator.synchronize()
start, end = torch.cuda.Event(True), torch.cuda.Event(True)
start.record()
for _ in range(iters):
fn()
end.record()
torch.accelerator.synchronize()
return start.elapsed_time(end) / iters * 1e3 # us
def time_graph(fn, warmup=10, iters=50, inner=32) -> float:
"""GPU time per call; ``inner`` calls per graph amortize the ~2 us replay."""
side = torch.cuda.Stream()
side.wait_stream(torch.cuda.current_stream())
with torch.cuda.stream(side):
for _ in range(warmup):
fn()
torch.cuda.current_stream().wait_stream(side)
graph = torch.cuda.CUDAGraph()
with torch.cuda.graph(graph):
for _ in range(inner):
fn()
for _ in range(3):
graph.replay()
torch.accelerator.synchronize()
start, end = torch.cuda.Event(True), torch.cuda.Event(True)
start.record()
for _ in range(iters):
graph.replay()
end.record()
torch.accelerator.synchronize()
return start.elapsed_time(end) / (iters * inner) * 1e3 # us
def main():
p = argparse.ArgumentParser()
p.add_argument("--nq", type=int, default=32, help="query heads per rank")
p.add_argument("--nkv", type=int, default=4, help="kv heads per rank")
p.add_argument("--kv-cache-dtype", default="auto", choices=["auto", "fp8"])
p.add_argument("--dtype", default="bfloat16", choices=["bfloat16", "float16"])
p.add_argument("--tokens", type=int, nargs="*", default=DEFAULT_TOKENS)
p.add_argument("--csv", default=None)
p.add_argument("--no-graph", action="store_true")
p.add_argument(
"--query-fp8", action="store_true", help="compare fused Q quantization"
)
args = p.parse_args()
device = "cuda"
dtype = getattr(torch, args.dtype)
if args.query_fp8:
if args.kv_cache_dtype != "fp8":
p.error("--query-fp8 requires --kv-cache-dtype fp8")
timer = time_eager if args.no_graph else time_graph
print(f"device: {torch.cuda.get_device_name()}; nq={args.nq}, nkv={args.nkv}")
print("layer,tokens,separate_us,fused_us,speedup")
rows = []
for use_rope, nt in itertools.product([True, False], args.tokens):
case = Case(nt, args.nq, args.nkv, dtype, args.kv_cache_dtype, device)
separate_us = timer(lambda c=case, r=use_rope: c.separate_query_quant(r))
case.qkv.copy_(case.pristine)
fused_us = timer(lambda c=case, r=use_rope: c.fused(r, quant_query=True))
row = (
"rope" if use_rope else "nope",
nt,
separate_us,
fused_us,
separate_us / fused_us,
)
rows.append(row)
print(f"{row[0]},{nt},{separate_us:.3f},{fused_us:.3f},{row[4]:.3f}")
if args.csv:
import csv
with open(args.csv, "w", newline="") as f:
writer = csv.writer(f)
writer.writerow(
["layer", "tokens", "separate_us", "fused_us", "speedup"]
)
writer.writerows(rows)
return
peak = _peak_bandwidth(device)
print(
f"device: {torch.cuda.get_device_name()} measured copy peak: {peak:.0f} GB/s"
)
print(
f"config: nq={args.nq} nkv={args.nkv} head_dim={HEAD_DIM} "
f"dtype={args.dtype} kv_cache={args.kv_cache_dtype}\n"
)
modes = ["rope"]
l2 = _l2_bytes(device)
print(
f"rows marked * have a working set inside the {l2 / 2**20:.0f} MiB L2, "
"so their bandwidth is not an HBM number\n"
)
header = (
f"{'layer':>5} {'tokens':>7} {'fused_us':>9} {'unfused_us':>11} "
f"{'tuned_us':>9} {'speedup':>8} {'GB/s':>8} {'SOL%':>6}"
)
rows = []
for mode, use_graph in itertools.product(modes, [False, True]):
if use_graph and args.no_graph:
continue
label = "graph" if use_graph else "eager"
print(f"── {mode} layer, {label} ──")
print(header)
timer = time_graph if use_graph else time_eager
for nt in args.tokens:
case = Case(nt, args.nq, args.nkv, dtype, args.kv_cache_dtype, device)
use_rope = mode == "rope"
fused_us = timer(lambda c=case, r=use_rope: c.fused(r))
baseline = case.unfused_rope
unfused_us = timer(baseline)
tuned_us = (
min(timer(lambda c=case, t=t: c.unfused_rope(t)) for t in (4, 8))
if use_rope
else unfused_us
)
nbytes = case.fused_bytes()
gbps = nbytes / (fused_us * 1e-6) / 1e9
sol = 100.0 * gbps / peak
resident = "*" if nbytes < l2 else " "
print(
f"{mode:>5} {nt:>7} {fused_us:>9.2f} {unfused_us:>11.2f} "
f"{tuned_us:>9.2f} {tuned_us / fused_us:>7.2f}x {gbps:>8.0f} "
f"{sol:>5.1f}%{resident}"
)
rows.append((mode, label, nt, fused_us, unfused_us, tuned_us, gbps, sol))
print()
if args.csv:
import csv
with open(args.csv, "w", newline="") as f:
w = csv.writer(f)
w.writerow(
[
"layer",
"timing",
"tokens",
"fused_us",
"unfused_us",
"tuned_us",
"gbps",
"sol",
]
)
w.writerows(rows)
print(f"wrote {args.csv}")
if __name__ == "__main__":
main()
csrc/fused_muse_spark_qknorm_rope_kv_insert_kernel.cu
+752
View File
@@ -0,0 +1,752 @@
/*
* SPDX-License-Identifier: Apache-2.0
* SPDX-FileCopyrightText: Copyright contributors to the vLLM project
*
* Fused Muse Spark attention pre-processing: weightless QK RMSNorm, GPT-J
* interleaved RoPE (skipped on NoPE layers, ``cos_sin_cache == None``) and
* the K/V insert of ``reshape_and_cache_flash``, over ``[q | k | v | o_gate]``
* rows.
*
* A group of ``head_dim * sizeof(scalar_t) / 16`` lanes owns one head (8 lanes
* for bf16 head_dim=64), so each lane moves one 16-byte vector, the rotated
* pair (2i, 2i+1) is lane-local, and the norm reduction is 3 shuffles.
*
* Slot enumeration per token:
* [0, nq) Q heads -> norm (+RoPE), in place or to
* ``q_fp8_out``
* [nq, nq+nkv) K heads -> norm (+RoPE), in place and into
* the K cache
* [nq+nkv, nq+2*nkv) V heads -> copied into the V cache (no norm)
* V slots exist only when a KV cache is bound.
*
* Norm and rotation run in fp32 and round once at the store. Both fp8 paths
* quantize after rounding to the model dtype, the same boundary
* ``reshape_and_cache_flash`` and ``scaled_fp8_quant`` use.
*/
#include <algorithm>
#include <cmath>
#include <cuda_runtime.h>
#include <type_traits>
#include "torch_utils.h"
#include "core/cuda_compat.h"
#include "type_convert.cuh"
#include "core/vec_dtypes/dtype_fp8.cuh"
#include "dispatch_utils.h"
#ifndef USE_ROCM
#include <cuda_fp8.h>
#include "quantization/w8a8/fp8/nvidia/quant_utils.cuh"
#else
#include <hip/hip_fp8.h>
#include "quantization/w8a8/fp8/amd/quant_utils.cuh"
#endif
#ifndef FINAL_MASK
#ifdef USE_ROCM
#define FINAL_MASK 0xffffffffffffffffULL
#else
#define FINAL_MASK 0xffffffffu
#endif
#endif
namespace vllm {
namespace muse_spark_fused_ops {
namespace {
inline int getSMVersion() {
auto* props = get_device_prop();
return props->major * 10 + props->minor;
}
inline int getSMCount() {
auto* props = get_device_prop();
return props->multiProcessorCount;
}
} // namespace
// 16 bytes per lane: the widest vector access, and the reason a head is owned
// by a group of lanes rather than a whole warp.
constexpr int kBytesPerLane = 16;
#ifdef USE_ROCM
using WarpMask = uint64_t;
constexpr unsigned kWarpSize = 64;
#else
using WarpMask = unsigned;
constexpr unsigned kWarpSize = 32;
#endif
// Slot -> (token, head) needs a division by a value only known at launch. The
// kernel is issue-bound, and a 32-bit hardware division is ~20 instructions of
// it, so the reciprocal is computed once on the host instead.
struct FastDiv {
uint32_t mul;
uint32_t shr;
int div;
};
inline FastDiv makeFastDiv(int denom) {
// The reciprocal form needs denom >= 2; denom == 1 is carried as a flag the
// device side selects on. Dividends stay below 2^31, which covers every slot
// index this kernel can form.
if (denom <= 1) return FastDiv{0u, 0u, 1};
int lg = 31 - __builtin_clz(denom);
if ((1 << lg) < denom) lg++;
uint32_t const p = 31 + static_cast<uint32_t>(lg);
uint64_t const m = ((1ull << p) + static_cast<uint64_t>(denom) - 1) /
static_cast<uint64_t>(denom);
return FastDiv{static_cast<uint32_t>(m), p - 32, denom};
}
__device__ __forceinline__ void fastDivmod(int& quo, int& rem, int src,
FastDiv const fd) {
int const q =
static_cast<int>(__umulhi(static_cast<uint32_t>(src), fd.mul) >> fd.shr);
// A select, not a branch: fd.div is uniform across the launch.
quo = (fd.div == 1) ? src : q;
rem = src - quo * fd.div;
}
// Sum across the LANES contiguous lanes that share one head. The group is
// LANES-aligned inside the warp, so an XOR butterfly over masks < LANES never
// leaves it. `group_mask` names exactly the group: whole groups are uniformly
// active or inactive, so a warp whose other groups bailed out still syncs.
template <int LANES>
__device__ __forceinline__ float groupReduceSum(float val,
WarpMask const group_mask) {
#pragma unroll
for (int mask = LANES >> 1; mask > 0; mask >>= 1) {
val += __shfl_xor_sync(group_mask, val, mask, 32);
}
return val;
}
// 16 contiguous bytes -> kElems fp32 registers, keeping the raw vector so a
// pass-through slot (V) can be re-stored without a conversion round trip.
template <typename scalar_t, int kElems>
__device__ __forceinline__ uint4 loadElems(scalar_t const* __restrict__ src,
float (&elems)[kElems]) {
using Converter = vllm::_typeConvert<scalar_t>;
using T2 = typename Converter::packed_hip_type;
uint4 raw = *reinterpret_cast<uint4 const*>(src);
auto const* p = reinterpret_cast<T2 const*>(&raw);
#pragma unroll
for (int i = 0; i < kElems / 2; i++) {
float2 f2 = Converter::convert(p[i]);
elems[2 * i] = f2.x;
elems[2 * i + 1] = f2.y;
}
return raw;
}
// The lane's slice of the rope table: kPairs cos and kPairs sin values, each
// run contiguous, so each is one vector load (16 bytes for the fp32 table, 8
// for a model-dtype one) instead of kPairs scalar ones. rotary_dim being a
// multiple of 16 is what keeps both runs naturally aligned.
template <typename rope_t, int kPairs>
__device__ __forceinline__ void loadRopeVec(rope_t const* __restrict__ cos_ptr,
rope_t const* __restrict__ sin_ptr,
float (&c)[kPairs],
float (&s)[kPairs]) {
static_assert(kPairs == 4, "rope vector load assumes 4 pairs per lane");
if constexpr (std::is_same_v<rope_t, float>) {
float4 const cv = VLLM_LDG(reinterpret_cast<float4 const*>(cos_ptr));
float4 const sv = VLLM_LDG(reinterpret_cast<float4 const*>(sin_ptr));
c[0] = cv.x, c[1] = cv.y, c[2] = cv.z, c[3] = cv.w;
s[0] = sv.x, s[1] = sv.y, s[2] = sv.z, s[3] = sv.w;
} else {
using Converter = vllm::_typeConvert<rope_t>;
using T2 = typename Converter::packed_hip_type;
uint2 const cv = VLLM_LDG(reinterpret_cast<uint2 const*>(cos_ptr));
uint2 const sv = VLLM_LDG(reinterpret_cast<uint2 const*>(sin_ptr));
auto const* cp = reinterpret_cast<T2 const*>(&cv);
auto const* sp = reinterpret_cast<T2 const*>(&sv);
#pragma unroll
for (int i = 0; i < kPairs / 2; i++) {
float2 const cf = Converter::convert(cp[i]);
float2 const sf = Converter::convert(sp[i]);
c[2 * i] = cf.x, c[2 * i + 1] = cf.y;
s[2 * i] = sf.x, s[2 * i + 1] = sf.y;
}
}
}
// kElems fp32 registers -> the 16-byte scalar_t vector they round to. Returned
// rather than stored so the fp8 cache path can reuse the *rounded* values,
// which is what reshape_and_cache_flash quantizes from.
template <typename scalar_t, int kElems>
__device__ __forceinline__ uint4 packElems(float const (&elems)[kElems]) {
using Converter = vllm::_typeConvert<scalar_t>;
using T2 = typename Converter::packed_hip_type;
uint4 raw;
auto* p = reinterpret_cast<T2*>(&raw);
#pragma unroll
for (int i = 0; i < kElems / 2; i++) {
p[i] = Converter::convert(make_float2(elems[2 * i], elems[2 * i + 1]));
}
return raw;
}
// Scatter one lane's 16 bytes into the KV cache. kAuto stores the model dtype
// verbatim; the fp8 dtypes quantize the value *after* it has been rounded to
// scalar_t, which is the boundary reshape_and_cache_flash uses.
//
// `inv_scale` is taken once per thread: per-element fp32 divisions would cost
// more than the rest of the kernel. The extra rounding is 2^-24 relative.
template <typename scalar_t, typename cache_t, Fp8KVCacheDataType kv_dt,
int kElems>
__device__ __forceinline__ void storeCache(cache_t* __restrict__ dst,
uint4 const& packed,
float const inv_scale) {
if constexpr (kv_dt == Fp8KVCacheDataType::kAuto) {
*reinterpret_cast<uint4*>(dst) = packed;
} else {
using Converter = vllm::_typeConvert<scalar_t>;
using HipT = typename Converter::hip_type;
auto const* rounded = reinterpret_cast<HipT const*>(&packed);
#ifndef USE_ROCM
constexpr __nv_fp8_interpretation_t kInterp =
(kv_dt == Fp8KVCacheDataType::kFp8E4M3) ? __NV_E4M3 : __NV_E5M2;
__nv_fp8x2_storage_t out[kElems / 2];
#pragma unroll
for (int i = 0; i < kElems / 2; i++) {
float2 v =
make_float2(Converter::convert(rounded[2 * i]) * inv_scale,
Converter::convert(rounded[2 * i + 1]) * inv_scale);
out[i] = __nv_cvt_float2_to_fp8x2(v, __NV_SATFINITE, kInterp);
}
// kElems fp8 bytes: 8 for the 2-byte dtypes this kernel serves.
static_assert(kElems == 8, "fp8 store assumes 8 elements per lane");
*reinterpret_cast<uint2*>(dst) = *reinterpret_cast<uint2 const*>(out);
#else
#pragma unroll
for (int i = 0; i < kElems; i++) {
float const v = Converter::convert(rounded[i]) * inv_scale;
dst[i] = fp8::scaled_convert<cache_t, float, kv_dt>(v, 1.0f);
}
#endif
}
}
// Grid is 1D over (token, head-chunk) pairs. A chunk is HEADS_PER_GROUP
// consecutive head slots of one token, served by kLanes contiguous lanes; the
// whole chunk shares the token's position and its slice of the rope table, so
// the slot division, the position load and the cos/sin load are paid once per
// chunk. `slots_per_token % HEADS_PER_GROUP == 0` keeps a chunk inside one
// token. `rope_t` is the cos/sin cache dtype, independent of the qkv dtype.
template <typename scalar_t, typename cache_t, Fp8KVCacheDataType kv_dt,
typename rope_t, int HEAD_DIM, int HEADS_PER_GROUP, bool kDoRope,
bool kInsertKV>
__global__
__launch_bounds__(256, 6) void fusedMuseSparkQKNormRopeKVInsertKernel(
scalar_t* __restrict__ qkv, // [N, qkv_stride] in/out
rope_t const* __restrict__ cos_sin_cache, // [max_pos, rotary_dim]
int64_t const* __restrict__ positions, // [N] i64
int64_t const* __restrict__ slot_mapping, // [N] i64 or nullptr
cache_t* __restrict__ kv_cache, // [nb, nkv, bs, 2*HEAD_DIM]
float const* __restrict__ k_scale, // [1] or nullptr
float const* __restrict__ v_scale, // [1] or nullptr
uint8_t* __restrict__ q_fp8_out, // [N, nq * HEAD_DIM] or nullptr
float const* __restrict__ q_scale, // [1] or nullptr
float const eps, int const rotary_dim, int const num_tokens, int const nq,
int const nkv, FastDiv const chunks_div, int64_t const qkv_stride,
FastDiv const block_div, int64_t const kv_s_block, int64_t const kv_s_head,
int64_t const kv_s_token) {
#if (!defined(__CUDA_ARCH__) || __CUDA_ARCH__ < 800) && !defined(USE_ROCM)
// _typeConvert<BFloat16> is unavailable on pre-Ampere; this kernel only ever
// runs with bf16/fp16 inputs. Discard the bf16 body there.
if constexpr (std::is_same_v<scalar_t, c10::BFloat16>) {
return;
} else {
#endif
constexpr int kElems = kBytesPerLane / sizeof(scalar_t);
constexpr int kLanes = HEAD_DIM / kElems;
constexpr int kPairs = kElems / 2;
constexpr int kH = HEADS_PER_GROUP;
static_assert(kLanes > 0 && kLanes <= 32 && (32 % kLanes) == 0,
"head_dim must give a power-of-two lane group <= 32");
int const tid = blockIdx.x * blockDim.x + threadIdx.x;
int const groupIdx = tid / kLanes;
int const dimBase = (tid % kLanes) * kElems;
int tokenIdx, chunk;
fastDivmod(tokenIdx, chunk, groupIdx, chunks_div);
int const firstSlot = chunk * kH;
// Out-of-range groups must not exit before the reductions below: the XOR
// butterfly needs every lane the mask names, and the mask covers the whole
// group. They ride along and skip every memory access instead.
bool const active = tokenIdx < num_tokens;
constexpr WarpMask kGroupBits =
(kLanes == 32) ? static_cast<WarpMask>(0xffffffffu)
: ((WarpMask{1} << kLanes) - WarpMask{1});
unsigned const group_shift =
(threadIdx.x & (kWarpSize - 1)) & ~static_cast<unsigned>(kLanes - 1);
WarpMask const group_mask = kGroupBits << group_shift;
// PDL: wait for the qkv-projection GEMM that produces `qkv`. No-op when
// PDL is not enabled on the launch.
#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900)
cudaGridDependencySynchronize();
#endif
// Q, K and V are contiguous head slots at the head of the row, so the slot
// index doubles as the head offset. The trailing o-gate is skipped by the
// row stride alone.
scalar_t* const base = qkv + static_cast<int64_t>(tokenIdx) * qkv_stride +
static_cast<int64_t>(firstSlot) * HEAD_DIM + dimBase;
// Issue every load in the chunk before touching any of it, so the group has
// kH loads in flight instead of one. An inactive group leaves these unset;
// nothing downstream reads them, since every store is gated on `active`.
uint4 raw[kH];
if (active) {
#pragma unroll
for (int h = 0; h < kH; h++) {
raw[h] = *reinterpret_cast<uint4 const*>(base + h * HEAD_DIM);
}
}
float cosv[kPairs], sinv[kPairs];
bool const laneRotates = kDoRope && dimBase < rotary_dim;
if constexpr (kDoRope) {
if (active && laneRotates) {
int64_t const pos = positions[tokenIdx];
rope_t const* cache_ptr = cos_sin_cache + pos * rotary_dim;
loadRopeVec<rope_t, kPairs>(cache_ptr + dimBase / 2,
cache_ptr + rotary_dim / 2 + dimBase / 2,
cosv, sinv);
}
}
int64_t sm = -1;
if constexpr (kInsertKV) {
if (active) sm = slot_mapping[tokenIdx];
}
int cache_block = 0, cache_tok = 0;
float k_inv = 1.0f, v_inv = 1.0f;
float const q_inv = q_fp8_out ? 1.0f / *q_scale : 1.0f;
if constexpr (kInsertKV) {
if (sm >= 0) {
fastDivmod(cache_block, cache_tok, static_cast<int>(sm), block_div);
}
if constexpr (kv_dt != Fp8KVCacheDataType::kAuto) {
k_inv = 1.0f / *k_scale;
v_inv = 1.0f / *v_scale;
}
}
#pragma unroll
for (int h = 0; h < kH; h++) {
int const slot = firstSlot + h;
bool const isV = kInsertKV && slot >= nq + nkv;
bool const isK = slot >= nq && slot < nq + nkv;
bool const isQK = active && !isV;
float elems[kElems];
uint4 packed = raw[h];
if (isQK) {
using Converter = vllm::_typeConvert<scalar_t>;
using T2 = typename Converter::packed_hip_type;
auto const* p = reinterpret_cast<T2 const*>(&raw[h]);
#pragma unroll
for (int i = 0; i < kElems / 2; i++) {
float2 const f2 = Converter::convert(p[i]);
elems[2 * i] = f2.x;
elems[2 * i + 1] = f2.y;
}
float sumsq = 0.0f;
#pragma unroll
for (int i = 0; i < kElems; i++) sumsq += elems[i] * elems[i];
sumsq = groupReduceSum<kLanes>(sumsq, group_mask);
float const rms_rcp =
rsqrtf(sumsq / static_cast<float>(HEAD_DIM) + eps);
#pragma unroll
for (int i = 0; i < kElems; i++) elems[i] *= rms_rcp;
if constexpr (kDoRope) {
if (laneRotates) {
#pragma unroll
for (int i = 0; i < kPairs; i++) {
float const v0 = elems[2 * i];
float const v1 = elems[2 * i + 1];
elems[2 * i] = v0 * cosv[i] - v1 * sinv[i];
elems[2 * i + 1] = v0 * sinv[i] + v1 * cosv[i];
}
}
}
// Rounding to scalar_t happens exactly once, here, and both the
// Q output and the fp8 cache quantization consume the same
// rounded value. V passes through with the vector it was loaded with.
packed = packElems<scalar_t, kElems>(elems);
if (q_fp8_out && slot < nq) {
int64_t const q_off =
(static_cast<int64_t>(tokenIdx) * nq + slot) * HEAD_DIM + dimBase;
storeCache<scalar_t, uint8_t, Fp8KVCacheDataType::kFp8E4M3, kElems>(
q_fp8_out + q_off, packed, q_inv);
} else {
*reinterpret_cast<uint4*>(base + h * HEAD_DIM) = packed;
}
}
if constexpr (kInsertKV) {
if (sm >= 0 && (isK || isV)) { // sm < 0 marks a padded token
int const head = isK ? (slot - nq) : (slot - nq - nkv);
int64_t const off = static_cast<int64_t>(cache_block) * kv_s_block +
head * kv_s_head +
static_cast<int64_t>(cache_tok) * kv_s_token +
(isK ? 0 : HEAD_DIM) + dimBase;
storeCache<scalar_t, cache_t, kv_dt, kElems>(kv_cache + off, packed,
isK ? k_inv : v_inv);
}
}
}
// PDL: let a dependent successor launch early.
#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900)
cudaTriggerProgrammaticLaunchCompletion();
#endif
#if (!defined(__CUDA_ARCH__) || __CUDA_ARCH__ < 800) && !defined(USE_ROCM)
}
#endif
}
template <typename scalar_t, typename cache_t, Fp8KVCacheDataType kv_dt,
typename rope_t, int HEAD_DIM, int HEADS_PER_GROUP>
void launchFusedMuseSparkTyped(
scalar_t* qkv, rope_t const* cos_sin_cache, int64_t const* positions,
int64_t const* slot_mapping, cache_t* kv_cache, float const* k_scale,
float const* v_scale, uint8_t* q_fp8_out, float const* q_scale,
float const eps, int const rotary_dim, int const num_tokens, int const nq,
int const nkv, int64_t const qkv_stride, int const block_size,
int64_t const kv_s_block, int64_t const kv_s_head, int64_t const kv_s_token,
bool const do_rope, bool const insert_kv, cudaStream_t stream) {
FastDiv const block_div = makeFastDiv(block_size);
constexpr int kElems = kBytesPerLane / sizeof(scalar_t);
constexpr int kLanes = HEAD_DIM / kElems;
int const slots_per_token = nq + nkv + (insert_kv ? nkv : 0);
int const chunks_per_token = slots_per_token / HEADS_PER_GROUP;
FastDiv const chunks_div = makeFastDiv(chunks_per_token);
int64_t const total_lanes =
static_cast<int64_t>(num_tokens) * chunks_per_token * kLanes;
if (total_lanes == 0) return;
constexpr int kBlockSize = 256;
int const grid =
static_cast<int>((total_lanes + kBlockSize - 1) / kBlockSize);
#ifndef USE_ROCM
// PDL wherever the hardware supports it (SM90+); a plain launch elsewhere.
static int const sm_version = getSMVersion();
cudaLaunchConfig_t config;
config.gridDim = dim3(grid);
config.blockDim = dim3(kBlockSize);
config.dynamicSmemBytes = 0;
config.stream = stream;
cudaLaunchAttribute attrs[1];
attrs[0].id = cudaLaunchAttributeProgrammaticStreamSerialization;
attrs[0].val.programmaticStreamSerializationAllowed = 1;
config.attrs = attrs;
config.numAttrs = (sm_version >= 90) ? 1 : 0;
#define LAUNCH_MUSE(DO_ROPE, INSERT) \
cudaLaunchKernelEx(&config, \
fusedMuseSparkQKNormRopeKVInsertKernel< \
scalar_t, cache_t, kv_dt, rope_t, HEAD_DIM, \
HEADS_PER_GROUP, DO_ROPE, INSERT>, \
qkv, cos_sin_cache, positions, slot_mapping, kv_cache, \
k_scale, v_scale, q_fp8_out, q_scale, eps, rotary_dim, \
num_tokens, nq, nkv, chunks_div, qkv_stride, block_div, \
kv_s_block, kv_s_head, kv_s_token)
#else
// clang-format off
#define LAUNCH_MUSE(DO_ROPE, INSERT) \
fusedMuseSparkQKNormRopeKVInsertKernel< \
scalar_t, cache_t, kv_dt, rope_t, HEAD_DIM, HEADS_PER_GROUP, DO_ROPE, \
INSERT><<<grid, kBlockSize, 0, stream>>>( \
qkv, cos_sin_cache, positions, slot_mapping, kv_cache, k_scale, \
v_scale, q_fp8_out, q_scale, eps, rotary_dim, num_tokens, nq, nkv, \
chunks_div, qkv_stride, block_div, kv_s_block, kv_s_head, kv_s_token)
// clang-format on
#endif
if (do_rope) {
if (insert_kv) {
LAUNCH_MUSE(true, true);
} else {
LAUNCH_MUSE(true, false);
}
} else {
if (insert_kv) {
LAUNCH_MUSE(false, true);
} else {
LAUNCH_MUSE(false, false);
}
}
#undef LAUNCH_MUSE
}
template <typename scalar_t, typename cache_t, Fp8KVCacheDataType kv_dt>
void launchFusedMuseSpark(
scalar_t* qkv, void const* cos_sin_cache, bool const rope_is_fp32,
int64_t const* positions, int64_t const* slot_mapping, cache_t* kv_cache,
float const* k_scale, float const* v_scale, uint8_t* q_fp8_out,
float const* q_scale, float const eps, int const rotary_dim,
int const num_tokens, int const nq, int const nkv, int const head_dim,
int64_t const qkv_stride, int const block_size, int64_t const kv_s_block,
int64_t const kv_s_head, int64_t const kv_s_token, bool const do_rope,
bool const insert_kv, cudaStream_t stream) {
// The group width must divide the slot count. Widen only while the grid
// still covers a full wave; decode-sized batches keep the narrow schedule.
int const slots_per_token = nq + nkv + (insert_kv ? nkv : 0);
constexpr int kElemsSel = kBytesPerLane / sizeof(scalar_t);
constexpr int kBlockSel = 256;
int const lanes_per_head = head_dim / kElemsSel;
int64_t const full_wave =
static_cast<int64_t>(getSMCount()) * (2048 / kBlockSel);
int heads_per_group = 1;
for (int h : {4, 2}) {
if (slots_per_token % h != 0) continue;
int64_t const blocks = (static_cast<int64_t>(num_tokens) *
(slots_per_token / h) * lanes_per_head +
kBlockSel - 1) /
kBlockSel;
if (blocks >= full_wave) {
heads_per_group = h;
break;
}
}
// TP4/TP8 amortize head addressing before filling a full CTA wave. RoPE
// benefits earlier; FP8 conversion keeps wider groups expensive for longer.
if (insert_kv && head_dim == 64 &&
((nq == 16 && nkv == 2) || (nq == 32 && nkv == 4))) {
int64_t const query_heads = static_cast<int64_t>(num_tokens) * nq;
if (do_rope && query_heads >= 24576) {
heads_per_group = std::max(heads_per_group, 2);
if constexpr (kv_dt == Fp8KVCacheDataType::kAuto) {
if (query_heads >= 49152) heads_per_group = 4;
}
} else if (!do_rope && query_heads >= 32768) {
if constexpr (kv_dt == Fp8KVCacheDataType::kAuto) {
heads_per_group = std::max(heads_per_group, 2);
}
}
}
#define DISPATCH_ARGS(RT, HD, HPG) \
launchFusedMuseSparkTyped<scalar_t, cache_t, kv_dt, RT, HD, HPG>( \
qkv, reinterpret_cast<RT const*>(cos_sin_cache), positions, \
slot_mapping, kv_cache, k_scale, v_scale, q_fp8_out, q_scale, eps, \
rotary_dim, num_tokens, nq, nkv, qkv_stride, block_size, kv_s_block, \
kv_s_head, kv_s_token, do_rope, insert_kv, stream)
#define DISPATCH_HPG(RT, HD) \
switch (heads_per_group) { \
case 4: \
DISPATCH_ARGS(RT, HD, 4); \
break; \
case 2: \
DISPATCH_ARGS(RT, HD, 2); \
break; \
default: \
DISPATCH_ARGS(RT, HD, 1); \
break; \
}
// A NoPE launch never reads the table, so it can share the fp32 branch.
#define DISPATCH_ROPE_T(HD) \
if (!do_rope || rope_is_fp32) { \
DISPATCH_HPG(float, HD); \
} else { \
DISPATCH_HPG(scalar_t, HD); \
}
switch (head_dim) {
case 64:
DISPATCH_ROPE_T(64);
break;
case 128:
DISPATCH_ROPE_T(128);
break;
case 256:
DISPATCH_ROPE_T(256);
break;
default:
STD_TORCH_CHECK(false, "unsupported head_dim: ", head_dim);
}
#undef DISPATCH_ROPE_T
#undef DISPATCH_HPG
#undef DISPATCH_ARGS
}
} // namespace muse_spark_fused_ops
} // namespace vllm
// clang-format off
#define CALL_FUSED_MUSE_SPARK(_RAW_T, CACHE_T, KV_DTYPE) \
vllm::muse_spark_fused_ops::launchFusedMuseSpark<st, CACHE_T, KV_DTYPE>( \
reinterpret_cast<st*>(qkv.data_ptr()), \
do_rope ? cos_sin_cache->data_ptr() : nullptr, rope_is_fp32, \
reinterpret_cast<int64_t const*>(positions.data_ptr()), \
insert_kv ? reinterpret_cast<int64_t const*>(slot_mapping->data_ptr()) \
: nullptr, \
insert_kv ? reinterpret_cast<CACHE_T*>(kv_cache->data_ptr()) : nullptr, \
needs_scales ? reinterpret_cast<float const*>(k_scale->data_ptr()) \
: nullptr, \
needs_scales ? reinterpret_cast<float const*>(v_scale->data_ptr()) \
: nullptr, \
q_fp8_out ? reinterpret_cast<uint8_t*>(q_fp8_out->data_ptr()) : nullptr, \
q_scale ? reinterpret_cast<float const*>(q_scale->data_ptr()) : nullptr, \
static_cast<float>(eps), static_cast<int>(rotary_dim), num_tokens, nq, \
nkv, hd, qkv_stride, static_cast<int>(block_size), kv_s_block, \
kv_s_head, kv_s_token, do_rope, insert_kv, stream)
// clang-format on
void fused_muse_spark_qknorm_rope_kv_insert(
torch::stable::Tensor& qkv, // [N, (nq + 2*nkv + gate)*hd]
torch::stable::Tensor const& positions, // [N] i64
int64_t num_heads, int64_t num_kv_heads, int64_t head_dim, double eps,
std::optional<torch::stable::Tensor> cos_sin_cache, // None => NoPE layer
int64_t rotary_dim,
std::optional<torch::stable::Tensor> slot_mapping, // [N] i64
std::optional<torch::stable::Tensor> kv_cache, // [nb, nkv, bs, 2*hd]
int64_t block_size, const std::string& kv_cache_dtype,
std::optional<torch::stable::Tensor> k_scale, // [1] fp32, fp8 cache only
std::optional<torch::stable::Tensor> v_scale,
std::optional<torch::stable::Tensor> q_fp8_out,
std::optional<torch::stable::Tensor> q_scale) {
STD_TORCH_CHECK(qkv.is_cuda() && qkv.dim() == 2 && qkv.stride(1) == 1,
"qkv must be a 2-D CUDA tensor with a contiguous row");
STD_TORCH_CHECK(
qkv.scalar_type() == torch::headeronly::ScalarType::Half ||
qkv.scalar_type() == torch::headeronly::ScalarType::BFloat16,
"qkv must be float16 or bfloat16");
STD_TORCH_CHECK(
positions.is_cuda() &&
positions.scalar_type() == torch::headeronly::ScalarType::Long,
"positions must be int64 CUDA");
int const num_tokens = static_cast<int>(qkv.size(0));
int const nq = static_cast<int>(num_heads);
int const nkv = static_cast<int>(num_kv_heads);
int const hd = static_cast<int>(head_dim);
int64_t const qkv_stride = qkv.stride(0);
STD_TORCH_CHECK(q_fp8_out.has_value() == q_scale.has_value(),
"q_fp8_out and q_scale must be provided together");
if (q_fp8_out) {
STD_TORCH_CHECK(
q_fp8_out->is_cuda() &&
q_fp8_out->get_device_index() == qkv.get_device_index() &&
q_fp8_out->scalar_type() ==
torch::headeronly::ScalarType::Float8_e4m3fn &&
q_fp8_out->dim() == 2 && q_fp8_out->is_contiguous() &&
q_fp8_out->size(0) == num_tokens &&
q_fp8_out->size(1) == static_cast<int64_t>(nq) * hd,
"q_fp8_out must be contiguous E4M3 [num_tokens, num_heads * head_dim] "
"on the qkv device");
STD_TORCH_CHECK(
q_scale->is_cuda() &&
q_scale->get_device_index() == qkv.get_device_index() &&
q_scale->scalar_type() == torch::headeronly::ScalarType::Float &&
q_scale->numel() == 1,
"q_scale must be a single fp32 value on the qkv device");
}
STD_TORCH_CHECK(hd == 64 || hd == 128 || hd == 256,
"head_dim must be 64, 128 or 256");
STD_TORCH_CHECK(qkv.size(1) >= static_cast<int64_t>(nq + 2 * nkv) * hd,
"qkv row must hold at least [q|k|v]");
// A lane moves 16 bytes, so every row start and every head start must land
// on a 16-byte boundary. Every supported head_dim is a multiple of 8 at 2
// bytes, so heads are fine; the row stride is the one that can break it.
STD_TORCH_CHECK(qkv_stride % 8 == 0,
"qkv row stride must be a multiple of 8 elements, got ",
qkv_stride);
bool const do_rope = cos_sin_cache.has_value();
bool rope_is_fp32 = true;
if (do_rope) {
STD_TORCH_CHECK(cos_sin_cache->is_cuda() && cos_sin_cache->dim() == 2 &&
cos_sin_cache->is_contiguous(),
"cos_sin_cache must be a contiguous 2-D CUDA tensor");
STD_TORCH_CHECK(cos_sin_cache->size(1) == rotary_dim,
"cos_sin_cache shape must be [max_pos, rotary_dim]");
STD_TORCH_CHECK(
cos_sin_cache->scalar_type() == torch::headeronly::ScalarType::Float ||
cos_sin_cache->scalar_type() == qkv.scalar_type(),
"cos_sin_cache must be float32 or match the qkv dtype");
rope_is_fp32 =
cos_sin_cache->scalar_type() == torch::headeronly::ScalarType::Float;
// A lane owns 16 bytes = 8 elements of the head and must be wholly inside
// or wholly outside the rotated prefix.
STD_TORCH_CHECK(rotary_dim > 0 && rotary_dim <= hd && rotary_dim % 16 == 0,
"rotary_dim must be a positive multiple of 16 and <= "
"head_dim");
}
vllm::Fp8KVCacheDataType const kv_dt =
vllm::get_fp8_kv_cache_data_type(kv_cache_dtype);
bool const insert_kv = kv_cache.has_value() && kv_cache->numel() > 0;
bool const needs_scales =
insert_kv && kv_dt != vllm::Fp8KVCacheDataType::kAuto;
int64_t kv_s_block = 0, kv_s_head = 0, kv_s_token = 0;
if (insert_kv) {
STD_TORCH_CHECK(
slot_mapping.has_value() && slot_mapping->is_cuda() &&
slot_mapping->scalar_type() == torch::headeronly::ScalarType::Long,
"insert mode requires an int64 CUDA slot_mapping");
STD_TORCH_CHECK(slot_mapping->numel() >= num_tokens,
"slot_mapping must cover every token");
STD_TORCH_CHECK(kv_cache->dim() == 4 && kv_cache->stride(3) == 1,
"kv_cache must be [nb, nkv, bs, 2*head_dim] with a "
"contiguous content dim");
STD_TORCH_CHECK(kv_cache->size(3) == 2 * hd,
"kv_cache content dim must be 2*head_dim");
STD_TORCH_CHECK(kv_cache->size(1) == nkv,
"kv_cache head dim must equal num_kv_heads");
STD_TORCH_CHECK(kv_cache->size(2) == block_size,
"kv_cache token dim must equal block_size");
if (kv_dt == vllm::Fp8KVCacheDataType::kAuto) {
STD_TORCH_CHECK(kv_cache->scalar_type() == qkv.scalar_type(),
"auto kv_cache dtype must match qkv");
} else {
STD_TORCH_CHECK(
kv_cache->scalar_type() == torch::headeronly::ScalarType::Byte ||
kv_cache->scalar_type() ==
torch::headeronly::ScalarType::Float8_e4m3fn,
"fp8 kv_cache must use uint8 or float8_e4m3fn storage");
STD_TORCH_CHECK(k_scale.has_value() && v_scale.has_value(),
"fp8 kv cache requires k_scale and v_scale");
STD_TORCH_CHECK(
k_scale->numel() == 1 && v_scale->numel() == 1 &&
k_scale->scalar_type() == torch::headeronly::ScalarType::Float &&
v_scale->scalar_type() == torch::headeronly::ScalarType::Float,
"k_scale and v_scale must be single fp32 values");
}
kv_s_block = kv_cache->stride(0);
kv_s_head = kv_cache->stride(1);
kv_s_token = kv_cache->stride(2);
}
const torch::stable::accelerator::DeviceGuard device_guard(
qkv.get_device_index());
auto stream = get_current_cuda_stream(qkv.get_device_index());
VLLM_STABLE_DISPATCH_HALF_TYPES(
qkv.scalar_type(), "fused_muse_spark_qknorm_rope_kv_insert", [&] {
using st = scalar_t;
DISPATCH_BY_KV_CACHE_DTYPE(qkv.scalar_type(), kv_cache_dtype,
CALL_FUSED_MUSE_SPARK);
});
}
#undef CALL_FUSED_MUSE_SPARK
csrc/fused_qknorm_rope_cache_kernel.cu
+567
View File
@@ -0,0 +1,567 @@
// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright contributors to the vLLM project
// Step5 fused QK norm, RoPE, and KV-cache update kernel.
// This kernel is built into `_C_stable_libtorch` and uses the libtorch stable
// ABI (torch::stable::Tensor / STD_TORCH_CHECK) instead of ATen and c10.
#include "torch_utils.h"
#include "dispatch_utils.h"
#include "type_convert.cuh"
#include "core/vec_dtypes/dtype_fp8.cuh"
#include "quantization/w8a8/fp8/nvidia/quant_utils.cuh"
#include <torch/csrc/stable/macros.h>
#include <cuda_runtime.h>
#include <algorithm>
#include <type_traits>
namespace vllm {
namespace {
constexpr int kWarpSize = 32;
constexpr int kThreadsPerBlock = 128;
template <typename scalar_t, typename weight_t, typename rope_t, int HEAD_DIM,
bool WIDE_192_ROTARY = false, bool FP8_CACHE = false>
__global__ void qknorm_rope_cache_kernel(
const void* __restrict__ qkv_void, void* __restrict__ q_out_void,
void* __restrict__ k_out_void, void* __restrict__ v_out_void,
const void* __restrict__ q_weight_void,
const void* __restrict__ k_weight_void, const void* __restrict__ cos_void,
const void* __restrict__ sin_void, const int64_t* __restrict__ positions,
const int64_t* __restrict__ slot_mapping, void* __restrict__ k_cache_void,
void* __restrict__ v_cache_void, int64_t qkv_token_stride,
int64_t q_token_stride, int64_t k_token_stride, int64_t v_token_stride,
int64_t cos_token_stride, int64_t sin_token_stride,
int64_t cache_block_stride, int64_t cache_page_stride,
int64_t cache_head_stride, int num_tokens, int num_slots, int num_q_heads,
int num_kv_heads, int rotary_pairs, int block_size, float epsilon,
int block_size_shift, float norm_weight_bias, const float* k_scale,
const float* v_scale) {
#if !defined(__CUDA_ARCH__) || __CUDA_ARCH__ < 800
if constexpr (std::is_same_v<scalar_t, torch::headeronly::BFloat16> ||
std::is_same_v<weight_t, torch::headeronly::BFloat16> ||
std::is_same_v<rope_t, torch::headeronly::BFloat16>) {
return;
} else {
#endif
using ScalarConverter = _typeConvert<scalar_t>;
using WeightConverter = _typeConvert<weight_t>;
using RopeConverter = _typeConvert<rope_t>;
using Scalar = typename ScalarConverter::hip_type;
using Weight = typename WeightConverter::hip_type;
using Rope = typename RopeConverter::hip_type;
// Preserve the original per-head reduction and activation rounding.
static_assert(!WIDE_192_ROTARY || HEAD_DIM == 192);
constexpr int kThreadsPerHead =
WIDE_192_ROTARY ? 32
: (HEAD_DIM <= 64 ? 8 : (HEAD_DIM <= 192 ? 16 : 32));
constexpr int kValuesPerVector =
WIDE_192_ROTARY ? 8 : (HEAD_DIM == 192 ? 4 : 8);
constexpr int kValuesPerTile = kThreadsPerHead * kValuesPerVector;
constexpr int kNumVectors =
(HEAD_DIM + kValuesPerTile - 1) / kValuesPerTile;
constexpr int kValuesPerThread = kValuesPerVector * kNumVectors;
constexpr int kTileSize = kValuesPerTile * kNumVectors;
constexpr int kHeadsPerBlock = kThreadsPerBlock / kThreadsPerHead;
const auto* qkv = static_cast<const Scalar*>(qkv_void);
auto* q_out = static_cast<Scalar*>(q_out_void);
auto* k_out = static_cast<Scalar*>(k_out_void);
auto* v_out = static_cast<Scalar*>(v_out_void);
const auto* q_weight = static_cast<const Weight*>(q_weight_void);
const auto* k_weight = static_cast<const Weight*>(k_weight_void);
const auto* cos = static_cast<const Rope*>(cos_void);
const auto* sin = static_cast<const Rope*>(sin_void);
auto* k_cache = static_cast<Scalar*>(k_cache_void);
auto* v_cache = static_cast<Scalar*>(v_cache_void);
const int head_in_block = threadIdx.x / kThreadsPerHead;
const int lane = threadIdx.x % kThreadsPerHead;
const int total_heads = num_q_heads + 2 * num_kv_heads;
int token;
int local_head;
const int64_t global_head =
static_cast<int64_t>(blockIdx.x) * kHeadsPerBlock + head_in_block;
token = global_head / total_heads;
// Reuse the quotient instead of spelling this as global_head %
// total_heads. nvcc otherwise emits a second runtime integer-division
// sequence for the flattened token/head mapping.
local_head = static_cast<int>(global_head -
static_cast<int64_t>(token) * total_heads);
if (token >= num_tokens) return;
const bool is_q = local_head < num_q_heads;
const bool is_k = !is_q && local_head < num_q_heads + num_kv_heads;
const int head = is_q ? local_head
: (is_k ? local_head - num_q_heads
: local_head - num_q_heads - num_kv_heads);
int64_t input_offset = token * qkv_token_stride;
Scalar* output;
int64_t output_offset;
if (is_q) {
input_offset += static_cast<int64_t>(head) * HEAD_DIM;
output = q_out;
output_offset =
token * q_token_stride + static_cast<int64_t>(head) * HEAD_DIM;
} else if (is_k) {
input_offset += static_cast<int64_t>(num_q_heads + head) * HEAD_DIM;
output = k_out;
output_offset =
token * k_token_stride + static_cast<int64_t>(head) * HEAD_DIM;
} else {
input_offset +=
static_cast<int64_t>(num_q_heads + num_kv_heads + head) * HEAD_DIM;
output = v_out;
output_offset =
token * v_token_stride + static_cast<int64_t>(head) * HEAD_DIM;
}
bool write_cache = false;
int64_t cache_offset = 0;
if (!is_q && token < num_slots) {
const int64_t slot = slot_mapping[token];
if (slot >= 0) {
int64_t block;
int64_t page;
if (block_size_shift >= 0) {
// vLLM production cache blocks are powers of two. Avoid the costly
// runtime int64 division on that path while retaining generic block
// sizes through the fallback below.
block = slot >> block_size_shift;
page = slot & (block_size - 1);
} else {
block = slot / block_size;
page = slot - block * block_size;
}
cache_offset = block * cache_block_stride + page * cache_page_stride +
static_cast<int64_t>(head) * cache_head_stride;
write_cache = true;
}
}
if (!is_q && !is_k) {
#pragma unroll
for (int block = 0; block < kNumVectors; ++block) {
#pragma unroll
for (int element = 0; element < kValuesPerVector; ++element) {
const int dim = block * kThreadsPerHead * kValuesPerVector +
lane * kValuesPerVector + element;
if constexpr (kTileSize > HEAD_DIM) {
if (dim >= HEAD_DIM) continue;
}
const Scalar value = qkv[input_offset + dim];
output[output_offset + dim] = value;
if (write_cache) {
if constexpr (FP8_CACHE) {
reinterpret_cast<uint8_t*>(v_cache_void)[cache_offset + dim] =
fp8::scaled_convert<uint8_t, float,
Fp8KVCacheDataType::kFp8E4M3>(
ScalarConverter::convert(value), *v_scale);
} else {
v_cache[cache_offset + dim] = value;
}
}
}
}
return;
}
float values[kValuesPerThread];
float sum_squares = 0.0f;
#pragma unroll
for (int block = 0; block < kNumVectors; ++block) {
#pragma unroll
for (int element = 0; element < kValuesPerVector; ++element) {
const int value_index = block * kValuesPerVector + element;
const int dim = block * kThreadsPerHead * kValuesPerVector +
lane * kValuesPerVector + element;
float value = 0.0f;
if constexpr (kTileSize > HEAD_DIM) {
if (dim < HEAD_DIM) {
value = ScalarConverter::convert(qkv[input_offset + dim]);
}
} else {
value = ScalarConverter::convert(qkv[input_offset + dim]);
}
values[value_index] = value;
// CuTeDSL lowers its fragment reduction to a sequential fma.rn
// chain. Keep this explicit: changing it changes RMSNorm bits.
sum_squares = __fmaf_rn(value, value, sum_squares);
}
}
const int warp_lane = threadIdx.x % kWarpSize;
const int subwarp_base = (warp_lane / kThreadsPerHead) * kThreadsPerHead;
constexpr uint32_t kSubwarpBits =
kThreadsPerHead == kWarpSize ? 0xffffffffu
: ((uint32_t{1} << kThreadsPerHead) - 1);
const uint32_t subwarp_mask = kSubwarpBits << subwarp_base;
#pragma unroll
for (int offset = 1; offset < kThreadsPerHead; offset <<= 1) {
// CuTeDSL's warp_reduce uses ascending XOR offsets (1, 2, 4, ...).
sum_squares +=
__shfl_xor_sync(subwarp_mask, sum_squares, offset, kWarpSize);
}
const float mean_square =
__fdiv_rn(sum_squares, static_cast<float>(HEAD_DIM));
const float inverse_rms = rsqrtf(mean_square + epsilon);
const Weight* weight = is_q ? q_weight : k_weight;
// CuTeDSL writes RMSNorm results to activation-typed shared memory and
// reloads them for RoPE. This intermediate FP16/BF16 rounding boundary
// is intentional and is necessary for bitwise output parity.
__shared__ Scalar normalized[kHeadsPerBlock * HEAD_DIM];
Scalar* normalized_head = normalized + head_in_block * HEAD_DIM;
#pragma unroll
for (int block = 0; block < kNumVectors; ++block) {
#pragma unroll
for (int element = 0; element < kValuesPerVector; ++element) {
const int value_index = block * kValuesPerVector + element;
const int dim = block * kThreadsPerHead * kValuesPerVector +
lane * kValuesPerVector + element;
if constexpr (kTileSize > HEAD_DIM) {
if (dim >= HEAD_DIM) continue;
}
const float x_hat = values[value_index] * inverse_rms;
const float norm_weight =
WeightConverter::convert(weight[dim]) + norm_weight_bias;
normalized_head[dim] = ScalarConverter::convert(x_hat * norm_weight);
}
}
__syncwarp(subwarp_mask);
// Apply NeoX-style RoPE. CuTeDSL assigns four rotary pairs to each
// participating lane and reloads activation-typed normalized values.
constexpr int kRotaryPairsPerThread = 4;
if (lane < rotary_pairs / kRotaryPairsPerThread) {
const int64_t position = positions[token];
const Rope* token_cos = cos + position * cos_token_stride;
const Rope* token_sin = sin + position * sin_token_stride;
#pragma unroll
for (int element = 0; element < kRotaryPairsPerThread; ++element) {
const int pair = lane * kRotaryPairsPerThread + element;
const int dim0 = pair;
const int dim1 = pair + rotary_pairs;
const float value0 = ScalarConverter::convert(normalized_head[dim0]);
const float value1 = ScalarConverter::convert(normalized_head[dim1]);
const float cos_value = RopeConverter::convert(token_cos[pair]);
const float sin_value = RopeConverter::convert(token_sin[pair]);
// Match CuTeDSL PTX exactly. The first half is mul + mul + sub; the
// second is one mul followed by one fused multiply-add.
const float rotated0 = __fsub_rn(__fmul_rn(value0, cos_value),
__fmul_rn(value1, sin_value));
const float rotated1 =
__fmaf_rn(value0, sin_value, __fmul_rn(value1, cos_value));
normalized_head[dim0] = ScalarConverter::convert(rotated0);
normalized_head[dim1] = ScalarConverter::convert(rotated1);
}
}
__syncwarp(subwarp_mask);
#pragma unroll
for (int block = 0; block < kNumVectors; ++block) {
#pragma unroll
for (int element = 0; element < kValuesPerVector; ++element) {
const int dim = block * kThreadsPerHead * kValuesPerVector +
lane * kValuesPerVector + element;
if constexpr (kTileSize > HEAD_DIM) {
if (dim >= HEAD_DIM) continue;
}
const Scalar value = normalized_head[dim];
output[output_offset + dim] = value;
if (write_cache) {
if constexpr (FP8_CACHE) {
reinterpret_cast<uint8_t*>(k_cache_void)[cache_offset + dim] =
fp8::scaled_convert<uint8_t, float,
Fp8KVCacheDataType::kFp8E4M3>(
ScalarConverter::convert(value), *k_scale);
} else {
k_cache[cache_offset + dim] = value;
}
}
}
}
#if !defined(__CUDA_ARCH__) || __CUDA_ARCH__ < 800
}
#endif
}
template <typename scalar_t, typename weight_t, typename rope_t, bool FP8_CACHE>
void launch_qknorm_rope_cache(
const torch::stable::Tensor& qkv, torch::stable::Tensor& q_out,
torch::stable::Tensor& k_out, torch::stable::Tensor& v_out,
const torch::stable::Tensor& q_weight,
const torch::stable::Tensor& k_weight, const torch::stable::Tensor& cos,
const torch::stable::Tensor& sin, const torch::stable::Tensor& positions,
const torch::stable::Tensor& slot_mapping, torch::stable::Tensor& k_cache,
torch::stable::Tensor& v_cache, int head_dim, int num_q_heads,
int num_kv_heads, int rotary_pairs, float epsilon, float norm_weight_bias,
cudaStream_t stream, const float* k_scale, const float* v_scale) {
const int total_heads = num_q_heads + 2 * num_kv_heads;
const int64_t total_work = qkv.size(0) * total_heads;
const int block_size = static_cast<int>(k_cache.size(1));
const int block_size_shift =
(block_size & (block_size - 1)) == 0
? __builtin_ctz(static_cast<unsigned int>(block_size))
: -1;
#define LAUNCH_QKNORM_ROPE_CACHE(HEAD_DIM, WIDE_192_ROTARY) \
do { \
constexpr int threads_per_head = \
WIDE_192_ROTARY ? 32 \
: (HEAD_DIM <= 64 ? 8 : (HEAD_DIM <= 192 ? 16 : 32)); \
constexpr int heads_per_block = kThreadsPerBlock / threads_per_head; \
const dim3 grid((total_work + heads_per_block - 1) / heads_per_block); \
const dim3 block(kThreadsPerBlock); \
qknorm_rope_cache_kernel<scalar_t, weight_t, rope_t, HEAD_DIM, \
WIDE_192_ROTARY, FP8_CACHE> \
<<<grid, block, 0, stream>>>( \
qkv.data_ptr(), q_out.data_ptr(), k_out.data_ptr(), \
v_out.data_ptr(), q_weight.data_ptr(), k_weight.data_ptr(), \
cos.data_ptr(), sin.data_ptr(), \
reinterpret_cast<const int64_t*>(positions.data_ptr()), \
reinterpret_cast<const int64_t*>(slot_mapping.data_ptr()), \
k_cache.data_ptr(), v_cache.data_ptr(), qkv.stride(0), \
q_out.stride(0), k_out.stride(0), v_out.stride(0), cos.stride(0), \
sin.stride(0), k_cache.stride(0), k_cache.stride(1), \
k_cache.stride(2), static_cast<int>(qkv.size(0)), \
static_cast<int>(slot_mapping.numel()), num_q_heads, num_kv_heads, \
rotary_pairs, block_size, epsilon, block_size_shift, \
norm_weight_bias, k_scale, v_scale); \
} while (false)
switch (head_dim) {
case 64:
LAUNCH_QKNORM_ROPE_CACHE(64, false);
break;
case 128:
LAUNCH_QKNORM_ROPE_CACHE(128, false);
break;
case 192:
if (rotary_pairs > 64) {
LAUNCH_QKNORM_ROPE_CACHE(192, true);
} else {
LAUNCH_QKNORM_ROPE_CACHE(192, false);
}
break;
case 256:
LAUNCH_QKNORM_ROPE_CACHE(256, false);
break;
default:
STD_TORCH_CHECK(false, "Unsupported head_dim: ", head_dim);
}
#undef LAUNCH_QKNORM_ROPE_CACHE
}
void check_cuda_tensor(const torch::stable::Tensor& tensor, const char* name) {
STD_TORCH_CHECK(tensor.is_cuda(), name, " must be a CUDA tensor");
}
void check_same_device(const torch::stable::Tensor& reference,
const torch::stable::Tensor& tensor, const char* name) {
STD_TORCH_CHECK(tensor.get_device_index() == reference.get_device_index(),
name, " must be on the same CUDA device as qkv");
}
} // namespace
void fused_qknorm_rope_cache(
torch::stable::Tensor& q_out, torch::stable::Tensor& k_out,
torch::stable::Tensor& v_out, torch::stable::Tensor& qkv,
torch::stable::Tensor& q_weight, torch::stable::Tensor& k_weight,
torch::stable::Tensor& cos, torch::stable::Tensor& sin,
torch::stable::Tensor& positions, torch::stable::Tensor& slot_mapping,
torch::stable::Tensor& k_cache, torch::stable::Tensor& v_cache,
int64_t head_dim, int64_t num_q_heads, int64_t num_kv_heads,
int64_t rotary_pairs, double epsilon, double norm_weight_bias,
const std::string& kv_cache_dtype,
std::optional<torch::stable::Tensor> k_scale,
std::optional<torch::stable::Tensor> v_scale) {
check_cuda_tensor(qkv, "qkv");
check_cuda_tensor(q_out, "q_out");
check_cuda_tensor(k_out, "k_out");
check_cuda_tensor(v_out, "v_out");
check_cuda_tensor(q_weight, "q_weight");
check_cuda_tensor(k_weight, "k_weight");
check_cuda_tensor(cos, "cos");
check_cuda_tensor(sin, "sin");
check_cuda_tensor(positions, "positions");
check_cuda_tensor(slot_mapping, "slot_mapping");
check_cuda_tensor(k_cache, "k_cache");
check_cuda_tensor(v_cache, "v_cache");
check_same_device(qkv, q_out, "q_out");
check_same_device(qkv, k_out, "k_out");
check_same_device(qkv, v_out, "v_out");
check_same_device(qkv, q_weight, "q_weight");
check_same_device(qkv, k_weight, "k_weight");
check_same_device(qkv, cos, "cos");
check_same_device(qkv, sin, "sin");
check_same_device(qkv, positions, "positions");
check_same_device(qkv, slot_mapping, "slot_mapping");
check_same_device(qkv, k_cache, "k_cache");
check_same_device(qkv, v_cache, "v_cache");
STD_TORCH_CHECK(qkv.dim() == 2 && qkv.stride(1) == 1,
"qkv must be a 2D tensor contiguous in its last dimension");
STD_TORCH_CHECK(q_out.dim() == 2 && q_out.is_contiguous(),
"q_out must be contiguous and 2D");
STD_TORCH_CHECK(k_out.dim() == 2 && k_out.is_contiguous(),
"k_out must be contiguous and 2D");
STD_TORCH_CHECK(v_out.dim() == 2 && v_out.is_contiguous(),
"v_out must be contiguous and 2D");
STD_TORCH_CHECK(q_weight.dim() == 1 && q_weight.is_contiguous(),
"q_weight must be contiguous and 1D");
STD_TORCH_CHECK(k_weight.dim() == 1 && k_weight.is_contiguous(),
"k_weight must be contiguous and 1D");
STD_TORCH_CHECK(cos.dim() == 2 && cos.stride(1) == 1,
"cos must be 2D and contiguous in its last dimension");
STD_TORCH_CHECK(sin.dim() == 2 && sin.stride(1) == 1,
"sin must be 2D and contiguous in its last dimension");
STD_TORCH_CHECK(
positions.scalar_type() == torch::headeronly::ScalarType::Long,
"positions must be an int64 tensor");
STD_TORCH_CHECK(positions.is_contiguous(), "positions must be contiguous");
STD_TORCH_CHECK(
slot_mapping.scalar_type() == torch::headeronly::ScalarType::Long,
"slot_mapping must be an int64 tensor");
STD_TORCH_CHECK(slot_mapping.is_contiguous(),
"slot_mapping must be contiguous");
STD_TORCH_CHECK(k_cache.dim() == 4 && v_cache.dim() == 4,
"K/V cache must have shape [blocks, block_size, heads, dim]");
for (int64_t d = 0; d < 4; ++d) {
STD_TORCH_CHECK(k_cache.size(d) == v_cache.size(d) &&
k_cache.stride(d) == v_cache.stride(d),
"K/V cache must have identical shapes and strides");
}
STD_TORCH_CHECK(k_cache.stride(3) == 1,
"K/V cache head dimension must be contiguous");
STD_TORCH_CHECK(num_q_heads > 0 && num_kv_heads > 0,
"Q/KV head counts must be positive");
STD_TORCH_CHECK(
head_dim == 64 || head_dim == 128 || head_dim == 192 || head_dim == 256,
"Unsupported head_dim");
STD_TORCH_CHECK(cos.size(0) == sin.size(0),
"cos/sin cache must have the same number of positions");
const int64_t num_tokens = qkv.size(0);
const int64_t packed_width = (num_q_heads + 2 * num_kv_heads) * head_dim;
STD_TORCH_CHECK(qkv.size(1) == packed_width,
"qkv width must equal (num_q_heads + 2 * num_kv_heads) * "
"head_dim");
STD_TORCH_CHECK(
q_out.size(0) == num_tokens && q_out.size(1) == num_q_heads * head_dim,
"q_out has an invalid shape");
STD_TORCH_CHECK(
k_out.size(0) == num_tokens && k_out.size(1) == num_kv_heads * head_dim,
"k_out has an invalid shape");
STD_TORCH_CHECK(
v_out.size(0) == k_out.size(0) && v_out.size(1) == k_out.size(1),
"v_out has an invalid shape");
STD_TORCH_CHECK(q_weight.numel() == head_dim && k_weight.numel() == head_dim,
"Q/K norm weights must have head_dim elements");
STD_TORCH_CHECK(positions.numel() == num_tokens,
"positions must have one entry per qkv token");
STD_TORCH_CHECK(rotary_pairs >= 0 && rotary_pairs * 2 <= head_dim &&
rotary_pairs % 4 == 0,
"rotary_pairs must be a multiple of 4 in [0, head_dim / 2]");
STD_TORCH_CHECK(cos.size(1) >= rotary_pairs && sin.size(1) >= rotary_pairs,
"cos/sin cache is smaller than rotary_pairs");
STD_TORCH_CHECK(
k_cache.size(2) == num_kv_heads && k_cache.size(3) == head_dim,
"K/V cache head shape does not match the kernel arguments");
STD_TORCH_CHECK(k_cache.size(1) > 0, "K/V cache block_size must be positive");
STD_TORCH_CHECK(slot_mapping.numel() <= num_tokens,
"slot_mapping cannot contain more entries than qkv tokens");
STD_TORCH_CHECK(qkv.scalar_type() == q_out.scalar_type() &&
qkv.scalar_type() == k_out.scalar_type() &&
qkv.scalar_type() == v_out.scalar_type(),
"qkv and Q/K/V outputs must have the same dtype");
const bool fp8_cache =
kv_cache_dtype == "fp8" || kv_cache_dtype == "fp8_e4m3";
STD_TORCH_CHECK(fp8_cache || kv_cache_dtype == "auto" ||
kv_cache_dtype == "bfloat16" ||
kv_cache_dtype == "float16",
"Unsupported KV cache dtype: ", kv_cache_dtype);
STD_TORCH_CHECK(k_cache.scalar_type() == v_cache.scalar_type(),
"K/V caches must have the same dtype");
if (fp8_cache) {
STD_TORCH_CHECK(
k_cache.scalar_type() == torch::headeronly::ScalarType::Byte ||
k_cache.scalar_type() ==
torch::headeronly::ScalarType::Float8_e4m3fn,
"FP8 cache requires uint8 or float8_e4m3fn storage");
STD_TORCH_CHECK(k_scale.has_value() && v_scale.has_value(),
"FP8 cache requires K/V scales");
for (const auto* scale : {&*k_scale, &*v_scale}) {
check_cuda_tensor(*scale, "KV scale");
check_same_device(qkv, *scale, "KV scale");
STD_TORCH_CHECK(
scale->numel() == 1 &&
scale->scalar_type() == torch::headeronly::ScalarType::Float,
"KV scales must be single FP32 values");
}
} else {
STD_TORCH_CHECK(qkv.scalar_type() == k_cache.scalar_type(),
"Unquantized cache must match QKV dtype");
}
STD_TORCH_CHECK(q_weight.scalar_type() == k_weight.scalar_type(),
"Q/K norm weights must have the same dtype");
STD_TORCH_CHECK(cos.scalar_type() == sin.scalar_type(),
"cos and sin must have the same dtype");
if (num_tokens == 0) return;
const torch::stable::accelerator::DeviceGuard device_guard(
qkv.get_device_index());
const cudaStream_t stream = get_current_cuda_stream(qkv.get_device_index());
VLLM_STABLE_DISPATCH_HALF_TYPES(
qkv.scalar_type(), "qknorm_rope_cache_input", [&] {
using input_t = scalar_t;
VLLM_STABLE_DISPATCH_FLOATING_TYPES(
q_weight.scalar_type(), "qknorm_rope_cache_weight", [&] {
using norm_weight_t = scalar_t;
VLLM_STABLE_DISPATCH_FLOATING_TYPES(
cos.scalar_type(), "qknorm_rope_cache_rope", [&] {
using rope_cache_t = scalar_t;
auto launch = [&](auto fp8_tag) {
launch_qknorm_rope_cache<input_t, norm_weight_t,
rope_cache_t,
decltype(fp8_tag)::value>(
qkv, q_out, k_out, v_out, q_weight, k_weight, cos,
sin, positions, slot_mapping, k_cache, v_cache,
static_cast<int>(head_dim),
static_cast<int>(num_q_heads),
static_cast<int>(num_kv_heads),
static_cast<int>(rotary_pairs),
static_cast<float>(epsilon),
static_cast<float>(norm_weight_bias), stream,
fp8_cache
? static_cast<const float*>(k_scale->data_ptr())
: nullptr,
fp8_cache
? static_cast<const float*>(v_scale->data_ptr())
: nullptr);
};
if (fp8_cache) {
launch(std::true_type{});
} else {
launch(std::false_type{});
}
});
});
});
STD_CUDA_KERNEL_LAUNCH_CHECK();
}
} // namespace vllm
csrc/muse_spark_mul_tanh_soft_cap_kernel.cu
+100
View File
@@ -0,0 +1,100 @@
// SPDX-License-Identifier: Apache-2.0
// SPDX-FileCopyrightText: Copyright contributors to the vLLM project
#include <cuda_runtime.h>
#include <torch/csrc/stable/accelerator.h>
#include <torch/csrc/stable/tensor.h>
#include <torch/headeronly/core/ScalarType.h>
#include <algorithm>
#include "dispatch_utils.h"
#include "quantization/vectorization_utils.cuh"
#include "torch_utils.h"
namespace vllm {
// 16 B of output per thread; low-precision inputs are widened in registers.
static constexpr int kMuseSoftCapVecSize = 4;
// blockIdx.x walks rows, blockIdx.y splits a row. Rows are the sampled tokens
// (often one, at decode) while a row is the whole vocab, so splitting the row
// is what keeps the GPU busy; a row-per-block grid would leave a single-token
// decode on one SM.
template <typename scalar_t>
__global__ void muse_spark_mul_tanh_soft_cap_kernel(
float* __restrict__ out, const scalar_t* __restrict__ input, int d,
float multiplier, float soft_cap_temp) {
const int64_t row_offset = static_cast<int64_t>(blockIdx.x) * d;
const scalar_t* __restrict__ in_row = input + row_offset;
float* __restrict__ out_row = out + row_offset;
const int tid = blockIdx.y * blockDim.x + threadIdx.x;
const int stride = gridDim.y * blockDim.x;
vectorize_with_alignment<kMuseSoftCapVecSize>(
in_row, out_row, d, tid, stride,
[=] __device__(float& dst, const scalar_t& src) {
// Multiply then divide, as MulTanhSoftCap.forward does, so fp32
// rounding matches the eager path.
dst = soft_cap_temp *
tanhf(static_cast<float>(src) * multiplier / soft_cap_temp);
});
}
} // namespace vllm
// out = soft_cap_temp * tanh(input * multiplier / soft_cap_temp), fp32.
void muse_spark_mul_tanh_soft_cap(torch::stable::Tensor& out,
torch::stable::Tensor& input,
double multiplier, double soft_cap_temp) {
STD_TORCH_CHECK(
input.scalar_type() == torch::headeronly::ScalarType::Float ||
input.scalar_type() == torch::headeronly::ScalarType::Half ||
input.scalar_type() == torch::headeronly::ScalarType::BFloat16,
"muse_spark_mul_tanh_soft_cap: input must be float32, float16 or "
"bfloat16");
STD_TORCH_CHECK(out.scalar_type() == torch::headeronly::ScalarType::Float,
"muse_spark_mul_tanh_soft_cap: out must be float32");
STD_TORCH_CHECK(input.is_contiguous(),
"muse_spark_mul_tanh_soft_cap: input must be contiguous");
STD_TORCH_CHECK(out.is_contiguous(),
"muse_spark_mul_tanh_soft_cap: out must be contiguous");
STD_TORCH_CHECK(out.numel() == input.numel(),
"muse_spark_mul_tanh_soft_cap: out and input must have the "
"same number of elements");
STD_TORCH_CHECK(
soft_cap_temp != 0.0,
"muse_spark_mul_tanh_soft_cap: soft_cap_temp must be nonzero");
if (input.numel() == 0) {
return;
}
const int d = static_cast<int>(input.size(input.dim() - 1));
const int64_t num_tokens = input.numel() / d;
constexpr int kThreads = 128;
const int64_t vecs_per_row =
(d + vllm::kMuseSoftCapVecSize - 1) / vllm::kMuseSoftCapVecSize;
const int64_t blocks_to_cover = (vecs_per_row + kThreads - 1) / kThreads;
constexpr int64_t kTargetBlocks = 32768;
const int64_t blocks_per_row = std::clamp<int64_t>(
kTargetBlocks / num_tokens, 1, std::max<int64_t>(blocks_to_cover, 1));
dim3 grid(static_cast<unsigned>(num_tokens),
static_cast<unsigned>(blocks_per_row));
dim3 block(kThreads);
const torch::stable::accelerator::DeviceGuard device_guard(
input.get_device_index());
const cudaStream_t stream = get_current_cuda_stream();
VLLM_STABLE_DISPATCH_FLOATING_TYPES(
input.scalar_type(), "muse_spark_mul_tanh_soft_cap", [&] {
vllm::muse_spark_mul_tanh_soft_cap_kernel<scalar_t>
<<<grid, block, 0, stream>>>(out.mutable_data_ptr<float>(),
input.const_data_ptr<scalar_t>(), d,
static_cast<float>(multiplier),
static_cast<float>(soft_cap_temp));
});
}
csrc/ops.h
+33
View File
@@ -363,6 +363,35 @@ minimax_allreduce_rms_qk(torch::stable::Tensor qkv,
int64_t const nranks, double const eps);
#endif
#ifndef USE_ROCM
namespace vllm {
void fused_qknorm_rope_cache(
torch::stable::Tensor& q_out, torch::stable::Tensor& k_out,
torch::stable::Tensor& v_out, torch::stable::Tensor& qkv,
torch::stable::Tensor& q_weight, torch::stable::Tensor& k_weight,
torch::stable::Tensor& cos, torch::stable::Tensor& sin,
torch::stable::Tensor& positions, torch::stable::Tensor& slot_mapping,
torch::stable::Tensor& k_cache, torch::stable::Tensor& v_cache,
int64_t head_dim, int64_t num_q_heads, int64_t num_kv_heads,
int64_t rotary_pairs, double epsilon, double norm_weight_bias,
const std::string& kv_cache_dtype,
std::optional<torch::stable::Tensor> k_scale,
std::optional<torch::stable::Tensor> v_scale);
} // namespace vllm
#endif
void fused_muse_spark_qknorm_rope_kv_insert(
torch::stable::Tensor& qkv, torch::stable::Tensor const& positions,
int64_t num_heads, int64_t num_kv_heads, int64_t head_dim, double eps,
std::optional<torch::stable::Tensor> cos_sin_cache, int64_t rotary_dim,
std::optional<torch::stable::Tensor> slot_mapping,
std::optional<torch::stable::Tensor> kv_cache, int64_t block_size,
const std::string& kv_cache_dtype,
std::optional<torch::stable::Tensor> k_scale,
std::optional<torch::stable::Tensor> v_scale,
std::optional<torch::stable::Tensor> q_fp8_out,
std::optional<torch::stable::Tensor> q_scale);
// Horizontally-fused MiniMax-M3 QK-norm + partial NeoX RoPE (+ optional KV /
// index-cache insert). Dense layer: norm+RoPE only; sparse layer: also packs
// the index branch and scatters k/v/index_k into their paged caches.
@@ -546,6 +575,10 @@ std::tuple<int64_t, torch::stable::Tensor> allocate_shared_buffer_and_handle(
int64_t open_mem_handle(torch::stable::Tensor& mem_handle);
void free_shared_buffer(int64_t buffer);
void muse_spark_mul_tanh_soft_cap(torch::stable::Tensor& out,
torch::stable::Tensor& input,
double multiplier, double soft_cap_temp);
// Activation kernels (shared CUDA/ROCm)
void silu_and_mul(torch::stable::Tensor& out, torch::stable::Tensor& input);
void silu_and_mul_clamp(torch::stable::Tensor& out,
csrc/torch_bindings.cpp
+31
View File
@@ -501,6 +501,27 @@ STABLE_TORCH_LIBRARY_FRAGMENT(_C, ops) {
"float eps) -> (Tensor, Tensor)");
#endif
#ifndef USE_ROCM
ops.def(
"fused_qknorm_rope_cache("
"Tensor! q_out, Tensor! k_out, Tensor! v_out, Tensor qkv, "
"Tensor q_weight, Tensor k_weight, Tensor cos, Tensor sin, "
"Tensor positions, Tensor slot_mapping, Tensor! k_cache, "
"Tensor! v_cache, int head_dim, int num_q_heads, int num_kv_heads, "
"int rotary_pairs, float epsilon, float norm_weight_bias, "
"str kv_cache_dtype=\"auto\", Tensor? k_scale=None, Tensor? "
"v_scale=None) -> ()");
#endif
ops.def(
"fused_muse_spark_qknorm_rope_kv_insert("
"Tensor! qkv, Tensor positions, int num_heads, int num_kv_heads, "
"int head_dim, float eps, Tensor? cos_sin_cache, int rotary_dim, "
"Tensor? slot_mapping, Tensor!? kv_cache, int block_size, "
"str kv_cache_dtype, Tensor? k_scale=None, Tensor? v_scale=None, "
"Tensor!? q_fp8_out=None, Tensor? q_scale=None) "
"-> ()");
// Horizontally-fused MiniMax-M3 QK-norm + partial NeoX RoPE + KV-insert.
ops.def(
"fused_minimax_m3_qknorm_rope_kv_insert("
@@ -598,6 +619,10 @@ STABLE_TORCH_LIBRARY_FRAGMENT(_C, ops) {
"y_q, Tensor! y_s, bool use_ue8m0) -> ()");
ops.def("weak_ref_tensor(Tensor input) -> Tensor");
ops.def(
"muse_spark_mul_tanh_soft_cap(Tensor! out, Tensor input, "
"float multiplier, float soft_cap_temp) -> ()");
// Activation function used in SwiGLU.
ops.def("silu_and_mul(Tensor! result, Tensor input) -> ()");
@@ -828,7 +853,11 @@ STABLE_TORCH_LIBRARY_IMPL(_C, CUDA, ops) {
TORCH_BOX(&fused_kimi_k3_mla_decode_q_concat_ds_mla_insert));
#ifndef USE_ROCM
ops.impl("minimax_allreduce_rms_qk", TORCH_BOX(&minimax_allreduce_rms_qk));
ops.impl("fused_qknorm_rope_cache",
TORCH_BOX(&vllm::fused_qknorm_rope_cache));
#endif
ops.impl("fused_muse_spark_qknorm_rope_kv_insert",
TORCH_BOX(&fused_muse_spark_qknorm_rope_kv_insert));
ops.impl("fused_minimax_m3_qknorm_rope_kv_insert",
TORCH_BOX(&fused_minimax_m3_qknorm_rope_kv_insert));
#ifdef VLLM_ENABLE_FUSED_KDA_DECODE
@@ -864,6 +893,8 @@ STABLE_TORCH_LIBRARY_IMPL(_C, CUDA, ops) {
TORCH_BOX(&persistent_masked_m_silu_mul_quant));
ops.impl("weak_ref_tensor", TORCH_BOX(&weak_ref_tensor));
ops.impl("silu_and_mul_quant", TORCH_BOX(&silu_and_mul_quant));
ops.impl("muse_spark_mul_tanh_soft_cap",
TORCH_BOX(&muse_spark_mul_tanh_soft_cap));
ops.impl("silu_and_mul", TORCH_BOX(&silu_and_mul));
ops.impl("mul_and_silu", TORCH_BOX(&mul_and_silu));
ops.impl("gelu_and_mul", TORCH_BOX(&gelu_and_mul));
docker/Dockerfile.rock_base
-453
View File
@@ -1,453 +0,0 @@
ARG BASE_IMAGE=ubuntu:22.04
ARG PYTHON_VERSION=3.12
ARG TORCH_VERSION=2.12.0+rocm10.0.0
ARG TORCHVISION_VERSION=0.27.0+rocm10.0.0
ARG TORCHAUDIO_VERSION=2.11.0+rocm10.0.0
ARG TRITON_BRANCH="669b31ac" # release/internal/3.8.x as of 09/29/2026
ARG TRITON_VERSION=3.8.0+git669b31ac
ARG TRITON_REPO="https://github.com/ROCm/triton.git"
ARG FA_BRANCH="0e60e394"
ARG FA_REPO="https://github.com/Dao-AILab/flash-attention.git"
ARG AITER_BRANCH="v0.1.25"
ARG AITER_REPO="https://github.com/ROCm/aiter.git"
ARG MORI_BRANCH="v1.2.3.post1"
ARG MORI_REPO="https://github.com/ROCm/mori.git"
ARG ROCM_SYSTEMS_REPO="https://github.com/ROCm/rocm-systems.git"
ARG ROCM_RUNTIME_COMMIT="25e14349f2606a6b95c4ccd45eb92a1835646cf2"
# TheRock configuration
ARG SITE_PACKAGES=/usr/local/lib/python${PYTHON_VERSION}/dist-packages
ARG ROCM_RELEASE_WHEELS_MULTIARCH_URL="https://stable.repo.amd.com/rocm/whl-next/" # For pre-built GPU images, packages, etc.
ARG ROCM_NIGHTLY_WHEELS_MULTIARCH_URL="https://nightly.repo.amd.com/rocm/whl-next/" # For torch, triton, etc. wheels with latest fixes.
ARG ROCM_SDK_VERSION=10.0.0
# Build configuration
ARG PYTORCH_ROCM_ARCH="gfx90a;gfx942;gfx950;gfx1100;gfx1101;gfx1200;gfx1201;gfx1150;gfx1151"
ARG AITER_ROCM_ARCH="gfx942;gfx950"
# Sccache configuration (only used in release pipeline)
ARG USE_SCCACHE
ARG SCCACHE_DOWNLOAD_URL
ARG SCCACHE_ENDPOINT
ARG SCCACHE_BUCKET_NAME=vllm-build-sccache
ARG SCCACHE_REGION_NAME=us-west-2
ARG SCCACHE_S3_NO_CREDENTIALS=0
FROM ${BASE_IMAGE} AS base
ARG PYTORCH_ROCM_ARCH
ENV PYTORCH_ROCM_ARCH="${PYTORCH_ROCM_ARCH}"
ARG AITER_ROCM_ARCH
ENV AITER_ROCM_ARCH="${AITER_ROCM_ARCH}"
ARG SITE_PACKAGES
ENV SITE_PACKAGES=${SITE_PACKAGES}
ENV SDK_DEV="$SITE_PACKAGES/_rocm_sdk_devel" \
SDK_CORE="$SITE_PACKAGES/_rocm_sdk_core"
ENV ROCM_PATH="$SDK_DEV" \
ROCM_HOME="$SDK_DEV" \
HIP_PATH="$SDK_DEV" \
HIP_CLANG_PATH="$SDK_DEV/lib/llvm/bin" \
HIP_DEVICE_LIB_PATH="$SDK_CORE/lib/llvm/amdgcn/bitcode"
ENV LD_LIBRARY_PATH="$SDK_DEV/lib:$SDK_DEV/lib/rocm_sysdeps/lib:$SDK_CORE/lib:$SDK_CORE/lib/rocm_sysdeps/lib"
ENV PATH="$SDK_DEV/bin:$SDK_DEV/lib/llvm/bin:$SDK_CORE/bin:$PATH"
# Required for RCCL in ROCm7.1
ENV HSA_NO_SCRATCH_RECLAIM=1
ARG PYTHON_VERSION
ENV PYTHON_VERSION=${PYTHON_VERSION}
RUN mkdir -p /app
WORKDIR /app
ENV DEBIAN_FRONTEND=noninteractive
# Install Python and other dependencies
RUN apt-get update -y \
&& apt-get install -y software-properties-common build-essential git curl sudo vim less libgfortran5 libopenmpi-dev libpci-dev liblzma-dev pkg-config libdrm-dev libnuma-dev kmod \
&& for i in 1 2 3; do \
add-apt-repository -y ppa:deadsnakes/ppa && break || \
{ echo "Attempt $i failed, retrying in 5s..."; sleep 5; }; \
done \
&& apt-get update -y \
&& apt-get install -y python${PYTHON_VERSION} python${PYTHON_VERSION}-dev python${PYTHON_VERSION}-venv \
python${PYTHON_VERSION}-lib2to3 python-is-python3 \
&& update-alternatives --install /usr/bin/python3 python3 /usr/bin/python${PYTHON_VERSION} 1 \
&& update-alternatives --set python3 /usr/bin/python${PYTHON_VERSION} \
&& ln -sf /usr/bin/python${PYTHON_VERSION}-config /usr/bin/python3-config \
&& curl -sS https://bootstrap.pypa.io/get-pip.py | python${PYTHON_VERSION} \
&& python3 --version && python3 -m pip --version
RUN pip install -U packaging 'cmake<4' ninja wheel 'setuptools<80' pybind11==3.0.4 Cython
RUN apt-get update && apt-get install -y libjpeg-dev libsox-dev libsox-fmt-all sox && rm -rf /var/lib/apt/lists/*
# Install ROCm SDK via TheRock
ARG ROCM_RELEASE_WHEELS_MULTIARCH_URL
ARG ROCM_NIGHTLY_WHEELS_MULTIARCH_URL
ARG ROCM_SDK_VERSION
RUN pip install --index-url ${ROCM_RELEASE_WHEELS_MULTIARCH_URL} \
--extra-index-url ${ROCM_NIGHTLY_WHEELS_MULTIARCH_URL} \
"rocm[libraries,devel]${ROCM_SDK_VERSION:+==${ROCM_SDK_VERSION}}" \
&& rocm-sdk init
# Install PyTorch wheels (nightly for latest torch, release as fallback)
ARG TORCH_VERSION
ARG TORCHAUDIO_VERSION
ARG TORCHVISION_VERSION
RUN pip install \
--index-url ${ROCM_RELEASE_WHEELS_MULTIARCH_URL} \
--extra-index-url ${ROCM_NIGHTLY_WHEELS_MULTIARCH_URL} \
"torch==${TORCH_VERSION}" \
"torchaudio==${TORCHAUDIO_VERSION}" \
"torchvision==${TORCHVISION_VERSION}"
# Install the AMD-compiled device kernels corresponding to PYTORCH_ROCM_ARCH:
# Possible values that can be in PYTORCH_ROCM_ARCH:
#
# gfx90a (MI210/250), gfx942 (MI300), gfx950 (MI350), gfx1100 (7900XTX),
# gfx1101 (7800XT), gfx1200 (9070XT), gfx1201 (9070), gfx1150/1151 (Strix APU)
# For each arch in PYTORCH_ROCM_ARCH, install the precompiled GPU code that
# PyTorch needs to run on that specific chip.
SHELL ["/bin/bash", "-c"]
RUN TORCH_VER=$(python3 -c "import torch; print(torch.__version__)") \
&& for arch in $(echo "${PYTORCH_ROCM_ARCH}" | tr ';' ' '); do \
echo "Installing device packages for ${arch}" \
&& pip install --index-url ${ROCM_RELEASE_WHEELS_MULTIARCH_URL} \
--extra-index-url ${ROCM_NIGHTLY_WHEELS_MULTIARCH_URL} \
amd-torch-device-${arch}==${TORCH_VER} \
rocm-sdk-device-${arch}==${ROCM_SDK_VERSION}; \
done
# Remove gfx90a xnack kpacks (ASAN artifacts that cause hipErrorInvalidImage on MI250)
RUN rm -f "${SITE_PACKAGES}/torch/.kpack/torch_gfx90a:xnack+.kpack" \
"${SITE_PACKAGES}/torch/.kpack/torch_gfx90a:xnack-.kpack"
RUN cd $SITE_PACKAGES/_rocm_sdk_core/share/amd_smi && pip install .
ENV PYTHONPATH=$SITE_PACKAGES/_rocm_sdk_core/share/amd_smi
# TheRock runtime fixes
RUN if [ -f "${SITE_PACKAGES}/rocm_sdk/__init__.py" ]; then \
sed -i 's/rtld_global: bool = True/rtld_global: bool = False/g' "${SITE_PACKAGES}/rocm_sdk/__init__.py"; \
fi \
&& if [ -f "${SITE_PACKAGES}/_rocm_sdk_core/lib/librocm_smi64.so.1" ] \
&& [ ! -e "${ROCM_PATH}/lib/librocm_smi64.so.7" ]; then \
ln -sfn "${SITE_PACKAGES}/_rocm_sdk_core/lib/librocm_smi64.so.1" "${ROCM_PATH}/lib/librocm_smi64.so.7"; \
fi \
&& printf '%s\n%s\n' "${ROCM_PATH}/lib" "${SITE_PACKAGES}/_rocm_sdk_core/lib" > /etc/ld.so.conf.d/rocm-pip.conf \
&& for f in ${SITE_PACKAGES}/_rocm_sdk_core/lib/libamdhip64.so*; do \
bn=$(basename "$f"); \
if [ -f "${ROCM_PATH}/lib/${bn}" ] && [ -f "$f" ] && [ ! -L "$f" ]; then \
rm -f "$f"; \
ln -sfn "${ROCM_PATH}/lib/${bn}" "$f"; \
fi; \
done \
&& for f in librocprofiler-sdk.so.1 librocprofiler-register.so.0 \
librocprofiler-sdk-roctx.so.1; do \
if [ -e "${SITE_PACKAGES}/_rocm_sdk_core/lib/${f}" ] \
&& [ -e "${SITE_PACKAGES}/_rocm_sdk_devel/lib/${f}" ]; then \
rm -f "${SITE_PACKAGES}/_rocm_sdk_core/lib/${f}"; \
ln -sfn "${SITE_PACKAGES}/_rocm_sdk_devel/lib/${f}" \
"${SITE_PACKAGES}/_rocm_sdk_core/lib/${f}"; \
fi; \
done \
&& ldconfig
# Install sccache if USE_SCCACHE is enabled (for release builds)
ARG USE_SCCACHE
ARG SCCACHE_DOWNLOAD_URL
ARG SCCACHE_ENDPOINT
ARG SCCACHE_BUCKET_NAME
ARG SCCACHE_REGION_NAME
ARG SCCACHE_S3_NO_CREDENTIALS
RUN if [ "$USE_SCCACHE" = "1" ]; then \
echo "Installing sccache..." \
&& SCCACHE_ARCH="x86_64" \
&& SCCACHE_VERSION="v0.8.1" \
&& SCCACHE_DL_URL="${SCCACHE_DOWNLOAD_URL:-https://github.com/mozilla/sccache/releases/download/${SCCACHE_VERSION}/sccache-${SCCACHE_VERSION}-${SCCACHE_ARCH}-unknown-linux-musl.tar.gz}" \
&& curl -L -o /tmp/sccache.tar.gz ${SCCACHE_DL_URL} \
&& tar -xzf /tmp/sccache.tar.gz -C /tmp \
&& mv /tmp/sccache-${SCCACHE_VERSION}-${SCCACHE_ARCH}-unknown-linux-musl/sccache /usr/bin/sccache \
&& chmod +x /usr/bin/sccache \
&& rm -rf /tmp/sccache.tar.gz /tmp/sccache-${SCCACHE_VERSION}-${SCCACHE_ARCH}-unknown-linux-musl \
&& sccache --version; \
fi
# Setup sccache for HIP compilation via HIP_CLANG_PATH
# This creates wrapper scripts in a separate directory and points HIP to use them
# This avoids modifying the original ROCm binaries which can break detection
# NOTE: HIP_CLANG_PATH is NOT set as ENV to avoid affecting downstream images (Dockerfile.rocm)
# Instead, each build stage should export HIP_CLANG_PATH=/opt/sccache-wrappers if USE_SCCACHE=1
RUN if [ "$USE_SCCACHE" = "1" ]; then \
echo "Setting up sccache wrappers for HIP compilation..." \
&& mkdir -p /opt/sccache-wrappers \
&& printf '#!/bin/bash\nexec sccache %s/lib/llvm/bin/clang++ "$@"\n' "${ROCM_PATH}" > /opt/sccache-wrappers/clang++ \
&& chmod +x /opt/sccache-wrappers/clang++ \
&& printf '#!/bin/bash\nexec sccache %s/lib/llvm/bin/clang "$@"\n' "${ROCM_PATH}" > /opt/sccache-wrappers/clang \
&& chmod +x /opt/sccache-wrappers/clang \
&& echo "sccache wrappers created in /opt/sccache-wrappers"; \
fi
ENV SCCACHE_BUCKET=${USE_SCCACHE:+${SCCACHE_BUCKET_NAME}}
ENV SCCACHE_REGION=${USE_SCCACHE:+${SCCACHE_REGION_NAME}}
ENV SCCACHE_S3_NO_CREDENTIALS=${USE_SCCACHE:+${SCCACHE_S3_NO_CREDENTIALS}}
ENV SCCACHE_IDLE_TIMEOUT=${USE_SCCACHE:+0}
###
### MORI Build
###
FROM base AS build_mori
ARG MORI_BRANCH
ARG MORI_REPO
RUN git clone ${MORI_REPO}
RUN cd mori \
&& git checkout ${MORI_BRANCH} \
&& git submodule update --init --recursive \
&& ln -sfn ${ROCM_PATH} /opt/rocm \
&& pip install "setuptools_scm>=6.2" \
&& CMAKE_PREFIX_PATH="${ROCM_PATH}:${ROCM_PATH}/lib/rocm_sysdeps" \
python3 setup.py bdist_wheel --dist-dir=dist && ls /app/mori/dist/*.whl
RUN mkdir -p /app/install && cp /app/mori/dist/*.whl /app/install
###
### FlashAttention Build
###
FROM base AS build_fa
ARG FA_BRANCH
ARG FA_REPO
ARG USE_SCCACHE
RUN git clone ${FA_REPO}
RUN cd flash-attention \
&& git checkout ${FA_BRANCH} \
&& git submodule update --init \
&& if [ "$USE_SCCACHE" = "1" ]; then \
export HIP_CLANG_PATH=/opt/sccache-wrappers \
&& sccache --show-stats; \
fi \
&& GPU_ARCHS=$(echo ${PYTORCH_ROCM_ARCH} | sed -e 's/;gfx1[0-9]\{3\}//g') python3 setup.py bdist_wheel --dist-dir=dist \
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi
RUN mkdir -p /app/install && cp /app/flash-attention/dist/*.whl /app/install
###
### AITER Build
###
FROM base AS build_aiter
ARG AITER_BRANCH
ARG AITER_REPO
ARG USE_SCCACHE
RUN git clone --recursive --branch ${AITER_BRANCH} ${AITER_REPO}
RUN cd aiter \
&& git submodule update --init --recursive \
&& pip install -r requirements.txt
# AITER_USE_SYSTEM_TRITON keeps setup.py from resolving its own triton, which would
# displace the pinned one already installed in base.
RUN pip install pyyaml && cd aiter \
&& if [ "$USE_SCCACHE" = "1" ]; then \
export HIP_CLANG_PATH=/opt/sccache-wrappers \
&& sccache --show-stats; \
fi \
&& PREBUILD_KERNELS=1 AITER_USE_SYSTEM_TRITON=1 GPU_ARCHS=${AITER_ROCM_ARCH} python3 setup.py bdist_wheel --dist-dir=dist \
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi \
&& ls /app/aiter/dist/*.whl
RUN mkdir -p /app/install && cp /app/aiter/dist/*.whl /app/install
###
### Triton Build
###
FROM base AS build_triton
ARG TRITON_BRANCH
ARG TRITON_REPO
RUN git clone ${TRITON_REPO}
RUN cd triton \
&& git checkout ${TRITON_BRANCH} \
&& if [ ! -f setup.py ]; then cd python; fi \
&& python3 setup.py bdist_wheel --dist-dir=dist \
&& mkdir -p /app/install && cp dist/*.whl /app/install
###
### ROCr + CLR build
###
# Brings in multi-stream perf improvements; drop after these arrive in an official release
FROM base AS build_rocm_runtime
ARG ROCM_SYSTEMS_REPO
ARG ROCM_RUNTIME_COMMIT
# libgl-dev: rocclr does find_package(OpenGL REQUIRED) unconditionally and the wheel image has
# no GL headers. The link must not pick up a libGL DT_NEEDED, which is asserted after the build.
RUN apt-get update \
&& apt-get install -y --no-install-recommends g++ libdw-dev xxd libgl-dev libglx-dev \
&& rm -rf /var/lib/apt/lists/*
RUN pip install --no-cache-dir CppHeaderParser
RUN git init -q /src && cd /src \
&& git remote add origin ${ROCM_SYSTEMS_REPO} \
&& git config core.sparseCheckout true \
&& git config remote.origin.promisor true \
&& git config remote.origin.partialclonefilter blob:none \
&& git sparse-checkout init --cone \
&& git sparse-checkout set projects/rocr-runtime projects/clr projects/hip shared cmake \
&& git fetch --filter=blob:none --depth 1 origin ${ROCM_RUNTIME_COMMIT} \
&& git checkout -q FETCH_HEAD
# libelf is taken from the SDK's own vendored sysdeps rather than apt -- the wheel image has no
# /usr/include/libelf.h -- so ROCr links against the copy the wheel itself was built against.
ENV SYSDEPS="$SDK_DEV/lib/rocm_sysdeps"
# ROCr first, and installed into $SDK_DEV, because the clr build below needs its headers.
RUN cd /src/projects/rocr-runtime \
&& cmake -B build -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_PREFIX_PATH="$SDK_DEV;$SYSDEPS" \
-DCMAKE_INSTALL_PREFIX="$SDK_DEV" \
-DLIBELF_INCLUDE_DIRS="$SYSDEPS/include" \
-DLIBELF_LIBRARIES="$SYSDEPS/lib/libelf.so" \
&& cmake --build build --parallel 16 \
&& cmake --install build \
&& cmake --install build --prefix /rocr-install --strip
RUN cd /src/projects/clr \
&& cmake -B build -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCLR_BUILD_HIP=ON \
-DCLR_BUILD_OCL=OFF \
-DROCM_KPACK_ENABLED=ON \
-DHIP_COMMON_DIR=/src/projects/hip \
-DROCM_PATH="$SDK_DEV" \
-DCMAKE_PREFIX_PATH="$SDK_DEV;$SYSDEPS" \
-DCMAKE_INSTALL_PREFIX="$SDK_DEV" \
-DHIP_LLVM_ROOT="$SDK_DEV/lib/llvm" \
&& cmake --build build --parallel 16 \
&& cmake --install build --prefix /clr-install --strip
RUN mkdir -p /staging/lib \
&& cp -P /rocr-install/lib/libhsa-runtime64.so* /staging/lib/ \
&& cp -P /clr-install/lib/libamdhip64.so* /staging/lib/ \
&& ls -l /staging/lib
###
### Final Build
###
# Wheel release stage -
# only includes dependencies used by wheel release pipeline
FROM base AS debs_wheel_release
RUN mkdir /app/debs
RUN --mount=type=bind,from=build_triton,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_fa,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_aiter,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
FROM base AS debs
RUN mkdir /app/debs
RUN --mount=type=bind,from=build_triton,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_fa,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_aiter,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_mori,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
FROM base AS final
ARG TRITON_VERSION
# torch 2.12 implicitly attaches rocprofiler-sdk; its queue interposition hangs at HSA
# teardown, so disable it. TODO: drop once ROCm's stock rocprofiler carries the fix.
ENV ROCPROFILER_QUEUE_INTERPOSITION=0
COPY --from=build_rocm_runtime /staging/lib/ ${SDK_DEV}/lib/
# Hard link the patched library onto every stock file name, in both SDK trees, so that the
# SONAME lookup and the hard coded paths all resolve to it. Hard links rather than symlinks
# because ldconfig rewrites SONAME symlinks but cannot re-point an inode. $SDK_CORE keeps its
# own libhsa-runtime64 copy, which is on LD_LIBRARY_PATH and in ld.so.conf; relinking it is
# belt and braces rather than strictly required, but it leaves no stock ROCr in the image.
RUN set -eux; \
cd "$SDK_DEV/lib"; \
ours_hsa="$(readlink -f libhsa-runtime64.so.1)"; \
ours_hip="$(readlink -f libamdhip64.so.7)"; \
for f in "$SDK_DEV"/lib/libhsa-runtime64.so.1* "$SDK_CORE"/lib/libhsa-runtime64.so.1* \
"$SDK_DEV"/lib/libamdhip64.so.7* "$SDK_CORE"/lib/libamdhip64.so.7*; do \
[ -f "$f" ] && [ ! -L "$f" ] || continue; \
case "${f##*/}" in libhsa*) ours="$ours_hsa";; *) ours="$ours_hip";; esac; \
[ "$f" -ef "$ours" ] || ln -f "$ours" "$f"; \
done
RUN ldconfig
# This is to prevent resolvers from checking METADATA and RECORD and clobbering
# the Triton source build we installed. Constraints files are unable to help
# with this. However, these entries are optional and can be blanked which
# solves the problem.
RUN TORCH_DIST_INFO=$(ls -d ${SITE_PACKAGES}/torch-*.dist-info) \
&& grep -q '^Requires-Dist: triton' "$TORCH_DIST_INFO/METADATA" \
&& sed -i '/^Requires-Dist: triton/d' "$TORCH_DIST_INFO/METADATA" \
&& sed -i "s|^\(${TORCH_DIST_INFO##*/}/METADATA\),.*|\1,,|" "$TORCH_DIST_INFO/RECORD"
RUN --mount=type=bind,from=debs,src=/app/debs,target=/install \
pip install /install/*.whl \
&& ! pip check 2>&1 | grep -iE '^(torch|triton) ' \
&& test "$(pip show triton | awk '/^Version:/ {print $2}')" = "${TRITON_VERSION}" \
# Create a constraints file to protect the versions we need, especially
# triton, requirements will try to reinstall repeatedly.
&& pip list --format=freeze \
| grep -E '^(torch|triton|torchaudio|torchvision|pybind11)==' > /etc/rocm-constraints.txt \
&& echo "ROCm constraints:" && cat /etc/rocm-constraints.txt \
&& test "$(wc -l < /etc/rocm-constraints.txt)" -eq 5 \
&& grep -q '^pybind11==3\.0\.' /etc/rocm-constraints.txt \
&& python3 -c "import torch; assert torch.version.hip, f'non-ROCm torch: {torch.__version__}'; print(f'torch={torch.__version__}')" \
&& python3 -c "import triton; print(f'triton={triton.__version__}')" \
&& pip list 2>/dev/null | grep -iE '^(torch|triton) '
ARG BASE_IMAGE
ARG PYTHON_VERSION
ARG ROCM_SDK_VERSION
ARG TORCH_VERSION
ARG TORCHAUDIO_VERSION
ARG TORCHVISION_VERSION
ARG PYTORCH_ROCM_ARCH
ARG AITER_ROCM_ARCH
ARG TRITON_BRANCH
ARG TRITON_REPO
ARG FA_BRANCH
ARG FA_REPO
ARG AITER_BRANCH
ARG AITER_REPO
ARG MORI_BRANCH
ARG MORI_REPO
ARG ROCM_SYSTEMS_REPO
ARG ROCM_RUNTIME_COMMIT
ARG TRITON_VERSION
# Record hashes for Triton and pybind11 so we can verify later.
RUN echo "BASE_IMAGE: ${BASE_IMAGE}" > /app/versions.txt \
&& echo "PYTHON_VERSION: ${PYTHON_VERSION}" >> /app/versions.txt \
&& echo "ROCM_SDK_VERSION: ${ROCM_SDK_VERSION}" >> /app/versions.txt \
&& echo "TORCH_VERSION: ${TORCH_VERSION}" >> /app/versions.txt \
&& echo "TORCHAUDIO_VERSION: ${TORCHAUDIO_VERSION}" >> /app/versions.txt \
&& echo "TORCHVISION_VERSION: ${TORCHVISION_VERSION}" >> /app/versions.txt \
&& echo "PYTORCH_ROCM_ARCH: ${PYTORCH_ROCM_ARCH}" >> /app/versions.txt \
&& echo "AITER_ROCM_ARCH: ${AITER_ROCM_ARCH}" >> /app/versions.txt \
&& echo "TRITON_BRANCH: ${TRITON_BRANCH}" >> /app/versions.txt \
&& echo "TRITON_REPO: ${TRITON_REPO}" >> /app/versions.txt \
&& echo "FA_BRANCH: ${FA_BRANCH}" >> /app/versions.txt \
&& echo "FA_REPO: ${FA_REPO}" >> /app/versions.txt \
&& echo "AITER_BRANCH: ${AITER_BRANCH}" >> /app/versions.txt \
&& echo "AITER_REPO: ${AITER_REPO}" >> /app/versions.txt \
&& echo "MORI_BRANCH: ${MORI_BRANCH}" >> /app/versions.txt \
&& echo "MORI_REPO: ${MORI_REPO}" >> /app/versions.txt \
&& echo "ROCM_SYSTEMS_REPO: ${ROCM_SYSTEMS_REPO}" >> /app/versions.txt \
&& echo "ROCM_RUNTIME_COMMIT: ${ROCM_RUNTIME_COMMIT}" >> /app/versions.txt \
# Fingerprint our triton so Dockerfile.rocm can prove it survived the build.
&& echo "TRITON_VERSION: $(pip show triton | awk '/^Version:/ {print $2}')" >> /app/versions.txt \
&& echo "TRITON_SHA256: $(find $(python3 -c 'import triton, os; print(os.path.dirname(triton.__file__))') \
-type f \( -name '*.py' -o -name '*.so' \) | sort | xargs sha256sum | sha256sum | cut -d' ' -f1)" \
>> /app/versions.txt \
&& echo "PYBIND11_VERSION: $(pip show pybind11 | awk '/^Version:/ {print $2}')" >> /app/versions.txt \
&& echo "PYBIND11_INTERNALS_VERSION: $(grep -rhE '^#[[:space:]]*define PYBIND11_INTERNALS_VERSION' \
$(python3 -c 'import pybind11; print(pybind11.get_include())') | awk '{print $NF}')" \
>> /app/versions.txt \
&& echo "PYBIND11_SHA256: $(find $(python3 -c 'import pybind11; print(pybind11.get_include())') \
-type f -name '*.h' | sort | xargs sha256sum | sha256sum | cut -d' ' -f1)" \
>> /app/versions.txt
docker/Dockerfile.rocm
+41 -41
View File
@@ -280,8 +280,9 @@ COPY --from=fetch_vllm \
${COMMON_WORKDIR}/vllm/requirements/rocm.txt \
requirements/
# Use constraints file to protect triton, pybind11, and torch dependencies.
RUN --mount=type=cache,id=vllm-rocm-uv,target=/root/.cache/uv \
uv pip install --system -r requirements/rocm.txt
uv pip install --system -c /etc/rocm-constraints.txt -r requirements/rocm.txt
# Fetch only the pinned ROCm Triton kernels package. Keeping this before the
# vLLM source COPYs makes the download reusable when native sources change.
@@ -322,8 +323,9 @@ ENV SETUPTOOLS_SCM_PRETEND_VERSION="0.0.0+rocm.csrc.build"
RUN --mount=type=bind,source=pyproject.toml,target=${COMMON_WORKDIR}/vllm/pyproject.toml \
--mount=type=cache,id=vllm-rocm-ccache,target=/root/.cache/ccache \
grep -Fq "set(TRITON_KERNELS_TAG \"${ROCM_TRITON_KERNELS_COMMIT}\")" \
{ grep -Fq "set(TRITON_KERNELS_TAG \"${ROCM_TRITON_KERNELS_COMMIT}\")" \
cmake/external_projects/triton_kernels.cmake \
|| echo "WARNING: Dockerfile.rocm ROCM_TRITON_KERNELS_COMMIT (${ROCM_TRITON_KERNELS_COMMIT}) does not match cmake TRITON_KERNELS_TAG; the Docker build uses the Dockerfile pin, cmake governs only non-Docker builds"; } \
&& export CCACHE_BASEDIR="$PWD" \
&& echo "=== ccache stats before ROCm native build ===" \
&& (ccache --show-stats || true) \
@@ -366,7 +368,8 @@ COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/requirements /requirements
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/benchmarks /benchmarks
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/tests /tests
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/examples /examples
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/tools /tools
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/tools/benchmark_helion_kernels.py /tools/benchmark_helion_kernels.py
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/tools/install_torchcodec_rocm.sh /tools/install_torchcodec_rocm.sh
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile /docker/Dockerfile
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile.cpu /docker/Dockerfile.cpu
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm /docker/
@@ -386,7 +389,6 @@ ARG NIXL_BRANCH="1683cf3b7f3d11674c03c5e861cea22339876c96"
ARG NIXL_REPO="https://github.com/ai-dynamo/nixl.git"
ARG UCX_BRANCH="96e58a16039f6d7d213bc967b8069238742c5194"
ARG UCX_REPO="https://github.com/openucx/ucx.git"
ENV ROCM_PATH=/opt/rocm
ENV UCX_HOME=/usr/local/ucx
ENV NIXL_HOME=/usr/local/nixl
@@ -541,7 +543,7 @@ RUN --mount=type=tmpfs,target=/tmp/lmcache-wheel \
c_ops="$(find /tmp/lmcache-wheel -type f -name 'c_ops*.so')"; \
test "$(find /tmp/lmcache-wheel -type f -name 'c_ops*.so' | wc -l)" -eq 1; \
expected="$(printf '%s' "${LMCACHE_ROCM_ARCH}" | tr ';' '\n' | sort -u)"; \
actual="$(/opt/rocm/llvm/bin/llvm-objdump --offloading "${c_ops}" \
actual="$(llvm-objdump --offloading "${c_ops}" \
| grep -oE 'gfx[0-9a-z]+' | sort -u)"; \
if [ "${actual}" != "${expected}" ]; then \
printf 'LMCache code objects mismatch\nexpected:\n%s\nactual:\n%s\n' \
@@ -549,32 +551,10 @@ RUN --mount=type=tmpfs,target=/tmp/lmcache-wheel \
exit 1; \
fi
# ROCShmem build stage - split from DeepEP so changing DEEPEP_BRANCH does not
# invalidate the slow ROCShmem build.
FROM base AS build_rocshmem
ARG ROCSHMEM_BRANCH="f0acb0c6"
ARG ROCSHMEM_REPO="https://github.com/ROCm/rocm-systems.git"
# DeepEP only supports gfx942 and gfx950; build ROCShmem for the same set so
# it can be linked against DeepEP without arch mismatches.
ARG DEEPEP_ROCM_ARCH="gfx942;gfx950"
ENV ROCM_PATH=/opt/rocm
ENV ROCSHMEM_DIR=/opt/rocshmem
RUN --mount=type=cache,target=/root/.cache/ccache \
git clone --no-checkout --filter=blob:none ${ROCSHMEM_REPO} \
&& cd rocm-systems \
&& git sparse-checkout set --cone projects/rocshmem \
&& git checkout ${ROCSHMEM_BRANCH} \
&& mkdir -p projects/rocshmem/build \
&& cd projects/rocshmem/build \
&& CC="ccache gcc" CXX="ccache g++" INSTALL_PREFIX=${ROCSHMEM_DIR} \
bash ../scripts/build_configs/all_backends \
-DROCM_PATH=${ROCM_PATH} \
-DGPU_TARGETS="${DEEPEP_ROCM_ARCH}" \
-DUSE_EXTERNAL_MPI=OFF
# DeepEP build stage - depends on ROCShmem, builds the HIP kernel wheel.
FROM build_rocshmem AS build_deepep
# DeepEP build stage. ROCShmem is provided by TheRock SDK
# ROCSHMEM_DIR set to the SDK install path so DeepEP finds the headers, etc.
FROM base AS build_deepep
ENV ROCSHMEM_DIR=${ROCM_PATH}
ARG DEEPEP_BRANCH="0f63d3e9"
ARG DEEPEP_REPO="https://github.com/ROCm/DeepEP.git"
ARG DEEPEP_NIC="cx7"
@@ -693,8 +673,9 @@ RUN if [ "$GIT_REPO_CHECK" != "0" ]; then \
# Install the packaging helpers before modifying requirements/rocm.txt.
RUN --mount=type=cache,target=/root/.cache/uv \
grep -Fq "set(TRITON_KERNELS_TAG \"${ROCM_TRITON_KERNELS_COMMIT}\")" \
{ grep -Fq "set(TRITON_KERNELS_TAG \"${ROCM_TRITON_KERNELS_COMMIT}\")" \
vllm/cmake/external_projects/triton_kernels.cmake \
|| echo "WARNING: Dockerfile.rocm ROCM_TRITON_KERNELS_COMMIT (${ROCM_TRITON_KERNELS_COMMIT}) does not match cmake TRITON_KERNELS_TAG; the Docker build uses the Dockerfile pin, cmake governs only non-Docker builds"; } \
&& cd vllm \
&& uv pip install --system setuptools_scm regex
@@ -750,7 +731,8 @@ COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/requirements /requir
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/benchmarks /benchmarks
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/tests /tests
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/examples /examples
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/tools /tools
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/tools/benchmark_helion_kernels.py /tools/benchmark_helion_kernels.py
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/tools/install_torchcodec_rocm.sh /tools/install_torchcodec_rocm.sh
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile /docker/Dockerfile
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile.cpu /docker/Dockerfile.cpu
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm /docker/
@@ -795,9 +777,6 @@ RUN --mount=type=bind,from=build_nixl,src=/app/install,target=/nixl_install \
--mount=type=bind,from=build_deepep,src=/app/deep_install,target=/deep_install \
uv pip install --system /nixl_install/*.whl /deep_install/*.whl
# Copy ROCShmem runtime libraries.
COPY --from=build_rocshmem /opt/rocshmem /opt/rocshmem
# RDMA userspace libraries plus FFmpeg dev libs needed by torchcodec.
RUN timeout --kill-after=30s 5m apt-get update -q -y --error-on=any \
-o Acquire::Retries=3 \
@@ -830,7 +809,8 @@ RUN --mount=type=cache,target=/root/.cache/uv \
# Pre-install shared ROCm runtime dependencies.
COPY requirements/common.txt requirements/rocm.txt /tmp/ci-base-requirements/
RUN --mount=type=cache,target=/root/.cache/uv \
uv pip install --system -r /tmp/ci-base-requirements/rocm.txt \
uv pip install --system -c /etc/rocm-constraints.txt \
-r /tmp/ci-base-requirements/rocm.txt \
&& rm -rf /tmp/ci-base-requirements
# Enable fast and less brittle model downloads in tests.
@@ -908,8 +888,6 @@ RUN mkdir src && mv vllm src/vllm
# Catch GPU<->CPU syncs in execute_model/sample_tokens during tests.
ENV VLLM_GPU_SYNC_CHECK=error
# Run the structural smoke test inside BuildKit, where the test image layers
# are already available. Export only a build-scoped success marker.
FROM test AS test_smoke
ARG ROCM_SMOKE_ID
RUN --network=none \
@@ -956,15 +934,37 @@ RUN --mount=type=bind,from=export_vllm,src=/,target=/install \
--mount=type=cache,target=/root/.cache/uv \
cd /install \
&& echo "opentelemetry-api<=1.40.0" > /tmp/otel-constraints.txt \
&& uv pip install --system -c /tmp/otel-constraints.txt -r requirements/rocm.txt \
&& uv pip install --system -c /etc/rocm-constraints.txt -c /tmp/otel-constraints.txt -r requirements/rocm.txt \
&& pip uninstall -y vllm \
&& uv pip install --system -c /tmp/otel-constraints.txt *.whl \
&& uv pip install --system -c /etc/rocm-constraints.txt -c /tmp/otel-constraints.txt *.whl \
&& rm /tmp/otel-constraints.txt
# Install NIXL ROCm wheel
RUN --mount=type=bind,from=build_nixl,src=/app/install,target=/nixl_install \
uv pip install --system /nixl_install/*.whl
# Recompute the base's triton fingerprint to detect resolver interference.
RUN EXPECTED=$(grep '^TRITON_SHA256: ' /app/versions.txt | cut -d' ' -f2) \
&& ACTUAL=$(find $(python3 -c 'import triton, os; print(os.path.dirname(triton.__file__))') \
-type f \( -name '*.py' -o -name '*.so' \) | sort | xargs sha256sum | sha256sum | cut -d' ' -f1) \
&& echo "triton sha256 expected=${EXPECTED} actual=${ACTUAL}" \
&& test -n "${EXPECTED}" \
&& test "${EXPECTED}" = "${ACTUAL}" \
&& python3 -c "import torch; assert torch.version.hip, f'non-ROCm torch: {torch.__version__}'; print(f'torch={torch.__version__}')"
# Recompute the base's pybind11 fingerprint to detect resolver interference.
RUN EXPECTED=$(grep '^PYBIND11_SHA256: ' /app/versions.txt | cut -d' ' -f2) \
&& EXPECTED_ABI=$(grep '^PYBIND11_INTERNALS_VERSION: ' /app/versions.txt | cut -d' ' -f2) \
&& INC=$(python3 -c 'import pybind11; print(pybind11.get_include())') \
&& ACTUAL=$(find "${INC}" -type f -name '*.h' | sort | xargs sha256sum | sha256sum | cut -d' ' -f1) \
&& ACTUAL_ABI=$(grep -rhE '^#[[:space:]]*define PYBIND11_INTERNALS_VERSION' "${INC}" | awk '{print $NF}') \
&& echo "pybind11 $(pip show pybind11 | awk '/^Version:/ {print $2}')" \
"sha256 expected=${EXPECTED} actual=${ACTUAL}" \
"internals expected=${EXPECTED_ABI} actual=${ACTUAL_ABI}" \
&& test -n "${EXPECTED}" \
&& test "${EXPECTED}" = "${ACTUAL}" \
&& test "${EXPECTED_ABI}" = "${ACTUAL_ABI}"
ARG COMMON_WORKDIR
ARG BASE_IMAGE
ARG NIC_BACKEND
docker/Dockerfile.rock → docker/Dockerfile.rocm_72
+49 -49
View File
@@ -3,7 +3,7 @@ ARG REMOTE_VLLM="0"
ARG VLLM_REPO="https://github.com/vllm-project/vllm.git"
ARG VLLM_BRANCH="main"
ARG COMMON_WORKDIR=/app
ARG BASE_IMAGE=rocm/vllm-dev:base
ARG BASE_IMAGE=vllm/vllm-openai-rocm:base-nightly-rocm72
ARG CI_BASE_IMAGE=rocm/vllm-dev:ci_base
ARG ROCM_TRITON_KERNELS_COMMIT=669b31acc1dd1b3fd93286afd5db67f65d9f7557
ARG INSTALL_LMCACHE=false
@@ -46,7 +46,7 @@ RUN --mount=type=cache,target=/root/.cache/pip \
# Note: mold is installed but not set as the system default linker because
# some packages use JIT compilation at runtime with flags mold does not support.
# Build stages opt in via LDFLAGS="-fuse-ld=mold".
# Remove sccache only if not using sccache (it exists in base image from Dockerfile.rocm_base)
# Remove sccache only if not using sccache (it exists in base image from Dockerfile.rocm_72_base)
ARG USE_SCCACHE
RUN if [ "$USE_SCCACHE" != "1" ]; then \
apt-get purge -y sccache || true; \
@@ -280,9 +280,8 @@ COPY --from=fetch_vllm \
${COMMON_WORKDIR}/vllm/requirements/rocm.txt \
requirements/
# Use constraints file to protect triton, pybind11, and torch dependencies.
RUN --mount=type=cache,id=vllm-rocm-uv,target=/root/.cache/uv \
uv pip install --system -c /etc/rocm-constraints.txt -r requirements/rocm.txt
uv pip install --system -r requirements/rocm.txt
# Fetch only the pinned ROCm Triton kernels package. Keeping this before the
# vLLM source COPYs makes the download reusable when native sources change.
@@ -323,9 +322,8 @@ ENV SETUPTOOLS_SCM_PRETEND_VERSION="0.0.0+rocm.csrc.build"
RUN --mount=type=bind,source=pyproject.toml,target=${COMMON_WORKDIR}/vllm/pyproject.toml \
--mount=type=cache,id=vllm-rocm-ccache,target=/root/.cache/ccache \
{ grep -Fq "set(TRITON_KERNELS_TAG \"${ROCM_TRITON_KERNELS_COMMIT}\")" \
grep -Fq "set(TRITON_KERNELS_TAG \"${ROCM_TRITON_KERNELS_COMMIT}\")" \
cmake/external_projects/triton_kernels.cmake \
|| echo "WARNING: Dockerfile.rock ROCM_TRITON_KERNELS_COMMIT (${ROCM_TRITON_KERNELS_COMMIT}) does not match cmake TRITON_KERNELS_TAG; the Docker build uses the Dockerfile pin, cmake governs only non-Docker builds"; } \
&& export CCACHE_BASEDIR="$PWD" \
&& echo "=== ccache stats before ROCm native build ===" \
&& (ccache --show-stats || true) \
@@ -368,12 +366,11 @@ COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/requirements /requirements
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/benchmarks /benchmarks
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/tests /tests
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/examples /examples
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/tools/benchmark_helion_kernels.py /tools/benchmark_helion_kernels.py
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/tools/install_torchcodec_rocm.sh /tools/install_torchcodec_rocm.sh
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/tools /tools
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile /docker/Dockerfile
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile.cpu /docker/Dockerfile.cpu
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm /docker/
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm_base /docker/Dockerfile.rocm_base
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm_72 /docker/
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm_72_base /docker/
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/ci-rocm.hcl /docker/ci-rocm.hcl
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/docker-bake.hcl /docker/docker-bake.hcl
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/docker-bake-rocm.hcl /docker/docker-bake-rocm.hcl
@@ -389,6 +386,7 @@ ARG NIXL_BRANCH="1683cf3b7f3d11674c03c5e861cea22339876c96"
ARG NIXL_REPO="https://github.com/ai-dynamo/nixl.git"
ARG UCX_BRANCH="96e58a16039f6d7d213bc967b8069238742c5194"
ARG UCX_REPO="https://github.com/openucx/ucx.git"
ENV ROCM_PATH=/opt/rocm
ENV UCX_HOME=/usr/local/ucx
ENV NIXL_HOME=/usr/local/nixl
@@ -543,7 +541,7 @@ RUN --mount=type=tmpfs,target=/tmp/lmcache-wheel \
c_ops="$(find /tmp/lmcache-wheel -type f -name 'c_ops*.so')"; \
test "$(find /tmp/lmcache-wheel -type f -name 'c_ops*.so' | wc -l)" -eq 1; \
expected="$(printf '%s' "${LMCACHE_ROCM_ARCH}" | tr ';' '\n' | sort -u)"; \
actual="$(llvm-objdump --offloading "${c_ops}" \
actual="$(/opt/rocm/llvm/bin/llvm-objdump --offloading "${c_ops}" \
| grep -oE 'gfx[0-9a-z]+' | sort -u)"; \
if [ "${actual}" != "${expected}" ]; then \
printf 'LMCache code objects mismatch\nexpected:\n%s\nactual:\n%s\n' \
@@ -551,10 +549,32 @@ RUN --mount=type=tmpfs,target=/tmp/lmcache-wheel \
exit 1; \
fi
# DeepEP build stage. ROCShmem is provided by TheRock SDK
# ROCSHMEM_DIR set to the SDK install path so DeepEP finds the headers, etc.
FROM base AS build_deepep
ENV ROCSHMEM_DIR=${ROCM_PATH}
# ROCShmem build stage - split from DeepEP so changing DEEPEP_BRANCH does not
# invalidate the slow ROCShmem build.
FROM base AS build_rocshmem
ARG ROCSHMEM_BRANCH="f0acb0c6"
ARG ROCSHMEM_REPO="https://github.com/ROCm/rocm-systems.git"
# DeepEP only supports gfx942 and gfx950; build ROCShmem for the same set so
# it can be linked against DeepEP without arch mismatches.
ARG DEEPEP_ROCM_ARCH="gfx942;gfx950"
ENV ROCM_PATH=/opt/rocm
ENV ROCSHMEM_DIR=/opt/rocshmem
RUN --mount=type=cache,target=/root/.cache/ccache \
git clone --no-checkout --filter=blob:none ${ROCSHMEM_REPO} \
&& cd rocm-systems \
&& git sparse-checkout set --cone projects/rocshmem \
&& git checkout ${ROCSHMEM_BRANCH} \
&& mkdir -p projects/rocshmem/build \
&& cd projects/rocshmem/build \
&& CC="ccache gcc" CXX="ccache g++" INSTALL_PREFIX=${ROCSHMEM_DIR} \
bash ../scripts/build_configs/all_backends \
-DROCM_PATH=${ROCM_PATH} \
-DGPU_TARGETS="${DEEPEP_ROCM_ARCH}" \
-DUSE_EXTERNAL_MPI=OFF
# DeepEP build stage - depends on ROCShmem, builds the HIP kernel wheel.
FROM build_rocshmem AS build_deepep
ARG DEEPEP_BRANCH="0f63d3e9"
ARG DEEPEP_REPO="https://github.com/ROCm/DeepEP.git"
ARG DEEPEP_NIC="cx7"
@@ -571,8 +591,8 @@ RUN --mount=type=cache,target=/root/.cache/ccache \
&& git checkout ${DEEPEP_BRANCH} \
&& LDFLAGS="-fuse-ld=mold" MAX_JOBS="${MAX_JOBS:-$(nproc)}" python3 setup.py --variant rocm --rocm-explicit-ctx --nic ${DEEPEP_NIC} --rocm-gfx942-fp8-fnuz-max 224 bdist_wheel --dist-dir=/app/deep_install
# MoRI runtime dependencies live in Dockerfile.rocm so NIC backend changes do
# not force users to rebuild the long-lived Dockerfile.rocm_base image.
# MoRI runtime dependencies live in Dockerfile.rocm_72 so NIC backend changes do
# not force users to rebuild the long-lived Dockerfile.rocm_72_base image.
FROM base AS mori_base
ARG NIC_BACKEND
ARG AINIC_VERSION
@@ -673,9 +693,8 @@ RUN if [ "$GIT_REPO_CHECK" != "0" ]; then \
# Install the packaging helpers before modifying requirements/rocm.txt.
RUN --mount=type=cache,target=/root/.cache/uv \
{ grep -Fq "set(TRITON_KERNELS_TAG \"${ROCM_TRITON_KERNELS_COMMIT}\")" \
grep -Fq "set(TRITON_KERNELS_TAG \"${ROCM_TRITON_KERNELS_COMMIT}\")" \
vllm/cmake/external_projects/triton_kernels.cmake \
|| echo "WARNING: Dockerfile.rock ROCM_TRITON_KERNELS_COMMIT (${ROCM_TRITON_KERNELS_COMMIT}) does not match cmake TRITON_KERNELS_TAG; the Docker build uses the Dockerfile pin, cmake governs only non-Docker builds"; } \
&& cd vllm \
&& uv pip install --system setuptools_scm regex
@@ -731,12 +750,11 @@ COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/requirements /requir
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/benchmarks /benchmarks
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/tests /tests
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/examples /examples
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/tools/benchmark_helion_kernels.py /tools/benchmark_helion_kernels.py
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/tools/install_torchcodec_rocm.sh /tools/install_torchcodec_rocm.sh
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/tools /tools
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile /docker/Dockerfile
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile.cpu /docker/Dockerfile.cpu
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm /docker/
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm_base /docker/Dockerfile.rocm_base
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm_72 /docker/
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm_72_base /docker/
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/ci-rocm.hcl /docker/ci-rocm.hcl
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/docker-bake.hcl /docker/docker-bake.hcl
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/docker-bake-rocm.hcl /docker/docker-bake-rocm.hcl
@@ -777,6 +795,9 @@ RUN --mount=type=bind,from=build_nixl,src=/app/install,target=/nixl_install \
--mount=type=bind,from=build_deepep,src=/app/deep_install,target=/deep_install \
uv pip install --system /nixl_install/*.whl /deep_install/*.whl
# Copy ROCShmem runtime libraries.
COPY --from=build_rocshmem /opt/rocshmem /opt/rocshmem
# RDMA userspace libraries plus FFmpeg dev libs needed by torchcodec.
RUN timeout --kill-after=30s 5m apt-get update -q -y --error-on=any \
-o Acquire::Retries=3 \
@@ -809,8 +830,7 @@ RUN --mount=type=cache,target=/root/.cache/uv \
# Pre-install shared ROCm runtime dependencies.
COPY requirements/common.txt requirements/rocm.txt /tmp/ci-base-requirements/
RUN --mount=type=cache,target=/root/.cache/uv \
uv pip install --system -c /etc/rocm-constraints.txt \
-r /tmp/ci-base-requirements/rocm.txt \
uv pip install --system -r /tmp/ci-base-requirements/rocm.txt \
&& rm -rf /tmp/ci-base-requirements
# Enable fast and less brittle model downloads in tests.
@@ -888,6 +908,8 @@ RUN mkdir src && mv vllm src/vllm
# Catch GPU<->CPU syncs in execute_model/sample_tokens during tests.
ENV VLLM_GPU_SYNC_CHECK=error
# Run the structural smoke test inside BuildKit, where the test image layers
# are already available. Export only a build-scoped success marker.
FROM test AS test_smoke
ARG ROCM_SMOKE_ID
RUN --network=none \
@@ -934,37 +956,15 @@ RUN --mount=type=bind,from=export_vllm,src=/,target=/install \
--mount=type=cache,target=/root/.cache/uv \
cd /install \
&& echo "opentelemetry-api<=1.40.0" > /tmp/otel-constraints.txt \
&& uv pip install --system -c /etc/rocm-constraints.txt -c /tmp/otel-constraints.txt -r requirements/rocm.txt \
&& uv pip install --system -c /tmp/otel-constraints.txt -r requirements/rocm.txt \
&& pip uninstall -y vllm \
&& uv pip install --system -c /etc/rocm-constraints.txt -c /tmp/otel-constraints.txt *.whl \
&& uv pip install --system -c /tmp/otel-constraints.txt *.whl \
&& rm /tmp/otel-constraints.txt
# Install NIXL ROCm wheel
RUN --mount=type=bind,from=build_nixl,src=/app/install,target=/nixl_install \
uv pip install --system /nixl_install/*.whl
# Recompute the base's triton fingerprint to detect resolver interference.
RUN EXPECTED=$(grep '^TRITON_SHA256: ' /app/versions.txt | cut -d' ' -f2) \
&& ACTUAL=$(find $(python3 -c 'import triton, os; print(os.path.dirname(triton.__file__))') \
-type f \( -name '*.py' -o -name '*.so' \) | sort | xargs sha256sum | sha256sum | cut -d' ' -f1) \
&& echo "triton sha256 expected=${EXPECTED} actual=${ACTUAL}" \
&& test -n "${EXPECTED}" \
&& test "${EXPECTED}" = "${ACTUAL}" \
&& python3 -c "import torch; assert torch.version.hip, f'non-ROCm torch: {torch.__version__}'; print(f'torch={torch.__version__}')"
# Recompute the base's pybind11 fingerprint to detect resolver interference.
RUN EXPECTED=$(grep '^PYBIND11_SHA256: ' /app/versions.txt | cut -d' ' -f2) \
&& EXPECTED_ABI=$(grep '^PYBIND11_INTERNALS_VERSION: ' /app/versions.txt | cut -d' ' -f2) \
&& INC=$(python3 -c 'import pybind11; print(pybind11.get_include())') \
&& ACTUAL=$(find "${INC}" -type f -name '*.h' | sort | xargs sha256sum | sha256sum | cut -d' ' -f1) \
&& ACTUAL_ABI=$(grep -rhE '^#[[:space:]]*define PYBIND11_INTERNALS_VERSION' "${INC}" | awk '{print $NF}') \
&& echo "pybind11 $(pip show pybind11 | awk '/^Version:/ {print $2}')" \
"sha256 expected=${EXPECTED} actual=${ACTUAL}" \
"internals expected=${EXPECTED_ABI} actual=${ACTUAL_ABI}" \
&& test -n "${EXPECTED}" \
&& test "${EXPECTED}" = "${ACTUAL}" \
&& test "${EXPECTED_ABI}" = "${ACTUAL_ABI}"
ARG COMMON_WORKDIR
ARG BASE_IMAGE
ARG NIC_BACKEND
docker/Dockerfile.rocm_72_base
+479
View File
@@ -0,0 +1,479 @@
ARG BASE_IMAGE=rocm/dev-ubuntu-22.04:7.2.3-complete
ARG TRITON_BRANCH="669b31a" # release/internal/3.8.x as of 09/17
ARG TRITON_REPO="https://github.com/ROCm/triton.git"
ARG PYTORCH_BRANCH="733fca1" # release/2.13 as of 09/17
ARG PYTORCH_REPO="https://github.com/ROCm/pytorch.git"
ARG PYTORCH_VISION_BRANCH="v0.28.0"
ARG PYTORCH_VISION_REPO="https://github.com/pytorch/vision.git"
ARG PYTORCH_AUDIO_BRANCH="v2.11.0"
ARG PYTORCH_AUDIO_REPO="https://github.com/pytorch/audio.git"
ARG FA_BRANCH="0e60e394"
ARG FA_REPO="https://github.com/Dao-AILab/flash-attention.git"
ARG AITER_BRANCH="v0.1.25"
ARG AITER_REPO="https://github.com/ROCm/aiter.git"
ARG MORI_BRANCH="v1.2.3.post1"
ARG MORI_REPO="https://github.com/ROCm/mori.git"
ARG ROCM_SYSTEMS_REPO="https://github.com/ROCm/rocm-systems.git"
ARG ROCPROFILER_SDK_COMMIT="2b22ab0195cc1461cd9abf3b969e9dd7c10af350"
ARG ROCM_RUNTIME_COMMIT="af70edc91f1dc5d228fd4c076df99e65e49c4521"
# Sccache configuration (only used in release pipeline)
ARG USE_SCCACHE
ARG SCCACHE_DOWNLOAD_URL
ARG SCCACHE_ENDPOINT
ARG SCCACHE_BUCKET_NAME=vllm-build-sccache
ARG SCCACHE_REGION_NAME=us-west-2
ARG SCCACHE_S3_NO_CREDENTIALS=0
FROM ${BASE_IMAGE} AS base
ENV PATH=/opt/rocm/llvm/bin:/opt/rocm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ENV ROCM_PATH=/opt/rocm
ENV LD_LIBRARY_PATH=/opt/rocm/lib:/usr/local/lib:
ARG PYTORCH_ROCM_ARCH=gfx90a;gfx942;gfx950;gfx1100;gfx1101;gfx1200;gfx1201;gfx1150;gfx1151
ENV PYTORCH_ROCM_ARCH=${PYTORCH_ROCM_ARCH}
ENV AITER_ROCM_ARCH=gfx942;gfx950
# Note: Do not set MORI_GPU_ARCHS here, it is automatically inferred at runtime
# Required for RCCL in ROCm7.1
ENV HSA_NO_SCRATCH_RECLAIM=1
ARG PYTHON_VERSION=3.12
ENV PYTHON_VERSION=${PYTHON_VERSION}
RUN mkdir -p /app
WORKDIR /app
ENV DEBIAN_FRONTEND=noninteractive
# Install Python and other dependencies (plus gcc-13, selected later in build_pytorch).
RUN apt-get update -y \
&& apt-get install -y software-properties-common git curl sudo vim less libgfortran5 libopenmpi-dev libpci-dev liblzma-dev pkg-config \
&& for i in 1 2 3; do \
add-apt-repository -y ppa:deadsnakes/ppa && add-apt-repository -y ppa:ubuntu-toolchain-r/test && break || \
{ echo "Attempt $i failed, retrying in 5s..."; sleep 5; }; \
done \
&& apt-get update -y \
&& apt-get install -y python${PYTHON_VERSION} python${PYTHON_VERSION}-dev python${PYTHON_VERSION}-venv \
python${PYTHON_VERSION}-lib2to3 python-is-python3 gcc-13 g++-13 \
&& update-alternatives --install /usr/bin/python3 python3 /usr/bin/python${PYTHON_VERSION} 1 \
&& update-alternatives --set python3 /usr/bin/python${PYTHON_VERSION} \
&& ln -sf /usr/bin/python${PYTHON_VERSION}-config /usr/bin/python3-config \
&& curl -sS https://bootstrap.pypa.io/get-pip.py | python${PYTHON_VERSION} \
&& python3 --version && python3 -m pip --version
RUN pip install -U packaging 'cmake<4' ninja wheel 'setuptools<80' 'pybind11<3.1.0' Cython
RUN apt-get update && apt-get install -y libjpeg-dev libsox-dev libsox-fmt-all sox && rm -rf /var/lib/apt/lists/*
# Install sccache if USE_SCCACHE is enabled (for release builds)
ARG USE_SCCACHE
ARG SCCACHE_DOWNLOAD_URL
ARG SCCACHE_ENDPOINT
ARG SCCACHE_BUCKET_NAME
ARG SCCACHE_REGION_NAME
ARG SCCACHE_S3_NO_CREDENTIALS
RUN if [ "$USE_SCCACHE" = "1" ]; then \
echo "Installing sccache..." \
&& SCCACHE_ARCH="x86_64" \
&& SCCACHE_VERSION="v0.8.1" \
&& SCCACHE_DL_URL="${SCCACHE_DOWNLOAD_URL:-https://github.com/mozilla/sccache/releases/download/${SCCACHE_VERSION}/sccache-${SCCACHE_VERSION}-${SCCACHE_ARCH}-unknown-linux-musl.tar.gz}" \
&& curl -L -o /tmp/sccache.tar.gz ${SCCACHE_DL_URL} \
&& tar -xzf /tmp/sccache.tar.gz -C /tmp \
&& mv /tmp/sccache-${SCCACHE_VERSION}-${SCCACHE_ARCH}-unknown-linux-musl/sccache /usr/bin/sccache \
&& chmod +x /usr/bin/sccache \
&& rm -rf /tmp/sccache.tar.gz /tmp/sccache-${SCCACHE_VERSION}-${SCCACHE_ARCH}-unknown-linux-musl \
&& sccache --version; \
fi
# Setup sccache for HIP compilation via HIP_CLANG_PATH
# This creates wrapper scripts in a separate directory and points HIP to use them
# This avoids modifying the original ROCm binaries which can break detection
# NOTE: HIP_CLANG_PATH is NOT set as ENV to avoid affecting downstream images (Dockerfile.rocm_72)
# Instead, each build stage should export HIP_CLANG_PATH=/opt/sccache-wrappers if USE_SCCACHE=1
RUN if [ "$USE_SCCACHE" = "1" ]; then \
echo "Setting up sccache wrappers for HIP compilation..." \
&& mkdir -p /opt/sccache-wrappers \
&& printf '#!/bin/bash\nexec sccache /opt/rocm/lib/llvm/bin/clang++ "$@"\n' > /opt/sccache-wrappers/clang++ \
&& chmod +x /opt/sccache-wrappers/clang++ \
&& printf '#!/bin/bash\nexec sccache /opt/rocm/lib/llvm/bin/clang "$@"\n' > /opt/sccache-wrappers/clang \
&& chmod +x /opt/sccache-wrappers/clang \
&& echo "sccache wrappers created in /opt/sccache-wrappers"; \
fi
# Set sccache environment variables only when USE_SCCACHE=1
# This prevents S3 config from leaking into images when sccache is not used
ARG USE_SCCACHE
ENV SCCACHE_BUCKET=${USE_SCCACHE:+${SCCACHE_BUCKET_NAME}}
ENV SCCACHE_REGION=${USE_SCCACHE:+${SCCACHE_REGION_NAME}}
ENV SCCACHE_S3_NO_CREDENTIALS=${USE_SCCACHE:+${SCCACHE_S3_NO_CREDENTIALS}}
ENV SCCACHE_IDLE_TIMEOUT=${USE_SCCACHE:+0}
###
### Triton Build
###
FROM base AS build_triton
ARG TRITON_BRANCH
ARG TRITON_REPO
RUN git clone ${TRITON_REPO}
RUN cd triton \
&& git checkout ${TRITON_BRANCH} \
&& git config --global user.email "you@example.com" && git config --global user.name "Your Name" \
&& if [ ! -f setup.py ]; then cd python; fi \
&& MAX_JOBS=16 python3 setup.py bdist_wheel --dist-dir=dist \
&& mkdir -p /app/install && cp dist/*.whl /app/install
###
### AMD SMI Build
###
FROM base AS build_amdsmi
RUN cd /opt/rocm/share/amd_smi \
&& pip wheel . --wheel-dir=dist
RUN mkdir -p /app/install && cp /opt/rocm/share/amd_smi/dist/*.whl /app/install
###
### Pytorch build
###
FROM base AS build_pytorch
ARG PYTORCH_BRANCH
ARG PYTORCH_REPO
ARG USE_SCCACHE
RUN apt-get update && apt-get install -y pkg-config liblzma-dev
# torch 2.12 emits float->__bf16 libcalls (__truncsfbf2) absent from gcc-11 libgcc; build with gcc-13.
RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-13 130 --slave /usr/bin/g++ g++ /usr/bin/g++-13
RUN git clone ${PYTORCH_REPO} pytorch
RUN cd pytorch && git checkout ${PYTORCH_BRANCH}
RUN cd pytorch \
&& pip install -r requirements.txt && git submodule update --init --recursive
RUN cd pytorch && python3 tools/amd_build/build_amd.py \
&& if [ "$USE_SCCACHE" = "1" ]; then \
export HIP_CLANG_PATH=/opt/sccache-wrappers \
&& export CMAKE_C_COMPILER_LAUNCHER=sccache \
&& export CMAKE_CXX_COMPILER_LAUNCHER=sccache \
&& sccache --show-stats; \
fi \
&& CMAKE_PREFIX_PATH=$(python3 -c 'import sys; print(sys.prefix)') python3 setup.py bdist_wheel --dist-dir=dist \
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi
RUN mkdir -p /app/install && cp /app/pytorch/dist/*.whl /app/install
###
### Shared PyTorch runtime for dependent builds
###
FROM base AS build_pytorch_runtime
RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-13 130 --slave /usr/bin/g++ g++ /usr/bin/g++-13
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install,ro \
pip install /install/*.whl
###
### TorchVision build
###
FROM build_pytorch_runtime AS build_torchvision
ARG PYTORCH_VISION_BRANCH
ARG PYTORCH_VISION_REPO
ARG USE_SCCACHE
RUN git clone ${PYTORCH_VISION_REPO} vision
RUN cd vision && git checkout ${PYTORCH_VISION_BRANCH} \
&& if [ "$USE_SCCACHE" = "1" ]; then \
export HIP_CLANG_PATH=/opt/sccache-wrappers \
&& export CMAKE_C_COMPILER_LAUNCHER=sccache \
&& export CMAKE_CXX_COMPILER_LAUNCHER=sccache; \
fi \
&& python3 setup.py bdist_wheel --dist-dir=dist \
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi
RUN mkdir -p /app/install && cp /app/vision/dist/*.whl /app/install
###
### TorchAudio build
###
FROM build_pytorch_runtime AS build_torchaudio
ARG PYTORCH_AUDIO_BRANCH
ARG PYTORCH_AUDIO_REPO
ARG USE_SCCACHE
RUN git clone ${PYTORCH_AUDIO_REPO} audio
RUN cd audio && git checkout ${PYTORCH_AUDIO_BRANCH} \
&& git submodule update --init --recursive
RUN cd audio && pip install -r requirements.txt
RUN cd audio \
&& if [ "$USE_SCCACHE" = "1" ]; then \
export HIP_CLANG_PATH=/opt/sccache-wrappers \
&& export CMAKE_C_COMPILER_LAUNCHER=sccache \
&& export CMAKE_CXX_COMPILER_LAUNCHER=sccache; \
fi \
&& python3 setup.py bdist_wheel --dist-dir=dist \
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi
RUN mkdir -p /app/install && cp /app/audio/dist/*.whl /app/install
###
### MORI Build
###
FROM base AS build_mori
ARG MORI_BRANCH
ARG MORI_REPO
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
pip install /install/*.whl
RUN git clone ${MORI_REPO}
RUN cd mori \
&& git checkout ${MORI_BRANCH} \
&& git submodule update --init --recursive \
&& python3 setup.py bdist_wheel --dist-dir=dist && ls /app/mori/dist/*.whl
RUN mkdir -p /app/install && cp /app/mori/dist/*.whl /app/install
###
### FlashAttention Build
###
FROM base AS build_fa
ARG FA_BRANCH
ARG FA_REPO
ARG USE_SCCACHE
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
pip install /install/*.whl
RUN git clone ${FA_REPO}
RUN cd flash-attention \
&& git checkout ${FA_BRANCH} \
&& git submodule update --init \
&& if [ "$USE_SCCACHE" = "1" ]; then \
export HIP_CLANG_PATH=/opt/sccache-wrappers \
&& sccache --show-stats; \
fi \
&& GPU_ARCHS=$(echo ${PYTORCH_ROCM_ARCH} | sed -e 's/;gfx1[0-9]\{3\}//g') python3 setup.py bdist_wheel --dist-dir=dist \
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi
RUN mkdir -p /app/install && cp /app/flash-attention/dist/*.whl /app/install
###
### AITER Build
###
FROM base AS build_aiter
ARG AITER_BRANCH
ARG AITER_REPO
ARG USE_SCCACHE
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
pip install /install/*.whl
RUN git clone --recursive --branch ${AITER_BRANCH} ${AITER_REPO}
RUN cd aiter \
&& git submodule update --init --recursive \
&& pip install -r requirements.txt
RUN pip install pyyaml && cd aiter \
&& if [ "$USE_SCCACHE" = "1" ]; then \
export HIP_CLANG_PATH=/opt/sccache-wrappers \
&& sccache --show-stats; \
fi \
&& PREBUILD_KERNELS=1 AITER_USE_SYSTEM_TRITON=1 GPU_ARCHS=${AITER_ROCM_ARCH} python3 setup.py bdist_wheel --dist-dir=dist \
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi \
&& ls /app/aiter/dist/*.whl
RUN mkdir -p /app/install && cp /app/aiter/dist/*.whl /app/install
###
### Final Build
###
# Wheel release stage -
# only includes dependencies used by wheel release pipeline
FROM base AS debs_wheel_release
RUN mkdir /app/debs
RUN --mount=type=bind,from=build_triton,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_fa,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_amdsmi,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_torchvision,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_torchaudio,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_aiter,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
# Full debs stage - includes Mori (used by Docker releases)
FROM base AS debs
RUN mkdir /app/debs
RUN --mount=type=bind,from=build_triton,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_fa,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_amdsmi,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_torchvision,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_torchaudio,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_aiter,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_mori,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
###
### ROCr + CLR build
###
# Pull in updates to ROCr + clr from ROCm 7.2.4 and beyond.
FROM base AS build_rocm_runtime
ARG ROCM_SYSTEMS_REPO
ARG ROCM_RUNTIME_COMMIT
RUN apt-get update \
&& apt-get install -y --no-install-recommends g++ \
libelf-dev libdrm-dev libnuma-dev libdw-dev xxd \
&& rm -rf /var/lib/apt/lists/*
RUN pip install --no-cache-dir CppHeaderParser
RUN git init -q /src && cd /src \
&& git remote add origin ${ROCM_SYSTEMS_REPO} \
&& git config core.sparseCheckout true \
&& git config remote.origin.promisor true \
&& git config remote.origin.partialclonefilter blob:none \
&& git sparse-checkout init --cone \
&& git sparse-checkout set projects/rocr-runtime projects/clr projects/hip shared cmake \
&& git fetch --filter=blob:none --depth 1 origin ${ROCM_RUNTIME_COMMIT} \
&& git checkout -q FETCH_HEAD
# ROCr's trap_handler and blit_shaders call find_package(Clang/LLVM REQUIRED) only to get
# IMPORTED executables. The ROCm dev image ships the binaries but no CMake package config, so
# supply one pointing at them.
RUN mkdir -p /opt/rocm-cmake-shim \
&& printf 'if(NOT TARGET clang)\n add_executable(clang IMPORTED GLOBAL)\n set_target_properties(clang PROPERTIES IMPORTED_LOCATION "/opt/rocm/llvm/bin/clang")\nendif()\nset(Clang_PACKAGE_VERSION "rocm-image-shim")\n' > /opt/rocm-cmake-shim/ClangConfig.cmake \
&& printf 'if(NOT TARGET llvm-objcopy)\n add_executable(llvm-objcopy IMPORTED GLOBAL)\n set_target_properties(llvm-objcopy PROPERTIES IMPORTED_LOCATION "/opt/rocm/llvm/bin/llvm-objcopy")\nendif()\nset(LLVM_FOUND TRUE)\nset(LLVM_PACKAGE_VERSION "rocm-image-shim")\n' > /opt/rocm-cmake-shim/LLVMConfig.cmake
# ROCr first, and installed into /opt/rocm, because the clr build below needs its headers
RUN cd /src/projects/rocr-runtime \
&& cmake -B build -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_PREFIX_PATH=/opt/rocm \
-DCMAKE_INSTALL_PREFIX=/opt/rocm \
-DClang_DIR=/opt/rocm-cmake-shim \
-DLLVM_DIR=/opt/rocm-cmake-shim \
&& cmake --build build --parallel 16 \
&& cmake --install build \
&& cmake --install build --prefix /rocr-install --strip
RUN cd /src/projects/clr \
&& cmake -B build -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCLR_BUILD_HIP=ON \
-DCLR_BUILD_OCL=OFF \
-DHIP_COMMON_DIR=/src/projects/hip \
-DROCM_PATH=/opt/rocm \
-DCMAKE_PREFIX_PATH=/opt/rocm \
-DCMAKE_INSTALL_PREFIX=/opt/rocm \
-DLLVM_DIR=/opt/rocm-cmake-shim \
-DHIP_LLVM_ROOT=/opt/rocm/llvm \
&& cmake --build build --parallel 16 \
&& cmake --install build --prefix /clr-install --strip
RUN mkdir -p /staging/lib \
&& cp -P /rocr-install/lib/libhsa-runtime64.so* /staging/lib/ \
&& cp -P /clr-install/lib/libamdhip64.so* /staging/lib/ \
&& ls -l /staging/lib
# rocprofiler-sdk 1.3.2 fixes torch 2.12 profiler on ROCm 7.2.3
FROM base AS build_rocprofiler_sdk
ARG ROCM_SYSTEMS_REPO
ARG ROCPROFILER_SDK_COMMIT
ARG USE_SCCACHE
RUN apt-get update \
&& apt-get install -y --no-install-recommends libdw-dev libelf-dev libsqlite3-dev libzstd-dev \
&& rm -rf /var/lib/apt/lists/*
RUN git init -q /src && cd /src \
&& git remote add origin ${ROCM_SYSTEMS_REPO} \
&& git config core.sparseCheckout true \
&& git config remote.origin.promisor true \
&& git config remote.origin.partialclonefilter blob:none \
&& git sparse-checkout init --cone \
&& git sparse-checkout set projects/rocprofiler-sdk shared cmake \
&& git fetch --filter=blob:none --depth 1 origin ${ROCPROFILER_SDK_COMMIT} \
&& git checkout -q FETCH_HEAD
# Patch the ROCm 7.2.3 build of rocprofiler-sdk to include queue-interposition fixes that are
# absent from the pinned 7.14 build. This resolves hangs on torch.profiler on ROCm.
# TODO: Drop these once ${ROCPROFILER_SDK_COMMIT} is bumped to a build that already
# contains them, such as ROCm 10.0. See https://github.com/vllm-project/vllm/issues/54087.
ARG ROCPROFILER_SDK_PR_7796=fae7fd57ffb8a76e811d6410b8a0cee1e94d710a
ARG ROCPROFILER_SDK_PR_7924=f8db22adbe17686a9b05ae03367dd1ec73011653
RUN cd /src \
&& HSA_PATHS='projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/hsa/*' \
&& curl -fsSL -o /tmp/qi-7796.patch "${ROCM_SYSTEMS_REPO%.git}/commit/${ROCPROFILER_SDK_PR_7796}.patch" \
&& curl -fsSL -o /tmp/qi-7924.patch "${ROCM_SYSTEMS_REPO%.git}/commit/${ROCPROFILER_SDK_PR_7924}.patch" \
&& git apply -p1 --include="$HSA_PATHS" --verbose /tmp/qi-7796.patch \
&& git apply -p1 --include="$HSA_PATHS" --verbose /tmp/qi-7924.patch
RUN cd /src/projects/rocprofiler-sdk \
&& if [ "$USE_SCCACHE" = "1" ]; then \
export CMAKE_C_COMPILER_LAUNCHER=sccache \
&& export CMAKE_CXX_COMPILER_LAUNCHER=sccache; \
fi \
&& cmake -B build -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_PREFIX_PATH=/opt/rocm \
-DCMAKE_INSTALL_PREFIX=/opt/rocm \
-DROCPROFILER_BUILD_TESTS=OFF \
-DROCPROFILER_BUILD_SAMPLES=OFF \
-DROCPROFILER_BUILD_DOCS=OFF \
&& cmake --build build --parallel "$(nproc)" \
&& cmake --install build --prefix /staging --strip
FROM base AS final
RUN --mount=type=bind,from=debs,src=/app/debs,target=/install \
pip install /install/*.whl
COPY --from=build_rocprofiler_sdk /staging/ /opt/rocm/
ENV ROCPROFILER_QUEUE_INTERPOSITION=0
COPY --from=build_rocm_runtime /staging/ /opt/rocm/
# Hard link the patched library onto every stock file name so that both the
# SONAME lookup and the hard coded paths resolve to it.
RUN set -eux; \
cd /opt/rocm/lib; \
ours_hsa="$(readlink -f libhsa-runtime64.so.1)"; \
ours_hip="$(readlink -f libamdhip64.so.7)"; \
for f in libhsa-runtime64.so.1.*; do \
[ "$f" = "${ours_hsa##*/}" ] || ln -f "$ours_hsa" "$f"; \
done; \
for f in libamdhip64.so.7.*; do \
[ "$f" = "${ours_hip##*/}" ] || ln -f "$ours_hip" "$f"; \
done
RUN ldconfig
ARG BASE_IMAGE
ARG TRITON_BRANCH
ARG TRITON_REPO
ARG PYTORCH_BRANCH
ARG PYTORCH_VISION_BRANCH
ARG PYTORCH_REPO
ARG PYTORCH_VISION_REPO
ARG PYTORCH_AUDIO_BRANCH
ARG PYTORCH_AUDIO_REPO
ARG FA_BRANCH
ARG FA_REPO
ARG AITER_BRANCH
ARG AITER_REPO
ARG MORI_BRANCH
ARG MORI_REPO
ARG ROCM_SYSTEMS_REPO
ARG ROCPROFILER_SDK_COMMIT
ARG ROCM_RUNTIME_COMMIT
RUN echo "BASE_IMAGE: ${BASE_IMAGE}" > /app/versions.txt \
&& echo "ROCM_SYSTEMS_REPO: ${ROCM_SYSTEMS_REPO}" >> /app/versions.txt \
&& echo "ROCPROFILER_SDK_COMMIT: ${ROCPROFILER_SDK_COMMIT}" >> /app/versions.txt \
&& echo "ROCM_RUNTIME_COMMIT: ${ROCM_RUNTIME_COMMIT}" >> /app/versions.txt \
&& echo "TRITON_BRANCH: ${TRITON_BRANCH}" >> /app/versions.txt \
&& echo "TRITON_REPO: ${TRITON_REPO}" >> /app/versions.txt \
&& echo "PYTORCH_BRANCH: ${PYTORCH_BRANCH}" >> /app/versions.txt \
&& echo "PYTORCH_VISION_BRANCH: ${PYTORCH_VISION_BRANCH}" >> /app/versions.txt \
&& echo "PYTORCH_REPO: ${PYTORCH_REPO}" >> /app/versions.txt \
&& echo "PYTORCH_VISION_REPO: ${PYTORCH_VISION_REPO}" >> /app/versions.txt \
&& echo "PYTORCH_AUDIO_BRANCH: ${PYTORCH_AUDIO_BRANCH}" >> /app/versions.txt \
&& echo "PYTORCH_AUDIO_REPO: ${PYTORCH_AUDIO_REPO}" >> /app/versions.txt \
&& echo "FA_BRANCH: ${FA_BRANCH}" >> /app/versions.txt \
&& echo "FA_REPO: ${FA_REPO}" >> /app/versions.txt \
&& echo "AITER_BRANCH: ${AITER_BRANCH}" >> /app/versions.txt \
&& echo "AITER_REPO: ${AITER_REPO}" >> /app/versions.txt \
&& echo "MORI_BRANCH: ${MORI_BRANCH}" >> /app/versions.txt \
&& echo "MORI_REPO: ${MORI_REPO}" >> /app/versions.txt
docker/Dockerfile.rocm_base
+267 -270
View File
@@ -1,12 +1,11 @@
ARG BASE_IMAGE=rocm/dev-ubuntu-22.04:7.2.3-complete
ARG TRITON_BRANCH="669b31a" # release/internal/3.8.x as of 09/17
ARG BASE_IMAGE=ubuntu:22.04
ARG PYTHON_VERSION=3.12
ARG TORCH_VERSION=2.12.0+rocm10.0.0
ARG TORCHVISION_VERSION=0.27.0+rocm10.0.0
ARG TORCHAUDIO_VERSION=2.11.0+rocm10.0.0
ARG TRITON_BRANCH="669b31ac" # release/internal/3.8.x as of 09/29/2026
ARG TRITON_VERSION=3.8.0+git669b31ac
ARG TRITON_REPO="https://github.com/ROCm/triton.git"
ARG PYTORCH_BRANCH="733fca1" # release/2.13 as of 09/17
ARG PYTORCH_REPO="https://github.com/ROCm/pytorch.git"
ARG PYTORCH_VISION_BRANCH="v0.28.0"
ARG PYTORCH_VISION_REPO="https://github.com/pytorch/vision.git"
ARG PYTORCH_AUDIO_BRANCH="v2.11.0"
ARG PYTORCH_AUDIO_REPO="https://github.com/pytorch/audio.git"
ARG FA_BRANCH="0e60e394"
ARG FA_REPO="https://github.com/Dao-AILab/flash-attention.git"
ARG AITER_BRANCH="v0.1.25"
@@ -14,8 +13,18 @@ ARG AITER_REPO="https://github.com/ROCm/aiter.git"
ARG MORI_BRANCH="v1.2.3.post1"
ARG MORI_REPO="https://github.com/ROCm/mori.git"
ARG ROCM_SYSTEMS_REPO="https://github.com/ROCm/rocm-systems.git"
ARG ROCPROFILER_SDK_COMMIT="2b22ab0195cc1461cd9abf3b969e9dd7c10af350"
ARG ROCM_RUNTIME_COMMIT="af70edc91f1dc5d228fd4c076df99e65e49c4521"
ARG ROCM_RUNTIME_COMMIT="25e14349f2606a6b95c4ccd45eb92a1835646cf2"
# TheRock configuration
ARG SITE_PACKAGES=/usr/local/lib/python${PYTHON_VERSION}/dist-packages
ARG ROCM_RELEASE_WHEELS_MULTIARCH_URL="https://stable.repo.amd.com/rocm/whl-next/" # For pre-built GPU images, packages, etc.
ARG ROCM_NIGHTLY_WHEELS_MULTIARCH_URL="https://nightly.repo.amd.com/rocm/whl-next/" # For torch, triton, etc. wheels with latest fixes.
ARG ROCM_SDK_VERSION=10.0.0
# Build configuration
ARG PYTORCH_ROCM_ARCH="gfx90a;gfx942;gfx950;gfx1100;gfx1101;gfx1200;gfx1201;gfx1150;gfx1151"
ARG AITER_ROCM_ARCH="gfx942;gfx950"
# Sccache configuration (only used in release pipeline)
ARG USE_SCCACHE
@@ -25,47 +34,128 @@ ARG SCCACHE_BUCKET_NAME=vllm-build-sccache
ARG SCCACHE_REGION_NAME=us-west-2
ARG SCCACHE_S3_NO_CREDENTIALS=0
FROM ${BASE_IMAGE} AS base
ENV PATH=/opt/rocm/llvm/bin:/opt/rocm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ENV ROCM_PATH=/opt/rocm
ENV LD_LIBRARY_PATH=/opt/rocm/lib:/usr/local/lib:
ARG PYTORCH_ROCM_ARCH
ENV PYTORCH_ROCM_ARCH="${PYTORCH_ROCM_ARCH}"
ARG AITER_ROCM_ARCH
ENV AITER_ROCM_ARCH="${AITER_ROCM_ARCH}"
ARG PYTORCH_ROCM_ARCH=gfx90a;gfx942;gfx950;gfx1100;gfx1101;gfx1200;gfx1201;gfx1150;gfx1151
ENV PYTORCH_ROCM_ARCH=${PYTORCH_ROCM_ARCH}
ENV AITER_ROCM_ARCH=gfx942;gfx950
# Note: Do not set MORI_GPU_ARCHS here, it is automatically inferred at runtime
ARG SITE_PACKAGES
ENV SITE_PACKAGES=${SITE_PACKAGES}
ENV SDK_DEV="$SITE_PACKAGES/_rocm_sdk_devel" \
SDK_CORE="$SITE_PACKAGES/_rocm_sdk_core"
ENV ROCM_PATH="$SDK_DEV" \
ROCM_HOME="$SDK_DEV" \
HIP_PATH="$SDK_DEV" \
HIP_CLANG_PATH="$SDK_DEV/lib/llvm/bin" \
HIP_DEVICE_LIB_PATH="$SDK_CORE/lib/llvm/amdgcn/bitcode"
ENV LD_LIBRARY_PATH="$SDK_DEV/lib:$SDK_DEV/lib/rocm_sysdeps/lib:$SDK_CORE/lib:$SDK_CORE/lib/rocm_sysdeps/lib"
ENV PATH="$SDK_DEV/bin:$SDK_DEV/lib/llvm/bin:$SDK_CORE/bin:$PATH"
# Required for RCCL in ROCm7.1
ENV HSA_NO_SCRATCH_RECLAIM=1
ARG PYTHON_VERSION=3.12
ARG PYTHON_VERSION
ENV PYTHON_VERSION=${PYTHON_VERSION}
RUN mkdir -p /app
WORKDIR /app
ENV DEBIAN_FRONTEND=noninteractive
# Install Python and other dependencies (plus gcc-13, selected later in build_pytorch).
# Install Python and other dependencies
RUN apt-get update -y \
&& apt-get install -y software-properties-common git curl sudo vim less libgfortran5 libopenmpi-dev libpci-dev liblzma-dev pkg-config \
&& apt-get install -y software-properties-common build-essential git curl sudo vim less libgfortran5 libopenmpi-dev libpci-dev liblzma-dev pkg-config libdrm-dev libnuma-dev kmod \
&& for i in 1 2 3; do \
add-apt-repository -y ppa:deadsnakes/ppa && add-apt-repository -y ppa:ubuntu-toolchain-r/test && break || \
add-apt-repository -y ppa:deadsnakes/ppa && break || \
{ echo "Attempt $i failed, retrying in 5s..."; sleep 5; }; \
done \
&& apt-get update -y \
&& apt-get install -y python${PYTHON_VERSION} python${PYTHON_VERSION}-dev python${PYTHON_VERSION}-venv \
python${PYTHON_VERSION}-lib2to3 python-is-python3 gcc-13 g++-13 \
python${PYTHON_VERSION}-lib2to3 python-is-python3 \
&& update-alternatives --install /usr/bin/python3 python3 /usr/bin/python${PYTHON_VERSION} 1 \
&& update-alternatives --set python3 /usr/bin/python${PYTHON_VERSION} \
&& ln -sf /usr/bin/python${PYTHON_VERSION}-config /usr/bin/python3-config \
&& curl -sS https://bootstrap.pypa.io/get-pip.py | python${PYTHON_VERSION} \
&& python3 --version && python3 -m pip --version
RUN pip install -U packaging 'cmake<4' ninja wheel 'setuptools<80' 'pybind11<3.1.0' Cython
RUN pip install -U packaging 'cmake<4' ninja wheel 'setuptools<80' pybind11==3.0.4 Cython
RUN apt-get update && apt-get install -y libjpeg-dev libsox-dev libsox-fmt-all sox && rm -rf /var/lib/apt/lists/*
# Install ROCm SDK via TheRock
ARG ROCM_RELEASE_WHEELS_MULTIARCH_URL
ARG ROCM_NIGHTLY_WHEELS_MULTIARCH_URL
ARG ROCM_SDK_VERSION
RUN pip install --index-url ${ROCM_RELEASE_WHEELS_MULTIARCH_URL} \
--extra-index-url ${ROCM_NIGHTLY_WHEELS_MULTIARCH_URL} \
"rocm[libraries,devel]${ROCM_SDK_VERSION:+==${ROCM_SDK_VERSION}}" \
&& rocm-sdk init
# Install PyTorch wheels (nightly for latest torch, release as fallback)
ARG TORCH_VERSION
ARG TORCHAUDIO_VERSION
ARG TORCHVISION_VERSION
RUN pip install \
--index-url ${ROCM_RELEASE_WHEELS_MULTIARCH_URL} \
--extra-index-url ${ROCM_NIGHTLY_WHEELS_MULTIARCH_URL} \
"torch==${TORCH_VERSION}" \
"torchaudio==${TORCHAUDIO_VERSION}" \
"torchvision==${TORCHVISION_VERSION}"
# Install the AMD-compiled device kernels corresponding to PYTORCH_ROCM_ARCH:
# Possible values that can be in PYTORCH_ROCM_ARCH:
#
# gfx90a (MI210/250), gfx942 (MI300), gfx950 (MI350), gfx1100 (7900XTX),
# gfx1101 (7800XT), gfx1200 (9070XT), gfx1201 (9070), gfx1150/1151 (Strix APU)
# For each arch in PYTORCH_ROCM_ARCH, install the precompiled GPU code that
# PyTorch needs to run on that specific chip.
SHELL ["/bin/bash", "-c"]
RUN TORCH_VER=$(python3 -c "import torch; print(torch.__version__)") \
&& for arch in $(echo "${PYTORCH_ROCM_ARCH}" | tr ';' ' '); do \
echo "Installing device packages for ${arch}" \
&& pip install --index-url ${ROCM_RELEASE_WHEELS_MULTIARCH_URL} \
--extra-index-url ${ROCM_NIGHTLY_WHEELS_MULTIARCH_URL} \
"torch[device-${arch}]==${TORCH_VER}" \
"torchvision[device-${arch}]==${TORCHVISION_VERSION}" \
"rocm[device-${arch}]==${ROCM_SDK_VERSION}"; \
done
# Remove gfx90a xnack kpacks (ASAN artifacts that cause hipErrorInvalidImage on MI250)
RUN rm -f "${SITE_PACKAGES}/torch/.kpack/torch_gfx90a:xnack+.kpack" \
"${SITE_PACKAGES}/torch/.kpack/torch_gfx90a:xnack-.kpack"
RUN cd $SITE_PACKAGES/_rocm_sdk_core/share/amd_smi && pip install .
ENV PYTHONPATH=$SITE_PACKAGES/_rocm_sdk_core/share/amd_smi
# TheRock runtime fixes
RUN if [ -f "${SITE_PACKAGES}/rocm_sdk/__init__.py" ]; then \
sed -i 's/rtld_global: bool = True/rtld_global: bool = False/g' "${SITE_PACKAGES}/rocm_sdk/__init__.py"; \
fi \
&& if [ -f "${SITE_PACKAGES}/_rocm_sdk_core/lib/librocm_smi64.so.1" ] \
&& [ ! -e "${ROCM_PATH}/lib/librocm_smi64.so.7" ]; then \
ln -sfn "${SITE_PACKAGES}/_rocm_sdk_core/lib/librocm_smi64.so.1" "${ROCM_PATH}/lib/librocm_smi64.so.7"; \
fi \
&& printf '%s\n%s\n' "${ROCM_PATH}/lib" "${SITE_PACKAGES}/_rocm_sdk_core/lib" > /etc/ld.so.conf.d/rocm-pip.conf \
&& for f in ${SITE_PACKAGES}/_rocm_sdk_core/lib/libamdhip64.so*; do \
bn=$(basename "$f"); \
if [ -f "${ROCM_PATH}/lib/${bn}" ] && [ -f "$f" ] && [ ! -L "$f" ]; then \
rm -f "$f"; \
ln -sfn "${ROCM_PATH}/lib/${bn}" "$f"; \
fi; \
done \
&& for f in librocprofiler-sdk.so.1 librocprofiler-register.so.0 \
librocprofiler-sdk-roctx.so.1; do \
if [ -e "${SITE_PACKAGES}/_rocm_sdk_core/lib/${f}" ] \
&& [ -e "${SITE_PACKAGES}/_rocm_sdk_devel/lib/${f}" ]; then \
rm -f "${SITE_PACKAGES}/_rocm_sdk_core/lib/${f}"; \
ln -sfn "${SITE_PACKAGES}/_rocm_sdk_devel/lib/${f}" \
"${SITE_PACKAGES}/_rocm_sdk_core/lib/${f}"; \
fi; \
done \
&& ldconfig
# Install sccache if USE_SCCACHE is enabled (for release builds)
ARG USE_SCCACHE
ARG SCCACHE_DOWNLOAD_URL
@@ -94,141 +184,35 @@ RUN if [ "$USE_SCCACHE" = "1" ]; then \
RUN if [ "$USE_SCCACHE" = "1" ]; then \
echo "Setting up sccache wrappers for HIP compilation..." \
&& mkdir -p /opt/sccache-wrappers \
&& printf '#!/bin/bash\nexec sccache /opt/rocm/lib/llvm/bin/clang++ "$@"\n' > /opt/sccache-wrappers/clang++ \
&& printf '#!/bin/bash\nexec sccache %s/lib/llvm/bin/clang++ "$@"\n' "${ROCM_PATH}" > /opt/sccache-wrappers/clang++ \
&& chmod +x /opt/sccache-wrappers/clang++ \
&& printf '#!/bin/bash\nexec sccache /opt/rocm/lib/llvm/bin/clang "$@"\n' > /opt/sccache-wrappers/clang \
&& printf '#!/bin/bash\nexec sccache %s/lib/llvm/bin/clang "$@"\n' "${ROCM_PATH}" > /opt/sccache-wrappers/clang \
&& chmod +x /opt/sccache-wrappers/clang \
&& echo "sccache wrappers created in /opt/sccache-wrappers"; \
fi
# Set sccache environment variables only when USE_SCCACHE=1
# This prevents S3 config from leaking into images when sccache is not used
ARG USE_SCCACHE
ENV SCCACHE_BUCKET=${USE_SCCACHE:+${SCCACHE_BUCKET_NAME}}
ENV SCCACHE_REGION=${USE_SCCACHE:+${SCCACHE_REGION_NAME}}
ENV SCCACHE_S3_NO_CREDENTIALS=${USE_SCCACHE:+${SCCACHE_S3_NO_CREDENTIALS}}
ENV SCCACHE_IDLE_TIMEOUT=${USE_SCCACHE:+0}
###
### Triton Build
###
FROM base AS build_triton
ARG TRITON_BRANCH
ARG TRITON_REPO
RUN git clone ${TRITON_REPO}
RUN cd triton \
&& git checkout ${TRITON_BRANCH} \
&& git config --global user.email "you@example.com" && git config --global user.name "Your Name" \
&& if [ ! -f setup.py ]; then cd python; fi \
&& MAX_JOBS=16 python3 setup.py bdist_wheel --dist-dir=dist \
&& mkdir -p /app/install && cp dist/*.whl /app/install
###
### AMD SMI Build
###
FROM base AS build_amdsmi
RUN cd /opt/rocm/share/amd_smi \
&& pip wheel . --wheel-dir=dist
RUN mkdir -p /app/install && cp /opt/rocm/share/amd_smi/dist/*.whl /app/install
###
### Pytorch build
###
FROM base AS build_pytorch
ARG PYTORCH_BRANCH
ARG PYTORCH_REPO
ARG USE_SCCACHE
RUN apt-get update && apt-get install -y pkg-config liblzma-dev
# torch 2.12 emits float->__bf16 libcalls (__truncsfbf2) absent from gcc-11 libgcc; build with gcc-13.
RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-13 130 --slave /usr/bin/g++ g++ /usr/bin/g++-13
RUN git clone ${PYTORCH_REPO} pytorch
RUN cd pytorch && git checkout ${PYTORCH_BRANCH}
RUN cd pytorch \
&& pip install -r requirements.txt && git submodule update --init --recursive
RUN cd pytorch && python3 tools/amd_build/build_amd.py \
&& if [ "$USE_SCCACHE" = "1" ]; then \
export HIP_CLANG_PATH=/opt/sccache-wrappers \
&& export CMAKE_C_COMPILER_LAUNCHER=sccache \
&& export CMAKE_CXX_COMPILER_LAUNCHER=sccache \
&& sccache --show-stats; \
fi \
&& CMAKE_PREFIX_PATH=$(python3 -c 'import sys; print(sys.prefix)') python3 setup.py bdist_wheel --dist-dir=dist \
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi
RUN mkdir -p /app/install && cp /app/pytorch/dist/*.whl /app/install
###
### Shared PyTorch runtime for dependent builds
###
FROM base AS build_pytorch_runtime
RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-13 130 --slave /usr/bin/g++ g++ /usr/bin/g++-13
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install,ro \
pip install /install/*.whl
###
### TorchVision build
###
FROM build_pytorch_runtime AS build_torchvision
ARG PYTORCH_VISION_BRANCH
ARG PYTORCH_VISION_REPO
ARG USE_SCCACHE
RUN git clone ${PYTORCH_VISION_REPO} vision
RUN cd vision && git checkout ${PYTORCH_VISION_BRANCH} \
&& if [ "$USE_SCCACHE" = "1" ]; then \
export HIP_CLANG_PATH=/opt/sccache-wrappers \
&& export CMAKE_C_COMPILER_LAUNCHER=sccache \
&& export CMAKE_CXX_COMPILER_LAUNCHER=sccache; \
fi \
&& python3 setup.py bdist_wheel --dist-dir=dist \
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi
RUN mkdir -p /app/install && cp /app/vision/dist/*.whl /app/install
###
### TorchAudio build
###
FROM build_pytorch_runtime AS build_torchaudio
ARG PYTORCH_AUDIO_BRANCH
ARG PYTORCH_AUDIO_REPO
ARG USE_SCCACHE
RUN git clone ${PYTORCH_AUDIO_REPO} audio
RUN cd audio && git checkout ${PYTORCH_AUDIO_BRANCH} \
&& git submodule update --init --recursive
RUN cd audio && pip install -r requirements.txt
RUN cd audio \
&& if [ "$USE_SCCACHE" = "1" ]; then \
export HIP_CLANG_PATH=/opt/sccache-wrappers \
&& export CMAKE_C_COMPILER_LAUNCHER=sccache \
&& export CMAKE_CXX_COMPILER_LAUNCHER=sccache; \
fi \
&& python3 setup.py bdist_wheel --dist-dir=dist \
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi
RUN mkdir -p /app/install && cp /app/audio/dist/*.whl /app/install
###
### MORI Build
###
FROM base AS build_mori
ARG MORI_BRANCH
ARG MORI_REPO
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
pip install /install/*.whl
RUN git clone ${MORI_REPO}
RUN cd mori \
&& git checkout ${MORI_BRANCH} \
&& git submodule update --init --recursive \
&& python3 setup.py bdist_wheel --dist-dir=dist && ls /app/mori/dist/*.whl
&& ln -sfn ${ROCM_PATH} /opt/rocm \
&& pip install "setuptools_scm>=6.2" \
&& CMAKE_PREFIX_PATH="${ROCM_PATH}:${ROCM_PATH}/lib/rocm_sysdeps" \
python3 setup.py bdist_wheel --dist-dir=dist && ls /app/mori/dist/*.whl
RUN mkdir -p /app/install && cp /app/mori/dist/*.whl /app/install
###
### FlashAttention Build
###
@@ -236,8 +220,6 @@ FROM base AS build_fa
ARG FA_BRANCH
ARG FA_REPO
ARG USE_SCCACHE
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
pip install /install/*.whl
RUN git clone ${FA_REPO}
RUN cd flash-attention \
&& git checkout ${FA_BRANCH} \
@@ -258,12 +240,12 @@ FROM base AS build_aiter
ARG AITER_BRANCH
ARG AITER_REPO
ARG USE_SCCACHE
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
pip install /install/*.whl
RUN git clone --recursive --branch ${AITER_BRANCH} ${AITER_REPO}
RUN cd aiter \
&& git submodule update --init --recursive \
&& pip install -r requirements.txt
# AITER_USE_SYSTEM_TRITON keeps setup.py from resolving its own triton, which would
# displace the pinned one already installed in base.
RUN pip install pyyaml && cd aiter \
&& if [ "$USE_SCCACHE" = "1" ]; then \
export HIP_CLANG_PATH=/opt/sccache-wrappers \
@@ -276,58 +258,30 @@ RUN mkdir -p /app/install && cp /app/aiter/dist/*.whl /app/install
###
### Final Build
### Triton Build
###
FROM base AS build_triton
ARG TRITON_BRANCH
ARG TRITON_REPO
RUN git clone ${TRITON_REPO}
RUN cd triton \
&& git checkout ${TRITON_BRANCH} \
&& if [ ! -f setup.py ]; then cd python; fi \
&& python3 setup.py bdist_wheel --dist-dir=dist \
&& mkdir -p /app/install && cp dist/*.whl /app/install
# Wheel release stage -
# only includes dependencies used by wheel release pipeline
FROM base AS debs_wheel_release
RUN mkdir /app/debs
RUN --mount=type=bind,from=build_triton,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_fa,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_amdsmi,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_torchvision,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_torchaudio,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_aiter,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
# Full debs stage - includes Mori (used by Docker releases)
FROM base AS debs
RUN mkdir /app/debs
RUN --mount=type=bind,from=build_triton,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_fa,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_amdsmi,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_torchvision,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_torchaudio,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_aiter,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_mori,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
###
### ROCr + CLR build
###
# Pull in updates to ROCr + clr from ROCm 7.2.4 and beyond.
# Brings in multi-stream perf improvements; drop after these arrive in an official release
FROM base AS build_rocm_runtime
ARG ROCM_SYSTEMS_REPO
ARG ROCM_RUNTIME_COMMIT
# libgl-dev: rocclr does find_package(OpenGL REQUIRED) unconditionally and the wheel image has
# no GL headers. The link must not pick up a libGL DT_NEEDED, which is asserted after the build.
RUN apt-get update \
&& apt-get install -y --no-install-recommends g++ \
libelf-dev libdrm-dev libnuma-dev libdw-dev xxd \
&& apt-get install -y --no-install-recommends g++ libdw-dev xxd libgl-dev libglx-dev \
&& rm -rf /var/lib/apt/lists/*
RUN pip install --no-cache-dir CppHeaderParser
RUN git init -q /src && cd /src \
@@ -339,20 +293,17 @@ RUN git init -q /src && cd /src \
&& git sparse-checkout set projects/rocr-runtime projects/clr projects/hip shared cmake \
&& git fetch --filter=blob:none --depth 1 origin ${ROCM_RUNTIME_COMMIT} \
&& git checkout -q FETCH_HEAD
# ROCr's trap_handler and blit_shaders call find_package(Clang/LLVM REQUIRED) only to get
# IMPORTED executables. The ROCm dev image ships the binaries but no CMake package config, so
# supply one pointing at them.
RUN mkdir -p /opt/rocm-cmake-shim \
&& printf 'if(NOT TARGET clang)\n add_executable(clang IMPORTED GLOBAL)\n set_target_properties(clang PROPERTIES IMPORTED_LOCATION "/opt/rocm/llvm/bin/clang")\nendif()\nset(Clang_PACKAGE_VERSION "rocm-image-shim")\n' > /opt/rocm-cmake-shim/ClangConfig.cmake \
&& printf 'if(NOT TARGET llvm-objcopy)\n add_executable(llvm-objcopy IMPORTED GLOBAL)\n set_target_properties(llvm-objcopy PROPERTIES IMPORTED_LOCATION "/opt/rocm/llvm/bin/llvm-objcopy")\nendif()\nset(LLVM_FOUND TRUE)\nset(LLVM_PACKAGE_VERSION "rocm-image-shim")\n' > /opt/rocm-cmake-shim/LLVMConfig.cmake
# ROCr first, and installed into /opt/rocm, because the clr build below needs its headers
# libelf is taken from the SDK's own vendored sysdeps rather than apt -- the wheel image has no
# /usr/include/libelf.h -- so ROCr links against the copy the wheel itself was built against.
ENV SYSDEPS="$SDK_DEV/lib/rocm_sysdeps"
# ROCr first, and installed into $SDK_DEV, because the clr build below needs its headers.
RUN cd /src/projects/rocr-runtime \
&& cmake -B build -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_PREFIX_PATH=/opt/rocm \
-DCMAKE_INSTALL_PREFIX=/opt/rocm \
-DClang_DIR=/opt/rocm-cmake-shim \
-DLLVM_DIR=/opt/rocm-cmake-shim \
-DCMAKE_PREFIX_PATH="$SDK_DEV;$SYSDEPS" \
-DCMAKE_INSTALL_PREFIX="$SDK_DEV" \
-DLIBELF_INCLUDE_DIRS="$SYSDEPS/include" \
-DLIBELF_LIBRARIES="$SYSDEPS/lib/libelf.so" \
&& cmake --build build --parallel 16 \
&& cmake --install build \
&& cmake --install build --prefix /rocr-install --strip
@@ -361,12 +312,12 @@ RUN cd /src/projects/clr \
-DCMAKE_BUILD_TYPE=Release \
-DCLR_BUILD_HIP=ON \
-DCLR_BUILD_OCL=OFF \
-DROCM_KPACK_ENABLED=ON \
-DHIP_COMMON_DIR=/src/projects/hip \
-DROCM_PATH=/opt/rocm \
-DCMAKE_PREFIX_PATH=/opt/rocm \
-DCMAKE_INSTALL_PREFIX=/opt/rocm \
-DLLVM_DIR=/opt/rocm-cmake-shim \
-DHIP_LLVM_ROOT=/opt/rocm/llvm \
-DROCM_PATH="$SDK_DEV" \
-DCMAKE_PREFIX_PATH="$SDK_DEV;$SYSDEPS" \
-DCMAKE_INSTALL_PREFIX="$SDK_DEV" \
-DHIP_LLVM_ROOT="$SDK_DEV/lib/llvm" \
&& cmake --build build --parallel 16 \
&& cmake --install build --prefix /clr-install --strip
RUN mkdir -p /staging/lib \
@@ -374,82 +325,115 @@ RUN mkdir -p /staging/lib \
&& cp -P /clr-install/lib/libamdhip64.so* /staging/lib/ \
&& ls -l /staging/lib
# rocprofiler-sdk 1.3.2 fixes torch 2.12 profiler on ROCm 7.2.3
FROM base AS build_rocprofiler_sdk
ARG ROCM_SYSTEMS_REPO
ARG ROCPROFILER_SDK_COMMIT
ARG USE_SCCACHE
RUN apt-get update \
&& apt-get install -y --no-install-recommends libdw-dev libelf-dev libsqlite3-dev libzstd-dev \
&& rm -rf /var/lib/apt/lists/*
RUN git init -q /src && cd /src \
&& git remote add origin ${ROCM_SYSTEMS_REPO} \
&& git config core.sparseCheckout true \
&& git config remote.origin.promisor true \
&& git config remote.origin.partialclonefilter blob:none \
&& git sparse-checkout init --cone \
&& git sparse-checkout set projects/rocprofiler-sdk shared cmake \
&& git fetch --filter=blob:none --depth 1 origin ${ROCPROFILER_SDK_COMMIT} \
&& git checkout -q FETCH_HEAD
# Patch the ROCm 7.2.3 build of rocprofiler-sdk to include queue-interposition fixes that are
# absent from the pinned 7.14 build. This resolves hangs on torch.profiler on ROCm.
# TODO: Drop these once ${ROCPROFILER_SDK_COMMIT} is bumped to a build that already
# contains them, such as ROCm 10.0. See https://github.com/vllm-project/vllm/issues/54087.
ARG ROCPROFILER_SDK_PR_7796=fae7fd57ffb8a76e811d6410b8a0cee1e94d710a
ARG ROCPROFILER_SDK_PR_7924=f8db22adbe17686a9b05ae03367dd1ec73011653
RUN cd /src \
&& HSA_PATHS='projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/hsa/*' \
&& curl -fsSL -o /tmp/qi-7796.patch "${ROCM_SYSTEMS_REPO%.git}/commit/${ROCPROFILER_SDK_PR_7796}.patch" \
&& curl -fsSL -o /tmp/qi-7924.patch "${ROCM_SYSTEMS_REPO%.git}/commit/${ROCPROFILER_SDK_PR_7924}.patch" \
&& git apply -p1 --include="$HSA_PATHS" --verbose /tmp/qi-7796.patch \
&& git apply -p1 --include="$HSA_PATHS" --verbose /tmp/qi-7924.patch
RUN cd /src/projects/rocprofiler-sdk \
&& if [ "$USE_SCCACHE" = "1" ]; then \
export CMAKE_C_COMPILER_LAUNCHER=sccache \
&& export CMAKE_CXX_COMPILER_LAUNCHER=sccache; \
fi \
&& cmake -B build -G Ninja \
-DCMAKE_BUILD_TYPE=Release \
-DCMAKE_PREFIX_PATH=/opt/rocm \
-DCMAKE_INSTALL_PREFIX=/opt/rocm \
-DROCPROFILER_BUILD_TESTS=OFF \
-DROCPROFILER_BUILD_SAMPLES=OFF \
-DROCPROFILER_BUILD_DOCS=OFF \
&& cmake --build build --parallel "$(nproc)" \
&& cmake --install build --prefix /staging --strip
###
### Final Build
###
# Wheel release stage -
# only includes dependencies used by wheel release pipeline
FROM base AS debs_wheel_release
RUN mkdir /app/debs
RUN --mount=type=bind,from=build_triton,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_fa,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_aiter,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
# The wheel index also serves a torch repacked to require our triton instead of
# the one TheRock built it with, and amdsmi, which TheRock ships only as sources
# inside rocm-sdk-core.
ARG TORCH_VERSION
ARG ROCM_RELEASE_WHEELS_MULTIARCH_URL
ARG ROCM_NIGHTLY_WHEELS_MULTIARCH_URL
COPY tools/vllm-rocm/therock_wheels.py /tmp/therock_wheels.py
# regex and pybase64 aren't installed here; fall back to the stdlib modules.
RUN sed -i -e 's/import regex as re/import re/' \
-e 's/import pybase64 as base64/import base64/' /tmp/therock_wheels.py \
&& pip download --no-deps --dest /tmp/torch-dl \
--index-url ${ROCM_RELEASE_WHEELS_MULTIARCH_URL} \
--extra-index-url ${ROCM_NIGHTLY_WHEELS_MULTIARCH_URL} \
"torch==${TORCH_VERSION}" \
&& TRITON_VER=$(basename /app/debs/triton-*.whl | cut -d- -f2) \
&& python3 /tmp/therock_wheels.py repack-torch /tmp/torch-dl/torch-*.whl \
--triton-version "${TRITON_VER}" --out-dir /app/debs \
&& python3 /tmp/therock_wheels.py patch-amdsmi "${SDK_CORE}/share/amd_smi" \
--out-dir /tmp/amd_smi \
&& pip wheel --no-deps --wheel-dir /app/debs /tmp/amd_smi \
&& rm -rf /tmp/torch-dl /tmp/amd_smi /tmp/therock_wheels.py \
&& ls -l /app/debs
FROM base AS debs
RUN mkdir /app/debs
RUN --mount=type=bind,from=build_triton,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_fa,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_aiter,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
RUN --mount=type=bind,from=build_mori,src=/app/install/,target=/install \
cp /install/*.whl /app/debs
FROM base AS final
RUN --mount=type=bind,from=debs,src=/app/debs,target=/install \
pip install /install/*.whl
ARG TRITON_VERSION
COPY --from=build_rocprofiler_sdk /staging/ /opt/rocm/
# torch 2.12 implicitly attaches rocprofiler-sdk; its queue interposition hangs at HSA
# teardown, so disable it. TODO: drop once ROCm's stock rocprofiler carries the fix.
ENV ROCPROFILER_QUEUE_INTERPOSITION=0
COPY --from=build_rocm_runtime /staging/ /opt/rocm/
# Hard link the patched library onto every stock file name so that both the
# SONAME lookup and the hard coded paths resolve to it.
COPY --from=build_rocm_runtime /staging/lib/ ${SDK_DEV}/lib/
# Hard link the patched library onto every stock file name, in both SDK trees, so that the
# SONAME lookup and the hard coded paths all resolve to it. Hard links rather than symlinks
# because ldconfig rewrites SONAME symlinks but cannot re-point an inode. $SDK_CORE keeps its
# own libhsa-runtime64 copy, which is on LD_LIBRARY_PATH and in ld.so.conf; relinking it is
# belt and braces rather than strictly required, but it leaves no stock ROCr in the image.
RUN set -eux; \
cd /opt/rocm/lib; \
cd "$SDK_DEV/lib"; \
ours_hsa="$(readlink -f libhsa-runtime64.so.1)"; \
ours_hip="$(readlink -f libamdhip64.so.7)"; \
for f in libhsa-runtime64.so.1.*; do \
[ "$f" = "${ours_hsa##*/}" ] || ln -f "$ours_hsa" "$f"; \
done; \
for f in libamdhip64.so.7.*; do \
[ "$f" = "${ours_hip##*/}" ] || ln -f "$ours_hip" "$f"; \
for f in "$SDK_DEV"/lib/libhsa-runtime64.so.1* "$SDK_CORE"/lib/libhsa-runtime64.so.1* \
"$SDK_DEV"/lib/libamdhip64.so.7* "$SDK_CORE"/lib/libamdhip64.so.7*; do \
[ -f "$f" ] && [ ! -L "$f" ] || continue; \
case "${f##*/}" in libhsa*) ours="$ours_hsa";; *) ours="$ours_hip";; esac; \
[ "$f" -ef "$ours" ] || ln -f "$ours" "$f"; \
done
RUN ldconfig
# This is to prevent resolvers from checking METADATA and RECORD and clobbering
# the Triton source build we installed. Constraints files are unable to help
# with this. However, these entries are optional and can be blanked which
# solves the problem.
RUN TORCH_DIST_INFO=$(ls -d ${SITE_PACKAGES}/torch-*.dist-info) \
&& grep -q '^Requires-Dist: triton' "$TORCH_DIST_INFO/METADATA" \
&& sed -i '/^Requires-Dist: triton/d' "$TORCH_DIST_INFO/METADATA" \
&& sed -i "s|^\(${TORCH_DIST_INFO##*/}/METADATA\),.*|\1,,|" "$TORCH_DIST_INFO/RECORD"
RUN --mount=type=bind,from=debs,src=/app/debs,target=/install \
pip install /install/*.whl \
&& ! pip check 2>&1 | grep -iE '^(torch|triton) ' \
&& test "$(pip show triton | awk '/^Version:/ {print $2}')" = "${TRITON_VERSION}" \
# Create a constraints file to protect the versions we need, especially
# triton, requirements will try to reinstall repeatedly.
&& pip list --format=freeze \
| grep -E '^(torch|triton|torchaudio|torchvision|pybind11)==' > /etc/rocm-constraints.txt \
&& echo "ROCm constraints:" && cat /etc/rocm-constraints.txt \
&& test "$(wc -l < /etc/rocm-constraints.txt)" -eq 5 \
&& grep -q '^pybind11==3\.0\.' /etc/rocm-constraints.txt \
&& python3 -c "import torch; assert torch.version.hip, f'non-ROCm torch: {torch.__version__}'; print(f'torch={torch.__version__}')" \
&& python3 -c "import triton; print(f'triton={triton.__version__}')" \
&& pip list 2>/dev/null | grep -iE '^(torch|triton) '
ARG BASE_IMAGE
ARG PYTHON_VERSION
ARG ROCM_SDK_VERSION
ARG TORCH_VERSION
ARG TORCHAUDIO_VERSION
ARG TORCHVISION_VERSION
ARG PYTORCH_ROCM_ARCH
ARG AITER_ROCM_ARCH
ARG TRITON_BRANCH
ARG TRITON_REPO
ARG PYTORCH_BRANCH
ARG PYTORCH_VISION_BRANCH
ARG PYTORCH_REPO
ARG PYTORCH_VISION_REPO
ARG PYTORCH_AUDIO_BRANCH
ARG PYTORCH_AUDIO_REPO
ARG FA_BRANCH
ARG FA_REPO
ARG AITER_BRANCH
@@ -457,23 +441,36 @@ ARG AITER_REPO
ARG MORI_BRANCH
ARG MORI_REPO
ARG ROCM_SYSTEMS_REPO
ARG ROCPROFILER_SDK_COMMIT
ARG ROCM_RUNTIME_COMMIT
ARG TRITON_VERSION
# Record hashes for Triton and pybind11 so we can verify later.
RUN echo "BASE_IMAGE: ${BASE_IMAGE}" > /app/versions.txt \
&& echo "ROCM_SYSTEMS_REPO: ${ROCM_SYSTEMS_REPO}" >> /app/versions.txt \
&& echo "ROCPROFILER_SDK_COMMIT: ${ROCPROFILER_SDK_COMMIT}" >> /app/versions.txt \
&& echo "ROCM_RUNTIME_COMMIT: ${ROCM_RUNTIME_COMMIT}" >> /app/versions.txt \
&& echo "PYTHON_VERSION: ${PYTHON_VERSION}" >> /app/versions.txt \
&& echo "ROCM_SDK_VERSION: ${ROCM_SDK_VERSION}" >> /app/versions.txt \
&& echo "TORCH_VERSION: ${TORCH_VERSION}" >> /app/versions.txt \
&& echo "TORCHAUDIO_VERSION: ${TORCHAUDIO_VERSION}" >> /app/versions.txt \
&& echo "TORCHVISION_VERSION: ${TORCHVISION_VERSION}" >> /app/versions.txt \
&& echo "PYTORCH_ROCM_ARCH: ${PYTORCH_ROCM_ARCH}" >> /app/versions.txt \
&& echo "AITER_ROCM_ARCH: ${AITER_ROCM_ARCH}" >> /app/versions.txt \
&& echo "TRITON_BRANCH: ${TRITON_BRANCH}" >> /app/versions.txt \
&& echo "TRITON_REPO: ${TRITON_REPO}" >> /app/versions.txt \
&& echo "PYTORCH_BRANCH: ${PYTORCH_BRANCH}" >> /app/versions.txt \
&& echo "PYTORCH_VISION_BRANCH: ${PYTORCH_VISION_BRANCH}" >> /app/versions.txt \
&& echo "PYTORCH_REPO: ${PYTORCH_REPO}" >> /app/versions.txt \
&& echo "PYTORCH_VISION_REPO: ${PYTORCH_VISION_REPO}" >> /app/versions.txt \
&& echo "PYTORCH_AUDIO_BRANCH: ${PYTORCH_AUDIO_BRANCH}" >> /app/versions.txt \
&& echo "PYTORCH_AUDIO_REPO: ${PYTORCH_AUDIO_REPO}" >> /app/versions.txt \
&& echo "FA_BRANCH: ${FA_BRANCH}" >> /app/versions.txt \
&& echo "FA_REPO: ${FA_REPO}" >> /app/versions.txt \
&& echo "AITER_BRANCH: ${AITER_BRANCH}" >> /app/versions.txt \
&& echo "AITER_REPO: ${AITER_REPO}" >> /app/versions.txt \
&& echo "MORI_BRANCH: ${MORI_BRANCH}" >> /app/versions.txt \
&& echo "MORI_REPO: ${MORI_REPO}" >> /app/versions.txt
&& echo "MORI_REPO: ${MORI_REPO}" >> /app/versions.txt \
&& echo "ROCM_SYSTEMS_REPO: ${ROCM_SYSTEMS_REPO}" >> /app/versions.txt \
&& echo "ROCM_RUNTIME_COMMIT: ${ROCM_RUNTIME_COMMIT}" >> /app/versions.txt \
# Fingerprint our triton so Dockerfile.rocm can prove it survived the build.
&& echo "TRITON_VERSION: $(pip show triton | awk '/^Version:/ {print $2}')" >> /app/versions.txt \
&& echo "TRITON_SHA256: $(find $(python3 -c 'import triton, os; print(os.path.dirname(triton.__file__))') \
-type f \( -name '*.py' -o -name '*.so' \) | sort | xargs sha256sum | sha256sum | cut -d' ' -f1)" \
>> /app/versions.txt \
&& echo "PYBIND11_VERSION: $(pip show pybind11 | awk '/^Version:/ {print $2}')" >> /app/versions.txt \
&& echo "PYBIND11_INTERNALS_VERSION: $(grep -rhE '^#[[:space:]]*define PYBIND11_INTERNALS_VERSION' \
$(python3 -c 'import pybind11; print(pybind11.get_include())') | awk '{print $NF}')" \
>> /app/versions.txt \
&& echo "PYBIND11_SHA256: $(find $(python3 -c 'import pybind11; print(pybind11.get_include())') \
-type f -name '*.h' | sort | xargs sha256sum | sha256sum | cut -d' ' -f1)" \
>> /app/versions.txt
docker/ci-rocm.hcl
+5 -29
View File
@@ -83,10 +83,6 @@ variable "UCX_BRANCH" {
default = ""
}
variable "ROCSHMEM_BRANCH" {
default = ""
}
variable "DEEPEP_BRANCH" {
default = ""
}
@@ -95,10 +91,6 @@ variable "NIXL_CACHE_KEY" {
default = ""
}
variable "ROCSHMEM_CACHE_KEY" {
default = ""
}
variable "DEEPEP_CACHE_KEY" {
default = ""
}
@@ -236,7 +228,7 @@ function "get_cache_to_rocm_rust" {
])
}
# Cache functions for upstream dependency stages (NIXL/UCX, ROCShmem, DeepEP).
# Cache functions for upstream dependency stages (NIXL/UCX, DeepEP).
# These stages are pinned to specific upstream commit hashes, so cache keys use
# those hashes rather than the Buildkite commit. This means the cache persists
# across all vLLM commits as long as the upstream dependency pins don't change.
@@ -245,8 +237,7 @@ function "get_cache_from_rocm_deps" {
params = []
result = compact([
NIXL_CACHE_KEY != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:nixl-rocm-${NIXL_CACHE_KEY}" : (NIXL_BRANCH != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:nixl-rocm-${NIXL_BRANCH}-ucx-${UCX_BRANCH}" : ""),
ROCSHMEM_CACHE_KEY != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:rocshmem-rocm-${ROCSHMEM_CACHE_KEY}" : (ROCSHMEM_BRANCH != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:rocshmem-rocm-${ROCSHMEM_BRANCH}" : ""),
DEEPEP_CACHE_KEY != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:deepep-rocm-${DEEPEP_CACHE_KEY}" : (DEEPEP_BRANCH != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:deepep-rocm-${DEEPEP_BRANCH}-rocshmem-${ROCSHMEM_BRANCH}" : ""),
DEEPEP_CACHE_KEY != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:deepep-rocm-${DEEPEP_CACHE_KEY}" : (DEEPEP_BRANCH != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:deepep-rocm-${DEEPEP_BRANCH}" : ""),
])
}
@@ -257,17 +248,10 @@ function "get_cache_to_rocm_nixl" {
])
}
function "get_cache_to_rocm_rocshmem" {
params = []
result = compact([
ROCSHMEM_CACHE_KEY != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:rocshmem-rocm-${ROCSHMEM_CACHE_KEY},mode=min" : (ROCSHMEM_BRANCH != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:rocshmem-rocm-${ROCSHMEM_BRANCH},mode=min" : ""),
])
}
function "get_cache_to_rocm_deepep" {
params = []
result = compact([
DEEPEP_CACHE_KEY != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:deepep-rocm-${DEEPEP_CACHE_KEY},mode=min" : (DEEPEP_BRANCH != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:deepep-rocm-${DEEPEP_BRANCH}-rocshmem-${ROCSHMEM_BRANCH},mode=min" : ""),
DEEPEP_CACHE_KEY != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:deepep-rocm-${DEEPEP_CACHE_KEY},mode=min" : (DEEPEP_BRANCH != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:deepep-rocm-${DEEPEP_BRANCH},mode=min" : ""),
])
}
@@ -385,14 +369,6 @@ target "nixl-rocm-ci" {
output = ["type=cacheonly"]
}
target "rocshmem-rocm-ci" {
inherits = ["_common-rocm", "_ci-rocm"]
target = "build_rocshmem"
cache-from = get_cache_from_rocm_deps()
cache-to = get_cache_to_rocm_rocshmem()
output = ["type=cacheonly"]
}
target "deepep-rocm-ci" {
inherits = ["_common-rocm", "_ci-rocm"]
target = "build_deepep"
@@ -418,7 +394,7 @@ target "ci-base-rocm-ci" {
CI_BASE_IMAGE_TAG != "" ? "type=registry,ref=${CI_BASE_IMAGE_TAG}" : "",
CI_BASE_TRUSTED_CONTENT_REF != "" ? "type=registry,ref=${CI_BASE_TRUSTED_CONTENT_REF}" : "",
]),
# Import upstream dependency caches so NIXL/ROCShmem/DeepEP stages
# Import upstream dependency caches so NIXL/DeepEP stages
# are cache hits even when ci_base itself needs rebuilding.
get_cache_from_rocm_deps(),
)
@@ -431,5 +407,5 @@ target "ci-base-rocm-ci" {
# Group for ci_base builds -- exports dependency stage caches alongside the
# ci_base image so future rebuilds can reuse them independently.
group "ci-base-rocm-ci-with-deps" {
targets = ["nixl-rocm-ci", "rocshmem-rocm-ci", "deepep-rocm-ci", "ci-base-rocm-ci"]
targets = ["nixl-rocm-ci", "deepep-rocm-ci", "ci-base-rocm-ci"]
}
docker/docker-bake-rocm.hcl
-4
View File
@@ -65,10 +65,6 @@ variable "UCX_BRANCH" {
default = ""
}
variable "ROCSHMEM_BRANCH" {
default = ""
}
variable "DEEPEP_BRANCH" {
default = ""
}
docs/cli/README.md
+1 -1
View File
@@ -136,7 +136,7 @@ vllm bench {forward, latency, serve, throughput}
### forward
Benchmark the model forward of exact batches: each case lists its requests as `[N*]Q[@K][c][:D]` (query length Q, KV length K including the query, D draft tokens). A prefill's prefix is a prefix-cache hit unless `c` marks a chunked prefill; they differ only on models with SWA bounded replay, which recompute the last window of a hit. Each request's prefix length varies by ±50% by default (`--kv-len-jitter`), identically on every DP rank. Each forward is captured as a one-shot CUDA graph; sampling and CPU overhead aren't measured. Use `torchrun`, one process per GPU, for DP/TP/EP. CUDA only for now.
Benchmark the model forward of exact batches: each case lists its requests as `[N*]Q[@K][c][:D]` (query length Q, KV length K including the query, D draft tokens). A prefill's prefix is a prefix-cache hit unless `c` marks a chunked prefill; they differ only on models with SWA bounded replay, which recompute the last window of a hit. Each request's prefix length varies by ±50% by default (`--kv-len-jitter`), identically on every DP rank. Each forward is captured as a one-shot CUDA graph; sampling and CPU overhead aren't measured. Use `torchrun`, one process per GPU, for DP/TP/EP. CUDA only for now. Add `--profile-dir DIR` to write a Perfetto trace and a per-kernel GPU time summary of each case; `python -m vllm.benchmarks.forward.profile A.kernels.json [B.kernels.json]` prints a summary or compares two. Under Nsight Systems, `nsys profile --capture-range=cudaProfilerApi --capture-range-end=repeat --cuda-graph-trace=node` records just those replays, one report per case.
```bash
vllm bench forward \
docs/getting_started/installation/gpu.rocm.inc.md
+59 -41
View File
@@ -1,7 +1,7 @@
<!-- markdownlint-disable MD041 MD051 -->
--8<-- [start:installation]
vLLM supports AMD GPUs with ROCm 6.3 or above. Pre-built wheels are available for ROCm 7.0 and ROCm 7.2.1.
vLLM supports AMD GPUs with ROCm 6.3 or above. Pre-built wheels are available for ROCm 10.0 (the default) and ROCm 7.2.
#### Prebuilt Wheels
@@ -9,6 +9,8 @@ vLLM supports AMD GPUs with ROCm 6.3 or above. Pre-built wheels are available fo
| ------------ | -------------- | ------------ | ----------------- | ------------------ |
| `rocm700` | 3.12 | 7.0 | >= 2.35 | `0.14.0` to `0.18.0` |
| `rocm721` | 3.12 | 7.2.1 | >= 2.35 | Nightly releases after commit `171775f306a333a9cf105bfd533bf3e113d401d9` |
| `rocm72` | 3.12 | 7.2 | >= 2.39 | Nightly releases on the legacy ROCm 7.2 stack |
| `rocm100` | 3.12 | 10.0 | >= 2.35 | Nightly releases (default) |
--8<-- [end:installation]
--8<-- [start:requirements]
@@ -40,12 +42,17 @@ If you need a different ROCm version or want to use an existing PyTorch installa
source .venv/bin/activate
```
To install the latest version of vLLM for Python 3.12, ROCm 7.0 and `glibc >= 2.35`.
To install the latest version of vLLM for Python 3.12, ROCm 10.0 and `glibc >= 2.35`:
```bash
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/ --upgrade
```
!!! tip
The ROCm 10.0 wheels install the ROCm SDK from pip ([TheRock](https://github.com/ROCm/TheRock)), including the GPU kernels for every supported architecture.
Wheels for the legacy ROCm 7.2 stack, which use a system ROCm installation, remain available at `https://wheels.vllm.ai/rocm/${VLLM_VERSION}/rocm72`.
!!! tip
You can find out about which ROCm version the latest vLLM supports by checking the `vllm` package in index in extra-index-url <https://wheels.vllm.ai/rocm/> at [https://wheels.vllm.ai/rocm/vllm](https://wheels.vllm.ai/rocm/vllm) .
@@ -195,9 +202,9 @@ install it without re-running the full installation:
and place it in `vllm/vllm-rs`.
If you see an error about a wheel not being found, the wheel for your base
commit and ROCm patch version might not be available. Check the available
variants under `https://wheels.vllm.ai/rocm/<commit>/`. For example, ROCm 7.2.1
uses the `rocm721` variant.
commit and ROCm version might not be available. Check the available
variants under `https://wheels.vllm.ai/rocm/<commit>/`. Variants are named by ROCm
major and minor version, e.g. ROCm 7.2.3 uses the `rocm72` variant.
There are more environment variables to control the behavior of Python-only
build:
@@ -208,7 +215,7 @@ build:
- `VLLM_PRECOMPILED_WHEEL_COMMIT`: override the full commit hash used to
download the pre-compiled wheel.
- `VLLM_PRECOMPILED_WHEEL_VARIANT`: specify the ROCm variant subdirectory, e.g.,
`rocm700` or `rocm721`. If not specified, the variant is auto-detected based
`rocm72` or `rocm100`. If not specified, the variant is auto-detected based
on your system's ROCm version. An explicitly specified variant must match the
detected environment.
@@ -226,23 +233,34 @@ You can find more information about vLLM's wheels in
#### Full build (with compilation) {#full-build}
!!! tip
- If you found that the following installation step does not work for you, please refer to [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base). Dockerfile is a form of installation steps.
- The steps below follow [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base), which pins the validated version of every component. If a step does not work for you, refer to it; a Dockerfile is a form of installation steps.
- The legacy ROCm 7.2 stack uses a system ROCm installation instead; its steps are in [docker/Dockerfile.rocm_72_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_72_base).
0. Install prerequisites (skip if you are already in an environment/docker with the following installed):
0. Install the ROCm SDK and PyTorch (skip if you are already in an environment or docker image with them installed, e.g. `vllm/vllm-openai-rocm:base-nightly`).
- [ROCm](https://rocm.docs.amd.com/en/latest/deploy/linux/index.html)
- [PyTorch](https://pytorch.org/)
For installing PyTorch, you can start from a fresh docker image, e.g, `rocm/pytorch:rocm7.0_ubuntu22.04_py3.10_pytorch_release_2.8.0`, `rocm/pytorch-nightly`. If you are using docker image, you can skip to Step 3.
Alternatively, you can install PyTorch using PyTorch wheels. You can check PyTorch installation guide in PyTorch [Getting Started](https://pytorch.org/get-started/locally/). Example:
ROCm 10.0 is installed from pip ([TheRock](https://github.com/ROCm/TheRock)), together with PyTorch and the GPU kernels for your GPU's architecture. In a Python 3.12 environment:
```bash
# Install PyTorch
pip uninstall torch -y
pip install --no-cache-dir torch torchvision --index-url https://download.pytorch.org/whl/nightly/rocm7.0
# Your GPU's architecture, e.g. gfx942 for MI300 or gfx950 for MI350
export GPU_ARCH=$(uvx --from rocm-bootstrap rocm-bootstrap-detect --unique)
export ROCM_INDEX=https://stable.repo.amd.com/rocm/whl-next/
pip install --index-url ${ROCM_INDEX} \
"rocm[libraries,devel,device-${GPU_ARCH}]==10.0.0" \
"torch[device-${GPU_ARCH}]==2.12.0+rocm10.0.0" \
"torchvision[device-${GPU_ARCH}]==0.27.0+rocm10.0.0"
rocm-sdk init
# Point builds at the pip-installed ROCm SDK
export ROCM_PATH=$(rocm-sdk path --root)
export ROCM_HOME=${ROCM_PATH} HIP_PATH=${ROCM_PATH}
export PATH=$(rocm-sdk path --bin):${PATH}
```
!!! note
- The validated `ROCM_SDK_VERSION`, `TORCH_VERSION` and `TORCHVISION_VERSION` can be found in [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base).
- To support several GPU architectures, add a `device-<arch>` extra for each of them.
1. Install [Triton for ROCm](https://github.com/ROCm/triton.git)
Install ROCm's Triton following the instructions from [ROCm/triton](https://github.com/ROCm/triton.git)
@@ -252,8 +270,7 @@ You can find more information about vLLM's wheels in
pip uninstall -y triton
git clone https://github.com/ROCm/triton.git
cd triton
# git checkout $TRITON_BRANCH
git checkout f9e5bf54
git checkout $TRITON_BRANCH
if [ ! -f setup.py ]; then cd python; fi
python3 setup.py install
cd ../..
@@ -265,17 +282,14 @@ You can find more information about vLLM's wheels in
2. Optionally, if you choose to use CK flash attention, you can install [flash attention for ROCm](https://github.com/Dao-AILab/flash-attention.git)
Install ROCm's flash attention (v2.8.0) following the instructions from [ROCm/flash-attention](https://github.com/Dao-AILab/flash-attention#amd-rocm-support)
For example, for ROCm 7.0, suppose your gfx arch is `gfx942`. To get your gfx architecture, run `rocminfo |grep gfx`.
Install ROCm's flash attention following the instructions from [ROCm/flash-attention](https://github.com/Dao-AILab/flash-attention#amd-rocm-support)
```bash
git clone https://github.com/Dao-AILab/flash-attention.git
cd flash-attention
# git checkout $FA_BRANCH
git checkout 0e60e394
git checkout $FA_BRANCH
git submodule update --init
GPU_ARCHS="gfx942" python3 setup.py install
GPU_ARCHS="${GPU_ARCH}" python3 setup.py install
cd ..
```
@@ -290,7 +304,8 @@ You can find more information about vLLM's wheels in
cd aiter
git checkout $AITER_BRANCH_OR_COMMIT
git submodule sync; git submodule update --init --recursive
python3 setup.py develop
# Keep the Triton installed in step 1
AITER_USE_SYSTEM_TRITON=1 python3 setup.py develop
```
!!! note
@@ -300,32 +315,34 @@ You can find more information about vLLM's wheels in
4. Optionally, if you want to use MORI for EP or PD disaggregation, you can install [MORI](https://github.com/ROCm/mori) using the following steps:
```bash
pip install "setuptools_scm>=6.2"
git clone https://github.com/ROCm/mori.git
cd mori
git checkout $MORI_BRANCH_OR_COMMIT
git submodule sync; git submodule update --init --recursive
MORI_GPU_ARCHS="gfx942;gfx950" python3 setup.py install
# The ROCm SDK keeps its CMake configs for system dependencies (e.g. NUMA) under rocm_sysdeps
CMAKE_PREFIX_PATH="${ROCM_PATH}:${ROCM_PATH}/lib/rocm_sysdeps" \
MORI_GPU_ARCHS="gfx942;gfx950" python3 setup.py install
```
!!! note
- You will need to config the `$MORI_BRANCH_OR_COMMIT` for your purpose.
- The validated `$MORI_BRANCH_OR_COMMIT` can be found in the [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base).
5. Build vLLM. For example, vLLM on ROCM 7.0 can be built with the following steps:
5. Build vLLM. For example, vLLM on ROCm 10.0 can be built with the following steps:
???+ console "Commands"
```bash
pip install --upgrade pip
# Build & install AMD SMI
pip install /opt/rocm/share/amd_smi
# Install AMD SMI, which the ROCm SDK ships as sources
SDK_CORE=$(python3 -c "import _rocm_sdk_core, os; print(os.path.dirname(_rocm_sdk_core.__file__))")
pip install ${SDK_CORE}/share/amd_smi
# It loads the SDK's libamd_smi relative to its own directory
export PYTHONPATH=${SDK_CORE}/share/amd_smi
# Install dependencies
pip install --upgrade numba \
scipy \
huggingface-hub \
setuptools_scm
pip install -r requirements/rocm.txt
# To build for a single architecture (e.g., MI300) for faster installation (recommended):
@@ -352,8 +369,9 @@ You can find more information about vLLM's wheels in
vLLM offers official Docker images for deployment.
The images can be used to run OpenAI compatible server and are available on Docker Hub as [vllm/vllm-openai-rocm](https://hub.docker.com/r/vllm/vllm-openai-rocm/tags).
- `vllm/vllm-openai-rocm:latest` — stable release
- `vllm/vllm-openai-rocm:nightly` — preview build from the latest development branch, use this if you want the latest features and fixes
- `vllm/vllm-openai-rocm:latest` — stable release, built on ROCm 10.0 (also tagged `latest-rocm100`)
- `vllm/vllm-openai-rocm:nightly` — preview build from the latest development branch on ROCm 10.0 (also tagged `nightly-rocm100`), use this if you want the latest features and fixes
- `vllm/vllm-openai-rocm:latest-rocm72` / `nightly-rocm72` — the same builds on the legacy ROCm 7.2 stack, kept for a transition period
```bash
docker run --rm \
@@ -409,7 +427,7 @@ You can build and run vLLM from source via the provided [docker/Dockerfile.rocm]
??? info "(Optional) Build an image with ROCm software stack"
Build a docker image from [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base) which setup ROCm software stack needed by the vLLM.
**This step is optional as this rocm_base image is usually prebuilt and store at [Docker Hub](https://hub.docker.com/r/rocm/vllm-dev) under tag `rocm/vllm-dev:base` to speed up user experience.**
**This step is optional as this rocm_base image is usually prebuilt and store at [Docker Hub](https://hub.docker.com/r/rocm/vllm-dev) under tag `rocm/vllm-dev:base` to speed up user experience.** For the legacy ROCm 7.2 stack, the base image is built from [docker/Dockerfile.rocm_72_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_72_base) and published as `vllm/vllm-openai-rocm:base-nightly-rocm72`.
If you choose to build this rocm_base image yourself, the steps are as follows.
It is important that the user kicks off the docker build using buildkit. Either the user put `DOCKER_BUILDKIT=1` as environment variable when calling docker build command, or the user needs to set up buildkit in the docker daemon configuration `/etc/docker/daemon.json` as follows and restart the daemon:
@@ -422,7 +440,7 @@ You can build and run vLLM from source via the provided [docker/Dockerfile.rocm]
}
```
To build vllm on ROCm 7.0 for MI200 and MI300 series, you can use the default:
To build vllm on ROCm 10.0 for all supported GPUs, you can use the default:
```bash
DOCKER_BUILDKIT=1 docker build \
@@ -441,15 +459,15 @@ It is important that the user kicks off the docker build using buildkit. Either
}
```
[docker/Dockerfile.rocm](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm) uses ROCm 7.0 by default, but also supports ROCm 5.7, 6.0, 6.1, 6.2, 6.3, and 6.4, in older vLLM branches.
[docker/Dockerfile.rocm](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm) uses ROCm 10.0 (installed from TheRock wheels) by default. The legacy ROCm 7.2 stack is available as [docker/Dockerfile.rocm_72](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_72) with [docker/Dockerfile.rocm_72_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_72_base); older vLLM branches support ROCm 5.7 through 7.0.
It provides flexibility to customize the build of docker image using the following arguments:
- `BASE_IMAGE`: specifies the base image used when running `docker build`. The default value `rocm/vllm-dev:base` is an image published and maintained by AMD. It is being built using [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base)
- `BASE_IMAGE`: specifies the base image used when running `docker build`. The default value `rocm/vllm-dev:base` is an image published and maintained by AMD. It is being built using [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base). For [docker/Dockerfile.rocm_72](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_72), the default is `vllm/vllm-openai-rocm:base-nightly-rocm72`
- `ARG_PYTORCH_ROCM_ARCH`: Allows to override the gfx architecture values from the base docker image
Their values can be passed in when running `docker build` with `--build-arg` options.
To build vllm on ROCm 7.0 for MI200 and MI300 series, you can use the default (which build a docker image with `vllm serve` as entrypoint):
To build vllm on ROCm 10.0 for all supported GPUs, you can use the default (which build a docker image with `vllm serve` as entrypoint):
```bash
DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.rocm -t vllm/vllm-openai-rocm .
pyproject.toml
+2 -1
View File
@@ -201,7 +201,8 @@ extend-exclude = ["tests/models/fixtures/*", "tests/prompts/*", "tests/tokenizer
"tests/v1/engine/test_fast_incdec_prefix_err.py", ".git/*", "csrc/cpu/sgl-kernels/*",
"rust/src/chat/src/renderer/deepseek_v32/fixtures/*", "rust/src/parser/**",
"rust/src/text/src/output/decoded.rs",
"rust/src/tokenizer/src/incremental.rs"]
"rust/src/tokenizer/src/incremental.rs",
"recipes/models/*"]
ignore-hidden = false
[tool.typos.default]
recipes/README.md
+40
View File
@@ -0,0 +1,40 @@
# vLLM model recipes
This directory holds the deployment recipes for models served by vLLM, one
YAML per model at `models/<hf_org>/<hf_repo>.yaml`. The path mirrors the
Hugging Face ID, so `models/Qwen/Qwen3.8-27B.yaml` renders at
[recipes.vllm.ai/Qwen/Qwen3.8-27B](https://recipes.vllm.ai/Qwen/Qwen3.8-27B).
The site, the command builder, and the JSON API are built by
[vllm-project/recipes](https://github.com/vllm-project/recipes). That
repository owns the schema and the shared vocabulary (hardware profiles,
serving strategies, KV-offload deployments) and pins this directory to a
commit. Omni (image, video, audio) recipes stay in vllm-project/recipes for
now (moving to vllm-omni), and Ascend NPU content is inline in these recipes
(moving to vllm-ascend overlays).
## Editing a recipe
Start a new recipe from
[`templates/recipe.yaml`](https://github.com/vllm-project/recipes/blob/main/templates/recipe.yaml).
Fields are defined in
[SCHEMA.md](https://github.com/vllm-project/recipes/blob/main/SCHEMA.md) and
checked against [`recipe.schema.json`](https://recipes.vllm.ai/recipe.schema.json)
on every build; an unknown or misspelled key is an error. For completion and
inline errors in YAML-aware editors, start the file with
`# yaml-language-server: $schema=https://recipes.vllm.ai/recipe.schema.json`.
Follow the recipes repository's
[add-recipe skill](https://github.com/vllm-project/recipes/blob/main/.agents/skills/add-recipe/SKILL.md)
for the schema and the validation steps. To validate a change exactly as this
repository's `Recipes` PR check does, run from a checkout of
`vllm-project/recipes`:
```bash
node scripts/validate-source.mjs --source vllm --dir ~/vllm/recipes
```
To preview the site with your changes, add a gitignored `sources.local.yaml`
to that checkout with `overrides: { vllm: ~/vllm/recipes }` and run `pnpm dev`.
This works once vllm-project/recipes reads this directory as its `vllm`
source (vllm-project/recipes#1095).
recipes/manifest.yaml
+4
View File
@@ -0,0 +1,4 @@
# Recipe source manifest, read by the vllm-project/recipes renderer
# (scripts/assemble-content.mjs). Its sources.yaml grants this directory's
# role and scope; this file only declares the schema the recipes target.
schema_version: 1
recipes/models/ByteDance-Seed/Seed-OSS-36B-Instruct.yaml
+164
View File
@@ -0,0 +1,164 @@
meta:
title: "Seed-OSS-36B"
slug: "seed-oss-36b"
provider: "Seed (ByteDance)"
description: "ByteDance Seed-OSS 36B dense model with unique 'thinking budget' control and 512K context support"
date_added: 2025-08-28
date_updated: 2026-04-17
tasks:
- text
related_recipes: []
model:
model_id: "ByteDance-Seed/Seed-OSS-36B-Instruct"
min_vllm_version: "0.11.0"
architecture: dense
parameter_count: "36B"
active_parameters: "36B"
context_length: 524288
base_args: []
base_env: {}
dependencies:
- note: "Pinned transformers commit required for Seed-OSS tokenizer compatibility"
command: "uv pip install git+https://github.com/huggingface/transformers.git@56d68c6706ee052b445e1e476056ed92ac5eb383"
features:
tool_calling:
description: "Seed-OSS tool-call parser with automatic tool choice"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "seed_oss"
opt_in_features: []
variants:
default:
precision: bf16
vram_minimum_gb: 86
description: "Native BF16 weights on 8x GPU (TP=8)"
extra_args:
- "--tensor-parallel-size"
- "8"
compatible_strategies:
- single_node_tp
- multi_node_tp
kv_offload_support:
offloading_cpu: verified
offloading_fs: verified
hardware_overrides:
amd:
extra_args: []
extra_env:
VLLM_ROCM_USE_AITER: "1"
strategy_overrides: {}
guide: |
## Overview
Seed-OSS-36B is a dense language model from ByteDance Seed with a unique **thinking
budget** feature for controlled reasoning, and up to 512K context. Users can choose
tensor-parallel (low latency) or data-parallel (high throughput).
## Prerequisites
- Hardware: 8x GPU recommended (TP=8); also runs on consumer hardware like RTX 3090
- vLLM >= 0.11.0 (support may require main branch)
- Latest transformers for compatibility
### Install vLLM (NVIDIA)
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
uv pip install git+https://github.com/huggingface/transformers.git@56d68c6706ee052b445e1e476056ed92ac5eb383
```
### Install vLLM (AMD ROCm)
```bash
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
```
## Launch commands
NVIDIA:
```bash
vllm serve ByteDance-Seed/Seed-OSS-36B-Instruct \
--host localhost --port 8000 \
--tensor-parallel-size 8 \
--enable-auto-tool-choice \
--tool-call-parser seed_oss
```
AMD:
```bash
export VLLM_ROCM_USE_AITER=1
vllm serve ByteDance-Seed/Seed-OSS-36B-Instruct \
--tensor-parallel-size 8 \
--enable-auto-tool-choice --tool-call-parser seed_oss \
--trust-remote-code
```
Tuning:
- `--max-model-len=65536` works well; max is 512K.
- `--max-num-batched-tokens=32768` for prompt-heavy; reduce to 8K–16K for latency.
- `--gpu-memory-utilization=0.95` to maximize KV cache.
## Thinking Budget
Control the model's chain-of-thought length via `chat_template_kwargs`. Recommended
values are multiples of 512 (512, 1K, 2K, 4K, 8K, 16K). Use `0` for direct answers.
```python
from openai import OpenAI
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
model = client.models.list().data[0].id
response = client.chat.completions.create(
model=model,
messages=[
{"role": "system", "content": "You are a helpful assistant"},
{"role": "user", "content": "Janet's ducks lay 16 eggs per day..."},
],
extra_body={"chat_template_kwargs": {"thinking_budget": 512}},
)
print(response.choices[0].message.content)
```
curl:
```bash
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "ByteDance-Seed/Seed-OSS-36B-Instruct",
"messages": [{"role": "user", "content": "Explain quantum computing"}],
"chat_template_kwargs": {"thinking_budget": 512}
}'
```
The model emits `<seed:think>` blocks with `<seed:cot_budget_reflect>` markers that
report token usage against the budget.
## Benchmarking
```bash
vllm bench serve \
--backend vllm --model ByteDance-Seed/Seed-OSS-36B-Instruct \
--endpoint /v1/completions --host localhost --port 8000 \
--dataset-name random --random-input 800 --random-output 100 \
--request-rate 2 --num-prompt 100
```
## References
- [Seed-OSS-36B-Instruct on Hugging Face](https://huggingface.co/ByteDance-Seed/Seed-OSS-36B-Instruct)
recipes/models/Google/diffusiongemma-26B-A4B-it.yaml
+358
View File
@@ -0,0 +1,358 @@
meta:
title: "DiffusionGemma 26B-A4B IT"
slug: "diffusiongemma-26b-a4b-it"
provider: "Google"
description: "Google's DiffusionGemma — a block-diffusion language model built on Gemma 4's MoE backbone (26B total / 4B active). Generates tokens via iterative denoising over a fixed-length canvas rather than left-to-right autoregressive decoding, enabling higher throughput with parallel block generation."
date_added: 2026-06-11
date_updated: 2026-09-23
tasks:
- multimodal
- text
performance_headline: "Block-diffusion MoE — 26B total / 4B active, canvas-based parallel generation with ~1.9x throughput vs autoregressive baseline"
related_recipes:
- google/gemma-4-26B-A4B-it
- google/gemma-4-31B-it
hardware:
h100: verified
h200: verified
dgx_spark_gb10: verified
xeon6: verified
arc_pro_b60: verified
arc_pro_b70: verified
model:
model_id: "google/diffusiongemma-26B-A4B-it"
min_vllm_version: "0.24.0"
docker_image: "vllm/vllm-openai:gemma"
architecture: moe
parameter_count: "26B"
active_parameters: "4B"
context_length: 262144
base_args:
- "--max-num-seqs"
- "4"
base_env: {}
dependencies: []
features:
tool_calling:
description: "Enable automatic tool choice with Gemma 4 parser and chat template"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "gemma4"
- "--chat-template"
- "examples/tool_chat_template_gemma4.jinja"
reasoning:
description: "Enable structured thinking/reasoning output"
args:
- "--reasoning-parser"
- "gemma4"
opt_in_features: []
variants:
default:
precision: bf16
vram_minimum_gb: 64
description: "Full BF16 — single 80 GB NVIDIA GPU (H100/H200). Requires --max-num-seqs 4 to avoid OOM from diffusion state buffers."
supported_hardware: [h100, h200, dgx_spark_gb10, xeon6]
fp8:
model_id: "RedHatAI/diffusiongemma-26B-A4B-it-FP8-dynamic"
precision: fp8
vram_minimum_gb: 32
description: "FP8 (E4M3) weights with dynamic per-token activation quantization"
supported_hardware: [h100, h200, dgx_spark_gb10]
nvfp4:
model_id: "RedHatAI/diffusiongemma-26B-A4B-it-NVFP4"
precision: nvfp4
label: "NVFP4 - RedHatAI"
vram_minimum_gb: 24
description: "NVFP4 (4-bit) quantized weights"
supported_hardware: [dgx_spark_gb10]
nvidia_nvfp4:
model_id: "nvidia/diffusiongemma-26B-A4B-it-NVFP4"
precision: nvfp4
label: "NVFP4 - NVIDIA"
vram_minimum_gb: 24
description: "NVIDIA NVFP4 (4-bit) quantized weights"
supported_hardware: [dgx_spark_gb10]
hardware_overrides:
dgx_spark_gb10:
extra_args:
- "--gpu-memory-utilization"
- "0.8"
- "--max-num-seqs"
- "8"
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "gemma4"
- "--load-format"
- "fastsafetensors"
- "--attention-backend"
- "TRITON_ATTN"
- "--enable-prefix-caching"
- "--diffusion-config"
- '{"canvas_length":256}'
- "--override-generation-config"
- '{"max_new_tokens": null}'
- "--reasoning-parser"
- "gemma4"
- "--default-chat-template-kwargs"
- '{"enable_thinking": true}'
compatible_strategies:
- single_node_tp
hardware_overrides:
cpu: {}
xpu:
extra_args:
- "--enforce-eager"
- "--attention-backend"
- "TRITON_ATTN"
- "--tensor-parallel-size"
- "4"
strategy_overrides:
single_node_tp:
tp: 1
guide: |
## Overview
[DiffusionGemma 26B-A4B](https://huggingface.co/google/diffusiongemma-26B-A4B-it) is a block-diffusion language model built on Gemma 4's MoE backbone. Instead of generating tokens one at a time (autoregressive), it generates blocks of 256 tokens simultaneously via iterative denoising over a fixed canvas — trading higher time-to-first-token for significantly higher per-request generation throughput.
### Key Architecture Features
- **Block Diffusion**: Generates tokens in 256-token canvas blocks via iterative denoising (up to 48 steps per block).
- **MoE Backbone**: Same Gemma 4 architecture — 128 fine-grained experts with top-8 routing, 26B total / 4B active parameters.
- **Entropy-Bound Sampling**: Uses an entropy-based sampler (`diffusion_sampler: entropy_bound`, `entropy_bound: 0.1`) for the denoising process.
- **Multimodal**: Supports text + images via the Gemma 4 vision encoder.
- **Thinking Mode**: Supports structured reasoning via `<|channel>thought\n...<channel|>` delimiters.
- **Function Calling**: Supports Gemma 4's tool-call protocol (works best in thinking mode).
### Important Deployment Notes
DiffusionGemma requires several specific flags that differ from standard Gemma 4:
| Flag | Why |
|------|-----|
| `--max-num-seqs 4` | The diffusion state buffers (`self_conditioning_probs`) pre-allocate `max_seqs × canvas_length × vocab_size` tensors. With Gemma's 262K vocab and canvas_length=256, higher values cause OOM. |
| `--generation-config vllm` | The checkpoint's `generation_config.json` sets `max_tokens: 256`, which would override per-request limits. This flag ignores it. |
| `--gpu-memory-utilization 0.85` | Leaves headroom for activation memory during denoising. |
| `--hf-overrides '{"diffusion_sampler":"entropy_bound","diffusion_entropy_bound":0.1}'` | Configures the entropy-bound denoising sampler. |
| `--diffusion-config '{"canvas_length": 256}'` | Sets the canvas block size for generation. |
## Prerequisites
DiffusionGemma requires a vLLM build with diffusion model support, available in the
Gemma docker image:
```bash
docker pull vllm/vllm-openai:gemma
```
## Deployment
### Single GPU (H100/H200, BF16)
```bash
docker run -itd --name diffusiongemma \
--ipc=host --network host --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:gemma \
--model google/diffusiongemma-26B-A4B-it \
--max-model-len 262144 \
--max-num-seqs 4 \
--gpu-memory-utilization 0.85 \
--host 0.0.0.0 --port 8000
```
### Intel Xeon 6 (CPU, BF16)
DiffusionGemma runs on Intel Xeon 6 CPUs using vLLM's CPU backend with AMX BF16 acceleration.
The generated example uses TP=1. Adjust `--tensor-parallel-size` based on the
detected NUMA topology and deployment requirements.
```bash
docker run \
--privileged --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai-cpu:latest-x86_64 google/diffusiongemma-26B-A4B-it \
--max-num-seqs 4 \
--tensor-parallel-size 1 \
--enable-auto-tool-choice \
--tool-call-parser gemma4 \
--chat-template examples/tool_chat_template_gemma4.jinja \
--reasoning-parser gemma4
```
### Intel XPU (B60 / B70, BF16)
Validated on 4x Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card)
with the official vLLM XPU image `vllm/vllm-openai-xpu:latest`. `--enforce-eager`
and `--attention-backend TRITON_ATTN` are required on this backend. Select Arc Pro
B60/B70 in the hardware picker above for the exact command.
### Full-Featured Server (text + image + thinking + tools)
```bash
docker run -itd --name diffusiongemma \
--ipc=host --network host --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:gemma \
--model google/diffusiongemma-26B-A4B-it \
--max-model-len 262144 \
--max-num-seqs 4 \
--gpu-memory-utilization 0.85 \
--mm-processor-kwargs '{"max_soft_tokens": 1120}' \
--limit-mm-per-prompt '{"image": 7}' \
--enable-auto-tool-choice \
--tool-call-parser gemma4 \
--reasoning-parser gemma4 \
--host 0.0.0.0 --port 8000
```
### Jev-Style Structured Decisions (Nightly)
vLLM's structured-read support for DiffusionGemma can answer yes/no, multiple-choice,
and scored questions with a confidence value. The example server seeds a canvas with
the response template, leaves only the answer slots noisy, and reads a probability
distribution from every slot after one denoising step. It exposes the Jev-compatible
`/v1/systemone` endpoint; this is an example server rather than a standard vLLM endpoint.
Use the pinned nightly image that contains the merged structured-read support:
```bash
docker run -d --name dgemma \
--gpus '"device=0"' \
--ipc=host \
-p 8000:8000 \
-p 8011:8011 \
-v "$HOME/.cache/huggingface:/root/.cache/huggingface" \
vllm/vllm-openai:nightly-e9757321527ca1ecd514c07c1418dd2c53da3d19 \
google/diffusiongemma-26B-A4B-it \
--served-model-name dgemma \
--diffusion-config '{"canvas_length":64}' \
--max-logprobs 32 \
--enable-prefix-caching
```
Wait for the vLLM server to become healthy, then start the example server in the same
container:
```bash
until curl -fsS localhost:8000/health >/dev/null; do sleep 2; done
docker exec -d dgemma \
python /vllm-workspace/examples/features/structured_diffusion/structured_server.py \
--upstream http://127.0.0.1:8000 \
--tokenizer google/diffusiongemma-26B-A4B-it \
--canvas 64
```
After the example server becomes healthy, send a Jev-style decision request:
```bash
until curl -fsS localhost:8011/health >/dev/null; do sleep 2; done
curl -sS localhost:8011/v1/systemone \
-H 'content-type: application/json' \
-d '{
"model": "jev-latest",
"state": {"ticket": "Everything is down, demo at noon."},
"questions": {
"urgent": {
"type": "noul",
"instructions": "Needs a reply within the hour?"
}
}
}'
```
## Client Usage
### Text Generation
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="google/diffusiongemma-26B-A4B-it",
messages=[{"role": "user", "content": "Write a poem about the ocean."}],
max_tokens=512,
temperature=0.7,
)
print(response.choices[0].message.content)
```
### Offline Inference
```python
from vllm import LLM, SamplingParams
from transformers import AutoTokenizer
model_path = "google/diffusiongemma-26B-A4B-it"
tokenizer = AutoTokenizer.from_pretrained(model_path)
llm = LLM(
model=model_path,
max_num_seqs=4,
gpu_memory_utilization=0.85,
hf_overrides={
"diffusion_sampler": "entropy_bound",
"diffusion_entropy_bound": 0.1,
},
diffusion_config={"canvas_length": 256},
)
messages = [{"role": "user", "content": "Explain quantum computing in simple terms."}]
prompt = tokenizer.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
outputs = llm.generate(prompt, SamplingParams(temperature=0.0, max_tokens=1024))
print(outputs[0].outputs[0].text)
```
### Thinking Mode
Launch the server with `--reasoning-parser gemma4`, then enable per-request:
```python
response = client.chat.completions.create(
model="google/diffusiongemma-26B-A4B-it",
messages=[{"role": "user", "content": "What is the derivative of x^3 * ln(x)?"}],
max_tokens=32768,
extra_body={"chat_template_kwargs": {"enable_thinking": True}},
)
```
## Performance Characteristics
Compared to the autoregressive Gemma 4 26B-A4B baseline (single H100, concurrency=1, SPEED-Bench):
| Metric | Gemma 4 26B-A4B (baseline) | DiffusionGemma 26B-A4B |
|--------|---------------------------|------------------------|
| Output TPS | 199 tok/s | 375 tok/s (**1.9×**) |
| E2E Request Time (mean) | 2.87s | 0.88s (**3.3× faster**) |
| TTFT (mean) | 53ms | 489ms (higher due to canvas denoising setup) |
| Per-request Gen TPS (mean) | 205 tok/s | 1,282 tok/s (**6.2×**) |
The diffusion model trades higher time-to-first-token for significantly higher generation throughput. It generates fewer total tokens per prompt on average (shorter outputs) but completes requests much faster.
## Known Limitations
- **TTFT**: ~10× higher than autoregressive baseline — the model must denoise an entire canvas block before emitting the first token.
- **`--max-num-seqs`**: Must be kept low (≤4) due to the large diffusion state tensors. Higher values cause CUDA OOM.
- **Audio**: Not supported — the diffusion checkpoints do not include an audio encoder.
- **Function Calling**: Works best in thinking mode. Non-thinking mode may answer directly instead of emitting tool calls.
## References
- [Model card](https://huggingface.co/google/diffusiongemma-26B-A4B-it)
- [vLLM structured-read support for DiffusionGemma](https://github.com/vllm-project/vllm/pull/57250)
- [djev-run Cloud Run deployment](https://github.com/taeold/djev-run)
- [Gemma 4 26B-A4B (autoregressive baseline)](https://huggingface.co/google/gemma-4-26B-A4B-it)
- [Gemma docs](https://ai.google.dev/gemma/docs)
recipes/models/Google/gemma-4-12B-it.yaml
+316
View File
@@ -0,0 +1,316 @@
meta:
title: "Gemma 4 12B IT"
slug: "gemma-4-12b-it"
provider: "Google"
description: "Google's encoder-free unified Gemma 4 dense model (12B) with native text, image, and audio, plus thinking mode and tool-use protocol."
date_added: 2026-06-03
date_updated: 2026-09-09
tasks:
- multimodal
- text
performance_headline: "Encoder-free unified multimodal model with audio, structured thinking, and function calling — runs on a single 40 GB+ GPU"
related_recipes:
- google/gemma-4-E4B-it
- google/gemma-4-26B-A4B-it
- google/gemma-4-31B-it
hardware:
h100: verified
arc_pro_b60: verified
arc_pro_b70: verified
model:
model_id: "google/gemma-4-12B-it"
min_vllm_version: "0.23.0"
docker_image: "vllm/vllm-openai:gemma4-unified"
install:
docker: {}
pip: {}
architecture: dense
parameter_count: "12B"
active_parameters: "12B"
context_length: 131072
base_args: []
base_env: {}
dependencies:
- note: "Audio extras — only needed when serving the audio modality"
command: 'uv pip install "vllm[audio]"'
optional: true
features:
tool_calling:
description: "Enable automatic tool choice with Gemma 4 parser and chat template"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "gemma4"
- "--chat-template"
- "examples/tool_chat_template_gemma4.jinja"
reasoning:
description: "Enable structured thinking/reasoning output"
args:
- "--reasoning-parser"
- "gemma4"
spec_decoding:
description: "MTP speculative decoding for accelerated inference"
args:
- "--speculative-config"
- '{"model":"google/gemma-4-12B-it-assistant","num_speculative_tokens":4}'
opt_in_features:
- spec_decoding
variants:
default:
precision: bf16
vram_minimum_gb: 29
description: "Full BF16 — single 40 GB+ NVIDIA GPU"
nvfp4:
model_id: "RedHatAI/gemma-4-12B-it-NVFP4"
precision: nvfp4
vram_minimum_gb: 8
description: "RedHatAI compressed-tensors NVFP4 quantization for Blackwell GPUs"
extra_args:
- "--kv-cache-dtype"
- "fp8"
w4a16:
model_id: "google/gemma-4-12B-it-qat-w4a16-ct"
precision: int4
vram_minimum_gb: 9
description: "W4A16 QAT (4-bit weights, 16-bit activations, group_size=32) via compressed-tensors — runs on any GPU"
fp8:
model_id: "RedHatAI/gemma-4-12B-it-FP8-Dynamic"
precision: fp8
vram_minimum_gb: 15
description: "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization"
extra_args:
- "--kv-cache-dtype"
- "fp8"
compatible_strategies:
- single_node_tp
- multi_node_tp
hardware_overrides:
xpu:
extra_args:
- "--enforce-eager"
- "--attention-backend"
- "flash_attn"
- "--trust-remote-code"
- "--gpu-memory-utilization"
- "0.9"
- "--no-enable-prefix-caching"
- "--max-num-batched-tokens"
- "8192"
- "--max-model-len"
- "5120"
- "--tensor-parallel-size"
- "2"
strategy_overrides:
single_node_tp:
tp: 1
guide: |
## Overview
[Gemma 4](https://ai.google.dev/gemma/docs) is Google's most capable open model family, featuring a unified multimodal architecture that natively processes text, images, and audio. The **12B Unified** model is *encoder-free*: instead of dedicated vision/audio encoders, it projects raw image patches and audio waveforms directly into the decoder's embedding space through lightweight linear layers, so every modality flows into a single decoder-only transformer. Gemma 4 models support structured thinking/reasoning, function calling with a custom tool-use protocol, and dynamic vision resolution — all available through vLLM's OpenAI-compatible API.
### Key Features
- **Encoder-free multimodality**: Text, image, and audio inputs handled by a single decoder-only transformer — no separate encoders to load, reducing multimodal latency.
- **Dual Attention**: Alternating sliding-window (local, 1024 tokens) and global attention; the final layer is always global, with unified Keys/Values and Proportional RoPE on global layers for long-context efficiency.
- **Thinking Mode**: Structured reasoning via `<|channel>thought\n...<channel|>` delimiters.
- **Function Calling**: Custom tool-call protocol with dedicated special tokens.
- **Native System Prompt Support**: First-class `system` role.
### Supported Variants
Dense:
- `google/gemma-4-E2B-it` (effective 2B)
- `google/gemma-4-E4B-it` (effective 4B)
- `google/gemma-4-12B-it` (11.95B, encoder-free)
- `google/gemma-4-31B-it` (31B)
MoE:
- `google/gemma-4-26B-A4B-it` (26B total / 4B active)
> **Context length**: this recipe pins `context_length` to the value in `config.json` (`max_position_embeddings = 131072`, i.e. 128K). The Gemma 4 model card markets the 12B at up to 256K — raise `--max-model-len` only if your vLLM build accepts it.
TPU support is provided through [vLLM TPU](https://github.com/vllm-project/tpu-inference) with recipes for [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) and [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/).
> **Nightly required**: support for the encoder-free 12B Unified model landed in [vllm-project/vllm#44429](https://github.com/vllm-project/vllm/pull/44429) and has not yet shipped in a stable release. Use the pinned Docker image (recommended) or a nightly pip wheel.
## Prerequisites
### Docker (recommended)
```bash
docker pull vllm/vllm-openai:gemma4-unified # NVIDIA (CUDA 13; append -cu129 for CUDA 12.9 hosts)
docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70)
```
TPU images are published separately by [vllm-project/tpu-inference](https://github.com/vllm-project/tpu-inference); see the Trillium / Ironwood tpu-recipes below for the pinned tag.
### pip (nightly, NVIDIA CUDA)
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --pre \
--extra-index-url https://wheels.vllm.ai/nightly/cu129 \
--extra-index-url https://download.pytorch.org/whl/cu129 \
--index-strategy unsafe-best-match
```
## Deployment Configurations
### Quick Start (Single GPU, BF16)
The 12B fits comfortably on a single 40 GB+ GPU.
```bash
vllm serve google/gemma-4-12B-it \
--max-model-len 32768 \
--gpu-memory-utilization 0.90
```
### Full-Featured Server Launch
Enables text, image, audio, thinking, and tool calling:
```bash
vllm serve google/gemma-4-12B-it \
--max-model-len 16384 \
--gpu-memory-utilization 0.90 \
--enable-auto-tool-choice \
--reasoning-parser gemma4 \
--tool-call-parser gemma4 \
--chat-template examples/tool_chat_template_gemma4.jinja \
--limit-mm-per-prompt '{"image": 4, "audio": 1}' \
--async-scheduling \
--host 0.0.0.0 \
--port 8000
```
### Docker (NVIDIA)
```bash
docker run -itd --name gemma4 \
--ipc=host --network host --shm-size 16G --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:gemma4-unified \
--model google/gemma-4-12B-it \
--max-model-len 32768 \
--gpu-memory-utilization 0.90 \
--host 0.0.0.0 --port 8000
```
On CUDA 12.9 hosts, use the `vllm/vllm-openai:gemma4-unified-cu129` tag instead.
### Docker (Intel XPU B60 / B70, TP=2, Full-Featured)
Validated on 2× Intel Arc Pro B60 / B70 with the official vLLM XPU image.
```bash
docker run --device /dev/dri \
-v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
--privileged --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai-xpu:latest \
google/gemma-4-12B-it \
--enforce-eager \
--attention-backend flash_attn \
--host 0.0.0.0 \
--port 8000 \
--trust-remote-code \
--enable-auto-tool-choice \
--reasoning-parser gemma4 \
--tool-call-parser gemma4 \
--chat-template examples/tool_chat_template_gemma4.jinja \
--limit-mm-per-prompt '{"image": 4, "audio": 1}' \
--async-scheduling \
--gpu-memory-utilization 0.9 \
--no-enable-prefix-caching \
--max-num-batched-tokens 8192 \
--max-model-len 5120 \
--tensor-parallel-size 2
```
### Docker (Cloud TPU — Trillium / Ironwood)
TPU uses the separate `vllm/vllm-tpu` image (no pip wheel). Pull the tag specified by the upstream [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) or [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/) recipe, then run:
```bash
docker run -itd --name gemma4-tpu \
--privileged --network host --shm-size 16G \
-v /dev/shm:/dev/shm -e HF_TOKEN=$HF_TOKEN \
vllm/vllm-tpu:latest \
--model google/gemma-4-12B-it \
--max-model-len 16384 \
--disable_chunked_mm_input \
--host 0.0.0.0 --port 8000
```
## Client Usage
### Text Generation
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="google/gemma-4-12B-it",
messages=[{"role": "user", "content": "Write a poem about the ocean."}],
max_tokens=512, temperature=0.7,
)
print(response.choices[0].message.content)
```
### Image Understanding
```python
response = client.chat.completions.create(
model="google/gemma-4-12B-it",
messages=[{"role": "user", "content": [
{"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg"}},
{"type": "text", "text": "Describe this image in detail."},
]}],
max_tokens=1024,
)
```
### Audio
Requires `uv pip install "vllm[audio]"`.
```bash
vllm serve google/gemma-4-12B-it \
--max-model-len 8192 \
--limit-mm-per-prompt '{"image": 4, "audio": 1}'
```
### Thinking Mode
```bash
vllm serve google/gemma-4-12B-it \
--max-model-len 16384 \
--enable-auto-tool-choice \
--reasoning-parser gemma4 \
--tool-call-parser gemma4 \
--chat-template examples/tool_chat_template_gemma4.jinja
```
Enable thinking per-request via `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`, or default-on with `--default-chat-template-kwargs '{"enable_thinking": true}'`.
### Structured Outputs
vLLM guided decoding constrains output to a JSON schema. Include semantic instructions in the system prompt — the model does not see schema descriptions.
## Configuration Tips
- Set `--max-model-len` to match your workload.
- `--gpu-memory-utilization 0.90–0.95` maximizes KV cache.
- Image-only workloads: pass `--limit-mm-per-prompt.audio 0`.
- Text-only workloads: pass `--limit-mm-per-prompt '{"image": 0, "audio": 0}'` to skip MM profiling.
- `--async-scheduling` improves throughput.
- FP8 KV cache (`--kv-cache-dtype fp8`) saves ~50% KV memory.
## Speculative Decoding (MTP)
Enable the **Spec Decoding** feature toggle (above) or add `--speculative-config` manually to use MTP drafting with the [assistant model](https://huggingface.co/google/gemma-4-12B-it-assistant). Recommended `num_speculative_tokens`: 4–8 for this model.
> **Note:** MTP speculative decoding for Gemma 4 requires the nightly build — the pinned `vllm/vllm-openai:gemma4-unified` image or a nightly pip wheel. The standard `:latest` stable tag does not include this feature.
## References
- [Model card](https://huggingface.co/google/gemma-4-12B-it)
- [vLLM support PR (#44429)](https://github.com/vllm-project/vllm/pull/44429)
- [Assistant (MTP draft) model](https://huggingface.co/google/gemma-4-12B-it-assistant)
- [Gemma docs](https://ai.google.dev/gemma/docs)
- [vLLM Gemma 4 tool-call template](https://github.com/vllm-project/vllm/blob/main/examples/tool_chat_template_gemma4.jinja)
- [TPU recipes: Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4)
- [TPU recipes: Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/)
recipes/models/Google/gemma-4-26B-A4B-it.yaml
+602
View File
@@ -0,0 +1,602 @@
meta:
title: "Gemma 4 26B-A4B IT"
slug: "gemma-4-26b-a4b-it"
provider: "Google"
description: "Google's Gemma 4 MoE multimodal model (26B total / 4B active) with 128 fine-grained experts, top-8 routing, thinking mode, and tool-use protocol."
date_added: 2026-04-02
date_updated: 2026-09-27
tasks:
- multimodal
- text
performance_headline: "MoE multimodal model — 26B total / 4B active, 128 experts with top-8 routing"
related_recipes:
- google/gemma-4-E2B-it
- google/gemma-4-E4B-it
- google/gemma-4-31B-it
platforms:
- id: modal
blurb: "Serverless deploy via the Gemma 4 Modal script bundled with this recipe."
script: https://github.com/vllm-project/recipes/blob/main/Google/gemma4-modal.py
install: |
curl -O https://raw.githubusercontent.com/vllm-project/recipes/main/Google/gemma4-modal.py
pip install modal
modal setup
modal deploy gemma4-modal.py
modal run gemma4-modal.py
hardware:
h100: verified
h200: verified
mi300x: verified
mi325x: verified
mi355x: verified
trillium: verified
ironwood: verified
xeon6: verified
dgx_spark_gb10: verified
dgx_station_gb300: verified
rtx_pro_6000: verified
rtx_5090: verified
arc_pro_b60: verified
arc_pro_b70: verified
model:
model_id: "google/gemma-4-26B-A4B-it"
min_vllm_version: "0.25.0"
architecture: moe
parameter_count: "26B"
active_parameters: "4B"
context_length: 131072
base_args: []
base_env: {}
dependencies: []
features:
tool_calling:
description: "Enable automatic tool choice with Gemma 4 parser and chat template"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "gemma4"
- "--chat-template"
- "examples/tool_chat_template_gemma4.jinja"
hardware_overrides:
dgx_spark_gb10:
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "gemma4"
reasoning:
description: "Enable structured thinking/reasoning output"
args:
- "--reasoning-parser"
- "gemma4"
text_only:
description: "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
args:
- "--language-model-only"
spec_decoding:
description: "MTP speculative decoding for accelerated inference"
default_mode: mtp
modes:
mtp:
label: "MTP"
description: "Built-in Multi-Token Prediction, 4 draft tokens."
args:
- "--speculative-config"
- '{"model":"google/gemma-4-26B-A4B-it-assistant","num_speculative_tokens":4}'
# The single-GPU Blackwell desktops run the draft MoE on the Triton
# backend — the fused CUTLASS path the drafter picks by default is not
# built for GB10/GB300/RTX PRO in the validated images.
hardware_overrides:
dgx_spark_gb10: &gemma4_mtp_triton
args:
- "--speculative-config"
- '{"method":"mtp","model":"google/gemma-4-26B-A4B-it-assistant","num_speculative_tokens":4,"moe_backend":"triton"}'
dgx_station_gb300: *gemma4_mtp_triton
rtx_pro_6000: *gemma4_mtp_triton
eagle3:
label: "Eagle3"
description: "Eagle3 speculative decoding with externally trained draft model; the served checkpoint is unchanged."
variants:
- default
args:
- "--speculative-config"
- '{"model": "RedHatAI/gemma-4-26B-A4B-it-speculator.eagle3", "num_speculative_tokens": 3, "method": "eagle3"}'
opt_in_features:
- text_only
- spec_decoding
variants:
default:
precision: bf16
vram_minimum_gb: 64
description: "Full BF16 — single 80 GB NVIDIA GPU or 1x MI300X/MI325X/MI350X/MI355X or 2x Xeon6/Xeon5 NUMA nodes"
supported_hardware: [h100, h200, mi300x, mi325x, mi355x, trillium, ironwood, xeon6, dgx_spark_gb10, dgx_station_gb300, rtx_pro_6000, rtx_5090]
hardware_overrides:
dgx_spark_gb10:
extra_env:
VLLM_USE_V2_MODEL_RUNNER: "1"
dgx_station_gb300:
extra_env:
VLLM_USE_V2_MODEL_RUNNER: "1"
rtx_pro_6000:
extra_env:
VLLM_USE_V2_MODEL_RUNNER: "1"
extra_args:
- "--gpu-memory-utilization"
- "0.9"
fp8:
model_id: "RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic"
precision: fp8
vram_minimum_gb: 32
description: "FP8 (E4M3) weights with dynamic per-token activation quantization — Hopper or Blackwell"
supported_hardware: [h100, h200, dgx_spark_gb10, dgx_station_gb300, rtx_pro_6000, rtx_5090]
extra_args:
- "--kv-cache-dtype"
- "fp8"
hardware_overrides:
dgx_spark_gb10:
extra_env:
VLLM_USE_V2_MODEL_RUNNER: "1"
dgx_station_gb300:
extra_env:
VLLM_USE_V2_MODEL_RUNNER: "1"
rtx_pro_6000:
extra_env:
VLLM_USE_V2_MODEL_RUNNER: "1"
extra_args:
- "--gpu-memory-utilization"
- "0.9"
rtx_5090:
extra_env:
VLLM_USE_V2_MODEL_RUNNER: "1"
extra_args:
- "--gpu-memory-utilization"
- "0.92"
- "--max-model-len"
- "131072"
- "--max-num-seqs"
- "1"
nvfp4:
model_id: "RedHatAI/gemma-4-26B-A4B-it-NVFP4"
precision: nvfp4
label: "NVFP4 - RedHatAI"
vram_minimum_gb: 16
description: "NVFP4 (4-bit) quantized weights — requires Blackwell (B200/B300)"
supported_hardware: [dgx_spark_gb10, dgx_station_gb300, rtx_pro_6000, rtx_5090]
extra_args:
- "--kv-cache-dtype"
- "fp8"
hardware_overrides:
dgx_spark_gb10:
extra_env:
VLLM_USE_V2_MODEL_RUNNER: "1"
dgx_station_gb300:
extra_env:
VLLM_USE_V2_MODEL_RUNNER: "1"
rtx_pro_6000:
extra_env:
VLLM_USE_V2_MODEL_RUNNER: "1"
extra_args:
- "--gpu-memory-utilization"
- "0.9"
rtx_5090:
extra_env:
VLLM_USE_V2_MODEL_RUNNER: "1"
extra_args:
- "--gpu-memory-utilization"
- "0.9"
- "--max-num-batched-tokens"
- "4096"
- "--max-num-seqs"
- "4"
nvidia_nvfp4:
model_id: "nvidia/Gemma-4-26B-A4B-NVFP4"
precision: nvfp4
label: "NVFP4 - NVIDIA"
vram_minimum_gb: 16
description: "NVIDIA ModelOpt NVFP4 (4-bit) quantized weights — validated on the single-GPU Blackwell desktops (DGX Spark GB10, DGX Station GB300, RTX PRO 6000)"
supported_hardware: [dgx_spark_gb10, dgx_station_gb300, rtx_pro_6000]
hardware_overrides:
dgx_spark_gb10:
# vLLM 0.24 and the generic nightly both fail on GB10 with this
# checkpoint; the dated ARM64 community build is the tested runtime, so
# the wheel tab is hidden here rather than offering a command that
# won't start.
docker_image: "eugr/spark-vllm:nightly-20260704"
# This image's entrypoint is nvidia_entrypoint.sh, not `vllm serve`.
docker_serve_command: "vllm serve"
install:
pip: false
extra_env:
VLLM_USE_V2_MODEL_RUNNER: "1"
extra_args:
# GB10 shares its 128 GB with the CPU, so leave headroom for the host.
- "--gpu-memory-utilization"
- "0.8"
- "--max-model-len"
- "262144"
- "--max-num-seqs"
- "8"
- "--max-num-batched-tokens"
- "8192"
- "--enable-prefix-caching"
- "--load-format"
- "fastsafetensors"
dgx_station_gb300:
extra_env:
VLLM_USE_V2_MODEL_RUNNER: "1"
extra_args:
# 252 GB on one GB300 — full context at high utilization and a 64-way
# batch.
- "--gpu-memory-utilization"
- "0.92"
- "--max-model-len"
- "262144"
- "--max-num-seqs"
- "64"
- "--max-num-batched-tokens"
- "8192"
- "--enable-chunked-prefill"
- "--enable-prefix-caching"
- "--load-format"
- "fastsafetensors"
rtx_pro_6000:
extra_env:
VLLM_USE_V2_MODEL_RUNNER: "1"
extra_args:
- "--gpu-memory-utilization"
- "0.92"
- "--max-model-len"
- "262144"
- "--max-num-seqs"
- "8"
- "--max-num-batched-tokens"
- "8192"
- "--enable-prefix-caching"
- "--load-format"
- "fastsafetensors"
rtx_5090:
extra_env:
VLLM_USE_V2_MODEL_RUNNER: "1"
extra_args:
- "--kv-cache-dtype"
- "fp8"
- "--gpu-memory-utilization"
- "0.9"
- "--max-num-batched-tokens"
- "4096"
- "--enable-prefix-caching"
- "--enable-chunked-prefill"
- "--async-scheduling"
- "--max-num-seqs"
- "4"
compatible_strategies:
- single_node_tp
- single_node_tep
- single_node_dep
- multi_node_tp
- multi_node_tep
- multi_node_dep
hardware_overrides:
xpu:
extra_args:
- "--enforce-eager"
- "--attention-backend"
- "flash_attn"
- "--trust-remote-code"
- "--gpu-memory-utilization"
- "0.9"
- "--no-enable-prefix-caching"
- "--max-num-batched-tokens"
- "8192"
- "--max-model-len"
- "5120"
- "--tensor-parallel-size"
- "4"
strategy_overrides:
single_node_tp:
tp: 1
guide: |
## Overview
[Gemma 4 26B-A4B](https://huggingface.co/google/gemma-4-26B-A4B-it) is the Mixture-of-Experts member of Google's Gemma 4 family — 26B total parameters with only 4B active per token via 128 fine-grained experts and top-8 routing. It supports text + images natively, structured thinking, function calling, and dynamic vision resolution.
Intel Xeon 6 CPU support remains verified. Benchmark-specific TP/DP, CPU binding,
sequence limits, and repro-only timeouts remain deployment choices.
### Key Features
- **MoE**: 128 fine-grained experts with top-8 routing and custom GELU-activated FFN.
- **Multimodal**: Text + images natively (video via custom frame-extraction pipeline). Audio is only supported on the smaller E2B/E4B variants.
- **Dual Attention**: Alternating sliding-window (local) and global attention with different head dimensions.
- **Thinking Mode**: Structured reasoning via `<|channel>thought\n...<channel|>` delimiters.
- **Function Calling**: Custom tool-call protocol with dedicated special tokens.
- **Dynamic Vision Resolution**: Per-request configurable vision token budget (70, 140, 280, 560, 1120 tokens).
TPU support is provided through [vLLM TPU](https://github.com/vllm-project/tpu-inference) with recipes for [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) and [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/).
## Prerequisites
- Hardware: Intel Xeon 6/Xeon 5 CPUs
- vLLM >= 0.25.0
### pip (NVIDIA CUDA)
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --pre \
--extra-index-url https://wheels.vllm.ai/nightly/cu129 \
--extra-index-url https://download.pytorch.org/whl/cu129 \
--index-strategy unsafe-best-match
```
### pip (AMD ROCm: MI300X, MI325X, MI350X, MI355X)
Requires Python 3.12, ROCm 7.2.1, glibc >= 2.35 (Ubuntu 22.04+).
```bash
uv venv --python 3.12
source .venv/bin/activate
uv pip install vllm --pre \
--extra-index-url https://wheels.vllm.ai/rocm/nightly/rocm721 --upgrade
```
### pip (Intel Xeon 6 CPUs)
For Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions.
### Docker
```bash
docker pull vllm/vllm-openai:gemma4-0505-cu129 # NVIDIA Hopper (H100/H200, CUDA 12.9)
docker pull vllm/vllm-openai:gemma4-0505-cu130 # NVIDIA Blackwell (B200/B300, CUDA 13.0)
docker pull vllm/vllm-openai-rocm:latest # AMD
docker pull vllm/vllm-openai-cpu:latest-x86_64 # For Intel Xeon 6
docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70)
```
TPU images are published separately by [vllm-project/tpu-inference](https://github.com/vllm-project/tpu-inference); see the Trillium / Ironwood tpu-recipes below for the pinned tag.
## Deployment Configurations
### 26B MoE on 1x A100/H100 (BF16)
```bash
vllm serve google/gemma-4-26B-A4B-it \
--max-model-len 32768 \
--gpu-memory-utilization 0.90
```
### Full-Featured Server Launch
Enables text, image, thinking, and tool calling:
```bash
vllm serve google/gemma-4-26B-A4B-it \
--max-model-len 16384 \
--gpu-memory-utilization 0.90 \
--enable-auto-tool-choice \
--reasoning-parser gemma4 \
--tool-call-parser gemma4 \
--chat-template examples/tool_chat_template_gemma4.jinja \
--limit-mm-per-prompt.image 4 \
--async-scheduling \
--host 0.0.0.0 \
--port 8000
```
### Docker (NVIDIA)
```bash
docker run -itd --name gemma4-moe \
--ipc=host --network host --shm-size 16G --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:gemma4-0505-cu129 \
--model google/gemma-4-26B-A4B-it \
--max-model-len 32768 \
--gpu-memory-utilization 0.90 \
--host 0.0.0.0 --port 8000
```
Swap `vllm/vllm-openai:gemma4-0505-cu129` for `vllm/vllm-openai:gemma4-0505-cu130` on Blackwell (B200/B300).
### Docker (AMD MI300X/MI325X/MI350X/MI355X)
```bash
docker run -itd --name gemma4-rocm \
--ipc=host --network=host --privileged \
--cap-add=CAP_SYS_ADMIN --device=/dev/kfd --device=/dev/dri \
--group-add=video --cap-add=SYS_PTRACE \
--security-opt=seccomp=unconfined --shm-size 16G \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai-rocm:latest \
--model google/gemma-4-26B-A4B-it \
--host 0.0.0.0 --port 8000
```
### Docker (Cloud TPU — Trillium / Ironwood)
TPU uses the separate `vllm/vllm-tpu` image (no pip wheel). Pull the tag specified by the upstream [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) or [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/) recipe, then run:
```bash
docker run -itd --name gemma4-tpu \
--privileged --network host --shm-size 16G \
-v /dev/shm:/dev/shm -e HF_TOKEN=$HF_TOKEN \
vllm/vllm-tpu:latest \
--model google/gemma-4-26B-A4B-it \
--tensor-parallel-size 8 \
--max-model-len 16384 \
--disable_chunked_mm_input \
--host 0.0.0.0 --port 8000
```
Trillium requires a 4-chip slice minimum; Ironwood runs on a single chip.
### Intel Xeon 6 Deployment via Docker
Launch the x86 CPU vLLM Docker container for `google/gemma-4-26B-A4B-it`:
```bash
docker run \
--privileged --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai-cpu:latest-x86_64 google/gemma-4-26B-A4B-it \
--tensor-parallel-size 1 \
--enable-auto-tool-choice \
--tool-call-parser gemma4 \
--chat-template examples/tool_chat_template_gemma4.jinja \
--reasoning-parser gemma4
```
For additional Intel Xeon 6 deployment details, see the Intel Software Catalog entries for [Gemma 4 26B-A4B IT](https://aiswcatalog.intel.com/models/google-gemma-4-26b-a4b-it).
The following settings are intentionally not prescribed as portable CPU model
defaults because they depend on the workload, hardware, or runtime environment:
- `--max-num-batched-tokens`: scheduler/throughput tuning.
- `--max-num-seqs`: concurrency and scheduler-capacity tuning.
- `--gpu-memory-utilization`: platform memory-budget tuning.
- `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning.
- `VLLM_ENGINE_ITERATION_TIMEOUT_S`: operational runtime timeout.
These settings may still appear in validated hardware-specific overrides. Tune
them at deployment time based on platform resources, workload shape, and
latency/throughput goals.
### Docker (Intel XPU B60 / B70, TP=4, Full-Featured)
Validated on 4x Intel Arc Pro B60 / B70 with the official vLLM XPU image.
```bash
docker run --device /dev/dri \
-v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
--privileged --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai-xpu:latest \
google/gemma-4-26B-A4B-it \
--enforce-eager \
--attention-backend flash_attn \
--host 0.0.0.0 \
--port 8000 \
--trust-remote-code \
--enable-auto-tool-choice \
--reasoning-parser gemma4 \
--tool-call-parser gemma4 \
--chat-template examples/tool_chat_template_gemma4.jinja \
--limit-mm-per-prompt.image 4 \
--async-scheduling \
--gpu-memory-utilization 0.9 \
--no-enable-prefix-caching \
--max-num-batched-tokens 8192 \
--max-model-len 5120 \
--tensor-parallel-size 4
```
## Client Usage
### Text Generation
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="google/gemma-4-26B-A4B-it",
messages=[{"role": "user", "content": "Write a poem about the ocean."}],
max_tokens=512, temperature=0.7,
)
print(response.choices[0].message.content)
```
### Image Understanding
```python
response = client.chat.completions.create(
model="google/gemma-4-26B-A4B-it",
messages=[{"role": "user", "content": [
{"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg"}},
{"type": "text", "text": "Describe this image in detail."},
]}],
max_tokens=1024,
)
```
### Thinking Mode
```bash
vllm serve google/gemma-4-26B-A4B-it \
--max-model-len 16384 \
--reasoning-parser gemma4 \
--tool-call-parser gemma4 \
--enable-auto-tool-choice \
--chat-template examples/tool_chat_template_gemma4.jinja
```
Enable per-request via `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`.
### Dynamic Vision Resolution
Supported values: 70, 140, 280 (default), 560, 1120 tokens/image.
```bash
vllm serve google/gemma-4-26B-A4B-it \
--mm-processor-kwargs '{"max_soft_tokens": 560}'
```
## Configuration Tips
- Set `--max-model-len` to match your workload.
- `--gpu-memory-utilization 0.90-0.95` maximizes KV cache.
- Text-only workloads: `--limit-mm-per-prompt '{"image": 0, "audio": 0}'`.
- `--async-scheduling` improves throughput.
- FP8 KV cache (`--kv-cache-dtype fp8`) saves ~50% KV memory.
- For MoE, TEP (tensor-expert parallelism) and DEP (data-expert parallelism) strategies scale better than pure TP at large node counts.
## Quantized Variants
Three pre-quantized checkpoints are available:
- [`RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic`](https://huggingface.co/RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic) — FP8 (E4M3) weights with dynamic per-token activation quantization; runs on Hopper and Blackwell.
- [`RedHatAI/gemma-4-26B-A4B-it-NVFP4`](https://huggingface.co/RedHatAI/gemma-4-26B-A4B-it-NVFP4) — NVFP4 (4-bit) weights; requires Blackwell (B200/B300).
- [`nvidia/Gemma-4-26B-A4B-NVFP4`](https://huggingface.co/nvidia/Gemma-4-26B-A4B-NVFP4) — NVIDIA ModelOpt NVFP4 (4-bit) weights; this is the checkpoint validated on the single-GPU Blackwell desktops below.
Pick them from the **Variant** dropdown above, or pass the repo id directly to `vllm serve`.
### NVFP4 on the Blackwell desktops
[`nvidia/Gemma-4-26B-A4B-NVFP4`](https://huggingface.co/nvidia/Gemma-4-26B-A4B-NVFP4)
has been run end to end on DGX Spark (GB10), DGX Station (GB300) and RTX PRO
6000, in each case on one GPU at TP1 with MTP drafting against
[`google/gemma-4-26B-A4B-it-assistant`](https://huggingface.co/google/gemma-4-26B-A4B-it-assistant).
Select the **NVIDIA NVFP4** variant, enable **Spec Decoding**, and switch the
hardware pill — memory utilization, batch size, prefill scheduling and the
drafter's MoE backend all differ per box and are encoded per hardware, so the
generated command is the tested one. Highlights:
- **DGX Spark (GB10)** — 128 GB shared with the CPU, so `--gpu-memory-utilization 0.8`
and an 8-way batch. Serve from `eugr/spark-vllm:nightly-20260704`: vLLM 0.24
and the generic nightly both fail on this checkpoint, so the pip tab is
hidden on this hardware.
- **DGX Station (GB300)** — 252 GB on one GPU, so `--gpu-memory-utilization 0.92`
with chunked prefill and a 64-way batch.
- **RTX PRO 6000** — 96 GB, `--gpu-memory-utilization 0.92` and an 8-way batch.
All three run the full 262,144-token context, above the 131,072 in
`config.json`, and pass `"moe_backend":"triton"` in `--speculative-config`
because the drafter's default fused MoE path is not built in these images.
The GB300 and RTX PRO 6000 runs used an internal NVIDIA nightly vLLM build.
As noted under Speculative Decoding below, Gemma 4 MTP has not landed in a
stable release, so the default `vllm/vllm-openai:latest` tag will not serve
these commands with Spec Decoding enabled — use a nightly wheel or a nightly
image on those two boxes.
## Throughput vs Latency
| Goal | TP | `--max-num-seqs` | Notes |
|------|----|------------------|-------|
| Max throughput | 1-2 | 256-512 | Best tok/s per GPU |
| Min latency | 4-8 | 8-16 | Best TTFT/TPOT |
| Balanced | 2 | 128 | Mixed workloads |
## Speculative Decoding (MTP)
Enable the **Spec Decoding** feature toggle (above) or add `--speculative-config` manually to use MTP drafting with the [assistant model](https://huggingface.co/google/gemma-4-26B-A4B-it-assistant). Recommended `num_speculative_tokens`: 4 for this model. See the [Gemma 4 usage guide](../../Google/Gemma4) for details and benchmarks.
> **Note:** MTP speculative decoding for Gemma 4 is only available on the vLLM nightly build — it has not yet landed in a stable release. Install via the nightly wheel (`uv pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly/cu129 …`) or use the `vllm/vllm-openai:gemma4-0505-cu129` / `vllm/vllm-openai:gemma4-0505-cu130` images above; the standard `:latest` stable tag does not include this feature.
## References
- [Model card](https://huggingface.co/google/gemma-4-26B-A4B-it)
- [FP8 variant](https://huggingface.co/RedHatAI/gemma-4-26B-A4B-it-FP8-dynamic)
- [NVFP4 variant](https://huggingface.co/RedHatAI/gemma-4-26B-A4B-it-NVFP4)
- [NVIDIA ModelOpt NVFP4 variant](https://huggingface.co/nvidia/Gemma-4-26B-A4B-NVFP4)
- [Gemma docs](https://ai.google.dev/gemma/docs)
- [vLLM Gemma 4 tool-call template](https://github.com/vllm-project/vllm/blob/main/examples/tool_chat_template_gemma4.jinja)
- [TPU recipes: Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4)
- [TPU recipes: Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/)
- [Xeon 6 recipe: Gemma 4 26B-A4B IT](https://aiswcatalog.intel.com/models/google-gemma-4-26b-a4b-it)
recipes/models/Google/gemma-4-31B-it.yaml
+425
View File
@@ -0,0 +1,425 @@
meta:
title: "Gemma 4 31B IT"
slug: "gemma-4-31b-it"
provider: "Google"
description: "Google's unified multimodal Gemma 4 dense model (31B) with native text, image, and audio, plus thinking mode and tool-use protocol."
date_added: 2026-04-02
date_updated: 2026-09-09
tasks:
- multimodal
- text
performance_headline: "Unified multimodal model with structured thinking, function calling, dynamic vision resolution"
related_recipes: []
hardware:
h100: verified
mi300x: verified
mi325x: verified
mi355x: verified
trillium: verified
ironwood: verified
arc_pro_b60: verified
arc_pro_b70: verified
model:
model_id: "google/gemma-4-31B-it"
min_vllm_version: "0.19.1"
architecture: dense
parameter_count: "31B"
active_parameters: "31B"
context_length: 262144
base_args: []
base_env: {}
dependencies:
- note: "Audio extras — only needed when serving the audio modality"
command: 'uv pip install "vllm[audio]"'
optional: true
features:
tool_calling:
description: "Enable automatic tool choice with Gemma 4 parser and chat template"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "gemma4"
- "--chat-template"
- "examples/tool_chat_template_gemma4.jinja"
reasoning:
description: "Enable structured thinking/reasoning output"
args:
- "--reasoning-parser"
- "gemma4"
text_only:
description: "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
args:
- "--language-model-only"
spec_decoding:
description: "MTP speculative decoding for accelerated inference"
default_mode: mtp
modes:
mtp:
label: "MTP"
description: "Built-in Multi-Token Prediction, 4 draft tokens."
args:
- "--speculative-config"
- '{"model":"google/gemma-4-31B-it-assistant","num_speculative_tokens":4}'
dflash:
label: "DFlash"
description: "DFlash speculative decoding with an externally trained draft model; the served checkpoint is unchanged."
variants:
- default
args:
- "--speculative-config"
- '{"model": "RedHatAI/gemma-4-31B-it-speculator.dflash", "num_speculative_tokens": 7, "method": "dflash"}'
dspark:
label: "DSpark"
description: "DSpark drafting with an externally trained draft model; the served checkpoint is unchanged."
variants:
- default
args:
- "--speculative-config"
- '{"model": "RedHatAI/gemma-4-31B-it-speculator.dspark", "num_speculative_tokens": 7, "method": "dspark"}'
eagle3:
label: "Eagle3"
description: "Eagle3 speculative decoding with externally trained draft model; the served checkpoint is unchanged."
variants:
- default
args:
- "--speculative-config"
- '{"model": "RedHatAI/gemma-4-31B-it-speculator.eagle3", "num_speculative_tokens": 3, "method": "eagle3"}'
opt_in_features:
- text_only
- spec_decoding
variants:
default:
precision: bf16
vram_minimum_gb: 75
description: "Full BF16 — single 80 GB NVIDIA GPU or 1x MI300X/MI325X/MI350X/MI355X"
fp8:
model_id: "RedHatAI/gemma-4-31B-it-FP8-dynamic"
precision: fp8
vram_minimum_gb: 38
description: "FP8 (E4M3) linear weights with dynamic per-token activation quantization (vision tower stays BF16) — Hopper or Blackwell"
extra_args:
- "--kv-cache-dtype"
- "fp8"
nvfp4:
model_id: "nvidia/gemma-4-31B-it-NVFP4"
precision: nvfp4
vram_minimum_gb: 19
description: "NVIDIA NVFP4 quantized weights for Blackwell GPUs"
extra_args:
- "--kv-cache-dtype"
- "fp8"
redhat_nvfp4:
model_id: "RedHatAI/gemma-4-31B-it-NVFP4"
precision: nvfp4
label: "NVFP4 (RedHat)"
vram_minimum_gb: 19
description: "RedHatAI compressed-tensors NVFP4 quantization for Blackwell GPUs"
extra_args:
- "--kv-cache-dtype"
- "fp8"
w4a16:
model_id: "google/gemma-4-31B-it-qat-w4a16-ct"
precision: int4
vram_minimum_gb: 20
description: "W4A16 QAT (4-bit weights, 16-bit activations, group_size=32) via compressed-tensors — runs on any GPU"
redhat_fp8:
model_id: "RedHatAI/gemma-4-31B-it-FP8-block"
precision: fp8
label: "FP8 Block (RedHat)"
vram_minimum_gb: 38
description: "RedHatAI FP8 block quantization"
extra_args:
- "--kv-cache-dtype"
- "fp8"
compatible_strategies:
- single_node_tp
- multi_node_tp
hardware_overrides:
xpu:
extra_args:
- "--enforce-eager"
- "--attention-backend"
- "flash_attn"
- "--trust-remote-code"
- "--gpu-memory-utilization"
- "0.9"
- "--no-enable-prefix-caching"
- "--max-num-batched-tokens"
- "8192"
- "--max-model-len"
- "5120"
- "--tensor-parallel-size"
- "4"
strategy_overrides:
single_node_tp:
tp: 1
guide: |
## Overview
[Gemma 4](https://ai.google.dev/gemma/docs) is Google's most capable open model family, featuring a unified multimodal architecture that natively processes text, images, and audio. Gemma 4 models support structured thinking/reasoning, function calling with a custom tool-use protocol, and dynamic vision resolution — all available through vLLM's OpenAI-compatible API.
### Key Features
- **Multimodal**: Text + images natively (video via custom frame-extraction pipeline). The smaller E2B and E4B models also support audio.
- **MoE variant**: 128 fine-grained experts with top-8 routing and custom GELU-activated FFN (Gemma 4 26B-A4B).
- **Dual Attention**: Alternating sliding-window (local) and global attention with different head dimensions.
- **Thinking Mode**: Structured reasoning via `<|channel>thought\n...<channel|>` delimiters.
- **Function Calling**: Custom tool-call protocol with dedicated special tokens.
- **Dynamic Vision Resolution**: Per-request configurable vision token budget (70, 140, 280, 560, 1120 tokens).
### Supported Variants
Dense:
- `google/gemma-4-E2B-it` (effective 2B)
- `google/gemma-4-E4B-it` (effective 4B)
- `google/gemma-4-31B-it` (31B)
MoE:
- `google/gemma-4-26B-A4B-it` (26B total / 4B active)
TPU support is provided through [vLLM TPU](https://github.com/vllm-project/tpu-inference) with recipes for [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) and [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/).
## Prerequisites
### pip (NVIDIA CUDA)
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --pre \
--extra-index-url https://wheels.vllm.ai/nightly/cu129 \
--extra-index-url https://download.pytorch.org/whl/cu129 \
--index-strategy unsafe-best-match
```
### pip (AMD ROCm: MI300X, MI325X, MI350X, MI355X)
Requires Python 3.12, ROCm 7.2.1, glibc >= 2.35 (Ubuntu 22.04+).
```bash
uv venv --python 3.12
source .venv/bin/activate
uv pip install vllm --pre \
--extra-index-url https://wheels.vllm.ai/rocm/nightly/rocm721 --upgrade
```
### Docker
```bash
docker pull vllm/vllm-openai:gemma4-0505-cu129 # NVIDIA Hopper (H100/H200, CUDA 12.9)
docker pull vllm/vllm-openai:gemma4-0505-cu130 # NVIDIA Blackwell (B200/B300, CUDA 13.0)
docker pull vllm/vllm-openai-rocm:latest # AMD
docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70)
```
TPU images are published separately by [vllm-project/tpu-inference](https://github.com/vllm-project/tpu-inference); see the Trillium / Ironwood tpu-recipes below for the pinned tag.
## Deployment Configurations
### Quick Start (Single GPU)
```bash
vllm serve google/gemma-4-E4B-it \
--max-model-len <n_of_tokens> # up to 131072
```
### 31B Dense on 2xA100/H100 (TP=2, BF16)
```bash
vllm serve google/gemma-4-31B-it \
--tensor-parallel-size 2 \
--max-model-len 32768 \
--gpu-memory-utilization 0.90
```
### 26B MoE on 1xA100/H100 (BF16)
```bash
vllm serve google/gemma-4-26B-A4B-it \
--max-model-len 32768 \
--gpu-memory-utilization 0.90
```
### Full-Featured Server Launch
Enables text, image, audio, thinking, and tool calling:
```bash
vllm serve google/gemma-4-31B-it \
--tensor-parallel-size 2 \
--max-model-len 16384 \
--gpu-memory-utilization 0.90 \
--enable-auto-tool-choice \
--reasoning-parser gemma4 \
--tool-call-parser gemma4 \
--chat-template examples/tool_chat_template_gemma4.jinja \
--limit-mm-per-prompt '{"image": 4, "audio": 1}' \
--async-scheduling \
--host 0.0.0.0 \
--port 8000
```
### Docker (NVIDIA)
```bash
docker run -itd --name gemma4 \
--ipc=host --network host --shm-size 16G --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:gemma4-0505-cu129 \
--model google/gemma-4-31B-it \
--tensor-parallel-size 2 \
--max-model-len 32768 \
--gpu-memory-utilization 0.90 \
--host 0.0.0.0 --port 8000
```
Swap `vllm/vllm-openai:gemma4-0505-cu129` for `vllm/vllm-openai:gemma4-0505-cu130` on Blackwell (B200/B300).
### Docker (AMD MI300X/MI325X/MI350X/MI355X)
```bash
docker run -itd --name gemma4-rocm \
--ipc=host --network=host --privileged \
--cap-add=CAP_SYS_ADMIN --device=/dev/kfd --device=/dev/dri \
--group-add=video --cap-add=SYS_PTRACE \
--security-opt=seccomp=unconfined --shm-size 16G \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai-rocm:latest \
--model google/gemma-4-31B-it \
--host 0.0.0.0 --port 8000
```
### Docker (Intel XPU B60 / B70, TP=4, Full-Featured)
Validated on 4× Intel Arc Pro B60 / B70 with the official vLLM XPU image.
```bash
docker run --device /dev/dri \
-v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
--privileged --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai-xpu:latest \
google/gemma-4-31B-it \
--enforce-eager \
--attention-backend flash_attn \
--host 0.0.0.0 \
--port 8000 \
--trust-remote-code \
--enable-auto-tool-choice \
--reasoning-parser gemma4 \
--tool-call-parser gemma4 \
--chat-template examples/tool_chat_template_gemma4.jinja \
--limit-mm-per-prompt '{"image": 4, "audio": 1}' \
--async-scheduling \
--gpu-memory-utilization 0.9 \
--no-enable-prefix-caching \
--max-num-batched-tokens 8192 \
--max-model-len 5120 \
--tensor-parallel-size 4
```
### Docker (Cloud TPU — Trillium / Ironwood)
TPU uses the separate `vllm/vllm-tpu` image (no pip wheel). Pull the tag specified by the upstream [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) or [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/) recipe, then run:
```bash
docker run -itd --name gemma4-tpu \
--privileged --network host --shm-size 16G \
-v /dev/shm:/dev/shm -e HF_TOKEN=$HF_TOKEN \
vllm/vllm-tpu:latest \
--model google/gemma-4-31B-it \
--tensor-parallel-size 8 \
--max-model-len 16384 \
--disable_chunked_mm_input \
--host 0.0.0.0 --port 8000
```
## Client Usage
### Text Generation
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="google/gemma-4-31B-it",
messages=[{"role": "user", "content": "Write a poem about the ocean."}],
max_tokens=512, temperature=0.7,
)
print(response.choices[0].message.content)
```
### Image Understanding
```python
response = client.chat.completions.create(
model="google/gemma-4-31B-it",
messages=[{"role": "user", "content": [
{"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg"}},
{"type": "text", "text": "Describe this image in detail."},
]}],
max_tokens=1024,
)
```
### Dynamic Vision Resolution
Supported values: 70, 140, 280 (default), 560, 1120 tokens/image.
```bash
vllm serve google/gemma-4-31B-it \
--mm-processor-kwargs '{"max_soft_tokens": 560}'
```
### Audio (E2B / E4B)
Requires `uv pip install "vllm[audio]"`.
```bash
vllm serve google/gemma-4-E2B-it \
--max-model-len 8192 \
--limit-mm-per-prompt '{"image": 4, "audio": 1}'
```
### Thinking Mode
```bash
vllm serve google/gemma-4-31B-it \
--max-model-len 16384 \
--enable-auto-tool-choice \
--reasoning-parser gemma4 \
--tool-call-parser gemma4 \
--chat-template examples/tool_chat_template_gemma4.jinja
```
Enable thinking per-request via `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`, or default-on with `--default-chat-template-kwargs '{"enable_thinking": true}'`.
### Structured Outputs
vLLM guided decoding constrains output to a JSON schema. Include semantic instructions in the system prompt — the model does not see schema descriptions.
## Configuration Tips
- Set `--max-model-len` to match your workload.
- `--gpu-memory-utilization 0.90–0.95` maximizes KV cache.
- Image-only workloads: pass `--limit-mm-per-prompt.audio 0`.
- Text-only workloads: pass `--limit-mm-per-prompt '{"image": 0, "audio": 0}'` to skip MM profiling.
- `--async-scheduling` improves throughput.
- FP8 KV cache (`--kv-cache-dtype fp8`) saves ~50% KV memory.
## Quantized Variants
Two pre-quantized checkpoints are available:
- [`RedHatAI/gemma-4-31B-it-FP8-dynamic`](https://huggingface.co/RedHatAI/gemma-4-31B-it-FP8-dynamic) — FP8 (E4M3) linear weights with dynamic per-token activation quantization (vision tower stays BF16); runs on Hopper and Blackwell.
- [`nvidia/gemma-4-31B-it-NVFP4`](https://huggingface.co/nvidia/gemma-4-31B-it-NVFP4) — NVFP4 (4-bit) weights; requires Blackwell (B200/B300).
Pick them from the **Variant** dropdown above, or pass the repo id directly to `vllm serve`.
## Throughput vs Latency
| Goal | TP | `--max-num-seqs` | Notes |
|------|----|------------------|-------|
| Max throughput | 1-2 | 256-512 | Best tok/s per GPU |
| Min latency | 4-8 | 8-16 | Best TTFT/TPOT |
| Balanced | 2 | 128 | Mixed workloads |
## Speculative Decoding (MTP)
Enable the **Spec Decoding** feature toggle (above) or add `--speculative-config` manually to use MTP drafting with the [assistant model](https://huggingface.co/google/gemma-4-31B-it-assistant). Recommended `num_speculative_tokens`: 4–8 for this model. See the [Gemma 4 usage guide](../../Google/Gemma4) for details and benchmarks.
> **Note:** MTP speculative decoding for Gemma 4 is only available on the vLLM nightly build — it has not yet landed in a stable release. Install via the nightly wheel (`uv pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly/cu129 …`) or use the `vllm/vllm-openai:gemma4-0505-cu129` / `vllm/vllm-openai:gemma4-0505-cu130` images above; the standard `:latest` stable tag does not include this feature.
## References
- [Model card](https://huggingface.co/google/gemma-4-31B-it)
- [FP8 variant](https://huggingface.co/RedHatAI/gemma-4-31B-it-FP8-dynamic)
- [NVFP4 variant](https://huggingface.co/nvidia/gemma-4-31B-it-NVFP4)
- [Gemma docs](https://ai.google.dev/gemma/docs)
- [vLLM Gemma 4 tool-call template](https://github.com/vllm-project/vllm/blob/main/examples/tool_chat_template_gemma4.jinja)
- [TPU recipes: Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4)
- [TPU recipes: Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/)
recipes/models/Google/gemma-4-E2B-it.yaml
+322
View File
@@ -0,0 +1,322 @@
meta:
title: "Gemma 4 E2B IT"
slug: "gemma-4-e2b-it"
provider: "Google"
description: "Google's compact Gemma 4 multimodal model (effective 2B) with native text, image, and audio, plus thinking mode and tool-use protocol."
date_added: 2026-04-02
date_updated: 2026-09-12
tasks:
- multimodal
- text
performance_headline: "Compact unified multimodal model with audio, thinking, and function calling — runs on a single 24 GB+ GPU"
related_recipes:
- google/gemma-4-E4B-it
- google/gemma-4-31B-it
- google/gemma-4-26B-A4B-it
hardware:
h100: verified
mi300x: verified
mi325x: verified
mi355x: verified
trillium: verified
ironwood: verified
xeon6: verified
model:
model_id: "google/gemma-4-E2B-it"
min_vllm_version: "0.19.1"
architecture: dense
parameter_count: "5B"
active_parameters: "5B"
context_length: 131072
base_args: []
base_env: {}
dependencies:
- note: "Audio extras — needed to serve the audio modality"
command: 'uv pip install "vllm[audio]"'
optional: true
features:
tool_calling:
description: "Enable automatic tool choice with Gemma 4 parser and chat template"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "gemma4"
- "--chat-template"
- "examples/tool_chat_template_gemma4.jinja"
reasoning:
description: "Enable structured thinking/reasoning output"
args:
- "--reasoning-parser"
- "gemma4"
text_only:
description: "Skip loading the vision and audio encoders for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
args:
- "--language-model-only"
spec_decoding:
description: "MTP speculative decoding with centroids masking for accelerated inference"
args:
- "--speculative-config"
- '{"model":"google/gemma-4-E2B-it-assistant","num_speculative_tokens":2}'
opt_in_features:
- text_only
- spec_decoding
variants:
default:
precision: bf16
vram_minimum_gb: 13
description: "Full BF16 — single 24 GB+ NVIDIA GPU or 1x MI300X/MI325X/MI350X/MI355X or 1x Xeon6/Xeon5 NUMA node"
supported_hardware: [h100, mi300x, mi325x, mi355x, trillium, ironwood, xeon6]
fp8:
model_id: "RedHatAI/gemma-4-E2B-it-FP8-dynamic"
precision: fp8
vram_minimum_gb: 6
description: "FP8 (E4M3) linear weights with dynamic per-token activation quantization (vision/audio encoders stay BF16) — Hopper or Blackwell"
supported_hardware: [h100]
extra_args:
- "--kv-cache-dtype"
- "fp8"
w4a16:
model_id: "google/gemma-4-E2B-it-qat-w4a16-ct"
precision: int4
vram_minimum_gb: 8
description: "W4A16 QAT (4-bit weights, 16-bit activations, group_size=32) via compressed-tensors — runs on any GPU"
supported_hardware: [h100, mi300x, mi325x, mi355x]
mobile:
model_id: "google/gemma-4-E2B-it-qat-mobile-ct"
precision: int2/4/8
vram_minimum_gb: 4
description: "Mobile QAT — mixed int2/4/8 compressed-tensors (int2 embeddings + lm_head + deep MLP, int4 attention + early MLP, int8 vision + gates). 3.6× weight compression vs BF16, up to 1.25× faster decode at low batch"
supported_hardware: [h100, mi300x, mi325x, mi355x]
compatible_strategies:
- single_node_tp
- multi_node_tp
strategy_overrides:
single_node_tp:
tp: 1
guide: |
## Overview
[Gemma 4 E2B](https://huggingface.co/google/gemma-4-E2B-it) is the smallest member of Google's Gemma 4 family — an effective-2B unified multimodal model that natively processes text, images, and audio, with structured thinking/reasoning, function calling, and dynamic vision resolution. It runs comfortably on a single 24 GB+ GPU.
Intel Xeon 6 CPU support remains verified. No additional benchmark-specific CPU
tuning is prescribed for general deployment.
### Key Features
- **Multimodal**: Text + images + audio natively (video via custom frame-extraction pipeline).
- **Dual Attention**: Alternating sliding-window (local) and global attention with different head dimensions.
- **Thinking Mode**: Structured reasoning via `<|channel>thought\n...<channel|>` delimiters.
- **Function Calling**: Custom tool-call protocol with dedicated special tokens.
- **Dynamic Vision Resolution**: Per-request configurable vision token budget (70, 140, 280, 560, 1120 tokens).
TPU support is provided through [vLLM TPU](https://github.com/vllm-project/tpu-inference) with recipes for [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) and [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/).
## Prerequisites
- Hardware: Intel Xeon 6/Xeon 5 CPUs
- vLLM >= 0.19.1
### pip (NVIDIA CUDA)
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --pre \
--extra-index-url https://wheels.vllm.ai/nightly/cu129 \
--extra-index-url https://download.pytorch.org/whl/cu129 \
--index-strategy unsafe-best-match
```
### pip (AMD ROCm: MI300X, MI325X, MI350X, MI355X)
Requires Python 3.12, ROCm 7.2.1, glibc >= 2.35 (Ubuntu 22.04+).
```bash
uv venv --python 3.12
source .venv/bin/activate
uv pip install vllm --pre \
--extra-index-url https://wheels.vllm.ai/rocm/nightly/rocm721 --upgrade
```
### pip (Intel Xeon 6 CPUs)
For Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions.
### Docker
```bash
docker pull vllm/vllm-openai:gemma4-0505-cu129 # NVIDIA Hopper (H100/H200, CUDA 12.9)
docker pull vllm/vllm-openai:gemma4-0505-cu130 # NVIDIA Blackwell (B200/B300, CUDA 13.0)
docker pull vllm/vllm-openai-rocm:latest # AMD
docker pull vllm/vllm-openai-cpu:latest-x86_64 # For Intel Xeon 6
```
## Deployment Configurations
### Quick Start (Single GPU)
```bash
vllm serve google/gemma-4-E2B-it \
--max-model-len 32768
```
### With Audio Support
```bash
vllm serve google/gemma-4-E2B-it \
--max-model-len 8192 \
--limit-mm-per-prompt '{"image": 4, "audio": 1}'
```
### Full-Featured Server Launch
Enables text, image, audio, thinking, and tool calling:
```bash
vllm serve google/gemma-4-E2B-it \
--max-model-len 16384 \
--gpu-memory-utilization 0.90 \
--enable-auto-tool-choice \
--reasoning-parser gemma4 \
--tool-call-parser gemma4 \
--chat-template examples/tool_chat_template_gemma4.jinja \
--limit-mm-per-prompt '{"image": 4, "audio": 1}' \
--async-scheduling \
--host 0.0.0.0 \
--port 8000
```
### Docker (NVIDIA)
```bash
docker run -itd --name gemma4-e2b \
--ipc=host --network host --shm-size 16G --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:gemma4-0505-cu129 \
--model google/gemma-4-E2B-it \
--max-model-len 32768 \
--host 0.0.0.0 --port 8000
```
Swap `vllm/vllm-openai:gemma4-0505-cu129` for `vllm/vllm-openai:gemma4-0505-cu130` on Blackwell (B200/B300).
### Docker (AMD MI300X/MI325X/MI350X/MI355X)
```bash
docker run -itd --name gemma4-rocm \
--ipc=host --network=host --privileged \
--cap-add=CAP_SYS_ADMIN --device=/dev/kfd --device=/dev/dri \
--group-add=video --cap-add=SYS_PTRACE \
--security-opt=seccomp=unconfined --shm-size 16G \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai-rocm:latest \
--model google/gemma-4-E2B-it \
--host 0.0.0.0 --port 8000
```
### Docker (Cloud TPU — Trillium / Ironwood)
TPU uses the separate `vllm/vllm-tpu` image (no pip wheel). Pull the tag specified by the upstream [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) or [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/) recipe, then run:
```bash
docker run -itd --name gemma4-tpu \
--privileged --network host --shm-size 16G \
-v /dev/shm:/dev/shm -e HF_TOKEN=$HF_TOKEN \
vllm/vllm-tpu:latest \
--model google/gemma-4-E2B-it \
--max-model-len 16384 \
--disable_chunked_mm_input \
--host 0.0.0.0 --port 8000
```
### Intel Xeon 6 Deployment via Docker
Launch the x86 CPU vLLM Docker container for `google/gemma-4-E2B-it`:
```bash
docker run \
--privileged --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai-cpu:latest-x86_64 google/gemma-4-E2B-it \
--tensor-parallel-size 1 \
--enable-auto-tool-choice \
--tool-call-parser gemma4 \
--chat-template examples/tool_chat_template_gemma4.jinja \
--reasoning-parser gemma4
```
The following settings are intentionally not prescribed as portable CPU model
defaults because they depend on the workload, hardware, or runtime environment:
- `--max-num-batched-tokens`: scheduler/throughput tuning.
- `--max-num-seqs`: concurrency and scheduler-capacity tuning.
- `--gpu-memory-utilization`: platform memory-budget tuning.
- `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning.
- `VLLM_ENGINE_ITERATION_TIMEOUT_S`: operational runtime timeout.
These settings may still appear in validated hardware-specific overrides. Tune
them at deployment time based on platform resources, workload shape, and
latency/throughput goals.
## Client Usage
### Audio Transcription
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="google/gemma-4-E2B-it",
messages=[{"role": "user", "content": [
{"type": "audio_url", "audio_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/2/22/Beatbox_by_Wikipedia_user_Wikipedia_Brown.ogg"}},
{"type": "text", "text": "Provide a verbatim, word-for-word transcription of the audio."},
]}],
max_tokens=512,
)
print(response.choices[0].message.content)
```
### Image Understanding
```python
response = client.chat.completions.create(
model="google/gemma-4-E2B-it",
messages=[{"role": "user", "content": [
{"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg"}},
{"type": "text", "text": "Describe this image in detail."},
]}],
max_tokens=1024,
)
```
### Thinking Mode
Launch with reasoning parser, then enable per-request:
```bash
vllm serve google/gemma-4-E2B-it \
--max-model-len 16384 \
--reasoning-parser gemma4 \
--tool-call-parser gemma4 \
--enable-auto-tool-choice \
--chat-template examples/tool_chat_template_gemma4.jinja
```
Enable per-request via `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`.
## Configuration Tips
- Set `--max-model-len` to match your workload (max 131072).
- Image-only workloads: `--limit-mm-per-prompt.audio 0`.
- Text-only workloads: `--limit-mm-per-prompt '{"image": 0, "audio": 0}'` to skip MM profiling.
- `--async-scheduling` improves throughput.
- FP8 KV cache (`--kv-cache-dtype fp8`) saves ~50% KV memory.
## Quantized Variant
[`RedHatAI/gemma-4-E2B-it-FP8-dynamic`](https://huggingface.co/RedHatAI/gemma-4-E2B-it-FP8-dynamic) is a pre-quantized FP8 (E4M3) checkpoint — linear weights with dynamic per-token activation quantization, vision/audio encoders kept in BF16. Runs on Hopper and Blackwell. Pick the **fp8** variant above, or pass the repo id directly to `vllm serve`.
## Speculative Decoding (MTP)
Enable the **Spec Decoding** feature toggle (above) or add `--speculative-config` manually to use MTP drafting with the [assistant model](https://huggingface.co/google/gemma-4-E2B-it-assistant). Recommended `num_speculative_tokens`: 2 for this model. The E2B assistant uses centroids masking for efficient sparse logit computation. See the [Gemma 4 usage guide](../../Google/Gemma4) for details and benchmarks.
> **Note:** MTP speculative decoding for Gemma 4 is only available on the vLLM nightly build — it has not yet landed in a stable release. Install via the nightly wheel (`uv pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly/cu129 …`) or use the `vllm/vllm-openai:gemma4-0505-cu129` / `vllm/vllm-openai:gemma4-0505-cu130` images above; the standard `:latest` stable tag does not include this feature.
## References
- [Model card](https://huggingface.co/google/gemma-4-E2B-it)
- [FP8 variant](https://huggingface.co/RedHatAI/gemma-4-E2B-it-FP8-dynamic)
- [Gemma docs](https://ai.google.dev/gemma/docs)
- [vLLM Gemma 4 tool-call template](https://github.com/vllm-project/vllm/blob/main/examples/tool_chat_template_gemma4.jinja)
- [TPU recipes: Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4)
- [TPU recipes: Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/)
recipes/models/Google/gemma-4-E4B-it.yaml
+322
View File
@@ -0,0 +1,322 @@
meta:
title: "Gemma 4 E4B IT"
slug: "gemma-4-e4b-it"
provider: "Google"
description: "Google's compact Gemma 4 multimodal model (effective 4B) with native text, image, and audio, plus thinking mode and tool-use protocol."
date_added: 2026-04-02
date_updated: 2026-09-12
tasks:
- multimodal
- text
performance_headline: "Effective-4B unified multimodal model with audio, thinking, and function calling — runs on a single 24 GB+ GPU"
related_recipes:
- google/gemma-4-E2B-it
- google/gemma-4-31B-it
- google/gemma-4-26B-A4B-it
hardware:
h100: verified
mi300x: verified
mi325x: verified
mi355x: verified
trillium: verified
ironwood: verified
xeon6: verified
model:
model_id: "google/gemma-4-E4B-it"
min_vllm_version: "0.19.1"
architecture: dense
parameter_count: "8B"
active_parameters: "8B"
context_length: 131072
base_args: []
base_env: {}
dependencies:
- note: "Audio extras — needed to serve the audio modality"
command: 'uv pip install "vllm[audio]"'
optional: true
features:
tool_calling:
description: "Enable automatic tool choice with Gemma 4 parser and chat template"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "gemma4"
- "--chat-template"
- "examples/tool_chat_template_gemma4.jinja"
reasoning:
description: "Enable structured thinking/reasoning output"
args:
- "--reasoning-parser"
- "gemma4"
text_only:
description: "Skip loading the vision and audio encoders for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
args:
- "--language-model-only"
spec_decoding:
description: "MTP speculative decoding with centroids masking for accelerated inference"
args:
- "--speculative-config"
- '{"model":"google/gemma-4-E4B-it-assistant","num_speculative_tokens":4}'
opt_in_features:
- text_only
- spec_decoding
variants:
default:
precision: bf16
vram_minimum_gb: 20
description: "Full BF16 — single 24 GB+ NVIDIA GPU or 1x MI300X/MI325X/MI350X/MI355Xi or 2x Xeon6/Xeon5 NUMA nodes"
supported_hardware: [h100, mi300x, mi325x, mi355x, trillium, ironwood, xeon6]
fp8:
model_id: "RedHatAI/gemma-4-E4B-it-FP8-dynamic"
precision: fp8
vram_minimum_gb: 10
description: "FP8 (E4M3) linear weights with dynamic per-token activation quantization (vision/audio encoders stay BF16) — Hopper or Blackwell"
supported_hardware: [h100]
extra_args:
- "--kv-cache-dtype"
- "fp8"
w4a16:
model_id: "google/gemma-4-E4B-it-qat-w4a16-ct"
precision: int4
vram_minimum_gb: 10
description: "W4A16 QAT (4-bit weights, 16-bit activations, group_size=32) via compressed-tensors — runs on any GPU"
supported_hardware: [h100, mi300x, mi325x, mi355x]
mobile:
model_id: "google/gemma-4-E4B-it-qat-mobile-ct"
precision: int2/4/8
vram_minimum_gb: 5
description: "Mobile QAT — mixed int2/4/8 compressed-tensors (int2 embeddings + lm_head + audio, int4 LLM attention + MLP, int8 vision + gates). 4.1× weight compression vs BF16, up to 1.5× faster decode at low batch"
supported_hardware: [h100, mi300x, mi325x, mi355x]
compatible_strategies:
- single_node_tp
- multi_node_tp
strategy_overrides:
single_node_tp:
tp: 1
guide: |
## Overview
[Gemma 4 E4B](https://huggingface.co/google/gemma-4-E4B-it) is Google's effective-4B unified multimodal model — text + images + audio in a single model, with structured thinking/reasoning, function calling, and dynamic vision resolution. It fits on a single 24 GB+ GPU.
Intel Xeon 6 CPU support remains verified. No additional benchmark-specific CPU
tuning is prescribed for general deployment.
### Key Features
- **Multimodal**: Text + images + audio natively (video via custom frame-extraction pipeline).
- **Dual Attention**: Alternating sliding-window (local) and global attention with different head dimensions.
- **Thinking Mode**: Structured reasoning via `<|channel>thought\n...<channel|>` delimiters.
- **Function Calling**: Custom tool-call protocol with dedicated special tokens.
- **Dynamic Vision Resolution**: Per-request configurable vision token budget (70, 140, 280, 560, 1120 tokens).
TPU support is provided through [vLLM TPU](https://github.com/vllm-project/tpu-inference) with recipes for [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) and [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/).
## Prerequisites
- Hardware: Intel Xeon 6/Xeon 5 CPUs
- vLLM >= 0.19.1
### pip (NVIDIA CUDA)
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --pre \
--extra-index-url https://wheels.vllm.ai/nightly/cu129 \
--extra-index-url https://download.pytorch.org/whl/cu129 \
--index-strategy unsafe-best-match
```
### pip (AMD ROCm: MI300X, MI325X, MI350X, MI355X)
Requires Python 3.12, ROCm 7.2.1, glibc >= 2.35 (Ubuntu 22.04+).
```bash
uv venv --python 3.12
source .venv/bin/activate
uv pip install vllm --pre \
--extra-index-url https://wheels.vllm.ai/rocm/nightly/rocm721 --upgrade
```
### pip (Intel Xeon 6 CPUs)
For Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions.
### Docker
```bash
docker pull vllm/vllm-openai:gemma4-0505-cu129 # NVIDIA Hopper (H100/H200, CUDA 12.9)
docker pull vllm/vllm-openai:gemma4-0505-cu130 # NVIDIA Blackwell (B200/B300, CUDA 13.0)
docker pull vllm/vllm-openai-rocm:latest # AMD
docker pull vllm/vllm-openai-cpu:latest-x86_64 # For Intel Xeon 6
```
## Deployment Configurations
### Quick Start (Single GPU)
```bash
vllm serve google/gemma-4-E4B-it \
--max-model-len 32768
```
### With Audio Support
```bash
vllm serve google/gemma-4-E4B-it \
--max-model-len 8192 \
--limit-mm-per-prompt '{"image": 4, "audio": 1}'
```
### Full-Featured Server Launch
Enables text, image, audio, thinking, and tool calling:
```bash
vllm serve google/gemma-4-E4B-it \
--max-model-len 16384 \
--gpu-memory-utilization 0.90 \
--enable-auto-tool-choice \
--reasoning-parser gemma4 \
--tool-call-parser gemma4 \
--chat-template examples/tool_chat_template_gemma4.jinja \
--limit-mm-per-prompt '{"image": 4, "audio": 1}' \
--async-scheduling \
--host 0.0.0.0 \
--port 8000
```
### Docker (NVIDIA)
```bash
docker run -itd --name gemma4-e4b \
--ipc=host --network host --shm-size 16G --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:gemma4-0505-cu129 \
--model google/gemma-4-E4B-it \
--max-model-len 32768 \
--host 0.0.0.0 --port 8000
```
Swap `vllm/vllm-openai:gemma4-0505-cu129` for `vllm/vllm-openai:gemma4-0505-cu130` on Blackwell (B200/B300).
### Docker (AMD MI300X/MI325X/MI350X/MI355X)
```bash
docker run -itd --name gemma4-rocm \
--ipc=host --network=host --privileged \
--cap-add=CAP_SYS_ADMIN --device=/dev/kfd --device=/dev/dri \
--group-add=video --cap-add=SYS_PTRACE \
--security-opt=seccomp=unconfined --shm-size 16G \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai-rocm:latest \
--model google/gemma-4-E4B-it \
--host 0.0.0.0 --port 8000
```
### Docker (Cloud TPU — Trillium / Ironwood)
TPU uses the separate `vllm/vllm-tpu` image (no pip wheel). Pull the tag specified by the upstream [Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4) or [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/) recipe, then run:
```bash
docker run -itd --name gemma4-tpu \
--privileged --network host --shm-size 16G \
-v /dev/shm:/dev/shm -e HF_TOKEN=$HF_TOKEN \
vllm/vllm-tpu:latest \
--model google/gemma-4-E4B-it \
--max-model-len 16384 \
--disable_chunked_mm_input \
--host 0.0.0.0 --port 8000
```
### Intel Xeon 6 Deployment via Docker
Launch the x86 CPU vLLM Docker container for `google/gemma-4-E4B-it`:
```bash
docker run \
--privileged --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai-cpu:latest-x86_64 google/gemma-4-E4B-it \
--tensor-parallel-size 1 \
--enable-auto-tool-choice \
--tool-call-parser gemma4 \
--chat-template examples/tool_chat_template_gemma4.jinja \
--reasoning-parser gemma4
```
For additional Intel Xeon 6 deployment details, see the Intel Software Catalog entries for [Gemma 4 E4B IT](https://aiswcatalog.intel.com/models/google-gemma-4-e4b-it).
The following settings are intentionally not prescribed as portable CPU model
defaults because they depend on the workload, hardware, or runtime environment:
- `--max-num-batched-tokens`: scheduler/throughput tuning.
- `--max-num-seqs`: concurrency and scheduler-capacity tuning.
- `--gpu-memory-utilization`: platform memory-budget tuning.
- `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning.
- `VLLM_ENGINE_ITERATION_TIMEOUT_S`: operational runtime timeout.
These settings may still appear in validated hardware-specific overrides. Tune
them at deployment time based on platform resources, workload shape, and
latency/throughput goals.
## Client Usage
### Audio Transcription
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="google/gemma-4-E4B-it",
messages=[{"role": "user", "content": [
{"type": "audio_url", "audio_url": {"url": "https://example.com/audio.wav"}},
{"type": "text", "text": "Transcribe this audio."},
]}],
max_tokens=512,
)
print(response.choices[0].message.content)
```
### Image Understanding
```python
response = client.chat.completions.create(
model="google/gemma-4-E4B-it",
messages=[{"role": "user", "content": [
{"type": "image_url", "image_url": {"url": "https://upload.wikimedia.org/wikipedia/commons/thumb/3/3a/Cat03.jpg/1200px-Cat03.jpg"}},
{"type": "text", "text": "Describe this image in detail."},
]}],
max_tokens=1024,
)
```
### Thinking Mode
```bash
vllm serve google/gemma-4-E4B-it \
--max-model-len 16384 \
--reasoning-parser gemma4 \
--tool-call-parser gemma4 \
--enable-auto-tool-choice \
--chat-template examples/tool_chat_template_gemma4.jinja
```
Enable per-request via `extra_body={"chat_template_kwargs": {"enable_thinking": True}}`.
## Configuration Tips
- Set `--max-model-len` to match your workload (max 131072).
- Image-only workloads: `--limit-mm-per-prompt.audio 0`.
- Text-only workloads: `--limit-mm-per-prompt '{"image": 0, "audio": 0}'` to skip MM profiling.
- `--async-scheduling` improves throughput.
- FP8 KV cache (`--kv-cache-dtype fp8`) saves ~50% KV memory.
## Quantized Variant
[`RedHatAI/gemma-4-E4B-it-FP8-dynamic`](https://huggingface.co/RedHatAI/gemma-4-E4B-it-FP8-dynamic) is a pre-quantized FP8 (E4M3) checkpoint — linear weights with dynamic per-token activation quantization, vision/audio encoders kept in BF16. Runs on Hopper and Blackwell. Pick the **fp8** variant above, or pass the repo id directly to `vllm serve`.
## Speculative Decoding (MTP)
Enable the **Spec Decoding** feature toggle (above) or add `--speculative-config` manually to use MTP drafting with the [assistant model](https://huggingface.co/google/gemma-4-E4B-it-assistant). Recommended `num_speculative_tokens`: 4 for this model. The E4B assistant uses centroids masking for efficient sparse logit computation. See the [Gemma 4 usage guide](../../Google/Gemma4) for details and benchmarks.
> **Note:** MTP speculative decoding for Gemma 4 is only available on the vLLM nightly build — it has not yet landed in a stable release. Install via the nightly wheel (`uv pip install -U vllm --pre --extra-index-url https://wheels.vllm.ai/nightly/cu129 …`) or use the `vllm/vllm-openai:gemma4-0505-cu129` / `vllm/vllm-openai:gemma4-0505-cu130` images above; the standard `:latest` stable tag does not include this feature.
## References
- [Model card](https://huggingface.co/google/gemma-4-E4B-it)
- [FP8 variant](https://huggingface.co/RedHatAI/gemma-4-E4B-it-FP8-dynamic)
- [Gemma docs](https://ai.google.dev/gemma/docs)
- [vLLM Gemma 4 tool-call template](https://github.com/vllm-project/vllm/blob/main/examples/tool_chat_template_gemma4.jinja)
- [TPU recipes: Trillium](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Gemma4)
- [TPU recipes: Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/blob/main/inference/ironwood/vLLM/Gemma4/)
- [Xeon 6 recipe: Gemma 4 E4B IT](https://aiswcatalog.intel.com/models/google-gemma-4-e4b-it)
recipes/models/Google/translategemma-27b-it.yaml
+122
View File
@@ -0,0 +1,122 @@
meta:
title: "TranslateGemma 27B IT"
slug: "translategemma-27b-it"
provider: "Google"
description: "Lightweight open translation model from Google (based on Gemma 3) supporting 55 languages. Served via the vLLM-optimized Infomaniak-AI checkpoint."
date_added: 2026-03-06
date_updated: 2026-04-17
tasks:
- text
performance_headline: "Deployable on laptops/desktops and cloud GPUs; vLLM-optimized checkpoint removes custom JSON inputs"
related_recipes: []
model:
model_id: "google/translategemma-27b-it"
min_vllm_version: "0.14.1"
architecture: dense
parameter_count: "27B"
active_parameters: "27B"
context_length: 131072
base_args: []
base_env: {}
features: {}
opt_in_features: []
variants:
default:
precision: bf16
vram_minimum_gb: 65
description: "Original Google checkpoint in BF16 (has vLLM compatibility issues — prefer vllm-optimized variant)"
vllm_optimized:
model_id: "Infomaniak-AI/vllm-translategemma-27b-it"
precision: bf16
vram_minimum_gb: 65
description: "Infomaniak-AI vLLM-optimized checkpoint — recommended. Fixes RoPE config, EOS token, and replaces custom JSON inputs with string delimiters."
small_4b:
model_id: "google/translategemma-4b-it"
precision: bf16
vram_minimum_gb: 10
description: "4B variant for lower-resource deployments (prefer Infomaniak-AI/vllm-translategemma-4b-it)"
compatible_strategies:
- single_node_tp
- multi_node_tp
hardware_overrides: {}
strategy_overrides: {}
guide: |
## Overview
[TranslateGemma](https://huggingface.co/collections/google/translategemma) is a family of lightweight, state-of-the-art open translation models from Google, based on the Gemma 3 family. TranslateGemma models handle translation across 55 languages and are small enough to deploy on laptops, desktops, and modest cloud GPU environments.
### Original Models
- [google/translategemma-27b-it](https://huggingface.co/google/translategemma-27b-it)
- [google/translategemma-4b-it](https://huggingface.co/google/translategemma-4b-it)
### Optimized vLLM Models
- [Infomaniak-AI/vllm-translategemma-27b-it](https://huggingface.co/Infomaniak-AI/vllm-translategemma-27b-it)
- [Infomaniak-AI/vllm-translategemma-4b-it](https://huggingface.co/Infomaniak-AI/vllm-translategemma-4b-it)
### Why use the vLLM-optimized models?
The original Google models have compatibility issues with standard inference engines like vLLM. The optimized versions from Infomaniak-AI ([detailed changes](https://huggingface.co/Infomaniak-AI/vllm-translategemma-27b-it#changes-from-original-model)):
- **vLLM Compatibility**: Originals require custom JSON parameters (`source_lang_code`/`target_lang_code`). The optimized version uses string delimiters.
- **RoPE Simplification**: Originals use a complex RoPE configuration for sliding attention. Optimized uses a standard linear RoPE format (`factor: 8.0`).
- **EOS Token Fix**: Corrects the EOS token from `<end_of_turn>` to `<eos>`.
## Prerequisites
### Docker
```bash
docker pull vllm/vllm-openai:latest
```
## Deployment Configurations
Verified for both 4B and 27B:
```bash
docker run -itd --name google-translategemma-27b-it \
--ipc=host \
--network host \
--shm-size 16G \
--gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:latest \
Infomaniak-AI/vllm-translategemma-27b-it \
--served-model-name translategemma-27b-it \
--gpu-memory-utilization 0.8 \
--host 0.0.0.0 \
--port 8000
```
## Client Usage
Tips:
- **Prompt Delimiters**: Encode language metadata directly in the content string:
`<<<source>>>{src_lang}<<<target>>>{tgt_lang}<<<text>>>{text}`
- **Language Codes**: ISO 639-1 Alpha-2 (e.g. `en`, `zh`) and regional variants (e.g. `en_US`, `zh_CN`).
- **Context Limit**: ~2K tokens.
### cURL
```bash
curl -X POST http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "translategemma-27b-it",
"messages": [{
"role": "user",
"content": "<<<source>>>en<<<target>>>zh<<<text>>>We distribute two models for language identification, which can recognize 176 languages."
}]
}'
```
## References
- [TranslateGemma collection](https://huggingface.co/collections/google/translategemma)
- [Infomaniak-AI optimized 27B](https://huggingface.co/Infomaniak-AI/vllm-translategemma-27b-it)
- [Infomaniak-AI optimized 4B](https://huggingface.co/Infomaniak-AI/vllm-translategemma-4b-it)
recipes/models/IFM/Jais-3-0.9B.yaml
+109
View File
@@ -0,0 +1,109 @@
meta:
title: "Jais-3-0.9B"
slug: "jais-3-0.9b"
provider: "IFM"
description: "Compact dense model for mathematics, code, instruction following, and STEM tasks"
date_added: 2026-10-05
date_updated: 2026-10-05
tasks:
- text
performance_headline: "1.08B stored parameters including embeddings, with 131K context and native K2-Horizon parsing"
related_recipes:
- "IFM/Jais-3-3.7B"
hardware:
h200: verified
model:
model_id: "IFM/Jais-3-0.9B"
architecture: dense
parameter_count: "1.08B"
active_parameters: "1.08B"
context_length: 131072
base_args:
- "--trust-remote-code"
- "--chat-template-content-format"
- "string"
- "--dtype"
- "bfloat16"
- "--max-model-len"
- "131072"
- "--gpu-memory-utilization"
- "0.85"
base_env: {}
features:
reasoning:
description: "Extract Jais-3 reasoning into message.reasoning"
args: ["--reasoning-parser", "k2_horizon"]
tool_calling:
description: "Enable automatic tool choice with the K2-Horizon parser"
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
variants:
default:
precision: bf16
vram_minimum_gb: 3
description: "Released BF16 checkpoint; the command starts at an 8K runtime context"
compatible_strategies:
- single_node_tp
- multi_node_tp
- multi_node_tp_pp
hardware_overrides: {}
guide: |
## Overview
Jais-3-0.9B is a compact dense model in the IFM Jais-3
family, with 1.08B stored parameters including embeddings and a 131,072-token
context window.
Jais-3 is an open-weight IFM model family built for transparent
foundation-model research, staged checkpoint analysis, and practical
deployment. The series spans compact dense models such as Jais-3-0.9B
for local experimentation, dense mid-size and large models for high-quality
research workloads, and mixture-of-expert (MoE) models for higher-capacity
serving.
## Launch Commands
Jais-3-0.9B's `config.json` already enables YaRN, so the full 131,072-token context needs no RoPE override.
```bash
vllm serve IFM/Jais-3-0.9B \
--trust-remote-code \
--chat-template-content-format string \
--dtype bfloat16 \
--max-model-len 131072 \
--gpu-memory-utilization 0.85 \
--reasoning-parser k2_horizon \
--enable-auto-tool-choice \
--tool-call-parser k2_horizon
```
Lower `--max-model-len` (e.g. 8192) to reduce KV-cache memory.
## Client usage
```python
from openai import OpenAI
client = OpenAI(
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
)
resp = client.chat.completions.create(
model="IFM/Jais-3-0.9B",
messages=[{"role": "user", "content": "لدينا عددان مجموعهما 50 وحاصل ضربهما 600. ما هما العددان؟ اشرح خطوات الحل."}],
temperature=1.0, top_p=0.95, max_tokens=2048,
)
print(resp.choices[0].message.content)
```
### Thinking modes
The model supports selectable thinking through `chat_template_kwargs`, per
request or server-wide via `--default-chat-template-kwargs`:
- `{"reasoning_effort": "high"}` — full thinking (default).
- `{"reasoning_effort": "low"}` — no thinking.
## References
- [IFM/Jais-3-0.9B on Hugging Face](https://huggingface.co/IFM/Jais-3-0.9B)
- [Jais 3 Collection](https://huggingface.co/collections/IFM/jais-3)
recipes/models/IFM/Jais-3-3.7B.yaml
+88
View File
@@ -0,0 +1,88 @@
meta:
title: "Jais-3-3.7B"
slug: "jais-3-3.7b"
provider: "IFM"
description: "Dense model for long-context research, serving, and downstream adaptation"
date_added: 2026-10-05
date_updated: 2026-10-05
tasks: [text]
performance_headline: "5.06B stored parameters including embeddings, with up to 512K context"
related_recipes: ["IFM/Jais-3-0.9B", "IFM/Jais-3-7B"]
hardware:
h200: verified
model:
model_id: "IFM/Jais-3-3.7B"
architecture: dense
parameter_count: "5.06B"
active_parameters: "5.06B"
context_length: 524288
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16"]
base_env: {}
features:
reasoning:
description: "Extract Jais-3 reasoning into message.reasoning"
args: ["--reasoning-parser", "k2_horizon"]
tool_calling:
description: "Enable automatic tool choice with the K2-Horizon parser"
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
variants:
default:
precision: bf16
vram_minimum_gb: 13
description: "BF16 serving runtime with a 131K context baseline"
compatible_strategies: [single_node_tp, multi_node_tp, multi_node_tp_pp]
hardware_overrides: {}
guide: |
## Overview
Jais-3-3.7B is a small dense model in the IFM Jais-3 family,
with 5.06B stored parameters including embeddings and support for up to
524,288 tokens of context.
Jais-3 is an open-weight IFM model family built for transparent
foundation-model research, staged checkpoint analysis, and practical
deployment. The series spans compact dense models for local experimentation,
dense mid-size and large models for high-quality research workloads, and
mixture-of-expert (MoE) models for higher-capacity serving.
## Launch Commands
```bash
vllm serve IFM/Jais-3-3.7B \
--trust-remote-code \
--chat-template-content-format string \
--dtype bfloat16 \
--reasoning-parser k2_horizon \
--enable-auto-tool-choice \
--tool-call-parser k2_horizon
```
## Client usage
```python
from openai import OpenAI
client = OpenAI(
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
)
resp = client.chat.completions.create(
model="IFM/Jais-3-3.7B",
messages=[{"role": "user", "content": "لدينا عددان مجموعهما 50 وحاصل ضربهما 600. ما هما العددان؟ اشرح خطوات الحل."}],
temperature=1.0, top_p=0.95, max_tokens=2048,
)
print(resp.choices[0].message.content)
```
### Thinking modes
The model supports selectable thinking through `chat_template_kwargs`, per
request or server-wide via `--default-chat-template-kwargs`:
- `{"reasoning_effort": "high"}` — full thinking (default).
- `{"reasoning_effort": "low"}` — no thinking.
## References
- [IFM/Jais-3-3.7B on Hugging Face](https://huggingface.co/IFM/Jais-3-3.7B)
- [Jais 3 Collection](https://huggingface.co/collections/IFM/jais-3)
recipes/models/IFM/Jais-3-32B.yaml
+93
View File
@@ -0,0 +1,93 @@
meta:
title: "Jais-3-32B"
slug: "jais-3-32b"
provider: "IFM"
description: "Dense long-context model for reasoning research and high-capacity serving"
date_added: 2026-10-05
date_updated: 2026-10-05
tasks: [text]
performance_headline: "34.78B stored parameters including embeddings, with native vLLM execution and up to 512K context"
related_recipes: ["IFM/Jais-3-7B", "IFM/Jais-3-MoVA-36B-A4B"]
hardware:
h200: verified
model:
model_id: "IFM/Jais-3-32B"
architecture: dense
parameter_count: "34.78B"
active_parameters: "34.78B"
context_length: 524288
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16"]
base_env: {}
features:
reasoning:
description: "Extract Jais-3 reasoning into message.reasoning"
args: ["--reasoning-parser", "k2_horizon"]
tool_calling:
description: "Enable automatic tool choice with the K2-Horizon parser"
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
variants:
default:
precision: bf16
vram_minimum_gb: 84
description: "BF16 serving runtime with native vLLM execution"
compatible_strategies: [single_node_tp, multi_node_tp, multi_node_tp_pp]
hardware_overrides: {}
strategy_overrides:
single_node_tp:
tp: 2
guide: |
## Overview
Jais-3-32B is a large dense model in the IFM Jais-3 family,
with 34.78B stored parameters including embeddings and support for up to
524,288 tokens of context.
Jais-3 is an open-weight IFM model family built for transparent
foundation-model research, staged checkpoint analysis, and practical
deployment. The series spans compact dense models for local experimentation,
dense mid-size and large models such as Jais-3-32B for high-quality
research workloads, and mixture-of-expert (MoE) models for higher-capacity
serving.
## Launch Commands
```bash
vllm serve IFM/Jais-3-32B \
--tensor-parallel-size 2 \
--trust-remote-code \
--chat-template-content-format string \
--dtype bfloat16 \
--reasoning-parser k2_horizon \
--enable-auto-tool-choice \
--tool-call-parser k2_horizon
```
## Client usage
```python
from openai import OpenAI
client = OpenAI(
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
)
resp = client.chat.completions.create(
model="IFM/Jais-3-32B",
messages=[{"role": "user", "content": "لدينا عددان مجموعهما 50 وحاصل ضربهما 600. ما هما العددان؟ اشرح خطوات الحل."}],
temperature=1.0, top_p=0.95, max_tokens=2048,
)
print(resp.choices[0].message.content)
```
### Thinking modes
The model supports selectable thinking through `chat_template_kwargs`, per
request or server-wide via `--default-chat-template-kwargs`:
- `{"reasoning_effort": "high"}` — full thinking (default).
- `{"reasoning_effort": "low"}` — no thinking.
## References
- [IFM/Jais-3-32B on Hugging Face](https://huggingface.co/IFM/Jais-3-32B)
- [Jais 3 Collection](https://huggingface.co/collections/IFM/jais-3)
recipes/models/IFM/Jais-3-375B-A23B.yaml
+148
View File
@@ -0,0 +1,148 @@
meta:
title: "Jais-3-375B-A23B"
slug: "jais-3-375b-a23b"
provider: "IFM"
description: "Frontier-scale sparse MoE model for long-context research and high-capacity serving"
date_added: 2026-10-05
date_updated: 2026-10-05
tasks: [text]
performance_headline: "379.17B stored parameters including embeddings / 26.67B active parameters per token"
related_recipes: ["IFM/Jais-3-MoVA-36B-A4B"]
hardware:
h200: verified
mi355x: verified
model:
model_id: "IFM/Jais-3-375B-A23B"
architecture: moe
parameter_count: "379.17B"
active_parameters: "26.67B"
context_length: 524288
# --enable-expert-parallel is not here: in base_args it would reach the AMD
# path too, which deliberately runs pure TP (see strategy_hardware). The TEP
# strategies emit it themselves; multi_node_tp_pp does not, so it carries its
# own copy in strategy_overrides below.
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16"]
base_env: {}
features:
reasoning:
description: "Extract Jais-3 reasoning into message.reasoning"
args: ["--reasoning-parser", "k2_horizon"]
tool_calling:
description: "Enable automatic tool choice with the K2-Horizon parser"
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
variants:
default:
precision: bf16
vram_minimum_gb: 911
description: "BF16 serving runtime with an eight-GPU tensor-parallel baseline"
compatible_strategies: [single_node_tep, single_node_tp, multi_node_tep, multi_node_tp_pp]
default_strategy: single_node_tep
# On AMD the expert-parallel dispatch is slower than replicating the experts
# across a pure tensor-parallel shard, so the TEP strategies are opted out and
# single-node TP is the offered path. multi_node_tp_pp is left available: EP
# works there, it is just not the fast default at single-node scale.
strategy_hardware:
single_node_tep:
amd: unsupported
multi_node_tep:
amd: unsupported
hardware_overrides:
amd:
extra_args:
- "--attention-backend"
- "ROCM_AITER_UNIFIED_ATTN"
- "--gpu-memory-utilization"
- "0.9"
extra_env:
VLLM_ROCM_USE_AITER: "1"
NCCL_MIN_NCHANNELS: "112"
strategy_overrides:
single_node_tep:
tp: 8
# multi_node_tp_pp emits no EP flag of its own, and this recipe has always
# run it with expert parallel on. Restored here rather than in base_args so
# it stays off the AMD single-node TP path.
multi_node_tp_pp:
extra_args: ["--enable-expert-parallel"]
guide: |
## Overview
Jais-3-375B-A23B is a frontier-scale mixture-of-expert (MoE) model in the IFM
Jais-3 family, with 379.17B stored parameters including embeddings,
26.67B active parameters per token, and support for up to 524,288 tokens of
context.
Jais-3 is an open-weight IFM model family built for transparent
foundation-model research, staged checkpoint analysis, and practical
deployment. The series spans compact dense models for local experimentation,
dense mid-size and large models for high-quality research workloads, and
mixture-of-expert (MoE) models such as Jais-3-375B-A23B for frontier-scale
research and higher-capacity serving.
## Launch Commands
```bash
vllm serve IFM/Jais-3-375B-A23B \
--tensor-parallel-size 8 \
--trust-remote-code \
--chat-template-content-format string \
--dtype bfloat16 \
--enable-expert-parallel \
--reasoning-parser k2_horizon \
--enable-auto-tool-choice \
--tool-call-parser k2_horizon
```
### AMD MI355X
Verified on an 8-GPU MI355X (gfx950) node. AITER supplies the fused MoE and
the unified attention backend:
```bash
export VLLM_ROCM_USE_AITER=1
export NCCL_MIN_NCHANNELS=112
vllm serve IFM/Jais-3-375B-A23B \
--tensor-parallel-size 8 \
--trust-remote-code \
--chat-template-content-format string \
--dtype bfloat16 \
--max-model-len 131072 \
--gpu-memory-utilization 0.9 \
--attention-backend ROCM_AITER_UNIFIED_ATTN \
--generation-config auto \
--reasoning-parser k2_horizon \
--enable-auto-tool-choice \
--tool-call-parser k2_horizon
```
## Client usage
```python
from openai import OpenAI
client = OpenAI(
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
)
resp = client.chat.completions.create(
model="IFM/Jais-3-375B-A23B",
messages=[{"role": "user", "content": "لدينا عددان مجموعهما 50 وحاصل ضربهما 600. ما هما العددان؟ اشرح خطوات الحل."}],
temperature=1.0, top_p=0.95, max_tokens=2048,
)
print(resp.choices[0].message.content)
```
### Thinking modes
The model supports selectable thinking through `chat_template_kwargs`, per
request or server-wide via `--default-chat-template-kwargs`:
- `{"reasoning_effort": "high"}` — full thinking (default).
- `{"reasoning_effort": "low"}` — no thinking.
## References
- [IFM/Jais-3-375B-A23B on Hugging Face](https://huggingface.co/IFM/Jais-3-375B-A23B)
- [Jais 3 Collection](https://huggingface.co/collections/IFM/jais-3)
recipes/models/IFM/Jais-3-7B.yaml
+89
View File
@@ -0,0 +1,89 @@
meta:
title: "Jais-3-7B"
slug: "jais-3-7b"
provider: "IFM"
description: "Dense model for long-context research, fine-tuning, and cost-conscious deployment"
date_added: 2026-10-05
date_updated: 2026-10-05
tasks: [text]
performance_headline: "9B stored parameters including embeddings, with up to 512K context"
related_recipes: ["IFM/Jais-3-3.7B", "IFM/Jais-3-32B"]
hardware:
h200: verified
model:
model_id: "IFM/Jais-3-7B"
architecture: dense
parameter_count: "9B"
active_parameters: "9B"
context_length: 524288
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16"]
base_env: {}
features:
reasoning:
description: "Extract Jais-3 reasoning into message.reasoning"
args: ["--reasoning-parser", "k2_horizon"]
tool_calling:
description: "Enable automatic tool choice with the K2-Horizon parser"
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
variants:
default:
precision: bf16
vram_minimum_gb: 22
description: "BF16 serving runtime with a 131K context baseline"
compatible_strategies: [single_node_tp, multi_node_tp, multi_node_tp_pp]
hardware_overrides: {}
guide: |
## Overview
Jais-3-7B is a compact dense model in the IFM Jais-3 family,
with 9B stored parameters including embeddings and support for up to 524,288
tokens of context.
Jais-3 is an open-weight IFM model family built for transparent
foundation-model research, staged checkpoint analysis, and practical
deployment. The series spans compact dense models such as Jais-3-7B for
local experimentation, dense mid-size and large models for high-quality
research workloads, and mixture-of-expert (MoE) models for higher-capacity
serving.
## Launch Commands
```bash
vllm serve IFM/Jais-3-7B \
--trust-remote-code \
--chat-template-content-format string \
--dtype bfloat16 \
--reasoning-parser k2_horizon \
--enable-auto-tool-choice \
--tool-call-parser k2_horizon
```
## Client usage
```python
from openai import OpenAI
client = OpenAI(
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
)
resp = client.chat.completions.create(
model="IFM/Jais-3-7B",
messages=[{"role": "user", "content": "لدينا عددان مجموعهما 50 وحاصل ضربهما 600. ما هما العددان؟ اشرح خطوات الحل."}],
temperature=1.0, top_p=0.95, max_tokens=2048,
)
print(resp.choices[0].message.content)
```
### Thinking modes
The model supports selectable thinking through `chat_template_kwargs`, per
request or server-wide via `--default-chat-template-kwargs`:
- `{"reasoning_effort": "high"}` — full thinking (default).
- `{"reasoning_effort": "low"}` — no thinking.
## References
- [IFM/Jais-3-7B on Hugging Face](https://huggingface.co/IFM/Jais-3-7B)
- [Jais 3 Collection](https://huggingface.co/collections/IFM/jais-3)
recipes/models/IFM/Jais-3-MoVA-36B-A4B.yaml
+96
View File
@@ -0,0 +1,96 @@
meta:
title: "Jais-3-MoVA-36B-A4B"
slug: "jais-3-mova-36b-a4b"
provider: "IFM"
description: "Sparse MoVA and MoE long-context model for research and production-style serving experiments"
date_added: 2026-10-05
date_updated: 2026-10-05
tasks: [text]
performance_headline: "37.44B stored parameters including embeddings / 5.95B active parameters per token"
related_recipes: ["IFM/Jais-3-32B", "IFM/Jais-3-375B-A23B"]
hardware:
h200: verified
model:
model_id: "IFM/Jais-3-MoVA-36B-A4B"
architecture: moe
parameter_count: "37.44B"
active_parameters: "5.95B"
context_length: 524288
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16", "--enable-expert-parallel"]
base_env: {}
features:
reasoning:
description: "Extract Jais-3 reasoning into message.reasoning"
args: ["--reasoning-parser", "k2_horizon"]
tool_calling:
description: "Enable automatic tool choice with the K2-Horizon parser"
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
variants:
default:
precision: bf16
vram_minimum_gb: 90
description: "BF16 serving runtime with native vLLM execution"
compatible_strategies: [single_node_tep, multi_node_tep, multi_node_tp_pp]
default_strategy: single_node_tep
hardware_overrides: {}
strategy_overrides:
single_node_tep:
tp: 2
guide: |
## Overview
Jais-3-MoVA-36B-A4B is a mixture-of-expert (MoE) model in the IFM Jais-3
family. It combines MoVA attention and MoE layers, with 37.44B stored parameters
including embeddings, 5.95B active parameters per token, and support for up
to 524,288 tokens of context.
Jais-3 is an open-weight IFM model family built for transparent
foundation-model research, staged checkpoint analysis, and practical
deployment. The series spans compact dense models for local experimentation,
dense mid-size and large models for high-quality research workloads, and
mixture-of-expert (MoE) models such as Jais-3-MoVA-36B-A4B for higher-capacity
serving.
## Launch Commands
```bash
vllm serve IFM/Jais-3-MoVA-36B-A4B \
--tensor-parallel-size 2 \
--trust-remote-code \
--chat-template-content-format string \
--dtype bfloat16 \
--enable-expert-parallel \
--reasoning-parser k2_horizon \
--enable-auto-tool-choice \
--tool-call-parser k2_horizon
```
## Client usage
```python
from openai import OpenAI
client = OpenAI(
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
)
resp = client.chat.completions.create(
model="IFM/Jais-3-MoVA-36B-A4B",
messages=[{"role": "user", "content": "لدينا عددان مجموعهما 50 وحاصل ضربهما 600. ما هما العددان؟ اشرح خطوات الحل."}],
temperature=1.0, top_p=0.95, max_tokens=2048,
)
print(resp.choices[0].message.content)
```
### Thinking modes
The model supports selectable thinking through `chat_template_kwargs`, per
request or server-wide via `--default-chat-template-kwargs`:
- `{"reasoning_effort": "high"}` — full thinking (default).
- `{"reasoning_effort": "low"}` — no thinking.
## References
- [IFM/Jais-3-MoVA-36B-A4B on Hugging Face](https://huggingface.co/IFM/Jais-3-MoVA-36B-A4B)
- [Jais 3 Collection](https://huggingface.co/collections/IFM/jais-3)
recipes/models/IFM/K2-Horizon-0.9B.yaml
+110
View File
@@ -0,0 +1,110 @@
meta:
title: "K2-Horizon-0.9B"
slug: "k2-horizon-0.9b"
provider: "IFM"
description: "Compact dense model for mathematics, code, instruction following, and STEM tasks"
date_added: 2026-08-31
date_updated: 2026-08-31
tasks:
- text
performance_headline: "1.08B stored parameters including embeddings, with 131K context and native K2-Horizon parsing"
related_recipes:
- "IFM/K2-Horizon-3.7B"
hardware:
h200: verified
model:
model_id: "IFM/K2-Horizon-0.9B"
architecture: dense
parameter_count: "1.08B"
active_parameters: "1.08B"
context_length: 131072
base_args:
- "--trust-remote-code"
- "--chat-template-content-format"
- "string"
- "--dtype"
- "bfloat16"
- "--max-model-len"
- "131072"
- "--gpu-memory-utilization"
- "0.85"
base_env: {}
features:
reasoning:
description: "Extract K2-Horizon reasoning into message.reasoning"
args: ["--reasoning-parser", "k2_horizon"]
tool_calling:
description: "Enable automatic tool choice with the K2-Horizon parser"
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
variants:
default:
precision: bf16
vram_minimum_gb: 3
description: "Released BF16 checkpoint; the command starts at an 8K runtime context"
compatible_strategies:
- single_node_tp
- multi_node_tp
- multi_node_tp_pp
hardware_overrides: {}
guide: |
## Overview
K2-Horizon-0.9B is a compact dense model in the IFM K2-Horizon
family, with 1.08B stored parameters including embeddings and a 131,072-token
context window.
K2-Horizon is an open-weight IFM model family built for transparent
foundation-model research, staged checkpoint analysis, and practical
deployment. The series spans compact dense models such as K2-Horizon-0.9B
for local experimentation, dense mid-size and large models for high-quality
research workloads, and mixture-of-expert (MoE) models for higher-capacity
serving.
## Launch Commands
K2-Horizon-0.9B's `config.json` already enables YaRN, so the full 131,072-token context needs no RoPE override.
```bash
vllm serve IFM/K2-Horizon-0.9B \
--trust-remote-code \
--chat-template-content-format string \
--dtype bfloat16 \
--max-model-len 131072 \
--gpu-memory-utilization 0.85 \
--reasoning-parser k2_horizon \
--enable-auto-tool-choice \
--tool-call-parser k2_horizon
```
Lower `--max-model-len` (e.g. 8192) to reduce KV-cache memory.
## Client usage
```python
from openai import OpenAI
client = OpenAI(
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
)
resp = client.chat.completions.create(
model="IFM/K2-Horizon-0.9B",
messages=[{"role": "user", "content": "Give me three primes above 100."}],
temperature=1.0, top_p=0.95, max_tokens=2048,
)
print(resp.choices[0].message.content)
```
### Thinking modes
The model supports selectable thinking through `chat_template_kwargs`, per
request or server-wide via `--default-chat-template-kwargs`:
- `{"reasoning_effort": "high"}` — full thinking (default).
- `{"reasoning_effort": "medium"}` — faster thinking.
- `{"reasoning_effort": "low"}` — fastest thinking.
## References
- [IFM/K2-Horizon-0.9B on Hugging Face](https://huggingface.co/IFM/K2-Horizon-0.9B)
- [K2 Horizon Collection](https://huggingface.co/collections/IFM/k2-horizon)
recipes/models/IFM/K2-Horizon-3.7B.yaml
+89
View File
@@ -0,0 +1,89 @@
meta:
title: "K2-Horizon-3.7B"
slug: "k2-horizon-3.7b"
provider: "IFM"
description: "Dense model for long-context research, serving, and downstream adaptation"
date_added: 2026-08-31
date_updated: 2026-09-02
tasks: [text]
performance_headline: "5.06B stored parameters including embeddings, with up to 512K context"
related_recipes: ["IFM/K2-Horizon-0.9B", "IFM/K2-Horizon-7B"]
hardware:
h200: verified
model:
model_id: "IFM/K2-Horizon-3.7B"
architecture: dense
parameter_count: "5.06B"
active_parameters: "5.06B"
context_length: 524288
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16"]
base_env: {}
features:
reasoning:
description: "Extract K2-Horizon reasoning into message.reasoning"
args: ["--reasoning-parser", "k2_horizon"]
tool_calling:
description: "Enable automatic tool choice with the K2-Horizon parser"
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
variants:
default:
precision: bf16
vram_minimum_gb: 13
description: "BF16 serving runtime with a 131K context baseline"
compatible_strategies: [single_node_tp, multi_node_tp, multi_node_tp_pp]
hardware_overrides: {}
guide: |
## Overview
K2-Horizon-3.7B is a small dense model in the IFM K2-Horizon family,
with 5.06B stored parameters including embeddings and support for up to
524,288 tokens of context.
K2-Horizon is an open-weight IFM model family built for transparent
foundation-model research, staged checkpoint analysis, and practical
deployment. The series spans compact dense models for local experimentation,
dense mid-size and large models for high-quality research workloads, and
mixture-of-expert (MoE) models for higher-capacity serving.
## Launch Commands
```bash
vllm serve IFM/K2-Horizon-3.7B \
--trust-remote-code \
--chat-template-content-format string \
--dtype bfloat16 \
--reasoning-parser k2_horizon \
--enable-auto-tool-choice \
--tool-call-parser k2_horizon
```
## Client usage
```python
from openai import OpenAI
client = OpenAI(
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
)
resp = client.chat.completions.create(
model="IFM/K2-Horizon-3.7B",
messages=[{"role": "user", "content": "Give me three primes above 100."}],
temperature=1.0, top_p=0.95, max_tokens=2048,
)
print(resp.choices[0].message.content)
```
### Thinking modes
The model supports selectable thinking through `chat_template_kwargs`, per
request or server-wide via `--default-chat-template-kwargs`:
- `{"reasoning_effort": "high"}` — full thinking (default).
- `{"reasoning_effort": "medium"}` — faster thinking.
- `{"reasoning_effort": "low"}` — fastest thinking.
## References
- [IFM/K2-Horizon-3.7B on Hugging Face](https://huggingface.co/IFM/K2-Horizon-3.7B)
- [K2 Horizon Collection](https://huggingface.co/collections/IFM/k2-horizon)
recipes/models/IFM/K2-Horizon-32B.yaml
+94
View File
@@ -0,0 +1,94 @@
meta:
title: "K2-Horizon-32B"
slug: "k2-horizon-32b"
provider: "IFM"
description: "Dense long-context model for reasoning research and high-capacity serving"
date_added: 2026-08-31
date_updated: 2026-09-02
tasks: [text]
performance_headline: "34.78B stored parameters including embeddings, with native vLLM execution and up to 512K context"
related_recipes: ["IFM/K2-Horizon-7B", "IFM/K2-Horizon-MoVA-36B-A4B"]
hardware:
h200: verified
model:
model_id: "IFM/K2-Horizon-32B"
architecture: dense
parameter_count: "34.78B"
active_parameters: "34.78B"
context_length: 524288
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16"]
base_env: {}
features:
reasoning:
description: "Extract K2-Horizon reasoning into message.reasoning"
args: ["--reasoning-parser", "k2_horizon"]
tool_calling:
description: "Enable automatic tool choice with the K2-Horizon parser"
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
variants:
default:
precision: bf16
vram_minimum_gb: 84
description: "BF16 serving runtime with native vLLM execution"
compatible_strategies: [single_node_tp, multi_node_tp, multi_node_tp_pp]
hardware_overrides: {}
strategy_overrides:
single_node_tp:
tp: 2
guide: |
## Overview
K2-Horizon-32B is a large dense model in the IFM K2-Horizon family,
with 34.78B stored parameters including embeddings and support for up to
524,288 tokens of context.
K2-Horizon is an open-weight IFM model family built for transparent
foundation-model research, staged checkpoint analysis, and practical
deployment. The series spans compact dense models for local experimentation,
dense mid-size and large models such as K2-Horizon-32B for high-quality
research workloads, and mixture-of-expert (MoE) models for higher-capacity
serving.
## Launch Commands
```bash
vllm serve IFM/K2-Horizon-32B \
--tensor-parallel-size 2 \
--trust-remote-code \
--chat-template-content-format string \
--dtype bfloat16 \
--reasoning-parser k2_horizon \
--enable-auto-tool-choice \
--tool-call-parser k2_horizon
```
## Client usage
```python
from openai import OpenAI
client = OpenAI(
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
)
resp = client.chat.completions.create(
model="IFM/K2-Horizon-32B",
messages=[{"role": "user", "content": "Give me three primes above 100."}],
temperature=1.0, top_p=0.95, max_tokens=2048,
)
print(resp.choices[0].message.content)
```
### Thinking modes
The model supports selectable thinking through `chat_template_kwargs`, per
request or server-wide via `--default-chat-template-kwargs`:
- `{"reasoning_effort": "high"}` — full thinking (default).
- `{"reasoning_effort": "medium"}` — faster thinking.
- `{"reasoning_effort": "low"}` — fastest thinking.
## References
- [IFM/K2-Horizon-32B on Hugging Face](https://huggingface.co/IFM/K2-Horizon-32B)
- [K2 Horizon Collection](https://huggingface.co/collections/IFM/k2-horizon)
recipes/models/IFM/K2-Horizon-375B-A23B.yaml
+149
View File
@@ -0,0 +1,149 @@
meta:
title: "K2-Horizon-375B-A23B"
slug: "k2-horizon-375b-a23b"
provider: "IFM"
description: "Frontier-scale sparse MoE model for long-context research and high-capacity serving"
date_added: 2026-08-31
date_updated: 2026-09-08
tasks: [text]
performance_headline: "379.17B stored parameters including embeddings / 26.67B active parameters per token"
related_recipes: ["IFM/K2-Horizon-MoVA-36B-A4B"]
hardware:
h200: verified
mi355x: verified
model:
model_id: "IFM/K2-Horizon-375B-A23B"
architecture: moe
parameter_count: "379.17B"
active_parameters: "26.67B"
context_length: 524288
# --enable-expert-parallel is not here: in base_args it would reach the AMD
# path too, which deliberately runs pure TP (see strategy_hardware). The TEP
# strategies emit it themselves; multi_node_tp_pp does not, so it carries its
# own copy in strategy_overrides below.
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16"]
base_env: {}
features:
reasoning:
description: "Extract K2-Horizon reasoning into message.reasoning"
args: ["--reasoning-parser", "k2_horizon"]
tool_calling:
description: "Enable automatic tool choice with the K2-Horizon parser"
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
variants:
default:
precision: bf16
vram_minimum_gb: 911
description: "BF16 serving runtime with an eight-GPU tensor-parallel baseline"
compatible_strategies: [single_node_tep, single_node_tp, multi_node_tep, multi_node_tp_pp]
default_strategy: single_node_tep
# On AMD the expert-parallel dispatch is slower than replicating the experts
# across a pure tensor-parallel shard, so the TEP strategies are opted out and
# single-node TP is the offered path. multi_node_tp_pp is left available: EP
# works there, it is just not the fast default at single-node scale.
strategy_hardware:
single_node_tep:
amd: unsupported
multi_node_tep:
amd: unsupported
hardware_overrides:
amd:
extra_args:
- "--attention-backend"
- "ROCM_AITER_UNIFIED_ATTN"
- "--gpu-memory-utilization"
- "0.9"
extra_env:
VLLM_ROCM_USE_AITER: "1"
NCCL_MIN_NCHANNELS: "112"
strategy_overrides:
single_node_tep:
tp: 8
# multi_node_tp_pp emits no EP flag of its own, and this recipe has always
# run it with expert parallel on. Restored here rather than in base_args so
# it stays off the AMD single-node TP path.
multi_node_tp_pp:
extra_args: ["--enable-expert-parallel"]
guide: |
## Overview
K2-Horizon-375B-A23B is a frontier-scale mixture-of-expert (MoE) model in the IFM
K2-Horizon family, with 379.17B stored parameters including embeddings,
26.67B active parameters per token, and support for up to 524,288 tokens of
context.
K2-Horizon is an open-weight IFM model family built for transparent
foundation-model research, staged checkpoint analysis, and practical
deployment. The series spans compact dense models for local experimentation,
dense mid-size and large models for high-quality research workloads, and
mixture-of-expert (MoE) models such as K2-Horizon-375B-A23B for frontier-scale
research and higher-capacity serving.
## Launch Commands
```bash
vllm serve IFM/K2-Horizon-375B-A23B \
--tensor-parallel-size 8 \
--trust-remote-code \
--chat-template-content-format string \
--dtype bfloat16 \
--enable-expert-parallel \
--reasoning-parser k2_horizon \
--enable-auto-tool-choice \
--tool-call-parser k2_horizon
```
### AMD MI355X
Verified on an 8-GPU MI355X (gfx950) node. AITER supplies the fused MoE and
the unified attention backend:
```bash
export VLLM_ROCM_USE_AITER=1
export NCCL_MIN_NCHANNELS=112
vllm serve IFM/K2-Horizon-375B-A23B \
--tensor-parallel-size 8 \
--trust-remote-code \
--chat-template-content-format string \
--dtype bfloat16 \
--max-model-len 131072 \
--gpu-memory-utilization 0.9 \
--attention-backend ROCM_AITER_UNIFIED_ATTN \
--generation-config auto \
--reasoning-parser k2_horizon \
--enable-auto-tool-choice \
--tool-call-parser k2_horizon
```
## Client usage
```python
from openai import OpenAI
client = OpenAI(
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
)
resp = client.chat.completions.create(
model="IFM/K2-Horizon-375B-A23B",
messages=[{"role": "user", "content": "Give me three primes above 100."}],
temperature=1.0, top_p=0.95, max_tokens=2048,
)
print(resp.choices[0].message.content)
```
### Thinking modes
The model supports selectable thinking through `chat_template_kwargs`, per
request or server-wide via `--default-chat-template-kwargs`:
- `{"reasoning_effort": "high"}` — full thinking (default).
- `{"reasoning_effort": "medium"}` — faster thinking.
- `{"reasoning_effort": "low"}` — fastest thinking.
## References
- [IFM/K2-Horizon-375B-A23B on Hugging Face](https://huggingface.co/IFM/K2-Horizon-375B-A23B)
- [K2 Horizon Collection](https://huggingface.co/collections/IFM/k2-horizon)
recipes/models/IFM/K2-Horizon-7B.yaml
+90
View File
@@ -0,0 +1,90 @@
meta:
title: "K2-Horizon-7B"
slug: "k2-horizon-7b"
provider: "IFM"
description: "Dense model for long-context research, fine-tuning, and cost-conscious deployment"
date_added: 2026-08-31
date_updated: 2026-09-02
tasks: [text]
performance_headline: "9B stored parameters including embeddings, with up to 512K context"
related_recipes: ["IFM/K2-Horizon-3.7B", "IFM/K2-Horizon-32B"]
hardware:
h200: verified
model:
model_id: "IFM/K2-Horizon-7B"
architecture: dense
parameter_count: "9B"
active_parameters: "9B"
context_length: 524288
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16"]
base_env: {}
features:
reasoning:
description: "Extract K2-Horizon reasoning into message.reasoning"
args: ["--reasoning-parser", "k2_horizon"]
tool_calling:
description: "Enable automatic tool choice with the K2-Horizon parser"
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
variants:
default:
precision: bf16
vram_minimum_gb: 22
description: "BF16 serving runtime with a 131K context baseline"
compatible_strategies: [single_node_tp, multi_node_tp, multi_node_tp_pp]
hardware_overrides: {}
guide: |
## Overview
K2-Horizon-7B is a compact dense model in the IFM K2-Horizon family,
with 9B stored parameters including embeddings and support for up to 524,288
tokens of context.
K2-Horizon is an open-weight IFM model family built for transparent
foundation-model research, staged checkpoint analysis, and practical
deployment. The series spans compact dense models such as K2-Horizon-7B for
local experimentation, dense mid-size and large models for high-quality
research workloads, and mixture-of-expert (MoE) models for higher-capacity
serving.
## Launch Commands
```bash
vllm serve IFM/K2-Horizon-7B \
--trust-remote-code \
--chat-template-content-format string \
--dtype bfloat16 \
--reasoning-parser k2_horizon \
--enable-auto-tool-choice \
--tool-call-parser k2_horizon
```
## Client usage
```python
from openai import OpenAI
client = OpenAI(
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
)
resp = client.chat.completions.create(
model="IFM/K2-Horizon-7B",
messages=[{"role": "user", "content": "Give me three primes above 100."}],
temperature=1.0, top_p=0.95, max_tokens=2048,
)
print(resp.choices[0].message.content)
```
### Thinking modes
The model supports selectable thinking through `chat_template_kwargs`, per
request or server-wide via `--default-chat-template-kwargs`:
- `{"reasoning_effort": "high"}` — full thinking (default).
- `{"reasoning_effort": "medium"}` — faster thinking.
- `{"reasoning_effort": "low"}` — fastest thinking.
## References
- [IFM/K2-Horizon-7B on Hugging Face](https://huggingface.co/IFM/K2-Horizon-7B)
- [K2 Horizon Collection](https://huggingface.co/collections/IFM/k2-horizon)
recipes/models/IFM/K2-Horizon-MoVA-36B-A4B.yaml
+97
View File
@@ -0,0 +1,97 @@
meta:
title: "K2-Horizon-MoVA-36B-A4B"
slug: "k2-horizon-mova-36b-a4b"
provider: "IFM"
description: "Sparse MoVA and MoE long-context model for research and production-style serving experiments"
date_added: 2026-08-31
date_updated: 2026-09-03
tasks: [text]
performance_headline: "37.44B stored parameters including embeddings / 5.95B active parameters per token"
related_recipes: ["IFM/K2-Horizon-32B", "IFM/K2-Horizon-375B-A23B"]
hardware:
h200: verified
model:
model_id: "IFM/K2-Horizon-MoVA-36B-A4B"
architecture: moe
parameter_count: "37.44B"
active_parameters: "5.95B"
context_length: 524288
base_args: ["--trust-remote-code", "--chat-template-content-format", "string", "--dtype", "bfloat16", "--enable-expert-parallel"]
base_env: {}
features:
reasoning:
description: "Extract K2-Horizon reasoning into message.reasoning"
args: ["--reasoning-parser", "k2_horizon"]
tool_calling:
description: "Enable automatic tool choice with the K2-Horizon parser"
args: ["--enable-auto-tool-choice", "--tool-call-parser", "k2_horizon"]
variants:
default:
precision: bf16
vram_minimum_gb: 90
description: "BF16 serving runtime with native vLLM execution"
compatible_strategies: [single_node_tep, multi_node_tep, multi_node_tp_pp]
default_strategy: single_node_tep
hardware_overrides: {}
strategy_overrides:
single_node_tep:
tp: 2
guide: |
## Overview
K2-Horizon-MoVA-36B-A4B is a mixture-of-expert (MoE) model in the IFM K2-Horizon
family. It combines MoVA attention and MoE layers, with 37.44B stored parameters
including embeddings, 5.95B active parameters per token, and support for up
to 524,288 tokens of context.
K2-Horizon is an open-weight IFM model family built for transparent
foundation-model research, staged checkpoint analysis, and practical
deployment. The series spans compact dense models for local experimentation,
dense mid-size and large models for high-quality research workloads, and
mixture-of-expert (MoE) models such as K2-Horizon-MoVA-36B-A4B for higher-capacity
serving.
## Launch Commands
```bash
vllm serve IFM/K2-Horizon-MoVA-36B-A4B \
--tensor-parallel-size 2 \
--trust-remote-code \
--chat-template-content-format string \
--dtype bfloat16 \
--enable-expert-parallel \
--reasoning-parser k2_horizon \
--enable-auto-tool-choice \
--tool-call-parser k2_horizon
```
## Client usage
```python
from openai import OpenAI
client = OpenAI(
api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600
)
resp = client.chat.completions.create(
model="IFM/K2-Horizon-MoVA-36B-A4B",
messages=[{"role": "user", "content": "Give me three primes above 100."}],
temperature=1.0, top_p=0.95, max_tokens=2048,
)
print(resp.choices[0].message.content)
```
### Thinking modes
The model supports selectable thinking through `chat_template_kwargs`, per
request or server-wide via `--default-chat-template-kwargs`:
- `{"reasoning_effort": "high"}` — full thinking (default).
- `{"reasoning_effort": "medium"}` — faster thinking.
- `{"reasoning_effort": "low"}` — fastest thinking.
## References
- [IFM/K2-Horizon-MoVA-36B-A4B on Hugging Face](https://huggingface.co/IFM/K2-Horizon-MoVA-36B-A4B)
- [K2 Horizon Collection](https://huggingface.co/collections/IFM/k2-horizon)
recipes/models/JetBrains/Mellum2-12B-A2.5B-Instruct.yaml
+153
View File
@@ -0,0 +1,153 @@
meta:
title: "Mellum2-12B-A2.5B-Instruct"
slug: "mellum2-12b-a2.5b-instruct"
provider: "JetBrains"
description: "JetBrains' instruction-tuned code MoE (12B total / 2.5B active) that answers directly without an externalized chain of thought — low-latency coding and tool use"
date_added: 2026-06-02
date_updated: 2026-06-02
tasks:
- text
performance_headline: "78.4 EvalPlus, 67.1 MultiPL-E — direct answers, fits on a single GPU"
related_recipes:
- "JetBrains/Mellum2-12B-A2.5B-Thinking"
hardware:
h200: verified
mi300x: verified
mi325x: verified
mi355x: verified
model:
model_id: "JetBrains/Mellum2-12B-A2.5B-Instruct"
min_vllm_version: "0.23.0"
architecture: moe
parameter_count: "12B"
active_parameters: "2.5B"
context_length: 131072
base_args:
- "--max-model-len"
- "131072"
- "--trust-remote-code"
base_env: {}
features:
tool_calling:
description: "Hermes tool-call parser for function calling"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "hermes"
opt_in_features: []
variants:
default:
precision: bf16
vram_minimum_gb: 29
description: "Native bfloat16 weights; fits comfortably on a single H200/H100/A100"
compatible_strategies:
- single_node_tp
- single_node_tep
- single_node_dep
hardware_overrides: {}
strategy_overrides:
single_node_tp:
tp: 1
guide: |
## Overview
[Mellum2-12B-A2.5B-Instruct](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Instruct)
is JetBrains' instruction-tuned code assistant. It shares the same Mixture-of-Experts
backbone as the rest of the Mellum2 family — 64 experts (8 activated per token), 12B total
/ 2.5B active parameters, sliding-window + full-attention layers, 131,072-token context —
but is post-trained (SFT + RLVR on math, coding, tool use, instruction following, reasoning,
and knowledge) to **answer directly, without an externalized chain of thought**. For complex
debugging, multi-step planning, or math/reasoning-heavy tasks where you want explicit
reasoning traces, use the
[Thinking](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Thinking) variant instead.
## Prerequisites
- Hardware: a single H200, H100, or A100 (~29 GB at bf16) is plenty. On AMD,
use MI300X / MI325X / MI355X with TP=1.
- vLLM **nightly** — `MellumForCausalLM` support landed after v0.22.0 and is not yet in a
stable release. Install the nightly wheels until the next tagged release ships.
### Install vLLM (nightly)
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --extra-index-url https://wheels.vllm.ai/nightly
```
## Launch command
Unlike the Thinking checkpoint, Instruct does not emit `<think>` blocks, so no
`--reasoning-parser` is needed.
### Single GPU
```bash
vllm serve JetBrains/Mellum2-12B-A2.5B-Instruct \
--max-model-len 131072 \
--trust-remote-code \
--tensor-parallel-size 1
# Optional: add tool calling.
vllm serve JetBrains/Mellum2-12B-A2.5B-Instruct \
--max-model-len 131072 \
--trust-remote-code \
--tensor-parallel-size 1 \
--enable-auto-tool-choice \
--tool-call-parser hermes
```
### Docker (AMD MI300X / MI325X / MI355X)
MI300X / MI325X / MI355X GPUs have larger per-GPU HBM, so you can use
`--max-model-len auto`.
```bash
# Plain serving
vllm serve JetBrains/Mellum2-12B-A2.5B-Instruct \
--max-model-len auto \
--trust-remote-code \
--tensor-parallel-size 1
# Add tool calling.
vllm serve JetBrains/Mellum2-12B-A2.5B-Instruct \
--max-model-len auto \
--trust-remote-code \
--tensor-parallel-size 1 \
--enable-auto-tool-choice \
--tool-call-parser hermes
```
## Client usage
JetBrains recommends sampling at `temperature=0.6`, `top_p=0.95`, `top_k=20`.
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
model="JetBrains/Mellum2-12B-A2.5B-Instruct",
messages=[{"role": "user", "content": "Write a Python function to reverse a string."}],
max_tokens=81920,
temperature=0.6,
top_p=0.95,
extra_body={"top_k": 20},
)
print(resp.choices[0].message.content)
```
## References
- [Model card](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Instruct)
- [vLLM support PR #43992](https://github.com/vllm-project/vllm/pull/43992)
- [Mellum2 Technical Report](https://arxiv.org/abs/2605.31268)
recipes/models/JetBrains/Mellum2-12B-A2.5B-Thinking.yaml
+146
View File
@@ -0,0 +1,146 @@
meta:
title: "Mellum2-12B-A2.5B-Thinking"
slug: "mellum2-12b-a2.5b-thinking"
provider: "JetBrains"
description: "JetBrains' reasoning-augmented code MoE (12B total / 2.5B active) that emits explicit <think> chains for debugging, planning, and agentic coding"
date_added: 2026-06-02
date_updated: 2026-06-02
tasks:
- text
performance_headline: "69.9 LiveCodeBench v6, 58.4 AIME — fits on a single GPU"
related_recipes:
- "JetBrains/Mellum2-12B-A2.5B-Instruct"
hardware:
h200: verified
model:
model_id: "JetBrains/Mellum2-12B-A2.5B-Thinking"
min_vllm_version: "0.23.0"
architecture: moe
parameter_count: "12B"
active_parameters: "2.5B"
context_length: 131072
base_args:
- "--max-model-len"
- "131072"
base_env: {}
features:
reasoning:
description: "Parse the model's <think>...</think> reasoning blocks (Qwen3-style parser)"
args:
- "--reasoning-parser"
- "qwen3"
tool_calling:
description: "Hermes tool-call parser for function calling"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "hermes"
spec_decoding:
description: "DFlash speculative decoding with an externally trained draft model; the served checkpoint is unchanged."
args:
- "--speculative-config"
- '{"model": "RedHatAI/Mellum2-12B-A2.5B-Thinking-speculator.dflash", "num_speculative_tokens": 7, "method": "dflash"}'
opt_in_features:
- spec_decoding
variants:
default:
precision: bf16
vram_minimum_gb: 29
description: "Native bfloat16 weights; fits comfortably on a single H200/H100/A100"
fp8:
model_id: "RedHatAI/Mellum2-12B-A2.5B-Thinking-FP8-Dynamic"
precision: fp8
vram_minimum_gb: 15
description: "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization"
extra_args:
- "--kv-cache-dtype"
- "fp8"
nvfp4:
model_id: "RedHatAI/Mellum2-12B-A2.5B-Thinking-NVFP4"
precision: nvfp4
vram_minimum_gb: 8
description: "RedHatAI NVFP4 quantization"
compatible_strategies:
- single_node_tp
- single_node_tep
- single_node_dep
strategy_overrides:
single_node_tp:
tp: 1
guide: |
## Overview
[Mellum2-12B-A2.5B-Thinking](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Thinking)
is JetBrains' reasoning-augmented code assistant. It uses a Mixture-of-Experts architecture
with 64 experts (8 activated per token) — 12B total parameters, 2.5B active — combining
sliding-window and full-attention layers for a 131,072-token context. The model emits its
chain-of-thought inside `<think>...</think>` blocks before the final answer, making it
suited to complex debugging, multi-step planning, and agentic workflows. For direct,
low-latency answers without reasoning traces, use the
[Instruct](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Instruct) variant instead.
## Prerequisites
- Hardware: a single H200, H100, or A100 (~29 GB at bf16) is plenty
- vLLM **nightly** — `MellumForCausalLM` support landed after v0.22.0 and is not yet in a
stable release. Install the nightly wheels until the next tagged release ships.
### Install vLLM (nightly)
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --extra-index-url https://wheels.vllm.ai/nightly
```
## Launch command
```bash
# With reasoning (recommended for the Thinking checkpoint)
vllm serve JetBrains/Mellum2-12B-A2.5B-Thinking \
--max-model-len 131072 \
--reasoning-parser qwen3
# Add tool calling
vllm serve JetBrains/Mellum2-12B-A2.5B-Thinking \
--max-model-len 131072 \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser hermes
```
## Client usage
JetBrains recommends sampling at `temperature=0.6`, `top_p=0.95`, `top_k=20` for the
Thinking checkpoint.
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
resp = client.chat.completions.create(
model="JetBrains/Mellum2-12B-A2.5B-Thinking",
messages=[{"role": "user", "content": "Is 1024 a power of 2? Explain your reasoning."}],
max_tokens=81920,
temperature=0.6,
top_p=0.95,
extra_body={"top_k": 20},
)
print(resp.choices[0].message.content)
```
## References
- [Model card](https://huggingface.co/JetBrains/Mellum2-12B-A2.5B-Thinking)
- [vLLM support PR #43992](https://github.com/vllm-project/vllm/pull/43992)
- [Mellum2 Technical Report](https://arxiv.org/abs/2605.31268)
recipes/models/LiquidAI/LFM2.5-1.2B-Base.yaml
+137
View File
@@ -0,0 +1,137 @@
meta:
title: "LFM2.5 1.2B Base"
slug: "lfm2.5-1.2b-base"
provider: "LiquidAI"
description: "Liquid AI's 1.2B pretrained base model on the LFM2 hybrid conv+attention backbone — a text-completion and fine-tuning foundation (no chat template)."
date_added: 2026-06-25
date_updated: 2026-06-24
tasks:
- text
performance_headline: "1.2B pretrained base (no chat template) — completions endpoint, ideal as a fine-tuning base"
related_recipes:
- LiquidAI/LFM2.5-1.2B-Instruct
- LiquidAI/LFM2.5-350M
hardware:
h100: verified
h200: verified
b200: verified
model:
model_id: "LiquidAI/LFM2.5-1.2B-Base"
min_vllm_version: "0.23.0"
architecture: dense
parameter_count: "1.2B"
active_parameters: "1.2B"
context_length: 32768
base_args: []
base_env: {}
variants:
default:
precision: bf16
vram_minimum_gb: 3
description: "Full BF16 — single 8 GB+ NVIDIA GPU with room for KV cache"
compatible_strategies:
- single_node_tp
hardware_overrides:
# Blackwell (B200) only: route RMSNorm to the bundled oink kernels (measured ~+2.6% on B200,
# output identical). Inert (falls back to native) on non-Blackwell GPUs.
blackwell:
extra_env:
VLLM_USE_OINK_OPS: "1"
strategy_overrides:
single_node_tp:
tp: 1
guide: |
## Overview
[LFM2.5-1.2B-Base](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Base) is the pretrained base
checkpoint behind [Liquid AI's](https://www.liquid.ai/) LFM2.5-1.2B instruct models, built on
the **LFM2 hybrid backbone** (short-range gated convolution blocks interleaved with grouped-query
attention). It is **not** instruction-tuned and has no chat template — use the
`/v1/completions` endpoint (raw text continuation), or fine-tune it for your own task.
### Key Features
- **Pretrained base**: No chat template, no instruction tuning — a clean foundation for fine-tuning or raw completion.
- **Hybrid backbone**: Gated short convolutions interleaved with grouped-query attention — a smaller KV cache and lower decode latency than a same-size full-attention transformer.
- **32K context**: 32,768-token context window.
- **Native vLLM support**: Served via the `Lfm2ForCausalLM` architecture.
### Supported Variants
Dense:
- `LiquidAI/LFM2.5-230M` (230M)
- `LiquidAI/LFM2.5-350M` (350M)
- `LiquidAI/LFM2.5-1.2B-Instruct` (1.2B)
- `LiquidAI/LFM2.5-1.2B-Thinking` (1.2B, reasoning)
- `LiquidAI/LFM2.5-1.2B-JP` / `LiquidAI/LFM2.5-1.2B-JP-202606` (Japanese)
- `LiquidAI/LFM2.5-1.2B-Base` (pretrained base)
MoE:
- `LiquidAI/LFM2.5-8B-A1B` (8B total / ~1B active)
Vision-Language:
- `LiquidAI/LFM2.5-VL-450M`, `LiquidAI/LFM2.5-VL-1.6B`
See the [LFM2.5 usage guide](../../LiquidAI/LFM2.5) for the full family.
## Prerequisites
- **Hardware**: 1× GPU with ≥8 GB VRAM.
- **vLLM**: ≥ 0.23.0.
### pip (NVIDIA CUDA)
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```
## Deployment Configurations
### Quick Start (Single GPU, BF16)
```bash
vllm serve LiquidAI/LFM2.5-1.2B-Base
```
### Docker (NVIDIA)
```bash
docker run -itd --name lfm2.5-base \
--ipc=host --network host --shm-size 16G --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:latest \
--model LiquidAI/LFM2.5-1.2B-Base \
--host 0.0.0.0 --port 8000
```
## Client Usage
### Text Completion
This is a base model — use the **completions** endpoint, not chat. Recommended sampling:
temperature `0.3`, `min_p 0.15`, `repetition_penalty 1.05`.
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.completions.create(
model="LiquidAI/LFM2.5-1.2B-Base",
prompt="The three laws of thermodynamics are:",
temperature=0.3,
extra_body={"min_p": 0.15, "repetition_penalty": 1.05},
)
print(response.choices[0].text)
```
## Configuration Tips
- Use `/v1/completions` (raw continuation) — there is no chat template on a base model.
- For fine-tuning, this checkpoint is the recommended starting point for custom LFM2.5-1.2B tasks.
- Set `--max-model-len` to match your workload (up to 32K).
- Sampling presets are per-request client defaults — don't bake them into `vllm serve`.
## References
- [Model card](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Base)
- [LFM2.5 usage guide](../../LiquidAI/LFM2.5)
- [Liquid AI](https://www.liquid.ai/)
- [vLLM supported models](https://docs.vllm.ai/en/latest/models/supported_models.html)
recipes/models/LiquidAI/LFM2.5-1.2B-Instruct.yaml
+186
View File
@@ -0,0 +1,186 @@
meta:
title: "LFM2.5 1.2B Instruct"
slug: "lfm2.5-1.2b-instruct"
provider: "LiquidAI"
description: "Liquid AI's 1.2B instruction-tuned model on the LFM2 hybrid conv+attention backbone, with tool calling and a 32K context window on a single small GPU."
date_added: 2026-06-25
date_updated: 2026-06-24
tasks:
- text
performance_headline: "1.2B hybrid (gated conv + GQA) instruct model with tool calling — runs on a single 8 GB+ GPU"
related_recipes:
- LiquidAI/LFM2.5-350M
- LiquidAI/LFM2.5-1.2B-Thinking
- LiquidAI/LFM2.5-8B-A1B
hardware:
h100: verified
h200: verified
b200: verified
model:
model_id: "LiquidAI/LFM2.5-1.2B-Instruct"
min_vllm_version: "0.23.0"
architecture: dense
parameter_count: "1.2B"
active_parameters: "1.2B"
context_length: 32768
base_args: []
base_env: {}
features:
tool_calling:
description: "Surface LFM2.5's Pythonic tool calls (wrapped in <|tool_call_start|>…<|tool_call_end|>) as OpenAI tool_calls via vLLM's native lfm2 parser"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "lfm2"
variants:
default:
precision: bf16
vram_minimum_gb: 3
description: "Full BF16 — single 8 GB+ NVIDIA GPU with room for KV cache"
compatible_strategies:
- single_node_tp
hardware_overrides:
# Blackwell (B200) only: route RMSNorm to the bundled oink kernels (measured ~+2.6% on B200,
# output identical). Inert (falls back to native) on non-Blackwell GPUs.
blackwell:
extra_env:
VLLM_USE_OINK_OPS: "1"
strategy_overrides:
single_node_tp:
tp: 1
guide: |
## Overview
[LFM2.5-1.2B-Instruct](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Instruct) is a 1.2B
instruction-tuned model from [Liquid AI](https://www.liquid.ai/), built on the **LFM2 hybrid
backbone**: short-range gated convolution blocks interleaved with grouped-query attention.
The hybrid design keeps the KV cache small and decode fast, so the model serves comfortably on
a single small GPU while supporting a 32K context window and Pythonic tool calling — all
through vLLM's OpenAI-compatible API.
### Key Features
- **Hybrid backbone**: Gated short convolutions interleaved with grouped-query attention — a smaller KV cache and lower decode latency than a same-size full-attention transformer.
- **32K context**: 32,768-token context window.
- **Tool calling**: Pythonic tool calls (`<|tool_call_start|>…<|tool_call_end|>`) surfaced as OpenAI `tool_calls` by vLLM's native `lfm2` parser.
- **Native vLLM support**: Served via the `Lfm2ForCausalLM` architecture.
- **Edge-ready family**: Day-one support across llama.cpp, MLX, ONNX, vLLM, and SGLang.
### Supported Variants
Dense:
- `LiquidAI/LFM2.5-230M` (230M)
- `LiquidAI/LFM2.5-350M` (350M)
- `LiquidAI/LFM2.5-1.2B-Instruct` (1.2B)
- `LiquidAI/LFM2.5-1.2B-Thinking` (1.2B, reasoning)
- `LiquidAI/LFM2.5-1.2B-JP` / `LiquidAI/LFM2.5-1.2B-JP-202606` (Japanese)
- `LiquidAI/LFM2.5-1.2B-Base` (pretrained base)
MoE:
- `LiquidAI/LFM2.5-8B-A1B` (8B total / ~1B active)
Vision-Language:
- `LiquidAI/LFM2.5-VL-450M`, `LiquidAI/LFM2.5-VL-1.6B`
See the [LFM2.5 usage guide](../../LiquidAI/LFM2.5) for the full family.
## Prerequisites
- **Hardware**: 1× GPU with ≥8 GB VRAM (e.g. L4, A10, RTX 4090, H100).
- **vLLM**: ≥ 0.23.0.
### pip (NVIDIA CUDA)
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```
## Deployment Configurations
### Quick Start (Single GPU, BF16)
```bash
vllm serve LiquidAI/LFM2.5-1.2B-Instruct
```
Cap the context to fit a smaller GPU (the model supports up to 32K):
```bash
vllm serve LiquidAI/LFM2.5-1.2B-Instruct --max-model-len 32768
```
### Full-Featured Server Launch
Enables tool calling:
```bash
vllm serve LiquidAI/LFM2.5-1.2B-Instruct \
--enable-auto-tool-choice \
--tool-call-parser lfm2 \
--host 0.0.0.0 --port 8000
```
### Docker (NVIDIA)
```bash
docker run -itd --name lfm2.5-1.2b \
--ipc=host --network host --shm-size 16G --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:latest \
--model LiquidAI/LFM2.5-1.2B-Instruct \
--host 0.0.0.0 --port 8000
```
## Client Usage
### Text Generation
The model card recommends temperature `0.1`, `top_k 50`, `repetition_penalty 1.05`. `top_k` and
`repetition_penalty` are vLLM extra sampling params — pass them via `extra_body`.
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="LiquidAI/LFM2.5-1.2B-Instruct",
messages=[{"role": "user", "content": "What is C. elegans? Answer in one sentence."}],
temperature=0.1,
extra_body={"top_k": 50, "repetition_penalty": 1.05},
)
print(response.choices[0].message.content)
```
### Tool Calling
Launch with `--enable-auto-tool-choice --tool-call-parser lfm2`, then pass `tools=[…]`. The
`lfm2` parser converts the model's Pythonic call into a standard `tool_calls` array.
```python
response = client.chat.completions.create(
model="LiquidAI/LFM2.5-1.2B-Instruct",
messages=[{"role": "user", "content": "What's the weather in Paris?"}],
tools=[{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get current weather for a location",
"parameters": {"type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"]},
},
}],
temperature=0.1,
extra_body={"top_k": 50, "repetition_penalty": 1.05},
)
print(response.choices[0].message.tool_calls)
```
### Structured Outputs
vLLM's guided decoding constrains output to a JSON schema via `response_format`. The model does
not see the schema itself — put semantic instructions (units, formatting) in the system prompt.
## Configuration Tips
- Set `--max-model-len` to match your workload (up to 32K); lowering it frees VRAM for KV cache.
- `--gpu-memory-utilization 0.90–0.95` maximizes KV cache capacity.
- FP8 KV cache (`--kv-cache-dtype fp8`) roughly halves KV memory.
- Sampling presets are per-request client defaults — don't bake them into `vllm serve`.
## References
- [Model card](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Instruct)
- [LFM2.5 usage guide](../../LiquidAI/LFM2.5)
- [Liquid AI](https://www.liquid.ai/)
- [vLLM supported models](https://docs.vllm.ai/en/latest/models/supported_models.html)
recipes/models/LiquidAI/LFM2.5-1.2B-JP-202606.yaml
+158
View File
@@ -0,0 +1,158 @@
meta:
title: "LFM2.5 1.2B JP (202606)"
slug: "lfm2.5-1.2b-jp-202606"
provider: "LiquidAI"
description: "Liquid AI's updated (2026-06) 1.2B Japanese chat model on the LFM2 hybrid conv+attention backbone — adds tool calling, with a 32K context window."
date_added: 2026-06-25
date_updated: 2026-06-24
tasks:
- text
performance_headline: "Updated Japanese-tuned 1.2B hybrid chat model with tool calling — single small GPU"
related_recipes:
- LiquidAI/LFM2.5-1.2B-JP
- LiquidAI/LFM2.5-1.2B-Instruct
hardware:
h100: verified
h200: verified
b200: verified
model:
model_id: "LiquidAI/LFM2.5-1.2B-JP-202606"
min_vllm_version: "0.23.0"
architecture: dense
parameter_count: "1.2B"
active_parameters: "1.2B"
context_length: 32768
base_args: []
base_env: {}
features:
tool_calling:
description: "Surface LFM2.5's Pythonic tool calls (wrapped in <|tool_call_start|>…<|tool_call_end|>) as OpenAI tool_calls via vLLM's native lfm2 parser"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "lfm2"
variants:
default:
precision: bf16
vram_minimum_gb: 3
description: "Full BF16 — single 8 GB+ NVIDIA GPU with room for KV cache"
compatible_strategies:
- single_node_tp
hardware_overrides:
# Blackwell (B200) only: route RMSNorm to the bundled oink kernels (measured ~+2.6% on B200,
# output identical). Inert (falls back to native) on non-Blackwell GPUs.
blackwell:
extra_env:
VLLM_USE_OINK_OPS: "1"
strategy_overrides:
single_node_tp:
tp: 1
guide: |
## Overview
[LFM2.5-1.2B-JP-202606](https://huggingface.co/LiquidAI/LFM2.5-1.2B-JP-202606) is the updated
(June 2026) Japanese-specialized 1.2B chat model from [Liquid AI](https://www.liquid.ai/),
adding tool calling on top of the **LFM2 hybrid backbone** (short-range gated convolution
blocks interleaved with grouped-query attention). It serves on a single small GPU through
vLLM's OpenAI-compatible API.
### Key Features
- **Japanese-specialized**: Updated June 2026 checkpoint tuned for Japanese chat.
- **Hybrid backbone**: Gated short convolutions interleaved with grouped-query attention — a smaller KV cache and lower decode latency than a same-size full-attention transformer.
- **Tool calling**: Pythonic tool calls (`<|tool_call_start|>…<|tool_call_end|>`) surfaced as OpenAI `tool_calls` by vLLM's native `lfm2` parser.
- **32K context**: 32,768-token context window.
- **Native vLLM support**: Served via the `Lfm2ForCausalLM` architecture.
### Supported Variants
Dense:
- `LiquidAI/LFM2.5-230M` (230M)
- `LiquidAI/LFM2.5-350M` (350M)
- `LiquidAI/LFM2.5-1.2B-Instruct` (1.2B)
- `LiquidAI/LFM2.5-1.2B-Thinking` (1.2B, reasoning)
- `LiquidAI/LFM2.5-1.2B-JP` / `LiquidAI/LFM2.5-1.2B-JP-202606` (Japanese)
- `LiquidAI/LFM2.5-1.2B-Base` (pretrained base)
MoE:
- `LiquidAI/LFM2.5-8B-A1B` (8B total / ~1B active)
Vision-Language:
- `LiquidAI/LFM2.5-VL-450M`, `LiquidAI/LFM2.5-VL-1.6B`
See the [LFM2.5 usage guide](../../LiquidAI/LFM2.5) for the full family.
## Prerequisites
- **Hardware**: 1× GPU with ≥8 GB VRAM.
- **vLLM**: ≥ 0.23.0.
### pip (NVIDIA CUDA)
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```
## Deployment Configurations
### Quick Start (Single GPU, BF16)
```bash
vllm serve LiquidAI/LFM2.5-1.2B-JP-202606
```
### Full-Featured Server Launch
Enables tool calling:
```bash
vllm serve LiquidAI/LFM2.5-1.2B-JP-202606 \
--enable-auto-tool-choice \
--tool-call-parser lfm2 \
--host 0.0.0.0 --port 8000
```
### Docker (NVIDIA)
```bash
docker run -itd --name lfm2.5-jp-202606 \
--ipc=host --network host --shm-size 16G --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:latest \
--model LiquidAI/LFM2.5-1.2B-JP-202606 \
--host 0.0.0.0 --port 8000
```
## Client Usage
### Text Generation
The updated checkpoint uses temperature `0.1`, `top_k 50`, `repetition_penalty 1.05` (`top_k`
and `repetition_penalty` ride in `extra_body`).
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="LiquidAI/LFM2.5-1.2B-JP-202606",
messages=[{"role": "user", "content": "京都でおすすめの観光スポットを3つ教えてください。"}],
temperature=0.1,
extra_body={"top_k": 50, "repetition_penalty": 1.05},
)
print(response.choices[0].message.content)
```
### Tool Calling
Launch with `--enable-auto-tool-choice --tool-call-parser lfm2`, then pass `tools=[…]`; the
`lfm2` parser converts the model's Pythonic call into a standard `tool_calls` array.
## Configuration Tips
- Set `--max-model-len` to match your workload (up to 32K); lowering it frees VRAM for KV cache.
- `--gpu-memory-utilization 0.90–0.95` maximizes KV cache capacity.
- Sampling presets are per-request client defaults — don't bake them into `vllm serve`.
## References
- [Model card](https://huggingface.co/LiquidAI/LFM2.5-1.2B-JP-202606)
- [LFM2.5 usage guide](../../LiquidAI/LFM2.5)
- [Liquid AI](https://www.liquid.ai/)
- [vLLM supported models](https://docs.vllm.ai/en/latest/models/supported_models.html)
recipes/models/LiquidAI/LFM2.5-1.2B-JP.yaml
+137
View File
@@ -0,0 +1,137 @@
meta:
title: "LFM2.5 1.2B JP"
slug: "lfm2.5-1.2b-jp"
provider: "LiquidAI"
description: "Liquid AI's 1.2B Japanese-specialized chat model on the LFM2 hybrid conv+attention backbone, with a 32K context window on a single small GPU."
date_added: 2026-06-25
date_updated: 2026-06-24
tasks:
- text
performance_headline: "Japanese-tuned 1.2B hybrid chat model — single small GPU"
related_recipes:
- LiquidAI/LFM2.5-1.2B-JP-202606
- LiquidAI/LFM2.5-1.2B-Instruct
hardware:
h100: verified
h200: verified
b200: verified
model:
model_id: "LiquidAI/LFM2.5-1.2B-JP"
min_vllm_version: "0.23.0"
architecture: dense
parameter_count: "1.2B"
active_parameters: "1.2B"
context_length: 32768
base_args: []
base_env: {}
variants:
default:
precision: bf16
vram_minimum_gb: 3
description: "Full BF16 — single 8 GB+ NVIDIA GPU with room for KV cache"
compatible_strategies:
- single_node_tp
hardware_overrides:
# Blackwell (B200) only: route RMSNorm to the bundled oink kernels (measured ~+2.6% on B200,
# output identical). Inert (falls back to native) on non-Blackwell GPUs.
blackwell:
extra_env:
VLLM_USE_OINK_OPS: "1"
strategy_overrides:
single_node_tp:
tp: 1
guide: |
## Overview
[LFM2.5-1.2B-JP](https://huggingface.co/LiquidAI/LFM2.5-1.2B-JP) is the Japanese-specialized
1.2B chat model from [Liquid AI](https://www.liquid.ai/), built on the **LFM2 hybrid backbone**
(short-range gated convolution blocks interleaved with grouped-query attention). It serves on a
single small GPU through vLLM's OpenAI-compatible API. For the latest Japanese checkpoint with
tool calling, see
[LFM2.5-1.2B-JP-202606](https://huggingface.co/LiquidAI/LFM2.5-1.2B-JP-202606).
### Key Features
- **Japanese-specialized**: Tuned for Japanese chat and instruction following.
- **Hybrid backbone**: Gated short convolutions interleaved with grouped-query attention — a smaller KV cache and lower decode latency than a same-size full-attention transformer.
- **32K context**: 32,768-token context window.
- **Native vLLM support**: Served via the `Lfm2ForCausalLM` architecture.
### Supported Variants
Dense:
- `LiquidAI/LFM2.5-230M` (230M)
- `LiquidAI/LFM2.5-350M` (350M)
- `LiquidAI/LFM2.5-1.2B-Instruct` (1.2B)
- `LiquidAI/LFM2.5-1.2B-Thinking` (1.2B, reasoning)
- `LiquidAI/LFM2.5-1.2B-JP` / `LiquidAI/LFM2.5-1.2B-JP-202606` (Japanese)
- `LiquidAI/LFM2.5-1.2B-Base` (pretrained base)
MoE:
- `LiquidAI/LFM2.5-8B-A1B` (8B total / ~1B active)
Vision-Language:
- `LiquidAI/LFM2.5-VL-450M`, `LiquidAI/LFM2.5-VL-1.6B`
See the [LFM2.5 usage guide](../../LiquidAI/LFM2.5) for the full family.
## Prerequisites
- **Hardware**: 1× GPU with ≥8 GB VRAM.
- **vLLM**: ≥ 0.23.0.
### pip (NVIDIA CUDA)
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```
## Deployment Configurations
### Quick Start (Single GPU, BF16)
```bash
vllm serve LiquidAI/LFM2.5-1.2B-JP
```
### Docker (NVIDIA)
```bash
docker run -itd --name lfm2.5-jp \
--ipc=host --network host --shm-size 16G --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:latest \
--model LiquidAI/LFM2.5-1.2B-JP \
--host 0.0.0.0 --port 8000
```
## Client Usage
### Text Generation
The card recommends temperature `0.3`, `min_p 0.15`, `repetition_penalty 1.05` for the JP
checkpoint (`min_p` and `repetition_penalty` ride in `extra_body`).
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="LiquidAI/LFM2.5-1.2B-JP",
messages=[{"role": "user", "content": "日本の四季について簡単に説明してください。"}],
temperature=0.3,
extra_body={"min_p": 0.15, "repetition_penalty": 1.05},
)
print(response.choices[0].message.content)
```
## Configuration Tips
- Set `--max-model-len` to match your workload (up to 32K); lowering it frees VRAM for KV cache.
- `--gpu-memory-utilization 0.90–0.95` maximizes KV cache capacity.
- Sampling presets are per-request client defaults — don't bake them into `vllm serve`.
## References
- [Model card](https://huggingface.co/LiquidAI/LFM2.5-1.2B-JP)
- [LFM2.5 usage guide](../../LiquidAI/LFM2.5)
- [Liquid AI](https://www.liquid.ai/)
- [vLLM supported models](https://docs.vllm.ai/en/latest/models/supported_models.html)
recipes/models/LiquidAI/LFM2.5-1.2B-Thinking.yaml
+180
View File
@@ -0,0 +1,180 @@
meta:
title: "LFM2.5 1.2B Thinking"
slug: "lfm2.5-1.2b-thinking"
provider: "LiquidAI"
description: "Liquid AI's 1.2B reasoning model on the LFM2 hybrid conv+attention backbone — <think> chain-of-thought, tool calling, and 32K context on a single small GPU."
date_added: 2026-06-25
date_updated: 2026-06-24
tasks:
- text
performance_headline: "1.2B hybrid reasoning model with <think> chain-of-thought and tool calling — single small GPU"
related_recipes:
- LiquidAI/LFM2.5-1.2B-Instruct
- LiquidAI/LFM2.5-8B-A1B
hardware:
h100: verified
h200: verified
b200: verified
model:
model_id: "LiquidAI/LFM2.5-1.2B-Thinking"
min_vllm_version: "0.23.0"
architecture: dense
parameter_count: "1.2B"
active_parameters: "1.2B"
context_length: 32768
base_args: []
base_env: {}
features:
reasoning:
description: "Split the model's <think>…</think> chain-of-thought into a separate reasoning field via vLLM's qwen3 parser"
args:
- "--reasoning-parser"
- "qwen3"
tool_calling:
description: "Surface LFM2.5's Pythonic tool calls (wrapped in <|tool_call_start|>…<|tool_call_end|>) as OpenAI tool_calls via vLLM's native lfm2 parser"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "lfm2"
variants:
default:
precision: bf16
vram_minimum_gb: 3
description: "Full BF16 — single 8 GB+ NVIDIA GPU with room for KV cache"
compatible_strategies:
- single_node_tp
hardware_overrides:
# Blackwell (B200) only: route RMSNorm to the bundled oink kernels (measured ~+2.6% on B200,
# output identical). Inert (falls back to native) on non-Blackwell GPUs.
blackwell:
extra_env:
VLLM_USE_OINK_OPS: "1"
strategy_overrides:
single_node_tp:
tp: 1
guide: |
## Overview
[LFM2.5-1.2B-Thinking](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Thinking) is the reasoning
variant of [Liquid AI's](https://www.liquid.ai/) 1.2B LFM2.5 model, built on the **LFM2 hybrid
backbone** (short-range gated convolution blocks interleaved with grouped-query attention). On
reasoning-heavy tasks it produces an explicit `<think>…</think>` chain-of-thought before its
final answer, so it reasons well above its weight class while still serving on a single small
GPU through vLLM's OpenAI-compatible API.
### Key Features
- **Hybrid backbone**: Gated short convolutions interleaved with grouped-query attention — a smaller KV cache and lower decode latency than a same-size full-attention transformer.
- **`<think>` reasoning**: Emits an explicit `<think>…</think>` chain-of-thought on non-trivial problems; vLLM's `qwen3` parser splits it into a separate `reasoning` field.
- **32K context**: 32,768-token context window.
- **Tool calling**: Pythonic tool calls surfaced as OpenAI `tool_calls` by vLLM's native `lfm2` parser.
- **Native vLLM support**: Served via the `Lfm2ForCausalLM` architecture.
### Supported Variants
Dense:
- `LiquidAI/LFM2.5-230M` (230M)
- `LiquidAI/LFM2.5-350M` (350M)
- `LiquidAI/LFM2.5-1.2B-Instruct` (1.2B)
- `LiquidAI/LFM2.5-1.2B-Thinking` (1.2B, reasoning)
- `LiquidAI/LFM2.5-1.2B-JP` / `LiquidAI/LFM2.5-1.2B-JP-202606` (Japanese)
- `LiquidAI/LFM2.5-1.2B-Base` (pretrained base)
MoE:
- `LiquidAI/LFM2.5-8B-A1B` (8B total / ~1B active, also a reasoning model)
Vision-Language:
- `LiquidAI/LFM2.5-VL-450M`, `LiquidAI/LFM2.5-VL-1.6B`
See the [LFM2.5 usage guide](../../LiquidAI/LFM2.5) for the full family.
## Prerequisites
- **Hardware**: 1× GPU with ≥8 GB VRAM.
- **vLLM**: ≥ 0.23.0.
### pip (NVIDIA CUDA)
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```
## Deployment Configurations
### Quick Start (Single GPU, BF16)
Enable the reasoning parser so the chain-of-thought is returned in `reasoning`:
```bash
vllm serve LiquidAI/LFM2.5-1.2B-Thinking \
--reasoning-parser qwen3
```
### Full-Featured Server Launch
Reasoning **and** tool calling:
```bash
vllm serve LiquidAI/LFM2.5-1.2B-Thinking \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser lfm2 \
--host 0.0.0.0 --port 8000
```
### Docker (NVIDIA)
```bash
docker run -itd --name lfm2.5-thinking \
--ipc=host --network host --shm-size 16G --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:latest \
--model LiquidAI/LFM2.5-1.2B-Thinking \
--reasoning-parser qwen3 \
--host 0.0.0.0 --port 8000
```
## Client Usage
### Reasoning Mode
The card recommends a low temperature for the reasoning model — temperature `0.05`, `top_k 50`,
`repetition_penalty 1.05`. Do **not** set `max_tokens` — capping it can truncate the
chain-of-thought mid-stream.
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="LiquidAI/LFM2.5-1.2B-Thinking",
messages=[{"role": "user", "content": "A snail climbs 3 ft up a 20 ft well each day and slides 2 ft back each night. How many days to reach the top?"}],
temperature=0.05,
extra_body={"top_k": 50, "repetition_penalty": 1.05},
)
msg = response.choices[0].message
print("reasoning:", msg.reasoning)
print("answer:", msg.content)
```
> **Note**: the model opens the `<think>` channel for non-trivial problems; a simple prompt may
> be answered directly, in which case `reasoning` is empty. That is expected behavior —
> the `qwen3` parser extracts the block whenever it is present.
### Tool Calling
Add `--enable-auto-tool-choice --tool-call-parser lfm2` at launch, then pass `tools=[…]`; the
model can reason about which tool to call before emitting it.
### Structured Outputs
vLLM's guided decoding constrains output to a JSON schema via `response_format`. The model does
not see the schema itself — put semantic instructions in the system prompt.
## Configuration Tips
- Give reasoning room: leave `max_tokens` unset so the `<think>` block and answer aren't truncated.
- Set `--max-model-len` to match your workload (up to 32K).
- `--gpu-memory-utilization 0.90–0.95` maximizes KV cache capacity.
- Sampling presets are per-request client defaults — don't bake them into `vllm serve`.
## References
- [Model card](https://huggingface.co/LiquidAI/LFM2.5-1.2B-Thinking)
- [LFM2.5 usage guide](../../LiquidAI/LFM2.5)
- [Liquid AI](https://www.liquid.ai/)
- [vLLM supported models](https://docs.vllm.ai/en/latest/models/supported_models.html)
recipes/models/LiquidAI/LFM2.5-230M.yaml
+184
View File
@@ -0,0 +1,184 @@
meta:
title: "LFM2.5 230M"
slug: "lfm2.5-230m"
provider: "LiquidAI"
description: "Liquid AI's most compact LFM2.5 chat model (230M) on the LFM2 hybrid conv+attention backbone — distilled from LFM2.5-350M, tool calling and 32K context, built for edge / on-device serving."
date_added: 2026-06-25
date_updated: 2026-06-25
tasks:
- text
performance_headline: "230M hybrid instruct model with tool calling — runs from low-cost CPUs to production GPUs, ideal for tightest on-device budgets"
related_recipes:
- LiquidAI/LFM2.5-350M
- LiquidAI/LFM2.5-1.2B-Instruct
- LiquidAI/LFM2.5-8B-A1B
hardware:
h100: verified
h200: verified
b200: verified
model:
model_id: "LiquidAI/LFM2.5-230M"
min_vllm_version: "0.23.0"
architecture: dense
parameter_count: "230M"
active_parameters: "230M"
context_length: 32768
base_args: []
base_env: {}
features:
tool_calling:
description: "Surface LFM2.5's Pythonic tool calls (wrapped in <|tool_call_start|>…<|tool_call_end|>) as OpenAI tool_calls via vLLM's native lfm2 parser"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "lfm2"
variants:
default:
precision: bf16
vram_minimum_gb: 1
description: "Full BF16 — fits essentially any GPU (~0.5 GB of weights)"
compatible_strategies:
- single_node_tp
hardware_overrides: {}
strategy_overrides:
single_node_tp:
tp: 1
guide: |
## Overview
[LFM2.5-230M](https://huggingface.co/LiquidAI/LFM2.5-230M) is the most compact chat model in
[Liquid AI's](https://www.liquid.ai/) LFM2.5 family, built on the **LFM2 hybrid backbone**:
short-range gated convolution blocks interleaved with grouped-query attention. It was
**distilled from LFM2.5-350M** and refined with multi-stage reinforcement learning, then
trained on a 19T-token budget — keeping memory and latency low enough for the tightest edge and
on-device budgets while still supporting tool calling and a 32K context window through vLLM's
OpenAI-compatible API.
### Key Features
- **Hybrid backbone**: 14 layers — 8 double-gated short-convolution blocks interleaved with 6 grouped-query attention blocks — for a smaller KV cache and lower decode latency than a same-size full-attention transformer.
- **32K context**: Long-context support (`context_length = 32768`).
- **Tool calling**: Pythonic tool calls (`<|tool_call_start|>…<|tool_call_end|>`) surfaced as OpenAI `tool_calls` by vLLM's native `lfm2` parser.
- **Native vLLM support**: Served via the `Lfm2ForCausalLM` architecture — no `--trust-remote-code` required.
- **Edge-ready**: ~0.5 GB of BF16 weights — runs from low-cost CPUs to production GPUs (213 tok/s decode on a Galaxy S25 Ultra, 42 tok/s on a Raspberry Pi 5).
Best suited for data extraction and lightweight on-device agentic pipelines; the model card
notes it is **not** recommended for reasoning-heavy workloads (advanced math, code generation,
creative writing) — reach for a larger sibling there.
### Supported Variants
Dense:
- `LiquidAI/LFM2.5-230M` (230M)
- `LiquidAI/LFM2.5-350M` (350M)
- `LiquidAI/LFM2.5-1.2B-Instruct` (1.2B)
- `LiquidAI/LFM2.5-1.2B-Thinking` (1.2B, reasoning)
- `LiquidAI/LFM2.5-1.2B-JP` / `LiquidAI/LFM2.5-1.2B-JP-202606` (Japanese)
- `LiquidAI/LFM2.5-1.2B-Base` (pretrained base)
MoE:
- `LiquidAI/LFM2.5-8B-A1B` (8B total / ~1B active)
Vision-Language:
- `LiquidAI/LFM2.5-VL-450M`, `LiquidAI/LFM2.5-VL-1.6B`
See the [LFM2.5 usage guide](../../LiquidAI/LFM2.5) for the full family.
## Prerequisites
- **Hardware**: 1× GPU (~1 GB VRAM for weights; any modern GPU works). Verified on H100.
- **vLLM**: ≥ 0.23.0 — the LFM2 architecture ships in the 0.23.0 stable release.
### pip (NVIDIA CUDA)
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```
## Deployment Configurations
### Quick Start (Single GPU, BF16)
```bash
vllm serve LiquidAI/LFM2.5-230M
```
### Full-Featured Server Launch
Enables tool calling:
```bash
vllm serve LiquidAI/LFM2.5-230M \
--enable-auto-tool-choice \
--tool-call-parser lfm2 \
--host 0.0.0.0 --port 8000
```
### Docker (NVIDIA)
```bash
docker run -itd --name lfm2.5-230m \
--ipc=host --network host --shm-size 16G --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:latest \
--model LiquidAI/LFM2.5-230M \
--host 0.0.0.0 --port 8000
```
## Client Usage
### Text Generation
The model card recommends temperature `0.1`, `top_k 50`, `repetition_penalty 1.05` (`top_k` and
`repetition_penalty` ride in `extra_body`).
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="LiquidAI/LFM2.5-230M",
messages=[{"role": "user", "content": "Give me three uses for baking soda."}],
temperature=0.1,
extra_body={"top_k": 50, "repetition_penalty": 1.05},
)
print(response.choices[0].message.content)
```
### Tool Calling
Launch with `--enable-auto-tool-choice --tool-call-parser lfm2`, then pass `tools=[…]`; the
`lfm2` parser converts the model's Pythonic call into a standard `tool_calls` array.
```python
response = client.chat.completions.create(
model="LiquidAI/LFM2.5-230M",
messages=[{"role": "user", "content": "What's the weather in Paris?"}],
tools=[{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get current weather for a location",
"parameters": {"type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"]},
},
}],
temperature=0.1,
extra_body={"top_k": 50, "repetition_penalty": 1.05},
)
print(response.choices[0].message.tool_calls)
```
### Structured Outputs
vLLM's guided decoding constrains output to a JSON schema via `response_format`. The model does
not see the schema itself — put semantic instructions in the system prompt. Data extraction is a
primary use case for this model, so structured outputs pair naturally with it.
## Configuration Tips
- At 230M the model fits any GPU; raise `--max-num-seqs` to push batch throughput.
- Set `--max-model-len` to match your workload (up to 32K).
- `--gpu-memory-utilization 0.90–0.95` maximizes KV cache capacity.
- Sampling presets are per-request client defaults — don't bake them into `vllm serve`.
## References
- [Model card](https://huggingface.co/LiquidAI/LFM2.5-230M)
- [Blog post](https://www.liquid.ai/blog/lfm2-5-230m)
- [LFM2.5 usage guide](../../LiquidAI/LFM2.5)
- [Liquid AI](https://www.liquid.ai/)
- [vLLM supported models](https://docs.vllm.ai/en/latest/models/supported_models.html)
recipes/models/LiquidAI/LFM2.5-350M.yaml
+181
View File
@@ -0,0 +1,181 @@
meta:
title: "LFM2.5 350M"
slug: "lfm2.5-350m"
provider: "LiquidAI"
description: "Liquid AI's smallest LFM2.5 chat model (350M) on the LFM2 hybrid conv+attention backbone — tool calling and 32K context, light enough for edge GPUs."
date_added: 2026-06-25
date_updated: 2026-06-24
tasks:
- text
performance_headline: "350M hybrid instruct model with tool calling — fits any GPU, ideal for edge / on-device serving"
related_recipes:
- LiquidAI/LFM2.5-1.2B-Instruct
- LiquidAI/LFM2.5-8B-A1B
hardware:
h100: verified
h200: verified
b200: verified
model:
model_id: "LiquidAI/LFM2.5-350M"
min_vllm_version: "0.23.0"
architecture: dense
parameter_count: "350M"
active_parameters: "350M"
context_length: 32768
base_args: []
base_env: {}
features:
tool_calling:
description: "Surface LFM2.5's Pythonic tool calls (wrapped in <|tool_call_start|>…<|tool_call_end|>) as OpenAI tool_calls via vLLM's native lfm2 parser"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "lfm2"
variants:
default:
precision: bf16
vram_minimum_gb: 1
description: "Full BF16 — fits essentially any GPU (~0.7 GB of weights)"
compatible_strategies:
- single_node_tp
hardware_overrides:
# Blackwell (B200) only: route RMSNorm to the bundled oink kernels (measured ~+2.6% on B200,
# output identical). Inert (falls back to native) on non-Blackwell GPUs.
blackwell:
extra_env:
VLLM_USE_OINK_OPS: "1"
strategy_overrides:
single_node_tp:
tp: 1
guide: |
## Overview
[LFM2.5-350M](https://huggingface.co/LiquidAI/LFM2.5-350M) is the smallest chat model in
[Liquid AI's](https://www.liquid.ai/) LFM2.5 family, built on the **LFM2 hybrid backbone**:
short-range gated convolution blocks interleaved with grouped-query attention. It shares that
backbone with its larger siblings, which keeps memory and latency low enough for edge and
on-device deployment while still supporting tool calling and a 32K context window — all
through vLLM's OpenAI-compatible API.
### Key Features
- **Hybrid backbone**: Gated short convolutions interleaved with grouped-query attention — a smaller KV cache and lower decode latency than a same-size full-attention transformer.
- **32K context**: 32,768-token context window.
- **Tool calling**: Pythonic tool calls (`<|tool_call_start|>…<|tool_call_end|>`) surfaced as OpenAI `tool_calls` by vLLM's native `lfm2` parser.
- **Native vLLM support**: Served via the `Lfm2ForCausalLM` architecture.
- **Edge-ready**: ~0.7 GB of BF16 weights — runs on commodity and on-device GPUs.
### Supported Variants
Dense:
- `LiquidAI/LFM2.5-230M` (230M)
- `LiquidAI/LFM2.5-350M` (350M)
- `LiquidAI/LFM2.5-1.2B-Instruct` (1.2B)
- `LiquidAI/LFM2.5-1.2B-Thinking` (1.2B, reasoning)
- `LiquidAI/LFM2.5-1.2B-JP` / `LiquidAI/LFM2.5-1.2B-JP-202606` (Japanese)
- `LiquidAI/LFM2.5-1.2B-Base` (pretrained base)
MoE:
- `LiquidAI/LFM2.5-8B-A1B` (8B total / ~1B active)
Vision-Language:
- `LiquidAI/LFM2.5-VL-450M`, `LiquidAI/LFM2.5-VL-1.6B`
See the [LFM2.5 usage guide](../../LiquidAI/LFM2.5) for the full family.
## Prerequisites
- **Hardware**: 1× GPU (~1 GB VRAM for weights; any modern GPU works).
- **vLLM**: ≥ 0.23.0.
### pip (NVIDIA CUDA)
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```
## Deployment Configurations
### Quick Start (Single GPU, BF16)
```bash
vllm serve LiquidAI/LFM2.5-350M
```
### Full-Featured Server Launch
Enables tool calling:
```bash
vllm serve LiquidAI/LFM2.5-350M \
--enable-auto-tool-choice \
--tool-call-parser lfm2 \
--host 0.0.0.0 --port 8000
```
### Docker (NVIDIA)
```bash
docker run -itd --name lfm2.5-350m \
--ipc=host --network host --shm-size 16G --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:latest \
--model LiquidAI/LFM2.5-350M \
--host 0.0.0.0 --port 8000
```
## Client Usage
### Text Generation
The model card recommends temperature `0.1`, `top_k 50`, `repetition_penalty 1.05` (`top_k` and
`repetition_penalty` ride in `extra_body`).
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="LiquidAI/LFM2.5-350M",
messages=[{"role": "user", "content": "Give me three uses for baking soda."}],
temperature=0.1,
extra_body={"top_k": 50, "repetition_penalty": 1.05},
)
print(response.choices[0].message.content)
```
### Tool Calling
Launch with `--enable-auto-tool-choice --tool-call-parser lfm2`, then pass `tools=[…]`; the
`lfm2` parser converts the model's Pythonic call into a standard `tool_calls` array.
```python
response = client.chat.completions.create(
model="LiquidAI/LFM2.5-350M",
messages=[{"role": "user", "content": "What's the weather in Paris?"}],
tools=[{
"type": "function",
"function": {
"name": "get_weather",
"description": "Get current weather for a location",
"parameters": {"type": "object", "properties": {"location": {"type": "string"}}, "required": ["location"]},
},
}],
temperature=0.1,
extra_body={"top_k": 50, "repetition_penalty": 1.05},
)
print(response.choices[0].message.tool_calls)
```
### Structured Outputs
vLLM's guided decoding constrains output to a JSON schema via `response_format`. The model does
not see the schema itself — put semantic instructions in the system prompt.
## Configuration Tips
- At 350M the model fits any GPU; raise `--max-num-seqs` to push batch throughput.
- Set `--max-model-len` to match your workload (up to 32K).
- `--gpu-memory-utilization 0.90–0.95` maximizes KV cache capacity.
- Sampling presets are per-request client defaults — don't bake them into `vllm serve`.
## References
- [Model card](https://huggingface.co/LiquidAI/LFM2.5-350M)
- [LFM2.5 usage guide](../../LiquidAI/LFM2.5)
- [Liquid AI](https://www.liquid.ai/)
- [vLLM supported models](https://docs.vllm.ai/en/latest/models/supported_models.html)
recipes/models/LiquidAI/LFM2.5-8B-A1B.yaml
+189
View File
@@ -0,0 +1,189 @@
meta:
title: "LFM2.5 8B A1B"
slug: "lfm2.5-8b-a1b"
provider: "LiquidAI"
description: "Liquid AI's 8B mixture-of-experts model (~1B active) on the LFM2 hybrid conv+attention backbone, with <think> reasoning and tool calling at ~1B decode cost."
date_added: 2026-06-25
date_updated: 2026-06-24
tasks:
- text
performance_headline: "8B-total / ~1B-active hybrid MoE with reasoning and tool calling — 8B quality at ~1B decode cost on a single GPU"
related_recipes:
- LiquidAI/LFM2.5-1.2B-Instruct
- LiquidAI/LFM2.5-1.2B-Thinking
hardware:
h100: verified
h200: verified
b200: verified
model:
model_id: "LiquidAI/LFM2.5-8B-A1B"
min_vllm_version: "0.23.0"
architecture: moe
parameter_count: "8B"
active_parameters: "1B"
context_length: 128000
base_args: []
base_env: {}
features:
reasoning:
description: "Split the model's <think>…</think> chain-of-thought into a separate reasoning field via vLLM's qwen3 parser"
args:
- "--reasoning-parser"
- "qwen3"
tool_calling:
description: "Surface LFM2.5's Pythonic tool calls (wrapped in <|tool_call_start|>…<|tool_call_end|>) as OpenAI tool_calls via vLLM's native lfm2 parser"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "lfm2"
variants:
default:
precision: bf16
vram_minimum_gb: 21
description: "Full BF16 — all 8B of experts live in VRAM; single 24 GB+ NVIDIA GPU"
compatible_strategies:
- single_node_tp
hardware_overrides:
# Blackwell (B200) only: route RMSNorm to the bundled oink kernels (measured ~+2.6% on B200,
# output identical). Inert (falls back to native) on non-Blackwell GPUs.
blackwell:
extra_env:
VLLM_USE_OINK_OPS: "1"
strategy_overrides:
single_node_tp:
tp: 1
guide: |
## Overview
[LFM2.5-8B-A1B](https://huggingface.co/LiquidAI/LFM2.5-8B-A1B) is [Liquid AI's](https://www.liquid.ai/)
mixture-of-experts model: ~8B total parameters with only ~1B activated per token. It pairs the
**LFM2 hybrid backbone** (short-range gated convolution blocks interleaved with grouped-query
attention) with sparse MoE feed-forward layers, so it reaches 8B-class quality at roughly 1B
decode cost. It supports explicit `<think>` reasoning and Pythonic tool calling through vLLM's
OpenAI-compatible API.
### Key Features
- **Hybrid + MoE**: Gated short convolutions + grouped-query attention with sparse expert FFNs — ~1B active parameters per token out of ~8B total.
- **`<think>` reasoning**: Emits an explicit `<think>…</think>` chain-of-thought on non-trivial problems; vLLM's `qwen3` parser splits it into `reasoning`.
- **Tool calling**: Pythonic tool calls surfaced as OpenAI `tool_calls` by vLLM's native `lfm2` parser.
- **128K context**: Long-context support (`max_position_embeddings = 128000`).
- **Native vLLM support**: Served via the `Lfm2MoeForCausalLM` architecture.
### Supported Variants
Dense:
- `LiquidAI/LFM2.5-230M` (230M)
- `LiquidAI/LFM2.5-350M` (350M)
- `LiquidAI/LFM2.5-1.2B-Instruct` (1.2B)
- `LiquidAI/LFM2.5-1.2B-Thinking` (1.2B, reasoning)
- `LiquidAI/LFM2.5-1.2B-JP` / `LiquidAI/LFM2.5-1.2B-JP-202606` (Japanese)
- `LiquidAI/LFM2.5-1.2B-Base` (pretrained base)
MoE:
- `LiquidAI/LFM2.5-8B-A1B` (8B total / ~1B active)
Vision-Language:
- `LiquidAI/LFM2.5-VL-450M`, `LiquidAI/LFM2.5-VL-1.6B`
See the [LFM2.5 usage guide](../../LiquidAI/LFM2.5) for the full family.
> **Sizing**: MoE keeps every expert resident in VRAM, so size the GPU for the full ~8B of
> weights (≈17 GB BF16 + KV cache) even though only ~1B is active per token.
## Prerequisites
- **Hardware**: 1× GPU with ≥24 GB VRAM.
- **vLLM**: ≥ 0.23.0.
### pip (NVIDIA CUDA)
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```
## Deployment Configurations
### Quick Start (Single GPU, BF16)
```bash
vllm serve LiquidAI/LFM2.5-8B-A1B
```
### Full-Featured Server Launch
Reasoning **and** tool calling:
```bash
vllm serve LiquidAI/LFM2.5-8B-A1B \
--reasoning-parser qwen3 \
--enable-auto-tool-choice \
--tool-call-parser lfm2 \
--host 0.0.0.0 --port 8000
```
### Multi-GPU
This model fits on a single 24 GB+ GPU, so tensor parallelism gives no meaningful speedup on
basically any GPU — there's nothing to split that helps. You can still try it on a multi-GPU
node if you want, but expect no throughput gain:
```bash
vllm serve LiquidAI/LFM2.5-8B-A1B --tensor-parallel-size 2
```
### Docker (NVIDIA)
```bash
docker run -itd --name lfm2.5-8b-a1b \
--ipc=host --network host --shm-size 16G --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:latest \
--model LiquidAI/LFM2.5-8B-A1B \
--reasoning-parser qwen3 \
--host 0.0.0.0 --port 8000
```
## Client Usage
### Reasoning Mode
The model card recommends temperature `0.2`, `top_k 80`, `repetition_penalty 1.05`. Leave
`max_tokens` unset so the `<think>` block and answer aren't truncated mid-stream.
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="LiquidAI/LFM2.5-8B-A1B",
messages=[{"role": "user", "content": "A snail climbs 3 ft up a 20 ft well each day and slides 2 ft back each night. How many days to reach the top?"}],
temperature=0.2,
extra_body={"top_k": 80, "repetition_penalty": 1.05},
)
msg = response.choices[0].message
print("reasoning:", msg.reasoning)
print("answer:", msg.content)
```
> **Note**: the model opens the `<think>` channel for non-trivial problems; a simple prompt may
> be answered directly, in which case `reasoning` is empty. That is expected behavior —
> the `qwen3` parser extracts the block whenever it is present.
### Tool Calling
Add `--enable-auto-tool-choice --tool-call-parser lfm2` at launch, then pass `tools=[…]`; the
`lfm2` parser converts the model's Pythonic call into a standard `tool_calls` array.
### Structured Outputs
vLLM's guided decoding constrains output to a JSON schema via `response_format`. The model does
not see the schema itself — put semantic instructions in the system prompt.
## Configuration Tips
- Size the GPU for the full ~8B of weights — all experts are resident even though ~1B is active.
- It fits on one GPU; tensor parallelism gives no real speedup on basically any GPU (you can still try `--tensor-parallel-size N`).
- Set `--max-model-len` to match your workload (up to 128K).
- `--gpu-memory-utilization 0.90–0.95` maximizes KV cache capacity.
- Sampling presets are per-request client defaults — don't bake them into `vllm serve`.
## References
- [Model card](https://huggingface.co/LiquidAI/LFM2.5-8B-A1B)
- [LFM2.5 usage guide](../../LiquidAI/LFM2.5)
- [Liquid AI](https://www.liquid.ai/)
- [vLLM supported models](https://docs.vllm.ai/en/latest/models/supported_models.html)
recipes/models/LiquidAI/LFM2.5-VL-1.6B.yaml
+155
View File
@@ -0,0 +1,155 @@
meta:
title: "LFM2.5 VL 1.6B"
slug: "lfm2.5-vl-1.6b"
provider: "LiquidAI"
description: "Liquid AI's 1.6B vision-language model — LFM2 hybrid LM backbone plus a SigLIP2 vision tower for image+text chat on a single small GPU."
date_added: 2026-06-25
date_updated: 2026-06-24
tasks:
- multimodal
performance_headline: "1.6B vision-language model (hybrid LM + SigLIP2 vision) — image understanding on a single small GPU"
related_recipes:
- LiquidAI/LFM2.5-VL-450M
- LiquidAI/LFM2.5-1.2B-Instruct
hardware:
h100: verified
h200: verified
b200: verified
model:
model_id: "LiquidAI/LFM2.5-VL-1.6B"
min_vllm_version: "0.23.0"
architecture: dense
parameter_count: "1.6B"
active_parameters: "1.6B"
context_length: 32768
base_args: []
base_env: {}
# TODO(vincentzed): the Lfm2VL encoder CUDA graph (vllm-project/vllm#44930, ~10-20% lower e2e
# latency at low batch) is NOT in 0.23.0 — add the encoder-CG compilation flag here once it
# ships in a stable release (>0.23.0).
variants:
default:
precision: bf16
vram_minimum_gb: 4
description: "Full BF16 (LM + vision encoder) — single 8 GB+ NVIDIA GPU"
compatible_strategies:
- single_node_tp
hardware_overrides:
# Blackwell (B200) only: route RMSNorm to the bundled oink kernels (measured ~+2.6% on B200,
# output identical). Inert (falls back to native) on non-Blackwell GPUs.
blackwell:
extra_env:
VLLM_USE_OINK_OPS: "1"
strategy_overrides:
single_node_tp:
tp: 1
guide: |
## Overview
[LFM2.5-VL-1.6B](https://huggingface.co/LiquidAI/LFM2.5-VL-1.6B) is the larger vision-language
model in [Liquid AI's](https://www.liquid.ai/) LFM2.5-VL family. It pairs the **LFM2 hybrid
language backbone** (short-range gated convolution blocks interleaved with grouped-query
attention) with a SigLIP2 vision encoder for image understanding, while remaining small enough
to serve on a single commodity GPU through vLLM's OpenAI-compatible API.
### Key Features
- **Vision-language**: SigLIP2 vision encoder on top of the LFM2 hybrid language backbone — single- and multi-image prompts.
- **Hybrid LM backbone**: Gated short convolutions interleaved with grouped-query attention — a smaller KV cache and lower decode latency than a same-size full-attention transformer.
- **32K context**: 32,768-token context window.
- **Native vLLM support**: Served via the `Lfm2VlForConditionalGeneration` architecture.
### Supported Variants
Vision-Language:
- `LiquidAI/LFM2.5-VL-450M` (450M)
- `LiquidAI/LFM2.5-VL-1.6B` (1.6B)
Text (same LFM2 family):
- Dense: `LiquidAI/LFM2.5-230M`, `LiquidAI/LFM2.5-350M`, `LiquidAI/LFM2.5-1.2B-Instruct`, `LiquidAI/LFM2.5-1.2B-Thinking`, `LiquidAI/LFM2.5-1.2B-JP`, `LiquidAI/LFM2.5-1.2B-JP-202606`, `LiquidAI/LFM2.5-1.2B-Base`
- MoE: `LiquidAI/LFM2.5-8B-A1B`
See the [LFM2.5 usage guide](../../LiquidAI/LFM2.5) for the full family.
## Prerequisites
- **Hardware**: 1× GPU with ≥8 GB VRAM.
- **vLLM**: ≥ 0.23.0.
### pip (NVIDIA CUDA)
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```
## Deployment Configurations
### Quick Start (Single GPU, BF16)
```bash
vllm serve LiquidAI/LFM2.5-VL-1.6B
```
### Multiple Images per Request
```bash
vllm serve LiquidAI/LFM2.5-VL-1.6B \
--limit-mm-per-prompt '{"image": 4}' \
--host 0.0.0.0 --port 8000
```
### Docker (NVIDIA)
```bash
docker run -itd --name lfm2.5-vl-1.6b \
--ipc=host --network host --shm-size 16G --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:latest \
--model LiquidAI/LFM2.5-VL-1.6B \
--host 0.0.0.0 --port 8000
```
## Client Usage
### Image Understanding
Send an image + text turn via the OpenAI chat API. The card recommends temperature `0.1`,
`min_p 0.15`, `repetition_penalty 1.05` (`min_p` and `repetition_penalty` ride in `extra_body`).
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="LiquidAI/LFM2.5-VL-1.6B",
messages=[{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"}},
{"type": "text", "text": "What is in this image?"},
],
}],
temperature=0.1,
extra_body={"min_p": 0.15, "repetition_penalty": 1.05},
)
print(response.choices[0].message.content)
```
### Multiple Images
Launch with `--limit-mm-per-prompt '{"image": N}'`, then include several `image_url` blocks in
one message to compare or reason across images.
## Performance
Tensor parallelism gives no speedup — the model fits on a single GPU.
## Configuration Tips
- `--limit-mm-per-prompt '{"image": N}'` caps images per request (default 1).
- Set `--max-model-len` to match your workload (up to 32K).
- `--gpu-memory-utilization 0.90–0.95` maximizes KV cache capacity.
- Sampling presets are per-request client defaults — don't bake them into `vllm serve`.
## References
- [Model card](https://huggingface.co/LiquidAI/LFM2.5-VL-1.6B)
- [LFM2.5 usage guide](../../LiquidAI/LFM2.5)
- [Liquid AI](https://www.liquid.ai/)
- [vLLM supported models](https://docs.vllm.ai/en/latest/models/supported_models.html)
recipes/models/LiquidAI/LFM2.5-VL-450M.yaml
+149
View File
@@ -0,0 +1,149 @@
meta:
title: "LFM2.5 VL 450M"
slug: "lfm2.5-vl-450m"
provider: "LiquidAI"
description: "Liquid AI's smallest vision-language model (450M) — LFM2 hybrid LM backbone plus a SigLIP2 vision tower for image+text chat, light enough for edge GPUs."
date_added: 2026-06-25
date_updated: 2026-06-24
tasks:
- multimodal
performance_headline: "450M vision-language model (hybrid LM + SigLIP2 vision) — image understanding light enough for edge / on-device serving"
related_recipes:
- LiquidAI/LFM2.5-VL-1.6B
- LiquidAI/LFM2.5-350M
hardware:
h100: verified
h200: verified
b200: verified
model:
model_id: "LiquidAI/LFM2.5-VL-450M"
min_vllm_version: "0.23.0"
architecture: dense
parameter_count: "450M"
active_parameters: "450M"
context_length: 32768
base_args: []
base_env: {}
variants:
default:
precision: bf16
vram_minimum_gb: 2
description: "Full BF16 (LM + vision encoder) — fits essentially any GPU"
compatible_strategies:
- single_node_tp
hardware_overrides:
# Blackwell (B200) only: route RMSNorm to the bundled oink kernels (measured ~+2.6% on B200,
# output identical). Inert (falls back to native) on non-Blackwell GPUs.
blackwell:
extra_env:
VLLM_USE_OINK_OPS: "1"
strategy_overrides:
single_node_tp:
tp: 1
guide: |
## Overview
[LFM2.5-VL-450M](https://huggingface.co/LiquidAI/LFM2.5-VL-450M) is the smallest vision-language
model in [Liquid AI's](https://www.liquid.ai/) LFM2.5-VL family — a SigLIP2 vision encoder on
top of the **LFM2 hybrid language backbone** (short-range gated convolution blocks interleaved
with grouped-query attention). It is light enough for edge and on-device image understanding,
served through vLLM's OpenAI-compatible API.
### Key Features
- **Vision-language**: SigLIP2 vision encoder on top of the LFM2 hybrid language backbone — single- and multi-image prompts.
- **Edge-ready**: ~1 GB of BF16 weights — runs on commodity and on-device GPUs.
- **Hybrid LM backbone**: Gated short convolutions interleaved with grouped-query attention — a smaller KV cache and lower decode latency than a same-size full-attention transformer.
- **32K context**: 32,768-token context window.
- **Native vLLM support**: Served via the `Lfm2VlForConditionalGeneration` architecture.
### Supported Variants
Vision-Language:
- `LiquidAI/LFM2.5-VL-450M` (450M)
- `LiquidAI/LFM2.5-VL-1.6B` (1.6B)
Text (same LFM2 family):
- Dense: `LiquidAI/LFM2.5-230M`, `LiquidAI/LFM2.5-350M`, `LiquidAI/LFM2.5-1.2B-Instruct`, `LiquidAI/LFM2.5-1.2B-Thinking`, `LiquidAI/LFM2.5-1.2B-JP`, `LiquidAI/LFM2.5-1.2B-JP-202606`, `LiquidAI/LFM2.5-1.2B-Base`
- MoE: `LiquidAI/LFM2.5-8B-A1B`
See the [LFM2.5 usage guide](../../LiquidAI/LFM2.5) for the full family.
## Prerequisites
- **Hardware**: 1× GPU (~1–2 GB VRAM for weights; any modern GPU works).
- **vLLM**: ≥ 0.23.0.
### pip (NVIDIA CUDA)
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```
## Deployment Configurations
### Quick Start (Single GPU, BF16)
```bash
vllm serve LiquidAI/LFM2.5-VL-450M
```
### Multiple Images per Request
```bash
vllm serve LiquidAI/LFM2.5-VL-450M \
--limit-mm-per-prompt '{"image": 4}' \
--host 0.0.0.0 --port 8000
```
### Docker (NVIDIA)
```bash
docker run -itd --name lfm2.5-vl-450m \
--ipc=host --network host --shm-size 16G --gpus all \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:latest \
--model LiquidAI/LFM2.5-VL-450M \
--host 0.0.0.0 --port 8000
```
## Client Usage
### Image Understanding
Send an image + text turn via the OpenAI chat API. The card recommends temperature `0.1`,
`min_p 0.15`, `repetition_penalty 1.05` (`min_p` and `repetition_penalty` ride in `extra_body`).
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="LiquidAI/LFM2.5-VL-450M",
messages=[{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"}},
{"type": "text", "text": "What is in this image?"},
],
}],
temperature=0.1,
extra_body={"min_p": 0.15, "repetition_penalty": 1.05},
)
print(response.choices[0].message.content)
```
### Multiple Images
Launch with `--limit-mm-per-prompt '{"image": N}'`, then include several `image_url` blocks in
one message to compare or reason across images.
## Configuration Tips
- At 450M the model fits any GPU; ideal for edge / on-device image understanding.
- `--limit-mm-per-prompt '{"image": N}'` caps images per request (default 1).
- Set `--max-model-len` to match your workload (up to 32K).
- Sampling presets are per-request client defaults — don't bake them into `vllm serve`.
## References
- [Model card](https://huggingface.co/LiquidAI/LFM2.5-VL-450M)
- [LFM2.5 usage guide](../../LiquidAI/LFM2.5)
- [Liquid AI](https://www.liquid.ai/)
- [vLLM supported models](https://docs.vllm.ai/en/latest/models/supported_models.html)
recipes/models/MiniMaxAI/MiniMax-M2.1.yaml
+179
View File
@@ -0,0 +1,179 @@
meta:
title: "MiniMax-M2.1"
slug: "minimax-m2.1"
provider: "MiniMax"
description: "MiniMax M2.1 MoE language model (230B total / 10B active) for coding, agent toolchains, and long-context reasoning — native FP8 checkpoint"
date_added: 2025-12-20
date_updated: 2026-04-17
tasks:
- text
performance_headline: "Updated M2 series MoE with strong SWE-Bench and Terminal-Bench performance, 196K context"
related_recipes: []
hardware:
h100: verified
mi300x: verified
mi325x: verified
mi355x: verified
model:
model_id: "MiniMaxAI/MiniMax-M2.1"
min_vllm_version: "0.11.1"
architecture: moe
parameter_count: "230B"
active_parameters: "10B"
context_length: 196608
base_args:
- "--trust-remote-code"
base_env: {}
features:
tool_calling:
description: "MiniMax M2 tool call parser with automatic tool choice"
args:
- "--tool-call-parser"
- "minimax_m2"
- "--enable-auto-tool-choice"
reasoning:
description: "MiniMax M2 reasoning parser for chain-of-thought extraction"
args:
- "--reasoning-parser"
- "minimax_m2"
opt_in_features: []
variants:
default:
precision: fp8
vram_minimum_gb: 276
description: "Native FP8 checkpoint — 4x H200/H20/H100 or 4x A100/A800 for weights, plus KV cache headroom"
compatible_strategies:
- single_node_tp
- single_node_tep
- single_node_dep
- multi_node_tp
- multi_node_tp_pp
- multi_node_tep
- multi_node_dep
- pd_cluster
kv_offload_support:
offloading_cpu: verified
offloading_fs: verified
hardware_overrides:
amd:
extra_args:
- "--tensor-parallel-size"
- "2"
- "--attention-backend"
- "ROCM_AITER_FA"
extra_env:
VLLM_ROCM_USE_AITER: "1"
VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT: "1"
strategy_overrides:
single_node_tp:
# Pure TP=8 fails on the FP8 block-quantized gate_up weights:
# intermediate_size / 8 = 192 is not divisible by block_n = 128.
# Pin TP=4 on 8-GPU nodes; for full-node utilization use
# single_node_tep (TP+EP) or single_node_dep (DP+EP).
tp: 4
guide: |
## Overview
[MiniMax-M2.1](https://huggingface.co/MiniMaxAI/MiniMax-M2.1) is part of the MiniMax
M2 series of advanced MoE language models. It retains the M2 architecture (10B active,
230B total) with improvements over the original M2 release. Supports 196K context per sequence.
## Prerequisites
- **OS:** Linux
- **Python:** 3.10 - 3.13
- **NVIDIA:** compute capability >= 7.0; ~220 GB for weights + 240 GB per 1M context tokens
- **AMD:** MI300X / MI325X / MI350X / MI355X with ROCm 7.0+
### Install vLLM (NVIDIA)
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```
### Docker (dedicated M2-series image)
```bash
docker run --gpus all \
-p 8000:8000 \
--ipc=host \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:minimax27 MiniMaxAI/MiniMax-M2.1 \
--tensor-parallel-size 4 \
--tool-call-parser minimax_m2 \
--reasoning-parser minimax_m2 \
--enable-auto-tool-choice \
--trust-remote-code
```
## Launching the Server
### NVIDIA — TP4
```bash
vllm serve MiniMaxAI/MiniMax-M2.1 \
--tensor-parallel-size 4 \
--tool-call-parser minimax_m2 \
--reasoning-parser minimax_m2 \
--enable-auto-tool-choice \
--trust-remote-code
```
Pure TP8 is not supported. For >4 GPUs use DP+EP or TP+EP.
### TP4+EP (recommended for H100)
```bash
vllm serve MiniMaxAI/MiniMax-M2.1 \
--tensor-parallel-size 4 \
--enable-expert-parallel \
--tool-call-parser minimax_m2 \
--reasoning-parser minimax_m2 \
--enable-auto-tool-choice
```
### AMD ROCm — TP2
```bash
VLLM_ROCM_USE_AITER=1 vllm serve MiniMaxAI/MiniMax-M2.1 \
--tensor-parallel-size 2 \
--tool-call-parser minimax_m2 \
--reasoning-parser minimax_m2 \
--enable-auto-tool-choice \
--trust-remote-code
```
## Benchmarking
```bash
vllm bench serve \
--backend vllm \
--model MiniMaxAI/MiniMax-M2.1 \
--endpoint /v1/completions \
--dataset-name random \
--random-input 2048 \
--random-output 1024 \
--max-concurrency 10 \
--num-prompt 100
```
## Troubleshooting
- See [MiniMax-M2](./MiniMax-M2.yaml) for shared troubleshooting notes
(latest vLLM fusion behavior, nightly vs stable, DeepGEMM, AITER).
## References
- [Model card](https://huggingface.co/MiniMaxAI/MiniMax-M2.1)
- [MiniMax](https://www.minimax.io/)
recipes/models/MiniMaxAI/MiniMax-M2.5.yaml
+249
View File
@@ -0,0 +1,249 @@
meta:
title: "MiniMax-M2.5"
slug: "minimax-m2.5"
provider: "MiniMax"
description: "MiniMax M2.5 MoE language model (230B total / 10B active) for coding, agent toolchains, and long-context reasoning — native FP8 checkpoint"
date_added: 2026-02-13
date_updated: 2026-09-14
tasks:
- text
performance_headline: "Refreshed M2 series MoE with strong SWE-Bench and Terminal-Bench performance, 196K context"
related_recipes: []
hardware:
h100: verified
h200: verified
mi300x: verified
mi325x: verified
mi355x: verified
model:
model_id: "MiniMaxAI/MiniMax-M2.5"
min_vllm_version: "0.20.2"
default_frontend: rust
docker_image: "vllm/vllm-openai:v0.28.0"
architecture: moe
parameter_count: "230B"
active_parameters: "10B"
context_length: 196608
base_args:
- "--trust-remote-code"
base_env: {}
features:
tool_calling:
description: "MiniMax M2 tool call parser with automatic tool choice"
args:
- "--tool-call-parser"
- "minimax_m2"
- "--enable-auto-tool-choice"
reasoning:
description: "MiniMax M2 reasoning parser for chain-of-thought extraction"
args:
- "--reasoning-parser"
- "minimax_m2"
opt_in_features: []
variants:
default:
precision: fp8
vram_minimum_gb: 276
description: "Native FP8 checkpoint — 4x H200/H20/H100 or 4x A100/A800 for weights, plus KV cache headroom"
nvfp4:
model_id: "nvidia/MiniMax-M2.5-NVFP4"
precision: nvfp4
vram_minimum_gb: 138
description: "NVIDIA NVFP4 quantized weights for Blackwell GPUs"
extra_args:
- "--kv-cache-dtype"
- "fp8"
redhat_nvfp4:
model_id: "RedHatAI/MiniMax-M2.5-NVFP4"
precision: nvfp4
label: "NVFP4 (RedHat)"
vram_minimum_gb: 146
description: "RedHatAI NVFP4 quantization"
compatible_strategies:
- single_node_tp
- single_node_tep
- single_node_dep
- multi_node_tp
- multi_node_tp_pp
- multi_node_tep
- multi_node_dep
- pd_cluster
kv_offload_support:
offloading_cpu: verified
offloading_fs: verified
hardware_overrides:
nvidia:
extra_args:
- "--compilation-config"
- '{"cudagraph_mode":"PIECEWISE"}'
hopper:
extra_args:
- "--compilation-config"
- '{"cudagraph_mode":"PIECEWISE"}'
- "--max-num-seqs"
- "512"
- "--max-num-batched-tokens"
- "32768"
- "--kv-cache-dtype"
- "fp8"
- "--moe-backend"
- "triton"
- "--attention-backend"
- "FLASHINFER"
- "--enable-flashinfer-autotune"
extra_env:
PYTHONNOUSERSITE: "1"
VLLM_USE_DEEP_GEMM: "0"
VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER: "0"
VLLM_FLOAT32_MATMUL_PRECISION: "high"
amd:
extra_args:
- "--tensor-parallel-size"
- "2"
- "--attention-backend"
- "ROCM_AITER_FA"
extra_env:
VLLM_ROCM_USE_AITER: "1"
VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT: "1"
strategy_overrides:
single_node_tp:
# Pure TP=8 fails on the FP8 block-quantized gate_up weights:
# intermediate_size / 8 = 192 is not divisible by block_n = 128.
# Pin TP=4 on 8-GPU nodes; for full-node utilization use
# single_node_tep (TP+EP) or single_node_dep (DP+EP).
tp: 4
guide: |
## Overview
[MiniMax-M2.5](https://huggingface.co/MiniMaxAI/MiniMax-M2.5) is part of the MiniMax
M2 series of advanced MoE language models. It retains the M2 architecture (10B active,
230B total) and a 196K context per sequence.
## Prerequisites
- **OS:** Linux
- **Python:** 3.10 - 3.13
- **NVIDIA:** compute capability >= 7.0; ~220 GB for weights + 240 GB per 1M context tokens
- **AMD:** MI300X / MI325X / MI350X / MI355X with ROCm 7.0+
### Install vLLM (NVIDIA)
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```
### Docker (dedicated M2-series image)
```bash
docker run --gpus all \
-p 8000:8000 \
--ipc=host \
-v ~/.cache/huggingface:/root/.cache/huggingface \
-e PYTHONNOUSERSITE=1 \
-e VLLM_USE_DEEP_GEMM=0 \
-e VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER=0 \
-e VLLM_FLOAT32_MATMUL_PRECISION=high \
vllm/vllm-openai:v0.28.0 MiniMaxAI/MiniMax-M2.5 \
--tensor-parallel-size 4 \
--max-num-seqs 512 \
--max-num-batched-tokens 32768 \
--kv-cache-dtype fp8 \
--moe-backend triton \
--attention-backend FLASHINFER \
--enable-flashinfer-autotune \
--tool-call-parser minimax_m2 \
--reasoning-parser minimax_m2 \
--enable-auto-tool-choice \
--compilation-config '{"cudagraph_mode":"PIECEWISE"}' \
--trust-remote-code
```
## Launching the Server
### NVIDIA H200 — TP4
```bash
PYTHONNOUSERSITE=1 \
VLLM_USE_DEEP_GEMM=0 \
VLLM_BLOCKSCALE_FP8_GEMM_FLASHINFER=0 \
VLLM_FLOAT32_MATMUL_PRECISION=high \
vllm serve MiniMaxAI/MiniMax-M2.5 \
--tensor-parallel-size 4 \
--max-num-seqs 512 \
--max-num-batched-tokens 32768 \
--kv-cache-dtype fp8 \
--moe-backend triton \
--attention-backend FLASHINFER \
--enable-flashinfer-autotune \
--tool-call-parser minimax_m2 \
--reasoning-parser minimax_m2 \
--compilation-config '{"cudagraph_mode":"PIECEWISE"}' \
--enable-auto-tool-choice \
--trust-remote-code
```
For H200 FP8, choose the attention backend by sequence length for best performance:
shorter sequences (e.g. 1024) should keep the command above with `--attention-backend FLASHINFER`
and `--enable-flashinfer-autotune`, while longer input sequences (e.g. 8192)
should remove those flags to use the auto-selected FlashAttention backend.
Pure TP8 is not supported. For >4 GPUs use DP+EP or TP+EP.
### TP4+EP (recommended for H100)
```bash
vllm serve MiniMaxAI/MiniMax-M2.5 \
--tensor-parallel-size 4 \
--enable-expert-parallel \
--tool-call-parser minimax_m2 \
--reasoning-parser minimax_m2 \
--compilation-config '{"cudagraph_mode":"PIECEWISE"}' \
--enable-auto-tool-choice
```
### AMD ROCm — TP2
```bash
VLLM_ROCM_USE_AITER=1 vllm serve MiniMaxAI/MiniMax-M2.5 \
--tensor-parallel-size 2 \
--tool-call-parser minimax_m2 \
--reasoning-parser minimax_m2 \
--enable-auto-tool-choice \
--trust-remote-code
```
## Benchmarking
```bash
vllm bench serve \
--backend vllm \
--model MiniMaxAI/MiniMax-M2.5 \
--endpoint /v1/completions \
--dataset-name random \
--random-input 2048 \
--random-output 1024 \
--max-concurrency 10 \
--num-prompt 100
```
## Troubleshooting
- See [MiniMax-M2](./MiniMax-M2.yaml) for shared troubleshooting notes
(latest vLLM fusion behavior, nightly vs stable, DeepGEMM, AITER).
## References
- [Model card](https://huggingface.co/MiniMaxAI/MiniMax-M2.5)
- [MiniMax](https://www.minimax.io/)
recipes/models/MiniMaxAI/MiniMax-M2.7.yaml
+252
View File
@@ -0,0 +1,252 @@
meta:
title: "MiniMax-M2.7"
slug: "minimax-m2.7"
provider: "MiniMax"
description: "MiniMax M2.7 MoE language model (230B total / 10B active) — latest M2 release for coding, agent toolchains, and long-context reasoning with native FP8"
date_added: 2026-04-12
date_updated: 2026-09-14
tasks:
- text
performance_headline: "Latest M2 series release; verified accuracy on AIME25, GPQA-D, GSM8K; 196K context"
related_recipes: []
hardware:
h100: verified
h200: verified
mi300x: verified
mi325x: verified
mi355x: verified
dgx_station_gb300: verified
rtx_pro_6000_2x: verified
model:
model_id: "MiniMaxAI/MiniMax-M2.7"
min_vllm_version: "0.20.0"
default_frontend: rust
architecture: moe
parameter_count: "230B"
active_parameters: "10B"
context_length: 196608
base_args:
- "--trust-remote-code"
base_env: {}
features:
tool_calling:
description: "MiniMax M2 tool call parser with automatic tool choice"
args:
- "--tool-call-parser"
- "minimax_m2"
- "--enable-auto-tool-choice"
reasoning:
description: "MiniMax M2 reasoning parser for chain-of-thought extraction"
args:
- "--reasoning-parser"
- "minimax_m2"
opt_in_features: []
variants:
default:
precision: fp8
vram_minimum_gb: 276
description: "Native FP8 checkpoint — 4x H200/H20/H100 or 4x A100/A800 for weights, plus KV cache headroom"
nvfp4:
model_id: "nvidia/MiniMax-M2.7-NVFP4"
precision: nvfp4
vram_minimum_gb: 138
description: "NVIDIA NVFP4 quantized weights for Blackwell GPUs"
extra_args:
- "--kv-cache-dtype"
- "fp8"
hardware_overrides:
dgx_station_gb300:
extra_args:
- "--kv-cache-dtype"
- "fp8"
- "--gpu-memory-utilization"
- "0.92"
- "--load-format"
- "fastsafetensors"
- "--max-num-batched-tokens"
- "8192"
- "--enable-chunked-prefill"
- "--enable-prefix-caching"
rtx_pro_6000_2x:
extra_args:
- "--load-format"
- "fastsafetensors"
- "--enable-chunked-prefill"
- "--enable-prefix-caching"
- "--max-num-batched-tokens"
- "8192"
- "--gpu-memory-utilization"
- "0.95"
compatible_strategies:
- single_node_tp
- single_node_tep
- single_node_dep
- multi_node_tp
- multi_node_tp_pp
- multi_node_tep
- multi_node_dep
- pd_cluster
kv_offload_support:
offloading_cpu: verified
offloading_fs: verified
hardware_overrides:
amd:
extra_args:
- "--tensor-parallel-size"
- "2"
- "--attention-backend"
- "ROCM_AITER_FA"
extra_env:
VLLM_ROCM_USE_AITER: "1"
VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT: "1"
strategy_overrides:
single_node_tp:
# Pure TP=8 fails on the FP8 block-quantized gate_up weights:
# intermediate_size / 8 = 192 is not divisible by block_n = 128.
# Pin TP=4 on 8-GPU nodes; for full-node utilization use
# single_node_tep (TP+EP) or single_node_dep (DP+EP).
tp: 4
guide: |
## Overview
[MiniMax-M2.7](https://huggingface.co/MiniMaxAI/MiniMax-M2.7) is the latest release
in the MiniMax M2 series. Like earlier M2 variants, it ships with 10B active
parameters out of 230B total, and supports a 196K context per sequence. MiniMax has
verified M2.7 accuracy on AIME25, GPQA-D, and GSM8K at vLLM commit
`0f3ce4c74b1875791d6604e006b6e905fde9f698`.
## Prerequisites
- **OS:** Linux
- **Python:** 3.10 - 3.13
- **NVIDIA:** compute capability >= 7.0; ~220 GB for weights + 240 GB per 1M context tokens
- **AMD:** MI300X / MI325X / MI350X / MI355X with ROCm 7.0+
### Install vLLM (NVIDIA)
```bash
uv pip install -U vllm --extra-index-url https://wheels.vllm.ai/nightly
```
### Install vLLM (AMD ROCm)
```bash
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
```
### Docker (dedicated M2-series image)
```bash
docker run --gpus all \
-p 8000:8000 \
--ipc=host \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:minimax27 MiniMaxAI/MiniMax-M2.7 \
--tensor-parallel-size 4 \
--tool-call-parser minimax_m2 \
--reasoning-parser minimax_m2 \
--enable-auto-tool-choice \
--trust-remote-code
```
## Launching the Server
### NVIDIA — TP4 (4x H200/H20/H100 or 4x A100/A800)
```bash
vllm serve MiniMaxAI/MiniMax-M2.7 \
--tensor-parallel-size 4 \
--tool-call-parser minimax_m2 \
--reasoning-parser minimax_m2 \
--enable-auto-tool-choice \
--trust-remote-code
```
Pure TP8 is not supported. For >4 GPUs use DP+EP or TP+EP:
### DP8+EP
```bash
vllm serve MiniMaxAI/MiniMax-M2.7 \
--data-parallel-size 8 \
--enable-expert-parallel \
--tool-call-parser minimax_m2 \
--reasoning-parser minimax_m2 \
--enable-auto-tool-choice
```
### TP4+EP (recommended for H100)
```bash
vllm serve MiniMaxAI/MiniMax-M2.7 \
--tensor-parallel-size 4 \
--enable-expert-parallel \
--tool-call-parser minimax_m2 \
--reasoning-parser minimax_m2 \
--enable-auto-tool-choice
```
### TP8+EP
```bash
vllm serve MiniMaxAI/MiniMax-M2.7 \
--tensor-parallel-size 8 \
--enable-expert-parallel \
--tool-call-parser minimax_m2 \
--reasoning-parser minimax_m2 \
--enable-auto-tool-choice
```
### AMD ROCm — TP2
```bash
VLLM_ROCM_USE_AITER=1 vllm serve MiniMaxAI/MiniMax-M2.7 \
--tensor-parallel-size 2 \
--tool-call-parser minimax_m2 \
--reasoning-parser minimax_m2 \
--enable-auto-tool-choice \
--trust-remote-code
```
## Benchmarking
```bash
vllm bench serve \
--backend vllm \
--model MiniMaxAI/MiniMax-M2.7 \
--endpoint /v1/completions \
--dataset-name random \
--random-input 2048 \
--random-output 1024 \
--max-concurrency 10 \
--num-prompt 100
```
## Troubleshooting
- **MiniMax QK norm fusion:** Latest vLLM applies the fusion automatically at
runtime when the fused CUDA op is available; older vLLM builds used an
explicit `fuse_minimax_qk_norm` compile pass.
- **Corrupted output on stable release:** Upgrade to a nightly after commit
`cf3eacfe58fa9e745c2854782ada884a9f992cf7`.
- **Verified accuracy:** Use the pinned commit
`0f3ce4c74b1875791d6604e006b6e905fde9f698` if reproducing MiniMax's reported results.
- **DeepGEMM:** vLLM uses DeepGEMM by default; install via
[install_deepgemm.sh](https://github.com/vllm-project/vllm/blob/main/tools/install_deepgemm.sh)
if missing.
- **AITER first launch:** Initial AMD launch JIT-compiles optimized kernels; subsequent launches reuse cached kernels.
## References
- [Model card](https://huggingface.co/MiniMaxAI/MiniMax-M2.7)
- [MiniMax](https://www.minimax.io/)
- [vLLM PR #43410 (automatic MiniMax QK norm fusion)](https://github.com/vllm-project/vllm/pull/43410)
recipes/models/MiniMaxAI/MiniMax-M2.yaml
+244
View File
@@ -0,0 +1,244 @@
meta:
title: "MiniMax-M2"
slug: "minimax-m2"
provider: "MiniMax"
description: "MiniMax M2 MoE language model (230B total / 10B active) for coding, agent toolchains, and long-context reasoning — native FP8 checkpoint, with an NVFP4 variant for Blackwell"
date_added: 2025-10-26
date_updated: 2026-05-11
tasks:
- text
performance_headline: "Open-source MoE with strong SWE-Bench and Terminal-Bench performance, 196K context"
related_recipes: []
hardware:
h100: verified
h200: verified
mi300x: verified
mi325x: verified
mi355x: verified
model:
model_id: "MiniMaxAI/MiniMax-M2"
min_vllm_version: "0.11.1"
architecture: moe
parameter_count: "230B"
active_parameters: "10B"
context_length: 196608
base_args:
- "--trust-remote-code"
base_env: {}
dependencies:
- note: "Optional: DeepGEMM FP8 MoE kernels for throughput (skip on B200 — known FlashInfer FP8 MoE error)"
command: "export VLLM_USE_DEEP_GEMM=1"
optional: true
features:
tool_calling:
description: "MiniMax M2 tool call parser with automatic tool choice"
args:
- "--tool-call-parser"
- "minimax_m2"
- "--enable-auto-tool-choice"
reasoning:
description: "MiniMax M2 reasoning parser for chain-of-thought extraction"
args:
- "--reasoning-parser"
- "minimax_m2"
opt_in_features: []
variants:
default:
precision: fp8
vram_minimum_gb: 276
description: "Native FP8 checkpoint — 4x H200/H20/H100 or 4x A100/A800 for weights, plus KV cache headroom"
nvfp4:
model_id: "RedHatAI/MiniMax-M2-NVFP4"
precision: nvfp4
vram_minimum_gb: 138
description: "NVFP4 (4-bit) quantized weights — requires Blackwell (B200/B300); roughly half the VRAM of the FP8 checkpoint"
extra_args:
- "--kv-cache-dtype"
- "fp8"
compatible_strategies:
- single_node_tp
- single_node_tep
- single_node_dep
- multi_node_tp
- multi_node_tp_pp
- multi_node_tep
- multi_node_dep
- pd_cluster
kv_offload_support:
offloading_cpu: verified
offloading_fs: verified
hardware_overrides:
amd:
# First launch JIT-compiles AITER kernels (CK FP8 MoE, RMSNorm, activation);
# expect a slow cold start on MI300X/MI325X/MI350X/MI355X.
extra_args:
- "--tensor-parallel-size"
- "2"
- "--attention-backend"
- "ROCM_AITER_FA"
extra_env:
VLLM_ROCM_USE_AITER: "1"
VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT: "1"
strategy_overrides:
single_node_tp:
# Pure TP=8 fails on the FP8 block-quantized gate_up weights:
# intermediate_size / 8 = 192 is not divisible by block_n = 128.
# Pin TP=4 on 8-GPU nodes; for full-node utilization use
# single_node_tep (TP+EP) or single_node_dep (DP+EP).
tp: 4
guide: |
## Overview
[MiniMax-M2](https://huggingface.co/MiniMaxAI/MiniMax-M2) is an advanced MoE language
model from [MiniMax](https://www.minimax.io/). Highlights:
- Superior intelligence — #1 among open-source models globally on math, science, coding, tool use
- Advanced coding — multi-file edits, run-fix loops, test-validated repairs (SWE-Bench, Terminal-Bench)
- Agent performance — plans and executes complex toolchains across shell, browser, and code
- Efficient design — 10B active / 230B total for low latency and high throughput
- 196K context length per sequence
## Prerequisites
- **OS:** Linux
- **Python:** 3.10 - 3.13
- **NVIDIA:** compute capability >= 7.0; ~220 GB for weights + 240 GB per 1M context tokens
- **AMD:** MI300X / MI325X / MI350X / MI355X with ROCm 7.0+
### Install vLLM (NVIDIA, stable)
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```
### Install vLLM (NVIDIA, nightly)
If you hit corrupted output, upgrade to a nightly after commit
`cf3eacfe58fa9e745c2854782ada884a9f992cf7`:
```bash
uv pip install -U vllm --extra-index-url https://wheels.vllm.ai/nightly
```
### Install vLLM (AMD ROCm)
```bash
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
```
### Docker (NVIDIA, dedicated M2 image)
```bash
docker run --gpus all \
-p 8000:8000 \
--ipc=host \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai:minimax27 MiniMaxAI/MiniMax-M2 \
--tensor-parallel-size 4 \
--tool-call-parser minimax_m2 \
--reasoning-parser minimax_m2 \
--enable-auto-tool-choice \
--trust-remote-code
```
## Launching the Server
### NVIDIA — TP4 (4x H200/H20/H100 or 4x A100/A800)
```bash
vllm serve MiniMaxAI/MiniMax-M2 \
--tensor-parallel-size 4 \
--tool-call-parser minimax_m2 \
--reasoning-parser minimax_m2 \
--enable-auto-tool-choice \
--trust-remote-code
```
Pure TP8 is not supported. For >4 GPUs use DP+EP or TP+EP:
### TP4+EP (recommended for H100)
```bash
vllm serve MiniMaxAI/MiniMax-M2 \
--tensor-parallel-size 4 \
--enable-expert-parallel \
--tool-call-parser minimax_m2 \
--reasoning-parser minimax_m2 \
--enable-auto-tool-choice
```
### DP8+EP
```bash
vllm serve MiniMaxAI/MiniMax-M2 \
--data-parallel-size 8 \
--enable-expert-parallel \
--tool-call-parser minimax_m2 \
--reasoning-parser minimax_m2 \
--enable-auto-tool-choice
```
### AMD ROCm — TP2
```bash
VLLM_ROCM_USE_AITER=1 vllm serve MiniMaxAI/MiniMax-M2 \
--tensor-parallel-size 2 \
--tool-call-parser minimax_m2 \
--reasoning-parser minimax_m2 \
--enable-auto-tool-choice \
--trust-remote-code
```
## Benchmarking
```bash
vllm bench serve \
--backend vllm \
--model MiniMaxAI/MiniMax-M2 \
--endpoint /v1/completions \
--dataset-name random \
--random-input 2048 \
--random-output 1024 \
--max-concurrency 10 \
--num-prompt 100
```
## Troubleshooting
- **MiniMax QK norm fusion:** Latest vLLM applies the fusion automatically at
runtime when the fused CUDA op is available; older vLLM builds used an
explicit `fuse_minimax_qk_norm` compile pass.
- **Corrupted output on stable release:** Upgrade to a nightly after commit
`cf3eacfe58fa9e745c2854782ada884a9f992cf7`.
- **DeepGEMM:** vLLM uses DeepGEMM by default; install via
[install_deepgemm.sh](https://github.com/vllm-project/vllm/blob/main/tools/install_deepgemm.sh)
when missing.
- **AITER first launch:** AMD initial launch JIT-compiles CK-based FP8 MoE, RMSNorm,
and activation kernels. Subsequent launches use cached kernels.
## Quantized Variant (NVFP4)
[`RedHatAI/MiniMax-M2-NVFP4`](https://huggingface.co/RedHatAI/MiniMax-M2-NVFP4) is an NVFP4 (4-bit)
checkpoint — roughly half the VRAM of the native FP8 checkpoint. Select the **nvfp4** variant
above (it sets `--kv-cache-dtype fp8`; on vLLM v0.28.0+ the FlashInfer NVFP4 MoE
kernels are selected automatically), or pass the repo id
directly to `vllm serve`. NVFP4 requires a Blackwell GPU (compute capability 10.0+).
## References
- [Model card](https://huggingface.co/MiniMaxAI/MiniMax-M2)
- [NVFP4 variant](https://huggingface.co/RedHatAI/MiniMax-M2-NVFP4)
- [MiniMax](https://www.minimax.io/)
- [vLLM PR #43410 (automatic MiniMax QK norm fusion)](https://github.com/vllm-project/vllm/pull/43410)
recipes/models/MiniMaxAI/MiniMax-M3.yaml
+973
View File
@@ -0,0 +1,973 @@
meta:
title: "MiniMax-M3"
slug: "minimax-m3"
provider: "MiniMax"
description: "MiniMax M3 vision-language MoE (427B total / 26B active) for frontier coding, agent toolchains, and 1M-token reasoning via MSA sparse attention — native multimodal (image + video + computer use); BF16 plus MXFP8, NVIDIA Blackwell NVFP4, and AMD MI355X MXFP4 variants. Runs on NVIDIA (Hopper/Blackwell) and AMD CDNA4/CDNA3."
date_added: 2026-06-12
date_updated: 2026-10-06
tasks:
- text
- multimodal
performance_headline: "Frontier coding and agent (SWE-Bench Pro 59.0, Terminal-Bench 2.1 66.0); MSA sparse attention; 1M context"
related_recipes: []
hardware:
h200: verified
b200: verified
b300: verified
mi300x: verified
mi355x: verified
model:
model_id: "MiniMaxAI/MiniMax-M3"
# Opt in to SMG gRPC PD; parsing runs on the router, not vLLM.
smg_grpc:
reasoning_parser: minimax_m3
tool_call_parser: minimax_m3
min_vllm_version: "0.24.0"
nightly_required: true
docker_image:
nvidia: "vllm/vllm-openai:minimax-m3"
amd: "vllm/vllm-openai-rocm:nightly"
# Docker-only: MiniMax-M3 support hasn't shipped in a stable wheel, and the
# dedicated image is the supported path — hide the pip tab.
architecture: moe
# 427.04B total from safetensors metadata (BF16: 426.99B + F32: 46.3M).
# Active: 57 MoE layers × (4 routed + 1 shared) × 56.6M + 60-layer attn
# 6.6B + 3 dense FFN 0.68B + embeddings 2.46B ≈ 25.9B per token.
parameter_count: "427B"
active_parameters: "26B"
# config.max_position_embeddings is 524288; MiniMax officially advertises 1M
# via RoPE scaling — using the advertised number here.
context_length: 1048576
base_args:
# MSA's sparse_block_size is 128 — KV cache block size must match.
- "--block-size"
- "128"
base_env: {}
features:
tool_calling:
description: "MiniMax M3 tool call parser with automatic tool choice"
args:
- "--tool-call-parser"
- "minimax_m3"
- "--enable-auto-tool-choice"
reasoning:
description: "MiniMax M3 reasoning parser (supports `thinking` and `non-thinking` modes, switchable per request)"
args:
- "--reasoning-parser"
- "minimax_m3"
spec_decoding:
description: "Eagle3 speculative decoding with an Inferact MiniMax-M3 draft head — accelerates decoding. Choose between the standard MHA head and the GQA head (16× smaller draft KV cache — better for long-context or high-batch deployments)."
default_mode: eagle3
modes:
eagle3:
label: "MHA"
description: "Standard Eagle3 draft head (Multi-Head Attention, 64 KV heads)."
args:
- "--speculative-config"
- '{"method": "eagle3", "model": "Inferact/MiniMax-M3-EAGLE3", "num_speculative_tokens": 3, "attention_backend": "FLASH_ATTN"}'
hardware_overrides:
amd:
args:
- "--speculative-config"
- '{"method": "eagle3", "model": "Inferact/MiniMax-M3-EAGLE3", "num_speculative_tokens": 3, "attention_backend": "TRITON_ATTN"}'
eagle3_gqa:
label: "GQA"
description: "GQA Eagle3 draft head (Grouped Query Attention, 4 KV heads vs 64) — 16× smaller draft KV cache. Prefer for long-context or large-batch deployments where draft KV memory is the constraint."
args:
- "--speculative-config"
- '{"method": "eagle3", "model": "Inferact/MiniMax-M3-EAGLE3-GQA", "num_speculative_tokens": 3, "attention_backend": "FLASH_ATTN"}'
hardware_overrides:
amd:
args:
- "--speculative-config"
- '{"method": "eagle3", "model": "Inferact/MiniMax-M3-EAGLE3-GQA", "num_speculative_tokens": 3, "attention_backend": "TRITON_ATTN"}'
text_only:
description: "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
args:
- "--language-model-only"
thinking_always_on:
description: "Pin `thinking_mode: enabled` server-side — every turn reasons before answering, including after tool results. Recommended for agentic workflows."
args:
- "--default-chat-template-kwargs"
- '{"thinking_mode": "enabled"}'
encoder_parallel:
description: "Run the vision encoder data-parallel instead of tensor-parallel — avoids TP comm overhead on the small encoder. Also enables the host-shared-memory multimodal processor cache and the per-brand encoder attention backend (FlashInfer on NVIDIA, AITER FlashAttention on AMD). Mutually exclusive with text_only."
# No platform-agnostic `args`: this recipe targets only NVIDIA + AMD, both
# covered by the per-brand overrides below. (A feature's top-level `args`
# is just the fallback used when no hardware_overrides.<gen> matches.)
hardware_overrides:
# Per-brand vision-encoder config — each block carries shm + tp-mode plus
# its attention backend: FlashInfer on NVIDIA, AITER FlashAttention on AMD.
nvidia:
args:
- "--mm-encoder-attn-backend"
- "FLASHINFER"
- "--mm-processor-cache-type"
- "shm"
- "--mm-encoder-tp-mode"
- "data"
amd:
args:
- "--mm-encoder-attn-backend"
- "ROCM_AITER_FA"
- "--mm-processor-cache-type"
- "shm"
- "--mm-encoder-tp-mode"
- "data"
opt_in_features:
- spec_decoding
- text_only
- thinking_always_on
- encoder_parallel
variants:
default:
precision: bf16
# 427B × 2 bytes (BF16) + 46M × 4 (F32 norms), × 1.2 headroom ≈ 1025 GB.
# 8x H200/H20 (1128 GB) is a tight fit; multi-node TP recommended for
# KV-cache headroom at long context.
# On AMD, 8x MI350X/MI355X fits comfortably.
vram_minimum_gb: 1025
description: "Native BF16 checkpoint — multi-node TP recommended for long-context headroom"
hardware_overrides:
h200:
# After ~1 TB of BF16 weights, 8x H200 has ~3.5 GiB/GPU left for KV;
# vLLM estimates ~83K max tokens, so the 1M default fails at startup
# ("KV cache is needed"). Cap at 64K; use MXFP8 or multi-node for more.
extra_args:
- "--max-model-len"
- "65536"
mxfp8:
model_id: "MiniMaxAI/MiniMax-M3-MXFP8"
precision: mxfp8
# 427B × 1 byte (MXFP8) + F32 norms, × 1.2 ≈ 513 GB → fits 8x H200/H20
# comfortably, ~4 GPUs for weights alone on Blackwell (B200/B300) or AMD MI350X/MI355X (gfx950)).
vram_minimum_gb: 513
description: "NVIDIA-quantized MXFP8 weights — Blackwell (B200/B300) for native MX tensor cores, and AMD CDNA4 (MI350X/MI355X, gfx950) for native MXFP8 Matrix Cores."
extra_args:
- "--attention_config.indexer_kv_dtype"
- "fp8"
hardware_overrides:
blackwell:
extra_env:
VLLM_FLOAT32_MATMUL_PRECISION: "high"
h100:
# 8x H100 80GB (640 GB node) fits the ~513 GB MXFP8 weights but not the
# full 1M-token KV cache — cap the context to 128K so the engine starts.
# H200 (1128 GB node) has the headroom and keeps the full window.
extra_args:
- "--max-model-len"
- "131072"
h200:
# Humming runs the dense MXFP8 linear layers; the atomic-add env var
# enables Marlin's atomic reduction on the MXFP8 MoE path. This image
# also supports the FP8 MSA indexer cache on H200.
docker_image: "vllm/vllm-openai:nightly-1a001d584244bd21829a5e5bd4a376ea141c0ce3"
extra_args:
- "--attention-backend"
- "FLASH_ATTN"
- "--linear-backend"
- "humming"
- "--kv-cache-dtype"
- "fp8"
- "--gpu-memory-utilization"
- "0.95"
- "--enable-prefix-caching"
extra_env:
VLLM_MARLIN_USE_ATOMIC_ADD: "1"
mi300x:
docker_image: "vllm/vllm-openai-rocm:nightly"
extra_args:
- "--language-model-only"
- "--async-scheduling"
- "--max-num-batched-tokens"
- "16384"
extra_env:
VLLM_ENGINE_READY_TIMEOUT_S: "3600"
VLLM_USE_BREAKABLE_CUDAGRAPH: "0"
VLLM_ROCM_USE_AITER: "1"
VLLM_ROCM_USE_AITER_MHA: "0"
TORCH_BLAS_PREFER_HIPBLASLT: "1"
NCCL_MIN_NCHANNELS: "112"
GPU_MAX_HW_QUEUES: "2"
mi355x:
docker_image: "vllm/vllm-openai-rocm:nightly"
extra_args:
# run linear-layer GEMMs through the bf16 hipblasLT on MI355X.
- "--linear-backend"
- "emulation"
extra_env:
VLLM_ROCM_USE_AITER: "1"
VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS: "1"
VLLM_ROCM_QUICK_REDUCE_QUANTIZATION: "INT4"
nvfp4:
model_id: "nvidia/MiniMax-M3-NVFP4"
precision: nvfp4
vram_minimum_gb: 257
description: "NVIDIA-quantized NVFP4 weights (ModelOpt) for Blackwell (B200/B300) — native FP4 tensor cores, ~half the VRAM of MXFP8."
extra_args:
- "--trust-remote-code"
- "--kv-cache-dtype"
- "fp8"
# FlashInfer TRT-LLM attention kernels with an FP8 indexer KV cache.
- "--attention_config.backend"
- "FLASHINFER"
- "--attention_config.use_trtllm_attention"
- "true"
- "--attention_config.indexer_kv_dtype"
- "fp8"
- "--attention_config.minimax_m3_msa_decode_backend"
- "cutlass"
extra_env:
VLLM_FLOAT32_MATMUL_PRECISION: "high"
VLLM_FLASHINFER_ALLREDUCE_BACKEND: "trtllm"
mxfp4:
model_id: "amd/MiniMax-M3-MXFP4"
precision: mxfp4
# 427B × 0.5 bytes (MXFP4) × 1.2 headroom = 256.2 GB.
vram_minimum_gb: 257
description: "AMD Quark OCP MXFP4 weights and dynamic MXFP4 activations for MI355X (gfx950)."
supported_hardware:
- mi355x
# Match the validated InferenceX single-node baseline. TP8 also leaves
# practical headroom for KV cache and the multimodal encoder.
tp: 8
hardware_overrides:
mi355x:
docker_image: "vllm/vllm-openai-rocm:nightly"
extra_args:
- "--trust-remote-code"
- "--compilation-config"
- '{"cudagraph_mode":"FULL_DECODE_ONLY"}'
# Keep the multimodal encoder loaded and run it data-parallel on MI355X.
- "--mm-encoder-tp-mode"
- "data"
- "--mm-encoder-attn-backend"
- "ROCM_AITER_FA"
compatible_strategies:
- single_node_tp
- single_node_tep
- single_node_dep
- multi_node_tp
- multi_node_tp_pp
- multi_node_tep
- multi_node_dep
- pd_cluster
hardware_overrides:
amd:
# ROCm MI3xx: MSA sparse attention runs on the Triton backend — this is the
# main-model attention and is always needed on AMD. The vision-encoder
# flags (data-parallel + AITER FlashAttention) are gated behind the
# encoder_parallel feature, mirroring NVIDIA.
extra_args:
- "--attention-backend"
- "TRITON_ATTN"
extra_env:
# Run MiniMax-M3 with CUDA graphs on AMD MI3xx; avoids the decode
# breakable-cudagraph path that otherwise forces eager (per @hongxiayang).
VLLM_USE_BREAKABLE_CUDAGRAPH: "0"
VLLM_ROCM_USE_AITER_MOE: "1"
VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS: "1"
strategy_overrides:
pd_cluster:
hardware_overrides:
blackwell:
extra_args:
- "--attention_config.backend"
- "FLASHINFER"
- "--attention_config.use_trtllm_attention"
- "true"
- "--attention_config.indexer_kv_dtype"
- "fp8"
guide: |
## Overview
[MiniMax-M3](https://huggingface.co/MiniMaxAI/MiniMax-M3) is a frontier
vision-language MoE model from [MiniMax](https://www.minimax.io/).
- **MSA (MiniMax Sparse Attention)** — scalable sparse-attention architecture
that lifts the context window to **1M tokens**. MiniMax reports per-token
compute at 1M context reduced to ~1/20 of the previous generation, with
>9× prefill and >15× decode speedup vs dense baselines.
- **Frontier coding and agent capabilities** — SWE-Bench Pro 59.0%,
Terminal-Bench 2.1 66.0%, SWE-fficiency 34.8%, KernelBench Hard 28.8%,
MCP Atlas 74.2%.
- **Native multimodal** — image + video inputs, plus computer-use; trained
multimodally from step 0.
- **Two reasoning modes** — `thinking` (complex reasoning / agents) and
`non-thinking` (latency-sensitive), switchable per request.
## Prerequisites
- **OS:** Linux
- **Python:** 3.10 - 3.13
- **NVIDIA:** compute capability >= 9.0 (Hopper) recommended; 8x H200 / H20
for a tight single-node BF16 fit, or multi-node TP for long-context headroom
- **AMD:** MI350X/MI355X (gfx950), MI300X/MI325X (gfx942), ROCm 7.2+. BF16
needs TP=8; the MXFP8 variant runs from TP=4.
- **`--block-size 128` is mandatory on every platform** (MSA sparse/index cache).
### Docker (NVIDIA)
MiniMax-M3 support has not yet shipped in a stable vLLM release — use the
dedicated Docker image:
```bash
docker pull vllm/vllm-openai:minimax-m3
```
### Docker (AMD ROCm)
Use the nightly image, which is the image the AMD variants of this recipe pin:
```bash
docker pull vllm/vllm-openai-rocm:nightly
```
```bash
docker run --rm -it --device /dev/kfd --device /dev/dri --group-add video \
--cap-add SYS_PTRACE --security-opt seccomp=unconfined --ipc=host \
--shm-size=16g -p 8000:8000 \
--entrypoint /bin/bash \
vllm/vllm-openai-rocm:nightly
```
## Launching the Server
### NVIDIA — TP8 (8x H200 / H20)
```bash
vllm serve MiniMaxAI/MiniMax-M3 \
--tensor-parallel-size 8 \
--block-size 128 \
--tool-call-parser minimax_m3 \
--reasoning-parser minimax_m3 \
--enable-auto-tool-choice
```
#### TP8 + Expert Parallel
```bash
vllm serve MiniMaxAI/MiniMax-M3 \
--tensor-parallel-size 8 \
--enable-expert-parallel \
--block-size 128 \
--tool-call-parser minimax_m3 \
--reasoning-parser minimax_m3 \
--enable-auto-tool-choice
```
#### DP8 + Expert Parallel
```bash
vllm serve MiniMaxAI/MiniMax-M3 \
--data-parallel-size 8 \
--enable-expert-parallel \
--block-size 128 \
--tool-call-parser minimax_m3 \
--reasoning-parser minimax_m3 \
--enable-auto-tool-choice
```
### AMD ROCm (MI350X/MI355X (gfx950), MI300X/MI325X (gfx942))
On AMD MI300X / MI325X / MI355X, run with CUDA graphs and set the following
before any of the serve commands below. It avoids the MiniMax-M3 decode breakable-cudagraph path that would
otherwise force eager execution (per @hongxiayang):
```bash
export VLLM_USE_BREAKABLE_CUDAGRAPH=0
```
For gfx950: Prefer using the MXFP8 variant `MiniMaxAI/MiniMax-M3-MXFP8` for TP=4 and a smaller
footprint. Use TP=8 for lower latency or long context length, or the default bf16 model.
#### TP8 (Text or Vision)
```bash
vllm serve MiniMaxAI/MiniMax-M3 \
--tensor-parallel-size 8 \
--block-size 128 \
--attention-backend TRITON_ATTN \
--mm-encoder-tp-mode data \
--mm-encoder-attn-backend ROCM_AITER_FA \
--tool-call-parser minimax_m3 \
--reasoning-parser minimax_m3 \
--enable-auto-tool-choice
```
Add the vision-encoder flags (`--mm-encoder-tp-mode data
--mm-encoder-attn-backend ROCM_AITER_FA`) for multimodal serving.
#### TP8 + `--quantization fp8_per_channel`
Online **per-channel FP8 (PTPC)** quantization of the BF16 checkpoint halves the
weight footprint (≈100 → **≈50 GiB/GPU**, ~1.75× more KV cache) and lifts
batched throughput with **gsm8k unchanged from BF16** (lossless);
These env vars and the flag are for the **BF16 checkpoint only**.
```bash
export VLLM_USE_BREAKABLE_CUDAGRAPH=0
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_AITER_MOE=0
vllm serve MiniMaxAI/MiniMax-M3 \
--tensor-parallel-size 8 \
--block-size 128 \
--attention-backend TRITON_ATTN \
--quantization fp8_per_channel \
--tool-call-parser minimax_m3 \
--reasoning-parser minimax_m3 \
--enable-auto-tool-choice
```
#### TP8 + Expert Parallel
```bash
vllm serve MiniMaxAI/MiniMax-M3 \
--tensor-parallel-size 8 \
--enable-expert-parallel \
--block-size 128 \
--attention-backend TRITON_ATTN \
--mm-encoder-tp-mode data \
--mm-encoder-attn-backend ROCM_AITER_FA \
--tool-call-parser minimax_m3 \
--reasoning-parser minimax_m3 \
--enable-auto-tool-choice
```
#### DP8 + Expert Parallel
```bash
vllm serve MiniMaxAI/MiniMax-M3 \
--data-parallel-size 8 \
--enable-expert-parallel \
--block-size 128 \
--attention-backend TRITON_ATTN \
--mm-encoder-tp-mode data \
--mm-encoder-attn-backend ROCM_AITER_FA \
--tool-call-parser minimax_m3 \
--reasoning-parser minimax_m3 \
--enable-auto-tool-choice
```
## FP8 KV Cache
Add `--kv-cache-dtype fp8` to any command for ~1.5× the KV pool — lossless
in our testing across the full native context. Especially worth it for high
concurrency or long context, where KV is the binding constraint.
## Context Length & GPU Memory
The full **1M-token** window (`context_length: 1048576`) needs a large KV
cache. To save GPU memory, you can optionally cap the context with
`--max-model-len`:
```bash
vllm serve MiniMaxAI/MiniMax-M3 \
--tensor-parallel-size 8 \
--block-size 128 \
--max-model-len 131072 # 128K instead of the full 1M
```
AMD ROCm notes: Native context is 512K. To go past it, supply a YaRN rope_scaling on the text config
(a top-level override silently misses the decoder's config) and allow the long max length.
TP=8 + fp8 KV is the practical combo at 1M:
```bash
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
vllm serve MiniMaxAI/MiniMax-M3 \
--block-size 128 \
--kv-cache-dtype fp8 \
--tensor-parallel-size 8 \
--max-model-len 1048576 \
--attention-backend TRITON_ATTN \
--mm-encoder-tp-mode data \
--mm-encoder-attn-backend ROCM_AITER_FA \
--tool-call-parser minimax_m3 \
--enable-auto-tool-choice \
--reasoning-parser minimax_m3 \
--hf-overrides '{"text_config":{"rope_scaling":{"rope_type":"yarn","factor":2.0,"original_max_position_embeddings":524288}}}'
```
- Set `--max-model-len` to the longest prompt + output you actually serve
(e.g. `32768`, `131072`, `262144`). A smaller window frees KV-pool headroom
for higher concurrency and lets the model fit on fewer GPUs; if you need the
full 1M window, consider scaling out with multi-node TP instead.
## Client Usage
Recommended sampling parameters (from the model card):
- `temperature = 1.0`
- `top_p = 0.95`
- `top_k = 40`
Default system prompt:
```
You are a helpful assistant. Your name is MiniMax-M3 and is built by MiniMax.
```
Example chat request:
```bash
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"model": "MiniMaxAI/MiniMax-M3",
"temperature": 1.0,
"top_p": 0.95,
"messages": [
{"role": "system", "content": "You are a helpful assistant. Your name is MiniMax-M3 and is built by MiniMax."},
{"role": "user", "content": "Explain MSA sparse attention in 3 bullets."}
]
}'
```
## Thinking Modes
M3 reasoning is controlled by the `thinking_mode`, there are three values:
- **`enabled`** — the model thinks before every response, including after tool results.
Use for complex reasoning and agents.
- **`disabled`** — no thinking; the model answers directly. Use for
latency-sensitive turns.
- **`adaptive`** *(default when unset)* — the model decides whether to think
based on the task.
Pass it per request through `chat_template_kwargs`. The same value also tunes
the `minimax_m3` reasoning parser, so `reasoning` and `content` are
split correctly in every mode.
```python
# Start the MiniMax-M3 model by referring to the command above first.
from openai import OpenAI
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
response = client.chat.completions.create(
model="MiniMaxAI/MiniMax-M3",
messages=[{"role": "user", "content": "Prove there are infinitely many primes."}],
extra_body={"chat_template_kwargs": {"thinking_mode": "enabled"}},
)
msg = response.choices[0].message
print(msg.reasoning) # the <mm:think> block
print(msg.content) # the final answer
```
## Benchmarking
```bash
vllm bench serve \
--backend vllm \
--model MiniMaxAI/MiniMax-M3 \
--endpoint /v1/completions \
--dataset-name random \
--random-input 2048 \
--random-output 1024 \
--max-concurrency 10 \
--num-prompt 100
```
## Quantized Variant (MXFP8)
[`MiniMaxAI/MiniMax-M3-MXFP8`](https://huggingface.co/MiniMaxAI/MiniMax-M3-MXFP8)
is an MXFP8 checkpoint quantized by NVIDIA from the original FP16 weights —
roughly half the VRAM of the BF16 release. Select the **mxfp8** variant above,
or pass the repo id directly to `vllm serve`:
```bash
vllm serve MiniMaxAI/MiniMax-M3-MXFP8 \
--tensor-parallel-size 8 \
--block-size 128 \
--tool-call-parser minimax_m3 \
--reasoning-parser minimax_m3 \
--enable-auto-tool-choice
```
For best MXFP8 throughput, prefer Blackwell (B200/B300) for native MX tensor cores,
or AMD CDNA4 (MI350X/MI355X, gfx950) for native MXFP8 matrix cores.
On AMD MI300X, the **mxfp8** variant stacks two accuracy-safe scheduling
levers from the InferenceX MI300X tuning recipe and avoids the high-concurrency
expert-parallel regression by using plain TP8:
```bash
export VLLM_ENGINE_READY_TIMEOUT_S=3600
export VLLM_USE_BREAKABLE_CUDAGRAPH=0
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_AITER_MHA=0
export TORCH_BLAS_PREFER_HIPBLASLT=1
export NCCL_MIN_NCHANNELS="${NCCL_MIN_NCHANNELS:-112}"
export GPU_MAX_HW_QUEUES="${GPU_MAX_HW_QUEUES:-2}"
vllm serve MiniMaxAI/MiniMax-M3-MXFP8 \
--tensor-parallel-size 8 \
--block-size 128 \
--language-model-only \
--max-model-len "$MAX_MODEL_LEN" \
--attention-backend TRITON_ATTN \
--async-scheduling \
--max-num-batched-tokens 16384 \
--tool-call-parser minimax_m3 \
--reasoning-parser minimax_m3 \
--enable-auto-tool-choice
```
On AMD MI355X the **mxfp8** variant additionally enables the AITER kernels,
the fused shared-experts MoE, and INT4 QuickReduce, and pulls the nightly ROCm
image (`vllm/vllm-openai-rocm:nightly`) — selected automatically in the command
builder. If you launch by hand:
```bash
export VLLM_USE_BREAKABLE_CUDAGRAPH=0
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4
vllm serve MiniMaxAI/MiniMax-M3-MXFP8 \
--tensor-parallel-size 8 \
--block-size 128 \
--attention-backend TRITON_ATTN \
--tool-call-parser minimax_m3 \
--reasoning-parser minimax_m3 \
--enable-auto-tool-choice
```
### MXFP8 high-concurrency options on MI355X
The command above is the default serving configuration and is tuned for the
general case. **AITER page-16 sparse paged attention** helps specifically at **long context and
high concurrency**. It maps MiniMax-M3's top-k 128-token MSA blocks onto AITER page-16 block tables
and runs AITER Gluon paged attention over only the selected KV pages.
The high-concurrency configuration is:
```bash
# Long context + high concurrency only (isl >= 8k, conc >= 64). Regresses
# short-context / low-concurrency serving.
export VLLM_USE_BREAKABLE_CUDAGRAPH=0
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1
export VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4
export VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16=0
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION_MIN_SIZE_KB=256
vllm serve MiniMaxAI/MiniMax-M3-MXFP8 \
--tensor-parallel-size 4 \
--block-size 128 \
--language-model-only \
--moe-backend aiter \
--kv-cache-dtype fp8 \
--max-num-batched-tokens 32768 \
--attention-backend TRITON_ATTN \
--linear-backend emulation \
--tool-call-parser minimax_m3 \
--reasoning-parser minimax_m3 \
--enable-auto-tool-choice
```
Note: Outside the long-context/high-batch
regime the shuffled layout costs more than it saves; leave it at `0`:
```bash
export VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=0
```
## Quantized Variant (NVFP4 on Blackwell)
[`nvidia/MiniMax-M3-NVFP4`](https://huggingface.co/nvidia/MiniMax-M3-NVFP4)
is an NVFP4 checkpoint quantized by NVIDIA with ModelOpt. It requires a vLLM
build with MiniMax-M3 NVFP4 support (vllm-project/vllm PR #46380).
Select the **nvfp4** variant and **B200** hardware in the command builder, or
launch the tested B200 baseline directly:
```bash
VLLM_FLOAT32_MATMUL_PRECISION=high \
vllm serve nvidia/MiniMax-M3-NVFP4 \
--tensor-parallel-size 8 \
--trust-remote-code \
--block-size 128 \
--tool-call-parser minimax_m3 \
--reasoning-parser minimax_m3 \
--enable-auto-tool-choice
```
Pair it with EAGLE3 speculative decoding (the *Spec decoding* feature) for
faster decode — the validated B200 benchmark runs the NVFP4 target against the
`Inferact/MiniMax-M3-EAGLE3` draft head (3 speculative tokens):
```bash
VLLM_FLOAT32_MATMUL_PRECISION=high \
vllm serve nvidia/MiniMax-M3-NVFP4 \
--tensor-parallel-size 8 \
--trust-remote-code \
--block-size 128 \
--language-model-only \
--speculative-config '{"method": "eagle3", "model": "Inferact/MiniMax-M3-EAGLE3", "num_speculative_tokens": 3, "attention_backend": "FLASH_ATTN"}'
```
For long-context or high-batch workloads where draft KV memory is the binding
constraint, swap in the **GQA** variant:
```bash
VLLM_FLOAT32_MATMUL_PRECISION=high \
vllm serve nvidia/MiniMax-M3-NVFP4 \
--tensor-parallel-size 8 \
--trust-remote-code \
--block-size 128 \
--language-model-only \
--speculative-config '{"method": "eagle3", "model": "Inferact/MiniMax-M3-EAGLE3-GQA", "num_speculative_tokens": 3, "attention_backend": "FLASH_ATTN"}'
```
You may also consider FlashInfer for draft attention if it performs better for
your workload, and local argmax reduction to reduce communication traffic:
```bash
--speculative-config '{"method":"eagle3","model":"Inferact/MiniMax-M3-EAGLE3-GQA","num_speculative_tokens":3,"attention_backend":"FLASHINFER","use_local_argmax_reduction":true}'
```
## Quantized Variant (MXFP4 on MI355X)
[`amd/MiniMax-M3-MXFP4`](https://huggingface.co/amd/MiniMax-M3-MXFP4)
uses static OCP MXFP4 weights and dynamic OCP MXFP4 activations produced with
AMD Quark. It targets AMD Instinct MI355X (gfx950) and requires a ROCm
nightly containing the Quark MXFP4 fix from vLLM PR #45794.
Select the **mxfp4** variant and **MI355X** hardware in the command builder,
or launch the tested TP8 baseline directly:
```bash
vllm serve amd/MiniMax-M3-MXFP4 \
--tensor-parallel-size 8 \
--trust-remote-code \
--block-size 128 \
--attention-backend TRITON_ATTN \
--mm-encoder-tp-mode data \
--mm-encoder-attn-backend ROCM_AITER_FA \
--tool-call-parser minimax_m3 \
--enable-auto-tool-choice \
--reasoning-parser minimax_m3
```
### MXFP4 benchmark reproduction (InferenceX MI355X sweep)
The command above is the default serving configuration. The
[SemiAnalysis InferenceX](https://inferencex.semianalysis.com/) MI355X
agentic-coding lane for this checkpoint is a **separate benchmark
configuration** — single-node, text-only, GPU-resident, and driven by the GQA
EAGLE3 draft head. It is not the recipe default and not a general serving
recommendation.
The lane is the `minimaxm3-fp4-mi355x-vllm-agentic-mtp` entry in
`configs/amd-master.yaml`, launched by
`benchmarks/single_node/agentic/minimaxm3_fp4_mi355x_mtp.sh` in
[SemiAnalysisAI/InferenceX](https://github.com/SemiAnalysisAI/InferenceX):
| Setting | Benchmark lane |
|---------|----------------|
| Served checkpoint | `amd/MiniMax-M3-MXFP4` |
| Hardware | one MI355X node |
| Scenario | agentic-coding trace replay |
| Parallelism | TP4 and TP2 — no expert parallelism, no KV offload |
| TP4 concurrency sweep | 1, 4, 5, 8, 10, 12, 15, 20, 24, 32 |
| TP2 concurrency sweep | 1, 2 |
| Draft head | `Inferact/MiniMax-M3-EAGLE3-GQA`, 3 speculative tokens |
| Attention | `ROCM_AITER_UNIFIED_ATTN` for both the target and the draft |
| Indexer K cache | `fp8` (`--attention-config`); MiniMax-M3's default is bf16 |
| Model path | text-only (`--language-model-only`), prefix caching enabled |
Differences from the default command above: TP4/TP2 instead of TP8, the
text-only language-model path, `--kv-cache-dtype fp8`, the FP8 sparse-attention
indexer K cache, the AITER MXFP4 MoE backend, unified AITER attention instead
of `TRITON_ATTN`, the GQA EAGLE3 draft, and a hand-tuned decode
`cudagraph_capture_sizes` list sized for the lane's `--max-num-seqs` range. The
lane also pins thinking server-side with `--default-chat-template-kwargs`, and
it passes no `--hf-overrides` — the pinned image supplies the sparse-PA
index-cache and top-k refresh behavior itself, and re-adding the override
would rewrite the served model config.
Run it on a current official `vllm/vllm-openai-rocm:nightly`. The lane pins one
exact nightly digest rather than a release tag: the AITER shuffled KV-cache
layout that MiniMax-M3 sparse paged attention relies on was changed by the
KV-cache layout standardization in
[vllm-project/vllm#51718](https://github.com/vllm-project/vllm/pull/51718),
and the nightly restores it. The lane's pinned digest also carries the AITER
indexer scoring and top-k kernels for the MiniMax-M3 sparse-attention path from
[vllm-project/vllm#52664](https://github.com/vllm-project/vllm/pull/52664),
which is what makes the FP8 indexer K cache and the unconditional shuffled
layout below worthwhile. Don't hard-code a transient tag here — the digest
validated for these numbers lives in the InferenceX config.
#### Environment
```bash
# TP4 and TP2 are the only topologies this lane validates.
TP_SIZE="${TP_SIZE:-4}"
CONC="${CONC:-32}"
if [ "$TP_SIZE" -ne 4 ] && [ "$TP_SIZE" -ne 2 ]; then
echo "TP_SIZE must be 2 or 4 for this benchmark lane" >&2
exit 1
fi
# Shuffled KV-cache layout, now unconditional. Earlier digests needed the
# page-16 sparse-attention fast path's one-KV-head-per-rank property, so TP2
# (two of MiniMax-M3's four KV heads per rank) had to fall back to 0; the
# pinned digest's AITER indexer kernels serve both widths from the shuffled
# layout, and the lane runs TP4 and TP2 with it enabled.
export VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=1
# Kernel selection — required to match the benchmarked MoE path.
# Without these (and `--moe-backend aiter` on the serve command), vLLM's MXFP4
# MoE oracle selects the TRITON_UNFUSED fallback on gfx950 instead of the
# AITER_MXFP4_MXFP4 kernel the InferenceX lane actually runs.
export VLLM_ENGINE_READY_TIMEOUT_S=3600
export VLLM_EXECUTE_MODEL_TIMEOUT_SECONDS=1800
export VLLM_USE_BREAKABLE_CUDAGRAPH=0
export VLLM_ROCM_USE_AITER=1
export VLLM_ROCM_USE_AITER_MOE=1
export VLLM_ROCM_USE_AITER_FUSION_SHARED_EXPERTS=1
# Benchmark-only — accuracy-sensitive; NOT part of the default command.
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION=INT4
export VLLM_ROCM_QUICK_REDUCE_CAST_BF16_TO_FP16=0
export VLLM_ROCM_QUICK_REDUCE_QUANTIZATION_MIN_SIZE_KB=256
```
#### Serving and evaluation (real target verification)
This is the form to use for the lane's accuracy evaluation and for any normal
serving. The draft's proposals are verified against the real target model:
```bash
vllm serve amd/MiniMax-M3-MXFP4 \
--port "${PORT:-8000}" \
--tensor-parallel-size "$TP_SIZE" \
--trust-remote-code \
--block-size 128 \
--gpu-memory-utilization 0.90 \
--enable-chunked-prefill \
--max-num-batched-tokens 32768 \
--compilation-config '{"cudagraph_mode":"FULL_DECODE_ONLY", "cudagraph_capture_sizes": [1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,20,22,24,26,28,30,32,34,36,48,64,72,80,88,96,104,112,120,128]}' \
--language-model-only \
--enable-prefix-caching \
--attention-backend ROCM_AITER_UNIFIED_ATTN \
--moe-backend aiter \
--kv-cache-dtype fp8 \
--attention-config '{"indexer_kv_dtype": "fp8"}' \
--tool-call-parser minimax_m3 \
--reasoning-parser minimax_m3 \
--enable-auto-tool-choice \
--default-chat-template-kwargs '{"thinking_mode":"enabled"}' \
--max-num-seqs "$((2 * CONC))" \
--stream-interval 20 \
--speculative-config '{"method": "eagle3", "model": "Inferact/MiniMax-M3-EAGLE3-GQA", "num_speculative_tokens": 3, "attention_backend": "ROCM_AITER_UNIFIED_ATTN"}'
```
#### Throughput measurement only (synthetic acceptance)
The throughput arm of the sweep replaces only the `--speculative-config` with
a synthetic rejection sampler so that every concurrency point is measured
against the same acceptance rate:
```bash
--speculative-config '{"method": "eagle3", "model": "Inferact/MiniMax-M3-EAGLE3-GQA", "num_speculative_tokens": 3, "attention_backend": "ROCM_AITER_UNIFIED_ATTN", "rejection_sample_method": "synthetic", "synthetic_acceptance_length": 2.78}'
```
`2.78` is **not** a tuning knob and **not** a production shortcut. It is the
committed golden EAGLE3-GQA acceptance length for MiniMax-M3 with thinking
enabled at 3 speculative tokens, taken from
[`golden_al_distribution/minimaxm3_eagle3_gqa.yaml`](https://github.com/SemiAnalysisAI/InferenceX/blob/main/golden_al_distribution/minimaxm3_eagle3_gqa.yaml).
Synthetic rejection standardizes throughput comparisons; it bypasses real
verification, so it must never be used for accuracy evaluation or for serving
users. Use the real-verification form above for everything else.
Selecting the AITER MXFP4 MoE backend (`--moe-backend aiter`) is a
kernel-selection choice (same MXFP4 math, faster kernel), not a precision
reduction. The INT4 all-reduce, the fp8 KV cache, and the fp8 indexer K cache
are throughput optimizations for high concurrency and are **kept out of the
default command pending accuracy validation** — `--attention-config
'{"indexer_kv_dtype": "fp8"}'` quantizes the sparse-attention indexer's K
cache away from MiniMax-M3's bf16 default and depends on the indexer kernels
in the pinned digest. The raised `--gpu-memory-utilization 0.90` and the
explicit `cudagraph_capture_sizes` list are likewise lane-specific: the list
is sized for this sweep's `--max-num-seqs` range (`2 × concurrency`, so at
most 64 here) and capping graph capture at 128 would shrink CUDA-graph
coverage on a general deployment that serves wider batches. Performance
figures on the InferenceX dashboard are **reported externally by SemiAnalysis
and have not been reproduced in this repository.**
#### References for this benchmark lane
- Current source PR: [SemiAnalysisAI/InferenceX#2959](https://github.com/SemiAnalysisAI/InferenceX/pull/2959)
(image bump, fp8 indexer K cache, tuned decode graphs) — reviewed at head
`43c23db28f400f643f79dc490699910a010d3360`
- Full sweep run for that lane: [actions/runs/34609443618](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/34609443618)
— all TP4 and TP2 throughput and real-verification eval jobs passed on
commit `c738b5030e2277cc701ab15e115cb6584e0cacd0`, whose lane script and
config entry are byte-identical to the reviewed head
- Original lane PR: [SemiAnalysisAI/InferenceX#2781](https://github.com/SemiAnalysisAI/InferenceX/pull/2781)
([sweep](https://github.com/SemiAnalysisAI/InferenceX/actions/runs/33239952571))
- Golden GQA acceptance-length curve: [`minimaxm3_eagle3_gqa.yaml`](https://github.com/SemiAnalysisAI/InferenceX/blob/main/golden_al_distribution/minimaxm3_eagle3_gqa.yaml)
- KV-cache layout change the image pin compensates for: [vllm-project/vllm#51718](https://github.com/vllm-project/vllm/pull/51718)
- AITER indexer scoring and top-k kernels the pinned digest adds: [vllm-project/vllm#52664](https://github.com/vllm-project/vllm/pull/52664)
## Troubleshooting
- **`--block-size` mismatch.** MSA's sparse block size is 128; the vLLM KV
cache block size must match. Using the default (16) misaligns the sparse
attention indexing (on AMD it crashes with `No common block size for 16`).
- **Parsers.** `--tool-call-parser` and `--reasoning-parser` both use
`minimax_m3` — distinct from `minimax_m2` used by earlier releases.
- **Long context KV cache.** See *Context Length & GPU Memory* above — cap
`--max-model-len` or scale to multi-node TP if you OOM.
- **Vision encoder.** The encoder is small, so at high TP the *Encoder
Parallel* option runs it data-parallel (`--mm-encoder-tp-mode data`) to
avoid TP comm overhead; it also turns on the vision-encoder attention
backend (FlashInfer on NVIDIA, `--mm-encoder-attn-backend FLASHINFER`;
AITER FlashAttention on AMD, `ROCM_AITER_FA`) and the host-shared-memory
multimodal processor cache (`--mm-processor-cache-type shm`). For **text-only** workloads enable *Text only*
(`--language-model-only`) to skip loading the encoder and free VRAM — it is
mutually exclusive with Encoder Parallel.
## EAGLE3 Speculative Decoding
Two Inferact draft heads are available as selectable modes in the *Spec decoding*
feature — both use `method: eagle3` with 3 speculative tokens and share
MiniMax-M3's embedding table and LM head:
| Mode | Repo | KV heads | Draft KV cache |
|------|------|----------|----------------|
| **MHA** (default) | `Inferact/MiniMax-M3-EAGLE3` | 64 | baseline |
| **GQA** | `Inferact/MiniMax-M3-EAGLE3-GQA` | 4 | **16× smaller** |
The GQA head uses Grouped Query Attention (4 KV heads grouped from 64 query
heads) — the only architectural difference from the MHA head. The 16× reduction
in draft KV state makes it preferable when draft memory is the binding constraint
(long-context serving or large batch sizes).
## References
- [Model card](https://huggingface.co/MiniMaxAI/MiniMax-M3)
- [MXFP8 variant](https://huggingface.co/MiniMaxAI/MiniMax-M3-MXFP8)
- [NVFP4 variant](https://huggingface.co/nvidia/MiniMax-M3-NVFP4)
- [AMD MXFP4 variant](https://huggingface.co/amd/MiniMax-M3-MXFP4)
- [EAGLE3 draft head (MHA)](https://huggingface.co/Inferact/MiniMax-M3-EAGLE3)
- [EAGLE3 draft head (GQA)](https://huggingface.co/Inferact/MiniMax-M3-EAGLE3-GQA)
- [MiniMax](https://www.minimax.io/)
- [MiniMax Agent](https://agent.minimax.io/)
- [MiniMax Platform](https://platform.minimax.io/)
- [Previous release: MiniMax-M2.7](https://huggingface.co/MiniMaxAI/MiniMax-M2.7)
recipes/models/OpenGVLab/InternVL3_5-8B.yaml
+143
View File
@@ -0,0 +1,143 @@
meta:
title: "InternVL3.5"
slug: "internvl3.5"
provider: "InternVL (OpenGVLab)"
description: "InternVL 3.5 vision-language models from Shanghai AI Lab with thinking-mode prompting"
date_added: 2025-09-20
date_updated: 2026-04-17
tasks:
- multimodal
related_recipes:
- internlm/Intern-S1
hardware:
h100: verified
mi300x: verified
mi325x: verified
mi355x: verified
model:
model_id: "OpenGVLab/InternVL3_5-8B"
min_vllm_version: "0.10.0"
architecture: dense
parameter_count: "8B"
active_parameters: "8B"
context_length: 40960
base_args:
- "--trust-remote-code"
base_env: {}
features:
text_only:
description: "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
args:
- "--language-model-only"
encoder_parallel:
description: "Run the vision encoder in data-parallel mode — avoids TP comm overhead on the small encoder. Mutually exclusive with text_only."
args:
- "--mm-encoder-tp-mode"
- "data"
opt_in_features:
- text_only
- encoder_parallel
variants:
default:
precision: bf16
vram_minimum_gb: 19
description: "BF16 weights for the 8B variant"
compatible_strategies:
- single_node_tp
- multi_node_tp
kv_offload_support:
offloading_cpu: verified
offloading_fs: verified
hardware_overrides: {}
strategy_overrides: {}
guide: |
## Overview
[InternVL3.5](https://github.com/OpenGVLab/InternVL) is a vision-language model developed
by Shanghai AI Laboratory. It supports single-image and multi-image prompts, plus an
optional "thinking mode" via a custom system prompt.
## Prerequisites
- Hardware: 1x GPU with >=20 GB VRAM (A100, L40S, H100, etc.)
- vLLM >= 0.10.0
### Install vLLM (CUDA)
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```
### Install vLLM (AMD ROCm MI300X/MI325X/MI355X)
```bash
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.14.1/rocm700
```
## Launch command
```bash
vllm serve OpenGVLab/InternVL3_5-8B --trust-remote-code
```
On AMD:
```bash
export VLLM_ROCM_USE_AITER=1
vllm serve OpenGVLab/InternVL3_5-8B --trust-remote-code
```
## Client Usage
Single image:
```python
from openai import OpenAI
client = OpenAI(api_key="", base_url="http://0.0.0.0:8000/v1")
model_name = client.models.list().data[0].id
response = client.chat.completions.create(
model=model_name,
messages=[{
"role": "user",
"content": [
{"type": "text", "text": "Describe the image."},
{"type": "image_url", "image_url": {"url": "https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg"}},
],
}],
temperature=0.0,
)
print(response.choices[0].message.content)
```
## Thinking Mode
Set a thinking system prompt and use `temperature=0.6` to mitigate repetition:
```python
THINKING_SYSTEM_PROMPT = """
You are an AI assistant that rigorously follows this response protocol:
1. First, conduct a detailed analysis of the question. Consider different angles, potential
solutions, and reason through the problem step-by-step. Enclose this entire thinking process
within <think> and </think> tags.
2. After the thinking section, provide a clear, concise, and direct answer to the user's
question. Separate the answer from the think section with a newline.
""".strip()
```
## References
- [InternVL3.5-8B](https://huggingface.co/OpenGVLab/InternVL3_5-8B)
- [InternVL GitHub](https://github.com/OpenGVLab/InternVL)
recipes/models/OpenMOSS-Team/MOSS-Transcribe-Diarize.yaml
+179
View File
@@ -0,0 +1,179 @@
meta:
title: "MOSS-Transcribe-Diarize"
slug: "moss-transcribe-diarize"
provider: "OpenMOSS"
description: "OpenMOSS's 0.9B end-to-end multi-speaker long-audio transcription model with timestamps and speaker labels, served through vLLM's OpenAI-compatible /v1/audio/transcriptions API."
date_added: 2026-07-08
date_updated: 2026-08-10
tasks:
- multimodal
performance_headline: "0.9B end-to-end transcription with speaker labels and timestamps"
related_recipes:
- OpenMOSS-Team/MOSS-TTS
model:
model_id: "OpenMOSS-Team/MOSS-Transcribe-Diarize"
min_vllm_version: "0.27.0"
architecture: dense
parameter_count: "0.9B"
active_parameters: "0.9B"
base_args:
- "--trust-remote-code"
base_env: {}
dependencies:
- note: "Audio extras are required for loading and preprocessing audio files."
command: 'uv pip install -U "vllm[audio]" --torch-backend auto'
features: {}
opt_in_features: []
variants:
default:
precision: bf16
vram_minimum_gb: 4
description: "MOSS-Transcribe-Diarize-0.9B - single-GPU long-form multi-speaker transcription"
compatible_strategies:
- single_node_tp
hardware_overrides: {}
strategy_overrides: {}
guide: |
## Overview
[MOSS-Transcribe-Diarize](https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize)
is a 0.9B end-to-end speech-to-text model for long-form multi-speaker
transcription. It uses a Whisper-style audio encoder and a Qwen3-style causal
decoder to generate structured transcripts with text, timestamps, and speaker
labels in a single pass.
Example output:
```text
[0.11][S01] Good morning! [1.03]
[1.11][S02] Morning, let's start the meeting. [2.42]
```
Key capabilities:
- End-to-end multi-speaker transcription without a separate ASR, diarization,
or alignment pipeline.
- Structured text output with speaker IDs and timestamps.
- Long-audio transcription for meetings, interviews, podcasts, and customer
support recordings.
- Prompt-based hotwords for names, organizations, product names, project
codes, and domain vocabulary.
## Prerequisites
Install vLLM with audio dependencies.
```bash
uv venv
source .venv/bin/activate
uv pip install -U "vllm[audio]" --torch-backend auto
```
## Launching with vLLM
```bash
vllm serve OpenMOSS-Team/MOSS-Transcribe-Diarize \
--trust-remote-code \
--served-model-name moss-mtd
```
The model uses custom Hugging Face processor code, so `--trust-remote-code` is
required.
## Client Usage
### OpenAI Transcription API
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
with open("meeting.wav", "rb") as audio_file:
transcription = client.audio.transcriptions.create(
model="moss-mtd",
file=audio_file,
response_format="diarized_json",
temperature=0,
)
print(transcription.text)
for segment in transcription.segments:
print(
f"[{segment.start:.2f} - {segment.end:.2f}] "
f"{segment.speaker}: {segment.text}"
)
```
### cURL
```bash
curl http://localhost:8000/v1/audio/transcriptions \
-F "model=moss-mtd" \
-F "file=@meeting.wav" \
-F "response_format=diarized_json" \
-F "temperature=0"
```
## Output Format
With `response_format=diarized_json`, vLLM returns an OpenAI-compatible
response with structured speaker segments:
```json
{
"task": "transcribe",
"duration": 5.1,
"text": "Let's review the deployment plan. The vLLM server is already running.",
"segments": [
{
"type": "transcript.text.segment",
"id": "seg_0",
"start": 0.41,
"end": 2.8,
"text": "Let's review the deployment plan.",
"speaker": "S01"
},
{
"type": "transcript.text.segment",
"id": "seg_1",
"start": 3.12,
"end": 5.1,
"text": "The vLLM server is already running.",
"speaker": "S02"
}
],
"usage": {
"type": "duration",
"seconds": 6
}
}
```
Speaker IDs such as `S01`, `S02`, and `S03` are generated by the model and are
intended to remain consistent within the audio.
Use `response_format=json` if you want the model's native timestamped
transcript instead.
```text
[0.41][S01] Let's review the deployment plan. [2.80]
[3.12][S02] The vLLM server is already running. [5.10]
```
## References
- [Model card](https://huggingface.co/OpenMOSS-Team/MOSS-Transcribe-Diarize)
- [GitHub repository](https://github.com/OpenMOSS/MOSS-Transcribe-Diarize)
- [vLLM audio support](https://docs.vllm.ai/en/latest/features/multimodal_inputs.html)
recipes/models/PaddlePaddle/PaddleOCR-VL-1.5.yaml
+204
View File
@@ -0,0 +1,204 @@
meta:
title: "PaddleOCR-VL-1.5"
slug: "paddleocr-vl-1.5"
provider: "PaddlePaddle"
description: "PaddleOCR-VL-1.5 (0.9B) — next-gen compact VLM for document parsing; adds text spotting, seal recognition, and Tibetan/Bengali"
date_added: 2026-05-11
date_updated: 2026-05-11
tasks:
- multimodal
related_recipes:
- "PaddlePaddle/PaddleOCR-VL"
model:
model_id: "PaddlePaddle/PaddleOCR-VL-1.5"
min_vllm_version: "0.11.1"
architecture: dense
parameter_count: "0.9B"
active_parameters: "0.9B"
context_length: 131072
base_args:
- "--trust-remote-code"
- "--max-num-batched-tokens"
- "16384"
- "--no-enable-prefix-caching"
- "--mm-processor-cache-gb"
- "0"
base_env: {}
dependencies:
- note: "PaddlePaddle runtime (install in a separate venv from vllm to avoid conflicts)"
command: "uv pip install paddlepaddle-gpu==3.2.1 --extra-index-url https://www.paddlepaddle.org.cn/packages/stable/cu126/"
- note: "PaddleOCR document-parsing helpers (1.5 ships under the same paddleocr[doc-parser] extra)"
command: 'uv pip install -U "paddleocr[doc-parser]"'
- note: "Safetensors loader used by the doc-parser path"
command: "uv pip install safetensors"
features:
text_only:
description: "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
args:
- "--language-model-only"
encoder_parallel:
description: "Run the vision encoder in data-parallel mode — avoids TP comm overhead on the small encoder. Mutually exclusive with text_only."
args:
- "--mm-encoder-tp-mode"
- "data"
opt_in_features:
- text_only
- encoder_parallel
variants:
default:
precision: bf16
vram_minimum_gb: 2
description: "BF16 weights — small footprint, runs on most GPUs"
compatible_strategies:
- single_node_tp
- multi_node_tp
kv_offload_support:
offloading_cpu: verified
offloading_fs: verified
hardware_overrides:
amd:
extra_args: []
extra_env:
SAFETENSORS_FAST_GPU: "1"
strategy_overrides: {}
guide: |
## Overview
[PaddleOCR-VL-1.5](https://huggingface.co/PaddlePaddle/PaddleOCR-VL-1.5) is the next
generation of PaddleOCR-VL — same 0.9B architecture (NaViT-style dynamic-resolution
vision encoder + ERNIE-4.5-0.3B LM), with substantial accuracy gains and new tasks:
- **94.5% on OmniDocBench v1.5** (SOTA)
- **Text spotting** — line-level localization + recognition in one pass
- **Seal recognition** — new task with SOTA results
- **Irregular-shape localization** — polygonal detection under skew/warping
- **Multilingual** — adds Tibetan and Bengali
- Cross-page table merging and paragraph-heading recognition
Architecture is identical to PaddleOCR-VL, so the vLLM launch command and feature
flags are the same.
## Prerequisites
- Hardware: 1x GPU (small VRAM footprint)
- vLLM >= 0.11.1 (nightly if not released yet)
- AMD ROCm wheels require Python 3.12, ROCm 7.2.3, and glibc >= 2.35.
### Install vLLM (NVIDIA)
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --pre \
--extra-index-url https://wheels.vllm.ai/nightly \
--extra-index-url https://download.pytorch.org/whl/cu129 \
--index-strategy unsafe-best-match
```
### Install vLLM (AMD ROCm)
```bash
uv venv --python 3.12
source .venv/bin/activate
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm
```
## Launch command
```bash
vllm serve PaddlePaddle/PaddleOCR-VL-1.5 \
--trust-remote-code \
--max-num-batched-tokens 16384 \
--no-enable-prefix-caching \
--mm-processor-cache-gb 0
```
AMD MI300X/MI325X/MI355X:
```bash
export SAFETENSORS_FAST_GPU=1
vllm serve PaddlePaddle/PaddleOCR-VL-1.5 \
--trust-remote-code \
--max-num-batched-tokens 16384 \
--mm-processor-cache-gb 0
```
Tip: OCR workloads don't benefit much from prefix caching or image reuse, so disable
those to avoid hashing/caching overhead.
## Client Usage
Task-specific prompts (note the two new tasks `spotting` and `seal`):
```python
from openai import OpenAI
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600)
TASKS = {
"ocr": "OCR:",
"table": "Table Recognition:",
"formula": "Formula Recognition:",
"chart": "Chart Recognition:",
"spotting": "Spotting:",
"seal": "Seal Recognition:",
}
response = client.chat.completions.create(
model="PaddlePaddle/PaddleOCR-VL-1.5",
messages=[{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": "https://.../receipt.png"}},
{"type": "text", "text": TASKS["spotting"]},
],
}],
temperature=0.0,
)
print(response.choices[0].message.content)
```
## Offline Inference with PP-DocLayoutV2
Use separate venvs for `vllm` and `paddlepaddle` to avoid conflicts. If you see
"The model PaddleOCR-VL-1.5-0.9B does not exist.", add
`--served-model-name PaddleOCR-VL-1.5-0.9B`.
```bash
uv pip install paddlepaddle-gpu==3.2.1 --extra-index-url https://www.paddlepaddle.org.cn/packages/stable/cu126/
uv pip install -U "paddleocr[doc-parser]"
uv pip install safetensors
```
```python
from paddleocr import PaddleOCRVL
pipeline = PaddleOCRVL(
vl_rec_model_name="PaddleOCR-VL-1.5-0.9B",
vl_rec_backend="vllm-server",
vl_rec_server_url="http://localhost:8000/v1",
layout_detection_model_name="PP-DocLayoutV2",
layout_detection_model_dir="/path/to/your/PP-DocLayoutV2/",
)
output = pipeline.predict("https://paddle-model-ecology.bj.bcebos.com/paddlex/imgs/demo_image/paddleocr_vl_demo.png")
for i, res in enumerate(output):
res.save_to_json(save_path=f"output_{i}.json")
res.save_to_markdown(save_path=f"output_{i}.md")
```
## References
- [PaddleOCR-VL-1.5 on Hugging Face](https://huggingface.co/PaddlePaddle/PaddleOCR-VL-1.5)
- [PaddleOCR GitHub](https://github.com/PaddlePaddle/PaddleOCR)
- [vLLM support tracking issue](https://github.com/vllm-project/vllm/issues/33554)
recipes/models/PaddlePaddle/PaddleOCR-VL-1.6.yaml
+187
View File
@@ -0,0 +1,187 @@
meta:
title: "PaddleOCR-VL-1.6"
slug: "paddleocr-vl-1.6"
provider: "PaddlePaddle"
description: "PaddleOCR-VL-1.6 (0.9B) — region-aware data optimization + progressive post-training; new SOTA 96.33% on OmniDocBench v1.6, drop-in replacement for 1.5"
date_added: 2026-07-11
date_updated: 2026-07-08
tasks:
- multimodal
related_recipes:
- "PaddlePaddle/PaddleOCR-VL-1.5"
- "PaddlePaddle/PaddleOCR-VL"
model:
model_id: "PaddlePaddle/PaddleOCR-VL-1.6"
min_vllm_version: "0.11.1"
architecture: dense
parameter_count: "0.9B"
active_parameters: "0.9B"
context_length: 131072
base_args:
- "--trust-remote-code"
- "--max-num-batched-tokens"
- "16384"
- "--no-enable-prefix-caching"
- "--mm-processor-cache-gb"
- "0"
base_env: {}
dependencies:
- note: "PaddlePaddle runtime (install in a separate venv from vllm to avoid conflicts)"
command: "uv pip install paddlepaddle-gpu==3.2.1 --extra-index-url https://www.paddlepaddle.org.cn/packages/stable/cu126/"
brand: NVIDIA
- note: "PaddleOCR document-parsing helpers — 1.6 requires >= 3.6.0 (pipeline_version v1.6)"
command: 'uv pip install -U "paddleocr[doc-parser]>=3.6.0"'
- note: "Safetensors loader used by the doc-parser path"
command: "uv pip install safetensors"
features:
text_only:
description: "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
args:
- "--language-model-only"
encoder_parallel:
description: "Run the vision encoder in data-parallel mode — avoids TP comm overhead on the small encoder. Mutually exclusive with text_only."
args:
- "--mm-encoder-tp-mode"
- "data"
opt_in_features:
- text_only
- encoder_parallel
variants:
default:
precision: bf16
vram_minimum_gb: 2
description: "BF16 weights — small footprint, runs on most GPUs"
compatible_strategies:
- single_node_tp
- multi_node_tp
kv_offload_support:
offloading_cpu: verified
offloading_fs: verified
hardware_overrides:
amd:
extra_args: []
extra_env:
SAFETENSORS_FAST_GPU: "1"
strategy_overrides: {}
guide: |
## Overview
[PaddleOCR-VL-1.6](https://huggingface.co/PaddlePaddle/PaddleOCR-VL-1.6) upgrades
PaddleOCR-VL-1.5 with a region-aware data optimization framework (targeted enhancement
of weak regions found in the previous model) and a progressive post-training recipe
based on curated data selection and reinforcement learning:
- **96.33% on OmniDocBench v1.6** (new SOTA), plus new records on OmniDocBench v1.5
and Real5-OmniDocBench (scanning, warping, screen-photography, illumination, skew)
- Major gains in table, Chinese ancient document, and Chinese rare character recognition
- Improved seal/stamp recognition, text spotting, and chart recognition
The architecture is **fully compatible with PaddleOCR-VL-1.5** (same 0.9B NaViT-style
dynamic-resolution vision encoder + ERNIE-4.5-0.3B LM), so migration is plug-and-play:
the vLLM launch command and feature flags are unchanged.
## Prerequisites
- Hardware: 1x GPU (small VRAM footprint)
- vLLM >= 0.11.1
### Install vLLM
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm
```
## Launch command
```bash
vllm serve PaddlePaddle/PaddleOCR-VL-1.6 \
--trust-remote-code \
--max-num-batched-tokens 16384 \
--no-enable-prefix-caching \
--mm-processor-cache-gb 0
```
Tip: OCR workloads don't benefit much from prefix caching or image reuse, so disable
those to avoid hashing/caching overhead.
## Client Usage
Task-specific prompts (same task set as 1.5, including `spotting` and `seal`):
```python
from openai import OpenAI
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600)
TASKS = {
"ocr": "OCR:",
"table": "Table Recognition:",
"formula": "Formula Recognition:",
"chart": "Chart Recognition:",
"spotting": "Spotting:",
"seal": "Seal Recognition:",
}
response = client.chat.completions.create(
model="PaddlePaddle/PaddleOCR-VL-1.6",
messages=[{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": "https://.../receipt.png"}},
{"type": "text", "text": TASKS["spotting"]},
],
}],
temperature=0.0,
)
print(response.choices[0].message.content)
```
## Offline Inference with PP-DocLayoutV3
Use separate venvs for `vllm` and `paddlepaddle` to avoid conflicts. The 1.6 pipeline
requires `paddleocr[doc-parser] >= 3.6.0` and `pipeline_version="v1.6"`, which
automatically uses [PP-DocLayoutV3](https://huggingface.co/PaddlePaddle/PP-DocLayoutV3)
for layout detection (PP-DocLayoutV2 belongs to the v1 pipeline). Pass
`vl_rec_api_model_name="PaddlePaddle/PaddleOCR-VL-1.6"` so the client matches the
model name the vLLM server exposes — otherwise it defaults to
`PaddleOCR-VL-1.6-0.9B` and fails with "The model PaddleOCR-VL-1.6-0.9B does not exist."
```bash
uv pip install paddlepaddle-gpu==3.2.1 --extra-index-url https://www.paddlepaddle.org.cn/packages/stable/cu126/
uv pip install -U "paddleocr[doc-parser]>=3.6.0"
uv pip install safetensors
```
```python
from paddleocr import PaddleOCRVL
pipeline = PaddleOCRVL(
pipeline_version="v1.6",
vl_rec_backend="vllm-server",
vl_rec_server_url="http://localhost:8000/v1",
vl_rec_api_model_name="PaddlePaddle/PaddleOCR-VL-1.6",
)
output = pipeline.predict("https://paddle-model-ecology.bj.bcebos.com/paddlex/imgs/demo_image/paddleocr_vl_demo.png")
for i, res in enumerate(output):
res.save_to_json(save_path=f"output_{i}.json")
res.save_to_markdown(save_path=f"output_{i}.md")
```
## References
- [PaddleOCR-VL-1.6 on Hugging Face](https://huggingface.co/PaddlePaddle/PaddleOCR-VL-1.6)
- [Technical report (arXiv)](https://arxiv.org/pdf/2606.03264)
- [PaddleOCR GitHub](https://github.com/PaddlePaddle/PaddleOCR)
- [vLLM support PR](https://github.com/vllm-project/vllm/pull/27758)
recipes/models/PaddlePaddle/PaddleOCR-VL.yaml
+190
View File
@@ -0,0 +1,190 @@
meta:
title: "PaddleOCR-VL"
slug: "paddleocr-vl"
provider: "PaddlePaddle"
description: "PaddleOCR-VL (0.9B) — compact vision-language model for document parsing, OCR, tables, formulas, charts"
date_added: 2025-11-04
date_updated: 2026-04-17
tasks:
- multimodal
related_recipes: []
model:
model_id: "PaddlePaddle/PaddleOCR-VL"
min_vllm_version: "0.11.1"
architecture: dense
parameter_count: "0.9B"
active_parameters: "0.9B"
context_length: 131072
base_args:
- "--trust-remote-code"
- "--max-num-batched-tokens"
- "16384"
- "--no-enable-prefix-caching"
- "--mm-processor-cache-gb"
- "0"
base_env: {}
dependencies:
- note: "PaddlePaddle runtime (install in a separate venv from vllm to avoid conflicts)"
command: "uv pip install paddlepaddle-gpu==3.2.1 --extra-index-url https://www.paddlepaddle.org.cn/packages/stable/cu126/"
- note: "PaddleOCR document-parsing helpers"
command: 'uv pip install -U "paddleocr[doc-parser]"'
- note: "Safetensors loader used by the doc-parser path"
command: "uv pip install safetensors"
features:
text_only:
description: "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
args:
- "--language-model-only"
encoder_parallel:
description: "Run the vision encoder in data-parallel mode — avoids TP comm overhead on the small encoder. Mutually exclusive with text_only."
args:
- "--mm-encoder-tp-mode"
- "data"
opt_in_features:
- text_only
- encoder_parallel
variants:
default:
precision: bf16
vram_minimum_gb: 2
description: "BF16 weights — small footprint, runs on most GPUs"
compatible_strategies:
- single_node_tp
- multi_node_tp
kv_offload_support:
offloading_cpu: verified
offloading_fs: verified
hardware_overrides:
amd:
extra_args: []
extra_env:
SAFETENSORS_FAST_GPU: "1"
strategy_overrides: {}
guide: |
## Overview
[PaddleOCR-VL](https://huggingface.co/PaddlePaddle/PaddleOCR-VL) is a SOTA resource-efficient
model for document parsing. Its core (PaddleOCR-VL-0.9B) combines a NaViT-style dynamic
resolution visual encoder with an ERNIE-4.5-0.3B language model, optimized for OCR,
tables, formulas, and chart recognition.
## Prerequisites
- Hardware: 1x GPU (small VRAM footprint)
- vLLM >= 0.11.1 (nightly if not released yet)
- AMD ROCm wheels require Python 3.12, ROCm 7.2.3, and glibc >= 2.35.
### Install vLLM (NVIDIA)
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --pre \
--extra-index-url https://wheels.vllm.ai/nightly \
--extra-index-url https://download.pytorch.org/whl/cu129 \
--index-strategy unsafe-best-match
```
### Install vLLM (AMD ROCm)
```bash
uv venv --python 3.12
source .venv/bin/activate
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm
```
## Launch command
```bash
vllm serve PaddlePaddle/PaddleOCR-VL \
--trust-remote-code \
--max-num-batched-tokens 16384 \
--no-enable-prefix-caching \
--mm-processor-cache-gb 0
```
AMD MI300X/MI325X/MI355X:
```bash
export SAFETENSORS_FAST_GPU=1
vllm serve PaddlePaddle/PaddleOCR-VL \
--trust-remote-code \
--max-num-batched-tokens 16384 \
--no-enable-prefix-caching \
--mm-processor-cache-gb 0
```
Tip: OCR workloads don't benefit much from prefix caching or image reuse, so disable
those to avoid hashing/caching overhead.
## Client Usage
Task-specific prompts:
```python
from openai import OpenAI
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600)
TASKS = {
"ocr": "OCR:",
"table": "Table Recognition:",
"formula": "Formula Recognition:",
"chart": "Chart Recognition:",
}
response = client.chat.completions.create(
model="PaddlePaddle/PaddleOCR-VL",
messages=[{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": "https://.../receipt.png"}},
{"type": "text", "text": TASKS["ocr"]},
],
}],
temperature=0.0,
)
print(response.choices[0].message.content)
```
## Offline Inference with PP-DocLayoutV2
Use separate venvs for `vllm` and `paddlepaddle` to avoid conflicts. If you see
"The model PaddleOCR-VL-0.9B does not exist.", add `--served-model-name PaddleOCR-VL-0.9B`.
```bash
uv pip install paddlepaddle-gpu==3.2.1 --extra-index-url https://www.paddlepaddle.org.cn/packages/stable/cu126/
uv pip install -U "paddleocr[doc-parser]"
uv pip install safetensors
```
```python
from paddleocr import PaddleOCRVL
pipeline = PaddleOCRVL(
vl_rec_backend="vllm-server",
vl_rec_server_url="http://localhost:8000/v1",
layout_detection_model_name="PP-DocLayoutV2",
layout_detection_model_dir="/path/to/your/PP-DocLayoutV2/",
)
output = pipeline.predict("https://paddle-model-ecology.bj.bcebos.com/paddlex/imgs/demo_image/paddleocr_vl_demo.png")
for i, res in enumerate(output):
res.save_to_json(save_path=f"output_{i}.json")
res.save_to_markdown(save_path=f"output_{i}.md")
```
## References
- [PaddleOCR-VL on Hugging Face](https://huggingface.co/PaddlePaddle/PaddleOCR-VL)
- [PaddleOCR GitHub](https://github.com/PaddlePaddle/PaddleOCR)
recipes/models/Qwen/QwQ-32B.yaml
+161
View File
@@ -0,0 +1,161 @@
meta:
title: "QwQ-32B"
slug: "qwq-32b"
provider: "Qwen"
description: "Qwen's 32B dense reasoning model, validated for Intel Xeon 6 CPU execution."
date_added: 2026-08-17
date_updated: 2026-09-12
tasks:
- text
performance_headline: "Validated for Intel Xeon 6 CPU execution"
related_recipes:
- "Qwen/Qwen3-32B"
hardware:
xeon6: verified
model:
model_id: "Qwen/QwQ-32B"
min_vllm_version: "0.7.3"
architecture: dense
parameter_count: "32B"
active_parameters: "32B"
context_length: 40960
base_args: []
base_env: {}
features:
tool_calling:
description: "Enable automatic tool choice with the Hermes parser"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "hermes"
reasoning:
description: "Extract QwQ reasoning content with the DeepSeek-R1 reasoning parser"
args:
- "--reasoning-parser"
- "deepseek_r1"
long_context:
description: "YaRN RoPE scaling to the full 131,072-token context, which the model card requires for prompts over 8,192 tokens. Static YaRN holds the scaling factor constant regardless of input length, so leave this off unless you actually serve long prompts."
args:
- "--hf-overrides"
- '{"max_position_embeddings":131072,"rope_parameters":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":32768}}'
- "--max-model-len"
- "131072"
opt_in_features:
- long_context
variants:
default:
precision: bf16
vram_minimum_gb: 77
description: "Full precision BF16"
supported_hardware:
- xeon6
w8a8_int8:
model_id: "RedHatAI/QwQ-32B-quantized.w8a8"
precision: int8
vram_minimum_gb: 39
description: "Red Hat AI INT8 W8A8 compressed-tensors checkpoint"
supported_hardware:
- xeon6
awq:
model_id: "Qwen/QwQ-32B-AWQ"
precision: int4
vram_minimum_gb: 20
description: "Qwen official AWQ 4-bit checkpoint"
supported_hardware:
- xeon6
extra_args:
- "--quantization"
- "awq"
compatible_strategies:
- single_node_tp
- multi_node_tp
kv_offload_support:
offloading_cpu: verified
offloading_fs: verified
hardware_overrides: {}
strategy_overrides: {}
guide: |
## Overview
QwQ-32B is Qwen's 32B dense reasoning model. The CPU model validation table
marks it supported on Intel Xeon 6.
## Prerequisites
- Hardware: Intel Xeon 6 CPUs
- vLLM CPU >= 0.8.5 for this validated Xeon 6 configuration
### pip (Intel Xeon 6 CPUs)
For Intel and AMD x86 CPUs, follow the
[CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels)
installation instructions.
### Docker (Intel Xeon 6 CPUs)
```bash
docker pull vllm/vllm-openai-cpu:latest-x86_64
```
## Intel Xeon 6
```bash
vllm serve Qwen/QwQ-32B \
--tensor-parallel-size 1 \
--enable-auto-tool-choice \
--tool-call-parser hermes \
--reasoning-parser deepseek_r1
```
Docker (the image entrypoint is `vllm serve`):
```bash
docker run \
--privileged --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai-cpu:latest-x86_64 Qwen/QwQ-32B \
--tensor-parallel-size 1 \
--enable-auto-tool-choice \
--tool-call-parser hermes \
--reasoning-parser deepseek_r1
```
W8A8 INT8:
```bash
vllm serve RedHatAI/QwQ-32B-quantized.w8a8
```
The example uses TP=1. Adjust tensor parallelism based on system topology;
topology-specific CPU binding is intentionally not hard-coded.
## Runtime and Platform Tuning
The following settings are intentionally not prescribed as portable CPU model
defaults because they depend on the workload, hardware, or runtime environment:
- `--max-num-batched-tokens`: scheduler/throughput tuning.
- `--max-num-seqs`: concurrency and scheduler-capacity tuning.
- `--gpu-memory-utilization`: platform memory-budget tuning.
- `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning.
- `VLLM_ENGINE_ITERATION_TIMEOUT_S`: operational runtime timeout.
These settings may still appear in validated hardware-specific overrides. Tune
them at deployment time based on platform resources, workload shape, and
latency/throughput goals.
## References
- [Model card](https://huggingface.co/Qwen/QwQ-32B)
- [W8A8 variant](https://huggingface.co/RedHatAI/QwQ-32B-quantized.w8a8)
- [vLLM reasoning outputs](https://docs.vllm.ai/en/latest/features/reasoning_outputs.html)
- [vLLM tool calling](https://docs.vllm.ai/en/latest/features/tool_calling.html)
recipes/models/Qwen/Qwen2.5-32B.yaml
+128
View File
@@ -0,0 +1,128 @@
meta:
title: "Qwen2.5-32B"
slug: "qwen2.5-32b"
provider: "Qwen"
description: "Qwen2.5 32B dense base (pretrained) language model for text completion — verified on TPU v6e (Trillium)."
date_added: 2026-06-05
date_updated: 2026-06-04
tasks:
- text
performance_headline: "Verified on TPU v6e (Trillium) with BF16, TP=4 on a 2x2 slice"
related_recipes: []
hardware:
trillium: verified
model:
model_id: "Qwen/Qwen2.5-32B"
min_vllm_version: "0.6.2"
architecture: dense
parameter_count: "32B"
active_parameters: "32B"
context_length: 131072
base_args: []
base_env: {}
features: {}
opt_in_features: []
variants:
default:
precision: bf16
vram_minimum_gb: 77
description: "Full precision BF16 — TPU v6e 2x2 (4 chips, TP=4) or a single 80GB+ GPU"
fp8:
model_id: "RedHatAI/Qwen2.5-32B-FP8-dynamic"
precision: fp8
vram_minimum_gb: 39
description: "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization"
extra_args:
- "--kv-cache-dtype"
- "fp8"
compatible_strategies:
- single_node_tp
- multi_node_tp
kv_offload_support:
offloading_cpu: verified
offloading_fs: verified
hardware_overrides: {}
strategy_overrides: {}
guide: |
## Overview
Qwen2.5-32B is the 32B dense **base** (pretrained, non-instruct) model in the Qwen2.5 series. It's a text-completion model — use the `/v1/completions` endpoint rather than chat, or pick `Qwen/Qwen2.5-32B-Instruct` if you need a chat-tuned variant. BF16 is the training precision and gives the best accuracy.
## Prerequisites
### NVIDIA
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```
### TPU (Trillium / v6e)
Use the official vLLM TPU image. The 32B model is served on a `v6e-4` slice (`--topology 2x2`, 4 chips, TP=4).
```bash
export HF_HOME=/dev/shm
export HF_TOKEN=<your HF token>
```
## Deployment Configurations
### TPU v6e (Trillium, 2x2 slice, TP=4)
Verified end-to-end on a 4-chip v6e slice.
```bash
export HF_HOME=/dev/shm
export HF_TOKEN=<your HF token>
export TP=4
export MAX_MODEL_LEN=4096
vllm serve Qwen/Qwen2.5-32B \
--seed 42 \
--gpu-memory-utilization 0.98 \
--max-num-batched-tokens 2048 \
--max-num-seqs 128 \
--tensor-parallel-size $TP \
--max-model-len $MAX_MODEL_LEN
```
### Single GPU / node (BF16)
```bash
vllm serve Qwen/Qwen2.5-32B \
--host 0.0.0.0 \
--port 8000 \
--tensor-parallel-size 2
```
## Configuration Tips
- `--max-model-len 4096` keeps memory in check on a 4-chip v6e slice; native context is 128K — raise it if you have headroom.
- vLLM has been V1-engine-only for a long time, so no `VLLM_USE_V1=1` export is needed even though the upstream TPU recipe still shows it.
- vLLM uses 90% of device memory by default; the TPU recipe pushes `--gpu-memory-utilization 0.98` to maximize KV cache.
- `--max-num-batched-tokens` and `--max-num-seqs` tune batch throughput vs latency.
## Benchmarking
Launch the server with `--no-enable-prefix-caching` for consistent measurements.
```bash
vllm bench serve \
--host 0.0.0.0 \
--port 8000 \
--model Qwen/Qwen2.5-32B \
--dataset-name random \
--random-input-len 1024 \
--random-output-len 1024 \
--num-prompts 128
```
## References
- [Model card](https://huggingface.co/Qwen/Qwen2.5-32B)
- [Qwen2.5-32B on Trillium (v6e)](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Qwen2.5-32B)
recipes/models/Qwen/Qwen2.5-VL-72B-Instruct.yaml
+205
View File
@@ -0,0 +1,205 @@
meta:
title: "Qwen2.5-VL-72B-Instruct"
slug: "qwen2.5-vl-72b-instruct"
provider: "Qwen"
description: "Qwen2.5-VL dense vision-language model (72B) for high-quality image and video understanding."
date_added: 2025-08-22
date_updated: 2026-04-17
tasks:
- multimodal
- text
performance_headline: "Verified on 4x A100 and 4x MI300X/MI325X/MI355X with BF16"
related_recipes:
- "Qwen/Qwen2.5-VL-7B-Instruct"
hardware:
mi300x: verified
mi325x: verified
mi355x: verified
model:
model_id: "Qwen/Qwen2.5-VL-72B-Instruct"
min_vllm_version: "0.10.2"
architecture: dense
parameter_count: "72B"
active_parameters: "72B"
context_length: 128000
base_args: []
base_env: {}
features:
text_only:
description: "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
args:
- "--language-model-only"
encoder_parallel:
description: "Run the vision encoder in data-parallel mode — avoids TP comm overhead on the small encoder. Mutually exclusive with text_only."
args:
- "--mm-encoder-tp-mode"
- "data"
opt_in_features:
- text_only
variants:
default:
precision: bf16
vram_minimum_gb: 173
description: "Full precision BF16 — 4x A100 80GB or 4x MI300X/MI325X/MI355X"
awq:
model_id: "Qwen/Qwen2.5-VL-72B-Instruct-AWQ"
precision: int4
vram_minimum_gb: 43
description: "AWQ 4-bit quantized weights"
extra_args:
- "--quantization"
- "awq"
fp8:
model_id: "RedHatAI/Qwen2.5-VL-72B-Instruct-FP8-dynamic"
precision: fp8
vram_minimum_gb: 87
description: "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization"
extra_args:
- "--kv-cache-dtype"
- "fp8"
compatible_strategies:
- single_node_tp
- multi_node_tp
- multi_node_tp_pp
kv_offload_support:
offloading_cpu: verified
offloading_fs: verified
hardware_overrides:
hopper:
extra_args:
- "--mm-encoder-tp-mode"
- "data"
extra_env: {}
blackwell:
extra_args:
- "--mm-encoder-tp-mode"
- "data"
extra_env: {}
amd:
# PR #125 verified on MI300X only; TP=2 for throughput, TP=8 for latency.
extra_args:
- "--mm-encoder-tp-mode"
- "data"
- "--limit-mm-per-prompt"
- '{"image":2,"video":0}'
extra_env:
VLLM_ROCM_USE_AITER: "1"
strategy_overrides: {}
guide: |
## Overview
This guide describes how to run Qwen2.5-VL series on the targeted accelerated stack. Since BF16 is the commonly used precision for Qwen2.5-VL training, using BF16 in inference ensures the best accuracy.
## Prerequisites
### NVIDIA
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```
### AMD ROCm (MI300X, MI325X, MI355X)
> Note: The vLLM wheel for ROCm requires Python 3.12, ROCm 7.0, and glibc >= 2.35. Use the Docker flow if your environment is incompatible.
```bash
uv venv
source .venv/bin/activate
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm
```
### TPU Deployment
TPU support has been verified for the smaller [Qwen2.5-VL-7B-Instruct](/Qwen/Qwen2.5-VL-7B-Instruct) on a single v6e chip — the 72B is not yet validated on Trillium. See the [upstream TPU recipe](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Qwen2.5-VL) (7B).
## Deployment Configurations
### 4xA100 (BF16, TP=4)
```bash
export CUDA_VISIBLE_DEVICES=0,1,2,3
vllm serve Qwen/Qwen2.5-VL-72B-Instruct \
--host 0.0.0.0 \
--port 8000 \
--tensor-parallel-size 4 \
--mm-encoder-tp-mode data \
--limit-mm-per-prompt '{"image":2,"video":0}'
```
### 4xMI300X/MI325X/MI355X (BF16, TP=4)
```bash
export CUDA_VISIBLE_DEVICES=0,1,2,3
export VLLM_ROCM_USE_AITER=1
vllm serve Qwen/Qwen2.5-VL-72B-Instruct \
--host 0.0.0.0 \
--port 8000 \
--tensor-parallel-size 4 \
--mm-encoder-tp-mode data \
--limit-mm-per-prompt '{"image":2,"video":0}'
```
### Qwen2.5-VL-7B-Instruct (DP=4)
For medium-size 7B model, data parallelism works better.
```bash
export CUDA_VISIBLE_DEVICES=0,1,2,3
vllm serve Qwen/Qwen2.5-VL-7B-Instruct \
--host 0.0.0.0 \
--port 8000 \
--data-parallel-size 4 \
--limit-mm-per-prompt '{"image":2,"video":0}'
```
## Configuration Tips
- `--max-model-len 65536` is usually good for most scenarios (native context is 128K).
- For A100-80GB devices, TP must be >= 2 to avoid OOM.
- `--limit-mm-per-prompt` caps incoming multimodal requests.
- `--mm-encoder-tp-mode data` deploys the small ViT encoder in DP fashion (ViT ≈ 675M vs 72B LM).
- vLLM uses 90% of GPU memory by default; set `--gpu-memory-utilization=0.95` to maximize KV cache.
## Benchmarking
Launch the server with `--no-enable-prefix-caching` to get consistent measurements.
### VisionArena-Chat
```bash
vllm bench serve \
--host 0.0.0.0 \
--port 8000 \
--backend openai-chat \
--endpoint /v1/chat/completions \
--model Qwen/Qwen2.5-VL-72B-Instruct \
--dataset-name hf \
--dataset-path lmarena-ai/VisionArena-Chat \
--num-prompts 128
```
### Random Synthetic
```bash
vllm bench serve \
--host 0.0.0.0 \
--port 8000 \
--model Qwen/Qwen2.5-VL-72B-Instruct \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 128
```
Workload mixes:
- Prompt-heavy: 8000 in / 1000 out
- Decode-heavy: 1000 in / 8000 out
- Balanced: 1000 in / 1000 out
## References
- [Model card](https://huggingface.co/Qwen/Qwen2.5-VL-72B-Instruct)
- [vLLM multimodal inputs guide](https://docs.vllm.ai/en/latest/features/multimodal_inputs.html)
recipes/models/Qwen/Qwen2.5-VL-7B-Instruct.yaml
+236
View File
@@ -0,0 +1,236 @@
meta:
title: "Qwen2.5-VL-7B-Instruct"
slug: "qwen2.5-vl-7b-instruct"
provider: "Qwen"
description: "Qwen2.5-VL dense vision-language model (7B) for image and video understanding — fits on a single TPU v6e chip, one GPU, or Intel Xeon 6 CPU."
date_added: 2025-08-22
date_updated: 2026-09-12
tasks:
- multimodal
- text
performance_headline: "Verified on TPU v6e (Trillium) and Intel Xeon 6 CPU with BF16"
related_recipes:
- "Qwen/Qwen2.5-VL-72B-Instruct"
hardware:
trillium: verified
xeon6: verified
model:
model_id: "Qwen/Qwen2.5-VL-7B-Instruct"
min_vllm_version: "0.10.2"
architecture: dense
parameter_count: "7B"
active_parameters: "7B"
context_length: 128000
base_args: []
base_env: {}
features:
text_only:
description: "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
args:
- "--language-model-only"
encoder_parallel:
description: "Run the vision encoder in data-parallel mode — avoids TP comm overhead on the small encoder. Mutually exclusive with text_only."
args:
- "--mm-encoder-tp-mode"
- "data"
opt_in_features:
- text_only
variants:
default:
precision: bf16
vram_minimum_gb: 17
description: "Full precision BF16 — fits on a single TPU v6e chip or one 24GB+ GPU"
supported_hardware: [trillium, xeon6]
awq:
model_id: "Qwen/Qwen2.5-VL-7B-Instruct-AWQ"
precision: int4
vram_minimum_gb: 5
description: "AWQ 4-bit quantized weights"
supported_hardware: [trillium]
extra_args:
- "--quantization"
- "awq"
fp8:
model_id: "RedHatAI/Qwen2.5-VL-7B-Instruct-FP8-Dynamic"
precision: fp8
vram_minimum_gb: 9
description: "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization"
supported_hardware: [trillium]
extra_args:
- "--kv-cache-dtype"
- "fp8"
compatible_strategies:
- single_node_tp
- multi_node_tp
kv_offload_support:
offloading_cpu: verified
offloading_fs: verified
hardware_overrides: {}
strategy_overrides: {}
guide: |
## Overview
Qwen2.5-VL-7B-Instruct is the small dense vision-language model in the Qwen2.5-VL series. At 7B it fits comfortably on a single accelerator — one TPU v6e (Trillium) chip, or a single 24GB+ GPU — and is also validated for Intel Xeon 6 CPU serving. BF16 is the precision used in training, so BF16 inference gives the best accuracy.
For the 72B sibling, see [Qwen2.5-VL-72B-Instruct](/Qwen/Qwen2.5-VL-72B-Instruct).
## Prerequisites
### NVIDIA
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```
### Intel Xeon 6 CPUs
- Hardware: Intel Xeon 6 CPUs
- vLLM CPU >= 0.8.5 for this validated Xeon 6 configuration
For Intel and AMD x86 CPUs, follow the
[CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels)
installation instructions.
```bash
docker pull vllm/vllm-openai-cpu:latest-x86_64
```
### TPU (Trillium / v6e)
Use the official vLLM TPU image. The 7B model fits on a single v6e chip (`v6e-1`, `--topology 1x1`).
```bash
export HF_HOME=/dev/shm
export HF_TOKEN=<your HF token>
docker run --rm --privileged --net=host \
--shm-size=16G \
-e HF_HOME -e HF_TOKEN \
vllm/vllm-tpu:latest \
vllm serve ...
```
## Deployment Configurations
### Intel Xeon 6
Choose tensor parallelism based on the system topology; the recipe does not
prescribe a fixed TP/DP layout or hard-code NUMA node IDs.
```bash
vllm serve Qwen/Qwen2.5-VL-7B-Instruct
```
Docker:
```bash
docker run \
--privileged --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai-cpu:latest-x86_64 Qwen/Qwen2.5-VL-7B-Instruct \
--tensor-parallel-size 1 \
--mm-encoder-tp-mode data
```
#### Runtime and Platform Tuning
The following settings are intentionally not prescribed as portable CPU model
defaults because they depend on the workload, hardware, or runtime environment:
- `--max-num-batched-tokens`: scheduler/throughput tuning.
- `--max-num-seqs`: concurrency and scheduler-capacity tuning.
- `--gpu-memory-utilization`: platform memory-budget tuning.
- `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning.
- `VLLM_ENGINE_ITERATION_TIMEOUT_S`: operational runtime timeout.
These settings may still appear in validated hardware-specific overrides. Tune
them at deployment time based on platform resources, workload shape, and
latency/throughput goals.
### TPU v6e (Trillium, single chip)
Verified end-to-end on a single v6e chip.
```bash
export HF_HOME=/dev/shm
export HF_TOKEN=<your HF token>
vllm serve Qwen/Qwen2.5-VL-7B-Instruct \
--tensor-parallel-size 1 \
--pipeline-parallel-size 1 \
--dtype bfloat16 \
--gpu-memory-utilization 0.98 \
--max-model-len 16384 \
--limit-mm-per-prompt '{"image":10,"video":0}' \
--mm-processor-kwargs '{"max_pixels":1003520}' \
--guided-decoding-backend xgrammar \
--disable-chunked-mm-input
```
### Single GPU (BF16, TP=1)
```bash
export CUDA_VISIBLE_DEVICES=0
vllm serve Qwen/Qwen2.5-VL-7B-Instruct \
--host 0.0.0.0 \
--port 8000 \
--limit-mm-per-prompt '{"image":2,"video":0}'
```
### Multi-GPU (DP=4)
For higher throughput across a node, data parallelism works better than TP for a model this size.
```bash
export CUDA_VISIBLE_DEVICES=0,1,2,3
vllm serve Qwen/Qwen2.5-VL-7B-Instruct \
--host 0.0.0.0 \
--port 8000 \
--data-parallel-size 4 \
--limit-mm-per-prompt '{"image":2,"video":0}'
```
## Configuration Tips
- `--max-model-len 16384` is a safe default on a single v6e chip (native context is 128K); raise it if you have headroom.
- `--limit-mm-per-prompt` caps incoming multimodal requests per prompt.
- `--mm-processor-kwargs '{"max_pixels":1003520}'` bounds the per-image resolution to control encoder cost.
- vLLM uses 90% of device memory by default; on TPU the recipe pushes `--gpu-memory-utilization 0.98` to maximize KV cache.
- `--disable-chunked-mm-input` is recommended on TPU for stable multimodal batching.
## Benchmarking
Launch the server with `--no-enable-prefix-caching` to get consistent measurements.
### VisionArena-Chat
```bash
vllm bench serve \
--host 0.0.0.0 \
--port 8000 \
--backend openai-chat \
--endpoint /v1/chat/completions \
--model Qwen/Qwen2.5-VL-7B-Instruct \
--dataset-name hf \
--dataset-path lmarena-ai/VisionArena-Chat \
--num-prompts 128
```
### Random Synthetic
```bash
vllm bench serve \
--host 0.0.0.0 \
--port 8000 \
--model Qwen/Qwen2.5-VL-7B-Instruct \
--dataset-name random \
--random-input-len 8000 \
--random-output-len 1000 \
--num-prompts 128
```
## References
- [Model card](https://huggingface.co/Qwen/Qwen2.5-VL-7B-Instruct)
- [Qwen2.5-VL on Trillium (v6e)](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Qwen2.5-VL)
- [vLLM multimodal inputs guide](https://docs.vllm.ai/en/latest/features/multimodal_inputs.html)
recipes/models/Qwen/Qwen3-1.7B.yaml
+129
View File
@@ -0,0 +1,129 @@
meta:
title: "Qwen3-1.7B"
slug: "qwen3-1.7b"
provider: "Qwen"
description: "Qwen3 1.7B dense model with hybrid thinking/non-thinking modes, validated for BF16 serving on Intel Xeon 6."
date_added: 2026-08-14
date_updated: 2026-09-12
tasks:
- text
performance_headline: "Validated on Intel Xeon 6 (GNR) with BF16 serving"
related_recipes:
- "Qwen/Qwen3-4B"
hardware:
xeon6: verified
model:
model_id: "Qwen/Qwen3-1.7B"
min_vllm_version: "0.8.5"
architecture: dense
parameter_count: "1.7B"
active_parameters: "1.7B"
context_length: 40960
base_args: []
base_env: {}
features:
tool_calling:
description: "Enable automatic tool choice with the Hermes parser"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "hermes"
reasoning:
description: "Enable Qwen3 thinking mode with the Qwen3 reasoning parser"
args:
- "--reasoning-parser"
- "qwen3"
opt_in_features: []
variants:
default:
precision: bf16
vram_minimum_gb: 5
description: "Full precision BF16"
supported_hardware: [xeon6]
compatible_strategies:
- single_node_tp
- multi_node_tp
hardware_overrides:
cpu:
extra_args:
- "--max-num-batched-tokens"
- "16384"
- "--gpu-memory-utilization"
- "0.8"
strategy_overrides: {}
guide: |
## Overview
Qwen3-1.7B is a compact dense Qwen3 model with hybrid thinking and non-thinking
modes. This recipe includes an Intel Xeon 6 CPU configuration validated with
BF16 serving.
## Prerequisites
- Hardware: 1x Xeon 6 CPUs
- vLLM >= 0.8.5
### pip (Intel Xeon 6 CPUs)
For Intel and AMD x86 CPUs, follow the
[CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels)
installation instructions.
### Docker (Intel Xeon 6 CPUs)
```bash
docker pull vllm/vllm-openai-cpu:latest-x86_64
```
## Intel Xeon 6
Choose the tensor parallel size based on the system topology and deployment
requirements. Add `--tensor-parallel-size <N>` when needed; the recipe does
not prescribe a Xeon 6 TP value or hard-code NUMA node IDs.
```bash
vllm serve Qwen/Qwen3-1.7B \
--max-num-batched-tokens 16384 \
--gpu-memory-utilization 0.8
```
Docker:
```bash
docker run \
--privileged --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai-cpu:latest-x86_64 Qwen/Qwen3-1.7B \
--tensor-parallel-size 1 \
--max-num-batched-tokens 16384 \
--gpu-memory-utilization 0.8 \
--enable-auto-tool-choice \
--tool-call-parser hermes \
--reasoning-parser qwen3
```
## Client Usage
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
response = client.chat.completions.create(
model="Qwen/Qwen3-1.7B",
messages=[{"role": "user", "content": "Explain tensor parallelism briefly."}],
)
print(response.choices[0].message.content)
```
## References
- [Model card](https://huggingface.co/Qwen/Qwen3-1.7B)
- [vLLM CPU installation](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/)
recipes/models/Qwen/Qwen3-14B.yaml
+152
View File
@@ -0,0 +1,152 @@
meta:
title: "Qwen3-14B"
slug: "qwen3-14b"
provider: "Qwen"
description: "Qwen3 14B dense model with hybrid thinking/non-thinking modes, validated for Intel Xeon 6 CPU serving."
date_added: 2026-08-17
date_updated: 2026-09-12
tasks:
- text
performance_headline: "Validated on Intel Xeon 6 CPU with BF16 serving"
related_recipes:
- "Qwen/Qwen3-4B"
- "Qwen/Qwen3-32B"
hardware:
xeon6: verified
model:
model_id: "Qwen/Qwen3-14B"
min_vllm_version: "0.8.5"
architecture: dense
parameter_count: "14B"
active_parameters: "14B"
context_length: 40960
base_args: []
base_env: {}
features:
tool_calling:
description: "Enable automatic tool choice with the Hermes parser"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "hermes"
reasoning:
description: "Enable Qwen3 thinking mode with the Qwen3 reasoning parser"
args:
- "--reasoning-parser"
- "qwen3"
long_context:
description: "YaRN RoPE scaling to the model card's validated 131,072-token context. Static YaRN holds the scaling factor constant regardless of input length, so leave this off unless you actually serve long prompts — it can cost quality below 32K."
args:
- "--hf-overrides"
- '{"max_position_embeddings":131072,"rope_parameters":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":32768}}'
- "--max-model-len"
- "131072"
opt_in_features:
- reasoning
- long_context
variants:
default:
precision: bf16
vram_minimum_gb: 34
description: "Full precision BF16"
supported_hardware:
- xeon6
fp8:
model_id: "Qwen/Qwen3-14B-FP8"
precision: fp8
vram_minimum_gb: 17
description: "Qwen official FP8 checkpoint"
awq:
model_id: "Qwen/Qwen3-14B-AWQ"
precision: int4
vram_minimum_gb: 10
description: "Qwen official AWQ 4-bit checkpoint"
supported_hardware:
- xeon6
extra_args:
- "--quantization"
- "awq"
compatible_strategies:
- single_node_tp
- multi_node_tp
kv_offload_support:
offloading_cpu: verified
offloading_fs: verified
hardware_overrides: {}
strategy_overrides: {}
guide: |
## Overview
Qwen3-14B is a Qwen3 dense model with hybrid thinking and non-thinking modes.
Intel Xeon 6 CPU serving was validated with BF16.
## Prerequisites
- Hardware: Intel Xeon 6 CPUs
- vLLM >= 0.8.5
### pip (Intel Xeon 6 CPUs)
Follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels)
installation instructions.
### Docker (Intel Xeon 6 CPUs)
```bash
docker pull vllm/vllm-openai-cpu:latest-x86_64
```
## Intel Xeon 6
Choose tensor parallelism based on the system topology; the recipe does not
prescribe a fixed TP/DP layout or hard-code NUMA node IDs.
```bash
vllm serve Qwen/Qwen3-14B
```
Docker:
```bash
docker run \
--privileged --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai-cpu:latest-x86_64 Qwen/Qwen3-14B \
--tensor-parallel-size 1 \
--enable-auto-tool-choice \
--tool-call-parser hermes
```
## Configuration Notes
- TP/DP values and explicit CPU bindings are intentionally not prescribed.
- The `qwen3` reasoning parser is available in vLLM 0.9.0 and later.
## Runtime and Platform Tuning
The following settings are intentionally not prescribed as portable CPU model
defaults because they depend on the workload, hardware, or runtime environment:
- `--max-num-batched-tokens`: scheduler/throughput tuning.
- `--max-num-seqs`: concurrency and scheduler-capacity tuning.
- `--gpu-memory-utilization`: platform memory-budget tuning.
- `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning.
- `VLLM_ENGINE_ITERATION_TIMEOUT_S`: operational runtime timeout.
These settings may still appear in validated hardware-specific overrides. Tune
them at deployment time based on platform resources, workload shape, and
latency/throughput goals.
## References
- [Model card](https://huggingface.co/Qwen/Qwen3-14B)
- [vLLM CPU installation](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/)
recipes/models/Qwen/Qwen3-235B-A22B-Instruct-2507.yaml
+181
View File
@@ -0,0 +1,181 @@
meta:
title: "Qwen3-235B-A22B-Instruct"
slug: "qwen3-235b-a22b-instruct-2507"
provider: "Qwen"
description: "Flagship Qwen3 MoE instruct model with 235B total and 22B active parameters, tuned for high-quality text generation."
date_added: 2025-10-15
date_updated: 2026-04-17
tasks:
- text
performance_headline: "Verified on 4x/8x H200, MI300X/MI325X/MI355X nodes (BF16 and FP8)"
related_recipes: []
hardware:
h200: verified
mi300x: verified
mi325x: verified
mi355x: verified
model:
model_id: "Qwen/Qwen3-235B-A22B-Instruct-2507"
min_vllm_version: "0.10.0"
architecture: moe
parameter_count: "235B"
active_parameters: "22B"
context_length: 262144
base_args:
- "--trust-remote-code"
base_env: {}
features:
tool_calling:
description: "Enable automatic tool choice with Hermes-compatible parser"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "hermes"
spec_decoding:
description: "Eagle3 speculative decoding with externally trained draft model; the served checkpoint is unchanged."
args:
- "--speculative-config"
- '{"model": "RedHatAI/Qwen3-235B-A22B-Instruct-2507-speculator.eagle3", "num_speculative_tokens": 3, "method": "eagle3"}'
opt_in_features:
- spec_decoding
variants:
default:
precision: bf16
vram_minimum_gb: 564
description: "Full precision BF16 — requires 4x H200 or 8x MI300X/MI325X/MI355X"
fp8:
model_id: "Qwen/Qwen3-235B-A22B-FP8"
precision: fp8
vram_minimum_gb: 240
# FP8 block_n=128 + moe_intermediate_size=1536 → TP must divide 12; cap at 4.
tp: 4
description: "Qwen official FP8 checkpoint for improved efficiency on SM90+"
nvfp4:
model_id: "nvidia/Qwen3-235B-A22B-Instruct-2507-NVFP4"
precision: nvfp4
vram_minimum_gb: 141
description: "NVIDIA NVFP4 quantized weights for Blackwell GPUs"
extra_args:
- "--kv-cache-dtype"
- "fp8"
redhat_nvfp4:
model_id: "RedHatAI/Qwen3-235B-A22B-Instruct-2507-NVFP4"
precision: nvfp4
label: "NVFP4 (RedHat)"
vram_minimum_gb: 152
description: "RedHatAI NVFP4 quantization"
compatible_strategies:
- single_node_tp
- single_node_tep
- single_node_dep
- multi_node_tp
- multi_node_tp_pp
- multi_node_dep
- multi_node_tep
- pd_cluster
kv_offload_support:
offloading_cpu: verified
offloading_fs: verified
hardware_overrides: {}
strategy_overrides: {}
guide: |
## Overview
[Qwen3](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507) is the flagship instruct MoE model in the Qwen3 series, with 235B total parameters and 22B active parameters. This guide covers deploying the model efficiently with vLLM on NVIDIA and AMD GPUs.
## Prerequisites
### NVIDIA CUDA (pip)
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend=auto
```
### AMD ROCm (pip)
> Note: The vLLM wheel for ROCm requires Python 3.12, ROCm 7.0, and glibc >= 2.35. Use the Docker flow if your environment is incompatible.
```bash
uv venv
source .venv/bin/activate
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.14.1/rocm700
```
## Deployment Configurations
### BF16 on MI300X/MI325X/MI355X (4 GPUs)
```bash
HIP_VISIBLE_DEVICES="4,5,6,7" \
VLLM_ROCM_USE_AITER=1 \
VLLM_ROCM_USE_AITER_MHA=0 \
SAFETENSORS_FAST_GPU=1 \
vllm serve Qwen/Qwen3-235B-A22B \
--trust-remote-code \
-tp 4 \
--swap-space 32 \
--distributed-executor-backend mp \
--max-num-batched-tokens 32768 \
--max-model-len 32768 \
--no-enable-prefix-caching \
--gpu-memory-utilization 0.8
```
### FP8 on MI300X/MI325X/MI355X (4 GPUs)
```bash
HIP_VISIBLE_DEVICES="4,5,6,7" \
VLLM_ROCM_USE_AITER=1 \
VLLM_ROCM_USE_AITER_MHA=0 \
SAFETENSORS_FAST_GPU=1 \
vllm serve Qwen/Qwen3-235B-A22B-FP8 \
--trust-remote-code \
-tp 4 \
--swap-space 16 \
--distributed-executor-backend mp \
--max-num-batched-tokens 32768 \
--max-model-len 32768 \
--no-enable-prefix-caching \
--gpu-memory-utilization 0.8
```
### TPU Deployment
- [Qwen3-32B on Trillium (v6e)](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Qwen3)
- [Qwen3-4B on Trillium (v6e)](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Qwen3)
## Client Usage
```bash
vllm bench serve \
--model "Qwen/Qwen3-235B-A22B-FP8" \
--dataset-name random \
--random-input-len 8192 \
--random-output-len 1024 \
--request-rate 10000 \
--num-prompts 16 \
--ignore-eos \
--trust-remote-code
```
## Troubleshooting
- Use `--max-num-batched-tokens` and `--max-model-len` to fit memory constraints on smaller nodes.
- If OOM at startup, lower `--gpu-memory-utilization` (e.g. 0.8) and disable prefix caching.
- AMD builds require `VLLM_ROCM_USE_AITER=1` for best performance.
## References
- [Model card](https://huggingface.co/Qwen/Qwen3-235B-A22B-Instruct-2507)
- [FP8 checkpoint](https://huggingface.co/Qwen/Qwen3-235B-A22B-FP8)
- [vLLM documentation](https://docs.vllm.ai/)
recipes/models/Qwen/Qwen3-30B-A3B.yaml
+229
View File
@@ -0,0 +1,229 @@
meta:
title: "Qwen3-30B-A3B"
slug: "qwen3-30b-a3b"
provider: "Qwen"
description: "Qwen3 MoE model with 30.5B total and 3.3B active parameters, validated for BF16 serving on Intel Xeon 6 and Intel Arc Pro B70 and W8A8 serving on Ascend A3."
date_added: 2026-08-14
date_updated: 2026-09-23
tasks:
- text
performance_headline: "Validated on Intel Xeon 6 (GNR), Ascend A3 W8A8 and 4x Intel Arc Pro B70 with BF16 serving"
related_recipes:
- "Qwen/Qwen3-235B-A22B-Instruct-2507"
hardware:
xeon6: verified
ascend_910c: verified
arc_pro_b70: verified
model:
model_id: "Qwen/Qwen3-30B-A3B"
min_vllm_version: "0.8.5"
architecture: moe
parameter_count: "30.5B"
active_parameters: "3.3B"
context_length: 40960
base_args: []
base_env: {}
features:
tool_calling:
description: "Enable automatic tool choice with the Hermes parser"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "hermes"
reasoning:
description: "Enable Qwen3 thinking mode with the Qwen3 reasoning parser"
args:
- "--reasoning-parser"
- "qwen3"
long_context:
description: "YaRN RoPE scaling to the model card's validated 131,072-token context. Static YaRN holds the scaling factor constant regardless of input length, so leave this off unless you actually serve long prompts — it can cost quality below 32K."
args:
- "--hf-overrides"
- '{"max_position_embeddings":131072,"rope_parameters":{"rope_type":"yarn","rope_theta":1000000,"factor":4.0,"original_max_position_embeddings":32768}}'
- "--max-model-len"
- "131072"
opt_in_features:
- long_context
hardware_opt_in_features:
ascend_910c:
- tool_calling
- reasoning
variants:
default:
precision: bf16
vram_minimum_gb: 74
description: "Full precision BF16"
supported_hardware: [xeon6, arc_pro_b70]
ascend_w8a8:
model_id: "Eco-Tech/Qwen3-30B-A3B-w8a8"
precision: int8
label: "W8A8 (Ascend A3)"
vram_minimum_gb: 37
description: "ModelSlim INT8 W8A8 checkpoint for Ascend 910C, verified with TP4 + expert parallelism."
supported_hardware: [ascend_910c]
tp: 4
min_vllm_version: "0.23.0"
extra_args:
- "--trust-remote-code"
- "--max-num-seqs"
- "100"
- "--max-model-len"
- "40960"
- "--max-num-batched-tokens"
- "16384"
- "--enable-expert-parallel"
- "--quantization"
- "ascend"
- "--distributed_executor_backend"
- "mp"
- "--no-enable-prefix-caching"
- "--async-scheduling"
- "--compilation-config"
- '{"cudagraph_mode": "FULL_DECODE_ONLY"}'
- "--additional-config"
- '{"enable_flashcomm1": true, "weight_nz_mode": 2}'
- "--gpu-memory-utilization"
- "0.95"
extra_env:
VLLM_USE_MODELSCOPE: "True"
HCCL_OP_EXPANSION_MODE: "AIV"
HCCL_BUFFSIZE: "1024"
OMP_PROC_BIND: "false"
OMP_NUM_THREADS: "1"
TASK_QUEUE_ENABLE: "1"
PYTORCH_NPU_ALLOC_CONF: "expandable_segments:True"
VLLM_USE_RUST_FRONTEND: "1"
hardware_overrides:
ascend_910c:
docker_image: "swr.cn-southwest-2.myhuaweicloud.com/base_image/ascend-ci/vllm-ascend:v0.23.0-a3"
compatible_strategies:
- single_node_tp
- multi_node_tp
hardware_overrides:
cpu:
extra_args:
- "--max-num-batched-tokens"
- "16384"
- "--gpu-memory-utilization"
- "0.8"
xpu:
extra_args:
- "--block-size"
- "64"
- "--enforce-eager"
- "--no-enable-prefix-caching"
- "--disable-sliding-window"
- "--max-model-len"
- "9472"
- "--max-num-batched-tokens"
- "8192"
- "--gpu-memory-utilization"
- "0.9"
strategy_overrides: {}
guide: |
## Overview
Qwen3-30B-A3B is a mixture-of-experts Qwen3 model with 30.5B total parameters
and 3.3B active parameters per token. This recipe includes an Intel Xeon 6 CPU
configuration validated with BF16 serving and an Ascend A3 W8A8 variant.
## Prerequisites
- Hardware: 4x Xeon 6 CPUs, or 4x Intel Arc Pro B70 (32 GB per card).
- vLLM >= 0.8.5
### pip (Intel Xeon 6 CPUs)
For Intel and AMD x86 CPUs, follow the
[CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels)
installation instructions.
### Docker (Intel Xeon 6 CPUs)
```bash
docker pull vllm/vllm-openai-cpu:latest-x86_64
```
### Docker (Intel Arc Pro B70)
```bash
docker pull vllm/vllm-openai-xpu:latest
```
## Intel Xeon 6
Choose the tensor parallel size based on the system topology and deployment
requirements. Add `--tensor-parallel-size <N>` when needed; the recipe does
not prescribe a Xeon 6 TP value or hard-code NUMA node IDs.
```bash
vllm serve Qwen/Qwen3-30B-A3B \
--max-num-batched-tokens 16384 \
--gpu-memory-utilization 0.8
```
Docker:
```bash
docker run \
--privileged --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai-cpu:latest-x86_64 Qwen/Qwen3-30B-A3B \
--tensor-parallel-size 1 \
--max-num-batched-tokens 16384 \
--gpu-memory-utilization 0.8 \
--enable-auto-tool-choice \
--tool-call-parser hermes \
--reasoning-parser qwen3
```
## Intel Arc Pro B70 (XPU)
Validated on 4x Intel Arc Pro B70 (32 GB per card) with the official vLLM XPU
image `vllm/vllm-openai-xpu:latest`, BF16, TP=4.
```bash
docker run --device /dev/dri \
-v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
--privileged --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai-xpu:latest \
Qwen/Qwen3-30B-A3B \
--tensor-parallel-size 4 \
--block-size 64 \
--enforce-eager \
--no-enable-prefix-caching \
--disable-sliding-window \
--max-model-len 9472 \
--max-num-batched-tokens 8192 \
--gpu-memory-utilization 0.9
```
## Client Usage
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
response = client.chat.completions.create(
model="Qwen/Qwen3-30B-A3B",
messages=[{"role": "user", "content": "Explain tensor parallelism briefly."}],
)
print(response.choices[0].message.content)
```
## References
- [Model card](https://huggingface.co/Qwen/Qwen3-30B-A3B)
- [Ascend A3 W8A8 checkpoint](https://www.modelscope.cn/models/Eco-Tech/Qwen3-30B-A3B-w8a8)
- [vLLM Ascend](https://github.com/vllm-project/vllm-ascend)
- [vLLM CPU installation](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/)
recipes/models/Qwen/Qwen3-32B.yaml
+249
View File
@@ -0,0 +1,249 @@
meta:
title: "Qwen3-32B"
slug: "qwen3-32b"
provider: "Qwen"
description: "Qwen3 32B dense model with hybrid thinking/non-thinking modes — verified on TPU v6e (Trillium)."
date_added: 2025-10-15
date_updated: 2026-09-10
tasks:
- text
performance_headline: "Verified on TPU v6e (Trillium) and v7 (Ironwood) with BF16"
related_recipes:
- "Qwen/Qwen3-4B"
hardware:
trillium: verified
ironwood: verified
arc_pro_b60: verified
arc_pro_b70: verified
model:
model_id: "Qwen/Qwen3-32B"
min_vllm_version: "0.8.5"
architecture: dense
parameter_count: "32B"
active_parameters: "32B"
context_length: 40960
base_args: []
base_env: {}
features:
tool_calling:
description: "Enable automatic tool choice with the Hermes parser"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "hermes"
reasoning:
description: "Enable Qwen3 thinking mode (chain-of-thought) with the Qwen3 reasoning parser"
args:
- "--reasoning-parser"
- "qwen3"
long_context:
description: "YaRN RoPE scaling to the model card's validated 131,072-token context. Static YaRN holds the scaling factor constant regardless of input length, so leave this off unless you actually serve long prompts — it can cost quality below 32K."
args:
- "--hf-overrides"
- '{"max_position_embeddings":131072,"rope_parameters":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":32768}}'
- "--max-model-len"
- "131072"
spec_decoding:
description: "Eagle3 speculative decoding with externally trained draft model; the served checkpoint is unchanged."
args:
- "--speculative-config"
- '{"model": "RedHatAI/Qwen3-32B-speculator.eagle3", "num_speculative_tokens": 3, "method": "eagle3"}'
opt_in_features:
- spec_decoding
- long_context
variants:
default:
precision: bf16
vram_minimum_gb: 77
description: "Full precision BF16 — TPU v6e 2x2 (4 chips, TP=4) or a single 80GB+ GPU"
fp8:
model_id: "Qwen/Qwen3-32B-FP8"
precision: fp8
vram_minimum_gb: 39
description: "Qwen official FP8 checkpoint — single 40 GB GPU"
awq:
model_id: "Qwen/Qwen3-32B-AWQ"
precision: int4
vram_minimum_gb: 20
description: "AWQ 4-bit quantized weights — fits on a single 24 GB GPU"
extra_args:
- "--quantization"
- "awq"
redhat_fp8:
model_id: "RedHatAI/Qwen3-32B-FP8-dynamic"
precision: fp8
label: "FP8 Dynamic (RedHat)"
vram_minimum_gb: 39
description: "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization"
extra_args:
- "--kv-cache-dtype"
- "fp8"
redhat_fp8_block:
model_id: "RedHatAI/Qwen3-32B-FP8-block"
precision: fp8
label: "FP8 Block (RedHat)"
vram_minimum_gb: 39
description: "RedHatAI FP8 block quantization"
extra_args:
- "--kv-cache-dtype"
- "fp8"
nvfp4:
model_id: "RedHatAI/Qwen3-32B-NVFP4"
precision: nvfp4
vram_minimum_gb: 22
description: "RedHatAI NVFP4 quantization"
redhat_nvfp4:
model_id: "RedHatAI/Qwen3-32B-NVFP4A16"
precision: nvfp4
label: "NVFP4 A16 (RedHat)"
vram_minimum_gb: 22
description: "RedHatAI NVFP4 with BF16 activations"
compatible_strategies:
- single_node_tp
- multi_node_tp
kv_offload_support:
offloading_cpu: verified
offloading_fs: verified
hardware_overrides:
xpu:
extra_args:
- "--gpu-memory-utilization"
- "0.90"
strategy_overrides: {}
guide: |
## Overview
Qwen3-32B is the flagship dense model in the Qwen3 series, with hybrid thinking / non-thinking modes. BF16 is the training precision and gives the best accuracy.
Toggle thinking mode per request with `enable_thinking` in `chat_template_kwargs`, or set the **Reasoning** feature to add the Qwen3 reasoning parser so `<think>` content is split into a separate field.
## Prerequisites
### NVIDIA
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```
### TPU (Trillium / v6e)
Use the official vLLM TPU image. The 32B model is served on a `v6e-4` slice (`--topology 2x2`, 4 chips, TP=4).
```bash
export HF_HOME=/dev/shm
export HF_TOKEN=<your HF token>
```
### Intel XPU (B60 / B70)
Use the official vLLM XPU image.
```bash
docker pull vllm/vllm-openai-xpu:latest
```
## Deployment Configurations
### TPU v6e (Trillium, 2x2 slice, TP=4)
Verified end-to-end on a 4-chip v6e slice.
```bash
export HF_HOME=/dev/shm
export HF_TOKEN=<your HF token>
export TP=4
export MAX_MODEL_LEN=4096
vllm serve Qwen/Qwen3-32B \
--seed 42 \
--gpu-memory-utilization 0.98 \
--max-num-batched-tokens 2048 \
--max-num-seqs 256 \
--tensor-parallel-size $TP \
--max-model-len $MAX_MODEL_LEN
```
### TPU v7 (Ironwood, single host, TP=2)
Verified end-to-end on a single Ironwood host (`tpu7x-standard-1t`, topology `1x1x1`). Served via the OpenAI API server entrypoint.
```bash
export HF_HOME=/data
export HUGGING_FACE_HUB_TOKEN=<your HF token>
export LAYOUT_Q_PROJ_AS_NDH=1
export USE_BATCHED_RPA_KERNEL=1
vllm serve Qwen/Qwen3-32B \
--host 0.0.0.0 \
--port 8000 \
--seed 42 \
--tensor-parallel-size 2 \
--data-parallel-size 1 \
--max-model-len 9216 \
--max-num-batched-tokens 4096 \
--max-num-seqs 135 \
--gpu-memory-utilization 0.98 \
--block-size 256 \
--kv-cache-dtype fp8 \
--no-enable-prefix-caching \
--async-scheduling
```
> Ironwood support ships in the nightly TPU image (e.g. `vllm/vllm-tpu:nightly-20260330-2f76400-8c0b626`).
### Single node (BF16, TP=2)
```bash
vllm serve Qwen/Qwen3-32B \
--host 0.0.0.0 \
--port 8000 \
--tensor-parallel-size 2
```
### Intel XPU (B60, B70)
Validated on 4× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the
official vLLM XPU image `vllm/vllm-openai-xpu:latest`.
```bash
docker run --device /dev/dri \
-v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
--privileged --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai-xpu:latest \
Qwen/Qwen3-32B \
--gpu-memory-utilization 0.90 \
--tensor-parallel-size 4 \
--max-model-len 8192
```
## Configuration Tips
- `--max-model-len 4096` keeps memory in check on a 4-chip v6e slice; native context is 40K — raise it if you have headroom.
- vLLM uses 90% of device memory by default; the TPU recipe pushes `--gpu-memory-utilization 0.98` to maximize KV cache.
- `--max-num-batched-tokens` and `--max-num-seqs` tune batch throughput vs latency.
- For thinking mode Qwen recommends `temperature=0.6, top_p=0.95, top_k=20`; for non-thinking, `temperature=0.7, top_p=0.8`.
## Benchmarking
Launch the server with `--no-enable-prefix-caching` for consistent measurements.
```bash
vllm bench serve \
--host 0.0.0.0 \
--port 8000 \
--model Qwen/Qwen3-32B \
--dataset-name random \
--random-input-len 1024 \
--random-output-len 1024 \
--num-prompts 128
```
## References
- [Model card](https://huggingface.co/Qwen/Qwen3-32B)
- [Qwen3 on Trillium (v6e)](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Qwen3)
- [Qwen3-32B on Ironwood (v7)](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/ironwood/vLLM/Qwen3-32B)
recipes/models/Qwen/Qwen3-4B.yaml
+176
View File
@@ -0,0 +1,176 @@
meta:
title: "Qwen3-4B"
slug: "qwen3-4b"
provider: "Qwen"
description: "Qwen3 4B dense model with hybrid thinking/non-thinking modes — fits on a single TPU v6e chip, one GPU or one Xeon 6 NUMA node."
date_added: 2025-10-15
date_updated: 2026-09-12
tasks:
- text
performance_headline: "Verified on TPU v6e (Trillium) with BF16 on a single chip"
related_recipes:
- "Qwen/Qwen3-32B"
hardware:
trillium: verified
xeon6: verified
model:
model_id: "Qwen/Qwen3-4B"
min_vllm_version: "0.8.5"
architecture: dense
parameter_count: "4B"
active_parameters: "4B"
context_length: 40960
base_args: []
base_env: {}
features:
tool_calling:
description: "Enable automatic tool choice with the Hermes parser"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "hermes"
reasoning:
description: "Enable Qwen3 thinking mode (chain-of-thought) with the Qwen3 reasoning parser"
args:
- "--reasoning-parser"
- "qwen3"
long_context:
description: "YaRN RoPE scaling to the model card's validated 131,072-token context. Static YaRN holds the scaling factor constant regardless of input length, so leave this off unless you actually serve long prompts — it can cost quality below 32K."
args:
- "--hf-overrides"
- '{"max_position_embeddings":131072,"rope_parameters":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":32768}}'
- "--max-model-len"
- "131072"
opt_in_features:
- long_context
variants:
default:
precision: bf16
vram_minimum_gb: 10
description: "Full precision BF16 — fits on a single TPU v6e chip or one 16GB+ GPU or one Xeon 6 NUMA node"
supported_hardware: [trillium, xeon6]
fp8:
model_id: "Qwen/Qwen3-4B-FP8"
precision: fp8
vram_minimum_gb: 5
description: "Qwen official FP8 checkpoint"
supported_hardware: [trillium]
redhat_fp8:
model_id: "RedHatAI/Qwen3-4B-FP8-dynamic"
precision: fp8
label: "FP8 Dynamic (RedHat)"
vram_minimum_gb: 5
description: "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization"
supported_hardware: [trillium]
extra_args:
- "--kv-cache-dtype"
- "fp8"
compatible_strategies:
- single_node_tp
- multi_node_tp
kv_offload_support:
offloading_cpu: verified
offloading_fs: verified
hardware_overrides: {}
strategy_overrides: {}
guide: |
## Overview
Qwen3-4B is the small dense model in the Qwen3 series, with hybrid thinking / non-thinking modes. At 4B it fits on a single accelerator — one TPU v6e (Trillium) chip, a single 16GB+ GPU or one Xeon 6 NUMA node — making it the cheapest entry point in the family. BF16 is the training precision and gives the best accuracy.
Toggle thinking mode per request with `enable_thinking` in `chat_template_kwargs`, or set the **Reasoning** feature to add the Qwen3 reasoning parser so `<think>` content is split into a separate field.
## Prerequisites
### NVIDIA
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```
### pip (Intel Xeon 6 CPUs)
For Intel and AMD x86 CPUs, follow the [CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels) installation instructions.
### TPU (Trillium / v6e)
Use the official vLLM TPU image. The 4B model fits on a single v6e chip (`v6e-1`, `--topology 1x1`).
```bash
export HF_HOME=/dev/shm
export HF_TOKEN=<your HF token>
```
## Deployment Configurations
### TPU v6e (Trillium, single chip, TP=1)
Verified end-to-end on a single v6e chip.
```bash
export HF_HOME=/dev/shm
export HF_TOKEN=<your HF token>
export TP=1
export MAX_MODEL_LEN=4096
vllm serve Qwen/Qwen3-4B \
--seed 42 \
--gpu-memory-utilization 0.98 \
--max-num-batched-tokens 1024 \
--max-num-seqs 128 \
--tensor-parallel-size $TP \
--max-model-len $MAX_MODEL_LEN
```
### Single GPU (BF16)
```bash
vllm serve Qwen/Qwen3-4B \
--host 0.0.0.0 \
--port 8000
```
### Intel Xeon 6 Deployment via Docker
Launch the x86 CPU vLLM Docker container for `Qwen/Qwen3-4B`:
```bash
docker run \
--privileged --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai-cpu:latest-x86_64 Qwen/Qwen3-4B \
--tensor-parallel-size 1 \
--enable-auto-tool-choice \
--tool-call-parser hermes \
--reasoning-parser qwen3
```
## Configuration Tips
- `--max-model-len 4096` is the TPU-recipe default; native context is 40K — raise it if you have headroom.
- vLLM uses 90% of device memory by default; the TPU recipe pushes `--gpu-memory-utilization 0.98` to maximize KV cache.
- For thinking mode Qwen recommends `temperature=0.6, top_p=0.95, top_k=20`; for non-thinking, `temperature=0.7, top_p=0.8`.
## Benchmarking
Launch the server with `--no-enable-prefix-caching` for consistent measurements.
```bash
vllm bench serve \
--host 0.0.0.0 \
--port 8000 \
--model Qwen/Qwen3-4B \
--dataset-name random \
--random-input-len 1024 \
--random-output-len 1024 \
--num-prompts 128
```
## References
- [Model card](https://huggingface.co/Qwen/Qwen3-4B)
- [Qwen3 on Trillium (v6e)](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/trillium/vLLM/Qwen3)
recipes/models/Qwen/Qwen3-8B.yaml
+138
View File
@@ -0,0 +1,138 @@
meta:
title: "Qwen3-8B"
slug: "qwen3-8b"
provider: "Qwen"
description: "Qwen3 8.2B dense model with hybrid thinking/non-thinking modes, validated for BF16 serving on Intel Xeon 6."
date_added: 2026-08-14
date_updated: 2026-09-12
tasks:
- text
performance_headline: "Validated on Intel Xeon 6 (GNR) with BF16 serving"
related_recipes:
- "Qwen/Qwen3-4B"
- "Qwen/Qwen3-32B"
hardware:
xeon6: verified
model:
model_id: "Qwen/Qwen3-8B"
min_vllm_version: "0.8.5"
architecture: dense
parameter_count: "8.2B"
active_parameters: "8.2B"
context_length: 40960
base_args: []
base_env: {}
features:
tool_calling:
description: "Enable automatic tool choice with the Hermes parser"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "hermes"
reasoning:
description: "Enable Qwen3 thinking mode with the Qwen3 reasoning parser"
args:
- "--reasoning-parser"
- "qwen3"
long_context:
description: "YaRN RoPE scaling to the model card's validated 131,072-token context. Static YaRN holds the scaling factor constant regardless of input length, so leave this off unless you actually serve long prompts — it can cost quality below 32K."
args:
- "--hf-overrides"
- '{"max_position_embeddings":131072,"rope_parameters":{"rope_type":"yarn","factor":4.0,"original_max_position_embeddings":32768}}'
- "--max-model-len"
- "131072"
opt_in_features:
- long_context
variants:
default:
precision: bf16
vram_minimum_gb: 20
description: "Full precision BF16"
supported_hardware: [xeon6]
compatible_strategies:
- single_node_tp
- multi_node_tp
hardware_overrides:
cpu:
extra_args:
- "--max-num-batched-tokens"
- "16384"
- "--gpu-memory-utilization"
- "0.8"
strategy_overrides: {}
guide: |
## Overview
Qwen3-8B is a dense Qwen3 model with hybrid thinking and non-thinking modes.
This recipe includes an Intel Xeon 6 CPU configuration validated with BF16
serving.
## Prerequisites
- Hardware: 2x Xeon 6 CPUs
- vLLM >= 0.8.5
### pip (Intel Xeon 6 CPUs)
For Intel and AMD x86 CPUs, follow the
[CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels)
installation instructions.
### Docker (Intel Xeon 6 CPUs)
```bash
docker pull vllm/vllm-openai-cpu:latest-x86_64
```
## Intel Xeon 6
Choose the tensor parallel size based on the system topology and deployment
requirements. Add `--tensor-parallel-size <N>` when needed; the recipe does
not prescribe a Xeon 6 TP value or hard-code NUMA node IDs.
```bash
vllm serve Qwen/Qwen3-8B \
--max-num-batched-tokens 16384 \
--gpu-memory-utilization 0.8
```
Docker:
```bash
docker run \
--privileged --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai-cpu:latest-x86_64 Qwen/Qwen3-8B \
--tensor-parallel-size 1 \
--max-num-batched-tokens 16384 \
--gpu-memory-utilization 0.8 \
--enable-auto-tool-choice \
--tool-call-parser hermes \
--reasoning-parser qwen3
```
## Client Usage
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="unused")
response = client.chat.completions.create(
model="Qwen/Qwen3-8B",
messages=[{"role": "user", "content": "Explain tensor parallelism briefly."}],
)
print(response.choices[0].message.content)
```
## References
- [Model card](https://huggingface.co/Qwen/Qwen3-8B)
- [vLLM CPU installation](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/)
recipes/models/Qwen/Qwen3-ASR-1.7B.yaml
+154
View File
@@ -0,0 +1,154 @@
meta:
title: "Qwen3-ASR-1.7B"
slug: "qwen3-asr-1.7b"
provider: "Qwen"
description: "Speech-to-text model supporting 11 languages, multiple accents, and singing voice with customizable text-context prompting."
date_added: 2026-01-29
date_updated: 2026-04-17
tasks:
- multimodal
performance_headline: "Accurate multilingual ASR, including singing voice; single-GPU serving"
related_recipes: []
model:
model_id: "Qwen/Qwen3-ASR-1.7B"
min_vllm_version: "0.16.0"
architecture: dense
parameter_count: "2.3B"
active_parameters: "2.3B"
context_length: 65536
base_args: []
base_env: {}
dependencies:
- note: "Audio extras — required for ASR input pre-processing (librosa, soundfile)"
command: 'uv pip install -U "vllm[audio]"'
features: {}
opt_in_features: []
variants:
default:
precision: bf16
vram_minimum_gb: 4
description: "Full precision BF16 — fits on a single mid-range GPU"
compatible_strategies:
- single_node_tp
- multi_node_tp
kv_offload_support:
offloading_cpu: verified
offloading_fs: verified
hardware_overrides:
amd:
extra_args: []
extra_env:
VLLM_ROCM_USE_AITER: "1"
SAFETENSORS_FAST_GPU: "1"
strategy_overrides: {}
guide: |
## Overview
Qwen3-ASR is a speech-to-text model that achieves accurate and robust recognition across 11 languages and multiple accents. It supports prompting the model with text context in any format to produce customized ASR results and performs well on singing-voice recognition. This guide demonstrates how to deploy Qwen3-ASR efficiently with vLLM.
## Prerequisites
Install vLLM with audio dependencies:
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --pre \
--extra-index-url https://wheels.vllm.ai/nightly/cu129 \
--extra-index-url https://download.pytorch.org/whl/cu129 \
--index-strategy unsafe-best-match
uv pip install "vllm[audio]"
```
## Launching with vLLM
```bash
vllm serve Qwen/Qwen3-ASR-1.7B
```
## Client Usage
### Chat Completions (OpenAI SDK)
```python
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
response = client.chat.completions.create(
model="Qwen/Qwen3-ASR-1.7B",
messages=[{
"role": "user",
"content": [{
"type": "audio_url",
"audio_url": {"url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_en.wav"},
}],
}],
)
print(response.choices[0].message.content)
```
### Transcription API
```python
import httpx
from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="EMPTY")
audio_url = "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_en.wav"
audio_file = httpx.get(audio_url).content
transcription = client.audio.transcriptions.create(
model="Qwen/Qwen3-ASR-1.7B",
file=audio_file,
)
print(transcription.text)
```
### cURL
```bash
curl http://localhost:8000/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{
"messages": [
{"role": "user", "content": [
{"type": "audio_url", "audio_url": {"url": "https://qianwen-res.oss-cn-beijing.aliyuncs.com/Qwen3-ASR-Repo/asr_en.wav"}}
]}
]
}'
```
### Offline Inference
```python
from vllm import LLM, SamplingParams
from vllm.assets.audio import AudioAsset
llm = LLM(model="Qwen/Qwen3-ASR-1.7B")
audio_asset = AudioAsset("winning_call")
conversation = [{
"role": "user",
"content": [{"type": "audio_url", "audio_url": {"url": audio_asset.url}}],
}]
sampling_params = SamplingParams(temperature=0.01, max_tokens=256)
outputs = llm.chat(conversation, sampling_params=sampling_params)
print(outputs[0].outputs[0].text)
```
## Troubleshooting
- Make sure `vllm[audio]` extras are installed or audio requests will fail.
- Use the nightly wheel until Qwen3-ASR support lands in the stable release.
## References
- [Model card](https://huggingface.co/Qwen/Qwen3-ASR-1.7B)
- [vLLM audio support](https://docs.vllm.ai/en/latest/features/multimodal_inputs.html)
recipes/models/Qwen/Qwen3-Coder-480B-A35B-Instruct.yaml
+206
View File
@@ -0,0 +1,206 @@
meta:
title: "Qwen3-Coder-480B-A35B-Instruct"
slug: "qwen3-coder-480b-a35b-instruct"
provider: "Qwen"
description: "Large coder MoE with 480B total / 35B active parameters, strong tool-use and code generation capabilities."
date_added: 2025-07-23
date_updated: 2026-04-17
tasks:
- text
performance_headline: "HumanEval 0.939, MBPP 0.918 (FP8). Recommended FP8 on 8x H200/H20 via DP=8"
related_recipes: []
hardware:
h200: verified
mi300x: verified
mi325x: verified
mi355x: verified
ironwood: verified
model:
model_id: "Qwen/Qwen3-Coder-480B-A35B-Instruct"
min_vllm_version: "0.10.0"
architecture: moe
parameter_count: "480B"
active_parameters: "35B"
context_length: 262144
base_args: []
base_env: {}
dependencies:
- note: "Optional: opt into DeepGEMM FP8 MoE kernels for extra throughput"
command: "export VLLM_USE_DEEP_GEMM=1"
optional: true
features:
tool_calling:
description: "Enable automatic tool choice with Qwen3 Coder parser"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "qwen3_coder"
opt_in_features: []
variants:
default:
precision: bf16
vram_minimum_gb: 1152
description: "Full BF16 — 8x H200/H20 (141GB × 8) recommended"
fp8:
model_id: "Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8"
precision: fp8
vram_minimum_gb: 576
# FP8 block_n=128 + moe_intermediate_size=2560 → TP must divide 20; cap at 4.
tp: 4
description: "Qwen official FP8 checkpoint — required for DP=8 serving"
nvfp4:
model_id: "nvidia/Qwen3-Coder-480B-A35B-Instruct-NVFP4"
precision: nvfp4
vram_minimum_gb: 288
description: "NVIDIA NVFP4 quantized weights for Blackwell GPUs"
extra_args:
- "--kv-cache-dtype"
- "fp8"
redhat_fp8:
model_id: "RedHatAI/Qwen3-Coder-480B-A35B-Instruct-FP8"
precision: fp8
label: "FP8 (RedHat)"
vram_minimum_gb: 576
description: "RedHatAI FP8 quantization"
extra_args:
- "--kv-cache-dtype"
- "fp8"
compatible_strategies:
- single_node_tp
- single_node_tep
- single_node_dep
- multi_node_tp
- multi_node_tp_pp
- multi_node_dep
- multi_node_tep
- pd_cluster
hardware_overrides:
hopper:
extra_args: []
extra_env:
VLLM_USE_DEEP_GEMM: "1"
amd:
# Verified on 8× MI300X / MI325X / MI355X (FP8 path uses --data-parallel-size 8
# instead of TP8 — swap strategy to DP if deploying the FP8 checkpoint).
extra_args: ["--attention-backend", "rocm_aiter_fa", "--moe-backend", "aiter"]
extra_env:
VLLM_ROCM_USE_AITER: "1"
strategy_overrides: {}
guide: |
## Overview
[Qwen3-Coder](https://github.com/QwenLM/Qwen3-Coder) is an advanced large language model created by the Qwen team. `Qwen3-Coder-480B-A35B-Instruct` is the flagship coder MoE with 480B total / 35B active parameters. vLLM supports it including tool calling; the guide below covers BF16 and FP8 serving on NVIDIA and AMD GPUs.
TPU support is provided through [vLLM TPU](https://github.com/vllm-project/tpu-inference) with a recipe for [Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/ironwood/vLLM/Qwen3-Coder-480B-A35B). The Ironwood docker command rendered by the hardware picker uses the `vllm/vllm-tpu` image; pin to the tag specified by the upstream recipe.
## Prerequisites
### CUDA
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```
### ROCm (MI300X, MI325X, MI355X)
```bash
uv venv
source .venv/bin/activate
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/
```
The ROCm wheel requires Python 3.12, ROCm 7.0, and glibc >= 2.35.
## Deployment Configurations
### 8xH200 / 8xH20 BF16
```bash
vllm serve Qwen/Qwen3-Coder-480B-A35B-Instruct \
--max-model-len 32000 \
--enable-expert-parallel \
--tensor-parallel-size 8 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder
```
### 8xH200 / 8xH20 FP8 (DP=8, recommended)
```bash
VLLM_USE_DEEP_GEMM=1 vllm serve Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
--max-model-len 131072 \
--enable-expert-parallel \
--data-parallel-size 8 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder
```
### 8xMI300X/MI325X/MI355X BF16
```bash
VLLM_ROCM_USE_AITER=1 vllm serve Qwen/Qwen3-Coder-480B-A35B-Instruct \
--max-model-len 32000 \
--attention-backend rocm_aiter_fa \
--moe-backend aiter \
--enable-expert-parallel \
--tensor-parallel-size 8 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder
```
### 8xMI300X/MI325X/MI355X FP8
```bash
VLLM_ROCM_USE_AITER=1 vllm serve Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
--trust-remote-code \
--max-model-len 131072 \
--attention-backend rocm_aiter_fa \
--moe-backend aiter \
--enable-expert-parallel \
--data-parallel-size 8 \
--enable-auto-tool-choice \
--tool-call-parser qwen3_coder
```
## Client Usage
```bash
vllm bench serve \
--backend vllm \
--model Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8 \
--endpoint /v1/completions \
--dataset-name random \
--random-input 2048 \
--random-output 1024 \
--max-concurrency 10 \
--num-prompt 100
```
## Troubleshooting
- **Context-length OOM**: A single H20 node cannot serve the native 262144 context. Reduce `--max-model-len` or raise `--gpu-memory-utilization`.
- **TP=8 failure on FP8**: Expect `ValueError: The output_size of gate's and up's weight = 320 is not divisible by weight quantization block_n = 128.` on FP8 with TP=8. Switch to `--data-parallel-size 8` instead.
- **DeepGEMM**: set `VLLM_USE_DEEP_GEMM=1` for faster FP8 matmul. Follow the [setup instructions](https://github.com/vllm-project/vllm/blob/main/benchmarks/kernels/deepgemm/README.md#setup) to install it.
- **Tool calling**: add `--tool-call-parser qwen3_coder` as shown above.
## Evaluation
| Dataset | Test Type | Pass@1 Score |
|------------|--------------------|--------------|
| HumanEval | Base tests | 0.939 |
| HumanEval+ | Base + extra tests | 0.902 |
| MBPP | Base tests | 0.918 |
| MBPP+ | Base + extra tests | 0.794 |
## References
- [Model card](https://huggingface.co/Qwen/Qwen3-Coder-480B-A35B-Instruct)
- [FP8 checkpoint](https://huggingface.co/Qwen/Qwen3-Coder-480B-A35B-Instruct-FP8)
- [Qwen3-Coder GitHub](https://github.com/QwenLM/Qwen3-Coder)
- [EvalPlus](https://github.com/evalplus/evalplus)
- [TPU recipes: Ironwood](https://github.com/AI-Hypercomputer/tpu-recipes/tree/main/inference/ironwood/vLLM/Qwen3-Coder-480B-A35B)
recipes/models/Qwen/Qwen3-Next-80B-A3B-Instruct.yaml
+232
View File
@@ -0,0 +1,232 @@
meta:
title: "Qwen3-Next-80B-A3B-Instruct"
slug: "qwen3-next-80b-a3b-instruct"
provider: "Qwen"
description: "Advanced Qwen3-Next MoE model (80B total / 3B active) with hybrid attention, highly sparse experts, and multi-token prediction."
date_added: 2025-09-11
date_updated: 2026-09-10
tasks:
- text
performance_headline: "Highly sparse MoE with MTP-accelerated decoding, runs on 4x H200/H20/A100/A800"
related_recipes: []
hardware:
h200: verified
mi300x: verified
mi325x: verified
mi355x: verified
model:
model_id: "Qwen/Qwen3-Next-80B-A3B-Instruct"
min_vllm_version: "0.17.0"
architecture: moe
parameter_count: "80B"
active_parameters: "3B"
context_length: 262144
base_args: []
base_env: {}
features:
tool_calling:
description: "Enable automatic tool choice with Hermes parser"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "hermes"
spec_decoding:
description: "Multi-token prediction speculative decoding for lower latency"
args:
- "--speculative-config"
- '{"method":"qwen3_next_mtp","num_speculative_tokens":2}'
- "--no-enable-chunked-prefill"
long_context:
description: "YaRN RoPE scaling to the model card's validated 1M-token context (factor 4.0 over the native 262,144). Static YaRN holds the scaling factor constant regardless of input length, so leave this off unless you actually serve long prompts."
args:
- "--hf-overrides"
- '{"rope_parameters":{"rope_type":"yarn","rope_theta":10000000,"partial_rotary_factor":0.25,"factor":4.0,"original_max_position_embeddings":262144}}'
- "--max-model-len"
- "1010000"
env:
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
opt_in_features:
- spec_decoding
- long_context
variants:
default:
precision: bf16
vram_minimum_gb: 192
description: "Full precision BF16 — fits on 4x H200/H20/A100/A800"
fp8:
model_id: "Qwen/Qwen3-Next-80B-A3B-Instruct-FP8"
precision: fp8
vram_minimum_gb: 96
description: "Qwen official FP8 checkpoint — recommended on SM90/SM100"
nvfp4:
model_id: "nvidia/Qwen3-Next-80B-A3B-Instruct-NVFP4"
precision: nvfp4
vram_minimum_gb: 48
description: "NVIDIA NVFP4 quantized weights for Blackwell GPUs"
extra_args:
- "--kv-cache-dtype"
- "fp8"
redhat_fp8:
model_id: "RedHatAI/Qwen3-Next-80B-A3B-Instruct-FP8-dynamic"
precision: fp8
label: "FP8 Dynamic (RedHat)"
vram_minimum_gb: 96
description: "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization"
extra_args:
- "--kv-cache-dtype"
- "fp8"
redhat_fp8_plain:
model_id: "RedHatAI/Qwen3-Next-80B-A3B-Instruct-FP8"
precision: fp8
label: "FP8 (RedHat)"
vram_minimum_gb: 96
description: "RedHatAI FP8 quantization"
extra_args:
- "--kv-cache-dtype"
- "fp8"
redhat_fp8_block:
model_id: "RedHatAI/Qwen3-Next-80B-A3B-Instruct-FP8-block"
precision: fp8
label: "FP8 Block (RedHat)"
vram_minimum_gb: 96
description: "RedHatAI FP8 block quantization"
extra_args:
- "--kv-cache-dtype"
- "fp8"
redhat_nvfp4:
model_id: "RedHatAI/Qwen3-Next-80B-A3B-Instruct-NVFP4"
precision: nvfp4
label: "NVFP4 (RedHat)"
vram_minimum_gb: 48
description: "RedHatAI NVFP4 quantization"
compatible_strategies:
- single_node_tp
- single_node_tep
- single_node_dep
- multi_node_tp
- multi_node_tp_pp
- multi_node_dep
- multi_node_tep
hardware_overrides:
blackwell:
extra_args:
- "--moe-backend"
- "flashinfer_trtllm"
- "--attention-backend"
- "FLASH_ATTN"
extra_env:
VLLM_USE_DEEP_GEMM: "0"
strategy_overrides: {}
guide: |
## Overview
[Qwen3-Next](https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd&from=research.latest-advancements-list) is an advanced LLM from the Qwen team featuring:
- A hybrid attention mechanism
- A highly sparse Mixture-of-Experts (MoE) structure
- Training-stability-friendly optimizations
- A multi-token prediction mechanism for faster inference
## Prerequisites
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend auto
```
## Deployment Configurations
Launch on 4x H200/H20 or 4x A100/A800 GPUs.
### Basic Multi-GPU (BF16)
```bash
vllm serve Qwen/Qwen3-Next-80B-A3B-Instruct \
--tensor-parallel-size 4 \
--served-model-name qwen3-next \
--enable-prefix-caching
```
If you hit `torch.AcceleratorError: CUDA error: an illegal memory access was encountered`, add `--compilation_config.cudagraph_mode=PIECEWISE`.
### FP8 (SM90/SM100)
```bash
vllm serve Qwen/Qwen3-Next-80B-A3B-Instruct-FP8 \
--tensor-parallel-size 4 \
--enable-prefix-caching
```
On SM100, accelerate with the FlashInfer TRTLLM MoE kernel:
```bash
VLLM_USE_DEEP_GEMM=0 \
vllm serve Qwen/Qwen3-Next-80B-A3B-Instruct-FP8 \
--tensor-parallel-size 4 \
--moe-backend flashinfer_trtllm \
--attention-backend FLASH_ATTN
```
### MTP (Multi-Token Prediction)
```bash
vllm serve Qwen/Qwen3-Next-80B-A3B-Instruct \
--tokenizer-mode auto --gpu-memory-utilization 0.8 \
--speculative-config '{"method": "qwen3_next_mtp", "num_speculative_tokens": 2}' \
--tensor-parallel-size 4 --no-enable-chunked-prefill
```
### Tool / Function Calling
```bash
vllm serve ... --tool-call-parser hermes --enable-auto-tool-choice
```
### AMD (MI300X/MI325X/MI355X)
```bash
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/0.14.1/rocm700
```
```bash
SAFETENSORS_FAST_GPU=1 \
VLLM_ALLOW_LONG_MAX_MODEL_LEN=1 \
vllm serve Qwen/Qwen3-Next-80B-A3B-Instruct \
--tensor-parallel-size 4 \
--max-model-len 32768 \
--no-enable-prefix-caching \
--trust-remote-code
```
## Client Usage
Benchmark:
```bash
vllm bench serve \
--backend vllm \
--model Qwen/Qwen3-Next-80B-A3B-Instruct \
--served-model-name qwen3-next \
--endpoint /v1/completions \
--dataset-name random \
--random-input 2048 \
--random-output 1024 \
--max-concurrency 10 \
--num-prompt 100
```
## Troubleshooting
- **Sub-optimal MoE performance warning**: Tune the MoE Triton kernel with [benchmark_moe](https://github.com/vllm-project/vllm/blob/main/benchmarks/kernels/benchmark_moe.py), then set `VLLM_TUNED_CONFIG_FOLDER` to the directory containing the generated config.
- **IMA error in DP mode**: add `--compilation_config.cudagraph_mode=PIECEWISE`.
- For more parallel topologies, see the [Data Parallel Deployment docs](https://docs.vllm.ai/en/latest/serving/data_parallel_deployment.html).
## References
- [Model card](https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct)
- [FP8 checkpoint](https://huggingface.co/Qwen/Qwen3-Next-80B-A3B-Instruct-FP8)
- [Qwen3-Next blog](https://qwen.ai/blog?id=4074cca80393150c248e508aa62983f9cb7d27cd)
recipes/models/Qwen/Qwen3-VL-235B-A22B-Instruct.yaml
+343
View File
@@ -0,0 +1,343 @@
meta:
title: "Qwen3-VL-235B-A22B-Instruct"
slug: "qwen3-vl-235b-a22b-instruct"
provider: "Qwen"
description: "Qwen3-VL flagship MoE vision-language model with 235B total / 22B active parameters, supporting images, video, and long context."
date_added: 2025-09-28
date_updated: 2026-04-17
tasks:
- multimodal
- text
performance_headline: "Strong on images, video, and text — #1 open model on text on lmarena.ai at release"
related_recipes: []
hardware:
h100: verified
h200: verified
b200: verified
mi300x: verified
mi325x: verified
mi355x: verified
model:
model_id: "Qwen/Qwen3-VL-235B-A22B-Instruct"
min_vllm_version: "0.11.0"
architecture: moe
parameter_count: "235B"
active_parameters: "22B"
context_length: 262144
base_args: []
base_env: {}
docker_image:
amd: "vllm/vllm-openai-rocm:v0.28.0"
dependencies:
- note: "Recommended for offline multimodal inference (image/video pre-processing helpers)"
command: "uv pip install qwen-vl-utils==0.0.14"
optional: true
features:
text_only:
description: "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
args:
- "--language-model-only"
encoder_parallel:
description: "Run the vision encoder in data-parallel mode — avoids TP comm overhead on the small encoder. Mutually exclusive with text_only."
args:
- "--mm-encoder-tp-mode"
- "data"
long_context:
description: "Extend the 262K context to 1M tokens with YaRN. Static scaling costs quality on
short prompts and the wider window inflates the KV budget, so it is opt-in."
args:
- "--hf-overrides"
- '{"text_config":{"rope_parameters":{"rope_type":"yarn","factor":3.0,"original_max_position_embeddings":262144,"mrope_section":[24,20,20],"mrope_interleaved":true}}}'
- "--max-model-len"
- "1000000"
env:
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
opt_in_features:
- text_only
- long_context
variants:
default:
precision: bf16
vram_minimum_gb: 564
description: "Full BF16 — ideal on H200/B200 with 8 GPUs"
fp8:
model_id: "Qwen/Qwen3-VL-235B-A22B-Instruct-FP8"
precision: fp8
vram_minimum_gb: 282
# FP8 block_n=128 + moe_intermediate_size=1536 → TP must divide 12; cap at 4.
tp: 4
description: "Qwen official FP8 checkpoint for optimal H100 memory efficiency"
nvfp4:
model_id: "nvidia/Qwen3-VL-235B-A22B-Instruct-NVFP4"
precision: nvfp4
vram_minimum_gb: 141
description: "NVIDIA NVFP4 quantized weights for Blackwell GPUs"
extra_args:
- "--kv-cache-dtype"
- "fp8"
redhat_fp8:
model_id: "RedHatAI/Qwen3-VL-235B-A22B-Instruct-FP8-dynamic"
precision: fp8
label: "FP8 Dynamic (RedHat)"
vram_minimum_gb: 282
description: "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization"
extra_args:
- "--kv-cache-dtype"
- "fp8"
redhat_fp8_block:
model_id: "RedHatAI/Qwen3-VL-235B-A22B-Instruct-FP8-block"
precision: fp8
label: "FP8 Block (RedHat)"
vram_minimum_gb: 282
description: "RedHatAI FP8 block quantization"
extra_args:
- "--kv-cache-dtype"
- "fp8"
redhat_nvfp4:
model_id: "RedHatAI/Qwen3-VL-235B-A22B-Instruct-NVFP4"
precision: nvfp4
label: "NVFP4 (RedHat)"
vram_minimum_gb: 150
description: "RedHatAI NVFP4 quantization"
compatible_strategies:
- single_node_tp
- single_node_tep
- single_node_dep
- multi_node_tp
- multi_node_tp_pp
- multi_node_dep
- multi_node_tep
- pd_cluster
kv_offload_support:
offloading_cpu: verified
offloading_fs: verified
hardware_overrides:
hopper:
extra_args:
- "--mm-encoder-tp-mode"
- "data"
- "--async-scheduling"
extra_env: {}
blackwell:
extra_args:
- "--mm-encoder-tp-mode"
- "data"
- "--async-scheduling"
extra_env: {}
amd:
extra_args:
- "--mm-encoder-tp-mode"
- "data"
- "--async-scheduling"
- "--enable-chunked-prefill"
extra_env:
VLLM_WORKER_MULTIPROC_METHOD: "spawn"
HIP_FORCE_DEV_KERNARG: "1"
VLLM_ROCM_USE_AITER: "1"
VLLM_ROCM_USE_AITER_MHA: "1"
VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT: "1"
SAFETENSORS_FAST_GPU: "1"
strategy_overrides: {}
guide: |
## Overview
[Qwen3-VL](https://github.com/QwenLM/Qwen3-VL) is the most powerful vision-language model in the Qwen series, delivering upgrades to text understanding & generation, visual perception & reasoning, extended context, spatial/video dynamics, and agent interaction. The flagship `Qwen3-VL-235B-A22B-Instruct` is a MoE model that requires at least 8 GPUs with ≥80 GB memory each (A100/H100/H200 class).
## Prerequisites
```bash
uv venv
source .venv/bin/activate
# Install vLLM >= 0.11.0
uv pip install -U vllm
# Install Qwen-VL utility library (recommended for offline inference)
uv pip install qwen-vl-utils==0.0.14
```
## Deployment Configurations
### H100 (Image + Video, FP8)
```bash
vllm serve Qwen/Qwen3-VL-235B-A22B-Instruct-FP8 \
--tensor-parallel-size 8 \
--mm-encoder-tp-mode data \
--enable-expert-parallel \
--async-scheduling
```
### H100 (Image-Only, FP8, TP4)
```bash
vllm serve Qwen/Qwen3-VL-235B-A22B-Instruct-FP8 \
--tensor-parallel-size 4 \
--limit-mm-per-prompt.video 0 \
--async-scheduling \
--gpu-memory-utilization 0.95 \
--max-num-seqs 128
```
### A100 & H100 (Image-Only, BF16)
```bash
vllm serve Qwen/Qwen3-VL-235B-A22B-Instruct \
--tensor-parallel-size 8 \
--limit-mm-per-prompt.video 0 \
--async-scheduling
```
### A100 & H100 (Image + Video, BF16)
```bash
vllm serve Qwen/Qwen3-VL-235B-A22B-Instruct \
--tensor-parallel-size 8 \
--max-model-len 128000 \
--async-scheduling
```
### H200 & B200
```bash
vllm serve Qwen/Qwen3-VL-235B-A22B-Instruct \
--tensor-parallel-size 8 \
--mm-encoder-tp-mode data \
--async-scheduling
```
### MI300X/MI325X/MI355X (BF16, TP8)
```bash
export SAFETENSORS_FAST_GPU="1"
export HIP_FORCE_DEV_KERNARG="1"
export HIP_VISIBLE_DEVICES="0,1,2,3,4,5,6,7"
export VLLM_WORKER_MULTIPROC_METHOD="spawn"
export VLLM_ROCM_USE_AITER="1"
export VLLM_ROCM_USE_AITER_MHA="1"
export VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT="1"
vllm serve Qwen/Qwen3-VL-235B-A22B-Instruct \
--tensor-parallel-size 8 \
--mm-encoder-tp-mode data \
--enable-expert-parallel \
--async-scheduling \
--gpu-memory-utilization 0.94 \
--max-model-len 32768 \
--max-num-seqs 10240 \
--max-num-batched-tokens 32768 \
--attention-backend ROCM_AITER_FA
```
### MI300X/MI325X/MI355X (FP8, TP8)
```bash
export SAFETENSORS_FAST_GPU="1"
export HIP_FORCE_DEV_KERNARG="1"
export HIP_VISIBLE_DEVICES="0,1,2,3,4,5,6,7"
export VLLM_WORKER_MULTIPROC_METHOD="spawn"
export VLLM_ROCM_USE_AITER="1"
export VLLM_ROCM_USE_AITER_MHA="1"
export VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT="1"
vllm serve Qwen/Qwen3-VL-235B-A22B-Instruct-FP8 \
--tensor-parallel-size 8 \
--mm-encoder-tp-mode data \
--enable-expert-parallel \
--async-scheduling \
--enable-chunked-prefill \
--limit-mm-per-prompt '{"image": 1}' \
--gpu-memory-utilization 0.94 \
--max-model-len 32768 \
--max-num-seqs 10240 \
--max-num-batched-tokens 32768 \
--kv-cache-dtype fp8 \
--compilation-config '{"mode": 3, "cudagraph_mode": "FULL_AND_PIECEWISE", "custom_ops": ["+rms_norm", "+quant_fp8"]}' \
--attention-backend ROCM_AITER_FA
```
> The above configuration runs efficiently at high concurrencies (up to 512) for context lengths up to 8k. If TPOT is not satisfactory, lower `--max-num-batched-tokens`. For longer contexts, `--max-num-seqs` and `--max-num-batched-tokens` may need retuning.
### MI300X / MI355X (FP8, EAGLE3 Speculative Decoding)
Adds EAGLE3 speculative decoding (`RedHatAI/Qwen3-VL-235B-A22B-Instruct-speculator.eagle3`) on top of the FP8 target. **`--attention-backend ROCM_AITER_FA` is mandatory** — the default `ROCM_ATTN` path makes SD slower than baseline on ROCm. Keep `VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT=0` and omit `--async-scheduling` (incompatible with SD).
```bash
export SAFETENSORS_FAST_GPU="1"
export HIP_FORCE_DEV_KERNARG="1"
export HIP_VISIBLE_DEVICES="0,1,2,3,4,5,6,7"
export VLLM_WORKER_MULTIPROC_METHOD="spawn"
export VLLM_ROCM_USE_AITER="1"
export VLLM_ROCM_USE_AITER_MHA="1"
export VLLM_ROCM_USE_AITER_RMSNORM="0"
export VLLM_ROCM_SHUFFLE_KV_CACHE_LAYOUT="0"
vllm serve Qwen/Qwen3-VL-235B-A22B-Instruct-FP8 \
--tensor-parallel-size 8 \
--gpu-memory-utilization 0.94 \
--distributed-executor-backend mp \
--enable-chunked-prefill \
--max-model-len 16384 \
--max-num-seqs 32 \
--max-num-batched-tokens 8192 \
--mm-encoder-tp-mode data \
--enable-expert-parallel \
--attention-backend ROCM_AITER_FA \
--compilation-config '{"mode": 3, "cudagraph_mode": "FULL_AND_PIECEWISE", "custom_ops": ["-rms_norm"], "pass_config": {"fuse_norm_quant":false}}' \
--speculative-config '{"model": "RedHatAI/Qwen3-VL-235B-A22B-Instruct-speculator.eagle3", "method": "eagle3", "num_speculative_tokens": 4}'
```
## Configuration Tips
- Use `--limit-mm-per-prompt.video 0` if your server only serves image inputs to save memory.
- `OMP_NUM_THREADS=1` reduces CPU contention during preprocessing.
- The model's context length is 262K. Reduce `--max-model-len` (e.g. 128000) if you don't need the full range.
- `--async-scheduling` overlaps scheduling with decoding for better throughput.
- `--mm-encoder-tp-mode data` deploys the vision encoder in data-parallel fashion for better performance.
- If your inputs are mostly unique, pass `--mm-processor-cache-gb 0` to skip caching overhead.
- Extend context to 1M tokens with the **Long Context** toggle, which applies YaRN via
`--hf-overrides` and raises `--max-model-len`. Static YaRN costs quality on short prompts.
Text-only mode: pass `--limit-mm-per-prompt.video 0 --limit-mm-per-prompt.image 0` to free memory for KV cache when serving text-only traffic.
## Client Usage
```python
import time
from openai import OpenAI
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1", timeout=3600)
messages = [{
"role": "user",
"content": [
{"type": "image_url", "image_url": {"url": "https://ofasys-multimodal-wlcb-3-toshanghai.oss-accelerate.aliyuncs.com/wpf272043/keepme/image/receipt.png"}},
{"type": "text", "text": "Read all the text in the image."},
],
}]
start = time.time()
response = client.chat.completions.create(
model="Qwen/Qwen3-VL-235B-A22B-Instruct",
messages=messages,
max_tokens=2048,
)
print(f"Response costs: {time.time() - start:.2f}s")
print(response.choices[0].message.content)
```
## Troubleshooting
- OOM on A100 / H100 BF16: reduce `--max-model-len`, drop to image-only, or switch to the FP8 checkpoint.
- If enabling `--mm-encoder-tp-mode data` raises memory pressure, lower `--gpu-memory-utilization`.
## References
- [Model card](https://huggingface.co/Qwen/Qwen3-VL-235B-A22B-Instruct)
- [FP8 checkpoint](https://huggingface.co/Qwen/Qwen3-VL-235B-A22B-Instruct-FP8)
- [Qwen3-VL GitHub](https://github.com/QwenLM/Qwen3-VL)
- [vLLM multimodal inputs guide](https://docs.vllm.ai/en/latest/features/multimodal_inputs.html)
recipes/models/Qwen/Qwen3-VL-30B-A3B-Instruct.yaml
+138
View File
@@ -0,0 +1,138 @@
meta:
title: "Qwen3-VL-30B-A3B-Instruct"
slug: "qwen3-vl-30b-a3b-instruct"
provider: "Qwen"
description: "Qwen3-VL MoE vision-language model with 30B total / 3B active parameters, supporting image, video, and text workloads."
date_added: 2026-08-17
date_updated: 2026-09-12
tasks:
- multimodal
- text
performance_headline: "Validated on Intel Xeon 6 CPU with BF16 serving"
related_recipes:
- "Qwen/Qwen3-VL-235B-A22B-Instruct"
hardware:
xeon6: verified
model:
model_id: "Qwen/Qwen3-VL-30B-A3B-Instruct"
min_vllm_version: "0.11.0"
architecture: moe
parameter_count: "30B"
active_parameters: "3B"
context_length: 262144
base_args: []
base_env: {}
dependencies:
- note: "Recommended for offline multimodal inference (image/video pre-processing helpers)"
command: "uv pip install qwen-vl-utils==0.0.14"
optional: true
features:
text_only:
description: "Skip loading the vision encoder for text-only workloads — frees memory for KV cache."
args:
- "--language-model-only"
encoder_parallel:
description: "Run the vision encoder in data-parallel mode."
args:
- "--mm-encoder-tp-mode"
- "data"
opt_in_features:
- text_only
variants:
default:
precision: bf16
vram_minimum_gb: 72
description: "Full precision BF16"
supported_hardware:
- xeon6
compatible_strategies:
- single_node_tp
- single_node_tep
- single_node_dep
- multi_node_tp
- multi_node_tp_pp
- multi_node_dep
- multi_node_tep
kv_offload_support:
offloading_cpu: verified
offloading_fs: verified
hardware_overrides: {}
strategy_overrides: {}
guide: |
## Overview
Qwen3-VL-30B-A3B-Instruct is a multimodal Qwen3-VL MoE model. Intel Xeon 6
CPU serving was validated with BF16.
## Prerequisites
- Hardware: Intel Xeon 6 CPUs
- vLLM >= 0.11.0
### pip (Intel Xeon 6 CPUs)
For Intel and AMD x86 CPUs, follow the
[CPU pre-built wheels](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/#pre-built-wheels)
installation instructions.
### Docker (Intel Xeon 6 CPUs)
```bash
docker pull vllm/vllm-openai-cpu:latest-x86_64
```
## Intel Xeon 6
```bash
vllm serve Qwen/Qwen3-VL-30B-A3B-Instruct \
--tensor-parallel-size 1 \
--mm-encoder-tp-mode data
```
Docker (the image entrypoint is `vllm serve`):
```bash
docker run \
--privileged --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai-cpu:latest-x86_64 Qwen/Qwen3-VL-30B-A3B-Instruct \
--tensor-parallel-size 1 \
--mm-encoder-tp-mode data
```
The example uses TP=1. Adjust TP/DP based on system topology; NUMA binding is
intentionally not hard-coded in the recipe.
## Configuration Notes
- TP/DP values and topology-specific CPU binding remain deployment choices.
## Runtime and Platform Tuning
The following settings are intentionally not prescribed as portable CPU model
defaults because they depend on the workload, hardware, or runtime environment:
- `--max-num-batched-tokens`: scheduler/throughput tuning.
- `--max-num-seqs`: concurrency and scheduler-capacity tuning.
- `--gpu-memory-utilization`: platform memory-budget tuning.
- `--no-enable-prefix-caching`: workload/benchmark cache-policy tuning.
- `VLLM_ENGINE_ITERATION_TIMEOUT_S`: operational runtime timeout.
These settings may still appear in validated hardware-specific overrides. Tune
them at deployment time based on platform resources, workload shape, and
latency/throughput goals.
## References
- [Model card](https://huggingface.co/Qwen/Qwen3-VL-30B-A3B-Instruct)
- [vLLM CPU installation](https://docs.vllm.ai/en/latest/getting_started/installation/cpu/)
recipes/models/Qwen/Qwen3.5-0.8B.yaml
+133
View File
@@ -0,0 +1,133 @@
meta:
title: "Qwen3.5-0.8B"
slug: "qwen3.5-0.8b"
provider: "Qwen"
description: "Qwen3.5 tiny dense multimodal model (0.8B) — ultra-low-VRAM / edge serving with 262K context"
date_added: 2026-04-22
date_updated: 2026-09-14
tasks:
- multimodal
- text
performance_headline: "Tiny Qwen3.5 dense for edge / draft-model use"
related_recipes:
- "Qwen/Qwen3.5-2B"
hardware:
arc_pro_b60: verified
arc_pro_b70: verified
model:
model_id: "Qwen/Qwen3.5-0.8B"
min_vllm_version: "0.17.0"
default_frontend: rust
architecture: dense
parameter_count: "0.8B"
active_parameters: "0.8B"
context_length: 262144
base_args:
- "--trust-remote-code"
base_env: {}
features:
tool_calling:
description: "Enable automatic tool choice with Qwen3 Coder parser"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "qwen3_coder"
reasoning:
description: "Enable chain-of-thought reasoning with Qwen3 parser"
args:
- "--reasoning-parser"
- "qwen3"
text_only:
description: "Skip loading the vision encoder for text-only workloads."
args:
- "--language-model-only"
opt_in_features:
- text_only
variants:
default:
precision: bf16
vram_minimum_gb: 2
description: "Full precision BF16 — runs on any modern GPU"
compatible_strategies:
- single_node_tp
hardware_overrides:
xpu:
extra_args:
- "--enforce-eager"
strategy_overrides: {}
guide: |
## Overview
[Qwen3.5-0.8B](https://huggingface.co/Qwen/Qwen3.5-0.8B) is the smallest
member of the Qwen3.5 family — same hybrid gated delta networks architecture
and 262K context, at a size suited to edge devices or as a draft model for
speculative decoding with larger Qwen3.5 checkpoints.
## Prerequisites
- **vLLM version:** >= 0.17.0
- **Hardware:** any modern GPU (>=4 GB VRAM) or Intel Arc Pro B60/B70
### Install vLLM
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend=auto
```
### Docker
```bash
docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70)
```
## Launching the Server
```bash
vllm serve Qwen/Qwen3.5-0.8B \
--max-model-len 262144 \
--reasoning-parser qwen3
```
### Docker (Intel XPU B60 / B70)
Validated on 1× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the
official vLLM XPU image `vllm/vllm-openai-xpu:latest`.
```bash
docker run --device /dev/dri \
-v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
--privileged --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai-xpu:latest \
Qwen/Qwen3.5-0.8B \
--reasoning-parser qwen3 \
--enforce-eager
```
## Client Usage
```python
from openai import OpenAI
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
resp = client.chat.completions.create(
model="Qwen/Qwen3.5-0.8B",
messages=[{"role": "user", "content": "Hi!"}],
max_tokens=64,
)
print(resp.choices[0].message.content)
```
## References
- [Model card](https://huggingface.co/Qwen/Qwen3.5-0.8B)
- [Base checkpoint](https://huggingface.co/Qwen/Qwen3.5-0.8B-Base)
recipes/models/Qwen/Qwen3.5-122B-A10B.yaml
+270
View File
@@ -0,0 +1,270 @@
meta:
title: "Qwen3.5-122B-A10B"
slug: "qwen3.5-122b-a10b"
provider: "Qwen"
description: "Mid-size Qwen3.5 multimodal MoE (122B total / 10B active) with gated delta networks, 256 experts, and 262K context"
date_added: 2026-04-22
date_updated: 2026-10-07
tasks:
- multimodal
- text
performance_headline: "Qwen3.5 mid-tier MoE — fits on 4x H200 BF16 or 2x H200 FP8"
related_recipes:
- "Qwen/Qwen3.5-397B-A17B"
- "Qwen/Qwen3.5-35B-A3B"
hardware:
mi325x: verified
mi355x: verified
model:
model_id: "Qwen/Qwen3.5-122B-A10B"
min_vllm_version: "0.17.0"
default_frontend: rust
docker_image:
# Stable release the AMD numbers in the guide were measured on (2026-10-07).
amd: "vllm/vllm-openai-rocm:v0.31.0"
architecture: moe
parameter_count: "122B"
active_parameters: "10B"
context_length: 262144
base_args:
- "--trust-remote-code"
base_env: {}
features:
tool_calling:
description: "Enable automatic tool choice with Qwen3 Coder parser"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "qwen3_coder"
reasoning:
description: "Enable chain-of-thought reasoning with Qwen3 parser"
args:
- "--reasoning-parser"
- "qwen3"
spec_decoding:
description: "Multi-token prediction speculative decoding for lower latency"
args:
- "--speculative-config"
- '{"method":"mtp","num_speculative_tokens":1}'
text_only:
description: "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
args:
- "--language-model-only"
encoder_parallel:
description: "Run the vision encoder in data-parallel mode — avoids TP comm overhead on the small encoder. Mutually exclusive with text_only."
args:
- "--mm-encoder-tp-mode"
- "data"
long_context:
description: "YaRN RoPE scaling past the native 262,144-token window to ~1,010,000 tokens. Static YaRN holds the scaling factor constant regardless of input length, so leave this off unless you actually serve long prompts; for ~524K, halve the factor to 2.0."
args:
- "--hf-overrides"
- '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}'
- "--max-model-len"
- "1010000"
env:
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
opt_in_features:
- spec_decoding
- text_only
- long_context
variants:
default:
precision: bf16
vram_minimum_gb: 293
description: "Full precision BF16 — requires 4x H200 or equivalent"
fp8:
model_id: "Qwen/Qwen3.5-122B-A10B-FP8"
precision: fp8
vram_minimum_gb: 147
description: "Qwen official FP8 checkpoint — fits on 2x H200"
hardware_overrides:
amd:
# fp8 KV keeps MTP's gain flat across context on ROCm (with a bf16 KV cache
# MTP decode slows down as the prompt grows), and it is what the AMD numbers
# in the guide were measured with.
extra_args:
- "--kv-cache-dtype"
- "fp8"
nvfp4:
model_id: "RedHatAI/Qwen3.5-122B-A10B-NVFP4"
precision: nvfp4
vram_minimum_gb: 74
description: "RedHatAI compressed-tensors NVFP4 quantization for Blackwell GPUs"
extra_args:
- "--kv-cache-dtype"
- "fp8"
gptq_int4:
model_id: "Qwen/Qwen3.5-122B-A10B-GPTQ-Int4"
precision: int4
vram_minimum_gb: 74
description: "GPTQ Int4 checkpoint — single-GPU serving on 80GB hardware"
redhat_fp8:
model_id: "RedHatAI/Qwen3.5-122B-A10B-FP8-dynamic"
precision: fp8
label: "FP8 Dynamic (RedHat)"
vram_minimum_gb: 147
description: "RedHatAI FP8 (E4M3) with dynamic per-token activation quantization"
extra_args:
- "--kv-cache-dtype"
- "fp8"
compatible_strategies:
- single_node_tp
- single_node_tep
- single_node_dep
- multi_node_tp
- multi_node_dep
- multi_node_tep
- pd_cluster
hardware_overrides:
amd:
# AITER is opt-in on ROCm and the image does not set it; without these two the
# server runs ROCM_ATTN + the Triton FP8 MoE and single-stream decode of this
# hybrid GDN model drops 4x from ~200 to ~12k prompt tokens. With AITER on, the
# auto-selection still picks ROCM_ATTN, so name the backend. ROCM_AITER_FA rather
# than ROCM_AITER_UNIFIED_ATTN: on gfx950 with v0.31.0 the unified backend runs
# AITER's new Gluon kernel, ~20% slower and with MTP decode that slows down as
# the prompt grows. Verified 2026-10-07 on one MI325X and one MI355X with the FP8
# checkpoint (see the AMD section of the guide).
extra_args:
- "--attention-backend"
- "ROCM_AITER_FA"
extra_env:
VLLM_ROCM_USE_AITER: "1"
strategy_overrides:
pd_cluster:
prefill:
env:
VLLM_SSM_CONV_STATE_LAYOUT: "DS"
decode:
env:
VLLM_SSM_CONV_STATE_LAYOUT: "DS"
guide: |
## Overview
[Qwen3.5-122B-A10B](https://huggingface.co/Qwen/Qwen3.5-122B-A10B) is a mid-tier
member of the Qwen3.5 family, sharing the gated delta networks MoE architecture
with 122B total parameters and 10B activated per token (256 experts). It is
multimodal (vision + text) and natively supports 262K context.
## Prerequisites
- **vLLM version:** >= 0.17.0
- **Hardware (BF16):** 4x H200 or 8x H100
- **Hardware (FP8):** 2x H200 or 4x H100
- **Hardware (Int4):** single 80 GB GPU
### Install vLLM
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend=auto
```
## Launching the Server
### BF16 on 4xH200 (TP4)
```bash
vllm serve Qwen/Qwen3.5-122B-A10B \
--tensor-parallel-size 4 \
--max-model-len 262144 \
--reasoning-parser qwen3
```
### FP8 on 2xH200 (TP2)
```bash
vllm serve Qwen/Qwen3.5-122B-A10B-FP8 \
--tensor-parallel-size 2 \
--max-model-len 262144 \
--reasoning-parser qwen3
```
### Throughput-focused (text-only, EP)
```bash
vllm serve Qwen/Qwen3.5-122B-A10B-FP8 \
-dp 4 --enable-expert-parallel \
--language-model-only \
--reasoning-parser qwen3 \
--enable-prefix-caching
```
### MTP speculative decoding
```bash
vllm serve Qwen/Qwen3.5-122B-A10B-FP8 \
--tensor-parallel-size 2 \
--speculative-config '{"method": "mtp", "num_speculative_tokens": 1}' \
--reasoning-parser qwen3
```
### AMD (MI325X / MI355X, single GPU, FP8)
The FP8 checkpoint fits one MI300X-class GPU. Three settings matter on ROCm and none is on by
default: `VLLM_ROCM_USE_AITER=1` switches the MoE to the AITER FP8 kernels,
`--attention-backend ROCM_AITER_FA` keeps decode flat across context (the auto-selected
`ROCM_ATTN` decays with prompt length on this hybrid GDN model), and `--kv-cache-dtype fp8`
keeps MTP flat across context too.
```bash
VLLM_ROCM_USE_AITER=1 vllm serve Qwen/Qwen3.5-122B-A10B-FP8 \
--attention-backend ROCM_AITER_FA \
--kv-cache-dtype fp8 --max-num-seqs 64 \
--reasoning-parser qwen3
```
`--max-num-seqs 64` because the default exceeds the Mamba block budget on 192 GB. MTP
(`--speculative-config '{"method":"mtp","num_speculative_tokens":1}'`) works with the same
flags. Single-stream decode (tokens/s at ~200 / ~2k / ~12k / ~30k prompt tokens, 128 output
tokens, prefix caching off), measured 2026-10-07 on `vllm/vllm-openai-rocm:v0.31.0`:
| | MI325X | MI355X |
|---|---|---|
| AITER + `ROCM_AITER_FA` | 128 / 127 / 127 / 127 | 143 / 143 / 143 / 143 |
| + MTP | 184 / 169 / 172 / 164 | 204 / 193 / 195 / 190 |
This recipe used `ROCM_AITER_UNIFIED_ATTN` before v0.31.0. Avoid it on MI350X/MI355X with
v0.31.0: there it runs AITER's new gfx950 Gluon kernel, which decodes at 110 tokens/s and,
with MTP, falls from 200 to 72 tokens/s between ~200 and ~30k prompt tokens
([vllm-project/vllm#60408](https://github.com/vllm-project/vllm/issues/60408)). MI325X is
unaffected (126 tokens/s; 182 to 162 with MTP).
## Client Usage
```python
from openai import OpenAI
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
resp = client.chat.completions.create(
model="Qwen/Qwen3.5-122B-A10B",
messages=[{"role": "user", "content": "Summarize the gated delta networks paper."}],
max_tokens=512,
)
print(resp.choices[0].message.content)
```
## Troubleshooting
- **CUDA graph / Mamba cache size error:** reduce `--max-cudagraph-capture-size`
(default 512). See [vLLM PR #34571](https://github.com/vllm-project/vllm/pull/34571).
- **Disable reasoning:** add `--default-chat-template-kwargs '{"enable_thinking": false}'`.
- **Prefix Caching (Mamba):** currently experimental in "align" mode.
## References
- [Model card](https://huggingface.co/Qwen/Qwen3.5-122B-A10B)
- [FP8 checkpoint](https://huggingface.co/Qwen/Qwen3.5-122B-A10B-FP8)
- [GPTQ-Int4 checkpoint](https://huggingface.co/Qwen/Qwen3.5-122B-A10B-GPTQ-Int4)
- [Qwen3.5-397B-A17B recipe](../Qwen3.5-397B-A17B)
recipes/models/Qwen/Qwen3.5-27B.yaml
+209
View File
@@ -0,0 +1,209 @@
meta:
title: "Qwen3.5-27B"
slug: "qwen3.5-27b"
provider: "Qwen"
description: "Qwen3.5 dense multimodal model (27B) with gated delta networks hybrid attention, MTP, and 262K context"
date_added: 2026-04-22
date_updated: 2026-09-14
tasks:
- multimodal
- text
performance_headline: "Qwen3.5 flagship dense — single-GPU FP8 or 2x GPU or 2-4x Intel Arc Pro B60/B70 BF16"
related_recipes:
- "Qwen/Qwen3.5-397B-A17B"
- "Qwen/Qwen3.5-35B-A3B"
- "Qwen/Qwen3.5-9B"
hardware:
arc_pro_b60: verified
arc_pro_b70: verified
model:
model_id: "Qwen/Qwen3.5-27B"
min_vllm_version: "0.17.0"
default_frontend: rust
architecture: dense
parameter_count: "27B"
active_parameters: "27B"
context_length: 262144
base_args:
- "--trust-remote-code"
base_env: {}
features:
tool_calling:
description: "Enable automatic tool choice with Qwen3 Coder parser"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "qwen3_coder"
reasoning:
description: "Enable chain-of-thought reasoning with Qwen3 parser"
args:
- "--reasoning-parser"
- "qwen3"
spec_decoding:
description: "Multi-token prediction speculative decoding for lower latency"
args:
- "--speculative-config"
- '{"method":"mtp","num_speculative_tokens":1}'
text_only:
description: "Skip loading the vision encoder for text-only workloads — frees VRAM for KV cache. Mutually exclusive with encoder_parallel."
args:
- "--language-model-only"
encoder_parallel:
description: "Run the vision encoder in data-parallel mode — avoids TP comm overhead on the small encoder. Mutually exclusive with text_only."
args:
- "--mm-encoder-tp-mode"
- "data"
long_context:
description: "YaRN RoPE scaling past the native 262,144-token window to ~1,010,000 tokens. Static YaRN holds the scaling factor constant regardless of input length, so leave this off unless you actually serve long prompts; for ~524K, halve the factor to 2.0."
args:
- "--hf-overrides"
- '{"text_config": {"rope_parameters": {"mrope_interleaved": true, "mrope_section": [11, 11, 10], "rope_type": "yarn", "rope_theta": 10000000, "partial_rotary_factor": 0.25, "factor": 4.0, "original_max_position_embeddings": 262144}}}'
- "--max-model-len"
- "1010000"
env:
VLLM_ALLOW_LONG_MAX_MODEL_LEN: "1"
opt_in_features:
- spec_decoding
- text_only
- long_context
variants:
default:
precision: bf16
vram_minimum_gb: 65
description: "Full precision BF16 — fits on 1x H200, 2x H100, 4x Intel Arc Pro B60, 2x Intel Arc Pro B70"
fp8:
model_id: "Qwen/Qwen3.5-27B-FP8"
precision: fp8
vram_minimum_gb: 33
description: "Qwen official FP8 checkpoint — single 40 GB GPU"
gptq_int4:
model_id: "Qwen/Qwen3.5-27B-GPTQ-Int4"
precision: int4
vram_minimum_gb: 17
description: "GPTQ Int4 checkpoint — fits on a single 24 GB GPU"
compatible_strategies:
- single_node_tp
- multi_node_tp
hardware_overrides:
xpu:
extra_args:
- "--enforce-eager"
strategy_overrides: {}
guide: |
## Overview
[Qwen3.5-27B](https://huggingface.co/Qwen/Qwen3.5-27B) is the flagship dense
model of the Qwen3.5 family. It uses the same gated delta networks hybrid
attention as its MoE siblings, supports vision+text input, and natively serves
262K context. MTP (multi-token prediction) is supported out of the box for
low-latency decoding.
## Prerequisites
- **vLLM version:** >= 0.17.0
- **Hardware (BF16):** 1x H200, 2x H100, 4x Intel Arc Pro B60, 2x Intel Arc Pro B70
- **Hardware (FP8):** single 40 GB GPU (H100/H200/L40S)
- **Hardware (Int4):** single 24 GB GPU
### Install vLLM
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend=auto
```
### Docker
```bash
docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70)
```
## Launching the Server
### Single-GPU FP8
```bash
vllm serve Qwen/Qwen3.5-27B-FP8 \
--max-model-len 262144 \
--reasoning-parser qwen3
```
### BF16 on 2xH100 (TP2)
```bash
vllm serve Qwen/Qwen3.5-27B \
--tensor-parallel-size 2 \
--max-model-len 262144 \
--reasoning-parser qwen3
```
### MTP speculative decoding
```bash
vllm serve Qwen/Qwen3.5-27B-FP8 \
--speculative-config '{"method": "mtp", "num_speculative_tokens": 1}' \
--reasoning-parser qwen3
```
### Text-only (skip vision encoder)
```bash
vllm serve Qwen/Qwen3.5-27B-FP8 \
--language-model-only \
--reasoning-parser qwen3 \
--enable-prefix-caching
```
### Docker (Intel XPU B60 / B70)
Validated on 4× Intel Arc Pro B60 or 2x Intel Arc Pro B70 (B60 24 GB, B70 32 GB per card) with the
official vLLM XPU image `vllm/vllm-openai-xpu:latest`.
```bash
docker run --device /dev/dri \
-v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
--privileged --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai-xpu:latest \
Qwen/Qwen3.5-27B \
--reasoning-parser qwen3 \
--tensor-parallel-size 4 \
--max-model-len 8192 \
--enforce-eager
```
## Client Usage
```python
from openai import OpenAI
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
resp = client.chat.completions.create(
model="Qwen/Qwen3.5-27B",
messages=[{"role": "user", "content": "Write a haiku about gated delta networks."}],
max_tokens=256,
)
print(resp.choices[0].message.content)
```
## Troubleshooting
- **CUDA graph / Mamba cache size error:** reduce `--max-cudagraph-capture-size`
(default 512). See [vLLM PR #34571](https://github.com/vllm-project/vllm/pull/34571).
- **Disable reasoning:** add `--default-chat-template-kwargs '{"enable_thinking": false}'`.
- **Prefix Caching (Mamba):** currently experimental in "align" mode.
## References
- [Model card](https://huggingface.co/Qwen/Qwen3.5-27B)
- [FP8 checkpoint](https://huggingface.co/Qwen/Qwen3.5-27B-FP8)
- [GPTQ-Int4 checkpoint](https://huggingface.co/Qwen/Qwen3.5-27B-GPTQ-Int4)
- [Qwen3.5-397B-A17B recipe](../Qwen3.5-397B-A17B)
recipes/models/Qwen/Qwen3.5-2B.yaml
+140
View File
@@ -0,0 +1,140 @@
meta:
title: "Qwen3.5-2B"
slug: "qwen3.5-2b"
provider: "Qwen"
description: "Qwen3.5 mini dense multimodal model (2B) — edge / low-VRAM serving with 262K context"
date_added: 2026-04-22
date_updated: 2026-09-14
tasks:
- multimodal
- text
performance_headline: "Edge-scale Qwen3.5 dense — fits on 8 GB GPUs or 1x Intel Arc Pro B60/B70"
related_recipes:
- "Qwen/Qwen3.5-4B"
- "Qwen/Qwen3.5-0.8B"
hardware:
arc_pro_b60: verified
arc_pro_b70: verified
model:
model_id: "Qwen/Qwen3.5-2B"
min_vllm_version: "0.17.0"
default_frontend: rust
architecture: dense
parameter_count: "2B"
active_parameters: "2B"
context_length: 262144
base_args:
- "--trust-remote-code"
base_env: {}
features:
tool_calling:
description: "Enable automatic tool choice with Qwen3 Coder parser"
args:
- "--enable-auto-tool-choice"
- "--tool-call-parser"
- "qwen3_coder"
reasoning:
description: "Enable chain-of-thought reasoning with Qwen3 parser"
args:
- "--reasoning-parser"
- "qwen3"
spec_decoding:
description: "Multi-token prediction speculative decoding for lower latency"
args:
- "--speculative-config"
- '{"method":"mtp","num_speculative_tokens":1}'
text_only:
description: "Skip loading the vision encoder for text-only workloads."
args:
- "--language-model-only"
opt_in_features:
- spec_decoding
- text_only
variants:
default:
precision: bf16
vram_minimum_gb: 5
description: "Full precision BF16 — fits on an 8 GB GPU or Intel Arc Pro B60/B70"
compatible_strategies:
- single_node_tp
hardware_overrides:
xpu:
extra_args:
- "--enforce-eager"
strategy_overrides: {}
guide: |
## Overview
[Qwen3.5-2B](https://huggingface.co/Qwen/Qwen3.5-2B) is a miniature dense
Qwen3.5 model — the full gated delta networks architecture, vision encoder,
and 262K context, in a form small enough for 8 GB consumer GPUs or edge
inference.
## Prerequisites
- **vLLM version:** >= 0.17.0
- **Hardware:** single 8 GB GPU or Intel Arc Pro B60/B70
### Install vLLM
```bash
uv venv
source .venv/bin/activate
uv pip install -U vllm --torch-backend=auto
```
### Docker
```bash
docker pull vllm/vllm-openai-xpu:latest # Intel XPU (B60 / B70)
```
## Launching the Server
```bash
vllm serve Qwen/Qwen3.5-2B \
--max-model-len 262144 \
--reasoning-parser qwen3
```
### Docker (Intel XPU B60 / B70)
Validated on 1× Intel Arc Pro B60 / B70 (B60 24 GB, B70 32 GB per card) with the
official vLLM XPU image `vllm/vllm-openai-xpu:latest`.
```bash
docker run --device /dev/dri \
-v /dev/dri/by-path:/dev/dri/by-path --shm-size=16g \
--privileged --ipc=host -p 8000:8000 \
-v ~/.cache/huggingface:/root/.cache/huggingface \
vllm/vllm-openai-xpu:latest \
Qwen/Qwen3.5-2B \
--reasoning-parser qwen3 \
--enforce-eager
```
## Client Usage
```python
from openai import OpenAI
client = OpenAI(api_key="EMPTY", base_url="http://localhost:8000/v1")
resp = client.chat.completions.create(
model="Qwen/Qwen3.5-2B",
messages=[{"role": "user", "content": "Hi!"}],
max_tokens=64,
)
print(resp.choices[0].message.content)
```
## References
- [Model card](https://huggingface.co/Qwen/Qwen3.5-2B)
- [Base checkpoint](https://huggingface.co/Qwen/Qwen3.5-2B-Base)

Some files were not shown because too many files have changed in this diff Show More