Compare commits
95
Commits
be95edb7f6
..
main
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
ff01a6b14f | ||
|
|
fe8f2140ee | ||
|
|
e80b6c642f | ||
|
|
371745ebd6 | ||
|
|
072c0886d5 | ||
|
|
6b24947628 | ||
|
|
494663940b | ||
|
|
3c13653738 | ||
|
|
953fa442fa | ||
|
|
3f1b04e045 | ||
|
|
2365177ddd | ||
|
|
1b2a666391 | ||
|
|
83c9a547bf | ||
|
|
b510857d37 | ||
|
|
7c9042c05a | ||
|
|
da9134bc89 | ||
|
|
b2456c622b | ||
|
|
03ee2e134d | ||
|
|
9c778e9ff8 | ||
|
|
7879f3687b | ||
|
|
255f14599e | ||
|
|
03e8300d2c | ||
|
|
5c1c903501 | ||
|
|
bafb263664 | ||
|
|
0c5dd18e7a | ||
|
|
763e3f7c91 | ||
|
|
e744eacf68 | ||
|
|
c0b6f74bee | ||
|
|
c61a1e73c6 | ||
|
|
237d99b4af | ||
|
|
2d8efee9a7 | ||
|
|
21e09ea959 | ||
|
|
837e87e1c5 | ||
|
|
4e964b2623 | ||
|
|
ec8b20fbf0 | ||
|
|
2b85d9cb65 | ||
|
|
026f6ab268 | ||
|
|
b515648641 | ||
|
|
f98ae6a9fd | ||
|
|
2067297b2f | ||
|
|
3bf25c5097 | ||
|
|
e56b1603af | ||
|
|
eb8de8d1a1 | ||
|
|
b05ede5616 | ||
|
|
61bb590877 | ||
|
|
c3de0b37e2 | ||
|
|
08a65e287e | ||
|
|
ad2688ab30 | ||
|
|
5be4266f5c | ||
|
|
90d6d871d6 | ||
|
|
d74f4180cd | ||
|
|
39983a5e7a | ||
|
|
41b8c15809 | ||
|
|
a0e7137c4b | ||
|
|
7b6310c109 | ||
|
|
556ac15eb9 | ||
|
|
dcd07967ca | ||
|
|
ce49c47cbb | ||
|
|
ce8853d27a | ||
|
|
71c20f73f2 | ||
|
|
b3a7e2ff6e | ||
|
|
6b6a6c5c98 | ||
|
|
f8112704d5 | ||
|
|
f019e3e105 | ||
|
|
783380af39 | ||
|
|
be25977532 | ||
|
|
523d74e2ac | ||
|
|
a94c315711 | ||
|
|
b544587408 | ||
|
|
2268d83bd9 | ||
|
|
28e92eeff2 | ||
|
|
b088eba944 | ||
|
|
596e530675 | ||
|
|
153a17166e | ||
|
|
9731f30a03 | ||
|
|
68ebea789a | ||
|
|
cc2e4fecf2 | ||
|
|
9eb7fb9157 | ||
|
|
663e535640 | ||
|
|
5bf3d9f665 | ||
|
|
307c5844ad | ||
|
|
3dc6def360 | ||
|
|
e7a5bdb454 | ||
|
|
ae8f41238c | ||
|
|
cffc0cfa31 | ||
|
|
706791d516 | ||
|
|
76081b9ee4 | ||
|
|
1a5b187066 | ||
|
|
8e25bbca19 | ||
|
|
8a6c0449b3 | ||
|
|
de7eeaa87b | ||
|
|
64175d3827 | ||
|
|
4ab99ff136 | ||
|
|
024c5188bf | ||
|
|
9a16d43ef0 |
@@ -31,6 +31,9 @@ Requests: `[N*]Q[@K][c][:D]`, comma-separated, per DP rank.
|
||||
| `2048@10240` | prefill after an 8192-token prefix-cache hit |
|
||||
| `2048@10240c` | chunked prefill: the prefix came from earlier chunks |
|
||||
|
||||
Verify cases require `--speculative-config` with `num_speculative_tokens`
|
||||
at least as large as the case's draft count; insufficient capacity is rejected.
|
||||
|
||||
Hit vs chunk only differs on models with SWA bounded replay (DSV4.1): a hit
|
||||
also recomputes the last window (128 tokens) of the prefix.
|
||||
|
||||
@@ -48,6 +51,46 @@ Key flags (plus all `EngineArgs`):
|
||||
|
||||
Read `median_us` (each replay is already the max over ranks).
|
||||
|
||||
## Kernel breakdown
|
||||
|
||||
Add `--profile-dir DIR` to see where the time goes. After timing, each case is
|
||||
replayed 3 more times under `torch.profiler`; every rank writes
|
||||
`DIR/<case>.rank<r>.kernels.json` (GPU time per kernel per forward, sorted)
|
||||
and `DIR/<case>.rank<r>.trace.json.gz` (open in Perfetto). Rank 0's top
|
||||
kernels are printed and included in `--output-json`.
|
||||
|
||||
```bash
|
||||
python -m vllm.benchmarks.forward.profile DIR/decode.rank0.kernels.json
|
||||
# Before/after a change, same case: per-kernel deltas, largest first.
|
||||
python -m vllm.benchmarks.forward.profile base/decode.rank0.kernels.json new/decode.rank0.kernels.json
|
||||
```
|
||||
|
||||
- Kernels are grouped by name without template or function arguments; the
|
||||
JSON keeps one full name per group in `example`.
|
||||
- Only the timed forward is included: the in-graph KV restore and rank
|
||||
rendezvous are excluded (`pre_forward_us`). `span_us` (first to last
|
||||
forward kernel) should be close to `median_us`, a bit higher from profiler
|
||||
overhead; `busy_us` can exceed it when streams overlap.
|
||||
- Compare the same rank across runs: under EP, ranks differ (e.g. collective
|
||||
wait time shows up inside communication kernels).
|
||||
|
||||
For Nsight Systems (NVTX, NCCL, CPU side, GPU metrics), skip `--profile-dir`:
|
||||
the two can't share CUPTI. Each case's 3 profiled replays run between
|
||||
`cudaProfilerStart`/`Stop` in an NVTX range `forward/<case>`, so a capture range
|
||||
records only them, one report per case (and rank):
|
||||
|
||||
```bash
|
||||
nsys profile --capture-range=cudaProfilerApi --capture-range-end=repeat \
|
||||
--cuda-graph-trace=node -o prof vllm bench forward ... # prof.1, prof.2, ...
|
||||
torchrun --nproc-per-node=4 --no-python nsys profile ... -o prof_rank%q{RANK} \
|
||||
python -m vllm.entrypoints.cli.main bench forward ...
|
||||
nsys stats --report cuda_gpu_kern_sum --format csv -o kern prof.1.nsys-rep
|
||||
```
|
||||
|
||||
Keep `--capture-range-end=repeat`: nsys's default stops after the first case
|
||||
and leaves the bench hanging. `nsys stats` also counts the KV restore copies
|
||||
that the JSON summary excludes.
|
||||
|
||||
## Caveats
|
||||
|
||||
- **Dummy weights lie for data-dependent kernels.** Dummy gates route every
|
||||
|
||||
@@ -5,9 +5,12 @@ description: Find vLLM deployment recipes and configs for LLMs and Hugging Face
|
||||
|
||||
# Model Recipes
|
||||
|
||||
Model serving recipes are maintained in the
|
||||
[models/ directory of vllm-project/recipes](https://github.com/vllm-project/recipes/tree/main/models).
|
||||
The published catalog is at
|
||||
Model serving recipes are maintained in this repository's
|
||||
[recipes/models/ directory](../../../recipes/models). Omni (image, video,
|
||||
audio) recipes stay in vllm-project/recipes for now (moving to vllm-omni),
|
||||
and Ascend NPU content is inline in these recipes (moving to vllm-ascend
|
||||
overlays). The published catalog, rendered by
|
||||
[vllm-project/recipes](https://github.com/vllm-project/recipes), is at
|
||||
[recipes.vllm.ai](https://recipes.vllm.ai).
|
||||
|
||||
## Usage notes
|
||||
@@ -146,6 +149,9 @@ uploads, untested hardware claims, or non-standard flags that only work for one
|
||||
setup. Before opening a pull request, confirm that the user ran the
|
||||
configuration and can share the results.
|
||||
|
||||
Follow the recipes repository's
|
||||
Edit the YAML under [recipes/models/](../../../recipes/models) in this
|
||||
repository, following the recipes repository's
|
||||
[add-recipe skill](https://github.com/vllm-project/recipes/blob/main/.agents/skills/add-recipe/SKILL.md)
|
||||
in a checkout of `vllm-project/recipes` or the user's intended fork.
|
||||
for the schema. Validate and preview it from a checkout of
|
||||
`vllm-project/recipes` pointed at this directory, as
|
||||
[recipes/README.md](../../../recipes/README.md) describes.
|
||||
|
||||
+208
-473
@@ -703,385 +703,6 @@ steps:
|
||||
DOCKER_BUILDKIT: "1"
|
||||
DOCKERHUB_USERNAME: "vllmbot"
|
||||
|
||||
# =============================================================================
|
||||
# ROCm Release Pipeline (x86_64 only)
|
||||
# =============================================================================
|
||||
#
|
||||
# vLLM version is determined by the Buildkite checkout (like CUDA pipeline).
|
||||
# To build a specific version, trigger the build from that branch/tag.
|
||||
#
|
||||
# Environment variables for ROCm builds (set via Buildkite UI or schedule):
|
||||
#
|
||||
# Note: ROCm version is determined by BASE_IMAGE in docker/Dockerfile.rocm_base
|
||||
#
|
||||
# =============================================================================
|
||||
|
||||
- group: "Build ROCm Wheel / Image "
|
||||
key: "build-rocm-wheel-image"
|
||||
depends_on: ~
|
||||
steps:
|
||||
# ROCm Job 1: Build ROCm Base Wheels (with S3 caching)
|
||||
- label: ":rocm: Build ROCm Base Image & Wheels"
|
||||
id: build-rocm-base-wheels
|
||||
depends_on: ~
|
||||
agents:
|
||||
queue: cpu_queue_release
|
||||
commands:
|
||||
- |
|
||||
set -euo pipefail
|
||||
|
||||
# Generate cache key
|
||||
CACHE_KEY=$$(.buildkite/scripts/cache-rocm-base-wheels.sh key)
|
||||
ECR_CACHE_TAG="public.ecr.aws/q9t5s3a7/vllm-release-repo:$${CACHE_KEY}-rocm-base"
|
||||
|
||||
echo "========================================"
|
||||
echo "ROCm Base Build Configuration"
|
||||
echo "========================================"
|
||||
echo " CACHE_KEY: $${CACHE_KEY}"
|
||||
echo " ECR_CACHE_TAG: $${ECR_CACHE_TAG}"
|
||||
echo "========================================"
|
||||
|
||||
# Login to ECR
|
||||
aws ecr-public get-login-password --region us-east-1 | \
|
||||
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
|
||||
|
||||
IMAGE_EXISTS=false
|
||||
WHEELS_EXIST=false
|
||||
|
||||
# Check ECR for Docker image
|
||||
|
||||
if docker manifest inspect "$${ECR_CACHE_TAG}" > /dev/null 2>&1; then
|
||||
IMAGE_EXISTS=true
|
||||
echo "ECR image cache HIT"
|
||||
fi
|
||||
|
||||
# Check S3 for wheels
|
||||
WHEEL_CACHE_STATUS=$(.buildkite/scripts/cache-rocm-base-wheels.sh check)
|
||||
if [ "$${WHEEL_CACHE_STATUS}" = "hit" ]; then
|
||||
WHEELS_EXIST=true
|
||||
echo "S3 wheels cache HIT"
|
||||
fi
|
||||
|
||||
|
||||
# Scenario 1: Both cached (best case)
|
||||
if [ "$${IMAGE_EXISTS}" = "true" ] && [ "$${WHEELS_EXIST}" = "true" ]; then
|
||||
echo ""
|
||||
echo "FULL CACHE HIT - Reusing both image and wheels"
|
||||
echo ""
|
||||
|
||||
# Download wheels
|
||||
.buildkite/scripts/cache-rocm-base-wheels.sh download
|
||||
|
||||
# Save ECR tag for downstream jobs
|
||||
buildkite-agent meta-data set "rocm-base-image-tag" "$${ECR_CACHE_TAG}"
|
||||
|
||||
# Scenario 2: Full rebuild needed
|
||||
else
|
||||
echo ""
|
||||
echo " CACHE MISS - Building from scratch..."
|
||||
echo ""
|
||||
|
||||
# Build full base image and push to ECR
|
||||
DOCKER_BUILDKIT=1 docker buildx build \
|
||||
--file docker/Dockerfile.rocm_base \
|
||||
--tag "$${ECR_CACHE_TAG}" \
|
||||
--build-arg USE_SCCACHE=1 \
|
||||
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
|
||||
--build-arg SCCACHE_REGION_NAME=us-west-2 \
|
||||
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
|
||||
--push \
|
||||
.
|
||||
|
||||
# Build wheel extraction stage
|
||||
DOCKER_BUILDKIT=1 docker buildx build \
|
||||
--file docker/Dockerfile.rocm_base \
|
||||
--tag rocm-base-debs:$${BUILDKITE_BUILD_NUMBER} \
|
||||
--target debs_wheel_release \
|
||||
--build-arg USE_SCCACHE=1 \
|
||||
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
|
||||
--build-arg SCCACHE_REGION_NAME=us-west-2 \
|
||||
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
|
||||
--load \
|
||||
.
|
||||
|
||||
# Extract and upload wheels
|
||||
mkdir -p artifacts/rocm-base-wheels
|
||||
cid=$(docker create rocm-base-debs:$${BUILDKITE_BUILD_NUMBER})
|
||||
docker cp $${cid}:/app/debs/. artifacts/rocm-base-wheels/
|
||||
docker rm $${cid}
|
||||
|
||||
.buildkite/scripts/cache-rocm-base-wheels.sh upload
|
||||
|
||||
# Cache base docker image to ECR
|
||||
docker push "$${ECR_CACHE_TAG}"
|
||||
|
||||
buildkite-agent meta-data set "rocm-base-image-tag" "$${ECR_CACHE_TAG}"
|
||||
|
||||
echo ""
|
||||
echo " Build complete - Image and wheels cached"
|
||||
fi
|
||||
|
||||
artifact_paths:
|
||||
- "artifacts/rocm-base-wheels/*.whl"
|
||||
env:
|
||||
DOCKER_BUILDKIT: "1"
|
||||
S3_BUCKET: "vllm-wheels"
|
||||
|
||||
# ROCm Job 2: Build vLLM ROCm Wheel
|
||||
- label: ":python: Build vLLM ROCm Wheel - x86_64"
|
||||
id: build-rocm-vllm-wheel
|
||||
depends_on:
|
||||
- step: build-rocm-base-wheels
|
||||
allow_failure: false
|
||||
agents:
|
||||
queue: cpu_queue_release
|
||||
timeout_in_minutes: 180
|
||||
commands:
|
||||
# Download artifacts and prepare Docker image
|
||||
- |
|
||||
set -euo pipefail
|
||||
|
||||
# Ensure git tags are up-to-date (Buildkite's default fetch doesn't update tags)
|
||||
# This fixes version detection when tags are moved/force-pushed
|
||||
echo "Fetching latest tags from origin..."
|
||||
git fetch --tags --force origin
|
||||
|
||||
# Log tag information for debugging version detection
|
||||
echo "========================================"
|
||||
echo "Git Tag Verification"
|
||||
echo "========================================"
|
||||
echo "Current HEAD: $(git rev-parse HEAD)"
|
||||
echo "git describe --tags: $(git describe --tags 2>/dev/null || echo 'No tags found')"
|
||||
echo ""
|
||||
echo "Recent tags (pointing to commits near HEAD):"
|
||||
git tag -l --sort=-creatordate | head -5
|
||||
echo "setuptools_scm version detection:"
|
||||
pip install -q setuptools_scm 2>/dev/null || true
|
||||
python3 -c "import setuptools_scm; print(' Detected version:', setuptools_scm.get_version())" 2>/dev/null || echo " (setuptools_scm not available in this environment)"
|
||||
echo "========================================"
|
||||
|
||||
# Download wheel artifacts from current build
|
||||
echo "Downloading wheel artifacts from current build"
|
||||
buildkite-agent artifact download "artifacts/rocm-base-wheels/*.whl" .
|
||||
|
||||
# Get ECR image tag from metadata (set by build-rocm-base-wheels)
|
||||
ECR_IMAGE_TAG="$$(buildkite-agent meta-data get rocm-base-image-tag 2>/dev/null || echo '')"
|
||||
if [ -z "$${ECR_IMAGE_TAG}" ]; then
|
||||
echo "ERROR: rocm-base-image-tag metadata not found"
|
||||
echo "This should have been set by the build-rocm-base-wheels job"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Pulling base Docker image from ECR: $${ECR_IMAGE_TAG}"
|
||||
|
||||
# Login to ECR
|
||||
aws ecr-public get-login-password --region us-east-1 | \
|
||||
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
|
||||
|
||||
# Pull base Docker image from ECR
|
||||
docker pull "$${ECR_IMAGE_TAG}"
|
||||
|
||||
echo "Loaded base image: $${ECR_IMAGE_TAG}"
|
||||
|
||||
# Prepare base wheels for Docker build context
|
||||
mkdir -p docker/context/base-wheels
|
||||
touch docker/context/base-wheels/.keep
|
||||
cp artifacts/rocm-base-wheels/*.whl docker/context/base-wheels/
|
||||
echo "Base wheels for vLLM build:"
|
||||
ls -lh docker/context/base-wheels/
|
||||
|
||||
echo "========================================"
|
||||
echo "Building vLLM wheel with:"
|
||||
echo " BUILDKITE_COMMIT: $${BUILDKITE_COMMIT}"
|
||||
echo " BUILDKITE_BRANCH: $${BUILDKITE_BRANCH}"
|
||||
echo " BASE_IMAGE: $${ECR_IMAGE_TAG}"
|
||||
echo "========================================"
|
||||
|
||||
# Build vLLM wheel using local checkout (REMOTE_VLLM=0)
|
||||
DOCKER_BUILDKIT=1 docker build \
|
||||
--file docker/Dockerfile.rocm \
|
||||
--target export_vllm_wheel_release \
|
||||
--output type=local,dest=rocm-dist \
|
||||
--build-arg BASE_IMAGE="$${ECR_IMAGE_TAG}" \
|
||||
--build-arg REMOTE_VLLM=0 \
|
||||
--build-arg GIT_REPO_CHECK=1 \
|
||||
--build-arg USE_SCCACHE=1 \
|
||||
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
|
||||
--build-arg SCCACHE_REGION_NAME=us-west-2 \
|
||||
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
|
||||
.
|
||||
echo "Built vLLM wheel:"
|
||||
ls -lh rocm-dist/*.whl
|
||||
# Copy wheel to artifacts directory
|
||||
mkdir -p artifacts/rocm-vllm-wheel
|
||||
cp rocm-dist/*.whl artifacts/rocm-vllm-wheel/
|
||||
echo "Final vLLM wheel:"
|
||||
ls -lh artifacts/rocm-vllm-wheel/
|
||||
artifact_paths:
|
||||
- "artifacts/rocm-vllm-wheel/*.whl"
|
||||
env:
|
||||
DOCKER_BUILDKIT: "1"
|
||||
S3_BUCKET: "vllm-wheels"
|
||||
|
||||
# ROCm Job 3: Upload Wheels to S3
|
||||
- label: ":s3: Upload ROCm Wheels to S3"
|
||||
id: upload-rocm-wheels
|
||||
depends_on:
|
||||
- step: build-rocm-vllm-wheel
|
||||
allow_failure: false
|
||||
agents:
|
||||
queue: cpu_queue_release
|
||||
timeout_in_minutes: 60
|
||||
commands:
|
||||
# Download all wheel artifacts and run upload
|
||||
- |
|
||||
set -euo pipefail
|
||||
|
||||
# Download artifacts from current build
|
||||
echo "Downloading artifacts from current build"
|
||||
buildkite-agent artifact download "artifacts/rocm-base-wheels/*.whl" .
|
||||
buildkite-agent artifact download "artifacts/rocm-vllm-wheel/*.whl" .
|
||||
|
||||
# # Run upload script
|
||||
bash .buildkite/scripts/upload-rocm-wheels.sh
|
||||
env:
|
||||
DOCKER_BUILDKIT: "1"
|
||||
S3_BUCKET: "vllm-wheels"
|
||||
|
||||
# ROCm Job 4: Annotate ROCm Wheel Release
|
||||
- label: ":memo: Annotate ROCm wheel release"
|
||||
id: annotate-rocm-release
|
||||
depends_on:
|
||||
- upload-rocm-wheels
|
||||
agents:
|
||||
queue: cpu_queue_release
|
||||
commands:
|
||||
- "bash .buildkite/scripts/annotate-rocm-release.sh"
|
||||
env:
|
||||
S3_BUCKET: "vllm-wheels"
|
||||
|
||||
# ROCm Job 5: Generate Root Index for ROCm Wheels (for release only)
|
||||
# This is the job to create https://wheels.vllm.ai/rocm/ index allowing
|
||||
# users to install with `uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/`
|
||||
- block: "Generate Root Index for ROCm Wheels for Release"
|
||||
key: block-generate-root-index-rocm-wheels
|
||||
depends_on: upload-rocm-wheels
|
||||
|
||||
- label: ":package: Generate Root Index for ROCm Wheels for Release"
|
||||
depends_on: block-generate-root-index-rocm-wheels
|
||||
id: generate-root-index-rocm-wheels
|
||||
agents:
|
||||
queue: cpu_queue_release
|
||||
commands:
|
||||
- "bash tools/vllm-rocm/generate-rocm-wheels-root-index.sh"
|
||||
env:
|
||||
S3_BUCKET: "vllm-wheels"
|
||||
VARIANT: "rocm723"
|
||||
|
||||
# ROCm Job 6: Build ROCm Release Docker Image
|
||||
- label: ":docker: Build release image - x86_64 - ROCm"
|
||||
id: build-rocm-release-image
|
||||
depends_on:
|
||||
- step: block-build-release-images
|
||||
allow_failure: true
|
||||
- step: build-rocm-base-wheels
|
||||
allow_failure: false
|
||||
agents:
|
||||
queue: cpu_queue_release
|
||||
timeout_in_minutes: 60
|
||||
commands:
|
||||
- |
|
||||
set -euo pipefail
|
||||
|
||||
# Login to ECR
|
||||
aws ecr-public get-login-password --region us-east-1 | \
|
||||
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
|
||||
|
||||
# Get ECR image tag from metadata (set by build-rocm-base-wheels)
|
||||
ECR_IMAGE_TAG="$$(buildkite-agent meta-data get rocm-base-image-tag 2>/dev/null || echo '')"
|
||||
if [ -z "$${ECR_IMAGE_TAG}" ]; then
|
||||
echo "ERROR: rocm-base-image-tag metadata not found"
|
||||
echo "This should have been set by the build-rocm-base-wheels job"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Pulling base Docker image from ECR: $${ECR_IMAGE_TAG}"
|
||||
|
||||
# Pull base Docker image from ECR
|
||||
docker pull "$${ECR_IMAGE_TAG}"
|
||||
|
||||
echo "Loaded base image: $${ECR_IMAGE_TAG}"
|
||||
|
||||
# Pass the base image ECR tag to downstream steps (nightly publish)
|
||||
buildkite-agent meta-data set "rocm-base-ecr-tag" "$${ECR_IMAGE_TAG}"
|
||||
|
||||
echo "========================================"
|
||||
echo "Building vLLM ROCm release image with:"
|
||||
echo " BASE_IMAGE: $${ECR_IMAGE_TAG}"
|
||||
echo " BUILDKITE_COMMIT: $${BUILDKITE_COMMIT}"
|
||||
echo "========================================"
|
||||
|
||||
# Build vLLM ROCm release image using cached base
|
||||
DOCKER_BUILDKIT=1 docker build \
|
||||
--build-arg max_jobs=16 \
|
||||
--build-arg BASE_IMAGE="$${ECR_IMAGE_TAG}" \
|
||||
--build-arg USE_SCCACHE=1 \
|
||||
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
|
||||
--build-arg SCCACHE_REGION_NAME=us-west-2 \
|
||||
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
|
||||
--build-arg INSTALL_LMCACHE=true \
|
||||
--tag public.ecr.aws/q9t5s3a7/vllm-release-repo:$${BUILDKITE_COMMIT}-rocm \
|
||||
--target vllm-openai \
|
||||
--progress plain \
|
||||
-f docker/Dockerfile.rocm .
|
||||
|
||||
# Push to ECR
|
||||
docker push public.ecr.aws/q9t5s3a7/vllm-release-repo:$${BUILDKITE_COMMIT}-rocm
|
||||
|
||||
echo ""
|
||||
echo " Successfully built and pushed ROCm release image"
|
||||
echo " Image: public.ecr.aws/q9t5s3a7/vllm-release-repo:$${BUILDKITE_COMMIT}-rocm"
|
||||
echo ""
|
||||
env:
|
||||
DOCKER_BUILDKIT: "1"
|
||||
S3_BUCKET: "vllm-wheels"
|
||||
|
||||
- label: "Publish nightly XPU image to DockerHub"
|
||||
depends_on:
|
||||
- create-manifest-xpu
|
||||
if: build.env("NIGHTLY") == "1"
|
||||
agents:
|
||||
queue: small_cpu_queue_release
|
||||
commands:
|
||||
- "bash .buildkite/scripts/xpu/push-nightly-builds-xpu.sh"
|
||||
- "bash .buildkite/scripts/cleanup-nightly-builds.sh nightly- vllm/vllm-openai-xpu"
|
||||
plugins:
|
||||
- docker-login#v3.0.0:
|
||||
username: vllmbot
|
||||
password-env: DOCKERHUB_TOKEN
|
||||
env:
|
||||
DOCKER_BUILDKIT: "1"
|
||||
DOCKERHUB_USERNAME: "vllmbot"
|
||||
|
||||
- label: "Publish nightly ROCm image to DockerHub"
|
||||
depends_on:
|
||||
- build-rocm-release-image
|
||||
if: build.env("NIGHTLY") == "1"
|
||||
agents:
|
||||
queue: small_cpu_queue_release
|
||||
commands:
|
||||
- "bash .buildkite/scripts/push-nightly-builds-rocm.sh"
|
||||
# Clean up old nightly builds (keep only last 14)
|
||||
- "bash .buildkite/scripts/cleanup-nightly-builds.sh nightly- vllm/vllm-openai-rocm"
|
||||
- "bash .buildkite/scripts/cleanup-nightly-builds.sh base-nightly- vllm/vllm-openai-rocm"
|
||||
plugins:
|
||||
- docker-login#v3.0.0:
|
||||
username: vllmbot
|
||||
password-env: DOCKERHUB_TOKEN
|
||||
env:
|
||||
DOCKER_BUILDKIT: "1"
|
||||
DOCKERHUB_USERNAME: "vllmbot"
|
||||
|
||||
- trigger: "perf-eval"
|
||||
label: "Trigger perf-eval (CUDA)"
|
||||
key: trigger-perf-eval-cuda
|
||||
@@ -1103,7 +724,7 @@ steps:
|
||||
key: trigger-perf-eval-rocm
|
||||
if: build.env("NIGHTLY") == "1" || build.env("RUN_PERF_EVAL") == "1"
|
||||
depends_on:
|
||||
- build-rocm-release-image
|
||||
- build-rocm100-release-image
|
||||
async: true
|
||||
soft_fail: true
|
||||
build:
|
||||
@@ -1112,138 +733,250 @@ steps:
|
||||
env:
|
||||
NIGHTLY: 1 # run all perf-eval workloads
|
||||
VLLM_COMMIT: "$BUILDKITE_COMMIT"
|
||||
VLLM_IMAGE_ROCM: "public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-rocm"
|
||||
VLLM_IMAGE_ROCM: "public.ecr.aws/q9t5s3a7/vllm-release-repo:$BUILDKITE_COMMIT-rocm100"
|
||||
|
||||
# =============================================================================
|
||||
# ROCk Release Pipeline (x86_64 only)
|
||||
# ROCm Release Pipelines (x86_64 only)
|
||||
# =============================================================================
|
||||
#
|
||||
# Same shape as the ROCm pipeline above (Jobs 1 and 6 only), but built from
|
||||
# docker/Dockerfile.rock_base and docker/Dockerfile.rock, which install the
|
||||
# ROCm SDK from TheRock wheels instead of the ROCm apt repos.
|
||||
# One group per ROCm stack, identified by its base Dockerfile (ROCM_BASE_DOCKERFILE).
|
||||
# The scripts derive the ROCm version, wheel variant, cache keys and image tags from
|
||||
# it (.buildkite/scripts/rocm/stack.sh), so both groups share one set of step
|
||||
# definitions, anchored on the ROCm 10.0 group.
|
||||
#
|
||||
# Note: ROCm SDK version is determined by ROCM_SDK_VERSION in
|
||||
# docker/Dockerfile.rock_base
|
||||
# - ROCm 10.0 (TheRock, docker/Dockerfile.rocm{,_base}) is the default: it owns the
|
||||
# :nightly / :latest images and the root wheel index.
|
||||
# - ROCm 7.2 (docker/Dockerfile.rocm_72{,_base}) is kept for a transition period as
|
||||
# rocm72 images and wheels.
|
||||
#
|
||||
# vLLM version is determined by the Buildkite checkout (like the CUDA pipeline).
|
||||
# =============================================================================
|
||||
|
||||
- group: "Build ROCk Image"
|
||||
key: "build-rock-image"
|
||||
- group: "Build ROCm 10.0 Wheel / Image"
|
||||
key: "build-rocm100"
|
||||
depends_on: ~
|
||||
steps:
|
||||
# ROCk Job 1: Build ROCk Base Image (with ECR caching)
|
||||
- label: ":rocm: Build ROCk Base Image"
|
||||
id: build-rock-base-image
|
||||
- label: ":rocm: Build ROCm 10.0 Base Image & Wheels"
|
||||
id: build-rocm100-base
|
||||
depends_on: ~
|
||||
agents:
|
||||
agents: &rocm_build_agents
|
||||
queue: cpu_queue_release
|
||||
timeout_in_minutes: 360
|
||||
commands:
|
||||
env: &rocm100_env
|
||||
ROCM_BASE_DOCKERFILE: "docker/Dockerfile.rocm_base"
|
||||
ROCM_DEFAULT_ARG: "--default"
|
||||
DOCKER_BUILDKIT: "1"
|
||||
S3_BUCKET: "vllm-wheels"
|
||||
DOCKERHUB_USERNAME: "vllmbot"
|
||||
commands: &rocm_base_commands
|
||||
- |
|
||||
set -euo pipefail
|
||||
|
||||
# The SDK version, torch wheels and component branches all live in
|
||||
# the Dockerfile, so its hash is the cache key.
|
||||
CACHE_KEY=$$(sha256sum docker/Dockerfile.rock_base | cut -c1-16)
|
||||
ECR_CACHE_TAG="public.ecr.aws/q9t5s3a7/vllm-release-repo:$${CACHE_KEY}-rock-base"
|
||||
|
||||
echo "========================================"
|
||||
echo "ROCk Base Build Configuration"
|
||||
echo "========================================"
|
||||
echo " CACHE_KEY: $${CACHE_KEY}"
|
||||
echo " ECR_CACHE_TAG: $${ECR_CACHE_TAG}"
|
||||
echo "========================================"
|
||||
|
||||
# Login to ECR
|
||||
source .buildkite/scripts/rocm/stack.sh "$${ROCM_BASE_DOCKERFILE}"
|
||||
SCCACHE_ARGS=(--build-arg USE_SCCACHE=1 --build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache
|
||||
--build-arg SCCACHE_REGION_NAME=us-west-2 --build-arg SCCACHE_S3_NO_CREDENTIALS=0)
|
||||
aws ecr-public get-login-password --region us-east-1 | \
|
||||
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
|
||||
|
||||
if docker manifest inspect "$${ECR_CACHE_TAG}" > /dev/null 2>&1; then
|
||||
echo "ECR image cache HIT"
|
||||
# Base image, cached in ECR by the base Dockerfile's hash
|
||||
if docker manifest inspect "$${ROCM_STACK_ECR_BASE}" > /dev/null 2>&1; then
|
||||
echo "ECR image cache HIT: $${ROCM_STACK_ECR_BASE}"
|
||||
else
|
||||
echo "CACHE MISS - Building from scratch..."
|
||||
|
||||
DOCKER_BUILDKIT=1 docker buildx build \
|
||||
--file docker/Dockerfile.rock_base \
|
||||
--tag "$${ECR_CACHE_TAG}" \
|
||||
--build-arg USE_SCCACHE=1 \
|
||||
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
|
||||
--build-arg SCCACHE_REGION_NAME=us-west-2 \
|
||||
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
|
||||
--push \
|
||||
.
|
||||
docker buildx build --file "$${ROCM_BASE_DOCKERFILE}" --tag "$${ROCM_STACK_ECR_BASE}" \
|
||||
"$${SCCACHE_ARGS[@]}" --push .
|
||||
fi
|
||||
|
||||
# Save ECR tag for downstream jobs
|
||||
buildkite-agent meta-data set "rock-base-image-tag" "$${ECR_CACHE_TAG}"
|
||||
env:
|
||||
DOCKER_BUILDKIT: "1"
|
||||
# Base wheels, cached in S3 (see cache-rocm-base-wheels.sh)
|
||||
CACHE=.buildkite/scripts/cache-rocm-base-wheels.sh
|
||||
if [ "$$($${CACHE} "$${ROCM_BASE_DOCKERFILE}" check)" = "hit" ]; then
|
||||
$${CACHE} "$${ROCM_BASE_DOCKERFILE}" download
|
||||
else
|
||||
DEBS="rocm-base-debs:$${BUILDKITE_BUILD_NUMBER}-$${ROCM_STACK_VARIANT}"
|
||||
docker buildx build --file "$${ROCM_BASE_DOCKERFILE}" --tag "$${DEBS}" \
|
||||
--target debs_wheel_release "$${SCCACHE_ARGS[@]}" --load .
|
||||
mkdir -p "$${ROCM_STACK_BASE_WHEELS_DIR}"
|
||||
cid=$$(docker create "$${DEBS}")
|
||||
docker cp "$${cid}":/app/debs/. "$${ROCM_STACK_BASE_WHEELS_DIR}/"
|
||||
docker rm "$${cid}"
|
||||
$${CACHE} "$${ROCM_BASE_DOCKERFILE}" upload
|
||||
fi
|
||||
buildkite-agent artifact upload "$${ROCM_STACK_BASE_WHEELS_DIR}/*.whl"
|
||||
|
||||
# ROCk Job 2: Build ROCk Release Docker Image
|
||||
- label: ":docker: Build release image - x86_64 - ROCk"
|
||||
id: build-rock-release-image
|
||||
- label: ":python: Build vLLM ROCm 10.0 Wheel - x86_64"
|
||||
id: build-rocm100-vllm-wheel
|
||||
depends_on: build-rocm100-base
|
||||
agents: *rocm_build_agents
|
||||
timeout_in_minutes: 180
|
||||
env: *rocm100_env
|
||||
commands: &rocm_vllm_wheel_commands
|
||||
- |
|
||||
set -euo pipefail
|
||||
source .buildkite/scripts/rocm/stack.sh "$${ROCM_BASE_DOCKERFILE}"
|
||||
# Buildkite's default fetch doesn't update moved tags; version detection needs them
|
||||
git fetch --tags --force origin
|
||||
buildkite-agent artifact download "$${ROCM_STACK_BASE_WHEELS_DIR}/*.whl" .
|
||||
aws ecr-public get-login-password --region us-east-1 | \
|
||||
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
|
||||
docker pull "$${ROCM_STACK_ECR_BASE}"
|
||||
mkdir -p docker/context/base-wheels
|
||||
touch docker/context/base-wheels/.keep
|
||||
cp "$${ROCM_STACK_BASE_WHEELS_DIR}"/*.whl docker/context/base-wheels/
|
||||
docker build --file "$${ROCM_STACK_DOCKERFILE}" --target export_vllm_wheel_release \
|
||||
--output type=local,dest=rocm-dist \
|
||||
--build-arg BASE_IMAGE="$${ROCM_STACK_ECR_BASE}" --build-arg REMOTE_VLLM=0 \
|
||||
--build-arg GIT_REPO_CHECK=1 --build-arg USE_SCCACHE=1 \
|
||||
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
|
||||
--build-arg SCCACHE_REGION_NAME=us-west-2 --build-arg SCCACHE_S3_NO_CREDENTIALS=0 .
|
||||
mkdir -p "$${ROCM_STACK_VLLM_WHEEL_DIR}"
|
||||
cp rocm-dist/*.whl "$${ROCM_STACK_VLLM_WHEEL_DIR}/"
|
||||
buildkite-agent artifact upload "$${ROCM_STACK_VLLM_WHEEL_DIR}/*.whl"
|
||||
|
||||
- label: ":s3: Upload ROCm 10.0 Wheels to S3"
|
||||
id: upload-rocm100-wheels
|
||||
depends_on: build-rocm100-vllm-wheel
|
||||
agents: *rocm_build_agents
|
||||
timeout_in_minutes: 60
|
||||
env: *rocm100_env
|
||||
commands: &rocm_upload_commands
|
||||
- |
|
||||
set -euo pipefail
|
||||
source .buildkite/scripts/rocm/stack.sh "$${ROCM_BASE_DOCKERFILE}"
|
||||
buildkite-agent artifact download "$${ROCM_STACK_BASE_WHEELS_DIR}/*.whl" .
|
||||
buildkite-agent artifact download "$${ROCM_STACK_VLLM_WHEEL_DIR}/*.whl" .
|
||||
bash .buildkite/scripts/upload-rocm-wheels.sh "$${ROCM_BASE_DOCKERFILE}"
|
||||
|
||||
- label: ":memo: Annotate ROCm 10.0 wheel release"
|
||||
id: annotate-rocm100-release
|
||||
depends_on: upload-rocm100-wheels
|
||||
agents: *rocm_build_agents
|
||||
env: *rocm100_env
|
||||
commands: &rocm_annotate_commands
|
||||
- bash .buildkite/scripts/annotate-rocm-release.sh "$${ROCM_BASE_DOCKERFILE}" $${ROCM_DEFAULT_ARG}
|
||||
|
||||
# Generate https://wheels.vllm.ai/rocm/ (the default stack's variant), so users can
|
||||
# `uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/` (release only)
|
||||
- block: "Generate Root Index for ROCm Wheels for Release"
|
||||
key: block-generate-root-index-rocm-wheels
|
||||
depends_on:
|
||||
- upload-rocm100-wheels
|
||||
- upload-rocm72-wheels
|
||||
|
||||
- label: ":package: Generate Root Index for ROCm Wheels for Release"
|
||||
depends_on: block-generate-root-index-rocm-wheels
|
||||
id: generate-root-index-rocm-wheels
|
||||
agents: *rocm_build_agents
|
||||
env: *rocm100_env
|
||||
commands:
|
||||
- "bash tools/vllm-rocm/generate-rocm-wheels-root-index.sh"
|
||||
|
||||
- label: ":docker: Build release image - x86_64 - ROCm 10.0"
|
||||
id: build-rocm100-release-image
|
||||
depends_on:
|
||||
- step: block-build-release-images
|
||||
allow_failure: true
|
||||
- step: build-rock-base-image
|
||||
- step: build-rocm100-base
|
||||
allow_failure: false
|
||||
agents:
|
||||
queue: cpu_queue_release
|
||||
agents: *rocm_build_agents
|
||||
timeout_in_minutes: 180
|
||||
commands:
|
||||
env: *rocm100_env
|
||||
commands: &rocm_release_image_commands
|
||||
- |
|
||||
set -euo pipefail
|
||||
|
||||
# Login to ECR
|
||||
source .buildkite/scripts/rocm/stack.sh "$${ROCM_BASE_DOCKERFILE}"
|
||||
aws ecr-public get-login-password --region us-east-1 | \
|
||||
docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
|
||||
|
||||
ECR_IMAGE_TAG="$$(buildkite-agent meta-data get rock-base-image-tag 2>/dev/null || echo '')"
|
||||
if [ -z "$${ECR_IMAGE_TAG}" ]; then
|
||||
echo "ERROR: rock-base-image-tag metadata not found"
|
||||
echo "This should have been set by the build-rock-base-image job"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
docker pull "$${ECR_IMAGE_TAG}"
|
||||
|
||||
# Pass the base image ECR tag to downstream steps (nightly publish)
|
||||
buildkite-agent meta-data set "rock-base-ecr-tag" "$${ECR_IMAGE_TAG}"
|
||||
|
||||
echo "========================================"
|
||||
echo "Building vLLM ROCk release image with:"
|
||||
echo " BASE_IMAGE: $${ECR_IMAGE_TAG}"
|
||||
echo " BUILDKITE_COMMIT: $${BUILDKITE_COMMIT}"
|
||||
echo "========================================"
|
||||
|
||||
DOCKER_BUILDKIT=1 docker build \
|
||||
--build-arg max_jobs=16 \
|
||||
--build-arg BASE_IMAGE="$${ECR_IMAGE_TAG}" \
|
||||
--build-arg USE_SCCACHE=1 \
|
||||
docker pull "$${ROCM_STACK_ECR_BASE}"
|
||||
docker build --file "$${ROCM_STACK_DOCKERFILE}" --target vllm-openai \
|
||||
--tag "$${ROCM_STACK_ECR_IMAGE}" --progress plain \
|
||||
--build-arg max_jobs=16 --build-arg BASE_IMAGE="$${ROCM_STACK_ECR_BASE}" \
|
||||
--build-arg INSTALL_LMCACHE=true --build-arg USE_SCCACHE=1 \
|
||||
--build-arg SCCACHE_BUCKET_NAME=vllm-build-sccache \
|
||||
--build-arg SCCACHE_REGION_NAME=us-west-2 \
|
||||
--build-arg SCCACHE_S3_NO_CREDENTIALS=0 \
|
||||
--build-arg INSTALL_LMCACHE=true \
|
||||
--tag public.ecr.aws/q9t5s3a7/vllm-release-repo:$${BUILDKITE_COMMIT}-rock \
|
||||
--target vllm-openai \
|
||||
--progress plain \
|
||||
-f docker/Dockerfile.rock .
|
||||
--build-arg SCCACHE_REGION_NAME=us-west-2 --build-arg SCCACHE_S3_NO_CREDENTIALS=0 .
|
||||
docker push "$${ROCM_STACK_ECR_IMAGE}"
|
||||
|
||||
docker push public.ecr.aws/q9t5s3a7/vllm-release-repo:$${BUILDKITE_COMMIT}-rock
|
||||
env:
|
||||
- label: "Publish nightly ROCm 10.0 image to DockerHub"
|
||||
id: publish-rocm100-nightly
|
||||
depends_on: build-rocm100-release-image
|
||||
if: build.env("NIGHTLY") == "1"
|
||||
agents: &rocm_publish_agents
|
||||
queue: small_cpu_queue_release
|
||||
env: *rocm100_env
|
||||
commands: &rocm_publish_nightly_commands
|
||||
- bash .buildkite/scripts/push-nightly-builds-rocm.sh "$${ROCM_BASE_DOCKERFILE}" $${ROCM_DEFAULT_ARG}
|
||||
plugins: &rocm_dockerhub_login
|
||||
- docker-login#v3.0.0:
|
||||
username: vllmbot
|
||||
password-env: DOCKERHUB_TOKEN
|
||||
|
||||
- group: "Build ROCm 7.2 Wheel / Image"
|
||||
key: "build-rocm72"
|
||||
depends_on: ~
|
||||
steps:
|
||||
- label: ":rocm: Build ROCm 7.2 Base Image & Wheels"
|
||||
id: build-rocm72-base
|
||||
depends_on: ~
|
||||
agents: *rocm_build_agents
|
||||
timeout_in_minutes: 360
|
||||
env: &rocm72_env
|
||||
ROCM_BASE_DOCKERFILE: "docker/Dockerfile.rocm_72_base"
|
||||
ROCM_DEFAULT_ARG: ""
|
||||
DOCKER_BUILDKIT: "1"
|
||||
S3_BUCKET: "vllm-wheels"
|
||||
DOCKERHUB_USERNAME: "vllmbot"
|
||||
commands: *rocm_base_commands
|
||||
|
||||
- label: "Publish nightly ROCk image to DockerHub"
|
||||
- label: ":python: Build vLLM ROCm 7.2 Wheel - x86_64"
|
||||
id: build-rocm72-vllm-wheel
|
||||
depends_on: build-rocm72-base
|
||||
agents: *rocm_build_agents
|
||||
timeout_in_minutes: 180
|
||||
env: *rocm72_env
|
||||
commands: *rocm_vllm_wheel_commands
|
||||
|
||||
- label: ":s3: Upload ROCm 7.2 Wheels to S3"
|
||||
id: upload-rocm72-wheels
|
||||
depends_on: build-rocm72-vllm-wheel
|
||||
agents: *rocm_build_agents
|
||||
timeout_in_minutes: 60
|
||||
env: *rocm72_env
|
||||
commands: *rocm_upload_commands
|
||||
|
||||
- label: ":memo: Annotate ROCm 7.2 wheel release"
|
||||
id: annotate-rocm72-release
|
||||
depends_on: upload-rocm72-wheels
|
||||
agents: *rocm_build_agents
|
||||
env: *rocm72_env
|
||||
commands: *rocm_annotate_commands
|
||||
|
||||
- label: ":docker: Build release image - x86_64 - ROCm 7.2"
|
||||
id: build-rocm72-release-image
|
||||
depends_on:
|
||||
- build-rock-release-image
|
||||
- step: block-build-release-images
|
||||
allow_failure: true
|
||||
- step: build-rocm72-base
|
||||
allow_failure: false
|
||||
agents: *rocm_build_agents
|
||||
timeout_in_minutes: 180
|
||||
env: *rocm72_env
|
||||
commands: *rocm_release_image_commands
|
||||
|
||||
- label: "Publish nightly ROCm 7.2 image to DockerHub"
|
||||
id: publish-rocm72-nightly
|
||||
depends_on: build-rocm72-release-image
|
||||
if: build.env("NIGHTLY") == "1"
|
||||
agents: *rocm_publish_agents
|
||||
env: *rocm72_env
|
||||
commands: *rocm_publish_nightly_commands
|
||||
plugins: *rocm_dockerhub_login
|
||||
|
||||
- label: "Publish nightly XPU image to DockerHub"
|
||||
depends_on:
|
||||
- create-manifest-xpu
|
||||
if: build.env("NIGHTLY") == "1"
|
||||
agents:
|
||||
queue: small_cpu_queue_release
|
||||
commands:
|
||||
- "bash .buildkite/scripts/push-nightly-builds-rocm.sh rock rocm100"
|
||||
# Clean up old nightly builds (keep only last 14)
|
||||
- "bash .buildkite/scripts/cleanup-nightly-builds.sh nightly-rocm100- vllm/vllm-openai-rocm"
|
||||
- "bash .buildkite/scripts/cleanup-nightly-builds.sh base-nightly-rocm100- vllm/vllm-openai-rocm"
|
||||
- "bash .buildkite/scripts/xpu/push-nightly-builds-xpu.sh"
|
||||
- "bash .buildkite/scripts/cleanup-nightly-builds.sh nightly- vllm/vllm-openai-xpu"
|
||||
plugins:
|
||||
- docker-login#v3.0.0:
|
||||
username: vllmbot
|
||||
@@ -1264,7 +997,8 @@ steps:
|
||||
- create-multi-arch-manifest-ubuntu2404
|
||||
- create-multi-arch-manifest-cuda-12-9-ubuntu2404
|
||||
- create-manifest-xpu
|
||||
- build-rocm-release-image
|
||||
- build-rocm100-release-image
|
||||
- build-rocm72-release-image
|
||||
- input-release-version
|
||||
# Wait for CPU builds if their block steps were unblocked, so publish
|
||||
# doesn't race the in-progress CPU build. allow_failure lets publish
|
||||
@@ -1289,6 +1023,7 @@ steps:
|
||||
- "cuda-13-0-ubuntu-24-04"
|
||||
- "cuda-12-9-ubuntu-24-04"
|
||||
- "rocm"
|
||||
- "rocm72"
|
||||
- "xpu"
|
||||
- "cpu"
|
||||
plugins:
|
||||
|
||||
@@ -2,15 +2,23 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
#
|
||||
# Generate Buildkite annotation for ROCm wheel release
|
||||
# Generate Buildkite annotation for a ROCm stack's wheel release.
|
||||
#
|
||||
# Usage: annotate-rocm-release.sh <base-dockerfile> [--default]
|
||||
# base-dockerfile e.g. docker/Dockerfile.rocm_base or docker/Dockerfile.rocm_72_base
|
||||
# --default the stack also owns the plain :latest / :v<ver> Docker Hub tags
|
||||
set -ex
|
||||
|
||||
# Extract build configuration from Dockerfile.rocm_base (single source of truth)
|
||||
# Extract ROCm version dynamically from Dockerfile.rocm_base
|
||||
# BASE_IMAGE format: rocm/dev-ubuntu-22.04:7.0-complete -> extracts "7.0"
|
||||
ROCM_VERSION=$(grep -E '^ARG BASE_IMAGE=' docker/Dockerfile.rocm_base | sed -E 's/.*:([0-9]+\.[0-9]+).*/\1/' || echo "unknown")
|
||||
PYTHON_VERSION=$(grep '^ARG PYTHON_VERSION=' docker/Dockerfile.rocm_base | sed 's/^ARG PYTHON_VERSION=//')
|
||||
PYTORCH_ROCM_ARCH=$(grep '^ARG PYTORCH_ROCM_ARCH=' docker/Dockerfile.rocm_base | sed 's/^ARG PYTORCH_ROCM_ARCH=//')
|
||||
# shellcheck source=.buildkite/scripts/rocm/stack.sh
|
||||
source "$(dirname "${BASH_SOURCE[0]}")/rocm/stack.sh" "${1:?Usage: $0 <base-dockerfile> [--default]}"
|
||||
TAG_VARIANTS=("-${ROCM_STACK_VARIANT}")
|
||||
if [[ "${2:-}" == "--default" ]]; then
|
||||
TAG_VARIANTS=("" "-${ROCM_STACK_VARIANT}")
|
||||
fi
|
||||
ROCM_VERSION="$ROCM_STACK_VERSION"
|
||||
VARIANT="$ROCM_STACK_VARIANT"
|
||||
PYTHON_VERSION=$(sed -nE 's/^ARG PYTHON_VERSION="?([^" ]+)"?.*/\1/p' "$ROCM_STACK_BASE_DOCKERFILE")
|
||||
PYTORCH_ROCM_ARCH=$(sed -nE 's/^ARG PYTORCH_ROCM_ARCH="?([^"]+)"?.*/\1/p' "$ROCM_STACK_BASE_DOCKERFILE")
|
||||
|
||||
# Get release version, default to 1.0.0.dev for nightly/per-commit builds
|
||||
RELEASE_VERSION=$(buildkite-agent meta-data get release-version 2>/dev/null || echo "")
|
||||
@@ -18,22 +26,32 @@ if [ -z "${RELEASE_VERSION}" ]; then
|
||||
RELEASE_VERSION="1.0.0.dev"
|
||||
fi
|
||||
|
||||
ROCM_BASE_CACHE_KEY=$(.buildkite/scripts/cache-rocm-base-wheels.sh key)
|
||||
|
||||
# S3 URLs
|
||||
S3_BUCKET="${S3_BUCKET:-vllm-wheels}"
|
||||
S3_REGION="${AWS_DEFAULT_REGION:-us-west-2}"
|
||||
S3_URL="http://${S3_BUCKET}.s3-website-${S3_REGION}.amazonaws.com"
|
||||
S3_HOST="${S3_BUCKET}.s3-website-${S3_REGION}.amazonaws.com"
|
||||
S3_URL="http://${S3_HOST}"
|
||||
WHEEL_PATH="rocm/${BUILDKITE_COMMIT}/${VARIANT}-wheels/"
|
||||
|
||||
# Format ROCm version for path (e.g., "7.1" -> "rocm710")
|
||||
ROCM_VERSION_PATH="rocm$(echo "${ROCM_VERSION}" | tr -d '.')"
|
||||
ROCM_PATH="rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}"
|
||||
buildkite-agent annotate --style 'success' --context 'rocm-release-workflow' << EOF
|
||||
## ROCm Wheel and Docker Image Releases
|
||||
DOCKER_CMDS="docker pull ${ROCM_STACK_ECR_BASE}
|
||||
docker pull ${ROCM_STACK_ECR_IMAGE}"
|
||||
for suffix in "${TAG_VARIANTS[@]}"; do
|
||||
for tag in "latest${suffix}" "v${RELEASE_VERSION}${suffix}"; do
|
||||
DOCKER_CMDS+="
|
||||
docker tag ${ROCM_STACK_ECR_BASE} vllm/vllm-openai-rocm:${tag}-base
|
||||
docker tag ${ROCM_STACK_ECR_IMAGE} vllm/vllm-openai-rocm:${tag}
|
||||
docker push vllm/vllm-openai-rocm:${tag}-base
|
||||
docker push vllm/vllm-openai-rocm:${tag}"
|
||||
done
|
||||
done
|
||||
|
||||
buildkite-agent annotate --style 'success' --context "rocm-release-workflow-${VARIANT}" << EOF
|
||||
## ROCm ${ROCM_VERSION} Wheel and Docker Image Releases
|
||||
### Build Configuration
|
||||
| Setting | Value |
|
||||
|---------|-------|
|
||||
| **ROCm Version** | ${ROCM_VERSION} |
|
||||
| **Index Variant** | ${VARIANT} |
|
||||
| **Python Version** | ${PYTHON_VERSION} |
|
||||
| **GPU Architectures** | ${PYTORCH_ROCM_ARCH} |
|
||||
| **Branch** | \`${BUILDKITE_BRANCH}\` |
|
||||
@@ -41,49 +59,20 @@ buildkite-agent annotate --style 'success' --context 'rocm-release-workflow' <<
|
||||
|
||||
### :package: Installation
|
||||
|
||||
**Install from this build (by commit):**
|
||||
|
||||
\`\`\`bash
|
||||
pip install vllm --extra-index-url ${S3_URL}/${ROCM_PATH}/ --trusted-host ${S3_BUCKET}.s3-website-${S3_REGION}.amazonaws.com
|
||||
|
||||
# Example for ROCm ${ROCM_VERSION}:
|
||||
pip install vllm --extra-index-url ${S3_URL}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/ --trusted-host ${S3_BUCKET}.s3-website-${S3_REGION}.amazonaws.com
|
||||
\`\`\`
|
||||
|
||||
**Install from nightly (if published):**
|
||||
|
||||
\`\`\`bash
|
||||
pip install vllm --extra-index-url ${S3_URL}/rocm/nightly/ --trusted-host ${S3_BUCKET}.s3-website-${S3_REGION}.amazonaws.com
|
||||
# This build (by commit)
|
||||
pip install vllm --extra-index-url ${S3_URL}/rocm/${BUILDKITE_COMMIT}/${VARIANT}/ --trusted-host ${S3_HOST}
|
||||
# Nightly (if published)
|
||||
pip install vllm --extra-index-url ${S3_URL}/rocm/nightly/${VARIANT}/ --trusted-host ${S3_HOST}
|
||||
\`\`\`
|
||||
|
||||
### :floppy_disk: Download Wheels Directly
|
||||
|
||||
\`\`\`bash
|
||||
# List all ROCm wheels
|
||||
aws s3 ls s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/
|
||||
# Download specific wheels
|
||||
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/vllm-*.whl .
|
||||
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/torch-*.whl .
|
||||
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/triton-*.whl .
|
||||
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/triton-kernels-*.whl .
|
||||
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/torchvision-*.whl .
|
||||
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/torchaudio-*.whl .
|
||||
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/amdsmi-*.whl .
|
||||
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/amd_aiter-*.whl .
|
||||
aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/flash-attn-*.whl .
|
||||
aws s3 ls s3://${S3_BUCKET}/${WHEEL_PATH}
|
||||
aws s3 cp --recursive --exclude "*" --include "*.whl" s3://${S3_BUCKET}/${WHEEL_PATH} .
|
||||
\`\`\`
|
||||
|
||||
### :gear: Included Packages
|
||||
- **vllm**: vLLM with ROCm support
|
||||
- **torch**: PyTorch built for ROCm ${ROCM_VERSION}
|
||||
- **triton**: Triton
|
||||
- **triton-kernels**: Triton kernels
|
||||
- **torchvision**: TorchVision for ROCm PyTorch
|
||||
- **torchaudio**: Torchaudio for ROCm PyTorch
|
||||
- **amdsmi**: AMD SMI Python bindings
|
||||
- **amd_aiter**: Aiter for ROCm
|
||||
- **flash-attn**: Flash Attention for ROCm
|
||||
|
||||
### :warning: Notes
|
||||
- These wheels are built for **ROCm ${ROCM_VERSION}** and will NOT work with CUDA GPUs
|
||||
- Supported GPU architectures: ${PYTORCH_ROCM_ARCH}
|
||||
@@ -91,23 +80,8 @@ aws s3 cp s3://${S3_BUCKET}/rocm/${BUILDKITE_COMMIT}/${ROCM_VERSION_PATH}/flash-
|
||||
|
||||
### :package: Docker Image Release
|
||||
|
||||
To download and upload the image:
|
||||
|
||||
\`\`\`
|
||||
docker pull public.ecr.aws/q9t5s3a7/vllm-release-repo:${BUILDKITE_COMMIT}-rocm-base
|
||||
docker pull public.ecr.aws/q9t5s3a7/vllm-release-repo:${BUILDKITE_COMMIT}-rocm
|
||||
|
||||
docker tag public.ecr.aws/q9t5s3a7/vllm-release-repo:${ROCM_BASE_CACHE_KEY}-rocm-base vllm/vllm-openai-rocm:${BUILDKITE_COMMIT}-base
|
||||
docker tag vllm/vllm-openai-rocm:${BUILDKITE_COMMIT}-base vllm/vllm-openai-rocm:latest-base
|
||||
docker tag vllm/vllm-openai-rocm:${BUILDKITE_COMMIT}-base vllm/vllm-openai-rocm:v${RELEASE_VERSION}-base
|
||||
docker push vllm/vllm-openai-rocm:latest-base
|
||||
docker push vllm/vllm-openai-rocm:v${RELEASE_VERSION}-base
|
||||
|
||||
docker tag public.ecr.aws/q9t5s3a7/vllm-release-repo:${BUILDKITE_COMMIT}-rocm vllm/vllm-openai-rocm:${BUILDKITE_COMMIT}
|
||||
docker tag vllm/vllm-openai-rocm:${BUILDKITE_COMMIT} vllm/vllm-openai-rocm:latest
|
||||
docker tag vllm/vllm-openai-rocm:${BUILDKITE_COMMIT} vllm/vllm-openai-rocm:v${RELEASE_VERSION}
|
||||
docker push vllm/vllm-openai-rocm:latest
|
||||
docker push vllm/vllm-openai-rocm:v${RELEASE_VERSION}
|
||||
${DOCKER_CMDS}
|
||||
\`\`\`
|
||||
|
||||
EOF
|
||||
|
||||
@@ -5,39 +5,33 @@
|
||||
# Cache helper for ROCm base wheels
|
||||
#
|
||||
# This script manages caching of pre-built ROCm base wheels (torch, triton, etc.)
|
||||
# to avoid rebuilding them when Dockerfile.rocm_base hasn't changed.
|
||||
# to avoid rebuilding them when the stack's base Dockerfile hasn't changed.
|
||||
#
|
||||
# Usage:
|
||||
# cache-rocm-base-wheels.sh check - Check if cache exists, outputs "hit" or "miss"
|
||||
# cache-rocm-base-wheels.sh upload - Upload wheels to cache
|
||||
# cache-rocm-base-wheels.sh download - Download wheels from cache
|
||||
# cache-rocm-base-wheels.sh key - Output the cache key
|
||||
# Usage: cache-rocm-base-wheels.sh <base-dockerfile> <command>
|
||||
# base-dockerfile e.g. docker/Dockerfile.rocm_base or docker/Dockerfile.rocm_72_base
|
||||
# Commands:
|
||||
# check - Check if cache exists, outputs "hit" or "miss"
|
||||
# upload - Upload wheels to cache
|
||||
# download - Download wheels from cache
|
||||
# key - Output the cache key
|
||||
# path - Output the full S3 cache path
|
||||
#
|
||||
# Environment variables:
|
||||
# S3_BUCKET - S3 bucket name (default: vllm-wheels)
|
||||
#
|
||||
# Note: ROCm version is determined by BASE_IMAGE in Dockerfile.rocm_base,
|
||||
# so changes to ROCm version are captured by the Dockerfile hash.
|
||||
# The cache key hashes the base Dockerfile and the repo files it COPYs in, which
|
||||
# capture the ROCm version and every component pin (see rocm/stack.sh).
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
BUCKET="${S3_BUCKET:-vllm-wheels}"
|
||||
DOCKERFILE="docker/Dockerfile.rocm_base"
|
||||
CACHE_PREFIX="rocm/cache"
|
||||
|
||||
# Generate hash from Dockerfile content + build args
|
||||
generate_cache_key() {
|
||||
# Include Dockerfile content
|
||||
if [[ ! -f "$DOCKERFILE" ]]; then
|
||||
echo "ERROR: Dockerfile not found: $DOCKERFILE" >&2
|
||||
exit 1
|
||||
fi
|
||||
local dockerfile_hash=$(sha256sum "$DOCKERFILE" | cut -c1-16)
|
||||
|
||||
echo "${dockerfile_hash}"
|
||||
}
|
||||
|
||||
CACHE_KEY=$(generate_cache_key)
|
||||
# shellcheck source=.buildkite/scripts/rocm/stack.sh
|
||||
source "$(dirname "${BASH_SOURCE[0]}")/rocm/stack.sh" "${1:?Usage: $0 <base-dockerfile> <command>}"
|
||||
shift
|
||||
CACHE_KEY="$ROCM_STACK_WHEEL_KEY"
|
||||
WHEELS_DIR="$ROCM_STACK_BASE_WHEELS_DIR"
|
||||
CACHE_PATH="s3://${BUCKET}/${CACHE_PREFIX}/${CACHE_KEY}/"
|
||||
|
||||
case "${1:-}" in
|
||||
@@ -67,19 +61,19 @@ case "${1:-}" in
|
||||
echo "Cache path: ${CACHE_PATH}"
|
||||
echo ""
|
||||
|
||||
if [[ ! -d "artifacts/rocm-base-wheels" ]]; then
|
||||
echo "ERROR: artifacts/rocm-base-wheels directory not found" >&2
|
||||
if [[ ! -d "${WHEELS_DIR}" ]]; then
|
||||
echo "ERROR: ${WHEELS_DIR} directory not found" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
WHEEL_COUNT=$(find artifacts/rocm-base-wheels -maxdepth 1 -name '*.whl' 2>/dev/null | wc -l)
|
||||
WHEEL_COUNT=$(find "${WHEELS_DIR}" -maxdepth 1 -name '*.whl' 2>/dev/null | wc -l)
|
||||
if [[ "$WHEEL_COUNT" -eq 0 ]]; then
|
||||
echo "ERROR: No wheels found in artifacts/rocm-base-wheels/" >&2
|
||||
echo "ERROR: No wheels found in ${WHEELS_DIR}/" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo "Uploading $WHEEL_COUNT wheels..."
|
||||
aws s3 cp --recursive artifacts/rocm-base-wheels/ "${CACHE_PATH}"
|
||||
aws s3 cp --recursive "${WHEELS_DIR}/" "${CACHE_PATH}"
|
||||
|
||||
echo ""
|
||||
echo "Cache upload complete!"
|
||||
@@ -93,17 +87,17 @@ case "${1:-}" in
|
||||
echo "Cache key: ${CACHE_KEY}"
|
||||
echo "Cache path: ${CACHE_PATH}"
|
||||
echo ""
|
||||
mkdir -p artifacts/rocm-base-wheels
|
||||
mkdir -p "${WHEELS_DIR}"
|
||||
|
||||
# Use sync with include/exclude to only download .whl files
|
||||
aws s3 sync "${CACHE_PATH}" artifacts/rocm-base-wheels/ \
|
||||
aws s3 sync "${CACHE_PATH}" "${WHEELS_DIR}/" \
|
||||
--exclude "*" \
|
||||
--include "*.whl"
|
||||
|
||||
echo ""
|
||||
echo "Downloaded wheels:"
|
||||
find artifacts/rocm-base-wheels -maxdepth 1 -name '*.whl' -exec ls -lh {} \;
|
||||
WHEEL_COUNT=$(find artifacts/rocm-base-wheels -maxdepth 1 -name '*.whl' 2>/dev/null | wc -l)
|
||||
find "${WHEELS_DIR}" -maxdepth 1 -name '*.whl' -exec ls -lh {} \;
|
||||
WHEEL_COUNT=$(find "${WHEELS_DIR}" -maxdepth 1 -name '*.whl' 2>/dev/null | wc -l)
|
||||
echo ""
|
||||
echo "Total: $WHEEL_COUNT wheels"
|
||||
echo "========================================"
|
||||
@@ -118,12 +112,12 @@ case "${1:-}" in
|
||||
;;
|
||||
|
||||
*)
|
||||
echo "Usage: $0 {check|upload|download|key|path}" >&2
|
||||
echo "Usage: $0 <base-dockerfile> {check|upload|download|key|path}" >&2
|
||||
echo "" >&2
|
||||
echo "Commands:" >&2
|
||||
echo " check - Check if cache exists, outputs 'hit' or 'miss'" >&2
|
||||
echo " upload - Upload wheels from artifacts/rocm-base-wheels/ to cache" >&2
|
||||
echo " download - Download wheels from cache to artifacts/rocm-base-wheels/" >&2
|
||||
echo " upload - Upload wheels from ${WHEELS_DIR}/ to cache" >&2
|
||||
echo " download - Download wheels from cache to ${WHEELS_DIR}/" >&2
|
||||
echo " key - Output the cache key" >&2
|
||||
echo " path - Output the full S3 cache path" >&2
|
||||
exit 1
|
||||
|
||||
@@ -17,7 +17,7 @@ DEFAULT_REPO_SLUG="vllm-project/vllm"
|
||||
DEFAULT_CI_HCL_SOURCE="docker/ci-rocm.hcl"
|
||||
DEFAULT_CI_BASE_CONTENT_FILES=".dockerignore requirements/common.txt requirements/rocm.txt requirements/test/rocm.txt tools/install_torchcodec_rocm.sh rust-toolchain.toml tests/vllm_test_utils"
|
||||
DEFAULT_CI_BASE_DOCKERFILE="docker/Dockerfile.rocm"
|
||||
DEFAULT_CI_BASE_DOCKERFILE_STAGES="base rust_toolchain_input_0 rust-toolchain-input rust-toolchain build_nixl lmcache_source build_lmcache build_rocshmem build_deepep mori_base ci_base"
|
||||
DEFAULT_CI_BASE_DOCKERFILE_STAGES="base rust_toolchain_input_0 rust-toolchain-input rust-toolchain build_nixl lmcache_source build_lmcache build_deepep mori_base ci_base"
|
||||
DEFAULT_CI_BASE_METADATA_VERSION="3"
|
||||
# ROCm CI forces REMOTE_VLLM=0, so content identity covers only the selected
|
||||
# local-source stages rather than unreachable remote-fetch alternatives.
|
||||
@@ -555,9 +555,6 @@ configure_custom_rocm_stages() {
|
||||
CI_BASE_DOCKERFILE_STAGES="${CI_BASE_DOCKERFILE_STAGES:-${stages}}"
|
||||
ROCM_CSRC_DOCKERFILE_STAGES="${ROCM_CSRC_DOCKERFILE_STAGES:-${stages}}"
|
||||
ROCM_RUST_DOCKERFILE_STAGES="${ROCM_RUST_DOCKERFILE_STAGES:-${stages}}"
|
||||
if [[ " ${stages} " != *" build_rocshmem "* ]]; then
|
||||
unset ROCSHMEM_BRANCH ROCSHMEM_CACHE_KEY DEEPEP_CACHE_KEY
|
||||
fi
|
||||
}
|
||||
|
||||
compose_dependency_cache_key() {
|
||||
@@ -1724,14 +1721,11 @@ ci_base_metadata_pairs() {
|
||||
metadata_pair "vllm.rocm.nixl_commit" "${NIXL_BRANCH:-$(resolve_dockerfile_arg_value "${dockerfile}" "NIXL_BRANCH")}"
|
||||
metadata_pair "vllm.rocm.ucx_repo" "$(resolve_dockerfile_arg_value "${dockerfile}" "UCX_REPO")"
|
||||
metadata_pair "vllm.rocm.ucx_commit" "${UCX_BRANCH:-$(resolve_dockerfile_arg_value "${dockerfile}" "UCX_BRANCH")}"
|
||||
metadata_pair "vllm.rocm.rocshmem_repo" "$(resolve_dockerfile_arg_value "${dockerfile}" "ROCSHMEM_REPO")"
|
||||
metadata_pair "vllm.rocm.rocshmem_commit" "${ROCSHMEM_BRANCH:-$(resolve_dockerfile_arg_value "${dockerfile}" "ROCSHMEM_BRANCH")}"
|
||||
metadata_pair "vllm.rocm.deepep_repo" "$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_REPO")"
|
||||
metadata_pair "vllm.rocm.deepep_commit" "${DEEPEP_BRANCH:-$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_BRANCH")}"
|
||||
metadata_pair "vllm.rocm.deepep_nic" "$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_NIC")"
|
||||
metadata_pair "vllm.rocm.deepep_rocm_arch" "$(resolve_dockerfile_arg_value "${dockerfile}" "DEEPEP_ROCM_ARCH")"
|
||||
metadata_pair "vllm.rocm.nixl_cache_key" "${NIXL_CACHE_KEY:-}"
|
||||
metadata_pair "vllm.rocm.rocshmem_cache_key" "${ROCSHMEM_CACHE_KEY:-}"
|
||||
metadata_pair "vllm.rocm.deepep_cache_key" "${DEEPEP_CACHE_KEY:-}"
|
||||
}
|
||||
|
||||
@@ -2314,7 +2308,7 @@ extract_dependency_pins() {
|
||||
return 0
|
||||
fi
|
||||
|
||||
for var in NIXL_BRANCH UCX_BRANCH ROCSHMEM_BRANCH DEEPEP_BRANCH; do
|
||||
for var in NIXL_BRANCH UCX_BRANCH DEEPEP_BRANCH; do
|
||||
if [[ -n "${!var:-}" ]]; then
|
||||
echo "Using provided ${var}: ${!var}"
|
||||
continue
|
||||
@@ -2336,17 +2330,14 @@ compute_dependency_cache_keys() {
|
||||
local dockerfile_rocm=""
|
||||
local nixl_branch=""
|
||||
local ucx_branch=""
|
||||
local rocshmem_branch=""
|
||||
local deepep_branch=""
|
||||
local nixl_material=""
|
||||
local rocshmem_material=""
|
||||
local deepep_material=""
|
||||
|
||||
bake_dir=$(dirname "${VLLM_BAKE_FILE}")
|
||||
dockerfile_rocm="${CI_BASE_DOCKERFILE:-${bake_dir}/Dockerfile.rocm}"
|
||||
nixl_branch=$(resolve_dockerfile_arg_value "${dockerfile_rocm}" "NIXL_BRANCH")
|
||||
ucx_branch=$(resolve_dockerfile_arg_value "${dockerfile_rocm}" "UCX_BRANCH")
|
||||
rocshmem_branch=$(resolve_dockerfile_arg_value "${dockerfile_rocm}" "ROCSHMEM_BRANCH")
|
||||
deepep_branch=$(resolve_dockerfile_arg_value "${dockerfile_rocm}" "DEEPEP_BRANCH")
|
||||
|
||||
if [[ -n "${nixl_branch}" && -n "${ucx_branch}" ]]; then
|
||||
@@ -2360,22 +2351,11 @@ compute_dependency_cache_keys() {
|
||||
echo "NIXL dependency cache key: ${NIXL_CACHE_KEY}"
|
||||
fi
|
||||
|
||||
if [[ -n "${rocshmem_branch}" ]]; then
|
||||
rocshmem_material=$(compose_stage_cache_material "${dockerfile_rocm}" "base build_rocshmem")
|
||||
ROCSHMEM_CACHE_KEY=$(
|
||||
compose_dependency_cache_key \
|
||||
"${rocshmem_branch}" \
|
||||
"${rocshmem_material}"
|
||||
)
|
||||
export ROCSHMEM_CACHE_KEY
|
||||
echo "ROCShmem dependency cache key: ${ROCSHMEM_CACHE_KEY}"
|
||||
fi
|
||||
|
||||
if [[ -n "${deepep_branch}" && -n "${rocshmem_branch}" ]]; then
|
||||
deepep_material=$(compose_stage_cache_material "${dockerfile_rocm}" "base build_rocshmem build_deepep")
|
||||
if [[ -n "${deepep_branch}" ]]; then
|
||||
deepep_material=$(compose_stage_cache_material "${dockerfile_rocm}" "base build_deepep")
|
||||
DEEPEP_CACHE_KEY=$(
|
||||
compose_dependency_cache_key \
|
||||
"${deepep_branch}-rocshmem-${rocshmem_branch}" \
|
||||
"${deepep_branch}" \
|
||||
"${deepep_material}"
|
||||
)
|
||||
export DEEPEP_CACHE_KEY
|
||||
@@ -2415,18 +2395,11 @@ dependency_cache_ref_for_target() {
|
||||
printf '%s\n' "${cache_repo}:nixl-rocm-${NIXL_BRANCH}-ucx-${UCX_BRANCH:-}"
|
||||
fi
|
||||
;;
|
||||
rocshmem-rocm-ci)
|
||||
if [[ -n "${ROCSHMEM_CACHE_KEY:-}" ]]; then
|
||||
printf '%s\n' "${cache_repo}:rocshmem-rocm-${ROCSHMEM_CACHE_KEY}"
|
||||
elif [[ -n "${ROCSHMEM_BRANCH:-}" ]]; then
|
||||
printf '%s\n' "${cache_repo}:rocshmem-rocm-${ROCSHMEM_BRANCH}"
|
||||
fi
|
||||
;;
|
||||
deepep-rocm-ci)
|
||||
if [[ -n "${DEEPEP_CACHE_KEY:-}" ]]; then
|
||||
printf '%s\n' "${cache_repo}:deepep-rocm-${DEEPEP_CACHE_KEY}"
|
||||
elif [[ -n "${DEEPEP_BRANCH:-}" ]]; then
|
||||
printf '%s\n' "${cache_repo}:deepep-rocm-${DEEPEP_BRANCH}-rocshmem-${ROCSHMEM_BRANCH:-}"
|
||||
printf '%s\n' "${cache_repo}:deepep-rocm-${DEEPEP_BRANCH}"
|
||||
fi
|
||||
;;
|
||||
esac
|
||||
@@ -2444,7 +2417,6 @@ add_dependency_cache_target() {
|
||||
resolve_ci_base_dependency_targets() {
|
||||
local mode="${ROCM_DEP_CACHE_EXPORT_MODE:-missing}"
|
||||
local nixl_ref=""
|
||||
local rocshmem_ref=""
|
||||
local deepep_ref=""
|
||||
|
||||
[[ "${TARGET}" == "ci-base-rocm-ci-with-deps" ]] || return 0
|
||||
@@ -2452,7 +2424,7 @@ resolve_ci_base_dependency_targets() {
|
||||
case "${mode}" in
|
||||
always)
|
||||
echo "ROCM_DEP_CACHE_EXPORT_MODE=always; exporting all dependency caches serially"
|
||||
for target in nixl-rocm-ci rocshmem-rocm-ci deepep-rocm-ci; do
|
||||
for target in nixl-rocm-ci deepep-rocm-ci; do
|
||||
if [[ -n "$(dependency_cache_ref_for_target "${target}")" ]]; then
|
||||
add_dependency_cache_target "${target}"
|
||||
fi
|
||||
@@ -2482,16 +2454,6 @@ resolve_ci_base_dependency_targets() {
|
||||
fi
|
||||
fi
|
||||
|
||||
if [[ "${mode}" != "always" && -n "${ROCSHMEM_CACHE_KEY:-}" ]]; then
|
||||
rocshmem_ref=$(dependency_cache_ref_for_target "rocshmem-rocm-ci")
|
||||
if dependency_cache_ref_exists "${rocshmem_ref}"; then
|
||||
echo "ROCShmem dependency cache exists: ${rocshmem_ref}"
|
||||
else
|
||||
echo "ROCShmem dependency cache missing; will seed: ${rocshmem_ref}"
|
||||
add_dependency_cache_target "rocshmem-rocm-ci"
|
||||
fi
|
||||
fi
|
||||
|
||||
if [[ "${mode}" != "always" && -n "${DEEPEP_CACHE_KEY:-}" ]]; then
|
||||
deepep_ref=$(dependency_cache_ref_for_target "deepep-rocm-ci")
|
||||
if dependency_cache_ref_exists "${deepep_ref}"; then
|
||||
@@ -2502,15 +2464,6 @@ resolve_ci_base_dependency_targets() {
|
||||
fi
|
||||
fi
|
||||
|
||||
# DeepEP inherits from ROCShmem. If ROCShmem is being seeded, seed DeepEP too
|
||||
# so the pair stays consistent for future ci_base rebuilds.
|
||||
if printf '%s\n' "${DEPENDENCY_CACHE_TARGETS[@]}" | grep -qx "rocshmem-rocm-ci" \
|
||||
&& ! printf '%s\n' "${DEPENDENCY_CACHE_TARGETS[@]}" | grep -qx "deepep-rocm-ci" \
|
||||
&& [[ -n "${DEEPEP_BRANCH:-}" ]]; then
|
||||
echo "ROCShmem cache is missing; also seeding DeepEP cache"
|
||||
add_dependency_cache_target "deepep-rocm-ci"
|
||||
fi
|
||||
|
||||
BAKE_TARGETS=("ci-base-rocm-ci")
|
||||
if [[ ${#DEPENDENCY_CACHE_TARGETS[@]} -eq 0 ]]; then
|
||||
echo "All dependency caches exist; building ci_base without dependency cache exports"
|
||||
|
||||
@@ -5,14 +5,17 @@ set -ex
|
||||
# Clean up old nightly builds from DockerHub, keeping only the last 14 builds
|
||||
# This script uses DockerHub API to list and delete old tags with specified prefix
|
||||
# Tags starting with "nightly-dev" are always excluded from deletion
|
||||
# Usage: cleanup-nightly-builds.sh [TAG_PREFIX] [REPO]
|
||||
# Usage: cleanup-nightly-builds.sh [TAG_PREFIX] [REPO] [EXCLUDE_PREFIX]
|
||||
# Example: cleanup-nightly-builds.sh "nightly-"
|
||||
# Example: cleanup-nightly-builds.sh "cu130-nightly-"
|
||||
# Example: cleanup-nightly-builds.sh "nightly-" "vllm/vllm-openai-rocm"
|
||||
# Example: cleanup-nightly-builds.sh "nightly-" "vllm/vllm-openai-rocm" "nightly-rocm"
|
||||
# (EXCLUDE_PREFIX skips tags owned by another variant sharing TAG_PREFIX)
|
||||
|
||||
# Get tag prefix and repo from arguments
|
||||
TAG_PREFIX="${1:-nightly-}"
|
||||
REPO="${2:-vllm/vllm-openai}"
|
||||
EXCLUDE_PREFIX="${3:-}"
|
||||
|
||||
echo "Cleaning up tags with prefix: $TAG_PREFIX in repository: $REPO"
|
||||
|
||||
@@ -57,7 +60,7 @@ get_all_tags() {
|
||||
|
||||
# Get both last_updated timestamp and tag name, separated by |
|
||||
# Exclude mutable architecture tags and nightly-dev tags from cleanup
|
||||
local tags=$(echo "$response" | jq -r --arg prefix "$TAG_PREFIX" '.results[] | select(.name | startswith($prefix)) | select(.name != ($prefix + "x86_64") and .name != ($prefix + "aarch64")) | select(.name | startswith("nightly-dev") | not) | "\(.last_updated)|\(.name)"')
|
||||
local tags=$(echo "$response" | jq -r --arg prefix "$TAG_PREFIX" --arg exclude "$EXCLUDE_PREFIX" '.results[] | select(.name | startswith($prefix)) | select($exclude == "" or (.name | startswith($exclude) | not)) | select(.name != ($prefix + "x86_64") and .name != ($prefix + "aarch64")) | select(.name | startswith("nightly-dev") | not) | "\(.last_updated)|\(.name)"')
|
||||
|
||||
if [ -z "$tags" ]; then
|
||||
break
|
||||
|
||||
@@ -12,7 +12,7 @@ from dataclasses import asdict, dataclass
|
||||
from datetime import datetime
|
||||
from pathlib import Path
|
||||
from typing import Any
|
||||
from urllib.parse import quote
|
||||
from urllib.parse import quote, unquote, urlparse
|
||||
|
||||
import regex as re
|
||||
|
||||
@@ -51,6 +51,8 @@ class WheelFileInfo:
|
||||
platform_tag: str
|
||||
variant: str | None
|
||||
filename: str
|
||||
# Absolute URL for files hosted elsewhere (see --external-links)
|
||||
url: str | None = None
|
||||
|
||||
|
||||
def parse_from_filename(file: str) -> WheelFileInfo:
|
||||
@@ -110,6 +112,29 @@ def parse_from_filename(file: str) -> WheelFileInfo:
|
||||
)
|
||||
|
||||
|
||||
def parse_external_url(url: str) -> WheelFileInfo:
|
||||
"""Parse an absolute wheel or sdist URL hosted on another index."""
|
||||
filename = unquote(urlparse(url).path.rsplit("/", 1)[-1])
|
||||
if filename.endswith(".whl"):
|
||||
info = parse_from_filename(filename)
|
||||
else:
|
||||
match = re.match(r"^(?P<name>.+)-(?P<version>[^-]+)\.(tar\.gz|zip)$", filename)
|
||||
if not match:
|
||||
raise ValueError(f"Unsupported external file: {filename}")
|
||||
info = WheelFileInfo(
|
||||
package_name=match.group("name"),
|
||||
version=match.group("version"),
|
||||
build_tag=None,
|
||||
python_tag="source",
|
||||
abi_tag="none",
|
||||
platform_tag="any",
|
||||
variant=None,
|
||||
filename=filename,
|
||||
)
|
||||
info.url = url
|
||||
return info
|
||||
|
||||
|
||||
def generate_project_list(subdir_names: list[str], comment: str = "") -> str:
|
||||
"""Generate project list HTML content linking to each project & variant subdirectory."""
|
||||
href_tags = []
|
||||
@@ -129,14 +154,19 @@ def generate_package_index_and_metadata(
|
||||
href_tags = []
|
||||
metadata = []
|
||||
for file in sorted(wheel_files, key=lambda x: x.filename):
|
||||
relative_path = (
|
||||
wheel_base_dir.relative_to(index_base_dir, walk_up=True) / file.filename
|
||||
)
|
||||
# handle with '+' in URL, and avoid double-encoding '/' and already-encoded '%2B'
|
||||
# NOTE: this is AWS S3 specific behavior!
|
||||
file_path_quoted = quote(relative_path.as_posix(), safe=":%/")
|
||||
if file.url:
|
||||
file_path_quoted = file.url
|
||||
else:
|
||||
relative_path = (
|
||||
wheel_base_dir.relative_to(index_base_dir, walk_up=True) / file.filename
|
||||
)
|
||||
# handle with '+' in URL, and avoid double-encoding '/' and already-encoded '%2B'
|
||||
# NOTE: this is AWS S3 specific behavior!
|
||||
file_path_quoted = quote(relative_path.as_posix(), safe=":%/")
|
||||
href_tags.append(f' <a href="{file_path_quoted}">{file.filename}</a><br/>')
|
||||
file_meta = asdict(file)
|
||||
if file_meta["url"] is None:
|
||||
del file_meta["url"]
|
||||
file_meta["path"] = file_path_quoted
|
||||
metadata.append(file_meta)
|
||||
index_str = INDEX_HTML_TEMPLATE.format(items="\n".join(href_tags), comment=comment)
|
||||
@@ -151,6 +181,7 @@ def generate_index_and_metadata(
|
||||
default_variant: str | None = None,
|
||||
alias_to_default: str | None = None,
|
||||
comment: str = "",
|
||||
external_urls: list[str] | None = None,
|
||||
):
|
||||
"""Generate index for all wheel files.
|
||||
|
||||
@@ -160,6 +191,8 @@ def generate_index_and_metadata(
|
||||
index_base_dir (Path): Base directory to store index files.
|
||||
default_variant (str | None): The default variant name, if any.
|
||||
alias_to_default (str | None): Alias variant name for the default variant, if any.
|
||||
external_urls (list[str] | None): Absolute URLs of files hosted on another index,
|
||||
listed in the generated indices without being copied.
|
||||
comment (str | None): Optional comment to include in the generated HTML files.
|
||||
|
||||
First, parse all wheel files to extract metadata.
|
||||
@@ -214,6 +247,7 @@ def generate_index_and_metadata(
|
||||
|
||||
"""
|
||||
parsed_files = [parse_from_filename(f) for f in whl_files]
|
||||
parsed_files += [parse_external_url(u) for u in external_urls or []]
|
||||
|
||||
if not parsed_files:
|
||||
print("No wheel files found, skipping index generation.")
|
||||
@@ -232,10 +266,14 @@ def generate_index_and_metadata(
|
||||
print(f"Detected ROCm variant from vllm: {rocm_variant}")
|
||||
break
|
||||
|
||||
# Apply ROCm variant to all wheels without a variant
|
||||
# Apply ROCm variant to all wheels without a variant. Dependencies with their
|
||||
# own ROCm local version (e.g. TheRock's torch-2.12.0+rocm10.0.0) belong to
|
||||
# the same variant rather than one of their own.
|
||||
if rocm_variant:
|
||||
for file in parsed_files:
|
||||
if file.variant is None:
|
||||
if file.variant is None or (
|
||||
file.variant.startswith("rocm") and file.variant != rocm_variant
|
||||
):
|
||||
file.variant = rocm_variant
|
||||
print(f"Inherited variant '{rocm_variant}' for {file.filename}")
|
||||
|
||||
@@ -366,6 +404,13 @@ if __name__ == "__main__":
|
||||
default=None,
|
||||
help="Alias variant name for the default variant",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--external-links",
|
||||
type=str,
|
||||
default=None,
|
||||
help="Optional file with one absolute URL per line for packages hosted on "
|
||||
"another index (listed alongside, but not copied into, this index)",
|
||||
)
|
||||
parser.add_argument(
|
||||
"--comment",
|
||||
type=str,
|
||||
@@ -463,5 +508,14 @@ if __name__ == "__main__":
|
||||
default_variant=None,
|
||||
alias_to_default=args.alias_to_default,
|
||||
comment=args.comment.strip(),
|
||||
external_urls=(
|
||||
[
|
||||
line.strip()
|
||||
for line in Path(args.external_links).read_text().splitlines()
|
||||
if line.strip() and not line.startswith("#")
|
||||
]
|
||||
if args.external_links
|
||||
else None
|
||||
),
|
||||
)
|
||||
print(f"Successfully generated index and metadata in {output_dir}")
|
||||
|
||||
@@ -11,9 +11,9 @@ set -euo pipefail
|
||||
TARGET="${1:-all}"
|
||||
case "${TARGET}" in
|
||||
cuda-13-0 | cuda-12-9 | cuda-13-0-ubuntu-24-04 | \
|
||||
cuda-12-9-ubuntu-24-04 | rocm | xpu | cpu | all) ;;
|
||||
cuda-12-9-ubuntu-24-04 | rocm | rocm72 | xpu | cpu | all) ;;
|
||||
*)
|
||||
echo "Usage: $0 {cuda-13-0|cuda-12-9|cuda-13-0-ubuntu-24-04|cuda-12-9-ubuntu-24-04|rocm|xpu|cpu|all}"
|
||||
echo "Usage: $0 {cuda-13-0|cuda-12-9|cuda-13-0-ubuntu-24-04|cuda-12-9-ubuntu-24-04|rocm|rocm72|xpu|cpu|all}"
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
@@ -143,23 +143,35 @@ if target_enabled cuda-12-9-ubuntu-24-04; then
|
||||
fi
|
||||
|
||||
# ---- ROCm ----
|
||||
# Each ROCm stack is identified by its base Dockerfile (see rocm/stack.sh). The
|
||||
# default stack also owns the plain :latest / :v<ver> tags.
|
||||
|
||||
publish_rocm_stack() {
|
||||
local base_dockerfile="$1" default="$2" suffix
|
||||
(
|
||||
# shellcheck source=.buildkite/scripts/rocm/stack.sh
|
||||
source .buildkite/scripts/rocm/stack.sh "$base_dockerfile"
|
||||
local suffixes=("-${ROCM_STACK_VARIANT}")
|
||||
[[ "$default" == "1" ]] && suffixes=("" "-${ROCM_STACK_VARIANT}")
|
||||
docker pull "$ROCM_STACK_ECR_IMAGE"
|
||||
docker pull "$ROCM_STACK_ECR_BASE"
|
||||
for suffix in "${suffixes[@]}"; do
|
||||
for tag in "latest${suffix}" "v${RELEASE_VERSION}${suffix}"; do
|
||||
docker tag "$ROCM_STACK_ECR_IMAGE" "vllm/vllm-openai-rocm:${tag}"
|
||||
docker tag "$ROCM_STACK_ECR_BASE" "vllm/vllm-openai-rocm:${tag}-base"
|
||||
docker push "vllm/vllm-openai-rocm:${tag}"
|
||||
docker push "vllm/vllm-openai-rocm:${tag}-base"
|
||||
done
|
||||
done
|
||||
)
|
||||
}
|
||||
|
||||
if target_enabled rocm; then
|
||||
ROCM_BASE_CACHE_KEY=$(.buildkite/scripts/cache-rocm-base-wheels.sh key)
|
||||
echo "ROCm base cache key: ${ROCM_BASE_CACHE_KEY}"
|
||||
publish_rocm_stack docker/Dockerfile.rocm_base 1
|
||||
fi
|
||||
|
||||
docker pull "public.ecr.aws/q9t5s3a7/vllm-release-repo:${COMMIT}-rocm"
|
||||
docker pull "public.ecr.aws/q9t5s3a7/vllm-release-repo:${ROCM_BASE_CACHE_KEY}-rocm-base"
|
||||
|
||||
docker tag "public.ecr.aws/q9t5s3a7/vllm-release-repo:${COMMIT}-rocm" vllm/vllm-openai-rocm:latest
|
||||
docker tag "public.ecr.aws/q9t5s3a7/vllm-release-repo:${COMMIT}-rocm" "vllm/vllm-openai-rocm:v${RELEASE_VERSION}"
|
||||
docker push vllm/vllm-openai-rocm:latest
|
||||
docker push "vllm/vllm-openai-rocm:v${RELEASE_VERSION}"
|
||||
|
||||
docker tag "public.ecr.aws/q9t5s3a7/vllm-release-repo:${ROCM_BASE_CACHE_KEY}-rocm-base" vllm/vllm-openai-rocm:latest-base
|
||||
docker tag "public.ecr.aws/q9t5s3a7/vllm-release-repo:${ROCM_BASE_CACHE_KEY}-rocm-base" "vllm/vllm-openai-rocm:v${RELEASE_VERSION}-base"
|
||||
docker push vllm/vllm-openai-rocm:latest-base
|
||||
docker push "vllm/vllm-openai-rocm:v${RELEASE_VERSION}-base"
|
||||
if target_enabled rocm72; then
|
||||
publish_rocm_stack docker/Dockerfile.rocm_72_base 0
|
||||
fi
|
||||
|
||||
# ---- XPU ----
|
||||
|
||||
@@ -2,42 +2,36 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
#
|
||||
# Push a ROCm-family nightly base image and nightly image from ECR to Docker Hub
|
||||
# under vllm/vllm-openai-rocm, as base-nightly[-<variant>], nightly[-<variant>]
|
||||
# and their -<commit> forms.
|
||||
# Run when NIGHTLY=1 after the matching build-*-release-image step has pushed to ECR.
|
||||
# Push a ROCm stack's nightly base image and nightly image from ECR to Docker Hub
|
||||
# under vllm/vllm-openai-rocm, as base-nightly-<variant>, nightly-<variant> and their
|
||||
# -<commit> forms, then prune old nightly tags. With --default, also push the plain
|
||||
# base-nightly and nightly tags for the same images.
|
||||
# Run when NIGHTLY=1 after the stack's release image has been pushed to ECR.
|
||||
#
|
||||
# Usage: push-nightly-builds-rocm.sh [ECR_TAG_SUFFIX TAG_VARIANT]
|
||||
# Pass both arguments or neither. With none, the default ROCm apt build is
|
||||
# published as :nightly. The TheRock build passes "rock" (the source ECR tag
|
||||
# suffix) and "rocm714" (the Docker Hub tag flavor), giving :nightly-rocm714.
|
||||
# Usage: push-nightly-builds-rocm.sh <base-dockerfile> [--default]
|
||||
# e.g. "docker/Dockerfile.rocm_base --default" (ROCm 10.0, also :nightly)
|
||||
# and "docker/Dockerfile.rocm_72_base" (ROCm 7.2, :nightly-rocm72).
|
||||
#
|
||||
# Local testing (no push to Docker Hub):
|
||||
# BASE_ECR_IMAGE=<full-base-image-reference> \
|
||||
# BUILDKITE_COMMIT=<commit-with-image-in-ecr> DRY_RUN=1 \
|
||||
# bash .buildkite/scripts/push-nightly-builds-rocm.sh
|
||||
# BUILDKITE_COMMIT=<commit-with-image-in-ecr> DRY_RUN=1 \
|
||||
# bash .buildkite/scripts/push-nightly-builds-rocm.sh docker/Dockerfile.rocm_base --default
|
||||
# Requires: AWS CLI configured (for ECR public login), Docker. For full run: Docker Hub login.
|
||||
|
||||
set -euxo pipefail
|
||||
|
||||
usage() {
|
||||
echo "Usage: $0 [ECR_TAG_SUFFIX TAG_VARIANT]" >&2
|
||||
echo "Usage: $0 <base-dockerfile> [--default]" >&2
|
||||
}
|
||||
|
||||
DOCKERHUB_REPO="vllm/vllm-openai-rocm"
|
||||
|
||||
PUSH_DEFAULT=0
|
||||
case "$#" in
|
||||
0)
|
||||
ECR_TAG_SUFFIX="rocm"
|
||||
TAG_VARIANT=""
|
||||
;;
|
||||
1) ;;
|
||||
2)
|
||||
if [[ -z "$1" || -z "$2" ]]; then
|
||||
if [[ "$2" != "--default" ]]; then
|
||||
usage
|
||||
exit 2
|
||||
fi
|
||||
ECR_TAG_SUFFIX="$1"
|
||||
TAG_VARIANT="$2"
|
||||
PUSH_DEFAULT=1
|
||||
;;
|
||||
*)
|
||||
usage
|
||||
@@ -49,61 +43,44 @@ esac
|
||||
BUILDKITE_COMMIT="${BUILDKITE_COMMIT:?Set BUILDKITE_COMMIT to the commit SHA that has the image in ECR (e.g. from a previous release pipeline run)}"
|
||||
DRY_RUN="${DRY_RUN:-0}"
|
||||
|
||||
ECR_REPO="public.ecr.aws/q9t5s3a7/vllm-release-repo"
|
||||
# shellcheck source=.buildkite/scripts/rocm/stack.sh
|
||||
source "$(dirname "${BASH_SOURCE[0]}")/rocm/stack.sh" "$1"
|
||||
DOCKERHUB_REPO="vllm/vllm-openai-rocm"
|
||||
VARIANT="$ROCM_STACK_VARIANT"
|
||||
|
||||
# Get the base image ECR tag (set by the build-*-release-image pipeline step)
|
||||
BASE_METADATA_KEY="${ECR_TAG_SUFFIX}-base-ecr-tag"
|
||||
if [[ -n "${BASE_ECR_IMAGE:-}" ]]; then
|
||||
if [[ "$DRY_RUN" != "1" ]]; then
|
||||
echo "ERROR: BASE_ECR_IMAGE may only be used with DRY_RUN=1" >&2
|
||||
exit 1
|
||||
fi
|
||||
BASE_ORIG_TAG="$BASE_ECR_IMAGE"
|
||||
elif ! BASE_ORIG_TAG="$(buildkite-agent meta-data get "$BASE_METADATA_KEY")"; then
|
||||
echo "ERROR: Failed to read required Buildkite metadata '$BASE_METADATA_KEY'" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [[ -z "$BASE_ORIG_TAG" ]]; then
|
||||
echo "ERROR: Required Buildkite metadata '$BASE_METADATA_KEY' is empty" >&2
|
||||
echo "Set BASE_ECR_IMAGE to the full ECR base image reference for local testing" >&2
|
||||
exit 1
|
||||
BASE_TAGS=("base-nightly-${VARIANT}" "base-nightly-${VARIANT}-${BUILDKITE_COMMIT}")
|
||||
TAGS=("nightly-${VARIANT}" "nightly-${VARIANT}-${BUILDKITE_COMMIT}")
|
||||
if [[ "$PUSH_DEFAULT" == "1" ]]; then
|
||||
BASE_TAGS+=("base-nightly" "base-nightly-${BUILDKITE_COMMIT}")
|
||||
TAGS+=("nightly" "nightly-${BUILDKITE_COMMIT}")
|
||||
fi
|
||||
|
||||
ORIG_TAG="${BUILDKITE_COMMIT}-${ECR_TAG_SUFFIX}"
|
||||
VARIANT_SUFFIX=""
|
||||
[[ -n "$TAG_VARIANT" ]] && VARIANT_SUFFIX="-$TAG_VARIANT"
|
||||
BASE_TAG_NAME="base-nightly${VARIANT_SUFFIX}"
|
||||
TAG_NAME="nightly${VARIANT_SUFFIX}"
|
||||
BASE_TAG_NAME_COMMIT="base-nightly${VARIANT_SUFFIX}-${BUILDKITE_COMMIT}"
|
||||
TAG_NAME_COMMIT="nightly${VARIANT_SUFFIX}-${BUILDKITE_COMMIT}"
|
||||
echo "Pushing base image $ROCM_STACK_ECR_BASE as: ${BASE_TAGS[*]}"
|
||||
echo "Pushing release image $ROCM_STACK_ECR_IMAGE as: ${TAGS[*]}"
|
||||
|
||||
echo "Pushing base image from ECR: $BASE_ORIG_TAG"
|
||||
echo "Pushing release image from ECR tag: $ORIG_TAG to Docker Hub as $DOCKERHUB_REPO:$TAG_NAME and $DOCKERHUB_REPO:$TAG_NAME_COMMIT"
|
||||
[[ "$DRY_RUN" == "1" ]] && echo "[DRY_RUN] Skipping push to Docker Hub"
|
||||
|
||||
# Login to ECR and pull the images built by build-*-release-image
|
||||
# Login to ECR and pull the stack's images
|
||||
aws ecr-public get-login-password --region us-east-1 | docker login --username AWS --password-stdin public.ecr.aws/q9t5s3a7
|
||||
docker pull "$BASE_ORIG_TAG"
|
||||
docker pull "${ECR_REPO}:${ORIG_TAG}"
|
||||
docker pull "$ROCM_STACK_ECR_BASE"
|
||||
docker pull "$ROCM_STACK_ECR_IMAGE"
|
||||
|
||||
# Tag for Docker Hub (base-nightly and base-nightly-<commit>, nightly and nightly-<commit>)
|
||||
docker tag "$BASE_ORIG_TAG" "$DOCKERHUB_REPO":"$BASE_TAG_NAME"
|
||||
docker tag "$BASE_ORIG_TAG" "$DOCKERHUB_REPO":"$BASE_TAG_NAME_COMMIT"
|
||||
docker tag "${ECR_REPO}:${ORIG_TAG}" "$DOCKERHUB_REPO":"$TAG_NAME"
|
||||
docker tag "${ECR_REPO}:${ORIG_TAG}" "$DOCKERHUB_REPO":"$TAG_NAME_COMMIT"
|
||||
for tag in "${BASE_TAGS[@]}"; do docker tag "$ROCM_STACK_ECR_BASE" "$DOCKERHUB_REPO:$tag"; done
|
||||
for tag in "${TAGS[@]}"; do docker tag "$ROCM_STACK_ECR_IMAGE" "$DOCKERHUB_REPO:$tag"; done
|
||||
|
||||
if [[ "$DRY_RUN" == "1" ]]; then
|
||||
echo "[DRY_RUN] Would push $DOCKERHUB_REPO:$BASE_TAG_NAME and $DOCKERHUB_REPO:$BASE_TAG_NAME_COMMIT"
|
||||
echo "[DRY_RUN] Would push $DOCKERHUB_REPO:$TAG_NAME and $DOCKERHUB_REPO:$TAG_NAME_COMMIT"
|
||||
echo "[DRY_RUN] Local tags created. Exiting without push."
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# Push to Docker Hub (docker-login plugin runs before this step in CI)
|
||||
docker push "$DOCKERHUB_REPO":"$BASE_TAG_NAME"
|
||||
docker push "$DOCKERHUB_REPO":"$BASE_TAG_NAME_COMMIT"
|
||||
docker push "$DOCKERHUB_REPO":"$TAG_NAME"
|
||||
docker push "$DOCKERHUB_REPO":"$TAG_NAME_COMMIT"
|
||||
for tag in "${BASE_TAGS[@]}" "${TAGS[@]}"; do docker push "$DOCKERHUB_REPO:$tag"; done
|
||||
echo "Pushed $DOCKERHUB_REPO: ${BASE_TAGS[*]} ${TAGS[*]}"
|
||||
|
||||
echo "Pushed $DOCKERHUB_REPO:$BASE_TAG_NAME and $DOCKERHUB_REPO:$BASE_TAG_NAME_COMMIT"
|
||||
echo "Pushed $DOCKERHUB_REPO:$TAG_NAME and $DOCKERHUB_REPO:$TAG_NAME_COMMIT"
|
||||
# Keep only the last 14 builds of each tag family. The plain nightly- families
|
||||
# exclude the variant ones sharing their prefix; commit SHAs never start with "rocm".
|
||||
CLEANUP=.buildkite/scripts/cleanup-nightly-builds.sh
|
||||
bash "$CLEANUP" "nightly-${VARIANT}-" "$DOCKERHUB_REPO"
|
||||
bash "$CLEANUP" "base-nightly-${VARIANT}-" "$DOCKERHUB_REPO"
|
||||
if [[ "$PUSH_DEFAULT" == "1" ]]; then
|
||||
bash "$CLEANUP" "nightly-" "$DOCKERHUB_REPO" "nightly-rocm"
|
||||
bash "$CLEANUP" "base-nightly-" "$DOCKERHUB_REPO" "base-nightly-rocm"
|
||||
fi
|
||||
|
||||
@@ -1,8 +1,8 @@
|
||||
#!/usr/bin/env bash
|
||||
# Shared Dockerfile selection for the normal AMD CI build chain.
|
||||
# CI_ROCM_DOCKERFILE_BASE and CI_ROCM_DOCKERFILE accept repository-relative paths.
|
||||
# For Rock, set CI_ROCM_DOCKERFILE_BASE=docker/Dockerfile.rock_base and
|
||||
# CI_ROCM_DOCKERFILE=docker/Dockerfile.rock.
|
||||
# For the legacy ROCm 7.2 stack, set CI_ROCM_DOCKERFILE_BASE=docker/Dockerfile.rocm_72_base
|
||||
# and CI_ROCM_DOCKERFILE=docker/Dockerfile.rocm_72.
|
||||
# Custom recipes must preserve the CI targets and artifact layout. Additional
|
||||
# COPY inputs also need the corresponding *_CONTENT_FILES overrides.
|
||||
|
||||
|
||||
@@ -0,0 +1,78 @@
|
||||
#!/usr/bin/env bash
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
#
|
||||
# Derive everything the release pipeline needs about a ROCm stack from its base
|
||||
# Dockerfile, so the stack's image is the single input to every ROCm script.
|
||||
#
|
||||
# Usage: source .buildkite/scripts/rocm/stack.sh <base-dockerfile>
|
||||
# e.g. docker/Dockerfile.rocm_base (ROCm 10.0) or docker/Dockerfile.rocm_72_base
|
||||
#
|
||||
# Exports:
|
||||
# ROCM_STACK_BASE_DOCKERFILE the base Dockerfile
|
||||
# ROCM_STACK_DOCKERFILE the matching vLLM Dockerfile (base name minus _base)
|
||||
# ROCM_STACK_THEROCK 1 if the ROCm SDK comes from TheRock wheels
|
||||
# ROCM_STACK_VERSION ROCm version, e.g. 10.0.0 or 7.2.3
|
||||
# ROCM_STACK_VARIANT wheel variant and Docker Hub tag flavor from major.minor,
|
||||
# same rule as setup.py (rocm100, rocm72)
|
||||
# ROCM_STACK_IMAGE_KEY hash of the base Dockerfile (base image cache key)
|
||||
# ROCM_STACK_WHEEL_KEY hash of the base Dockerfile plus files it COPYs from
|
||||
# the repo (base wheel cache key)
|
||||
# ROCM_STACK_ECR_BASE ECR tag of the cached base image
|
||||
# ROCM_STACK_ECR_IMAGE ECR tag of this commit's release image
|
||||
# ROCM_STACK_BASE_WHEELS_DIR local dir for base wheels (also the artifact path)
|
||||
# ROCM_STACK_VLLM_WHEEL_DIR local dir for the vLLM wheel (also the artifact path)
|
||||
|
||||
rocm_stack_init() {
|
||||
local base="$1"
|
||||
if [[ ! -f "$base" ]]; then
|
||||
echo "ROCm base Dockerfile not found: $base" >&2
|
||||
return 2
|
||||
fi
|
||||
ROCM_STACK_BASE_DOCKERFILE="$base"
|
||||
ROCM_STACK_DOCKERFILE="${base%_base}"
|
||||
|
||||
local sdk
|
||||
sdk=$(sed -nE 's/^ARG ROCM_SDK_VERSION=([0-9.]+).*/\1/p' "$base" | head -1)
|
||||
if [[ -n "$sdk" ]]; then
|
||||
ROCM_STACK_THEROCK=1
|
||||
ROCM_STACK_VERSION="$sdk"
|
||||
else
|
||||
ROCM_STACK_THEROCK=0
|
||||
# BASE_IMAGE format: rocm/dev-ubuntu-22.04:7.2.3-complete -> 7.2.3
|
||||
ROCM_STACK_VERSION=$(sed -nE 's/^ARG BASE_IMAGE=.*:([0-9]+(\.[0-9]+)+).*/\1/p' "$base" | head -1)
|
||||
fi
|
||||
if [[ -z "$ROCM_STACK_VERSION" ]]; then
|
||||
echo "Could not determine the ROCm version from $base" >&2
|
||||
return 2
|
||||
fi
|
||||
|
||||
local major minor
|
||||
IFS=. read -r major minor _ <<< "$ROCM_STACK_VERSION"
|
||||
ROCM_STACK_VARIANT="rocm${major}${minor}"
|
||||
|
||||
ROCM_STACK_IMAGE_KEY=$(sha256sum "$base" | cut -c1-16)
|
||||
local inputs
|
||||
inputs=$(grep -E '^COPY ' "$base" | grep -v -- '--from' | awk '{print $2}' || true)
|
||||
if [[ -z "$inputs" ]]; then
|
||||
ROCM_STACK_WHEEL_KEY="$ROCM_STACK_IMAGE_KEY"
|
||||
else
|
||||
# shellcheck disable=SC2086
|
||||
ROCM_STACK_WHEEL_KEY=$(sha256sum "$base" $inputs | sha256sum | cut -c1-16)
|
||||
fi
|
||||
|
||||
local ecr="public.ecr.aws/q9t5s3a7/vllm-release-repo"
|
||||
ROCM_STACK_ECR_BASE="${ecr}:${ROCM_STACK_IMAGE_KEY}-${ROCM_STACK_VARIANT}-base"
|
||||
ROCM_STACK_ECR_IMAGE="${ecr}:${BUILDKITE_COMMIT:-unknown}-${ROCM_STACK_VARIANT}"
|
||||
ROCM_STACK_BASE_WHEELS_DIR="artifacts/${ROCM_STACK_VARIANT}-base-wheels"
|
||||
ROCM_STACK_VLLM_WHEEL_DIR="artifacts/${ROCM_STACK_VARIANT}-vllm-wheel"
|
||||
|
||||
export ROCM_STACK_BASE_DOCKERFILE ROCM_STACK_DOCKERFILE ROCM_STACK_THEROCK \
|
||||
ROCM_STACK_VERSION ROCM_STACK_VARIANT ROCM_STACK_IMAGE_KEY \
|
||||
ROCM_STACK_WHEEL_KEY ROCM_STACK_ECR_BASE ROCM_STACK_ECR_IMAGE \
|
||||
ROCM_STACK_BASE_WHEELS_DIR ROCM_STACK_VLLM_WHEEL_DIR
|
||||
}
|
||||
|
||||
if [[ $# -gt 0 ]]; then
|
||||
rocm_stack_init "$1"
|
||||
fi
|
||||
@@ -4,21 +4,31 @@
|
||||
#
|
||||
# Upload ROCm wheels to S3 with proper index generation
|
||||
#
|
||||
# Usage: upload-rocm-wheels.sh <base-dockerfile>
|
||||
# base-dockerfile the ROCm stack, e.g. docker/Dockerfile.rocm_base (see rocm/stack.sh)
|
||||
#
|
||||
# Required environment variables:
|
||||
# AWS_ACCESS_KEY_ID, AWS_SECRET_ACCESS_KEY (or IAM role)
|
||||
# S3_BUCKET (default: vllm-wheels)
|
||||
#
|
||||
# S3 path structure:
|
||||
# s3://vllm-wheels/rocm/{commit}/ - All wheels for this commit
|
||||
# s3://vllm-wheels/rocm/nightly/ - Index pointing to latest nightly
|
||||
# s3://vllm-wheels/rocm/{version}/ - Index for release versions
|
||||
# s3://vllm-wheels/rocm/{commit}/<variant>-wheels/ - This stack's wheels for this commit
|
||||
# s3://vllm-wheels/rocm/{commit}/<variant>/ - Index per ROCm variant (e.g. rocm100)
|
||||
# s3://vllm-wheels/rocm/nightly/<variant>/ - Index pointing to latest nightly
|
||||
# s3://vllm-wheels/rocm/{version}/<variant>/ - Index for release versions
|
||||
# The top-level index.html at each prefix lists every variant present, since
|
||||
# several ROCm stacks publish into the same prefixes.
|
||||
|
||||
set -ex
|
||||
|
||||
# ======== Configuration ========
|
||||
BUCKET="${S3_BUCKET:-vllm-wheels}"
|
||||
# shellcheck source=.buildkite/scripts/rocm/stack.sh
|
||||
source "$(dirname "${BASH_SOURCE[0]}")/rocm/stack.sh" "${1:?Usage: $0 <base-dockerfile>}"
|
||||
ROCM_SUBPATH="rocm/${BUILDKITE_COMMIT}"
|
||||
S3_COMMIT_PREFIX="s3://$BUCKET/$ROCM_SUBPATH/"
|
||||
WHEEL_DIRS="$ROCM_STACK_BASE_WHEELS_DIR $ROCM_STACK_VLLM_WHEEL_DIR"
|
||||
WHEEL_SUBPATH="$ROCM_SUBPATH/${ROCM_STACK_VARIANT}-wheels"
|
||||
S3_WHEEL_PREFIX="s3://$BUCKET/$WHEEL_SUBPATH/"
|
||||
INDICES_OUTPUT_DIR="rocm-indices"
|
||||
|
||||
echo "========================================"
|
||||
@@ -49,9 +59,11 @@ source .buildkite/scripts/lib/manylinux.sh
|
||||
# ======== Part 1: Collect and prepare wheels ========
|
||||
|
||||
# Collect all wheels
|
||||
rm -rf all-rocm-wheels "$INDICES_OUTPUT_DIR"
|
||||
mkdir -p all-rocm-wheels
|
||||
cp artifacts/rocm-base-wheels/*.whl all-rocm-wheels/ 2>/dev/null || true
|
||||
cp artifacts/rocm-vllm-wheel/*.whl all-rocm-wheels/ 2>/dev/null || true
|
||||
for dir in $WHEEL_DIRS; do
|
||||
cp "$dir"/*.whl all-rocm-wheels/ 2>/dev/null || true
|
||||
done
|
||||
|
||||
WHEEL_COUNT=$(find all-rocm-wheels -maxdepth 1 -name '*.whl' 2>/dev/null | wc -l)
|
||||
echo "Total wheels to upload: $WHEEL_COUNT"
|
||||
@@ -84,9 +96,9 @@ ls -lh all-rocm-wheels/
|
||||
# ======== Part 2: Upload wheels to S3 ========
|
||||
|
||||
echo ""
|
||||
echo "Uploading wheels to $S3_COMMIT_PREFIX"
|
||||
echo "Uploading wheels to $S3_WHEEL_PREFIX"
|
||||
for wheel in all-rocm-wheels/*.whl; do
|
||||
aws s3 cp "$wheel" "$S3_COMMIT_PREFIX"
|
||||
aws s3 cp "$wheel" "$S3_WHEEL_PREFIX"
|
||||
done
|
||||
|
||||
# ======== Part 3: Generate and upload indices ========
|
||||
@@ -95,28 +107,63 @@ done
|
||||
echo ""
|
||||
echo "Generating indices..."
|
||||
obj_json="rocm-objects.json"
|
||||
aws s3api list-objects-v2 --bucket "$BUCKET" --prefix "$ROCM_SUBPATH/" --delimiter / --output json > "$obj_json"
|
||||
aws s3api list-objects-v2 --bucket "$BUCKET" --prefix "$WHEEL_SUBPATH/" --delimiter / --output json > "$obj_json"
|
||||
|
||||
mkdir -p "$INDICES_OUTPUT_DIR"
|
||||
|
||||
# Use the existing generate-nightly-index.py
|
||||
# HACK: Replace regex module with stdlib re (same as CUDA script)
|
||||
sed -i 's/import regex as re/import re/g' .buildkite/scripts/generate-nightly-index.py
|
||||
sed -i 's/import regex as re/import re/g' tools/vllm-rocm/therock_wheels.py
|
||||
sed -i 's/import pybase64 as base64/import base64/g' tools/vllm-rocm/therock_wheels.py
|
||||
|
||||
INDEX_ARGS=(--version "$ROCM_SUBPATH" --wheel-dir "${WHEEL_SUBPATH#rocm/}")
|
||||
# TheRock's SDK, device kernels and torchvision/torchaudio are linked, not hosted
|
||||
if [[ "$ROCM_STACK_THEROCK" == "1" ]]; then
|
||||
$PYTHON tools/vllm-rocm/therock_wheels.py external-links \
|
||||
--dockerfile "$ROCM_STACK_BASE_DOCKERFILE" \
|
||||
--torch-wheel "$(ls "$ROCM_STACK_BASE_WHEELS_DIR"/torch-*.whl)" > therock-external-links.txt
|
||||
cat therock-external-links.txt
|
||||
INDEX_ARGS+=(--external-links therock-external-links.txt)
|
||||
fi
|
||||
$PYTHON .buildkite/scripts/generate-nightly-index.py \
|
||||
--version "$ROCM_SUBPATH" \
|
||||
"${INDEX_ARGS[@]}" \
|
||||
--current-objects "$obj_json" \
|
||||
--output-dir "$INDICES_OUTPUT_DIR" \
|
||||
--comment "ROCm commit $BUILDKITE_COMMIT"
|
||||
|
||||
# Upload indices to commit directory
|
||||
echo "Uploading indices to $S3_COMMIT_PREFIX"
|
||||
aws s3 cp --recursive "$INDICES_OUTPUT_DIR/" "$S3_COMMIT_PREFIX"
|
||||
# Upload this build's variant indices under an S3 prefix (e.g. rocm/nightly), then
|
||||
# rebuild that prefix's top-level project list from every variant now present.
|
||||
publish_indices() {
|
||||
local prefix="$1"
|
||||
local listing variants merged
|
||||
echo "Uploading indices to s3://$BUCKET/$prefix/"
|
||||
aws s3 cp --recursive "$INDICES_OUTPUT_DIR/" "s3://$BUCKET/$prefix/" --exclude "index.html"
|
||||
listing=$(aws s3api list-objects-v2 --bucket "$BUCKET" --prefix "$prefix/" \
|
||||
--delimiter / --query 'CommonPrefixes[].Prefix' --output text)
|
||||
variants=$(tr '\t' '\n' <<< "$listing" | sed -n "s|^$prefix/\(rocm[0-9]*\)/$|\1|p" | sort -u)
|
||||
merged=$(mktemp)
|
||||
{
|
||||
echo '<!DOCTYPE html>'
|
||||
echo '<html>'
|
||||
echo ' <meta name="pypi:repository-version" content="1.0">'
|
||||
echo ' <body>'
|
||||
for variant in $variants; do
|
||||
echo " <a href=\"$variant/\">$variant/</a><br/>"
|
||||
done
|
||||
echo ' </body>'
|
||||
echo '</html>'
|
||||
} > "$merged"
|
||||
echo "Variants under $prefix/: $(echo "$variants" | tr '\n' ' ')"
|
||||
aws s3 cp "$merged" "s3://$BUCKET/$prefix/index.html"
|
||||
rm -f "$merged"
|
||||
}
|
||||
|
||||
publish_indices "$ROCM_SUBPATH"
|
||||
|
||||
# Only scheduled nightly builds should update the moving nightly index.
|
||||
if [[ "${NIGHTLY:-0}" == "1" ]]; then
|
||||
echo "Updating rocm/nightly/ index..."
|
||||
aws s3 cp --recursive "$INDICES_OUTPUT_DIR/" "s3://$BUCKET/rocm/nightly/"
|
||||
publish_indices "rocm/nightly"
|
||||
fi
|
||||
|
||||
# Extract version from vLLM wheel and update version-specific index
|
||||
@@ -129,8 +176,7 @@ if [ -n "$VLLM_WHEEL" ]; then
|
||||
echo "Pure version: $PURE_VERSION"
|
||||
|
||||
if [[ "$VERSION" != *"dev"* ]]; then
|
||||
echo "Updating rocm/$PURE_VERSION/ index..."
|
||||
aws s3 cp --recursive "$INDICES_OUTPUT_DIR/" "s3://$BUCKET/rocm/$PURE_VERSION/"
|
||||
publish_indices "rocm/$PURE_VERSION"
|
||||
fi
|
||||
fi
|
||||
|
||||
@@ -142,14 +188,14 @@ echo "ROCm Wheel Upload Complete!"
|
||||
echo "========================================"
|
||||
echo ""
|
||||
echo "Wheels available at:"
|
||||
echo " s3://$BUCKET/$ROCM_SUBPATH/"
|
||||
echo " s3://$BUCKET/$WHEEL_SUBPATH/"
|
||||
echo ""
|
||||
echo "Install command (by commit):"
|
||||
echo " pip install vllm --extra-index-url https://${BUCKET}.s3.amazonaws.com/$ROCM_SUBPATH/"
|
||||
echo " pip install vllm --extra-index-url https://${BUCKET}.s3.amazonaws.com/$ROCM_SUBPATH/$ROCM_STACK_VARIANT/"
|
||||
echo ""
|
||||
if [[ "${NIGHTLY:-0}" == "1" ]]; then
|
||||
echo "Install command (nightly):"
|
||||
echo " pip install vllm --extra-index-url https://${BUCKET}.s3.amazonaws.com/rocm/nightly/"
|
||||
echo " pip install vllm --extra-index-url https://${BUCKET}.s3.amazonaws.com/rocm/nightly/$ROCM_STACK_VARIANT/"
|
||||
fi
|
||||
echo ""
|
||||
echo "Wheel count: $WHEEL_COUNT"
|
||||
|
||||
+212
-300
@@ -399,27 +399,6 @@ steps:
|
||||
commands:
|
||||
- pytest -v -s v1/e2e/spec_decode/draft_model/
|
||||
|
||||
- label: ":amd: (MI250) Spec Decode Speculators + MTP" # TBD
|
||||
timeout_in_minutes: 75
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx90anightly, amdmi250]
|
||||
dind: false
|
||||
agent_pool: mi250_1
|
||||
optional: true
|
||||
working_dir: "/vllm-workspace/tests"
|
||||
source_file_dependencies:
|
||||
- vllm/v1/spec_decode/
|
||||
- vllm/v1/worker/gpu/spec_decode/
|
||||
- vllm/model_executor/model_loader/
|
||||
- vllm/v1/sample/
|
||||
- vllm/model_executor/layers/
|
||||
- vllm/transformers_utils/configs/speculators/
|
||||
- tests/v1/e2e/spec_decode/
|
||||
- vllm/platforms/rocm.py
|
||||
- vllm/v1/attention/backends/
|
||||
commands:
|
||||
- pytest -v -s v1/e2e/spec_decode/speculators/
|
||||
- pytest -v -s v1/e2e/spec_decode/mtp/
|
||||
|
||||
#------------------------------------------------------------- mi250 · misc ------------------------------------------------------------#
|
||||
|
||||
- label: ":amd: (MI300) Python-only Installation" # TBD
|
||||
@@ -1094,25 +1073,6 @@ steps:
|
||||
|
||||
#----------------------------------------------------------- mi300 · evals -----------------------------------------------------------#
|
||||
|
||||
- label: ":amd: (MI300) LM Eval Small Models" # TBD
|
||||
timeout_in_minutes: 45
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
|
||||
dind: false
|
||||
agent_pool: mi300_1
|
||||
working_dir: "/vllm-workspace/tests"
|
||||
source_file_dependencies:
|
||||
- csrc/
|
||||
- vllm/model_executor/layers/quantization
|
||||
- vllm/model_executor/models/
|
||||
- vllm/model_executor/model_loader/
|
||||
- vllm/v1/attention/backends/
|
||||
- vllm/v1/attention/selector.py
|
||||
- vllm/_aiter_ops.py
|
||||
- vllm/platforms/rocm.py
|
||||
optional: true
|
||||
commands:
|
||||
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-small.txt
|
||||
|
||||
- label: ":amd: (MI300) DeepSeek V2-Lite Prefetch Offload Accuracy" # TBD
|
||||
timeout_in_minutes: 30
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
|
||||
@@ -1153,23 +1113,6 @@ steps:
|
||||
commands:
|
||||
- pytest -s -v evals/mrcr/test_mrcr_correctness.py --config-list-file=evals/mrcr/configs/models-small.txt
|
||||
|
||||
- label: ":amd: (MI300) LM Eval KV-Offload" # TBD
|
||||
timeout_in_minutes: 55
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
|
||||
dind: false
|
||||
agent_pool: mi300_1
|
||||
optional: true
|
||||
working_dir: "/vllm-workspace/tests"
|
||||
source_file_dependencies:
|
||||
- vllm/distributed/kv_transfer/kv_connector/v1/offloading/
|
||||
- vllm/distributed/kv_transfer/kv_connector/v1/simple_cpu_offload_connector.py
|
||||
- vllm/v1/kv_offload/
|
||||
- vllm/v1/simple_kv_offload/
|
||||
- tests/evals/gsm8k/test_gsm8k_offloading.py
|
||||
- vllm/platforms/rocm.py
|
||||
commands:
|
||||
- pytest -s -v evals/gsm8k/test_gsm8k_offloading.py -k "nemotron-h-8b or gemma-4-e4b-it"
|
||||
|
||||
- label: ":amd: (MI300) LM Eval KV-Offload Medium" # TBD
|
||||
timeout_in_minutes: 70
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
|
||||
@@ -1188,24 +1131,6 @@ steps:
|
||||
commands:
|
||||
- pytest -s -v evals/gsm8k/test_gsm8k_offloading.py -k "qwen3.5-35b or deepseek-v2-lite"
|
||||
|
||||
- label: ":amd: (MI300) LM Eval KV-Offload Large" # TBD
|
||||
timeout_in_minutes: 65
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
|
||||
dind: false
|
||||
agent_pool: mi300_4
|
||||
num_gpus: 4
|
||||
optional: true
|
||||
working_dir: "/vllm-workspace/tests"
|
||||
source_file_dependencies:
|
||||
- vllm/distributed/kv_transfer/kv_connector/v1/offloading/
|
||||
- vllm/distributed/kv_transfer/kv_connector/v1/simple_cpu_offload_connector.py
|
||||
- vllm/v1/kv_offload/
|
||||
- vllm/v1/simple_kv_offload/
|
||||
- tests/evals/gsm8k/test_gsm8k_offloading.py
|
||||
- vllm/platforms/rocm.py
|
||||
commands:
|
||||
- VLLM_ROCM_USE_AITER=1 pytest -s -v evals/gsm8k/test_gsm8k_offloading.py -k "deepseek-v4-flash"
|
||||
|
||||
- label: ":amd: (MI300) LM Eval Small Models Harness" # TBD
|
||||
timeout_in_minutes: 35
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
|
||||
@@ -1383,52 +1308,6 @@ steps:
|
||||
commands:
|
||||
- VLLM_ENGINE_READY_TIMEOUT_S=1800 bash .buildkite/scripts/scheduled_integration_test/qwen3_next_mtp_async_eplb.sh 0.8 1319 8040
|
||||
|
||||
- label: ":amd: (MI300) LM Eval Large Models" # TBD
|
||||
timeout_in_minutes: 65
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
|
||||
dind: false
|
||||
agent_pool: mi300_8
|
||||
optional: true
|
||||
num_gpus: 8
|
||||
working_dir: "/vllm-workspace/tests"
|
||||
source_file_dependencies:
|
||||
- vllm/model_executor/models/
|
||||
- vllm/model_executor/model_loader/
|
||||
- vllm/model_executor/layers/quantization/
|
||||
- vllm/v1/attention/backends/
|
||||
- vllm/v1/attention/selector.py
|
||||
- vllm/model_executor/layers/layernorm.py
|
||||
- csrc/
|
||||
- vllm/_aiter_ops.py
|
||||
- vllm/platforms/rocm.py
|
||||
- tests/evals/
|
||||
commands:
|
||||
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
- export PYTORCH_ROCM_ARCH=gfx942
|
||||
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi3xx.txt
|
||||
|
||||
- label: ":amd: (MI300) LM Eval Large Models ROCm Harness" # TBD
|
||||
timeout_in_minutes: 40
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
|
||||
dind: false
|
||||
agent_pool: mi300_8
|
||||
optional: true
|
||||
num_gpus: 8
|
||||
working_dir: "/vllm-workspace/.buildkite/lm-eval-harness"
|
||||
source_file_dependencies:
|
||||
- vllm/model_executor/models/
|
||||
- vllm/model_executor/model_loader/
|
||||
- vllm/model_executor/layers/quantization/
|
||||
- vllm/v1/attention/backends/
|
||||
- vllm/v1/attention/selector.py
|
||||
- vllm/model_executor/layers/layernorm.py
|
||||
- csrc/
|
||||
- vllm/_aiter_ops.py
|
||||
- vllm/platforms/rocm.py
|
||||
commands:
|
||||
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
- pytest -s -v test_lm_eval_correctness.py --config-list-file=configs/models-large-rocm.txt --tp-size=8 --timeout=5400
|
||||
|
||||
- label: ":amd: (MI300) LM Eval Large Models FP8" # TBD
|
||||
timeout_in_minutes: 40
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
|
||||
@@ -1939,20 +1818,6 @@ steps:
|
||||
commands:
|
||||
- pytest -v -s models/language/pooling_mteb_test
|
||||
|
||||
- label: ":amd: (MI300) Language Models (Extended Pooling) Shard %N"
|
||||
timeout_in_minutes: 95
|
||||
parallelism: 4
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
|
||||
dind: false
|
||||
agent_pool: mi300_1
|
||||
optional: true
|
||||
working_dir: "/vllm-workspace/tests"
|
||||
source_file_dependencies:
|
||||
- vllm/
|
||||
- tests/models/language/pooling
|
||||
commands:
|
||||
- pytest -v -s models/language/pooling -m 'not core_model' --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB
|
||||
|
||||
- label: ":amd: (MI300) Language Models (Extended Generation)" # TBD
|
||||
timeout_in_minutes: 105
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
|
||||
@@ -1970,21 +1835,6 @@ steps:
|
||||
|
||||
#---------------------------------------------------- mi300 · models / multimodal ----------------------------------------------------#
|
||||
|
||||
- label: ":amd: (MI300) Multimodal Models (Extended Generation 1)" # TBD
|
||||
timeout_in_minutes: 100
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
|
||||
dind: false
|
||||
agent_pool: mi300_1
|
||||
optional: true
|
||||
working_dir: "/vllm-workspace/tests"
|
||||
source_file_dependencies:
|
||||
- vllm/
|
||||
- tests/models/multimodal/generation
|
||||
- tests/models/multimodal/test_mapping.py
|
||||
commands:
|
||||
- pytest -v -s models/multimodal/generation -m 'not core_model' --ignore models/multimodal/generation/test_common.py
|
||||
- pytest -v -s models/multimodal/test_mapping.py
|
||||
|
||||
- label: ":amd: (MI300) Multimodal Models (Extended Generation 2) Shard %N"
|
||||
timeout_in_minutes: 100
|
||||
parallelism: 4
|
||||
@@ -2013,19 +1863,6 @@ steps:
|
||||
commands:
|
||||
- pytest -v -s models/multimodal/generation/test_common.py -m 'split(group=1) and not core_model'
|
||||
|
||||
- label: ":amd: (MI300) Multimodal Models (PPL)"
|
||||
timeout_in_minutes: 90
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
|
||||
dind: false
|
||||
agent_pool: mi300_1
|
||||
optional: true
|
||||
working_dir: "/vllm-workspace/tests"
|
||||
source_file_dependencies:
|
||||
- vllm/
|
||||
- tests/models/multimodal/generation_ppl_test/
|
||||
commands:
|
||||
- pytest -v -s models/multimodal/generation_ppl_test/
|
||||
|
||||
- label: ":amd: (MI300) Multimodal Models (Standard) 3: llava + qwen2_vl" # TBD
|
||||
timeout_in_minutes: 55
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
|
||||
@@ -2057,19 +1894,6 @@ steps:
|
||||
commands:
|
||||
- pytest -v -s models/multimodal/processing/test_tensor_schema.py --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB
|
||||
|
||||
- label: ":amd: (MI300) Multimodal Models (Extended Pooling)" # TBD
|
||||
timeout_in_minutes: 75
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
|
||||
dind: false
|
||||
agent_pool: mi300_1
|
||||
optional: true
|
||||
working_dir: "/vllm-workspace/tests"
|
||||
source_file_dependencies:
|
||||
- vllm/
|
||||
- tests/models/multimodal/pooling
|
||||
commands:
|
||||
- pytest -v -s models/multimodal/pooling -m 'not core_model'
|
||||
|
||||
#----------------------------------------------------- mi300 · models / quantized -----------------------------------------------------#
|
||||
|
||||
- label: ":amd: (MI300) Quantized Models" # TBD
|
||||
@@ -2533,27 +2357,6 @@ steps:
|
||||
- VLLM_TP_SIZE=4 VLLM_NEEDLE_TRIALS=2 pytest -v -s v1/determinism/test_batch_invariance.py::test_v1_generation_is_deterministic_across_batch_sizes_with_needle -k TRITON_ATTN
|
||||
- VLLM_TEST_MODEL=allenai/OLMoE-1B-7B-0924 VLLM_TEST_SERVER_ARGS="--data-parallel-size 4 --enable-expert-parallel --max-model-len 4096" pytest -v -s v1/determinism/test_online_batch_invariance.py
|
||||
|
||||
- label: ":amd: (MI300) Spec Decode Speculators + MTP" # TBD
|
||||
timeout_in_minutes: 75
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
|
||||
dind: false
|
||||
agent_pool: mi300_1
|
||||
optional: true
|
||||
working_dir: "/vllm-workspace/tests"
|
||||
source_file_dependencies:
|
||||
- vllm/v1/spec_decode/
|
||||
- vllm/v1/worker/gpu/spec_decode/
|
||||
- vllm/model_executor/model_loader/
|
||||
- vllm/v1/sample/
|
||||
- vllm/model_executor/layers/
|
||||
- vllm/transformers_utils/configs/speculators/
|
||||
- tests/v1/e2e/spec_decode/
|
||||
- vllm/platforms/rocm.py
|
||||
- vllm/v1/attention/backends/
|
||||
commands:
|
||||
- pytest -v -s v1/e2e/spec_decode/speculators/
|
||||
- pytest -v -s v1/e2e/spec_decode/mtp/
|
||||
|
||||
- label: ":amd: (MI300) Spec Decode DeepSeek MTP Parallel Load"
|
||||
timeout_in_minutes: 45
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
|
||||
@@ -2581,60 +2384,6 @@ steps:
|
||||
commands:
|
||||
- pytest -v -s v1/e2e/spec_decode/test_mtp_parallel_load.py
|
||||
|
||||
- label: ":amd: (MI300) Spec Decode AL DFlash Nightly"
|
||||
timeout_in_minutes: 120
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
|
||||
dind: false
|
||||
agent_pool: mi300_1
|
||||
optional: true
|
||||
working_dir: "/vllm-workspace/tests"
|
||||
source_file_dependencies:
|
||||
- vllm/v1/spec_decode/
|
||||
- vllm/v1/worker/gpu/spec_decode/
|
||||
- vllm/model_executor/models/qwen3_dflash.py
|
||||
- vllm/model_executor/models/qwen3_dflash2.py
|
||||
- vllm/model_executor/models/qwen3_5.py
|
||||
- vllm/model_executor/models/laguna_dflash.py
|
||||
- tests/v1/e2e/spec_decode/
|
||||
- vllm/v1/attention/
|
||||
- vllm/v1/sample/
|
||||
- vllm/model_executor/model_loader/
|
||||
- vllm/model_executor/layers/
|
||||
- vllm/model_executor/models/qwen2.py
|
||||
- vllm/model_executor/models/qwen3.py
|
||||
- vllm/model_executor/models/laguna.py
|
||||
- vllm/platforms/rocm.py
|
||||
commands:
|
||||
- pytest -v -s v1/e2e/spec_decode/acceptance_rates/dflash/ -k "not dflash2"
|
||||
|
||||
- label: ":amd: (MI300) Spec Decode AL DSpark Nightly"
|
||||
timeout_in_minutes: 90
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
|
||||
dind: false
|
||||
agent_pool: mi300_1
|
||||
optional: true
|
||||
working_dir: "/vllm-workspace/tests"
|
||||
source_file_dependencies:
|
||||
- vllm/v1/spec_decode/
|
||||
- vllm/v1/worker/gpu/spec_decode/
|
||||
- vllm/model_executor/models/qwen3_dspark.py
|
||||
- vllm/model_executor/models/gemma4_dspark.py
|
||||
- tests/v1/e2e/spec_decode/
|
||||
- vllm/v1/attention/
|
||||
- vllm/v1/sample/
|
||||
- vllm/model_executor/model_loader/
|
||||
- vllm/model_executor/layers/
|
||||
- vllm/model_executor/models/qwen2.py
|
||||
- vllm/model_executor/models/qwen3.py
|
||||
- vllm/model_executor/models/qwen3_dflash.py
|
||||
- vllm/model_executor/models/gemma4.py
|
||||
- vllm/model_executor/models/gemma4_mm.py
|
||||
- vllm/model_executor/models/gemma4_unified.py
|
||||
- vllm/model_executor/models/gemma4_mtp.py
|
||||
- vllm/platforms/rocm.py
|
||||
commands:
|
||||
- pytest -v -s v1/e2e/spec_decode/acceptance_rates/dspark/
|
||||
|
||||
- label: ":amd: (MI300) V1 Spec Decode" # TBD
|
||||
timeout_in_minutes: 50
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
|
||||
@@ -2719,39 +2468,6 @@ steps:
|
||||
- pytest -v -s v1/engine --ignore v1/engine/test_preprocess_error_handling.py
|
||||
- pytest -v -s v1/test_tensor_ipc_queue.py
|
||||
|
||||
- label: ":amd: (MI300) Speculators Correctness Nightly"
|
||||
timeout_in_minutes: 40
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
|
||||
dind: false
|
||||
agent_pool: mi300_1
|
||||
optional: true
|
||||
working_dir: "/vllm-workspace/tests"
|
||||
source_file_dependencies:
|
||||
- vllm/config/speculative.py
|
||||
- vllm/engine/arg_utils.py
|
||||
- vllm/transformers_utils/config.py
|
||||
- vllm/transformers_utils/configs/speculators/
|
||||
- vllm/v1/spec_decode/
|
||||
- vllm/v1/worker/gpu/spec_decode/
|
||||
- vllm/v1/worker/gpu_model_runner.py
|
||||
- vllm/v1/sample/
|
||||
- vllm/v1/attention/backends/
|
||||
- vllm/v1/attention/selector.py
|
||||
- vllm/model_executor/model_loader/
|
||||
- vllm/model_executor/layers/
|
||||
- vllm/model_executor/models/llama_eagle3.py
|
||||
- vllm/model_executor/models/qwen3.py
|
||||
- vllm/model_executor/models/qwen3_dflash.py
|
||||
- vllm/model_executor/models/registry.py
|
||||
- vllm/_aiter_ops.py
|
||||
- tests/evals/gsm8k/
|
||||
- tests/v1/spec_decode/test_speculators_correctness.py
|
||||
- vllm/platforms/rocm.py
|
||||
- vllm/v1/worker/gpu/spec_decode/dflash/
|
||||
commands:
|
||||
- export VLLM_ALLOW_INSECURE_SERIALIZATION=1
|
||||
- pytest -v -s v1/spec_decode/test_speculators_correctness.py -m slow_test
|
||||
|
||||
- label: ":amd: (MI300) Extract Hidden States Integration" # TBD
|
||||
timeout_in_minutes: 30
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
|
||||
@@ -3106,22 +2822,6 @@ steps:
|
||||
- uv pip install --system -r /vllm-workspace/requirements/kv_connectors_rocm.txt
|
||||
- ATTENTION_BACKEND=TRITON_ATTN bash v1/kv_connector/nixl_integration/run_multi_connector_edge_case_test.sh
|
||||
|
||||
- label: ":amd: (MI300) V1 E2E Hybrid Chunked Prefill" # TBD
|
||||
timeout_in_minutes: 35
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx942nightly, amdmi300]
|
||||
dind: false
|
||||
agent_pool: mi300_4
|
||||
num_gpus: 4
|
||||
optional: true
|
||||
source_file_dependencies:
|
||||
- vllm/v1/attention/backends/utils.py
|
||||
- vllm/v1/worker/gpu_model_runner.py
|
||||
- tests/v1/e2e/test_hybrid_chunked_prefill.py
|
||||
- vllm/v1/core/
|
||||
working_dir: /vllm-workspace/tests
|
||||
commands:
|
||||
- pytest -v -s v1/e2e/test_hybrid_chunked_prefill.py
|
||||
|
||||
#########################################################################################################################################
|
||||
# #
|
||||
# MI355 (gfx950) tests #
|
||||
@@ -4604,6 +4304,82 @@ steps:
|
||||
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi355-nemotron-3-super.txt
|
||||
|
||||
- label: ":amd: (MI355) LM Eval Small Models" # TBD
|
||||
timeout_in_minutes: 45
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]
|
||||
dind: false
|
||||
agent_pool: mi355_dpx
|
||||
working_dir: "/vllm-workspace/tests"
|
||||
source_file_dependencies:
|
||||
- csrc/
|
||||
- vllm/model_executor/layers/quantization
|
||||
- vllm/model_executor/models/
|
||||
- vllm/model_executor/model_loader/
|
||||
- vllm/v1/attention/backends/
|
||||
- vllm/v1/attention/selector.py
|
||||
- vllm/_aiter_ops.py
|
||||
- vllm/platforms/rocm.py
|
||||
optional: true
|
||||
commands:
|
||||
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-small.txt
|
||||
|
||||
- label: ":amd: (MI355 DPX) LM Eval KV-Offload" # TBD
|
||||
timeout_in_minutes: 55
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]
|
||||
dind: false
|
||||
agent_pool: mi355_dpx
|
||||
optional: true
|
||||
working_dir: "/vllm-workspace/tests"
|
||||
source_file_dependencies:
|
||||
- vllm/distributed/kv_transfer/kv_connector/v1/offloading/
|
||||
- vllm/distributed/kv_transfer/kv_connector/v1/simple_cpu_offload_connector.py
|
||||
- vllm/v1/kv_offload/
|
||||
- vllm/v1/simple_kv_offload/
|
||||
- tests/evals/gsm8k/test_gsm8k_offloading.py
|
||||
- vllm/platforms/rocm.py
|
||||
commands:
|
||||
- pytest -s -v evals/gsm8k/test_gsm8k_offloading.py -k "nemotron-h-8b or gemma-4-e4b-it"
|
||||
|
||||
- label: ":amd: (MI355) LM Eval KV-Offload Large" # TBD
|
||||
timeout_in_minutes: 65
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]
|
||||
dind: false
|
||||
agent_pool: mi355_4
|
||||
num_gpus: 4
|
||||
optional: true
|
||||
working_dir: "/vllm-workspace/tests"
|
||||
source_file_dependencies:
|
||||
- vllm/distributed/kv_transfer/kv_connector/v1/offloading/
|
||||
- vllm/distributed/kv_transfer/kv_connector/v1/simple_cpu_offload_connector.py
|
||||
- vllm/v1/kv_offload/
|
||||
- vllm/v1/simple_kv_offload/
|
||||
- tests/evals/gsm8k/test_gsm8k_offloading.py
|
||||
- vllm/platforms/rocm.py
|
||||
commands:
|
||||
- VLLM_ROCM_USE_AITER=1 pytest -s -v evals/gsm8k/test_gsm8k_offloading.py -k "deepseek-v4-flash"
|
||||
|
||||
- label: ":amd: (MI355) LM Eval Large Models ROCm Harness" # TBD
|
||||
timeout_in_minutes: 40
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]
|
||||
dind: false
|
||||
agent_pool: mi355_8
|
||||
optional: true
|
||||
num_gpus: 8
|
||||
working_dir: "/vllm-workspace/.buildkite/lm-eval-harness"
|
||||
source_file_dependencies:
|
||||
- vllm/model_executor/models/
|
||||
- vllm/model_executor/model_loader/
|
||||
- vllm/model_executor/layers/quantization/
|
||||
- vllm/v1/attention/backends/
|
||||
- vllm/v1/attention/selector.py
|
||||
- vllm/model_executor/layers/layernorm.py
|
||||
- csrc/
|
||||
- vllm/_aiter_ops.py
|
||||
- vllm/platforms/rocm.py
|
||||
commands:
|
||||
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
- pytest -s -v test_lm_eval_correctness.py --config-list-file=configs/models-large-rocm.txt --tp-size=8 --timeout=5400
|
||||
|
||||
#--------------------------------------------------------- mi355 · examples ----------------------------------------------------------#
|
||||
|
||||
- label: ":amd: (MI355 DPX) Examples" # TBD
|
||||
@@ -5421,6 +5197,19 @@ steps:
|
||||
- pytest models/multimodal/generation/test_memory_leak.py -m core_model
|
||||
- cd .. && VLLM_WORKER_MULTIPROC_METHOD=spawn pytest -v -s tests/models/multimodal/generation/test_whisper.py -m core_model
|
||||
|
||||
- label: ":amd: (MI355 DPX) Multimodal Models (PPL)"
|
||||
timeout_in_minutes: 90
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]
|
||||
dind: false
|
||||
agent_pool: mi355_dpx
|
||||
optional: true
|
||||
working_dir: "/vllm-workspace/tests"
|
||||
source_file_dependencies:
|
||||
- vllm/
|
||||
- tests/models/multimodal/generation_ppl_test/
|
||||
commands:
|
||||
- pytest -v -s models/multimodal/generation_ppl_test/
|
||||
|
||||
#----------------------------------------------------- mi355 · models / quantized -----------------------------------------------------#
|
||||
|
||||
- label: ":amd: (MI355 DPX) Quantized Models" # TBD
|
||||
@@ -5565,6 +5354,27 @@ steps:
|
||||
commands:
|
||||
- pytest -v -s v1/e2e/spec_decode/draft_model/
|
||||
|
||||
- label: ":amd: (MI355 DPX) Spec Decode Speculators + MTP" # TBD
|
||||
timeout_in_minutes: 75
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]
|
||||
dind: false
|
||||
agent_pool: mi355_dpx
|
||||
optional: true
|
||||
working_dir: "/vllm-workspace/tests"
|
||||
source_file_dependencies:
|
||||
- vllm/v1/spec_decode/
|
||||
- vllm/v1/worker/gpu/spec_decode/
|
||||
- vllm/model_executor/model_loader/
|
||||
- vllm/v1/sample/
|
||||
- vllm/model_executor/layers/
|
||||
- vllm/transformers_utils/configs/speculators/
|
||||
- tests/v1/e2e/spec_decode/
|
||||
- vllm/platforms/rocm.py
|
||||
- vllm/v1/attention/backends/
|
||||
commands:
|
||||
- pytest -v -s v1/e2e/spec_decode/speculators/
|
||||
- pytest -v -s v1/e2e/spec_decode/mtp/
|
||||
|
||||
- label: ":amd: (MI355 DPX) Spec Decode Eagle 1: DeepSeek + Qwen"
|
||||
timeout_in_minutes: 60
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]
|
||||
@@ -6041,6 +5851,108 @@ steps:
|
||||
- if [ "$$BUILDKITE_PARALLEL_JOB" = "1" ]; then pytest -v -s "v1/e2e/spec_decode/acceptance_rates/mtp_other/test_mtp.py::test_gemma4_mtp_acceptance_lengths[False]"; fi
|
||||
- if [ "$$BUILDKITE_PARALLEL_JOB" = "2" ]; then pytest -v -s "v1/e2e/spec_decode/acceptance_rates/mtp_other/test_mtp.py::test_gemma4_mtp_acceptance_lengths[True]"; fi
|
||||
|
||||
- label: ":amd: (MI355 DPX) V1 E2E Hybrid Chunked Prefill" # TBD
|
||||
timeout_in_minutes: 35
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]
|
||||
dind: false
|
||||
agent_pool: mi355_dpx
|
||||
optional: true
|
||||
source_file_dependencies:
|
||||
- vllm/v1/attention/backends/utils.py
|
||||
- vllm/v1/worker/gpu_model_runner.py
|
||||
- tests/v1/e2e/test_hybrid_chunked_prefill.py
|
||||
- vllm/v1/core/
|
||||
working_dir: /vllm-workspace/tests
|
||||
commands:
|
||||
- pytest -v -s v1/e2e/test_hybrid_chunked_prefill.py
|
||||
|
||||
- label: ":amd: (MI355 DPX) Speculators Correctness Nightly"
|
||||
timeout_in_minutes: 40
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]
|
||||
dind: false
|
||||
agent_pool: mi355_dpx
|
||||
optional: true
|
||||
working_dir: "/vllm-workspace/tests"
|
||||
source_file_dependencies:
|
||||
- vllm/config/speculative.py
|
||||
- vllm/engine/arg_utils.py
|
||||
- vllm/transformers_utils/config.py
|
||||
- vllm/transformers_utils/configs/speculators/
|
||||
- vllm/v1/spec_decode/
|
||||
- vllm/v1/worker/gpu/spec_decode/
|
||||
- vllm/v1/worker/gpu_model_runner.py
|
||||
- vllm/v1/sample/
|
||||
- vllm/v1/attention/backends/
|
||||
- vllm/v1/attention/selector.py
|
||||
- vllm/model_executor/model_loader/
|
||||
- vllm/model_executor/layers/
|
||||
- vllm/model_executor/models/llama_eagle3.py
|
||||
- vllm/model_executor/models/qwen3.py
|
||||
- vllm/model_executor/models/qwen3_dflash.py
|
||||
- vllm/model_executor/models/registry.py
|
||||
- vllm/_aiter_ops.py
|
||||
- tests/evals/gsm8k/
|
||||
- tests/v1/spec_decode/test_speculators_correctness.py
|
||||
- vllm/platforms/rocm.py
|
||||
- vllm/v1/worker/gpu/spec_decode/dflash/
|
||||
commands:
|
||||
- export VLLM_ALLOW_INSECURE_SERIALIZATION=1
|
||||
- pytest -v -s v1/spec_decode/test_speculators_correctness.py -m slow_test
|
||||
|
||||
- label: ":amd: (MI355 DPX) Spec Decode AL DFlash Nightly"
|
||||
timeout_in_minutes: 120
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]
|
||||
dind: false
|
||||
agent_pool: mi355_dpx
|
||||
optional: true
|
||||
working_dir: "/vllm-workspace/tests"
|
||||
source_file_dependencies:
|
||||
- vllm/v1/spec_decode/
|
||||
- vllm/v1/worker/gpu/spec_decode/
|
||||
- vllm/model_executor/models/qwen3_dflash.py
|
||||
- vllm/model_executor/models/qwen3_dflash2.py
|
||||
- vllm/model_executor/models/qwen3_5.py
|
||||
- vllm/model_executor/models/laguna_dflash.py
|
||||
- tests/v1/e2e/spec_decode/
|
||||
- vllm/v1/attention/
|
||||
- vllm/v1/sample/
|
||||
- vllm/model_executor/model_loader/
|
||||
- vllm/model_executor/layers/
|
||||
- vllm/model_executor/models/qwen2.py
|
||||
- vllm/model_executor/models/qwen3.py
|
||||
- vllm/model_executor/models/laguna.py
|
||||
- vllm/platforms/rocm.py
|
||||
commands:
|
||||
- pytest -v -s v1/e2e/spec_decode/acceptance_rates/dflash/ -k "not dflash2"
|
||||
|
||||
- label: ":amd: (MI355 DPX) Spec Decode AL DSpark Nightly"
|
||||
timeout_in_minutes: 90
|
||||
mirror_hardwares: [amdexperimental, amdproduction, amdgfx950nightly, amdmi355]
|
||||
dind: false
|
||||
agent_pool: mi355_dpx
|
||||
optional: true
|
||||
working_dir: "/vllm-workspace/tests"
|
||||
source_file_dependencies:
|
||||
- vllm/v1/spec_decode/
|
||||
- vllm/v1/worker/gpu/spec_decode/
|
||||
- vllm/model_executor/models/qwen3_dspark.py
|
||||
- vllm/model_executor/models/gemma4_dspark.py
|
||||
- tests/v1/e2e/spec_decode/
|
||||
- vllm/v1/attention/
|
||||
- vllm/v1/sample/
|
||||
- vllm/model_executor/model_loader/
|
||||
- vllm/model_executor/layers/
|
||||
- vllm/model_executor/models/qwen2.py
|
||||
- vllm/model_executor/models/qwen3.py
|
||||
- vllm/model_executor/models/qwen3_dflash.py
|
||||
- vllm/model_executor/models/gemma4.py
|
||||
- vllm/model_executor/models/gemma4_mm.py
|
||||
- vllm/model_executor/models/gemma4_unified.py
|
||||
- vllm/model_executor/models/gemma4_mtp.py
|
||||
- vllm/platforms/rocm.py
|
||||
commands:
|
||||
- pytest -v -s v1/e2e/spec_decode/acceptance_rates/dspark/
|
||||
|
||||
#----------------------------------------------------------- mi355 · misc ------------------------------------------------------------#
|
||||
|
||||
- label: ":amd: (MI355 DPX) Regression" # TBD
|
||||
|
||||
@@ -238,9 +238,9 @@ steps:
|
||||
- pytest -v -s v1/e2e/test_hybrid_chunked_prefill.py
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI300) V1 E2E Hybrid Chunked Prefill"
|
||||
label: ":amd: (MI355 DPX) V1 E2E Hybrid Chunked Prefill"
|
||||
dind: false
|
||||
device: mi300_1
|
||||
device: mi355_dpx
|
||||
timeout_in_minutes: 35
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
|
||||
@@ -545,7 +545,6 @@ steps:
|
||||
num_devices: 1
|
||||
source_file_dependencies:
|
||||
- cmake/external_projects/deepgemm.cmake
|
||||
- tools/install_deepgemm.sh
|
||||
- tools/build_deepgemm_C.py
|
||||
- tools/setup_deepgemm_pythons.sh
|
||||
- tools/check_wheel_deepgemm.py
|
||||
@@ -939,7 +938,7 @@ steps:
|
||||
|
||||
- label: ":nvidia: (B200) FusedMoE Layer Kernels"
|
||||
key: kernels-fusedmoe-layer-test-2-b200s
|
||||
timeout_in_minutes: 90
|
||||
timeout_in_minutes: 45
|
||||
device: b200-k8s
|
||||
num_devices: 2
|
||||
source_file_dependencies:
|
||||
|
||||
@@ -15,9 +15,9 @@ steps:
|
||||
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-small.txt
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI300) LM Eval Small Models"
|
||||
label: ":amd: (MI355) LM Eval Small Models"
|
||||
dind: false
|
||||
device: mi300_1
|
||||
device: mi355_dpx
|
||||
timeout_in_minutes: 45
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
@@ -707,18 +707,17 @@ steps:
|
||||
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi355-nemotron-3-super.txt
|
||||
|
||||
- label: ":amd: (MI300) LM Eval Large Models"
|
||||
- label: ":amd: (MI355) LM Eval Large Models"
|
||||
key: amd-lm-eval-large-models
|
||||
dind: false
|
||||
device: mi300_8
|
||||
device: mi355_8
|
||||
optional: true
|
||||
timeout_in_minutes: 65
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
commands:
|
||||
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
- export PYTORCH_ROCM_ARCH=gfx942 # Limit Quark compilation to save time
|
||||
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-mi3xx.txt
|
||||
- pytest -s -v evals/gsm8k/test_gsm8k_correctness.py --config-list-file=configs/models-gfx950-large.txt
|
||||
source_file_dependencies:
|
||||
- vllm/_aiter_ops.py
|
||||
- vllm/platforms/rocm.py
|
||||
@@ -1298,9 +1297,9 @@ steps:
|
||||
- pytest -s -v evals/gsm8k/test_gsm8k_offloading.py -k "nemotron-h-8b or gemma-4-e4b-it"
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI300) LM Eval KV-Offload"
|
||||
label: ":amd: (MI355 DPX) LM Eval KV-Offload"
|
||||
dind: false
|
||||
device: mi300_1
|
||||
device: mi355_dpx
|
||||
timeout_in_minutes: 55
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
@@ -1360,9 +1359,9 @@ steps:
|
||||
- pytest -s -v evals/gsm8k/test_gsm8k_offloading.py -k "deepseek-v4-flash"
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI300) LM Eval KV-Offload Large"
|
||||
label: ":amd: (MI355) LM Eval KV-Offload Large"
|
||||
dind: false
|
||||
device: mi300_4
|
||||
device: mi355_4
|
||||
timeout_in_minutes: 65
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
@@ -1427,10 +1426,10 @@ steps:
|
||||
# Qwen2-57B has 28 attention heads and cannot run at TP8.
|
||||
- pytest -s -v test_lm_eval_correctness.py --config-list-file=configs/models-large.txt --tp-size=4
|
||||
|
||||
- label: ":amd: (MI300) LM Eval Large Models ROCm Harness"
|
||||
- label: ":amd: (MI355) LM Eval Large Models ROCm Harness"
|
||||
key: amd-lm-eval-large-models-rocm-harness
|
||||
dind: false
|
||||
device: mi300_8
|
||||
device: mi355_8
|
||||
optional: true
|
||||
timeout_in_minutes: 40
|
||||
depends_on:
|
||||
|
||||
@@ -164,6 +164,59 @@ steps:
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
|
||||
# The shared KV Connectors steps above exclude vllm/distributed/kv_transfer/ from
|
||||
# their triggers, and CUDA skips MoRIIO's mori/aiter-gated tests. Run the whole
|
||||
# MoRIIO suite on ROCm whenever the connector or its tests change.
|
||||
- label: ":amd: (MI355 DPX) MoRIIO Connector Unit Tests"
|
||||
key: amd-moriio-connector-unit-mi355
|
||||
dind: false
|
||||
device: mi355_dpx
|
||||
num_devices: 1
|
||||
timeout_in_minutes: 30
|
||||
working_dir: /vllm-workspace/tests
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
source_file_dependencies:
|
||||
- vllm/distributed/kv_transfer/kv_connector/v1/moriio/
|
||||
- vllm/distributed/kv_transfer/kv_connector/v1/base.py
|
||||
- vllm/distributed/kv_transfer/kv_connector/v1/ssm_conv_transfer_utils.py
|
||||
- examples/disaggregated/disaggregated_serving/moriio_toy_proxy_server.py
|
||||
- tests/v1/kv_connector/unit/test_moriio_
|
||||
- tests/v1/kv_connector/unit/utils.py
|
||||
- requirements/kv_connectors_rocm.txt
|
||||
commands:
|
||||
- bash /vllm-workspace/.buildkite/scripts/install-kv-connectors.sh
|
||||
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
- pytest -v -s -rs v1/kv_connector/unit/test_moriio_*.py
|
||||
|
||||
# Single-node MoRIIO WRITE P/D on dummy weights: prefill and decode on separate
|
||||
# GPUs over xGMI, compared exactly against a direct serve.
|
||||
- label: ":amd: (MI355) MoRIIO WRITE P/D Dummy-Weight E2E Shard %N"
|
||||
key: amd-moriio-write-dummy-e2e-mi355
|
||||
dind: false
|
||||
device: mi355_2
|
||||
num_devices: 2
|
||||
# ~3.5-5 min per case; pytest-shard hashes test ids, so with 20 cases the
|
||||
# largest of 6 shards holds 5 (~22 min on MI355). Every case picks free ports
|
||||
# for its servers and proxy and its own compile-cache dir.
|
||||
parallelism: 6
|
||||
timeout_in_minutes: 45
|
||||
working_dir: /vllm-workspace/tests
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
source_file_dependencies:
|
||||
- vllm/distributed/kv_transfer/kv_connector/v1/moriio/
|
||||
- vllm/distributed/kv_transfer/kv_connector/v1/base.py
|
||||
- vllm/distributed/kv_transfer/kv_connector/v1/ssm_conv_transfer_utils.py
|
||||
- vllm/model_executor/model_loader/dummy_loader.py
|
||||
- examples/disaggregated/disaggregated_serving/moriio_toy_proxy_server.py
|
||||
- tests/v1/kv_connector/moriio_integration/
|
||||
- requirements/kv_connectors_rocm.txt
|
||||
commands:
|
||||
- bash /vllm-workspace/.buildkite/scripts/install-kv-connectors.sh
|
||||
- export VLLM_WORKER_MULTIPROC_METHOD=spawn
|
||||
- pytest -v -s -rs v1/kv_connector/moriio_integration/test_write_dummy_e2e_rocm.py --shard-id=$$BUILDKITE_PARALLEL_JOB --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT
|
||||
|
||||
- label: ":nvidia: (H200 MIG 35GB) V1 Metrics + LM Eval"
|
||||
device: h200_35gb
|
||||
key: v1-metrics-lmeval
|
||||
@@ -214,6 +267,7 @@ steps:
|
||||
- vllm/v1/
|
||||
- tests/v1
|
||||
- tests/watermarking
|
||||
- examples/disaggregated/mooncake_connector/mooncake_connector_proxy.py
|
||||
device: cpu-small
|
||||
commands:
|
||||
# split the test to avoid interference
|
||||
|
||||
@@ -72,8 +72,9 @@ steps:
|
||||
- tests/models/test_deepseek_v41_pcp.py
|
||||
- tests/models/transformers/test_layer_registry_e2e.py
|
||||
- tests/models/test_deepseek_v41_replay_start.py
|
||||
- tests/models/test_deepseek_v4_vl_stream_weights.py
|
||||
commands:
|
||||
- pytest -v -s models/test_terratorch.py models/transformers/test_backend.py models/test_registry.py models/test_deepseek_v4_vl_rocm.py models/transformers/test_layer_registry_e2e.py models/test_deepseek_v41_replay_start.py models/test_deepseek_v41_decoder_replay_layers.py models/test_deepseek_v41_replay_batch.py models/test_deepseek_v41_pcp.py
|
||||
- pytest -v -s models/test_terratorch.py models/transformers/test_backend.py models/test_registry.py models/test_deepseek_v4_vl_rocm.py models/transformers/test_layer_registry_e2e.py models/test_deepseek_v41_replay_start.py models/test_deepseek_v41_decoder_replay_layers.py models/test_deepseek_v41_replay_batch.py models/test_deepseek_v41_pcp.py models/test_deepseek_v4_vl_stream_weights.py
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI355) Basic Models (Other)"
|
||||
|
||||
@@ -210,15 +210,18 @@ steps:
|
||||
- "!vllm/distributed/kv_transfer/"
|
||||
- tests/models/language/pooling
|
||||
commands:
|
||||
- pytest -v -s models/language/pooling -m 'not core_model'
|
||||
# Per-test timeout: a hung test fails with a stack dump instead of holding
|
||||
# the job until Buildkite cancels it, which also blocks the full run's
|
||||
# coverage collection. A whole shard takes 12-21 minutes.
|
||||
- pytest -v -s models/language/pooling -m 'not core_model' --timeout=900 --timeout-method=thread
|
||||
mirror:
|
||||
amd:
|
||||
parallelism: 4
|
||||
commands:
|
||||
- "pytest -v -s models/language/pooling -m 'not core_model' --num-shards=$$BUILDKITE_PARALLEL_JOB_COUNT --shard-id=$$BUILDKITE_PARALLEL_JOB"
|
||||
label: ":amd: (MI300) Language Models (Extended Pooling) Shard %N"
|
||||
label: ":amd: (MI355 DPX) Language Models (Extended Pooling) Shard %N"
|
||||
dind: false
|
||||
device: mi300_1
|
||||
device: mi355_dpx
|
||||
timeout_in_minutes: 95
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
|
||||
@@ -163,9 +163,9 @@ steps:
|
||||
- pytest -v -s models/multimodal/test_mapping.py
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI300) Multimodal Models (Extended Generation 1)"
|
||||
label: ":amd: (MI355 DPX) Multimodal Models (Extended Generation 1)"
|
||||
dind: false
|
||||
device: mi300_1
|
||||
device: mi355_dpx
|
||||
timeout_in_minutes: 100
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
@@ -182,9 +182,9 @@ steps:
|
||||
- pytest -v -s models/multimodal/generation_ppl_test/
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI300) Multimodal Models (PPL)"
|
||||
label: ":amd: (MI355 DPX) Multimodal Models (PPL)"
|
||||
dind: false
|
||||
device: mi300_1
|
||||
device: mi355_dpx
|
||||
timeout_in_minutes: 90
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
@@ -240,9 +240,9 @@ steps:
|
||||
- pytest -v -s models/multimodal/pooling -m 'not core_model'
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI300) Multimodal Models (Extended Pooling)"
|
||||
label: ":amd: (MI355 DPX) Multimodal Models (Extended Pooling)"
|
||||
dind: false
|
||||
device: mi300_1
|
||||
device: mi355_dpx
|
||||
timeout_in_minutes: 75
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
|
||||
@@ -137,9 +137,9 @@ steps:
|
||||
- pytest -v -s v1/e2e/spec_decode/speculators/
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI300) Spec Decode Speculators"
|
||||
label: ":amd: (MI355 DPX) Spec Decode Speculators"
|
||||
dind: false
|
||||
device: mi300_1
|
||||
device: mi355_dpx
|
||||
timeout_in_minutes: 40
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
@@ -167,9 +167,9 @@ steps:
|
||||
- pytest -v -s v1/e2e/spec_decode/mtp/gemma4/
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI300) Spec Decode MTP Gemma4"
|
||||
label: ":amd: (MI355 DPX) Spec Decode MTP Gemma4"
|
||||
dind: false
|
||||
device: mi300_1
|
||||
device: mi355_dpx
|
||||
timeout_in_minutes: 40
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
@@ -198,9 +198,9 @@ steps:
|
||||
- pytest -v -s v1/e2e/spec_decode/mtp/mimo/
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI300) Spec Decode MTP DeepSeek + MiMo"
|
||||
label: ":amd: (MI355 DPX) Spec Decode MTP DeepSeek + MiMo"
|
||||
dind: false
|
||||
device: mi300_1
|
||||
device: mi355_dpx
|
||||
timeout_in_minutes: 40
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
@@ -228,9 +228,9 @@ steps:
|
||||
- pytest -v -s v1/e2e/spec_decode/mtp/qwen3_5/
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI300) Spec Decode MTP Qwen3.5"
|
||||
label: ":amd: (MI355 DPX) Spec Decode MTP Qwen3.5"
|
||||
dind: false
|
||||
device: mi300_1
|
||||
device: mi355_dpx
|
||||
timeout_in_minutes: 40
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
@@ -387,9 +387,9 @@ steps:
|
||||
- pytest -v -s v1/spec_decode/test_speculators_correctness.py -m slow_test
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI300) Speculators Correctness Nightly"
|
||||
label: ":amd: (MI355 DPX) Speculators Correctness Nightly"
|
||||
dind: false
|
||||
device: mi300_1
|
||||
device: mi355_dpx
|
||||
timeout_in_minutes: 40
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
@@ -452,9 +452,9 @@ steps:
|
||||
- pytest -v -s v1/e2e/spec_decode/acceptance_rates/dflash/ -k "not dflash2"
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI300) Spec Decode AL DFlash Nightly"
|
||||
label: ":amd: (MI355 DPX) Spec Decode AL DFlash Nightly"
|
||||
dind: false
|
||||
device: mi300_1
|
||||
device: mi355_dpx
|
||||
timeout_in_minutes: 120
|
||||
commands:
|
||||
- pytest -v -s v1/e2e/spec_decode/acceptance_rates/dflash/ -k "not dflash2"
|
||||
@@ -531,9 +531,9 @@ steps:
|
||||
- pytest -v -s v1/e2e/spec_decode/acceptance_rates/dspark/
|
||||
mirror:
|
||||
amd:
|
||||
label: ":amd: (MI300) Spec Decode AL DSpark Nightly"
|
||||
label: ":amd: (MI355 DPX) Spec Decode AL DSpark Nightly"
|
||||
dind: false
|
||||
device: mi300_1
|
||||
device: mi355_dpx
|
||||
timeout_in_minutes: 90
|
||||
depends_on:
|
||||
- image-build-amd
|
||||
|
||||
+5
-1
@@ -8,7 +8,7 @@
|
||||
/vllm/distributed/kv_transfer/kv_connector/v1/hisparse @MatthewBonanni @LucasWilkinson
|
||||
/vllm/lora @jeejeelee
|
||||
/vllm/model_executor/layers/attention @LucasWilkinson @MatthewBonanni
|
||||
/vllm/model_executor/layers/fused_moe @mgoin @pavanimajety @zyongye @wzhao18
|
||||
/vllm/model_executor/layers/fused_moe @mgoin @pavanimajety @zyongye @wzhao18 @bnellnm
|
||||
/vllm/model_executor/layers/quantization @mgoin @robertgshaw2-redhat @tlrmchlsmth @yewentao256 @pavanimajety @zyongye
|
||||
/vllm/model_executor/layers/mamba @tdoublep @tomeras91 @ZJY0516
|
||||
/vllm/model_executor/layers/mamba/gdn/qwen_gdn_linear_attn.py @tdoublep @ZJY0516 @vadiklyutiy
|
||||
@@ -138,6 +138,7 @@
|
||||
/tests/evals @mgoin @vadiklyutiy @AndreasKaratzas @ZJY0516
|
||||
/tests/kernels @mgoin @tlrmchlsmth @WoosukKwon @yewentao256 @zyongye @AndreasKaratzas
|
||||
/tests/kernels/ir @ProExpertProg @tjtanaa
|
||||
/tests/kernels/moe @bnellnm
|
||||
/tests/models @DarkLight1337 @ywang96 @AndreasKaratzas
|
||||
/tests/multimodal @DarkLight1337 @ywang96 @NickLucche @shen-shanshan
|
||||
/tests/multimodal/media @Isotr0py
|
||||
@@ -164,6 +165,9 @@
|
||||
.readthedocs.yaml @hmellor
|
||||
mkdocs.yaml @hmellor
|
||||
|
||||
# Model recipes
|
||||
/recipes @esmeetu
|
||||
|
||||
# Linting
|
||||
/tools/pre_commit/.markdownlint.yaml @hmellor
|
||||
.pre-commit-config.yaml @hmellor
|
||||
|
||||
@@ -0,0 +1,28 @@
|
||||
name: Recipes
|
||||
|
||||
# Validate recipe changes against the renderer that publishes them
|
||||
# (recipes.vllm.ai). The check logic lives in vllm-project/recipes and tracks
|
||||
# its main branch, so a PR is held to the same validation that runs when
|
||||
# vllm-project/recipes picks up this directory after merge.
|
||||
|
||||
on:
|
||||
pull_request:
|
||||
paths:
|
||||
- 'recipes/**'
|
||||
- '.github/workflows/recipes.yml'
|
||||
|
||||
concurrency:
|
||||
group: ${{ github.workflow }}-${{ github.ref }}
|
||||
cancel-in-progress: true
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
jobs:
|
||||
validate:
|
||||
name: Recipes
|
||||
if: github.repository_owner == 'vllm-project'
|
||||
uses: vllm-project/recipes/.github/workflows/validate-recipes.yml@main
|
||||
with:
|
||||
source: vllm
|
||||
path: recipes
|
||||
@@ -621,6 +621,16 @@ def blocks_new_run(command: str, build: Mapping[str, Any]) -> bool:
|
||||
return is_active_build(build)
|
||||
|
||||
|
||||
def list_commit_builds(
|
||||
buildkite: BuildkiteClient, command: str, commit: str
|
||||
) -> list[dict[str, Any]]:
|
||||
builds = buildkite.list_builds(commit)
|
||||
if command not in AMD_CI_COMMANDS:
|
||||
return builds
|
||||
# bootstrap-amd.sh exits before uploading any test steps for PR webhook builds.
|
||||
return [build for build in builds if build.get("source") != "webhook"]
|
||||
|
||||
|
||||
def select_latest_build(
|
||||
builds: Sequence[dict[str, Any]],
|
||||
pr_number: int,
|
||||
@@ -968,7 +978,7 @@ def handle_run_ci(
|
||||
f"{ci_name} was already requested by this comment: {duplicate['web_url']}"
|
||||
)
|
||||
|
||||
current_builds = buildkite.list_builds(pr["head"]["sha"])
|
||||
current_builds = list_commit_builds(buildkite, command, pr["head"]["sha"])
|
||||
active_build = next(
|
||||
(
|
||||
build
|
||||
@@ -1023,7 +1033,7 @@ def handle_retry_failed(
|
||||
ci_name = ci_name_for_command(command)
|
||||
run_command = run_command_for_command(command)
|
||||
retry_command = retry_command_for_command(command)
|
||||
builds = buildkite.list_builds(pr["head"]["sha"])
|
||||
builds = list_commit_builds(buildkite, command, pr["head"]["sha"])
|
||||
build = select_latest_build(builds, pr["number"])
|
||||
if build:
|
||||
metadata = build.get("meta_data") or {}
|
||||
|
||||
@@ -883,6 +883,42 @@ class RunCiCommandTest(unittest.TestCase):
|
||||
self.assertEqual(buildkite.created_builds, [])
|
||||
self.assertIn("AMD CI is already running", github.comments[0])
|
||||
|
||||
def test_amd_run_ignores_finished_webhook_build_for_author(self) -> None:
|
||||
webhook_build = {
|
||||
"created_at": "2026-08-18T01:00:00Z",
|
||||
"meta_data": {},
|
||||
"number": 122,
|
||||
"pull_request": {"id": 42},
|
||||
"source": "webhook",
|
||||
"state": "passed",
|
||||
"web_url": "https://buildkite.example/amd-ci/builds/122",
|
||||
}
|
||||
ui_build = {**webhook_build, "source": "ui"}
|
||||
command_build = {
|
||||
**webhook_build,
|
||||
"meta_data": {"github-comment-id": "98"},
|
||||
"source": "api",
|
||||
}
|
||||
for build, allowed in (
|
||||
(webhook_build, True),
|
||||
(ui_build, False),
|
||||
(command_build, False),
|
||||
):
|
||||
with self.subTest(source=build["source"]):
|
||||
github = FakeGitHub(
|
||||
permission="read",
|
||||
pr=make_pr(labels=[{"name": "ready"}]),
|
||||
)
|
||||
buildkite = FakeBuildkite([[], [build]])
|
||||
|
||||
run(make_event(COMMAND_RUN_AMD_CI, "author"), github, buildkite)
|
||||
|
||||
self.assertEqual(bool(buildkite.created_builds), allowed)
|
||||
if not allowed:
|
||||
self.assertIn(
|
||||
"AMD CI already ran on this commit", github.comments[0]
|
||||
)
|
||||
|
||||
def test_run_all_sets_buildkite_environment(self) -> None:
|
||||
github = FakeGitHub()
|
||||
buildkite = FakeBuildkite([[], []])
|
||||
@@ -1229,6 +1265,29 @@ class RunCiCommandTest(unittest.TestCase):
|
||||
)
|
||||
self.assertIn("Use `/amd-ci run`", github.comments[0])
|
||||
|
||||
def test_amd_ci_retry_ignores_webhook_build_for_the_current_head(self) -> None:
|
||||
github = FakeGitHub(
|
||||
permission="read",
|
||||
pr=make_pr(labels=[{"name": "ready"}]),
|
||||
)
|
||||
webhook_build = {
|
||||
"created_at": "2026-08-18T01:00:00Z",
|
||||
"meta_data": {},
|
||||
"number": 122,
|
||||
"pull_request": {"id": 42},
|
||||
"source": "webhook",
|
||||
"state": "passed",
|
||||
"web_url": "https://buildkite.example/amd-ci/builds/122",
|
||||
}
|
||||
buildkite = FakeBuildkite([[webhook_build]])
|
||||
|
||||
run(make_event(COMMAND_RETRY_AMD_FAILED, "author"), github, buildkite)
|
||||
|
||||
self.assertEqual(buildkite.retry_calls, [])
|
||||
self.assertIn(
|
||||
"No AMD CI build exists for the current PR head", github.comments[0]
|
||||
)
|
||||
|
||||
def test_ci_retry_creates_filtered_build_for_new_head(self) -> None:
|
||||
github = FakeGitHub(
|
||||
behind=100,
|
||||
|
||||
@@ -277,6 +277,12 @@ repos:
|
||||
files: ^docker/(Dockerfile|versions\.json)$
|
||||
pass_filenames: false
|
||||
additional_dependencies: [dockerfile-parse]
|
||||
- id: check-dockerignore-tracked
|
||||
name: Check .dockerignore does not exclude tracked files
|
||||
entry: python tools/pre_commit/check_dockerignore_tracked.py
|
||||
language: python
|
||||
always_run: true
|
||||
pass_filenames: false
|
||||
- id: check-boolean-context-manager
|
||||
name: Check for boolean ops in with-statements
|
||||
entry: python tools/pre_commit/check_boolean_context_manager.py
|
||||
|
||||
@@ -415,6 +415,7 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
|
||||
"csrc/cuda_view.cu"
|
||||
"csrc/cuda_utils_kernels.cu"
|
||||
"csrc/activation_kernels.cu"
|
||||
"csrc/muse_spark_mul_tanh_soft_cap_kernel.cu"
|
||||
"csrc/ngram_embedding_kernels.cu"
|
||||
"csrc/vocab_parallel_embedding_kernels.cu"
|
||||
"csrc/quantization/activation_kernels.cu"
|
||||
@@ -423,6 +424,7 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
|
||||
"csrc/quantization/w8a8/fp8/per_token_group_quant.cu"
|
||||
"csrc/quantization/gptq/q_gemm.cu"
|
||||
"csrc/pos_encoding_kernels.cu"
|
||||
"csrc/fused_muse_spark_qknorm_rope_kv_insert_kernel.cu"
|
||||
"csrc/fused_qknorm_rope_kernel.cu"
|
||||
"csrc/fused_minimax_m3_qknorm_rope_kv_insert_kernel.cu"
|
||||
"csrc/layernorm_kernels.cu"
|
||||
@@ -441,6 +443,11 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
|
||||
"csrc/fused_deepseek_v4_qnorm_rope_kv_insert_kernel.cu"
|
||||
"csrc/fused_kimi_k3_mla_key_concat_kv_cache_kernel.cu")
|
||||
|
||||
if(VLLM_GPU_LANG STREQUAL "CUDA")
|
||||
list(APPEND VLLM_STABLE_EXT_SRC
|
||||
"csrc/fused_qknorm_rope_cache_kernel.cu")
|
||||
endif()
|
||||
|
||||
if(VLLM_GPU_LANG STREQUAL "CUDA" AND
|
||||
DEFINED CMAKE_CUDA_COMPILER_VERSION AND
|
||||
CMAKE_CUDA_COMPILER_VERSION VERSION_GREATER_EQUAL 12.0)
|
||||
@@ -1112,6 +1119,11 @@ if(VLLM_GPU_LANG STREQUAL "CUDA" OR VLLM_GPU_LANG STREQUAL "HIP")
|
||||
"9.0a;10.0f;12.0f" "${CUDA_ARCHS}")
|
||||
cuda_archs_loose_intersection(FUSED_GDN_DECODE_ARCHS
|
||||
"8.0;8.6;8.9;9.0a;10.0f;11.0f;12.0f" "${CUDA_ARCHS}")
|
||||
else()
|
||||
# The fused GDN decode kernel needs only cp.async and BF16 math. Only the
|
||||
# family-specific targets (10.0f, 11.0f, 12.0f) need CUDA >= 13.0.
|
||||
cuda_archs_loose_intersection(FUSED_GDN_DECODE_ARCHS
|
||||
"8.0;8.6;8.9;9.0a" "${CUDA_ARCHS}")
|
||||
endif()
|
||||
if(FUSED_KDA_DECODE_ARCHS)
|
||||
set(FUSED_KDA_DECODE_SRC
|
||||
|
||||
@@ -0,0 +1,326 @@
|
||||
# SPDX-License-Identifier: Apache-2.0
|
||||
# SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
"""Benchmark the fused Muse-Spark QK-norm + RoPE + KV-insert kernel vs unfused.
|
||||
|
||||
``tuned_us`` forces the cp.async schedule ``fused_qk_norm_rope`` only auto-selects
|
||||
on SM 9.0; ``unfused_us`` is what production runs.
|
||||
"""
|
||||
|
||||
import argparse
|
||||
import itertools
|
||||
|
||||
import torch
|
||||
|
||||
import vllm._custom_ops as ops
|
||||
|
||||
HEAD_DIM = 64
|
||||
DEFAULT_TOKENS = [1, 8, 64, 256, 1024, 4096, 8192, 16384, 32768, 65536]
|
||||
|
||||
|
||||
def make_cos_sin_cache(max_pos, rotary_dim, base, device):
|
||||
inv_freq = 1.0 / (
|
||||
base
|
||||
** (
|
||||
torch.arange(0, rotary_dim, 2, dtype=torch.float32, device=device)
|
||||
/ rotary_dim
|
||||
)
|
||||
)
|
||||
t = torch.arange(max_pos, dtype=torch.float32, device=device)
|
||||
freqs = torch.einsum("i,j->ij", t, inv_freq)
|
||||
return torch.cat((freqs.cos(), freqs.sin()), dim=-1)
|
||||
|
||||
|
||||
class Case:
|
||||
"""Everything a single (num_tokens, config) measurement needs."""
|
||||
|
||||
def __init__(self, num_tokens, nq, nkv, dtype, kv_cache_dtype, device):
|
||||
self.num_tokens = num_tokens
|
||||
self.nq, self.nkv = nq, nkv
|
||||
self.dtype = dtype
|
||||
self.kv_cache_dtype = kv_cache_dtype
|
||||
self.eps = 1e-5
|
||||
self.block_size = 128
|
||||
self.rotary_dim = HEAD_DIM
|
||||
|
||||
self.qsz, self.kvsz = nq * HEAD_DIM, nkv * HEAD_DIM
|
||||
self.row = self.qsz + 2 * self.kvsz + self.qsz # [q | k | v | o_gate]
|
||||
self.qkv = torch.randn(num_tokens, self.row, dtype=dtype, device=device)
|
||||
self.pristine = self.qkv.clone()
|
||||
|
||||
max_pos = 32768
|
||||
self.cos_sin = make_cos_sin_cache(max_pos, self.rotary_dim, 500_000.0, device)
|
||||
self.positions = torch.randint(
|
||||
0, max_pos, (num_tokens,), dtype=torch.int64, device=device
|
||||
)
|
||||
|
||||
num_blocks = (num_tokens + self.block_size - 1) // self.block_size + 1
|
||||
storage = torch.uint8 if kv_cache_dtype != "auto" else dtype
|
||||
self.kv_cache = torch.zeros(
|
||||
num_blocks,
|
||||
nkv,
|
||||
self.block_size,
|
||||
2 * HEAD_DIM,
|
||||
dtype=storage,
|
||||
device=device,
|
||||
)
|
||||
self.k_cache, self.v_cache = self.kv_cache.transpose(1, 2).split(
|
||||
HEAD_DIM, dim=-1
|
||||
)
|
||||
self.slot_mapping = torch.randperm(
|
||||
num_blocks * self.block_size, dtype=torch.int64, device=device
|
||||
)[:num_tokens]
|
||||
self.k_scale = torch.tensor(1.0, dtype=torch.float32, device=device)
|
||||
self.v_scale = torch.tensor(1.0, dtype=torch.float32, device=device)
|
||||
self.ones = torch.ones(HEAD_DIM, dtype=dtype, device=device)
|
||||
self.q_fp8 = torch.empty(
|
||||
num_tokens, self.qsz, dtype=torch.float8_e4m3fn, device=device
|
||||
)
|
||||
self.q_scale = torch.tensor(0.37, dtype=torch.float32, device=device)
|
||||
|
||||
def views(self):
|
||||
q, k, v, _ = self.qkv.split([self.qsz, self.kvsz, self.kvsz, self.qsz], dim=-1)
|
||||
return q, k, v
|
||||
|
||||
def fused(self, use_rope=True, quant_query=False):
|
||||
ops.fused_muse_spark_qknorm_rope_kv_insert(
|
||||
self.qkv,
|
||||
self.positions,
|
||||
self.nq,
|
||||
self.nkv,
|
||||
HEAD_DIM,
|
||||
self.eps,
|
||||
cos_sin_cache=self.cos_sin if use_rope else None,
|
||||
rotary_dim=self.rotary_dim if use_rope else 0,
|
||||
slot_mapping=self.slot_mapping,
|
||||
kv_cache=self.kv_cache,
|
||||
block_size=self.block_size,
|
||||
kv_cache_dtype=self.kv_cache_dtype,
|
||||
k_scale=self.k_scale if self.kv_cache_dtype != "auto" else None,
|
||||
v_scale=self.v_scale if self.kv_cache_dtype != "auto" else None,
|
||||
q_fp8_out=self.q_fp8 if quant_query else None,
|
||||
q_scale=self.q_scale if quant_query else None,
|
||||
)
|
||||
|
||||
def separate_query_quant(self, use_rope=True):
|
||||
self.fused(use_rope)
|
||||
ops.scaled_fp8_quant(self.qkv[:, : self.qsz], scale=self.q_scale)
|
||||
|
||||
def _insert(self):
|
||||
_, k, v = self.views()
|
||||
ops.reshape_and_cache_flash(
|
||||
k.view(-1, self.nkv, HEAD_DIM),
|
||||
v.view(-1, self.nkv, HEAD_DIM),
|
||||
self.k_cache,
|
||||
self.v_cache,
|
||||
self.slot_mapping,
|
||||
self.kv_cache_dtype,
|
||||
self.k_scale,
|
||||
self.v_scale,
|
||||
)
|
||||
|
||||
def unfused_rope(self, token_heads_per_warp: int = -1):
|
||||
ops.fused_qk_norm_rope(
|
||||
self.qkv,
|
||||
self.nq,
|
||||
self.nkv,
|
||||
self.nkv + self.nq, # v heads plus the o-gate
|
||||
HEAD_DIM,
|
||||
self.eps,
|
||||
self.ones,
|
||||
self.ones,
|
||||
self.cos_sin,
|
||||
False,
|
||||
self.positions,
|
||||
token_heads_per_warp,
|
||||
)
|
||||
self._insert()
|
||||
|
||||
def fused_bytes(self):
|
||||
el = self.qkv.element_size()
|
||||
cache_el = self.kv_cache.element_size()
|
||||
read = (self.nq + 2 * self.nkv) * HEAD_DIM * el
|
||||
write = (self.nq + self.nkv) * HEAD_DIM * el
|
||||
cache = 2 * self.nkv * HEAD_DIM * cache_el
|
||||
return self.num_tokens * (read + write + cache)
|
||||
|
||||
|
||||
def _l2_bytes(device) -> int:
|
||||
return torch.cuda.get_device_properties(device).L2_cache_size
|
||||
|
||||
|
||||
def _peak_bandwidth(device) -> float:
|
||||
"""Measured large-copy bandwidth, in GB/s -- the practical ceiling."""
|
||||
n = 1 << 28 # 256 Mi elements = 512 MiB each way at 2 bytes
|
||||
a = torch.empty(n, dtype=torch.bfloat16, device=device)
|
||||
b = torch.empty_like(a)
|
||||
for _ in range(3):
|
||||
b.copy_(a)
|
||||
torch.accelerator.synchronize()
|
||||
start, end = torch.cuda.Event(True), torch.cuda.Event(True)
|
||||
start.record()
|
||||
iters = 20
|
||||
for _ in range(iters):
|
||||
b.copy_(a)
|
||||
end.record()
|
||||
torch.accelerator.synchronize()
|
||||
ms = start.elapsed_time(end) / iters
|
||||
return (2 * a.numel() * a.element_size()) / (ms * 1e-3) / 1e9
|
||||
|
||||
|
||||
def time_eager(fn, warmup=20, iters=100) -> float:
|
||||
for _ in range(warmup):
|
||||
fn()
|
||||
torch.accelerator.synchronize()
|
||||
start, end = torch.cuda.Event(True), torch.cuda.Event(True)
|
||||
start.record()
|
||||
for _ in range(iters):
|
||||
fn()
|
||||
end.record()
|
||||
torch.accelerator.synchronize()
|
||||
return start.elapsed_time(end) / iters * 1e3 # us
|
||||
|
||||
|
||||
def time_graph(fn, warmup=10, iters=50, inner=32) -> float:
|
||||
"""GPU time per call; ``inner`` calls per graph amortize the ~2 us replay."""
|
||||
side = torch.cuda.Stream()
|
||||
side.wait_stream(torch.cuda.current_stream())
|
||||
with torch.cuda.stream(side):
|
||||
for _ in range(warmup):
|
||||
fn()
|
||||
torch.cuda.current_stream().wait_stream(side)
|
||||
graph = torch.cuda.CUDAGraph()
|
||||
with torch.cuda.graph(graph):
|
||||
for _ in range(inner):
|
||||
fn()
|
||||
for _ in range(3):
|
||||
graph.replay()
|
||||
torch.accelerator.synchronize()
|
||||
start, end = torch.cuda.Event(True), torch.cuda.Event(True)
|
||||
start.record()
|
||||
for _ in range(iters):
|
||||
graph.replay()
|
||||
end.record()
|
||||
torch.accelerator.synchronize()
|
||||
return start.elapsed_time(end) / (iters * inner) * 1e3 # us
|
||||
|
||||
|
||||
def main():
|
||||
p = argparse.ArgumentParser()
|
||||
p.add_argument("--nq", type=int, default=32, help="query heads per rank")
|
||||
p.add_argument("--nkv", type=int, default=4, help="kv heads per rank")
|
||||
p.add_argument("--kv-cache-dtype", default="auto", choices=["auto", "fp8"])
|
||||
p.add_argument("--dtype", default="bfloat16", choices=["bfloat16", "float16"])
|
||||
p.add_argument("--tokens", type=int, nargs="*", default=DEFAULT_TOKENS)
|
||||
p.add_argument("--csv", default=None)
|
||||
p.add_argument("--no-graph", action="store_true")
|
||||
p.add_argument(
|
||||
"--query-fp8", action="store_true", help="compare fused Q quantization"
|
||||
)
|
||||
args = p.parse_args()
|
||||
|
||||
device = "cuda"
|
||||
dtype = getattr(torch, args.dtype)
|
||||
if args.query_fp8:
|
||||
if args.kv_cache_dtype != "fp8":
|
||||
p.error("--query-fp8 requires --kv-cache-dtype fp8")
|
||||
timer = time_eager if args.no_graph else time_graph
|
||||
print(f"device: {torch.cuda.get_device_name()}; nq={args.nq}, nkv={args.nkv}")
|
||||
print("layer,tokens,separate_us,fused_us,speedup")
|
||||
rows = []
|
||||
for use_rope, nt in itertools.product([True, False], args.tokens):
|
||||
case = Case(nt, args.nq, args.nkv, dtype, args.kv_cache_dtype, device)
|
||||
separate_us = timer(lambda c=case, r=use_rope: c.separate_query_quant(r))
|
||||
case.qkv.copy_(case.pristine)
|
||||
fused_us = timer(lambda c=case, r=use_rope: c.fused(r, quant_query=True))
|
||||
row = (
|
||||
"rope" if use_rope else "nope",
|
||||
nt,
|
||||
separate_us,
|
||||
fused_us,
|
||||
separate_us / fused_us,
|
||||
)
|
||||
rows.append(row)
|
||||
print(f"{row[0]},{nt},{separate_us:.3f},{fused_us:.3f},{row[4]:.3f}")
|
||||
if args.csv:
|
||||
import csv
|
||||
|
||||
with open(args.csv, "w", newline="") as f:
|
||||
writer = csv.writer(f)
|
||||
writer.writerow(
|
||||
["layer", "tokens", "separate_us", "fused_us", "speedup"]
|
||||
)
|
||||
writer.writerows(rows)
|
||||
return
|
||||
peak = _peak_bandwidth(device)
|
||||
print(
|
||||
f"device: {torch.cuda.get_device_name()} measured copy peak: {peak:.0f} GB/s"
|
||||
)
|
||||
print(
|
||||
f"config: nq={args.nq} nkv={args.nkv} head_dim={HEAD_DIM} "
|
||||
f"dtype={args.dtype} kv_cache={args.kv_cache_dtype}\n"
|
||||
)
|
||||
|
||||
modes = ["rope"]
|
||||
l2 = _l2_bytes(device)
|
||||
print(
|
||||
f"rows marked * have a working set inside the {l2 / 2**20:.0f} MiB L2, "
|
||||
"so their bandwidth is not an HBM number\n"
|
||||
)
|
||||
header = (
|
||||
f"{'layer':>5} {'tokens':>7} {'fused_us':>9} {'unfused_us':>11} "
|
||||
f"{'tuned_us':>9} {'speedup':>8} {'GB/s':>8} {'SOL%':>6}"
|
||||
)
|
||||
rows = []
|
||||
for mode, use_graph in itertools.product(modes, [False, True]):
|
||||
if use_graph and args.no_graph:
|
||||
continue
|
||||
label = "graph" if use_graph else "eager"
|
||||
print(f"── {mode} layer, {label} ──")
|
||||
print(header)
|
||||
timer = time_graph if use_graph else time_eager
|
||||
for nt in args.tokens:
|
||||
case = Case(nt, args.nq, args.nkv, dtype, args.kv_cache_dtype, device)
|
||||
use_rope = mode == "rope"
|
||||
fused_us = timer(lambda c=case, r=use_rope: c.fused(r))
|
||||
baseline = case.unfused_rope
|
||||
unfused_us = timer(baseline)
|
||||
tuned_us = (
|
||||
min(timer(lambda c=case, t=t: c.unfused_rope(t)) for t in (4, 8))
|
||||
if use_rope
|
||||
else unfused_us
|
||||
)
|
||||
nbytes = case.fused_bytes()
|
||||
gbps = nbytes / (fused_us * 1e-6) / 1e9
|
||||
sol = 100.0 * gbps / peak
|
||||
resident = "*" if nbytes < l2 else " "
|
||||
print(
|
||||
f"{mode:>5} {nt:>7} {fused_us:>9.2f} {unfused_us:>11.2f} "
|
||||
f"{tuned_us:>9.2f} {tuned_us / fused_us:>7.2f}x {gbps:>8.0f} "
|
||||
f"{sol:>5.1f}%{resident}"
|
||||
)
|
||||
rows.append((mode, label, nt, fused_us, unfused_us, tuned_us, gbps, sol))
|
||||
print()
|
||||
|
||||
if args.csv:
|
||||
import csv
|
||||
|
||||
with open(args.csv, "w", newline="") as f:
|
||||
w = csv.writer(f)
|
||||
w.writerow(
|
||||
[
|
||||
"layer",
|
||||
"timing",
|
||||
"tokens",
|
||||
"fused_us",
|
||||
"unfused_us",
|
||||
"tuned_us",
|
||||
"gbps",
|
||||
"sol",
|
||||
]
|
||||
)
|
||||
w.writerows(rows)
|
||||
print(f"wrote {args.csv}")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -27,7 +27,6 @@ if(DEEPGEMM_SRC_DIR)
|
||||
set(deepgemm_SOURCE_DIR "${DEEPGEMM_SRC_DIR}")
|
||||
message(STATUS "DeepGEMM using local DEEPGEMM_SRC_DIR: ${deepgemm_SOURCE_DIR}")
|
||||
else()
|
||||
# Keep in sync with tools/install_deepgemm.sh
|
||||
set(_DEEPGEMM_UPSTREAM_REPO "https://github.com/vllm-project/DeepGEMM.git")
|
||||
# Pinned to the fork's dev branch: upstream's 26/09/30 release (2.8.1)
|
||||
# plus the SM120 port, the SM90 paged-MQA kv_block=32/next_n=4 port,
|
||||
|
||||
+4
-1
@@ -425,6 +425,7 @@ function(cuda_archs_loose_intersection OUT_CUDA_ARCHS SRC_CUDA_ARCHS TGT_CUDA_AR
|
||||
if("${_base}" IN_LIST _TGT_CUDA_ARCHS)
|
||||
list(REMOVE_ITEM _SRC_CUDA_ARCHS "${_arch}")
|
||||
list(REMOVE_ITEM _TGT_CUDA_ARCHS "${_base}")
|
||||
list(REMOVE_ITEM _TGT_CUDA_ARCHS "${_base}a" "${_base}f")
|
||||
list(APPEND _CUDA_ARCHS "${_arch}")
|
||||
endif()
|
||||
endif()
|
||||
@@ -464,8 +465,10 @@ function(cuda_archs_loose_intersection OUT_CUDA_ARCHS SRC_CUDA_ARCHS TGT_CUDA_AR
|
||||
string(REGEX REPLACE "[af]$" "" _base "${_arch}")
|
||||
if ("${_base}" IN_LIST _SRC_CUDA_ARCHS)
|
||||
list(REMOVE_ITEM _TGT_CUDA_ARCHS "${_arch}")
|
||||
list(REMOVE_ITEM _SRC_CUDA_ARCHS "${_base}")
|
||||
list(APPEND _CUDA_ARCHS "${_arch}")
|
||||
if (NOT "${_base}" IN_LIST _TGT_CUDA_ARCHS)
|
||||
list(REMOVE_ITEM _SRC_CUDA_ARCHS "${_base}")
|
||||
endif()
|
||||
endif()
|
||||
endif()
|
||||
endforeach()
|
||||
|
||||
@@ -1059,33 +1059,6 @@ void situ_and_mul(torch::stable::Tensor& out, // [..., d]
|
||||
});
|
||||
}
|
||||
|
||||
void masked_situ_and_mul(torch::stable::Tensor& out, // [E, T, d]
|
||||
torch::stable::Tensor& input, // [E, T, 2 * d]
|
||||
const torch::stable::Tensor& expert_num_tokens,
|
||||
double beta, double linear_beta) {
|
||||
int num_experts = input.size(0);
|
||||
int max_num_tokens = input.size(1);
|
||||
int d = input.size(2) / 2;
|
||||
if (num_experts == 0 || max_num_tokens == 0 || d == 0) {
|
||||
return;
|
||||
}
|
||||
constexpr int block_size = 256;
|
||||
dim3 grid((d + block_size - 1) / block_size, num_experts);
|
||||
dim3 block(block_size);
|
||||
const torch::stable::accelerator::DeviceGuard device_guard(
|
||||
input.get_device_index());
|
||||
const cudaStream_t stream = get_current_cuda_stream();
|
||||
VLLM_STABLE_DISPATCH_FLOATING_TYPES(
|
||||
input.scalar_type(), "masked_situ_and_mul_kernel", [&] {
|
||||
vllm::masked_situ_and_mul_kernel<scalar_t, true>
|
||||
<<<grid, block, 0, stream>>>(
|
||||
out.mutable_data_ptr<scalar_t>(),
|
||||
input.const_data_ptr<scalar_t>(),
|
||||
expert_num_tokens.const_data_ptr<int>(), max_num_tokens, d,
|
||||
(float)beta, (float)linear_beta);
|
||||
});
|
||||
}
|
||||
|
||||
#define LAUNCH_MASKED_ACT_AND_MUL(KERNEL, ACT_FIRST, HAS_CLAMP, LIMIT, ALPHA, \
|
||||
BETA) \
|
||||
vllm::masked_act_and_mul_kernel<scalar_t, KERNEL<scalar_t>, ACT_FIRST, \
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
// SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
//
|
||||
// Router GEMM: activation(T) x weight(fp32) -> fp32, M<=32, for the
|
||||
// Router GEMM: activation(T) x weight(fp32) -> fp32, M<=48, for the
|
||||
// supported (E, H) pairs listed at the bottom of this file.
|
||||
// Supports bf16 or fp32 activation; weight is always fp32.
|
||||
|
||||
@@ -214,8 +214,26 @@ void invokeFp32RouterGemm(float* output, InputT const* mat_a,
|
||||
// production-fidelity harness (CUDA-graph replay, per-layer cold weights).
|
||||
// Only enabled on the Blackwell family where it was validated; Hopper and
|
||||
// other shapes / fp32 activation keep the legacy geometry.
|
||||
if constexpr (std::is_same_v<InputT, __nv_bfloat16> && kNumExperts == 128 &&
|
||||
kHiddenDim == 6144) {
|
||||
if constexpr (std::is_same_v<InputT, __nv_bfloat16> && kNumExperts == 256 &&
|
||||
kHiddenDim == 5120) {
|
||||
if (!isBlackwellFamily()) {
|
||||
launchFp32RouterGemm<InputT, 128, 1, kNumTokens, kNumExperts, kHiddenDim>(
|
||||
output, mat_a, mat_b, stream);
|
||||
return;
|
||||
}
|
||||
if constexpr (kNumTokens > 16 && kNumTokens % 8 == 0) {
|
||||
constexpr int block_size = kNumTokens == 24 ? 160 : 128;
|
||||
launchFp32RouterGemm<InputT, block_size, 2, kNumTokens, kNumExperts,
|
||||
kHiddenDim, 4>(output, mat_a, mat_b, stream);
|
||||
} else {
|
||||
constexpr int block_size = kNumTokens == 1 ? 64 : 128;
|
||||
constexpr int token_groups = kNumTokens % 2 == 0 ? 2 : 1;
|
||||
launchFp32RouterGemm<InputT, block_size, 1, kNumTokens, kNumExperts,
|
||||
kHiddenDim, token_groups>(output, mat_a, mat_b,
|
||||
stream);
|
||||
}
|
||||
} else if constexpr (std::is_same_v<InputT, __nv_bfloat16> &&
|
||||
kNumExperts == 128 && kHiddenDim == 6144) {
|
||||
// MiniMax-M3. Legacy 128/1 only fills 128 blocks and pays the same
|
||||
// accumulator register cliffs; B300 sweep:
|
||||
// even M in [6, 10] : BS=384, EPB=1, 2 token groups (1.26-1.43x)
|
||||
@@ -261,6 +279,27 @@ void invokeFp32RouterGemm(float* output, InputT const* mat_a,
|
||||
launchFp32RouterGemm<InputT, 128, 1, kNumTokens, kNumExperts, kHiddenDim>(
|
||||
output, mat_a, mat_b, stream);
|
||||
}
|
||||
} else if constexpr (std::is_same_v<InputT, __nv_bfloat16> &&
|
||||
kNumExperts == 256 && kHiddenDim == 8192) {
|
||||
// MuseSpark1x: the 8MB weight makes this shape FFMA-bound past M~10, so
|
||||
// the wider block wins at every M and token groups clear the M >= 28
|
||||
// accumulator cliff. Thresholds are from a B300 sweep. BS=192/384 are not
|
||||
// instantiable: VPT=8 requires kBlockSize to divide kHiddenDim / 8 = 1024.
|
||||
if (!isBlackwellFamily()) {
|
||||
launchFp32RouterGemm<InputT, 128, 1, kNumTokens, kNumExperts, kHiddenDim>(
|
||||
output, mat_a, mat_b, stream);
|
||||
return;
|
||||
}
|
||||
if constexpr (kNumTokens >= 10 && kNumTokens % 2 == 0) {
|
||||
launchFp32RouterGemm<InputT, 256, 2, kNumTokens, kNumExperts, kHiddenDim,
|
||||
2>(output, mat_a, mat_b, stream);
|
||||
} else if constexpr (kNumTokens >= 13) {
|
||||
launchFp32RouterGemm<InputT, 256, 2, kNumTokens, kNumExperts, kHiddenDim>(
|
||||
output, mat_a, mat_b, stream);
|
||||
} else {
|
||||
launchFp32RouterGemm<InputT, 256, 1, kNumTokens, kNumExperts, kHiddenDim>(
|
||||
output, mat_a, mat_b, stream);
|
||||
}
|
||||
} else {
|
||||
launchFp32RouterGemm<InputT, 128, 1, kNumTokens, kNumExperts, kHiddenDim>(
|
||||
output, mat_a, mat_b, stream);
|
||||
@@ -268,8 +307,9 @@ void invokeFp32RouterGemm(float* output, InputT const* mat_a,
|
||||
}
|
||||
|
||||
// ---------------------------------------------------------------------------
|
||||
// Explicit instantiations: M=1..32, for both input types, for the supported
|
||||
// (E, H) pairs: (256, 3072) [MiniMax-M2/M2.5] and (128, 6144) [MiniMax-M3].
|
||||
// Explicit instantiations for both input types.
|
||||
// Supported (E, H) pairs: (256, 3072), (128, 6144), (256, 5120),
|
||||
// (352, 4096) [Step-5], and (256, 8192) [Muse Spark].
|
||||
// ---------------------------------------------------------------------------
|
||||
|
||||
#define INSTANTIATE(T, M, E, H) \
|
||||
@@ -315,5 +355,36 @@ INSTANTIATE_ALL(__nv_bfloat16, 256, 3072)
|
||||
INSTANTIATE_ALL(float, 128, 6144)
|
||||
INSTANTIATE_ALL(__nv_bfloat16, 128, 6144)
|
||||
|
||||
INSTANTIATE_ALL(float, 256, 5120)
|
||||
INSTANTIATE_ALL(__nv_bfloat16, 256, 5120)
|
||||
|
||||
INSTANTIATE_ALL(float, 352, 4096)
|
||||
INSTANTIATE_ALL(__nv_bfloat16, 352, 4096)
|
||||
INSTANTIATE_ALL(float, 256, 8192)
|
||||
INSTANTIATE_ALL(__nv_bfloat16, 256, 8192)
|
||||
|
||||
#define INSTANTIATE_LARGE(T, E, H) \
|
||||
INSTANTIATE(T, 33, E, H) \
|
||||
INSTANTIATE(T, 34, E, H) \
|
||||
INSTANTIATE(T, 35, E, H) \
|
||||
INSTANTIATE(T, 36, E, H) \
|
||||
INSTANTIATE(T, 37, E, H) \
|
||||
INSTANTIATE(T, 38, E, H) \
|
||||
INSTANTIATE(T, 39, E, H) \
|
||||
INSTANTIATE(T, 40, E, H) \
|
||||
INSTANTIATE(T, 41, E, H) \
|
||||
INSTANTIATE(T, 42, E, H) \
|
||||
INSTANTIATE(T, 43, E, H) \
|
||||
INSTANTIATE(T, 44, E, H) \
|
||||
INSTANTIATE(T, 45, E, H) \
|
||||
INSTANTIATE(T, 46, E, H) \
|
||||
INSTANTIATE(T, 47, E, H) \
|
||||
INSTANTIATE(T, 48, E, H)
|
||||
|
||||
INSTANTIATE_LARGE(float, 256, 5120)
|
||||
INSTANTIATE_LARGE(__nv_bfloat16, 256, 5120)
|
||||
|
||||
#undef INSTANTIATE_LARGE
|
||||
|
||||
#undef INSTANTIATE_ALL
|
||||
#undef INSTANTIATE
|
||||
|
||||
@@ -25,10 +25,13 @@ inline int getSMVersion() {
|
||||
static constexpr int FP32_MAX_TOKENS = 32;
|
||||
|
||||
// Supported (hidden_dim, num_experts) pairs (must match the instantiations in
|
||||
// fp32_router_gemm.cu): (3072, 256) for MiniMax-M2/M2.5, (6144, 128) for M3.
|
||||
// fp32_router_gemm.cu).
|
||||
static inline bool fp32_router_gemm_supported(int hidden_dim, int num_experts) {
|
||||
return (hidden_dim == 3072 && num_experts == 256) ||
|
||||
(hidden_dim == 6144 && num_experts == 128);
|
||||
(hidden_dim == 6144 && num_experts == 128) ||
|
||||
(hidden_dim == 5120 && num_experts == 256) ||
|
||||
(hidden_dim == 4096 && num_experts == 352) ||
|
||||
(hidden_dim == 8192 && num_experts == 256);
|
||||
}
|
||||
|
||||
// Forward declarations — 4 template params must match fp32_router_gemm.cu
|
||||
@@ -60,8 +63,7 @@ struct Fp32LoopUnroller<InputT, kNumExperts, kHiddenDim, kEnd, kEnd> {
|
||||
invokeFp32RouterGemm<InputT, kEnd, kNumExperts, kHiddenDim>(
|
||||
output, mat_a, mat_b, stream);
|
||||
} else {
|
||||
throw std::invalid_argument(
|
||||
"fp32_router_gemm: num_tokens must be in [1, 32]");
|
||||
throw std::invalid_argument("fp32_router_gemm: unsupported num_tokens");
|
||||
}
|
||||
}
|
||||
};
|
||||
@@ -77,6 +79,15 @@ void dispatchFp32RouterGemm(int num_experts, int hidden_dim, int num_tokens,
|
||||
} else if (num_experts == 128 && hidden_dim == 6144) {
|
||||
Fp32LoopUnroller<InputT, 128, 6144, 1, FP32_MAX_TOKENS>::unroll(
|
||||
num_tokens, output, mat_a, mat_b, stream);
|
||||
} else if (num_experts == 256 && hidden_dim == 5120) {
|
||||
Fp32LoopUnroller<InputT, 256, 5120, 1, 48>::unroll(num_tokens, output,
|
||||
mat_a, mat_b, stream);
|
||||
} else if (num_experts == 352 && hidden_dim == 4096) {
|
||||
Fp32LoopUnroller<InputT, 352, 4096, 1, FP32_MAX_TOKENS>::unroll(
|
||||
num_tokens, output, mat_a, mat_b, stream);
|
||||
} else if (num_experts == 256 && hidden_dim == 8192) {
|
||||
Fp32LoopUnroller<InputT, 256, 8192, 1, FP32_MAX_TOKENS>::unroll(
|
||||
num_tokens, output, mat_a, mat_b, stream);
|
||||
} else {
|
||||
throw std::invalid_argument(
|
||||
"fp32_router_gemm: unsupported (hidden_dim, num_experts) pair");
|
||||
@@ -111,9 +122,11 @@ void fp32_router_gemm(
|
||||
STD_TORCH_CHECK(
|
||||
fp32_router_gemm_supported(hidden_dim, num_experts),
|
||||
"fp32_router_gemm: supported (hidden_dim, num_experts) pairs are "
|
||||
"(3072, 256) and (6144, 128)");
|
||||
STD_TORCH_CHECK(num_tokens <= FP32_MAX_TOKENS,
|
||||
"fp32_router_gemm: num_tokens must be in [0, 32]");
|
||||
"(3072, 256), (6144, 128), (5120, 256), (4096, 352), and (8192, 256)");
|
||||
const int max_tokens =
|
||||
hidden_dim == 5120 && num_experts == 256 ? 48 : FP32_MAX_TOKENS;
|
||||
STD_TORCH_CHECK(num_tokens <= max_tokens,
|
||||
"fp32_router_gemm: num_tokens exceeds supported range");
|
||||
STD_TORCH_CHECK(
|
||||
mat_a.scalar_type() == torch::headeronly::ScalarType::Float ||
|
||||
mat_a.scalar_type() == torch::headeronly::ScalarType::BFloat16,
|
||||
|
||||
@@ -0,0 +1,752 @@
|
||||
/*
|
||||
* SPDX-License-Identifier: Apache-2.0
|
||||
* SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
*
|
||||
* Fused Muse Spark attention pre-processing: weightless QK RMSNorm, GPT-J
|
||||
* interleaved RoPE (skipped on NoPE layers, ``cos_sin_cache == None``) and
|
||||
* the K/V insert of ``reshape_and_cache_flash``, over ``[q | k | v | o_gate]``
|
||||
* rows.
|
||||
*
|
||||
* A group of ``head_dim * sizeof(scalar_t) / 16`` lanes owns one head (8 lanes
|
||||
* for bf16 head_dim=64), so each lane moves one 16-byte vector, the rotated
|
||||
* pair (2i, 2i+1) is lane-local, and the norm reduction is 3 shuffles.
|
||||
*
|
||||
* Slot enumeration per token:
|
||||
* [0, nq) Q heads -> norm (+RoPE), in place or to
|
||||
* ``q_fp8_out``
|
||||
* [nq, nq+nkv) K heads -> norm (+RoPE), in place and into
|
||||
* the K cache
|
||||
* [nq+nkv, nq+2*nkv) V heads -> copied into the V cache (no norm)
|
||||
* V slots exist only when a KV cache is bound.
|
||||
*
|
||||
* Norm and rotation run in fp32 and round once at the store. Both fp8 paths
|
||||
* quantize after rounding to the model dtype, the same boundary
|
||||
* ``reshape_and_cache_flash`` and ``scaled_fp8_quant`` use.
|
||||
*/
|
||||
|
||||
#include <algorithm>
|
||||
#include <cmath>
|
||||
#include <cuda_runtime.h>
|
||||
#include <type_traits>
|
||||
|
||||
#include "torch_utils.h"
|
||||
|
||||
#include "core/cuda_compat.h"
|
||||
#include "type_convert.cuh"
|
||||
#include "core/vec_dtypes/dtype_fp8.cuh"
|
||||
#include "dispatch_utils.h"
|
||||
|
||||
#ifndef USE_ROCM
|
||||
#include <cuda_fp8.h>
|
||||
#include "quantization/w8a8/fp8/nvidia/quant_utils.cuh"
|
||||
#else
|
||||
#include <hip/hip_fp8.h>
|
||||
#include "quantization/w8a8/fp8/amd/quant_utils.cuh"
|
||||
#endif
|
||||
|
||||
#ifndef FINAL_MASK
|
||||
#ifdef USE_ROCM
|
||||
#define FINAL_MASK 0xffffffffffffffffULL
|
||||
#else
|
||||
#define FINAL_MASK 0xffffffffu
|
||||
#endif
|
||||
#endif
|
||||
|
||||
namespace vllm {
|
||||
namespace muse_spark_fused_ops {
|
||||
|
||||
namespace {
|
||||
inline int getSMVersion() {
|
||||
auto* props = get_device_prop();
|
||||
return props->major * 10 + props->minor;
|
||||
}
|
||||
inline int getSMCount() {
|
||||
auto* props = get_device_prop();
|
||||
return props->multiProcessorCount;
|
||||
}
|
||||
} // namespace
|
||||
|
||||
// 16 bytes per lane: the widest vector access, and the reason a head is owned
|
||||
// by a group of lanes rather than a whole warp.
|
||||
constexpr int kBytesPerLane = 16;
|
||||
|
||||
#ifdef USE_ROCM
|
||||
using WarpMask = uint64_t;
|
||||
constexpr unsigned kWarpSize = 64;
|
||||
#else
|
||||
using WarpMask = unsigned;
|
||||
constexpr unsigned kWarpSize = 32;
|
||||
#endif
|
||||
|
||||
// Slot -> (token, head) needs a division by a value only known at launch. The
|
||||
// kernel is issue-bound, and a 32-bit hardware division is ~20 instructions of
|
||||
// it, so the reciprocal is computed once on the host instead.
|
||||
struct FastDiv {
|
||||
uint32_t mul;
|
||||
uint32_t shr;
|
||||
int div;
|
||||
};
|
||||
|
||||
inline FastDiv makeFastDiv(int denom) {
|
||||
// The reciprocal form needs denom >= 2; denom == 1 is carried as a flag the
|
||||
// device side selects on. Dividends stay below 2^31, which covers every slot
|
||||
// index this kernel can form.
|
||||
if (denom <= 1) return FastDiv{0u, 0u, 1};
|
||||
int lg = 31 - __builtin_clz(denom);
|
||||
if ((1 << lg) < denom) lg++;
|
||||
uint32_t const p = 31 + static_cast<uint32_t>(lg);
|
||||
uint64_t const m = ((1ull << p) + static_cast<uint64_t>(denom) - 1) /
|
||||
static_cast<uint64_t>(denom);
|
||||
return FastDiv{static_cast<uint32_t>(m), p - 32, denom};
|
||||
}
|
||||
|
||||
__device__ __forceinline__ void fastDivmod(int& quo, int& rem, int src,
|
||||
FastDiv const fd) {
|
||||
int const q =
|
||||
static_cast<int>(__umulhi(static_cast<uint32_t>(src), fd.mul) >> fd.shr);
|
||||
// A select, not a branch: fd.div is uniform across the launch.
|
||||
quo = (fd.div == 1) ? src : q;
|
||||
rem = src - quo * fd.div;
|
||||
}
|
||||
|
||||
// Sum across the LANES contiguous lanes that share one head. The group is
|
||||
// LANES-aligned inside the warp, so an XOR butterfly over masks < LANES never
|
||||
// leaves it. `group_mask` names exactly the group: whole groups are uniformly
|
||||
// active or inactive, so a warp whose other groups bailed out still syncs.
|
||||
template <int LANES>
|
||||
__device__ __forceinline__ float groupReduceSum(float val,
|
||||
WarpMask const group_mask) {
|
||||
#pragma unroll
|
||||
for (int mask = LANES >> 1; mask > 0; mask >>= 1) {
|
||||
val += __shfl_xor_sync(group_mask, val, mask, 32);
|
||||
}
|
||||
return val;
|
||||
}
|
||||
|
||||
// 16 contiguous bytes -> kElems fp32 registers, keeping the raw vector so a
|
||||
// pass-through slot (V) can be re-stored without a conversion round trip.
|
||||
template <typename scalar_t, int kElems>
|
||||
__device__ __forceinline__ uint4 loadElems(scalar_t const* __restrict__ src,
|
||||
float (&elems)[kElems]) {
|
||||
using Converter = vllm::_typeConvert<scalar_t>;
|
||||
using T2 = typename Converter::packed_hip_type;
|
||||
uint4 raw = *reinterpret_cast<uint4 const*>(src);
|
||||
auto const* p = reinterpret_cast<T2 const*>(&raw);
|
||||
#pragma unroll
|
||||
for (int i = 0; i < kElems / 2; i++) {
|
||||
float2 f2 = Converter::convert(p[i]);
|
||||
elems[2 * i] = f2.x;
|
||||
elems[2 * i + 1] = f2.y;
|
||||
}
|
||||
return raw;
|
||||
}
|
||||
|
||||
// The lane's slice of the rope table: kPairs cos and kPairs sin values, each
|
||||
// run contiguous, so each is one vector load (16 bytes for the fp32 table, 8
|
||||
// for a model-dtype one) instead of kPairs scalar ones. rotary_dim being a
|
||||
// multiple of 16 is what keeps both runs naturally aligned.
|
||||
template <typename rope_t, int kPairs>
|
||||
__device__ __forceinline__ void loadRopeVec(rope_t const* __restrict__ cos_ptr,
|
||||
rope_t const* __restrict__ sin_ptr,
|
||||
float (&c)[kPairs],
|
||||
float (&s)[kPairs]) {
|
||||
static_assert(kPairs == 4, "rope vector load assumes 4 pairs per lane");
|
||||
if constexpr (std::is_same_v<rope_t, float>) {
|
||||
float4 const cv = VLLM_LDG(reinterpret_cast<float4 const*>(cos_ptr));
|
||||
float4 const sv = VLLM_LDG(reinterpret_cast<float4 const*>(sin_ptr));
|
||||
c[0] = cv.x, c[1] = cv.y, c[2] = cv.z, c[3] = cv.w;
|
||||
s[0] = sv.x, s[1] = sv.y, s[2] = sv.z, s[3] = sv.w;
|
||||
} else {
|
||||
using Converter = vllm::_typeConvert<rope_t>;
|
||||
using T2 = typename Converter::packed_hip_type;
|
||||
uint2 const cv = VLLM_LDG(reinterpret_cast<uint2 const*>(cos_ptr));
|
||||
uint2 const sv = VLLM_LDG(reinterpret_cast<uint2 const*>(sin_ptr));
|
||||
auto const* cp = reinterpret_cast<T2 const*>(&cv);
|
||||
auto const* sp = reinterpret_cast<T2 const*>(&sv);
|
||||
#pragma unroll
|
||||
for (int i = 0; i < kPairs / 2; i++) {
|
||||
float2 const cf = Converter::convert(cp[i]);
|
||||
float2 const sf = Converter::convert(sp[i]);
|
||||
c[2 * i] = cf.x, c[2 * i + 1] = cf.y;
|
||||
s[2 * i] = sf.x, s[2 * i + 1] = sf.y;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// kElems fp32 registers -> the 16-byte scalar_t vector they round to. Returned
|
||||
// rather than stored so the fp8 cache path can reuse the *rounded* values,
|
||||
// which is what reshape_and_cache_flash quantizes from.
|
||||
template <typename scalar_t, int kElems>
|
||||
__device__ __forceinline__ uint4 packElems(float const (&elems)[kElems]) {
|
||||
using Converter = vllm::_typeConvert<scalar_t>;
|
||||
using T2 = typename Converter::packed_hip_type;
|
||||
uint4 raw;
|
||||
auto* p = reinterpret_cast<T2*>(&raw);
|
||||
#pragma unroll
|
||||
for (int i = 0; i < kElems / 2; i++) {
|
||||
p[i] = Converter::convert(make_float2(elems[2 * i], elems[2 * i + 1]));
|
||||
}
|
||||
return raw;
|
||||
}
|
||||
|
||||
// Scatter one lane's 16 bytes into the KV cache. kAuto stores the model dtype
|
||||
// verbatim; the fp8 dtypes quantize the value *after* it has been rounded to
|
||||
// scalar_t, which is the boundary reshape_and_cache_flash uses.
|
||||
//
|
||||
// `inv_scale` is taken once per thread: per-element fp32 divisions would cost
|
||||
// more than the rest of the kernel. The extra rounding is 2^-24 relative.
|
||||
template <typename scalar_t, typename cache_t, Fp8KVCacheDataType kv_dt,
|
||||
int kElems>
|
||||
__device__ __forceinline__ void storeCache(cache_t* __restrict__ dst,
|
||||
uint4 const& packed,
|
||||
float const inv_scale) {
|
||||
if constexpr (kv_dt == Fp8KVCacheDataType::kAuto) {
|
||||
*reinterpret_cast<uint4*>(dst) = packed;
|
||||
} else {
|
||||
using Converter = vllm::_typeConvert<scalar_t>;
|
||||
using HipT = typename Converter::hip_type;
|
||||
auto const* rounded = reinterpret_cast<HipT const*>(&packed);
|
||||
#ifndef USE_ROCM
|
||||
constexpr __nv_fp8_interpretation_t kInterp =
|
||||
(kv_dt == Fp8KVCacheDataType::kFp8E4M3) ? __NV_E4M3 : __NV_E5M2;
|
||||
__nv_fp8x2_storage_t out[kElems / 2];
|
||||
#pragma unroll
|
||||
for (int i = 0; i < kElems / 2; i++) {
|
||||
float2 v =
|
||||
make_float2(Converter::convert(rounded[2 * i]) * inv_scale,
|
||||
Converter::convert(rounded[2 * i + 1]) * inv_scale);
|
||||
out[i] = __nv_cvt_float2_to_fp8x2(v, __NV_SATFINITE, kInterp);
|
||||
}
|
||||
// kElems fp8 bytes: 8 for the 2-byte dtypes this kernel serves.
|
||||
static_assert(kElems == 8, "fp8 store assumes 8 elements per lane");
|
||||
*reinterpret_cast<uint2*>(dst) = *reinterpret_cast<uint2 const*>(out);
|
||||
#else
|
||||
#pragma unroll
|
||||
for (int i = 0; i < kElems; i++) {
|
||||
float const v = Converter::convert(rounded[i]) * inv_scale;
|
||||
dst[i] = fp8::scaled_convert<cache_t, float, kv_dt>(v, 1.0f);
|
||||
}
|
||||
#endif
|
||||
}
|
||||
}
|
||||
|
||||
// Grid is 1D over (token, head-chunk) pairs. A chunk is HEADS_PER_GROUP
|
||||
// consecutive head slots of one token, served by kLanes contiguous lanes; the
|
||||
// whole chunk shares the token's position and its slice of the rope table, so
|
||||
// the slot division, the position load and the cos/sin load are paid once per
|
||||
// chunk. `slots_per_token % HEADS_PER_GROUP == 0` keeps a chunk inside one
|
||||
// token. `rope_t` is the cos/sin cache dtype, independent of the qkv dtype.
|
||||
template <typename scalar_t, typename cache_t, Fp8KVCacheDataType kv_dt,
|
||||
typename rope_t, int HEAD_DIM, int HEADS_PER_GROUP, bool kDoRope,
|
||||
bool kInsertKV>
|
||||
__global__
|
||||
__launch_bounds__(256, 6) void fusedMuseSparkQKNormRopeKVInsertKernel(
|
||||
scalar_t* __restrict__ qkv, // [N, qkv_stride] in/out
|
||||
rope_t const* __restrict__ cos_sin_cache, // [max_pos, rotary_dim]
|
||||
int64_t const* __restrict__ positions, // [N] i64
|
||||
int64_t const* __restrict__ slot_mapping, // [N] i64 or nullptr
|
||||
cache_t* __restrict__ kv_cache, // [nb, nkv, bs, 2*HEAD_DIM]
|
||||
float const* __restrict__ k_scale, // [1] or nullptr
|
||||
float const* __restrict__ v_scale, // [1] or nullptr
|
||||
uint8_t* __restrict__ q_fp8_out, // [N, nq * HEAD_DIM] or nullptr
|
||||
float const* __restrict__ q_scale, // [1] or nullptr
|
||||
float const eps, int const rotary_dim, int const num_tokens, int const nq,
|
||||
int const nkv, FastDiv const chunks_div, int64_t const qkv_stride,
|
||||
FastDiv const block_div, int64_t const kv_s_block, int64_t const kv_s_head,
|
||||
int64_t const kv_s_token) {
|
||||
#if (!defined(__CUDA_ARCH__) || __CUDA_ARCH__ < 800) && !defined(USE_ROCM)
|
||||
// _typeConvert<BFloat16> is unavailable on pre-Ampere; this kernel only ever
|
||||
// runs with bf16/fp16 inputs. Discard the bf16 body there.
|
||||
if constexpr (std::is_same_v<scalar_t, c10::BFloat16>) {
|
||||
return;
|
||||
} else {
|
||||
#endif
|
||||
constexpr int kElems = kBytesPerLane / sizeof(scalar_t);
|
||||
constexpr int kLanes = HEAD_DIM / kElems;
|
||||
constexpr int kPairs = kElems / 2;
|
||||
constexpr int kH = HEADS_PER_GROUP;
|
||||
static_assert(kLanes > 0 && kLanes <= 32 && (32 % kLanes) == 0,
|
||||
"head_dim must give a power-of-two lane group <= 32");
|
||||
|
||||
int const tid = blockIdx.x * blockDim.x + threadIdx.x;
|
||||
int const groupIdx = tid / kLanes;
|
||||
int const dimBase = (tid % kLanes) * kElems;
|
||||
|
||||
int tokenIdx, chunk;
|
||||
fastDivmod(tokenIdx, chunk, groupIdx, chunks_div);
|
||||
int const firstSlot = chunk * kH;
|
||||
|
||||
// Out-of-range groups must not exit before the reductions below: the XOR
|
||||
// butterfly needs every lane the mask names, and the mask covers the whole
|
||||
// group. They ride along and skip every memory access instead.
|
||||
bool const active = tokenIdx < num_tokens;
|
||||
|
||||
constexpr WarpMask kGroupBits =
|
||||
(kLanes == 32) ? static_cast<WarpMask>(0xffffffffu)
|
||||
: ((WarpMask{1} << kLanes) - WarpMask{1});
|
||||
unsigned const group_shift =
|
||||
(threadIdx.x & (kWarpSize - 1)) & ~static_cast<unsigned>(kLanes - 1);
|
||||
WarpMask const group_mask = kGroupBits << group_shift;
|
||||
|
||||
// PDL: wait for the qkv-projection GEMM that produces `qkv`. No-op when
|
||||
// PDL is not enabled on the launch.
|
||||
#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900)
|
||||
cudaGridDependencySynchronize();
|
||||
#endif
|
||||
|
||||
// Q, K and V are contiguous head slots at the head of the row, so the slot
|
||||
// index doubles as the head offset. The trailing o-gate is skipped by the
|
||||
// row stride alone.
|
||||
scalar_t* const base = qkv + static_cast<int64_t>(tokenIdx) * qkv_stride +
|
||||
static_cast<int64_t>(firstSlot) * HEAD_DIM + dimBase;
|
||||
|
||||
// Issue every load in the chunk before touching any of it, so the group has
|
||||
// kH loads in flight instead of one. An inactive group leaves these unset;
|
||||
// nothing downstream reads them, since every store is gated on `active`.
|
||||
uint4 raw[kH];
|
||||
if (active) {
|
||||
#pragma unroll
|
||||
for (int h = 0; h < kH; h++) {
|
||||
raw[h] = *reinterpret_cast<uint4 const*>(base + h * HEAD_DIM);
|
||||
}
|
||||
}
|
||||
|
||||
float cosv[kPairs], sinv[kPairs];
|
||||
bool const laneRotates = kDoRope && dimBase < rotary_dim;
|
||||
if constexpr (kDoRope) {
|
||||
if (active && laneRotates) {
|
||||
int64_t const pos = positions[tokenIdx];
|
||||
rope_t const* cache_ptr = cos_sin_cache + pos * rotary_dim;
|
||||
loadRopeVec<rope_t, kPairs>(cache_ptr + dimBase / 2,
|
||||
cache_ptr + rotary_dim / 2 + dimBase / 2,
|
||||
cosv, sinv);
|
||||
}
|
||||
}
|
||||
|
||||
int64_t sm = -1;
|
||||
if constexpr (kInsertKV) {
|
||||
if (active) sm = slot_mapping[tokenIdx];
|
||||
}
|
||||
int cache_block = 0, cache_tok = 0;
|
||||
float k_inv = 1.0f, v_inv = 1.0f;
|
||||
float const q_inv = q_fp8_out ? 1.0f / *q_scale : 1.0f;
|
||||
if constexpr (kInsertKV) {
|
||||
if (sm >= 0) {
|
||||
fastDivmod(cache_block, cache_tok, static_cast<int>(sm), block_div);
|
||||
}
|
||||
if constexpr (kv_dt != Fp8KVCacheDataType::kAuto) {
|
||||
k_inv = 1.0f / *k_scale;
|
||||
v_inv = 1.0f / *v_scale;
|
||||
}
|
||||
}
|
||||
|
||||
#pragma unroll
|
||||
for (int h = 0; h < kH; h++) {
|
||||
int const slot = firstSlot + h;
|
||||
bool const isV = kInsertKV && slot >= nq + nkv;
|
||||
bool const isK = slot >= nq && slot < nq + nkv;
|
||||
bool const isQK = active && !isV;
|
||||
|
||||
float elems[kElems];
|
||||
uint4 packed = raw[h];
|
||||
if (isQK) {
|
||||
using Converter = vllm::_typeConvert<scalar_t>;
|
||||
using T2 = typename Converter::packed_hip_type;
|
||||
auto const* p = reinterpret_cast<T2 const*>(&raw[h]);
|
||||
#pragma unroll
|
||||
for (int i = 0; i < kElems / 2; i++) {
|
||||
float2 const f2 = Converter::convert(p[i]);
|
||||
elems[2 * i] = f2.x;
|
||||
elems[2 * i + 1] = f2.y;
|
||||
}
|
||||
|
||||
float sumsq = 0.0f;
|
||||
#pragma unroll
|
||||
for (int i = 0; i < kElems; i++) sumsq += elems[i] * elems[i];
|
||||
sumsq = groupReduceSum<kLanes>(sumsq, group_mask);
|
||||
float const rms_rcp =
|
||||
rsqrtf(sumsq / static_cast<float>(HEAD_DIM) + eps);
|
||||
#pragma unroll
|
||||
for (int i = 0; i < kElems; i++) elems[i] *= rms_rcp;
|
||||
|
||||
if constexpr (kDoRope) {
|
||||
if (laneRotates) {
|
||||
#pragma unroll
|
||||
for (int i = 0; i < kPairs; i++) {
|
||||
float const v0 = elems[2 * i];
|
||||
float const v1 = elems[2 * i + 1];
|
||||
elems[2 * i] = v0 * cosv[i] - v1 * sinv[i];
|
||||
elems[2 * i + 1] = v0 * sinv[i] + v1 * cosv[i];
|
||||
}
|
||||
}
|
||||
}
|
||||
// Rounding to scalar_t happens exactly once, here, and both the
|
||||
// Q output and the fp8 cache quantization consume the same
|
||||
// rounded value. V passes through with the vector it was loaded with.
|
||||
packed = packElems<scalar_t, kElems>(elems);
|
||||
if (q_fp8_out && slot < nq) {
|
||||
int64_t const q_off =
|
||||
(static_cast<int64_t>(tokenIdx) * nq + slot) * HEAD_DIM + dimBase;
|
||||
storeCache<scalar_t, uint8_t, Fp8KVCacheDataType::kFp8E4M3, kElems>(
|
||||
q_fp8_out + q_off, packed, q_inv);
|
||||
} else {
|
||||
*reinterpret_cast<uint4*>(base + h * HEAD_DIM) = packed;
|
||||
}
|
||||
}
|
||||
|
||||
if constexpr (kInsertKV) {
|
||||
if (sm >= 0 && (isK || isV)) { // sm < 0 marks a padded token
|
||||
int const head = isK ? (slot - nq) : (slot - nq - nkv);
|
||||
int64_t const off = static_cast<int64_t>(cache_block) * kv_s_block +
|
||||
head * kv_s_head +
|
||||
static_cast<int64_t>(cache_tok) * kv_s_token +
|
||||
(isK ? 0 : HEAD_DIM) + dimBase;
|
||||
storeCache<scalar_t, cache_t, kv_dt, kElems>(kv_cache + off, packed,
|
||||
isK ? k_inv : v_inv);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// PDL: let a dependent successor launch early.
|
||||
#if defined(__CUDA_ARCH__) && (__CUDA_ARCH__ >= 900)
|
||||
cudaTriggerProgrammaticLaunchCompletion();
|
||||
#endif
|
||||
#if (!defined(__CUDA_ARCH__) || __CUDA_ARCH__ < 800) && !defined(USE_ROCM)
|
||||
}
|
||||
#endif
|
||||
}
|
||||
|
||||
template <typename scalar_t, typename cache_t, Fp8KVCacheDataType kv_dt,
|
||||
typename rope_t, int HEAD_DIM, int HEADS_PER_GROUP>
|
||||
void launchFusedMuseSparkTyped(
|
||||
scalar_t* qkv, rope_t const* cos_sin_cache, int64_t const* positions,
|
||||
int64_t const* slot_mapping, cache_t* kv_cache, float const* k_scale,
|
||||
float const* v_scale, uint8_t* q_fp8_out, float const* q_scale,
|
||||
float const eps, int const rotary_dim, int const num_tokens, int const nq,
|
||||
int const nkv, int64_t const qkv_stride, int const block_size,
|
||||
int64_t const kv_s_block, int64_t const kv_s_head, int64_t const kv_s_token,
|
||||
bool const do_rope, bool const insert_kv, cudaStream_t stream) {
|
||||
FastDiv const block_div = makeFastDiv(block_size);
|
||||
constexpr int kElems = kBytesPerLane / sizeof(scalar_t);
|
||||
constexpr int kLanes = HEAD_DIM / kElems;
|
||||
|
||||
int const slots_per_token = nq + nkv + (insert_kv ? nkv : 0);
|
||||
int const chunks_per_token = slots_per_token / HEADS_PER_GROUP;
|
||||
FastDiv const chunks_div = makeFastDiv(chunks_per_token);
|
||||
int64_t const total_lanes =
|
||||
static_cast<int64_t>(num_tokens) * chunks_per_token * kLanes;
|
||||
if (total_lanes == 0) return;
|
||||
|
||||
constexpr int kBlockSize = 256;
|
||||
int const grid =
|
||||
static_cast<int>((total_lanes + kBlockSize - 1) / kBlockSize);
|
||||
|
||||
#ifndef USE_ROCM
|
||||
// PDL wherever the hardware supports it (SM90+); a plain launch elsewhere.
|
||||
static int const sm_version = getSMVersion();
|
||||
cudaLaunchConfig_t config;
|
||||
config.gridDim = dim3(grid);
|
||||
config.blockDim = dim3(kBlockSize);
|
||||
config.dynamicSmemBytes = 0;
|
||||
config.stream = stream;
|
||||
cudaLaunchAttribute attrs[1];
|
||||
attrs[0].id = cudaLaunchAttributeProgrammaticStreamSerialization;
|
||||
attrs[0].val.programmaticStreamSerializationAllowed = 1;
|
||||
config.attrs = attrs;
|
||||
config.numAttrs = (sm_version >= 90) ? 1 : 0;
|
||||
|
||||
#define LAUNCH_MUSE(DO_ROPE, INSERT) \
|
||||
cudaLaunchKernelEx(&config, \
|
||||
fusedMuseSparkQKNormRopeKVInsertKernel< \
|
||||
scalar_t, cache_t, kv_dt, rope_t, HEAD_DIM, \
|
||||
HEADS_PER_GROUP, DO_ROPE, INSERT>, \
|
||||
qkv, cos_sin_cache, positions, slot_mapping, kv_cache, \
|
||||
k_scale, v_scale, q_fp8_out, q_scale, eps, rotary_dim, \
|
||||
num_tokens, nq, nkv, chunks_div, qkv_stride, block_div, \
|
||||
kv_s_block, kv_s_head, kv_s_token)
|
||||
#else
|
||||
// clang-format off
|
||||
#define LAUNCH_MUSE(DO_ROPE, INSERT) \
|
||||
fusedMuseSparkQKNormRopeKVInsertKernel< \
|
||||
scalar_t, cache_t, kv_dt, rope_t, HEAD_DIM, HEADS_PER_GROUP, DO_ROPE, \
|
||||
INSERT><<<grid, kBlockSize, 0, stream>>>( \
|
||||
qkv, cos_sin_cache, positions, slot_mapping, kv_cache, k_scale, \
|
||||
v_scale, q_fp8_out, q_scale, eps, rotary_dim, num_tokens, nq, nkv, \
|
||||
chunks_div, qkv_stride, block_div, kv_s_block, kv_s_head, kv_s_token)
|
||||
// clang-format on
|
||||
#endif
|
||||
|
||||
if (do_rope) {
|
||||
if (insert_kv) {
|
||||
LAUNCH_MUSE(true, true);
|
||||
} else {
|
||||
LAUNCH_MUSE(true, false);
|
||||
}
|
||||
} else {
|
||||
if (insert_kv) {
|
||||
LAUNCH_MUSE(false, true);
|
||||
} else {
|
||||
LAUNCH_MUSE(false, false);
|
||||
}
|
||||
}
|
||||
#undef LAUNCH_MUSE
|
||||
}
|
||||
|
||||
template <typename scalar_t, typename cache_t, Fp8KVCacheDataType kv_dt>
|
||||
void launchFusedMuseSpark(
|
||||
scalar_t* qkv, void const* cos_sin_cache, bool const rope_is_fp32,
|
||||
int64_t const* positions, int64_t const* slot_mapping, cache_t* kv_cache,
|
||||
float const* k_scale, float const* v_scale, uint8_t* q_fp8_out,
|
||||
float const* q_scale, float const eps, int const rotary_dim,
|
||||
int const num_tokens, int const nq, int const nkv, int const head_dim,
|
||||
int64_t const qkv_stride, int const block_size, int64_t const kv_s_block,
|
||||
int64_t const kv_s_head, int64_t const kv_s_token, bool const do_rope,
|
||||
bool const insert_kv, cudaStream_t stream) {
|
||||
// The group width must divide the slot count. Widen only while the grid
|
||||
// still covers a full wave; decode-sized batches keep the narrow schedule.
|
||||
int const slots_per_token = nq + nkv + (insert_kv ? nkv : 0);
|
||||
constexpr int kElemsSel = kBytesPerLane / sizeof(scalar_t);
|
||||
constexpr int kBlockSel = 256;
|
||||
int const lanes_per_head = head_dim / kElemsSel;
|
||||
int64_t const full_wave =
|
||||
static_cast<int64_t>(getSMCount()) * (2048 / kBlockSel);
|
||||
int heads_per_group = 1;
|
||||
for (int h : {4, 2}) {
|
||||
if (slots_per_token % h != 0) continue;
|
||||
int64_t const blocks = (static_cast<int64_t>(num_tokens) *
|
||||
(slots_per_token / h) * lanes_per_head +
|
||||
kBlockSel - 1) /
|
||||
kBlockSel;
|
||||
if (blocks >= full_wave) {
|
||||
heads_per_group = h;
|
||||
break;
|
||||
}
|
||||
}
|
||||
|
||||
// TP4/TP8 amortize head addressing before filling a full CTA wave. RoPE
|
||||
// benefits earlier; FP8 conversion keeps wider groups expensive for longer.
|
||||
if (insert_kv && head_dim == 64 &&
|
||||
((nq == 16 && nkv == 2) || (nq == 32 && nkv == 4))) {
|
||||
int64_t const query_heads = static_cast<int64_t>(num_tokens) * nq;
|
||||
if (do_rope && query_heads >= 24576) {
|
||||
heads_per_group = std::max(heads_per_group, 2);
|
||||
if constexpr (kv_dt == Fp8KVCacheDataType::kAuto) {
|
||||
if (query_heads >= 49152) heads_per_group = 4;
|
||||
}
|
||||
} else if (!do_rope && query_heads >= 32768) {
|
||||
if constexpr (kv_dt == Fp8KVCacheDataType::kAuto) {
|
||||
heads_per_group = std::max(heads_per_group, 2);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#define DISPATCH_ARGS(RT, HD, HPG) \
|
||||
launchFusedMuseSparkTyped<scalar_t, cache_t, kv_dt, RT, HD, HPG>( \
|
||||
qkv, reinterpret_cast<RT const*>(cos_sin_cache), positions, \
|
||||
slot_mapping, kv_cache, k_scale, v_scale, q_fp8_out, q_scale, eps, \
|
||||
rotary_dim, num_tokens, nq, nkv, qkv_stride, block_size, kv_s_block, \
|
||||
kv_s_head, kv_s_token, do_rope, insert_kv, stream)
|
||||
|
||||
#define DISPATCH_HPG(RT, HD) \
|
||||
switch (heads_per_group) { \
|
||||
case 4: \
|
||||
DISPATCH_ARGS(RT, HD, 4); \
|
||||
break; \
|
||||
case 2: \
|
||||
DISPATCH_ARGS(RT, HD, 2); \
|
||||
break; \
|
||||
default: \
|
||||
DISPATCH_ARGS(RT, HD, 1); \
|
||||
break; \
|
||||
}
|
||||
|
||||
// A NoPE launch never reads the table, so it can share the fp32 branch.
|
||||
#define DISPATCH_ROPE_T(HD) \
|
||||
if (!do_rope || rope_is_fp32) { \
|
||||
DISPATCH_HPG(float, HD); \
|
||||
} else { \
|
||||
DISPATCH_HPG(scalar_t, HD); \
|
||||
}
|
||||
|
||||
switch (head_dim) {
|
||||
case 64:
|
||||
DISPATCH_ROPE_T(64);
|
||||
break;
|
||||
case 128:
|
||||
DISPATCH_ROPE_T(128);
|
||||
break;
|
||||
case 256:
|
||||
DISPATCH_ROPE_T(256);
|
||||
break;
|
||||
default:
|
||||
STD_TORCH_CHECK(false, "unsupported head_dim: ", head_dim);
|
||||
}
|
||||
#undef DISPATCH_ROPE_T
|
||||
#undef DISPATCH_HPG
|
||||
#undef DISPATCH_ARGS
|
||||
}
|
||||
|
||||
} // namespace muse_spark_fused_ops
|
||||
} // namespace vllm
|
||||
|
||||
// clang-format off
|
||||
#define CALL_FUSED_MUSE_SPARK(_RAW_T, CACHE_T, KV_DTYPE) \
|
||||
vllm::muse_spark_fused_ops::launchFusedMuseSpark<st, CACHE_T, KV_DTYPE>( \
|
||||
reinterpret_cast<st*>(qkv.data_ptr()), \
|
||||
do_rope ? cos_sin_cache->data_ptr() : nullptr, rope_is_fp32, \
|
||||
reinterpret_cast<int64_t const*>(positions.data_ptr()), \
|
||||
insert_kv ? reinterpret_cast<int64_t const*>(slot_mapping->data_ptr()) \
|
||||
: nullptr, \
|
||||
insert_kv ? reinterpret_cast<CACHE_T*>(kv_cache->data_ptr()) : nullptr, \
|
||||
needs_scales ? reinterpret_cast<float const*>(k_scale->data_ptr()) \
|
||||
: nullptr, \
|
||||
needs_scales ? reinterpret_cast<float const*>(v_scale->data_ptr()) \
|
||||
: nullptr, \
|
||||
q_fp8_out ? reinterpret_cast<uint8_t*>(q_fp8_out->data_ptr()) : nullptr, \
|
||||
q_scale ? reinterpret_cast<float const*>(q_scale->data_ptr()) : nullptr, \
|
||||
static_cast<float>(eps), static_cast<int>(rotary_dim), num_tokens, nq, \
|
||||
nkv, hd, qkv_stride, static_cast<int>(block_size), kv_s_block, \
|
||||
kv_s_head, kv_s_token, do_rope, insert_kv, stream)
|
||||
// clang-format on
|
||||
|
||||
void fused_muse_spark_qknorm_rope_kv_insert(
|
||||
torch::stable::Tensor& qkv, // [N, (nq + 2*nkv + gate)*hd]
|
||||
torch::stable::Tensor const& positions, // [N] i64
|
||||
int64_t num_heads, int64_t num_kv_heads, int64_t head_dim, double eps,
|
||||
std::optional<torch::stable::Tensor> cos_sin_cache, // None => NoPE layer
|
||||
int64_t rotary_dim,
|
||||
std::optional<torch::stable::Tensor> slot_mapping, // [N] i64
|
||||
std::optional<torch::stable::Tensor> kv_cache, // [nb, nkv, bs, 2*hd]
|
||||
int64_t block_size, const std::string& kv_cache_dtype,
|
||||
std::optional<torch::stable::Tensor> k_scale, // [1] fp32, fp8 cache only
|
||||
std::optional<torch::stable::Tensor> v_scale,
|
||||
std::optional<torch::stable::Tensor> q_fp8_out,
|
||||
std::optional<torch::stable::Tensor> q_scale) {
|
||||
STD_TORCH_CHECK(qkv.is_cuda() && qkv.dim() == 2 && qkv.stride(1) == 1,
|
||||
"qkv must be a 2-D CUDA tensor with a contiguous row");
|
||||
STD_TORCH_CHECK(
|
||||
qkv.scalar_type() == torch::headeronly::ScalarType::Half ||
|
||||
qkv.scalar_type() == torch::headeronly::ScalarType::BFloat16,
|
||||
"qkv must be float16 or bfloat16");
|
||||
STD_TORCH_CHECK(
|
||||
positions.is_cuda() &&
|
||||
positions.scalar_type() == torch::headeronly::ScalarType::Long,
|
||||
"positions must be int64 CUDA");
|
||||
|
||||
int const num_tokens = static_cast<int>(qkv.size(0));
|
||||
int const nq = static_cast<int>(num_heads);
|
||||
int const nkv = static_cast<int>(num_kv_heads);
|
||||
int const hd = static_cast<int>(head_dim);
|
||||
int64_t const qkv_stride = qkv.stride(0);
|
||||
|
||||
STD_TORCH_CHECK(q_fp8_out.has_value() == q_scale.has_value(),
|
||||
"q_fp8_out and q_scale must be provided together");
|
||||
if (q_fp8_out) {
|
||||
STD_TORCH_CHECK(
|
||||
q_fp8_out->is_cuda() &&
|
||||
q_fp8_out->get_device_index() == qkv.get_device_index() &&
|
||||
q_fp8_out->scalar_type() ==
|
||||
torch::headeronly::ScalarType::Float8_e4m3fn &&
|
||||
q_fp8_out->dim() == 2 && q_fp8_out->is_contiguous() &&
|
||||
q_fp8_out->size(0) == num_tokens &&
|
||||
q_fp8_out->size(1) == static_cast<int64_t>(nq) * hd,
|
||||
"q_fp8_out must be contiguous E4M3 [num_tokens, num_heads * head_dim] "
|
||||
"on the qkv device");
|
||||
STD_TORCH_CHECK(
|
||||
q_scale->is_cuda() &&
|
||||
q_scale->get_device_index() == qkv.get_device_index() &&
|
||||
q_scale->scalar_type() == torch::headeronly::ScalarType::Float &&
|
||||
q_scale->numel() == 1,
|
||||
"q_scale must be a single fp32 value on the qkv device");
|
||||
}
|
||||
|
||||
STD_TORCH_CHECK(hd == 64 || hd == 128 || hd == 256,
|
||||
"head_dim must be 64, 128 or 256");
|
||||
STD_TORCH_CHECK(qkv.size(1) >= static_cast<int64_t>(nq + 2 * nkv) * hd,
|
||||
"qkv row must hold at least [q|k|v]");
|
||||
// A lane moves 16 bytes, so every row start and every head start must land
|
||||
// on a 16-byte boundary. Every supported head_dim is a multiple of 8 at 2
|
||||
// bytes, so heads are fine; the row stride is the one that can break it.
|
||||
STD_TORCH_CHECK(qkv_stride % 8 == 0,
|
||||
"qkv row stride must be a multiple of 8 elements, got ",
|
||||
qkv_stride);
|
||||
|
||||
bool const do_rope = cos_sin_cache.has_value();
|
||||
bool rope_is_fp32 = true;
|
||||
if (do_rope) {
|
||||
STD_TORCH_CHECK(cos_sin_cache->is_cuda() && cos_sin_cache->dim() == 2 &&
|
||||
cos_sin_cache->is_contiguous(),
|
||||
"cos_sin_cache must be a contiguous 2-D CUDA tensor");
|
||||
STD_TORCH_CHECK(cos_sin_cache->size(1) == rotary_dim,
|
||||
"cos_sin_cache shape must be [max_pos, rotary_dim]");
|
||||
STD_TORCH_CHECK(
|
||||
cos_sin_cache->scalar_type() == torch::headeronly::ScalarType::Float ||
|
||||
cos_sin_cache->scalar_type() == qkv.scalar_type(),
|
||||
"cos_sin_cache must be float32 or match the qkv dtype");
|
||||
rope_is_fp32 =
|
||||
cos_sin_cache->scalar_type() == torch::headeronly::ScalarType::Float;
|
||||
// A lane owns 16 bytes = 8 elements of the head and must be wholly inside
|
||||
// or wholly outside the rotated prefix.
|
||||
STD_TORCH_CHECK(rotary_dim > 0 && rotary_dim <= hd && rotary_dim % 16 == 0,
|
||||
"rotary_dim must be a positive multiple of 16 and <= "
|
||||
"head_dim");
|
||||
}
|
||||
|
||||
vllm::Fp8KVCacheDataType const kv_dt =
|
||||
vllm::get_fp8_kv_cache_data_type(kv_cache_dtype);
|
||||
bool const insert_kv = kv_cache.has_value() && kv_cache->numel() > 0;
|
||||
bool const needs_scales =
|
||||
insert_kv && kv_dt != vllm::Fp8KVCacheDataType::kAuto;
|
||||
|
||||
int64_t kv_s_block = 0, kv_s_head = 0, kv_s_token = 0;
|
||||
if (insert_kv) {
|
||||
STD_TORCH_CHECK(
|
||||
slot_mapping.has_value() && slot_mapping->is_cuda() &&
|
||||
slot_mapping->scalar_type() == torch::headeronly::ScalarType::Long,
|
||||
"insert mode requires an int64 CUDA slot_mapping");
|
||||
STD_TORCH_CHECK(slot_mapping->numel() >= num_tokens,
|
||||
"slot_mapping must cover every token");
|
||||
STD_TORCH_CHECK(kv_cache->dim() == 4 && kv_cache->stride(3) == 1,
|
||||
"kv_cache must be [nb, nkv, bs, 2*head_dim] with a "
|
||||
"contiguous content dim");
|
||||
STD_TORCH_CHECK(kv_cache->size(3) == 2 * hd,
|
||||
"kv_cache content dim must be 2*head_dim");
|
||||
STD_TORCH_CHECK(kv_cache->size(1) == nkv,
|
||||
"kv_cache head dim must equal num_kv_heads");
|
||||
STD_TORCH_CHECK(kv_cache->size(2) == block_size,
|
||||
"kv_cache token dim must equal block_size");
|
||||
if (kv_dt == vllm::Fp8KVCacheDataType::kAuto) {
|
||||
STD_TORCH_CHECK(kv_cache->scalar_type() == qkv.scalar_type(),
|
||||
"auto kv_cache dtype must match qkv");
|
||||
} else {
|
||||
STD_TORCH_CHECK(
|
||||
kv_cache->scalar_type() == torch::headeronly::ScalarType::Byte ||
|
||||
kv_cache->scalar_type() ==
|
||||
torch::headeronly::ScalarType::Float8_e4m3fn,
|
||||
"fp8 kv_cache must use uint8 or float8_e4m3fn storage");
|
||||
STD_TORCH_CHECK(k_scale.has_value() && v_scale.has_value(),
|
||||
"fp8 kv cache requires k_scale and v_scale");
|
||||
STD_TORCH_CHECK(
|
||||
k_scale->numel() == 1 && v_scale->numel() == 1 &&
|
||||
k_scale->scalar_type() == torch::headeronly::ScalarType::Float &&
|
||||
v_scale->scalar_type() == torch::headeronly::ScalarType::Float,
|
||||
"k_scale and v_scale must be single fp32 values");
|
||||
}
|
||||
kv_s_block = kv_cache->stride(0);
|
||||
kv_s_head = kv_cache->stride(1);
|
||||
kv_s_token = kv_cache->stride(2);
|
||||
}
|
||||
|
||||
const torch::stable::accelerator::DeviceGuard device_guard(
|
||||
qkv.get_device_index());
|
||||
auto stream = get_current_cuda_stream(qkv.get_device_index());
|
||||
|
||||
VLLM_STABLE_DISPATCH_HALF_TYPES(
|
||||
qkv.scalar_type(), "fused_muse_spark_qknorm_rope_kv_insert", [&] {
|
||||
using st = scalar_t;
|
||||
DISPATCH_BY_KV_CACHE_DTYPE(qkv.scalar_type(), kv_cache_dtype,
|
||||
CALL_FUSED_MUSE_SPARK);
|
||||
});
|
||||
}
|
||||
|
||||
#undef CALL_FUSED_MUSE_SPARK
|
||||
@@ -0,0 +1,567 @@
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
// SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
// Step5 fused QK norm, RoPE, and KV-cache update kernel.
|
||||
|
||||
// This kernel is built into `_C_stable_libtorch` and uses the libtorch stable
|
||||
// ABI (torch::stable::Tensor / STD_TORCH_CHECK) instead of ATen and c10.
|
||||
#include "torch_utils.h"
|
||||
#include "dispatch_utils.h"
|
||||
#include "type_convert.cuh"
|
||||
#include "core/vec_dtypes/dtype_fp8.cuh"
|
||||
#include "quantization/w8a8/fp8/nvidia/quant_utils.cuh"
|
||||
|
||||
#include <torch/csrc/stable/macros.h>
|
||||
|
||||
#include <cuda_runtime.h>
|
||||
|
||||
#include <algorithm>
|
||||
#include <type_traits>
|
||||
|
||||
namespace vllm {
|
||||
|
||||
namespace {
|
||||
|
||||
constexpr int kWarpSize = 32;
|
||||
constexpr int kThreadsPerBlock = 128;
|
||||
|
||||
template <typename scalar_t, typename weight_t, typename rope_t, int HEAD_DIM,
|
||||
bool WIDE_192_ROTARY = false, bool FP8_CACHE = false>
|
||||
__global__ void qknorm_rope_cache_kernel(
|
||||
const void* __restrict__ qkv_void, void* __restrict__ q_out_void,
|
||||
void* __restrict__ k_out_void, void* __restrict__ v_out_void,
|
||||
const void* __restrict__ q_weight_void,
|
||||
const void* __restrict__ k_weight_void, const void* __restrict__ cos_void,
|
||||
const void* __restrict__ sin_void, const int64_t* __restrict__ positions,
|
||||
const int64_t* __restrict__ slot_mapping, void* __restrict__ k_cache_void,
|
||||
void* __restrict__ v_cache_void, int64_t qkv_token_stride,
|
||||
int64_t q_token_stride, int64_t k_token_stride, int64_t v_token_stride,
|
||||
int64_t cos_token_stride, int64_t sin_token_stride,
|
||||
int64_t cache_block_stride, int64_t cache_page_stride,
|
||||
int64_t cache_head_stride, int num_tokens, int num_slots, int num_q_heads,
|
||||
int num_kv_heads, int rotary_pairs, int block_size, float epsilon,
|
||||
int block_size_shift, float norm_weight_bias, const float* k_scale,
|
||||
const float* v_scale) {
|
||||
#if !defined(__CUDA_ARCH__) || __CUDA_ARCH__ < 800
|
||||
if constexpr (std::is_same_v<scalar_t, torch::headeronly::BFloat16> ||
|
||||
std::is_same_v<weight_t, torch::headeronly::BFloat16> ||
|
||||
std::is_same_v<rope_t, torch::headeronly::BFloat16>) {
|
||||
return;
|
||||
} else {
|
||||
#endif
|
||||
using ScalarConverter = _typeConvert<scalar_t>;
|
||||
using WeightConverter = _typeConvert<weight_t>;
|
||||
using RopeConverter = _typeConvert<rope_t>;
|
||||
using Scalar = typename ScalarConverter::hip_type;
|
||||
using Weight = typename WeightConverter::hip_type;
|
||||
using Rope = typename RopeConverter::hip_type;
|
||||
|
||||
// Preserve the original per-head reduction and activation rounding.
|
||||
static_assert(!WIDE_192_ROTARY || HEAD_DIM == 192);
|
||||
constexpr int kThreadsPerHead =
|
||||
WIDE_192_ROTARY ? 32
|
||||
: (HEAD_DIM <= 64 ? 8 : (HEAD_DIM <= 192 ? 16 : 32));
|
||||
constexpr int kValuesPerVector =
|
||||
WIDE_192_ROTARY ? 8 : (HEAD_DIM == 192 ? 4 : 8);
|
||||
constexpr int kValuesPerTile = kThreadsPerHead * kValuesPerVector;
|
||||
constexpr int kNumVectors =
|
||||
(HEAD_DIM + kValuesPerTile - 1) / kValuesPerTile;
|
||||
constexpr int kValuesPerThread = kValuesPerVector * kNumVectors;
|
||||
constexpr int kTileSize = kValuesPerTile * kNumVectors;
|
||||
constexpr int kHeadsPerBlock = kThreadsPerBlock / kThreadsPerHead;
|
||||
|
||||
const auto* qkv = static_cast<const Scalar*>(qkv_void);
|
||||
auto* q_out = static_cast<Scalar*>(q_out_void);
|
||||
auto* k_out = static_cast<Scalar*>(k_out_void);
|
||||
auto* v_out = static_cast<Scalar*>(v_out_void);
|
||||
const auto* q_weight = static_cast<const Weight*>(q_weight_void);
|
||||
const auto* k_weight = static_cast<const Weight*>(k_weight_void);
|
||||
const auto* cos = static_cast<const Rope*>(cos_void);
|
||||
const auto* sin = static_cast<const Rope*>(sin_void);
|
||||
auto* k_cache = static_cast<Scalar*>(k_cache_void);
|
||||
auto* v_cache = static_cast<Scalar*>(v_cache_void);
|
||||
|
||||
const int head_in_block = threadIdx.x / kThreadsPerHead;
|
||||
const int lane = threadIdx.x % kThreadsPerHead;
|
||||
const int total_heads = num_q_heads + 2 * num_kv_heads;
|
||||
int token;
|
||||
int local_head;
|
||||
const int64_t global_head =
|
||||
static_cast<int64_t>(blockIdx.x) * kHeadsPerBlock + head_in_block;
|
||||
token = global_head / total_heads;
|
||||
// Reuse the quotient instead of spelling this as global_head %
|
||||
// total_heads. nvcc otherwise emits a second runtime integer-division
|
||||
// sequence for the flattened token/head mapping.
|
||||
local_head = static_cast<int>(global_head -
|
||||
static_cast<int64_t>(token) * total_heads);
|
||||
|
||||
if (token >= num_tokens) return;
|
||||
|
||||
const bool is_q = local_head < num_q_heads;
|
||||
const bool is_k = !is_q && local_head < num_q_heads + num_kv_heads;
|
||||
const int head = is_q ? local_head
|
||||
: (is_k ? local_head - num_q_heads
|
||||
: local_head - num_q_heads - num_kv_heads);
|
||||
|
||||
int64_t input_offset = token * qkv_token_stride;
|
||||
Scalar* output;
|
||||
int64_t output_offset;
|
||||
if (is_q) {
|
||||
input_offset += static_cast<int64_t>(head) * HEAD_DIM;
|
||||
output = q_out;
|
||||
output_offset =
|
||||
token * q_token_stride + static_cast<int64_t>(head) * HEAD_DIM;
|
||||
} else if (is_k) {
|
||||
input_offset += static_cast<int64_t>(num_q_heads + head) * HEAD_DIM;
|
||||
output = k_out;
|
||||
output_offset =
|
||||
token * k_token_stride + static_cast<int64_t>(head) * HEAD_DIM;
|
||||
} else {
|
||||
input_offset +=
|
||||
static_cast<int64_t>(num_q_heads + num_kv_heads + head) * HEAD_DIM;
|
||||
output = v_out;
|
||||
output_offset =
|
||||
token * v_token_stride + static_cast<int64_t>(head) * HEAD_DIM;
|
||||
}
|
||||
|
||||
bool write_cache = false;
|
||||
int64_t cache_offset = 0;
|
||||
if (!is_q && token < num_slots) {
|
||||
const int64_t slot = slot_mapping[token];
|
||||
if (slot >= 0) {
|
||||
int64_t block;
|
||||
int64_t page;
|
||||
if (block_size_shift >= 0) {
|
||||
// vLLM production cache blocks are powers of two. Avoid the costly
|
||||
// runtime int64 division on that path while retaining generic block
|
||||
// sizes through the fallback below.
|
||||
block = slot >> block_size_shift;
|
||||
page = slot & (block_size - 1);
|
||||
} else {
|
||||
block = slot / block_size;
|
||||
page = slot - block * block_size;
|
||||
}
|
||||
cache_offset = block * cache_block_stride + page * cache_page_stride +
|
||||
static_cast<int64_t>(head) * cache_head_stride;
|
||||
write_cache = true;
|
||||
}
|
||||
}
|
||||
|
||||
if (!is_q && !is_k) {
|
||||
#pragma unroll
|
||||
for (int block = 0; block < kNumVectors; ++block) {
|
||||
#pragma unroll
|
||||
for (int element = 0; element < kValuesPerVector; ++element) {
|
||||
const int dim = block * kThreadsPerHead * kValuesPerVector +
|
||||
lane * kValuesPerVector + element;
|
||||
if constexpr (kTileSize > HEAD_DIM) {
|
||||
if (dim >= HEAD_DIM) continue;
|
||||
}
|
||||
const Scalar value = qkv[input_offset + dim];
|
||||
output[output_offset + dim] = value;
|
||||
if (write_cache) {
|
||||
if constexpr (FP8_CACHE) {
|
||||
reinterpret_cast<uint8_t*>(v_cache_void)[cache_offset + dim] =
|
||||
fp8::scaled_convert<uint8_t, float,
|
||||
Fp8KVCacheDataType::kFp8E4M3>(
|
||||
ScalarConverter::convert(value), *v_scale);
|
||||
} else {
|
||||
v_cache[cache_offset + dim] = value;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
float values[kValuesPerThread];
|
||||
float sum_squares = 0.0f;
|
||||
#pragma unroll
|
||||
for (int block = 0; block < kNumVectors; ++block) {
|
||||
#pragma unroll
|
||||
for (int element = 0; element < kValuesPerVector; ++element) {
|
||||
const int value_index = block * kValuesPerVector + element;
|
||||
const int dim = block * kThreadsPerHead * kValuesPerVector +
|
||||
lane * kValuesPerVector + element;
|
||||
float value = 0.0f;
|
||||
if constexpr (kTileSize > HEAD_DIM) {
|
||||
if (dim < HEAD_DIM) {
|
||||
value = ScalarConverter::convert(qkv[input_offset + dim]);
|
||||
}
|
||||
} else {
|
||||
value = ScalarConverter::convert(qkv[input_offset + dim]);
|
||||
}
|
||||
values[value_index] = value;
|
||||
// CuTeDSL lowers its fragment reduction to a sequential fma.rn
|
||||
// chain. Keep this explicit: changing it changes RMSNorm bits.
|
||||
sum_squares = __fmaf_rn(value, value, sum_squares);
|
||||
}
|
||||
}
|
||||
|
||||
const int warp_lane = threadIdx.x % kWarpSize;
|
||||
const int subwarp_base = (warp_lane / kThreadsPerHead) * kThreadsPerHead;
|
||||
constexpr uint32_t kSubwarpBits =
|
||||
kThreadsPerHead == kWarpSize ? 0xffffffffu
|
||||
: ((uint32_t{1} << kThreadsPerHead) - 1);
|
||||
const uint32_t subwarp_mask = kSubwarpBits << subwarp_base;
|
||||
#pragma unroll
|
||||
for (int offset = 1; offset < kThreadsPerHead; offset <<= 1) {
|
||||
// CuTeDSL's warp_reduce uses ascending XOR offsets (1, 2, 4, ...).
|
||||
sum_squares +=
|
||||
__shfl_xor_sync(subwarp_mask, sum_squares, offset, kWarpSize);
|
||||
}
|
||||
const float mean_square =
|
||||
__fdiv_rn(sum_squares, static_cast<float>(HEAD_DIM));
|
||||
const float inverse_rms = rsqrtf(mean_square + epsilon);
|
||||
const Weight* weight = is_q ? q_weight : k_weight;
|
||||
|
||||
// CuTeDSL writes RMSNorm results to activation-typed shared memory and
|
||||
// reloads them for RoPE. This intermediate FP16/BF16 rounding boundary
|
||||
// is intentional and is necessary for bitwise output parity.
|
||||
__shared__ Scalar normalized[kHeadsPerBlock * HEAD_DIM];
|
||||
Scalar* normalized_head = normalized + head_in_block * HEAD_DIM;
|
||||
#pragma unroll
|
||||
for (int block = 0; block < kNumVectors; ++block) {
|
||||
#pragma unroll
|
||||
for (int element = 0; element < kValuesPerVector; ++element) {
|
||||
const int value_index = block * kValuesPerVector + element;
|
||||
const int dim = block * kThreadsPerHead * kValuesPerVector +
|
||||
lane * kValuesPerVector + element;
|
||||
if constexpr (kTileSize > HEAD_DIM) {
|
||||
if (dim >= HEAD_DIM) continue;
|
||||
}
|
||||
const float x_hat = values[value_index] * inverse_rms;
|
||||
const float norm_weight =
|
||||
WeightConverter::convert(weight[dim]) + norm_weight_bias;
|
||||
normalized_head[dim] = ScalarConverter::convert(x_hat * norm_weight);
|
||||
}
|
||||
}
|
||||
|
||||
__syncwarp(subwarp_mask);
|
||||
|
||||
// Apply NeoX-style RoPE. CuTeDSL assigns four rotary pairs to each
|
||||
// participating lane and reloads activation-typed normalized values.
|
||||
constexpr int kRotaryPairsPerThread = 4;
|
||||
if (lane < rotary_pairs / kRotaryPairsPerThread) {
|
||||
const int64_t position = positions[token];
|
||||
const Rope* token_cos = cos + position * cos_token_stride;
|
||||
const Rope* token_sin = sin + position * sin_token_stride;
|
||||
#pragma unroll
|
||||
for (int element = 0; element < kRotaryPairsPerThread; ++element) {
|
||||
const int pair = lane * kRotaryPairsPerThread + element;
|
||||
const int dim0 = pair;
|
||||
const int dim1 = pair + rotary_pairs;
|
||||
const float value0 = ScalarConverter::convert(normalized_head[dim0]);
|
||||
const float value1 = ScalarConverter::convert(normalized_head[dim1]);
|
||||
const float cos_value = RopeConverter::convert(token_cos[pair]);
|
||||
const float sin_value = RopeConverter::convert(token_sin[pair]);
|
||||
|
||||
// Match CuTeDSL PTX exactly. The first half is mul + mul + sub; the
|
||||
// second is one mul followed by one fused multiply-add.
|
||||
const float rotated0 = __fsub_rn(__fmul_rn(value0, cos_value),
|
||||
__fmul_rn(value1, sin_value));
|
||||
const float rotated1 =
|
||||
__fmaf_rn(value0, sin_value, __fmul_rn(value1, cos_value));
|
||||
normalized_head[dim0] = ScalarConverter::convert(rotated0);
|
||||
normalized_head[dim1] = ScalarConverter::convert(rotated1);
|
||||
}
|
||||
}
|
||||
|
||||
__syncwarp(subwarp_mask);
|
||||
|
||||
#pragma unroll
|
||||
for (int block = 0; block < kNumVectors; ++block) {
|
||||
#pragma unroll
|
||||
for (int element = 0; element < kValuesPerVector; ++element) {
|
||||
const int dim = block * kThreadsPerHead * kValuesPerVector +
|
||||
lane * kValuesPerVector + element;
|
||||
if constexpr (kTileSize > HEAD_DIM) {
|
||||
if (dim >= HEAD_DIM) continue;
|
||||
}
|
||||
const Scalar value = normalized_head[dim];
|
||||
output[output_offset + dim] = value;
|
||||
if (write_cache) {
|
||||
if constexpr (FP8_CACHE) {
|
||||
reinterpret_cast<uint8_t*>(k_cache_void)[cache_offset + dim] =
|
||||
fp8::scaled_convert<uint8_t, float,
|
||||
Fp8KVCacheDataType::kFp8E4M3>(
|
||||
ScalarConverter::convert(value), *k_scale);
|
||||
} else {
|
||||
k_cache[cache_offset + dim] = value;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#if !defined(__CUDA_ARCH__) || __CUDA_ARCH__ < 800
|
||||
}
|
||||
#endif
|
||||
}
|
||||
|
||||
template <typename scalar_t, typename weight_t, typename rope_t, bool FP8_CACHE>
|
||||
void launch_qknorm_rope_cache(
|
||||
const torch::stable::Tensor& qkv, torch::stable::Tensor& q_out,
|
||||
torch::stable::Tensor& k_out, torch::stable::Tensor& v_out,
|
||||
const torch::stable::Tensor& q_weight,
|
||||
const torch::stable::Tensor& k_weight, const torch::stable::Tensor& cos,
|
||||
const torch::stable::Tensor& sin, const torch::stable::Tensor& positions,
|
||||
const torch::stable::Tensor& slot_mapping, torch::stable::Tensor& k_cache,
|
||||
torch::stable::Tensor& v_cache, int head_dim, int num_q_heads,
|
||||
int num_kv_heads, int rotary_pairs, float epsilon, float norm_weight_bias,
|
||||
cudaStream_t stream, const float* k_scale, const float* v_scale) {
|
||||
const int total_heads = num_q_heads + 2 * num_kv_heads;
|
||||
const int64_t total_work = qkv.size(0) * total_heads;
|
||||
const int block_size = static_cast<int>(k_cache.size(1));
|
||||
const int block_size_shift =
|
||||
(block_size & (block_size - 1)) == 0
|
||||
? __builtin_ctz(static_cast<unsigned int>(block_size))
|
||||
: -1;
|
||||
|
||||
#define LAUNCH_QKNORM_ROPE_CACHE(HEAD_DIM, WIDE_192_ROTARY) \
|
||||
do { \
|
||||
constexpr int threads_per_head = \
|
||||
WIDE_192_ROTARY ? 32 \
|
||||
: (HEAD_DIM <= 64 ? 8 : (HEAD_DIM <= 192 ? 16 : 32)); \
|
||||
constexpr int heads_per_block = kThreadsPerBlock / threads_per_head; \
|
||||
const dim3 grid((total_work + heads_per_block - 1) / heads_per_block); \
|
||||
const dim3 block(kThreadsPerBlock); \
|
||||
qknorm_rope_cache_kernel<scalar_t, weight_t, rope_t, HEAD_DIM, \
|
||||
WIDE_192_ROTARY, FP8_CACHE> \
|
||||
<<<grid, block, 0, stream>>>( \
|
||||
qkv.data_ptr(), q_out.data_ptr(), k_out.data_ptr(), \
|
||||
v_out.data_ptr(), q_weight.data_ptr(), k_weight.data_ptr(), \
|
||||
cos.data_ptr(), sin.data_ptr(), \
|
||||
reinterpret_cast<const int64_t*>(positions.data_ptr()), \
|
||||
reinterpret_cast<const int64_t*>(slot_mapping.data_ptr()), \
|
||||
k_cache.data_ptr(), v_cache.data_ptr(), qkv.stride(0), \
|
||||
q_out.stride(0), k_out.stride(0), v_out.stride(0), cos.stride(0), \
|
||||
sin.stride(0), k_cache.stride(0), k_cache.stride(1), \
|
||||
k_cache.stride(2), static_cast<int>(qkv.size(0)), \
|
||||
static_cast<int>(slot_mapping.numel()), num_q_heads, num_kv_heads, \
|
||||
rotary_pairs, block_size, epsilon, block_size_shift, \
|
||||
norm_weight_bias, k_scale, v_scale); \
|
||||
} while (false)
|
||||
|
||||
switch (head_dim) {
|
||||
case 64:
|
||||
LAUNCH_QKNORM_ROPE_CACHE(64, false);
|
||||
break;
|
||||
case 128:
|
||||
LAUNCH_QKNORM_ROPE_CACHE(128, false);
|
||||
break;
|
||||
case 192:
|
||||
if (rotary_pairs > 64) {
|
||||
LAUNCH_QKNORM_ROPE_CACHE(192, true);
|
||||
} else {
|
||||
LAUNCH_QKNORM_ROPE_CACHE(192, false);
|
||||
}
|
||||
break;
|
||||
case 256:
|
||||
LAUNCH_QKNORM_ROPE_CACHE(256, false);
|
||||
break;
|
||||
default:
|
||||
STD_TORCH_CHECK(false, "Unsupported head_dim: ", head_dim);
|
||||
}
|
||||
#undef LAUNCH_QKNORM_ROPE_CACHE
|
||||
}
|
||||
|
||||
void check_cuda_tensor(const torch::stable::Tensor& tensor, const char* name) {
|
||||
STD_TORCH_CHECK(tensor.is_cuda(), name, " must be a CUDA tensor");
|
||||
}
|
||||
|
||||
void check_same_device(const torch::stable::Tensor& reference,
|
||||
const torch::stable::Tensor& tensor, const char* name) {
|
||||
STD_TORCH_CHECK(tensor.get_device_index() == reference.get_device_index(),
|
||||
name, " must be on the same CUDA device as qkv");
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
void fused_qknorm_rope_cache(
|
||||
torch::stable::Tensor& q_out, torch::stable::Tensor& k_out,
|
||||
torch::stable::Tensor& v_out, torch::stable::Tensor& qkv,
|
||||
torch::stable::Tensor& q_weight, torch::stable::Tensor& k_weight,
|
||||
torch::stable::Tensor& cos, torch::stable::Tensor& sin,
|
||||
torch::stable::Tensor& positions, torch::stable::Tensor& slot_mapping,
|
||||
torch::stable::Tensor& k_cache, torch::stable::Tensor& v_cache,
|
||||
int64_t head_dim, int64_t num_q_heads, int64_t num_kv_heads,
|
||||
int64_t rotary_pairs, double epsilon, double norm_weight_bias,
|
||||
const std::string& kv_cache_dtype,
|
||||
std::optional<torch::stable::Tensor> k_scale,
|
||||
std::optional<torch::stable::Tensor> v_scale) {
|
||||
check_cuda_tensor(qkv, "qkv");
|
||||
check_cuda_tensor(q_out, "q_out");
|
||||
check_cuda_tensor(k_out, "k_out");
|
||||
check_cuda_tensor(v_out, "v_out");
|
||||
check_cuda_tensor(q_weight, "q_weight");
|
||||
check_cuda_tensor(k_weight, "k_weight");
|
||||
check_cuda_tensor(cos, "cos");
|
||||
check_cuda_tensor(sin, "sin");
|
||||
check_cuda_tensor(positions, "positions");
|
||||
check_cuda_tensor(slot_mapping, "slot_mapping");
|
||||
check_cuda_tensor(k_cache, "k_cache");
|
||||
check_cuda_tensor(v_cache, "v_cache");
|
||||
check_same_device(qkv, q_out, "q_out");
|
||||
check_same_device(qkv, k_out, "k_out");
|
||||
check_same_device(qkv, v_out, "v_out");
|
||||
check_same_device(qkv, q_weight, "q_weight");
|
||||
check_same_device(qkv, k_weight, "k_weight");
|
||||
check_same_device(qkv, cos, "cos");
|
||||
check_same_device(qkv, sin, "sin");
|
||||
check_same_device(qkv, positions, "positions");
|
||||
check_same_device(qkv, slot_mapping, "slot_mapping");
|
||||
check_same_device(qkv, k_cache, "k_cache");
|
||||
check_same_device(qkv, v_cache, "v_cache");
|
||||
|
||||
STD_TORCH_CHECK(qkv.dim() == 2 && qkv.stride(1) == 1,
|
||||
"qkv must be a 2D tensor contiguous in its last dimension");
|
||||
STD_TORCH_CHECK(q_out.dim() == 2 && q_out.is_contiguous(),
|
||||
"q_out must be contiguous and 2D");
|
||||
STD_TORCH_CHECK(k_out.dim() == 2 && k_out.is_contiguous(),
|
||||
"k_out must be contiguous and 2D");
|
||||
STD_TORCH_CHECK(v_out.dim() == 2 && v_out.is_contiguous(),
|
||||
"v_out must be contiguous and 2D");
|
||||
STD_TORCH_CHECK(q_weight.dim() == 1 && q_weight.is_contiguous(),
|
||||
"q_weight must be contiguous and 1D");
|
||||
STD_TORCH_CHECK(k_weight.dim() == 1 && k_weight.is_contiguous(),
|
||||
"k_weight must be contiguous and 1D");
|
||||
STD_TORCH_CHECK(cos.dim() == 2 && cos.stride(1) == 1,
|
||||
"cos must be 2D and contiguous in its last dimension");
|
||||
STD_TORCH_CHECK(sin.dim() == 2 && sin.stride(1) == 1,
|
||||
"sin must be 2D and contiguous in its last dimension");
|
||||
STD_TORCH_CHECK(
|
||||
positions.scalar_type() == torch::headeronly::ScalarType::Long,
|
||||
"positions must be an int64 tensor");
|
||||
STD_TORCH_CHECK(positions.is_contiguous(), "positions must be contiguous");
|
||||
STD_TORCH_CHECK(
|
||||
slot_mapping.scalar_type() == torch::headeronly::ScalarType::Long,
|
||||
"slot_mapping must be an int64 tensor");
|
||||
STD_TORCH_CHECK(slot_mapping.is_contiguous(),
|
||||
"slot_mapping must be contiguous");
|
||||
STD_TORCH_CHECK(k_cache.dim() == 4 && v_cache.dim() == 4,
|
||||
"K/V cache must have shape [blocks, block_size, heads, dim]");
|
||||
for (int64_t d = 0; d < 4; ++d) {
|
||||
STD_TORCH_CHECK(k_cache.size(d) == v_cache.size(d) &&
|
||||
k_cache.stride(d) == v_cache.stride(d),
|
||||
"K/V cache must have identical shapes and strides");
|
||||
}
|
||||
STD_TORCH_CHECK(k_cache.stride(3) == 1,
|
||||
"K/V cache head dimension must be contiguous");
|
||||
|
||||
STD_TORCH_CHECK(num_q_heads > 0 && num_kv_heads > 0,
|
||||
"Q/KV head counts must be positive");
|
||||
STD_TORCH_CHECK(
|
||||
head_dim == 64 || head_dim == 128 || head_dim == 192 || head_dim == 256,
|
||||
"Unsupported head_dim");
|
||||
STD_TORCH_CHECK(cos.size(0) == sin.size(0),
|
||||
"cos/sin cache must have the same number of positions");
|
||||
const int64_t num_tokens = qkv.size(0);
|
||||
const int64_t packed_width = (num_q_heads + 2 * num_kv_heads) * head_dim;
|
||||
STD_TORCH_CHECK(qkv.size(1) == packed_width,
|
||||
"qkv width must equal (num_q_heads + 2 * num_kv_heads) * "
|
||||
"head_dim");
|
||||
STD_TORCH_CHECK(
|
||||
q_out.size(0) == num_tokens && q_out.size(1) == num_q_heads * head_dim,
|
||||
"q_out has an invalid shape");
|
||||
STD_TORCH_CHECK(
|
||||
k_out.size(0) == num_tokens && k_out.size(1) == num_kv_heads * head_dim,
|
||||
"k_out has an invalid shape");
|
||||
STD_TORCH_CHECK(
|
||||
v_out.size(0) == k_out.size(0) && v_out.size(1) == k_out.size(1),
|
||||
"v_out has an invalid shape");
|
||||
STD_TORCH_CHECK(q_weight.numel() == head_dim && k_weight.numel() == head_dim,
|
||||
"Q/K norm weights must have head_dim elements");
|
||||
STD_TORCH_CHECK(positions.numel() == num_tokens,
|
||||
"positions must have one entry per qkv token");
|
||||
STD_TORCH_CHECK(rotary_pairs >= 0 && rotary_pairs * 2 <= head_dim &&
|
||||
rotary_pairs % 4 == 0,
|
||||
"rotary_pairs must be a multiple of 4 in [0, head_dim / 2]");
|
||||
STD_TORCH_CHECK(cos.size(1) >= rotary_pairs && sin.size(1) >= rotary_pairs,
|
||||
"cos/sin cache is smaller than rotary_pairs");
|
||||
STD_TORCH_CHECK(
|
||||
k_cache.size(2) == num_kv_heads && k_cache.size(3) == head_dim,
|
||||
"K/V cache head shape does not match the kernel arguments");
|
||||
STD_TORCH_CHECK(k_cache.size(1) > 0, "K/V cache block_size must be positive");
|
||||
STD_TORCH_CHECK(slot_mapping.numel() <= num_tokens,
|
||||
"slot_mapping cannot contain more entries than qkv tokens");
|
||||
STD_TORCH_CHECK(qkv.scalar_type() == q_out.scalar_type() &&
|
||||
qkv.scalar_type() == k_out.scalar_type() &&
|
||||
qkv.scalar_type() == v_out.scalar_type(),
|
||||
"qkv and Q/K/V outputs must have the same dtype");
|
||||
const bool fp8_cache =
|
||||
kv_cache_dtype == "fp8" || kv_cache_dtype == "fp8_e4m3";
|
||||
STD_TORCH_CHECK(fp8_cache || kv_cache_dtype == "auto" ||
|
||||
kv_cache_dtype == "bfloat16" ||
|
||||
kv_cache_dtype == "float16",
|
||||
"Unsupported KV cache dtype: ", kv_cache_dtype);
|
||||
STD_TORCH_CHECK(k_cache.scalar_type() == v_cache.scalar_type(),
|
||||
"K/V caches must have the same dtype");
|
||||
if (fp8_cache) {
|
||||
STD_TORCH_CHECK(
|
||||
k_cache.scalar_type() == torch::headeronly::ScalarType::Byte ||
|
||||
k_cache.scalar_type() ==
|
||||
torch::headeronly::ScalarType::Float8_e4m3fn,
|
||||
"FP8 cache requires uint8 or float8_e4m3fn storage");
|
||||
STD_TORCH_CHECK(k_scale.has_value() && v_scale.has_value(),
|
||||
"FP8 cache requires K/V scales");
|
||||
for (const auto* scale : {&*k_scale, &*v_scale}) {
|
||||
check_cuda_tensor(*scale, "KV scale");
|
||||
check_same_device(qkv, *scale, "KV scale");
|
||||
STD_TORCH_CHECK(
|
||||
scale->numel() == 1 &&
|
||||
scale->scalar_type() == torch::headeronly::ScalarType::Float,
|
||||
"KV scales must be single FP32 values");
|
||||
}
|
||||
} else {
|
||||
STD_TORCH_CHECK(qkv.scalar_type() == k_cache.scalar_type(),
|
||||
"Unquantized cache must match QKV dtype");
|
||||
}
|
||||
STD_TORCH_CHECK(q_weight.scalar_type() == k_weight.scalar_type(),
|
||||
"Q/K norm weights must have the same dtype");
|
||||
STD_TORCH_CHECK(cos.scalar_type() == sin.scalar_type(),
|
||||
"cos and sin must have the same dtype");
|
||||
|
||||
if (num_tokens == 0) return;
|
||||
|
||||
const torch::stable::accelerator::DeviceGuard device_guard(
|
||||
qkv.get_device_index());
|
||||
const cudaStream_t stream = get_current_cuda_stream(qkv.get_device_index());
|
||||
VLLM_STABLE_DISPATCH_HALF_TYPES(
|
||||
qkv.scalar_type(), "qknorm_rope_cache_input", [&] {
|
||||
using input_t = scalar_t;
|
||||
VLLM_STABLE_DISPATCH_FLOATING_TYPES(
|
||||
q_weight.scalar_type(), "qknorm_rope_cache_weight", [&] {
|
||||
using norm_weight_t = scalar_t;
|
||||
VLLM_STABLE_DISPATCH_FLOATING_TYPES(
|
||||
cos.scalar_type(), "qknorm_rope_cache_rope", [&] {
|
||||
using rope_cache_t = scalar_t;
|
||||
auto launch = [&](auto fp8_tag) {
|
||||
launch_qknorm_rope_cache<input_t, norm_weight_t,
|
||||
rope_cache_t,
|
||||
decltype(fp8_tag)::value>(
|
||||
qkv, q_out, k_out, v_out, q_weight, k_weight, cos,
|
||||
sin, positions, slot_mapping, k_cache, v_cache,
|
||||
static_cast<int>(head_dim),
|
||||
static_cast<int>(num_q_heads),
|
||||
static_cast<int>(num_kv_heads),
|
||||
static_cast<int>(rotary_pairs),
|
||||
static_cast<float>(epsilon),
|
||||
static_cast<float>(norm_weight_bias), stream,
|
||||
fp8_cache
|
||||
? static_cast<const float*>(k_scale->data_ptr())
|
||||
: nullptr,
|
||||
fp8_cache
|
||||
? static_cast<const float*>(v_scale->data_ptr())
|
||||
: nullptr);
|
||||
};
|
||||
if (fp8_cache) {
|
||||
launch(std::true_type{});
|
||||
} else {
|
||||
launch(std::false_type{});
|
||||
}
|
||||
});
|
||||
});
|
||||
});
|
||||
STD_CUDA_KERNEL_LAUNCH_CHECK();
|
||||
}
|
||||
|
||||
} // namespace vllm
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,70 +0,0 @@
|
||||
// auto generated by generate_kernels.py
|
||||
// clang-format off
|
||||
|
||||
#include "kernel.h"
|
||||
#include "marlin_template.h"
|
||||
|
||||
namespace MARLIN_NAMESPACE_NAME {
|
||||
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 256, 1, 8, 8, true, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 1, 8, 4, true, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 1, 4, 8, true, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 256, 1, 8, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 1, 8, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 1, 4, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 256, 2, 16, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 2, 8, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 2, 4, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 256, 3, 16, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 3, 8, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 3, 4, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 256, 4, 16, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 4, 8, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 4, 4, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 1, 8, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 1, 8, 4, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 1, 4, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
}
|
||||
@@ -1,100 +0,0 @@
|
||||
// auto generated by generate_kernels.py
|
||||
// clang-format off
|
||||
|
||||
#include "kernel.h"
|
||||
#include "marlin_template.h"
|
||||
|
||||
namespace MARLIN_NAMESPACE_NAME {
|
||||
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 1, 8, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 1, 8, 4, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 1, 4, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
}
|
||||
@@ -1,130 +0,0 @@
|
||||
// auto generated by generate_kernels.py
|
||||
// clang-format off
|
||||
|
||||
#include "kernel.h"
|
||||
#include "marlin_template.h"
|
||||
|
||||
namespace MARLIN_NAMESPACE_NAME {
|
||||
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
}
|
||||
@@ -1,130 +0,0 @@
|
||||
// auto generated by generate_kernels.py
|
||||
// clang-format off
|
||||
|
||||
#include "kernel.h"
|
||||
#include "marlin_template.h"
|
||||
|
||||
namespace MARLIN_NAMESPACE_NAME {
|
||||
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
}
|
||||
@@ -1,130 +0,0 @@
|
||||
// auto generated by generate_kernels.py
|
||||
// clang-format off
|
||||
|
||||
#include "kernel.h"
|
||||
#include "marlin_template.h"
|
||||
|
||||
namespace MARLIN_NAMESPACE_NAME {
|
||||
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
}
|
||||
@@ -1,40 +0,0 @@
|
||||
// auto generated by generate_kernels.py
|
||||
// clang-format off
|
||||
|
||||
#include "kernel.h"
|
||||
#include "marlin_template.h"
|
||||
|
||||
namespace MARLIN_NAMESPACE_NAME {
|
||||
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 256, 1, 8, 8, true, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 1, 8, 4, true, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 1, 4, 8, true, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 256, 1, 8, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 1, 8, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 1, 4, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 256, 2, 16, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 2, 8, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 2, 4, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 256, 3, 16, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 3, 8, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 3, 4, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 256, 4, 16, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 4, 8, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 4, 4, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
}
|
||||
@@ -1,70 +0,0 @@
|
||||
// auto generated by generate_kernels.py
|
||||
// clang-format off
|
||||
|
||||
#include "kernel.h"
|
||||
#include "marlin_template.h"
|
||||
|
||||
namespace MARLIN_NAMESPACE_NAME {
|
||||
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
}
|
||||
@@ -1,130 +0,0 @@
|
||||
// auto generated by generate_kernels.py
|
||||
// clang-format off
|
||||
|
||||
#include "kernel.h"
|
||||
#include "marlin_template.h"
|
||||
|
||||
namespace MARLIN_NAMESPACE_NAME {
|
||||
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
}
|
||||
@@ -1,130 +0,0 @@
|
||||
// auto generated by generate_kernels.py
|
||||
// clang-format off
|
||||
|
||||
#include "kernel.h"
|
||||
#include "marlin_template.h"
|
||||
|
||||
namespace MARLIN_NAMESPACE_NAME {
|
||||
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
}
|
||||
@@ -1,130 +0,0 @@
|
||||
// auto generated by generate_kernels.py
|
||||
// clang-format off
|
||||
|
||||
#include "kernel.h"
|
||||
#include "marlin_template.h"
|
||||
|
||||
namespace MARLIN_NAMESPACE_NAME {
|
||||
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
}
|
||||
@@ -1,106 +0,0 @@
|
||||
// auto generated by generate_kernels.py
|
||||
// clang-format off
|
||||
|
||||
#include "kernel.h"
|
||||
#include "marlin_template.h"
|
||||
|
||||
namespace MARLIN_NAMESPACE_NAME {
|
||||
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
}
|
||||
@@ -1,106 +0,0 @@
|
||||
// auto generated by generate_kernels.py
|
||||
// clang-format off
|
||||
|
||||
#include "kernel.h"
|
||||
#include "marlin_template.h"
|
||||
|
||||
namespace MARLIN_NAMESPACE_NAME {
|
||||
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
}
|
||||
@@ -1,106 +0,0 @@
|
||||
// auto generated by generate_kernels.py
|
||||
// clang-format off
|
||||
|
||||
#include "kernel.h"
|
||||
#include "marlin_template.h"
|
||||
|
||||
namespace MARLIN_NAMESPACE_NAME {
|
||||
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
}
|
||||
@@ -1,106 +0,0 @@
|
||||
// auto generated by generate_kernels.py
|
||||
// clang-format off
|
||||
|
||||
#include "kernel.h"
|
||||
#include "marlin_template.h"
|
||||
|
||||
namespace MARLIN_NAMESPACE_NAME {
|
||||
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
}
|
||||
File diff suppressed because it is too large
Load Diff
@@ -1,70 +0,0 @@
|
||||
// auto generated by generate_kernels.py
|
||||
// clang-format off
|
||||
|
||||
#include "kernel.h"
|
||||
#include "marlin_template.h"
|
||||
|
||||
namespace MARLIN_NAMESPACE_NAME {
|
||||
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 256, 1, 8, 8, true, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 1, 8, 4, true, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 1, 4, 8, true, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 256, 1, 8, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 1, 8, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 1, 4, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 256, 2, 16, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 2, 8, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 2, 4, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 256, 3, 16, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 3, 8, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 3, 4, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 256, 4, 16, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 4, 8, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), 128, 4, 4, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 1, 8, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 1, 8, 4, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 1, 4, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE2M1f.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
}
|
||||
@@ -1,100 +0,0 @@
|
||||
// auto generated by generate_kernels.py
|
||||
// clang-format off
|
||||
|
||||
#include "kernel.h"
|
||||
#include "marlin_template.h"
|
||||
|
||||
namespace MARLIN_NAMESPACE_NAME {
|
||||
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 1, 8, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 1, 8, 4, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 1, 4, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kFE4M3fn.id(), vllm::kBFloat16.id(), vllm::kFE8M0fnu.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
}
|
||||
@@ -1,130 +0,0 @@
|
||||
// auto generated by generate_kernels.py
|
||||
// clang-format off
|
||||
|
||||
#include "kernel.h"
|
||||
#include "marlin_template.h"
|
||||
|
||||
namespace MARLIN_NAMESPACE_NAME {
|
||||
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
}
|
||||
@@ -1,130 +0,0 @@
|
||||
// auto generated by generate_kernels.py
|
||||
// clang-format off
|
||||
|
||||
#include "kernel.h"
|
||||
#include "marlin_template.h"
|
||||
|
||||
namespace MARLIN_NAMESPACE_NAME {
|
||||
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
}
|
||||
@@ -1,130 +0,0 @@
|
||||
// auto generated by generate_kernels.py
|
||||
// clang-format off
|
||||
|
||||
#include "kernel.h"
|
||||
#include "marlin_template.h"
|
||||
|
||||
namespace MARLIN_NAMESPACE_NAME {
|
||||
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kBFloat16.id(), vllm::kU8B128.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
}
|
||||
@@ -1,40 +0,0 @@
|
||||
// auto generated by generate_kernels.py
|
||||
// clang-format off
|
||||
|
||||
#include "kernel.h"
|
||||
#include "marlin_template.h"
|
||||
|
||||
namespace MARLIN_NAMESPACE_NAME {
|
||||
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 256, 1, 8, 8, true, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 1, 8, 4, true, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 1, 4, 8, true, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 256, 1, 8, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 1, 8, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 1, 4, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 256, 2, 16, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 2, 8, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 2, 4, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 256, 3, 16, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 3, 8, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 3, 4, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 256, 4, 16, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 4, 8, 4, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE2M1f.id(), vllm::kFloat16.id(), vllm::kFE4M3fn.id(), 128, 4, 4, 8, false, 4, 1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
}
|
||||
@@ -1,70 +0,0 @@
|
||||
// auto generated by generate_kernels.py
|
||||
// clang-format off
|
||||
|
||||
#include "kernel.h"
|
||||
#include "marlin_template.h"
|
||||
|
||||
namespace MARLIN_NAMESPACE_NAME {
|
||||
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kFE4M3fn.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
}
|
||||
@@ -1,130 +0,0 @@
|
||||
// auto generated by generate_kernels.py
|
||||
// clang-format off
|
||||
|
||||
#include "kernel.h"
|
||||
#include "marlin_template.h"
|
||||
|
||||
namespace MARLIN_NAMESPACE_NAME {
|
||||
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
}
|
||||
@@ -1,130 +0,0 @@
|
||||
// auto generated by generate_kernels.py
|
||||
// clang-format off
|
||||
|
||||
#include "kernel.h"
|
||||
#include "marlin_template.h"
|
||||
|
||||
namespace MARLIN_NAMESPACE_NAME {
|
||||
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
}
|
||||
@@ -1,130 +0,0 @@
|
||||
// auto generated by generate_kernels.py
|
||||
// clang-format off
|
||||
|
||||
#include "kernel.h"
|
||||
#include "marlin_template.h"
|
||||
|
||||
namespace MARLIN_NAMESPACE_NAME {
|
||||
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, true, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kFloat16.id(), vllm::kU8B128.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
}
|
||||
@@ -1,106 +0,0 @@
|
||||
// auto generated by generate_kernels.py
|
||||
// clang-format off
|
||||
|
||||
#include "kernel.h"
|
||||
#include "marlin_template.h"
|
||||
|
||||
namespace MARLIN_NAMESPACE_NAME {
|
||||
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
}
|
||||
@@ -1,106 +0,0 @@
|
||||
// auto generated by generate_kernels.py
|
||||
// clang-format off
|
||||
|
||||
#include "kernel.h"
|
||||
#include "marlin_template.h"
|
||||
|
||||
namespace MARLIN_NAMESPACE_NAME {
|
||||
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
}
|
||||
@@ -1,106 +0,0 @@
|
||||
// auto generated by generate_kernels.py
|
||||
// clang-format off
|
||||
|
||||
#include "kernel.h"
|
||||
#include "marlin_template.h"
|
||||
|
||||
namespace MARLIN_NAMESPACE_NAME {
|
||||
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kBFloat16.id(), vllm::kBFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
}
|
||||
@@ -1,106 +0,0 @@
|
||||
// auto generated by generate_kernels.py
|
||||
// clang-format off
|
||||
|
||||
#include "kernel.h"
|
||||
#include "marlin_template.h"
|
||||
|
||||
namespace MARLIN_NAMESPACE_NAME {
|
||||
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, -1, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 2, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 4, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 1, 8, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 1, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 2, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 2, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 3, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 3, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 256, 4, 16, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 8, 4, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
template __global__ void Marlin<vllm::kS8.id(), vllm::kU4B8.id(), vllm::kFloat16.id(), vllm::kFloat16.id(), 128, 4, 4, 8, false, 4, 8, false>( MARLIN_KERNEL_PARAMS );
|
||||
|
||||
}
|
||||
@@ -0,0 +1,100 @@
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
// SPDX-FileCopyrightText: Copyright contributors to the vLLM project
|
||||
|
||||
#include <cuda_runtime.h>
|
||||
#include <torch/csrc/stable/accelerator.h>
|
||||
#include <torch/csrc/stable/tensor.h>
|
||||
#include <torch/headeronly/core/ScalarType.h>
|
||||
|
||||
#include <algorithm>
|
||||
|
||||
#include "dispatch_utils.h"
|
||||
#include "quantization/vectorization_utils.cuh"
|
||||
#include "torch_utils.h"
|
||||
|
||||
namespace vllm {
|
||||
|
||||
// 16 B of output per thread; low-precision inputs are widened in registers.
|
||||
static constexpr int kMuseSoftCapVecSize = 4;
|
||||
|
||||
// blockIdx.x walks rows, blockIdx.y splits a row. Rows are the sampled tokens
|
||||
// (often one, at decode) while a row is the whole vocab, so splitting the row
|
||||
// is what keeps the GPU busy; a row-per-block grid would leave a single-token
|
||||
// decode on one SM.
|
||||
template <typename scalar_t>
|
||||
__global__ void muse_spark_mul_tanh_soft_cap_kernel(
|
||||
float* __restrict__ out, const scalar_t* __restrict__ input, int d,
|
||||
float multiplier, float soft_cap_temp) {
|
||||
const int64_t row_offset = static_cast<int64_t>(blockIdx.x) * d;
|
||||
const scalar_t* __restrict__ in_row = input + row_offset;
|
||||
float* __restrict__ out_row = out + row_offset;
|
||||
|
||||
const int tid = blockIdx.y * blockDim.x + threadIdx.x;
|
||||
const int stride = gridDim.y * blockDim.x;
|
||||
|
||||
vectorize_with_alignment<kMuseSoftCapVecSize>(
|
||||
in_row, out_row, d, tid, stride,
|
||||
[=] __device__(float& dst, const scalar_t& src) {
|
||||
// Multiply then divide, as MulTanhSoftCap.forward does, so fp32
|
||||
// rounding matches the eager path.
|
||||
dst = soft_cap_temp *
|
||||
tanhf(static_cast<float>(src) * multiplier / soft_cap_temp);
|
||||
});
|
||||
}
|
||||
|
||||
} // namespace vllm
|
||||
|
||||
// out = soft_cap_temp * tanh(input * multiplier / soft_cap_temp), fp32.
|
||||
void muse_spark_mul_tanh_soft_cap(torch::stable::Tensor& out,
|
||||
torch::stable::Tensor& input,
|
||||
double multiplier, double soft_cap_temp) {
|
||||
STD_TORCH_CHECK(
|
||||
input.scalar_type() == torch::headeronly::ScalarType::Float ||
|
||||
input.scalar_type() == torch::headeronly::ScalarType::Half ||
|
||||
input.scalar_type() == torch::headeronly::ScalarType::BFloat16,
|
||||
"muse_spark_mul_tanh_soft_cap: input must be float32, float16 or "
|
||||
"bfloat16");
|
||||
STD_TORCH_CHECK(out.scalar_type() == torch::headeronly::ScalarType::Float,
|
||||
"muse_spark_mul_tanh_soft_cap: out must be float32");
|
||||
STD_TORCH_CHECK(input.is_contiguous(),
|
||||
"muse_spark_mul_tanh_soft_cap: input must be contiguous");
|
||||
STD_TORCH_CHECK(out.is_contiguous(),
|
||||
"muse_spark_mul_tanh_soft_cap: out must be contiguous");
|
||||
STD_TORCH_CHECK(out.numel() == input.numel(),
|
||||
"muse_spark_mul_tanh_soft_cap: out and input must have the "
|
||||
"same number of elements");
|
||||
STD_TORCH_CHECK(
|
||||
soft_cap_temp != 0.0,
|
||||
"muse_spark_mul_tanh_soft_cap: soft_cap_temp must be nonzero");
|
||||
|
||||
if (input.numel() == 0) {
|
||||
return;
|
||||
}
|
||||
|
||||
const int d = static_cast<int>(input.size(input.dim() - 1));
|
||||
const int64_t num_tokens = input.numel() / d;
|
||||
|
||||
constexpr int kThreads = 128;
|
||||
const int64_t vecs_per_row =
|
||||
(d + vllm::kMuseSoftCapVecSize - 1) / vllm::kMuseSoftCapVecSize;
|
||||
const int64_t blocks_to_cover = (vecs_per_row + kThreads - 1) / kThreads;
|
||||
constexpr int64_t kTargetBlocks = 32768;
|
||||
const int64_t blocks_per_row = std::clamp<int64_t>(
|
||||
kTargetBlocks / num_tokens, 1, std::max<int64_t>(blocks_to_cover, 1));
|
||||
|
||||
dim3 grid(static_cast<unsigned>(num_tokens),
|
||||
static_cast<unsigned>(blocks_per_row));
|
||||
dim3 block(kThreads);
|
||||
|
||||
const torch::stable::accelerator::DeviceGuard device_guard(
|
||||
input.get_device_index());
|
||||
const cudaStream_t stream = get_current_cuda_stream();
|
||||
VLLM_STABLE_DISPATCH_FLOATING_TYPES(
|
||||
input.scalar_type(), "muse_spark_mul_tanh_soft_cap", [&] {
|
||||
vllm::muse_spark_mul_tanh_soft_cap_kernel<scalar_t>
|
||||
<<<grid, block, 0, stream>>>(out.mutable_data_ptr<float>(),
|
||||
input.const_data_ptr<scalar_t>(), d,
|
||||
static_cast<float>(multiplier),
|
||||
static_cast<float>(soft_cap_temp));
|
||||
});
|
||||
}
|
||||
+33
-4
@@ -363,6 +363,35 @@ minimax_allreduce_rms_qk(torch::stable::Tensor qkv,
|
||||
int64_t const nranks, double const eps);
|
||||
#endif
|
||||
|
||||
#ifndef USE_ROCM
|
||||
namespace vllm {
|
||||
void fused_qknorm_rope_cache(
|
||||
torch::stable::Tensor& q_out, torch::stable::Tensor& k_out,
|
||||
torch::stable::Tensor& v_out, torch::stable::Tensor& qkv,
|
||||
torch::stable::Tensor& q_weight, torch::stable::Tensor& k_weight,
|
||||
torch::stable::Tensor& cos, torch::stable::Tensor& sin,
|
||||
torch::stable::Tensor& positions, torch::stable::Tensor& slot_mapping,
|
||||
torch::stable::Tensor& k_cache, torch::stable::Tensor& v_cache,
|
||||
int64_t head_dim, int64_t num_q_heads, int64_t num_kv_heads,
|
||||
int64_t rotary_pairs, double epsilon, double norm_weight_bias,
|
||||
const std::string& kv_cache_dtype,
|
||||
std::optional<torch::stable::Tensor> k_scale,
|
||||
std::optional<torch::stable::Tensor> v_scale);
|
||||
} // namespace vllm
|
||||
#endif
|
||||
|
||||
void fused_muse_spark_qknorm_rope_kv_insert(
|
||||
torch::stable::Tensor& qkv, torch::stable::Tensor const& positions,
|
||||
int64_t num_heads, int64_t num_kv_heads, int64_t head_dim, double eps,
|
||||
std::optional<torch::stable::Tensor> cos_sin_cache, int64_t rotary_dim,
|
||||
std::optional<torch::stable::Tensor> slot_mapping,
|
||||
std::optional<torch::stable::Tensor> kv_cache, int64_t block_size,
|
||||
const std::string& kv_cache_dtype,
|
||||
std::optional<torch::stable::Tensor> k_scale,
|
||||
std::optional<torch::stable::Tensor> v_scale,
|
||||
std::optional<torch::stable::Tensor> q_fp8_out,
|
||||
std::optional<torch::stable::Tensor> q_scale);
|
||||
|
||||
// Horizontally-fused MiniMax-M3 QK-norm + partial NeoX RoPE (+ optional KV /
|
||||
// index-cache insert). Dense layer: norm+RoPE only; sparse layer: also packs
|
||||
// the index branch and scatters k/v/index_k into their paged caches.
|
||||
@@ -546,6 +575,10 @@ std::tuple<int64_t, torch::stable::Tensor> allocate_shared_buffer_and_handle(
|
||||
int64_t open_mem_handle(torch::stable::Tensor& mem_handle);
|
||||
void free_shared_buffer(int64_t buffer);
|
||||
|
||||
void muse_spark_mul_tanh_soft_cap(torch::stable::Tensor& out,
|
||||
torch::stable::Tensor& input,
|
||||
double multiplier, double soft_cap_temp);
|
||||
|
||||
// Activation kernels (shared CUDA/ROCm)
|
||||
void silu_and_mul(torch::stable::Tensor& out, torch::stable::Tensor& input);
|
||||
void silu_and_mul_clamp(torch::stable::Tensor& out,
|
||||
@@ -579,10 +612,6 @@ void situ_and_mul_quant(
|
||||
int64_t group_size = 0,
|
||||
std::optional<torch::stable::Tensor> num_valid_tokens = std::nullopt,
|
||||
int64_t topk = 1);
|
||||
void masked_situ_and_mul(torch::stable::Tensor& out,
|
||||
torch::stable::Tensor& input,
|
||||
const torch::stable::Tensor& expert_num_tokens,
|
||||
double beta = 1.0, double linear_beta = -1.0);
|
||||
void masked_moe_activation(torch::stable::Tensor& out,
|
||||
torch::stable::Tensor& input,
|
||||
const torch::stable::Tensor& valid_token_counts,
|
||||
|
||||
@@ -135,6 +135,13 @@ hadamard_transform_kernel(b16* a, b16* out, int total_num_chunks) {
|
||||
|
||||
#pragma unroll
|
||||
for (int64_t k = 0; k < num_chunks; k++) {
|
||||
// The last warp may own fewer chunks than num_chunks: loading the
|
||||
// rest would read past the end of `a`. The cp.async.wait_group counts
|
||||
// below already assume only real_num_chunks groups were committed.
|
||||
if constexpr(enable_mask) {
|
||||
if (k >= real_num_chunks)
|
||||
break;
|
||||
}
|
||||
size_t shared_ptr = __cvta_generic_to_shared(b_frag_ptr);
|
||||
#if (__CUDA_ARCH__ >= 900) // SM90
|
||||
asm volatile(
|
||||
|
||||
+31
-4
@@ -501,6 +501,27 @@ STABLE_TORCH_LIBRARY_FRAGMENT(_C, ops) {
|
||||
"float eps) -> (Tensor, Tensor)");
|
||||
#endif
|
||||
|
||||
#ifndef USE_ROCM
|
||||
ops.def(
|
||||
"fused_qknorm_rope_cache("
|
||||
"Tensor! q_out, Tensor! k_out, Tensor! v_out, Tensor qkv, "
|
||||
"Tensor q_weight, Tensor k_weight, Tensor cos, Tensor sin, "
|
||||
"Tensor positions, Tensor slot_mapping, Tensor! k_cache, "
|
||||
"Tensor! v_cache, int head_dim, int num_q_heads, int num_kv_heads, "
|
||||
"int rotary_pairs, float epsilon, float norm_weight_bias, "
|
||||
"str kv_cache_dtype=\"auto\", Tensor? k_scale=None, Tensor? "
|
||||
"v_scale=None) -> ()");
|
||||
#endif
|
||||
|
||||
ops.def(
|
||||
"fused_muse_spark_qknorm_rope_kv_insert("
|
||||
"Tensor! qkv, Tensor positions, int num_heads, int num_kv_heads, "
|
||||
"int head_dim, float eps, Tensor? cos_sin_cache, int rotary_dim, "
|
||||
"Tensor? slot_mapping, Tensor!? kv_cache, int block_size, "
|
||||
"str kv_cache_dtype, Tensor? k_scale=None, Tensor? v_scale=None, "
|
||||
"Tensor!? q_fp8_out=None, Tensor? q_scale=None) "
|
||||
"-> ()");
|
||||
|
||||
// Horizontally-fused MiniMax-M3 QK-norm + partial NeoX RoPE + KV-insert.
|
||||
ops.def(
|
||||
"fused_minimax_m3_qknorm_rope_kv_insert("
|
||||
@@ -598,6 +619,10 @@ STABLE_TORCH_LIBRARY_FRAGMENT(_C, ops) {
|
||||
"y_q, Tensor! y_s, bool use_ue8m0) -> ()");
|
||||
ops.def("weak_ref_tensor(Tensor input) -> Tensor");
|
||||
|
||||
ops.def(
|
||||
"muse_spark_mul_tanh_soft_cap(Tensor! out, Tensor input, "
|
||||
"float multiplier, float soft_cap_temp) -> ()");
|
||||
|
||||
// Activation function used in SwiGLU.
|
||||
ops.def("silu_and_mul(Tensor! result, Tensor input) -> ()");
|
||||
|
||||
@@ -640,9 +665,6 @@ STABLE_TORCH_LIBRARY_FRAGMENT(_C, ops) {
|
||||
"situ_and_mul_quant(Tensor! out, Tensor! scale, Tensor input, "
|
||||
"float beta=1.0, float linear_beta=-1.0, int group_size=0, "
|
||||
"Tensor? num_valid_tokens=None, int topk=1) -> ()");
|
||||
ops.def(
|
||||
"masked_situ_and_mul(Tensor! out, Tensor input, Tensor "
|
||||
"expert_num_tokens, float beta=1.0, float linear_beta=-1.0) -> ()");
|
||||
ops.def(
|
||||
"masked_moe_activation(Tensor! out, Tensor input, Tensor "
|
||||
"valid_token_counts, str activation, float clamp_limit=0.0, float "
|
||||
@@ -831,7 +853,11 @@ STABLE_TORCH_LIBRARY_IMPL(_C, CUDA, ops) {
|
||||
TORCH_BOX(&fused_kimi_k3_mla_decode_q_concat_ds_mla_insert));
|
||||
#ifndef USE_ROCM
|
||||
ops.impl("minimax_allreduce_rms_qk", TORCH_BOX(&minimax_allreduce_rms_qk));
|
||||
ops.impl("fused_qknorm_rope_cache",
|
||||
TORCH_BOX(&vllm::fused_qknorm_rope_cache));
|
||||
#endif
|
||||
ops.impl("fused_muse_spark_qknorm_rope_kv_insert",
|
||||
TORCH_BOX(&fused_muse_spark_qknorm_rope_kv_insert));
|
||||
ops.impl("fused_minimax_m3_qknorm_rope_kv_insert",
|
||||
TORCH_BOX(&fused_minimax_m3_qknorm_rope_kv_insert));
|
||||
#ifdef VLLM_ENABLE_FUSED_KDA_DECODE
|
||||
@@ -867,6 +893,8 @@ STABLE_TORCH_LIBRARY_IMPL(_C, CUDA, ops) {
|
||||
TORCH_BOX(&persistent_masked_m_silu_mul_quant));
|
||||
ops.impl("weak_ref_tensor", TORCH_BOX(&weak_ref_tensor));
|
||||
ops.impl("silu_and_mul_quant", TORCH_BOX(&silu_and_mul_quant));
|
||||
ops.impl("muse_spark_mul_tanh_soft_cap",
|
||||
TORCH_BOX(&muse_spark_mul_tanh_soft_cap));
|
||||
ops.impl("silu_and_mul", TORCH_BOX(&silu_and_mul));
|
||||
ops.impl("mul_and_silu", TORCH_BOX(&mul_and_silu));
|
||||
ops.impl("gelu_and_mul", TORCH_BOX(&gelu_and_mul));
|
||||
@@ -875,7 +903,6 @@ STABLE_TORCH_LIBRARY_IMPL(_C, CUDA, ops) {
|
||||
ops.impl("swigluoai_and_mul", TORCH_BOX(&swigluoai_and_mul));
|
||||
ops.impl("situ_and_mul", TORCH_BOX(&situ_and_mul));
|
||||
ops.impl("situ_and_mul_quant", TORCH_BOX(&situ_and_mul_quant));
|
||||
ops.impl("masked_situ_and_mul", TORCH_BOX(&masked_situ_and_mul));
|
||||
ops.impl("masked_moe_activation", TORCH_BOX(&masked_moe_activation));
|
||||
ops.impl("gelu_new", TORCH_BOX(&gelu_new));
|
||||
ops.impl("gelu_fast", TORCH_BOX(&gelu_fast));
|
||||
|
||||
+5
-1
@@ -502,7 +502,11 @@ RUN --mount=type=cache,target=/opt/uv/cache \
|
||||
> /tmp/nccl-override.txt && \
|
||||
export UV_OVERRIDE=/tmp/nccl-override.txt; \
|
||||
fi && \
|
||||
export TORCH_CUDA_ARCH_LIST='9.0a 10.0a' && \
|
||||
DEEPEP_ARCH_LIST='9.0a 10.0a' && \
|
||||
if [ "$(printf '%s\n' 12.9 "$CUDA_VERSION" | sort -V | head -n1)" = 12.9 ]; then \
|
||||
DEEPEP_ARCH_LIST="$DEEPEP_ARCH_LIST 10.3a"; \
|
||||
fi && \
|
||||
export TORCH_CUDA_ARCH_LIST="$DEEPEP_ARCH_LIST" && \
|
||||
/tmp/install_python_libraries.sh \
|
||||
--workspace /tmp/ep_kernels_workspace \
|
||||
--mode wheel \
|
||||
|
||||
@@ -1,453 +0,0 @@
|
||||
ARG BASE_IMAGE=ubuntu:22.04
|
||||
ARG PYTHON_VERSION=3.12
|
||||
ARG TORCH_VERSION=2.12.0+rocm10.0.0
|
||||
ARG TORCHVISION_VERSION=0.27.0+rocm10.0.0
|
||||
ARG TORCHAUDIO_VERSION=2.11.0+rocm10.0.0
|
||||
ARG TRITON_BRANCH="669b31ac" # release/internal/3.8.x as of 09/29/2026
|
||||
ARG TRITON_VERSION=3.8.0+git669b31ac
|
||||
ARG TRITON_REPO="https://github.com/ROCm/triton.git"
|
||||
ARG FA_BRANCH="0e60e394"
|
||||
ARG FA_REPO="https://github.com/Dao-AILab/flash-attention.git"
|
||||
ARG AITER_BRANCH="v0.1.25"
|
||||
ARG AITER_REPO="https://github.com/ROCm/aiter.git"
|
||||
ARG MORI_BRANCH="v1.2.3.post1"
|
||||
ARG MORI_REPO="https://github.com/ROCm/mori.git"
|
||||
ARG ROCM_SYSTEMS_REPO="https://github.com/ROCm/rocm-systems.git"
|
||||
ARG ROCM_RUNTIME_COMMIT="25e14349f2606a6b95c4ccd45eb92a1835646cf2"
|
||||
|
||||
|
||||
# TheRock configuration
|
||||
ARG SITE_PACKAGES=/usr/local/lib/python${PYTHON_VERSION}/dist-packages
|
||||
ARG ROCM_RELEASE_WHEELS_MULTIARCH_URL="https://stable.repo.amd.com/rocm/whl-next/" # For pre-built GPU images, packages, etc.
|
||||
ARG ROCM_NIGHTLY_WHEELS_MULTIARCH_URL="https://nightly.repo.amd.com/rocm/whl-next/" # For torch, triton, etc. wheels with latest fixes.
|
||||
ARG ROCM_SDK_VERSION=10.0.0
|
||||
|
||||
# Build configuration
|
||||
ARG PYTORCH_ROCM_ARCH="gfx90a;gfx942;gfx950;gfx1100;gfx1101;gfx1200;gfx1201;gfx1150;gfx1151"
|
||||
ARG AITER_ROCM_ARCH="gfx942;gfx950"
|
||||
|
||||
# Sccache configuration (only used in release pipeline)
|
||||
ARG USE_SCCACHE
|
||||
ARG SCCACHE_DOWNLOAD_URL
|
||||
ARG SCCACHE_ENDPOINT
|
||||
ARG SCCACHE_BUCKET_NAME=vllm-build-sccache
|
||||
ARG SCCACHE_REGION_NAME=us-west-2
|
||||
ARG SCCACHE_S3_NO_CREDENTIALS=0
|
||||
|
||||
FROM ${BASE_IMAGE} AS base
|
||||
|
||||
ARG PYTORCH_ROCM_ARCH
|
||||
ENV PYTORCH_ROCM_ARCH="${PYTORCH_ROCM_ARCH}"
|
||||
ARG AITER_ROCM_ARCH
|
||||
ENV AITER_ROCM_ARCH="${AITER_ROCM_ARCH}"
|
||||
|
||||
ARG SITE_PACKAGES
|
||||
ENV SITE_PACKAGES=${SITE_PACKAGES}
|
||||
ENV SDK_DEV="$SITE_PACKAGES/_rocm_sdk_devel" \
|
||||
SDK_CORE="$SITE_PACKAGES/_rocm_sdk_core"
|
||||
ENV ROCM_PATH="$SDK_DEV" \
|
||||
ROCM_HOME="$SDK_DEV" \
|
||||
HIP_PATH="$SDK_DEV" \
|
||||
HIP_CLANG_PATH="$SDK_DEV/lib/llvm/bin" \
|
||||
HIP_DEVICE_LIB_PATH="$SDK_CORE/lib/llvm/amdgcn/bitcode"
|
||||
ENV LD_LIBRARY_PATH="$SDK_DEV/lib:$SDK_DEV/lib/rocm_sysdeps/lib:$SDK_CORE/lib:$SDK_CORE/lib/rocm_sysdeps/lib"
|
||||
ENV PATH="$SDK_DEV/bin:$SDK_DEV/lib/llvm/bin:$SDK_CORE/bin:$PATH"
|
||||
|
||||
# Required for RCCL in ROCm7.1
|
||||
ENV HSA_NO_SCRATCH_RECLAIM=1
|
||||
|
||||
ARG PYTHON_VERSION
|
||||
ENV PYTHON_VERSION=${PYTHON_VERSION}
|
||||
|
||||
RUN mkdir -p /app
|
||||
WORKDIR /app
|
||||
ENV DEBIAN_FRONTEND=noninteractive
|
||||
|
||||
# Install Python and other dependencies
|
||||
RUN apt-get update -y \
|
||||
&& apt-get install -y software-properties-common build-essential git curl sudo vim less libgfortran5 libopenmpi-dev libpci-dev liblzma-dev pkg-config libdrm-dev libnuma-dev kmod \
|
||||
&& for i in 1 2 3; do \
|
||||
add-apt-repository -y ppa:deadsnakes/ppa && break || \
|
||||
{ echo "Attempt $i failed, retrying in 5s..."; sleep 5; }; \
|
||||
done \
|
||||
&& apt-get update -y \
|
||||
&& apt-get install -y python${PYTHON_VERSION} python${PYTHON_VERSION}-dev python${PYTHON_VERSION}-venv \
|
||||
python${PYTHON_VERSION}-lib2to3 python-is-python3 \
|
||||
&& update-alternatives --install /usr/bin/python3 python3 /usr/bin/python${PYTHON_VERSION} 1 \
|
||||
&& update-alternatives --set python3 /usr/bin/python${PYTHON_VERSION} \
|
||||
&& ln -sf /usr/bin/python${PYTHON_VERSION}-config /usr/bin/python3-config \
|
||||
&& curl -sS https://bootstrap.pypa.io/get-pip.py | python${PYTHON_VERSION} \
|
||||
&& python3 --version && python3 -m pip --version
|
||||
|
||||
RUN pip install -U packaging 'cmake<4' ninja wheel 'setuptools<80' pybind11==3.0.4 Cython
|
||||
RUN apt-get update && apt-get install -y libjpeg-dev libsox-dev libsox-fmt-all sox && rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install ROCm SDK via TheRock
|
||||
ARG ROCM_RELEASE_WHEELS_MULTIARCH_URL
|
||||
ARG ROCM_NIGHTLY_WHEELS_MULTIARCH_URL
|
||||
ARG ROCM_SDK_VERSION
|
||||
RUN pip install --index-url ${ROCM_RELEASE_WHEELS_MULTIARCH_URL} \
|
||||
--extra-index-url ${ROCM_NIGHTLY_WHEELS_MULTIARCH_URL} \
|
||||
"rocm[libraries,devel]${ROCM_SDK_VERSION:+==${ROCM_SDK_VERSION}}" \
|
||||
&& rocm-sdk init
|
||||
|
||||
# Install PyTorch wheels (nightly for latest torch, release as fallback)
|
||||
ARG TORCH_VERSION
|
||||
ARG TORCHAUDIO_VERSION
|
||||
ARG TORCHVISION_VERSION
|
||||
RUN pip install \
|
||||
--index-url ${ROCM_RELEASE_WHEELS_MULTIARCH_URL} \
|
||||
--extra-index-url ${ROCM_NIGHTLY_WHEELS_MULTIARCH_URL} \
|
||||
"torch==${TORCH_VERSION}" \
|
||||
"torchaudio==${TORCHAUDIO_VERSION}" \
|
||||
"torchvision==${TORCHVISION_VERSION}"
|
||||
|
||||
# Install the AMD-compiled device kernels corresponding to PYTORCH_ROCM_ARCH:
|
||||
|
||||
# Possible values that can be in PYTORCH_ROCM_ARCH:
|
||||
#
|
||||
# gfx90a (MI210/250), gfx942 (MI300), gfx950 (MI350), gfx1100 (7900XTX),
|
||||
# gfx1101 (7800XT), gfx1200 (9070XT), gfx1201 (9070), gfx1150/1151 (Strix APU)
|
||||
|
||||
# For each arch in PYTORCH_ROCM_ARCH, install the precompiled GPU code that
|
||||
# PyTorch needs to run on that specific chip.
|
||||
SHELL ["/bin/bash", "-c"]
|
||||
RUN TORCH_VER=$(python3 -c "import torch; print(torch.__version__)") \
|
||||
&& for arch in $(echo "${PYTORCH_ROCM_ARCH}" | tr ';' ' '); do \
|
||||
echo "Installing device packages for ${arch}" \
|
||||
&& pip install --index-url ${ROCM_RELEASE_WHEELS_MULTIARCH_URL} \
|
||||
--extra-index-url ${ROCM_NIGHTLY_WHEELS_MULTIARCH_URL} \
|
||||
amd-torch-device-${arch}==${TORCH_VER} \
|
||||
rocm-sdk-device-${arch}==${ROCM_SDK_VERSION}; \
|
||||
done
|
||||
|
||||
# Remove gfx90a xnack kpacks (ASAN artifacts that cause hipErrorInvalidImage on MI250)
|
||||
RUN rm -f "${SITE_PACKAGES}/torch/.kpack/torch_gfx90a:xnack+.kpack" \
|
||||
"${SITE_PACKAGES}/torch/.kpack/torch_gfx90a:xnack-.kpack"
|
||||
|
||||
RUN cd $SITE_PACKAGES/_rocm_sdk_core/share/amd_smi && pip install .
|
||||
ENV PYTHONPATH=$SITE_PACKAGES/_rocm_sdk_core/share/amd_smi
|
||||
|
||||
# TheRock runtime fixes
|
||||
RUN if [ -f "${SITE_PACKAGES}/rocm_sdk/__init__.py" ]; then \
|
||||
sed -i 's/rtld_global: bool = True/rtld_global: bool = False/g' "${SITE_PACKAGES}/rocm_sdk/__init__.py"; \
|
||||
fi \
|
||||
&& if [ -f "${SITE_PACKAGES}/_rocm_sdk_core/lib/librocm_smi64.so.1" ] \
|
||||
&& [ ! -e "${ROCM_PATH}/lib/librocm_smi64.so.7" ]; then \
|
||||
ln -sfn "${SITE_PACKAGES}/_rocm_sdk_core/lib/librocm_smi64.so.1" "${ROCM_PATH}/lib/librocm_smi64.so.7"; \
|
||||
fi \
|
||||
&& printf '%s\n%s\n' "${ROCM_PATH}/lib" "${SITE_PACKAGES}/_rocm_sdk_core/lib" > /etc/ld.so.conf.d/rocm-pip.conf \
|
||||
&& for f in ${SITE_PACKAGES}/_rocm_sdk_core/lib/libamdhip64.so*; do \
|
||||
bn=$(basename "$f"); \
|
||||
if [ -f "${ROCM_PATH}/lib/${bn}" ] && [ -f "$f" ] && [ ! -L "$f" ]; then \
|
||||
rm -f "$f"; \
|
||||
ln -sfn "${ROCM_PATH}/lib/${bn}" "$f"; \
|
||||
fi; \
|
||||
done \
|
||||
&& for f in librocprofiler-sdk.so.1 librocprofiler-register.so.0 \
|
||||
librocprofiler-sdk-roctx.so.1; do \
|
||||
if [ -e "${SITE_PACKAGES}/_rocm_sdk_core/lib/${f}" ] \
|
||||
&& [ -e "${SITE_PACKAGES}/_rocm_sdk_devel/lib/${f}" ]; then \
|
||||
rm -f "${SITE_PACKAGES}/_rocm_sdk_core/lib/${f}"; \
|
||||
ln -sfn "${SITE_PACKAGES}/_rocm_sdk_devel/lib/${f}" \
|
||||
"${SITE_PACKAGES}/_rocm_sdk_core/lib/${f}"; \
|
||||
fi; \
|
||||
done \
|
||||
&& ldconfig
|
||||
|
||||
# Install sccache if USE_SCCACHE is enabled (for release builds)
|
||||
ARG USE_SCCACHE
|
||||
ARG SCCACHE_DOWNLOAD_URL
|
||||
ARG SCCACHE_ENDPOINT
|
||||
ARG SCCACHE_BUCKET_NAME
|
||||
ARG SCCACHE_REGION_NAME
|
||||
ARG SCCACHE_S3_NO_CREDENTIALS
|
||||
RUN if [ "$USE_SCCACHE" = "1" ]; then \
|
||||
echo "Installing sccache..." \
|
||||
&& SCCACHE_ARCH="x86_64" \
|
||||
&& SCCACHE_VERSION="v0.8.1" \
|
||||
&& SCCACHE_DL_URL="${SCCACHE_DOWNLOAD_URL:-https://github.com/mozilla/sccache/releases/download/${SCCACHE_VERSION}/sccache-${SCCACHE_VERSION}-${SCCACHE_ARCH}-unknown-linux-musl.tar.gz}" \
|
||||
&& curl -L -o /tmp/sccache.tar.gz ${SCCACHE_DL_URL} \
|
||||
&& tar -xzf /tmp/sccache.tar.gz -C /tmp \
|
||||
&& mv /tmp/sccache-${SCCACHE_VERSION}-${SCCACHE_ARCH}-unknown-linux-musl/sccache /usr/bin/sccache \
|
||||
&& chmod +x /usr/bin/sccache \
|
||||
&& rm -rf /tmp/sccache.tar.gz /tmp/sccache-${SCCACHE_VERSION}-${SCCACHE_ARCH}-unknown-linux-musl \
|
||||
&& sccache --version; \
|
||||
fi
|
||||
|
||||
# Setup sccache for HIP compilation via HIP_CLANG_PATH
|
||||
# This creates wrapper scripts in a separate directory and points HIP to use them
|
||||
# This avoids modifying the original ROCm binaries which can break detection
|
||||
# NOTE: HIP_CLANG_PATH is NOT set as ENV to avoid affecting downstream images (Dockerfile.rocm)
|
||||
# Instead, each build stage should export HIP_CLANG_PATH=/opt/sccache-wrappers if USE_SCCACHE=1
|
||||
RUN if [ "$USE_SCCACHE" = "1" ]; then \
|
||||
echo "Setting up sccache wrappers for HIP compilation..." \
|
||||
&& mkdir -p /opt/sccache-wrappers \
|
||||
&& printf '#!/bin/bash\nexec sccache %s/lib/llvm/bin/clang++ "$@"\n' "${ROCM_PATH}" > /opt/sccache-wrappers/clang++ \
|
||||
&& chmod +x /opt/sccache-wrappers/clang++ \
|
||||
&& printf '#!/bin/bash\nexec sccache %s/lib/llvm/bin/clang "$@"\n' "${ROCM_PATH}" > /opt/sccache-wrappers/clang \
|
||||
&& chmod +x /opt/sccache-wrappers/clang \
|
||||
&& echo "sccache wrappers created in /opt/sccache-wrappers"; \
|
||||
fi
|
||||
|
||||
ENV SCCACHE_BUCKET=${USE_SCCACHE:+${SCCACHE_BUCKET_NAME}}
|
||||
ENV SCCACHE_REGION=${USE_SCCACHE:+${SCCACHE_REGION_NAME}}
|
||||
ENV SCCACHE_S3_NO_CREDENTIALS=${USE_SCCACHE:+${SCCACHE_S3_NO_CREDENTIALS}}
|
||||
ENV SCCACHE_IDLE_TIMEOUT=${USE_SCCACHE:+0}
|
||||
|
||||
|
||||
###
|
||||
### MORI Build
|
||||
###
|
||||
FROM base AS build_mori
|
||||
ARG MORI_BRANCH
|
||||
ARG MORI_REPO
|
||||
RUN git clone ${MORI_REPO}
|
||||
RUN cd mori \
|
||||
&& git checkout ${MORI_BRANCH} \
|
||||
&& git submodule update --init --recursive \
|
||||
&& ln -sfn ${ROCM_PATH} /opt/rocm \
|
||||
&& pip install "setuptools_scm>=6.2" \
|
||||
&& CMAKE_PREFIX_PATH="${ROCM_PATH}:${ROCM_PATH}/lib/rocm_sysdeps" \
|
||||
python3 setup.py bdist_wheel --dist-dir=dist && ls /app/mori/dist/*.whl
|
||||
RUN mkdir -p /app/install && cp /app/mori/dist/*.whl /app/install
|
||||
|
||||
###
|
||||
### FlashAttention Build
|
||||
###
|
||||
FROM base AS build_fa
|
||||
ARG FA_BRANCH
|
||||
ARG FA_REPO
|
||||
ARG USE_SCCACHE
|
||||
RUN git clone ${FA_REPO}
|
||||
RUN cd flash-attention \
|
||||
&& git checkout ${FA_BRANCH} \
|
||||
&& git submodule update --init \
|
||||
&& if [ "$USE_SCCACHE" = "1" ]; then \
|
||||
export HIP_CLANG_PATH=/opt/sccache-wrappers \
|
||||
&& sccache --show-stats; \
|
||||
fi \
|
||||
&& GPU_ARCHS=$(echo ${PYTORCH_ROCM_ARCH} | sed -e 's/;gfx1[0-9]\{3\}//g') python3 setup.py bdist_wheel --dist-dir=dist \
|
||||
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi
|
||||
RUN mkdir -p /app/install && cp /app/flash-attention/dist/*.whl /app/install
|
||||
|
||||
|
||||
###
|
||||
### AITER Build
|
||||
###
|
||||
FROM base AS build_aiter
|
||||
ARG AITER_BRANCH
|
||||
ARG AITER_REPO
|
||||
ARG USE_SCCACHE
|
||||
RUN git clone --recursive --branch ${AITER_BRANCH} ${AITER_REPO}
|
||||
RUN cd aiter \
|
||||
&& git submodule update --init --recursive \
|
||||
&& pip install -r requirements.txt
|
||||
# AITER_USE_SYSTEM_TRITON keeps setup.py from resolving its own triton, which would
|
||||
# displace the pinned one already installed in base.
|
||||
RUN pip install pyyaml && cd aiter \
|
||||
&& if [ "$USE_SCCACHE" = "1" ]; then \
|
||||
export HIP_CLANG_PATH=/opt/sccache-wrappers \
|
||||
&& sccache --show-stats; \
|
||||
fi \
|
||||
&& PREBUILD_KERNELS=1 AITER_USE_SYSTEM_TRITON=1 GPU_ARCHS=${AITER_ROCM_ARCH} python3 setup.py bdist_wheel --dist-dir=dist \
|
||||
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi \
|
||||
&& ls /app/aiter/dist/*.whl
|
||||
RUN mkdir -p /app/install && cp /app/aiter/dist/*.whl /app/install
|
||||
|
||||
|
||||
###
|
||||
### Triton Build
|
||||
###
|
||||
FROM base AS build_triton
|
||||
ARG TRITON_BRANCH
|
||||
ARG TRITON_REPO
|
||||
RUN git clone ${TRITON_REPO}
|
||||
RUN cd triton \
|
||||
&& git checkout ${TRITON_BRANCH} \
|
||||
&& if [ ! -f setup.py ]; then cd python; fi \
|
||||
&& python3 setup.py bdist_wheel --dist-dir=dist \
|
||||
&& mkdir -p /app/install && cp dist/*.whl /app/install
|
||||
|
||||
|
||||
###
|
||||
### ROCr + CLR build
|
||||
###
|
||||
# Brings in multi-stream perf improvements; drop after these arrive in an official release
|
||||
FROM base AS build_rocm_runtime
|
||||
ARG ROCM_SYSTEMS_REPO
|
||||
ARG ROCM_RUNTIME_COMMIT
|
||||
# libgl-dev: rocclr does find_package(OpenGL REQUIRED) unconditionally and the wheel image has
|
||||
# no GL headers. The link must not pick up a libGL DT_NEEDED, which is asserted after the build.
|
||||
RUN apt-get update \
|
||||
&& apt-get install -y --no-install-recommends g++ libdw-dev xxd libgl-dev libglx-dev \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
RUN pip install --no-cache-dir CppHeaderParser
|
||||
RUN git init -q /src && cd /src \
|
||||
&& git remote add origin ${ROCM_SYSTEMS_REPO} \
|
||||
&& git config core.sparseCheckout true \
|
||||
&& git config remote.origin.promisor true \
|
||||
&& git config remote.origin.partialclonefilter blob:none \
|
||||
&& git sparse-checkout init --cone \
|
||||
&& git sparse-checkout set projects/rocr-runtime projects/clr projects/hip shared cmake \
|
||||
&& git fetch --filter=blob:none --depth 1 origin ${ROCM_RUNTIME_COMMIT} \
|
||||
&& git checkout -q FETCH_HEAD
|
||||
# libelf is taken from the SDK's own vendored sysdeps rather than apt -- the wheel image has no
|
||||
# /usr/include/libelf.h -- so ROCr links against the copy the wheel itself was built against.
|
||||
ENV SYSDEPS="$SDK_DEV/lib/rocm_sysdeps"
|
||||
# ROCr first, and installed into $SDK_DEV, because the clr build below needs its headers.
|
||||
RUN cd /src/projects/rocr-runtime \
|
||||
&& cmake -B build -G Ninja \
|
||||
-DCMAKE_BUILD_TYPE=Release \
|
||||
-DCMAKE_PREFIX_PATH="$SDK_DEV;$SYSDEPS" \
|
||||
-DCMAKE_INSTALL_PREFIX="$SDK_DEV" \
|
||||
-DLIBELF_INCLUDE_DIRS="$SYSDEPS/include" \
|
||||
-DLIBELF_LIBRARIES="$SYSDEPS/lib/libelf.so" \
|
||||
&& cmake --build build --parallel 16 \
|
||||
&& cmake --install build \
|
||||
&& cmake --install build --prefix /rocr-install --strip
|
||||
RUN cd /src/projects/clr \
|
||||
&& cmake -B build -G Ninja \
|
||||
-DCMAKE_BUILD_TYPE=Release \
|
||||
-DCLR_BUILD_HIP=ON \
|
||||
-DCLR_BUILD_OCL=OFF \
|
||||
-DROCM_KPACK_ENABLED=ON \
|
||||
-DHIP_COMMON_DIR=/src/projects/hip \
|
||||
-DROCM_PATH="$SDK_DEV" \
|
||||
-DCMAKE_PREFIX_PATH="$SDK_DEV;$SYSDEPS" \
|
||||
-DCMAKE_INSTALL_PREFIX="$SDK_DEV" \
|
||||
-DHIP_LLVM_ROOT="$SDK_DEV/lib/llvm" \
|
||||
&& cmake --build build --parallel 16 \
|
||||
&& cmake --install build --prefix /clr-install --strip
|
||||
RUN mkdir -p /staging/lib \
|
||||
&& cp -P /rocr-install/lib/libhsa-runtime64.so* /staging/lib/ \
|
||||
&& cp -P /clr-install/lib/libamdhip64.so* /staging/lib/ \
|
||||
&& ls -l /staging/lib
|
||||
|
||||
|
||||
###
|
||||
### Final Build
|
||||
###
|
||||
|
||||
# Wheel release stage -
|
||||
# only includes dependencies used by wheel release pipeline
|
||||
FROM base AS debs_wheel_release
|
||||
RUN mkdir /app/debs
|
||||
RUN --mount=type=bind,from=build_triton,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
RUN --mount=type=bind,from=build_fa,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
RUN --mount=type=bind,from=build_aiter,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
|
||||
FROM base AS debs
|
||||
RUN mkdir /app/debs
|
||||
RUN --mount=type=bind,from=build_triton,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
RUN --mount=type=bind,from=build_fa,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
RUN --mount=type=bind,from=build_aiter,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
RUN --mount=type=bind,from=build_mori,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
|
||||
FROM base AS final
|
||||
ARG TRITON_VERSION
|
||||
|
||||
# torch 2.12 implicitly attaches rocprofiler-sdk; its queue interposition hangs at HSA
|
||||
# teardown, so disable it. TODO: drop once ROCm's stock rocprofiler carries the fix.
|
||||
ENV ROCPROFILER_QUEUE_INTERPOSITION=0
|
||||
|
||||
COPY --from=build_rocm_runtime /staging/lib/ ${SDK_DEV}/lib/
|
||||
# Hard link the patched library onto every stock file name, in both SDK trees, so that the
|
||||
# SONAME lookup and the hard coded paths all resolve to it. Hard links rather than symlinks
|
||||
# because ldconfig rewrites SONAME symlinks but cannot re-point an inode. $SDK_CORE keeps its
|
||||
# own libhsa-runtime64 copy, which is on LD_LIBRARY_PATH and in ld.so.conf; relinking it is
|
||||
# belt and braces rather than strictly required, but it leaves no stock ROCr in the image.
|
||||
RUN set -eux; \
|
||||
cd "$SDK_DEV/lib"; \
|
||||
ours_hsa="$(readlink -f libhsa-runtime64.so.1)"; \
|
||||
ours_hip="$(readlink -f libamdhip64.so.7)"; \
|
||||
for f in "$SDK_DEV"/lib/libhsa-runtime64.so.1* "$SDK_CORE"/lib/libhsa-runtime64.so.1* \
|
||||
"$SDK_DEV"/lib/libamdhip64.so.7* "$SDK_CORE"/lib/libamdhip64.so.7*; do \
|
||||
[ -f "$f" ] && [ ! -L "$f" ] || continue; \
|
||||
case "${f##*/}" in libhsa*) ours="$ours_hsa";; *) ours="$ours_hip";; esac; \
|
||||
[ "$f" -ef "$ours" ] || ln -f "$ours" "$f"; \
|
||||
done
|
||||
|
||||
RUN ldconfig
|
||||
|
||||
# This is to prevent resolvers from checking METADATA and RECORD and clobbering
|
||||
# the Triton source build we installed. Constraints files are unable to help
|
||||
# with this. However, these entries are optional and can be blanked which
|
||||
# solves the problem.
|
||||
RUN TORCH_DIST_INFO=$(ls -d ${SITE_PACKAGES}/torch-*.dist-info) \
|
||||
&& grep -q '^Requires-Dist: triton' "$TORCH_DIST_INFO/METADATA" \
|
||||
&& sed -i '/^Requires-Dist: triton/d' "$TORCH_DIST_INFO/METADATA" \
|
||||
&& sed -i "s|^\(${TORCH_DIST_INFO##*/}/METADATA\),.*|\1,,|" "$TORCH_DIST_INFO/RECORD"
|
||||
|
||||
RUN --mount=type=bind,from=debs,src=/app/debs,target=/install \
|
||||
pip install /install/*.whl \
|
||||
&& ! pip check 2>&1 | grep -iE '^(torch|triton) ' \
|
||||
&& test "$(pip show triton | awk '/^Version:/ {print $2}')" = "${TRITON_VERSION}" \
|
||||
# Create a constraints file to protect the versions we need, especially
|
||||
# triton, requirements will try to reinstall repeatedly.
|
||||
&& pip list --format=freeze \
|
||||
| grep -E '^(torch|triton|torchaudio|torchvision|pybind11)==' > /etc/rocm-constraints.txt \
|
||||
&& echo "ROCm constraints:" && cat /etc/rocm-constraints.txt \
|
||||
&& test "$(wc -l < /etc/rocm-constraints.txt)" -eq 5 \
|
||||
&& grep -q '^pybind11==3\.0\.' /etc/rocm-constraints.txt \
|
||||
&& python3 -c "import torch; assert torch.version.hip, f'non-ROCm torch: {torch.__version__}'; print(f'torch={torch.__version__}')" \
|
||||
&& python3 -c "import triton; print(f'triton={triton.__version__}')" \
|
||||
&& pip list 2>/dev/null | grep -iE '^(torch|triton) '
|
||||
|
||||
ARG BASE_IMAGE
|
||||
ARG PYTHON_VERSION
|
||||
ARG ROCM_SDK_VERSION
|
||||
ARG TORCH_VERSION
|
||||
ARG TORCHAUDIO_VERSION
|
||||
ARG TORCHVISION_VERSION
|
||||
ARG PYTORCH_ROCM_ARCH
|
||||
ARG AITER_ROCM_ARCH
|
||||
ARG TRITON_BRANCH
|
||||
ARG TRITON_REPO
|
||||
ARG FA_BRANCH
|
||||
ARG FA_REPO
|
||||
ARG AITER_BRANCH
|
||||
ARG AITER_REPO
|
||||
ARG MORI_BRANCH
|
||||
ARG MORI_REPO
|
||||
ARG ROCM_SYSTEMS_REPO
|
||||
ARG ROCM_RUNTIME_COMMIT
|
||||
ARG TRITON_VERSION
|
||||
# Record hashes for Triton and pybind11 so we can verify later.
|
||||
RUN echo "BASE_IMAGE: ${BASE_IMAGE}" > /app/versions.txt \
|
||||
&& echo "PYTHON_VERSION: ${PYTHON_VERSION}" >> /app/versions.txt \
|
||||
&& echo "ROCM_SDK_VERSION: ${ROCM_SDK_VERSION}" >> /app/versions.txt \
|
||||
&& echo "TORCH_VERSION: ${TORCH_VERSION}" >> /app/versions.txt \
|
||||
&& echo "TORCHAUDIO_VERSION: ${TORCHAUDIO_VERSION}" >> /app/versions.txt \
|
||||
&& echo "TORCHVISION_VERSION: ${TORCHVISION_VERSION}" >> /app/versions.txt \
|
||||
&& echo "PYTORCH_ROCM_ARCH: ${PYTORCH_ROCM_ARCH}" >> /app/versions.txt \
|
||||
&& echo "AITER_ROCM_ARCH: ${AITER_ROCM_ARCH}" >> /app/versions.txt \
|
||||
&& echo "TRITON_BRANCH: ${TRITON_BRANCH}" >> /app/versions.txt \
|
||||
&& echo "TRITON_REPO: ${TRITON_REPO}" >> /app/versions.txt \
|
||||
&& echo "FA_BRANCH: ${FA_BRANCH}" >> /app/versions.txt \
|
||||
&& echo "FA_REPO: ${FA_REPO}" >> /app/versions.txt \
|
||||
&& echo "AITER_BRANCH: ${AITER_BRANCH}" >> /app/versions.txt \
|
||||
&& echo "AITER_REPO: ${AITER_REPO}" >> /app/versions.txt \
|
||||
&& echo "MORI_BRANCH: ${MORI_BRANCH}" >> /app/versions.txt \
|
||||
&& echo "MORI_REPO: ${MORI_REPO}" >> /app/versions.txt \
|
||||
&& echo "ROCM_SYSTEMS_REPO: ${ROCM_SYSTEMS_REPO}" >> /app/versions.txt \
|
||||
&& echo "ROCM_RUNTIME_COMMIT: ${ROCM_RUNTIME_COMMIT}" >> /app/versions.txt \
|
||||
# Fingerprint our triton so Dockerfile.rocm can prove it survived the build.
|
||||
&& echo "TRITON_VERSION: $(pip show triton | awk '/^Version:/ {print $2}')" >> /app/versions.txt \
|
||||
&& echo "TRITON_SHA256: $(find $(python3 -c 'import triton, os; print(os.path.dirname(triton.__file__))') \
|
||||
-type f \( -name '*.py' -o -name '*.so' \) | sort | xargs sha256sum | sha256sum | cut -d' ' -f1)" \
|
||||
>> /app/versions.txt \
|
||||
&& echo "PYBIND11_VERSION: $(pip show pybind11 | awk '/^Version:/ {print $2}')" >> /app/versions.txt \
|
||||
&& echo "PYBIND11_INTERNALS_VERSION: $(grep -rhE '^#[[:space:]]*define PYBIND11_INTERNALS_VERSION' \
|
||||
$(python3 -c 'import pybind11; print(pybind11.get_include())') | awk '{print $NF}')" \
|
||||
>> /app/versions.txt \
|
||||
&& echo "PYBIND11_SHA256: $(find $(python3 -c 'import pybind11; print(pybind11.get_include())') \
|
||||
-type f -name '*.h' | sort | xargs sha256sum | sha256sum | cut -d' ' -f1)" \
|
||||
>> /app/versions.txt
|
||||
+41
-41
@@ -280,8 +280,9 @@ COPY --from=fetch_vllm \
|
||||
${COMMON_WORKDIR}/vllm/requirements/rocm.txt \
|
||||
requirements/
|
||||
|
||||
# Use constraints file to protect triton, pybind11, and torch dependencies.
|
||||
RUN --mount=type=cache,id=vllm-rocm-uv,target=/root/.cache/uv \
|
||||
uv pip install --system -r requirements/rocm.txt
|
||||
uv pip install --system -c /etc/rocm-constraints.txt -r requirements/rocm.txt
|
||||
|
||||
# Fetch only the pinned ROCm Triton kernels package. Keeping this before the
|
||||
# vLLM source COPYs makes the download reusable when native sources change.
|
||||
@@ -322,8 +323,9 @@ ENV SETUPTOOLS_SCM_PRETEND_VERSION="0.0.0+rocm.csrc.build"
|
||||
|
||||
RUN --mount=type=bind,source=pyproject.toml,target=${COMMON_WORKDIR}/vllm/pyproject.toml \
|
||||
--mount=type=cache,id=vllm-rocm-ccache,target=/root/.cache/ccache \
|
||||
grep -Fq "set(TRITON_KERNELS_TAG \"${ROCM_TRITON_KERNELS_COMMIT}\")" \
|
||||
{ grep -Fq "set(TRITON_KERNELS_TAG \"${ROCM_TRITON_KERNELS_COMMIT}\")" \
|
||||
cmake/external_projects/triton_kernels.cmake \
|
||||
|| echo "WARNING: Dockerfile.rocm ROCM_TRITON_KERNELS_COMMIT (${ROCM_TRITON_KERNELS_COMMIT}) does not match cmake TRITON_KERNELS_TAG; the Docker build uses the Dockerfile pin, cmake governs only non-Docker builds"; } \
|
||||
&& export CCACHE_BASEDIR="$PWD" \
|
||||
&& echo "=== ccache stats before ROCm native build ===" \
|
||||
&& (ccache --show-stats || true) \
|
||||
@@ -366,7 +368,8 @@ COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/requirements /requirements
|
||||
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/benchmarks /benchmarks
|
||||
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/tests /tests
|
||||
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/examples /examples
|
||||
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/tools /tools
|
||||
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/tools/benchmark_helion_kernels.py /tools/benchmark_helion_kernels.py
|
||||
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/tools/install_torchcodec_rocm.sh /tools/install_torchcodec_rocm.sh
|
||||
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile /docker/Dockerfile
|
||||
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile.cpu /docker/Dockerfile.cpu
|
||||
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm /docker/
|
||||
@@ -386,7 +389,6 @@ ARG NIXL_BRANCH="1683cf3b7f3d11674c03c5e861cea22339876c96"
|
||||
ARG NIXL_REPO="https://github.com/ai-dynamo/nixl.git"
|
||||
ARG UCX_BRANCH="96e58a16039f6d7d213bc967b8069238742c5194"
|
||||
ARG UCX_REPO="https://github.com/openucx/ucx.git"
|
||||
ENV ROCM_PATH=/opt/rocm
|
||||
ENV UCX_HOME=/usr/local/ucx
|
||||
ENV NIXL_HOME=/usr/local/nixl
|
||||
|
||||
@@ -541,7 +543,7 @@ RUN --mount=type=tmpfs,target=/tmp/lmcache-wheel \
|
||||
c_ops="$(find /tmp/lmcache-wheel -type f -name 'c_ops*.so')"; \
|
||||
test "$(find /tmp/lmcache-wheel -type f -name 'c_ops*.so' | wc -l)" -eq 1; \
|
||||
expected="$(printf '%s' "${LMCACHE_ROCM_ARCH}" | tr ';' '\n' | sort -u)"; \
|
||||
actual="$(/opt/rocm/llvm/bin/llvm-objdump --offloading "${c_ops}" \
|
||||
actual="$(llvm-objdump --offloading "${c_ops}" \
|
||||
| grep -oE 'gfx[0-9a-z]+' | sort -u)"; \
|
||||
if [ "${actual}" != "${expected}" ]; then \
|
||||
printf 'LMCache code objects mismatch\nexpected:\n%s\nactual:\n%s\n' \
|
||||
@@ -549,32 +551,10 @@ RUN --mount=type=tmpfs,target=/tmp/lmcache-wheel \
|
||||
exit 1; \
|
||||
fi
|
||||
|
||||
# ROCShmem build stage - split from DeepEP so changing DEEPEP_BRANCH does not
|
||||
# invalidate the slow ROCShmem build.
|
||||
FROM base AS build_rocshmem
|
||||
ARG ROCSHMEM_BRANCH="f0acb0c6"
|
||||
ARG ROCSHMEM_REPO="https://github.com/ROCm/rocm-systems.git"
|
||||
# DeepEP only supports gfx942 and gfx950; build ROCShmem for the same set so
|
||||
# it can be linked against DeepEP without arch mismatches.
|
||||
ARG DEEPEP_ROCM_ARCH="gfx942;gfx950"
|
||||
ENV ROCM_PATH=/opt/rocm
|
||||
ENV ROCSHMEM_DIR=/opt/rocshmem
|
||||
|
||||
RUN --mount=type=cache,target=/root/.cache/ccache \
|
||||
git clone --no-checkout --filter=blob:none ${ROCSHMEM_REPO} \
|
||||
&& cd rocm-systems \
|
||||
&& git sparse-checkout set --cone projects/rocshmem \
|
||||
&& git checkout ${ROCSHMEM_BRANCH} \
|
||||
&& mkdir -p projects/rocshmem/build \
|
||||
&& cd projects/rocshmem/build \
|
||||
&& CC="ccache gcc" CXX="ccache g++" INSTALL_PREFIX=${ROCSHMEM_DIR} \
|
||||
bash ../scripts/build_configs/all_backends \
|
||||
-DROCM_PATH=${ROCM_PATH} \
|
||||
-DGPU_TARGETS="${DEEPEP_ROCM_ARCH}" \
|
||||
-DUSE_EXTERNAL_MPI=OFF
|
||||
|
||||
# DeepEP build stage - depends on ROCShmem, builds the HIP kernel wheel.
|
||||
FROM build_rocshmem AS build_deepep
|
||||
# DeepEP build stage. ROCShmem is provided by TheRock SDK
|
||||
# ROCSHMEM_DIR set to the SDK install path so DeepEP finds the headers, etc.
|
||||
FROM base AS build_deepep
|
||||
ENV ROCSHMEM_DIR=${ROCM_PATH}
|
||||
ARG DEEPEP_BRANCH="0f63d3e9"
|
||||
ARG DEEPEP_REPO="https://github.com/ROCm/DeepEP.git"
|
||||
ARG DEEPEP_NIC="cx7"
|
||||
@@ -693,8 +673,9 @@ RUN if [ "$GIT_REPO_CHECK" != "0" ]; then \
|
||||
|
||||
# Install the packaging helpers before modifying requirements/rocm.txt.
|
||||
RUN --mount=type=cache,target=/root/.cache/uv \
|
||||
grep -Fq "set(TRITON_KERNELS_TAG \"${ROCM_TRITON_KERNELS_COMMIT}\")" \
|
||||
{ grep -Fq "set(TRITON_KERNELS_TAG \"${ROCM_TRITON_KERNELS_COMMIT}\")" \
|
||||
vllm/cmake/external_projects/triton_kernels.cmake \
|
||||
|| echo "WARNING: Dockerfile.rocm ROCM_TRITON_KERNELS_COMMIT (${ROCM_TRITON_KERNELS_COMMIT}) does not match cmake TRITON_KERNELS_TAG; the Docker build uses the Dockerfile pin, cmake governs only non-Docker builds"; } \
|
||||
&& cd vllm \
|
||||
&& uv pip install --system setuptools_scm regex
|
||||
|
||||
@@ -750,7 +731,8 @@ COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/requirements /requir
|
||||
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/benchmarks /benchmarks
|
||||
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/tests /tests
|
||||
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/examples /examples
|
||||
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/tools /tools
|
||||
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/tools/benchmark_helion_kernels.py /tools/benchmark_helion_kernels.py
|
||||
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/tools/install_torchcodec_rocm.sh /tools/install_torchcodec_rocm.sh
|
||||
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile /docker/Dockerfile
|
||||
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile.cpu /docker/Dockerfile.cpu
|
||||
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm /docker/
|
||||
@@ -795,9 +777,6 @@ RUN --mount=type=bind,from=build_nixl,src=/app/install,target=/nixl_install \
|
||||
--mount=type=bind,from=build_deepep,src=/app/deep_install,target=/deep_install \
|
||||
uv pip install --system /nixl_install/*.whl /deep_install/*.whl
|
||||
|
||||
# Copy ROCShmem runtime libraries.
|
||||
COPY --from=build_rocshmem /opt/rocshmem /opt/rocshmem
|
||||
|
||||
# RDMA userspace libraries plus FFmpeg dev libs needed by torchcodec.
|
||||
RUN timeout --kill-after=30s 5m apt-get update -q -y --error-on=any \
|
||||
-o Acquire::Retries=3 \
|
||||
@@ -830,7 +809,8 @@ RUN --mount=type=cache,target=/root/.cache/uv \
|
||||
# Pre-install shared ROCm runtime dependencies.
|
||||
COPY requirements/common.txt requirements/rocm.txt /tmp/ci-base-requirements/
|
||||
RUN --mount=type=cache,target=/root/.cache/uv \
|
||||
uv pip install --system -r /tmp/ci-base-requirements/rocm.txt \
|
||||
uv pip install --system -c /etc/rocm-constraints.txt \
|
||||
-r /tmp/ci-base-requirements/rocm.txt \
|
||||
&& rm -rf /tmp/ci-base-requirements
|
||||
|
||||
# Enable fast and less brittle model downloads in tests.
|
||||
@@ -908,8 +888,6 @@ RUN mkdir src && mv vllm src/vllm
|
||||
# Catch GPU<->CPU syncs in execute_model/sample_tokens during tests.
|
||||
ENV VLLM_GPU_SYNC_CHECK=error
|
||||
|
||||
# Run the structural smoke test inside BuildKit, where the test image layers
|
||||
# are already available. Export only a build-scoped success marker.
|
||||
FROM test AS test_smoke
|
||||
ARG ROCM_SMOKE_ID
|
||||
RUN --network=none \
|
||||
@@ -956,15 +934,37 @@ RUN --mount=type=bind,from=export_vllm,src=/,target=/install \
|
||||
--mount=type=cache,target=/root/.cache/uv \
|
||||
cd /install \
|
||||
&& echo "opentelemetry-api<=1.40.0" > /tmp/otel-constraints.txt \
|
||||
&& uv pip install --system -c /tmp/otel-constraints.txt -r requirements/rocm.txt \
|
||||
&& uv pip install --system -c /etc/rocm-constraints.txt -c /tmp/otel-constraints.txt -r requirements/rocm.txt \
|
||||
&& pip uninstall -y vllm \
|
||||
&& uv pip install --system -c /tmp/otel-constraints.txt *.whl \
|
||||
&& uv pip install --system -c /etc/rocm-constraints.txt -c /tmp/otel-constraints.txt *.whl \
|
||||
&& rm /tmp/otel-constraints.txt
|
||||
|
||||
# Install NIXL ROCm wheel
|
||||
RUN --mount=type=bind,from=build_nixl,src=/app/install,target=/nixl_install \
|
||||
uv pip install --system /nixl_install/*.whl
|
||||
|
||||
# Recompute the base's triton fingerprint to detect resolver interference.
|
||||
RUN EXPECTED=$(grep '^TRITON_SHA256: ' /app/versions.txt | cut -d' ' -f2) \
|
||||
&& ACTUAL=$(find $(python3 -c 'import triton, os; print(os.path.dirname(triton.__file__))') \
|
||||
-type f \( -name '*.py' -o -name '*.so' \) | sort | xargs sha256sum | sha256sum | cut -d' ' -f1) \
|
||||
&& echo "triton sha256 expected=${EXPECTED} actual=${ACTUAL}" \
|
||||
&& test -n "${EXPECTED}" \
|
||||
&& test "${EXPECTED}" = "${ACTUAL}" \
|
||||
&& python3 -c "import torch; assert torch.version.hip, f'non-ROCm torch: {torch.__version__}'; print(f'torch={torch.__version__}')"
|
||||
|
||||
# Recompute the base's pybind11 fingerprint to detect resolver interference.
|
||||
RUN EXPECTED=$(grep '^PYBIND11_SHA256: ' /app/versions.txt | cut -d' ' -f2) \
|
||||
&& EXPECTED_ABI=$(grep '^PYBIND11_INTERNALS_VERSION: ' /app/versions.txt | cut -d' ' -f2) \
|
||||
&& INC=$(python3 -c 'import pybind11; print(pybind11.get_include())') \
|
||||
&& ACTUAL=$(find "${INC}" -type f -name '*.h' | sort | xargs sha256sum | sha256sum | cut -d' ' -f1) \
|
||||
&& ACTUAL_ABI=$(grep -rhE '^#[[:space:]]*define PYBIND11_INTERNALS_VERSION' "${INC}" | awk '{print $NF}') \
|
||||
&& echo "pybind11 $(pip show pybind11 | awk '/^Version:/ {print $2}')" \
|
||||
"sha256 expected=${EXPECTED} actual=${ACTUAL}" \
|
||||
"internals expected=${EXPECTED_ABI} actual=${ACTUAL_ABI}" \
|
||||
&& test -n "${EXPECTED}" \
|
||||
&& test "${EXPECTED}" = "${ACTUAL}" \
|
||||
&& test "${EXPECTED_ABI}" = "${ACTUAL_ABI}"
|
||||
|
||||
ARG COMMON_WORKDIR
|
||||
ARG BASE_IMAGE
|
||||
ARG NIC_BACKEND
|
||||
|
||||
@@ -3,7 +3,7 @@ ARG REMOTE_VLLM="0"
|
||||
ARG VLLM_REPO="https://github.com/vllm-project/vllm.git"
|
||||
ARG VLLM_BRANCH="main"
|
||||
ARG COMMON_WORKDIR=/app
|
||||
ARG BASE_IMAGE=rocm/vllm-dev:base
|
||||
ARG BASE_IMAGE=vllm/vllm-openai-rocm:base-nightly-rocm72
|
||||
ARG CI_BASE_IMAGE=rocm/vllm-dev:ci_base
|
||||
ARG ROCM_TRITON_KERNELS_COMMIT=669b31acc1dd1b3fd93286afd5db67f65d9f7557
|
||||
ARG INSTALL_LMCACHE=false
|
||||
@@ -46,7 +46,7 @@ RUN --mount=type=cache,target=/root/.cache/pip \
|
||||
# Note: mold is installed but not set as the system default linker because
|
||||
# some packages use JIT compilation at runtime with flags mold does not support.
|
||||
# Build stages opt in via LDFLAGS="-fuse-ld=mold".
|
||||
# Remove sccache only if not using sccache (it exists in base image from Dockerfile.rocm_base)
|
||||
# Remove sccache only if not using sccache (it exists in base image from Dockerfile.rocm_72_base)
|
||||
ARG USE_SCCACHE
|
||||
RUN if [ "$USE_SCCACHE" != "1" ]; then \
|
||||
apt-get purge -y sccache || true; \
|
||||
@@ -280,9 +280,8 @@ COPY --from=fetch_vllm \
|
||||
${COMMON_WORKDIR}/vllm/requirements/rocm.txt \
|
||||
requirements/
|
||||
|
||||
# Use constraints file to protect triton, pybind11, and torch dependencies.
|
||||
RUN --mount=type=cache,id=vllm-rocm-uv,target=/root/.cache/uv \
|
||||
uv pip install --system -c /etc/rocm-constraints.txt -r requirements/rocm.txt
|
||||
uv pip install --system -r requirements/rocm.txt
|
||||
|
||||
# Fetch only the pinned ROCm Triton kernels package. Keeping this before the
|
||||
# vLLM source COPYs makes the download reusable when native sources change.
|
||||
@@ -323,9 +322,8 @@ ENV SETUPTOOLS_SCM_PRETEND_VERSION="0.0.0+rocm.csrc.build"
|
||||
|
||||
RUN --mount=type=bind,source=pyproject.toml,target=${COMMON_WORKDIR}/vllm/pyproject.toml \
|
||||
--mount=type=cache,id=vllm-rocm-ccache,target=/root/.cache/ccache \
|
||||
{ grep -Fq "set(TRITON_KERNELS_TAG \"${ROCM_TRITON_KERNELS_COMMIT}\")" \
|
||||
grep -Fq "set(TRITON_KERNELS_TAG \"${ROCM_TRITON_KERNELS_COMMIT}\")" \
|
||||
cmake/external_projects/triton_kernels.cmake \
|
||||
|| echo "WARNING: Dockerfile.rock ROCM_TRITON_KERNELS_COMMIT (${ROCM_TRITON_KERNELS_COMMIT}) does not match cmake TRITON_KERNELS_TAG; the Docker build uses the Dockerfile pin, cmake governs only non-Docker builds"; } \
|
||||
&& export CCACHE_BASEDIR="$PWD" \
|
||||
&& echo "=== ccache stats before ROCm native build ===" \
|
||||
&& (ccache --show-stats || true) \
|
||||
@@ -368,12 +366,11 @@ COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/requirements /requirements
|
||||
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/benchmarks /benchmarks
|
||||
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/tests /tests
|
||||
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/examples /examples
|
||||
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/tools/benchmark_helion_kernels.py /tools/benchmark_helion_kernels.py
|
||||
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/tools/install_torchcodec_rocm.sh /tools/install_torchcodec_rocm.sh
|
||||
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/tools /tools
|
||||
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile /docker/Dockerfile
|
||||
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile.cpu /docker/Dockerfile.cpu
|
||||
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm /docker/
|
||||
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm_base /docker/Dockerfile.rocm_base
|
||||
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm_72 /docker/
|
||||
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm_72_base /docker/
|
||||
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/ci-rocm.hcl /docker/ci-rocm.hcl
|
||||
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/docker-bake.hcl /docker/docker-bake.hcl
|
||||
COPY --from=build_vllm ${COMMON_WORKDIR}/vllm/docker/docker-bake-rocm.hcl /docker/docker-bake-rocm.hcl
|
||||
@@ -389,6 +386,7 @@ ARG NIXL_BRANCH="1683cf3b7f3d11674c03c5e861cea22339876c96"
|
||||
ARG NIXL_REPO="https://github.com/ai-dynamo/nixl.git"
|
||||
ARG UCX_BRANCH="96e58a16039f6d7d213bc967b8069238742c5194"
|
||||
ARG UCX_REPO="https://github.com/openucx/ucx.git"
|
||||
ENV ROCM_PATH=/opt/rocm
|
||||
ENV UCX_HOME=/usr/local/ucx
|
||||
ENV NIXL_HOME=/usr/local/nixl
|
||||
|
||||
@@ -543,7 +541,7 @@ RUN --mount=type=tmpfs,target=/tmp/lmcache-wheel \
|
||||
c_ops="$(find /tmp/lmcache-wheel -type f -name 'c_ops*.so')"; \
|
||||
test "$(find /tmp/lmcache-wheel -type f -name 'c_ops*.so' | wc -l)" -eq 1; \
|
||||
expected="$(printf '%s' "${LMCACHE_ROCM_ARCH}" | tr ';' '\n' | sort -u)"; \
|
||||
actual="$(llvm-objdump --offloading "${c_ops}" \
|
||||
actual="$(/opt/rocm/llvm/bin/llvm-objdump --offloading "${c_ops}" \
|
||||
| grep -oE 'gfx[0-9a-z]+' | sort -u)"; \
|
||||
if [ "${actual}" != "${expected}" ]; then \
|
||||
printf 'LMCache code objects mismatch\nexpected:\n%s\nactual:\n%s\n' \
|
||||
@@ -551,10 +549,32 @@ RUN --mount=type=tmpfs,target=/tmp/lmcache-wheel \
|
||||
exit 1; \
|
||||
fi
|
||||
|
||||
# DeepEP build stage. ROCShmem is provided by TheRock SDK
|
||||
# ROCSHMEM_DIR set to the SDK install path so DeepEP finds the headers, etc.
|
||||
FROM base AS build_deepep
|
||||
ENV ROCSHMEM_DIR=${ROCM_PATH}
|
||||
# ROCShmem build stage - split from DeepEP so changing DEEPEP_BRANCH does not
|
||||
# invalidate the slow ROCShmem build.
|
||||
FROM base AS build_rocshmem
|
||||
ARG ROCSHMEM_BRANCH="f0acb0c6"
|
||||
ARG ROCSHMEM_REPO="https://github.com/ROCm/rocm-systems.git"
|
||||
# DeepEP only supports gfx942 and gfx950; build ROCShmem for the same set so
|
||||
# it can be linked against DeepEP without arch mismatches.
|
||||
ARG DEEPEP_ROCM_ARCH="gfx942;gfx950"
|
||||
ENV ROCM_PATH=/opt/rocm
|
||||
ENV ROCSHMEM_DIR=/opt/rocshmem
|
||||
|
||||
RUN --mount=type=cache,target=/root/.cache/ccache \
|
||||
git clone --no-checkout --filter=blob:none ${ROCSHMEM_REPO} \
|
||||
&& cd rocm-systems \
|
||||
&& git sparse-checkout set --cone projects/rocshmem \
|
||||
&& git checkout ${ROCSHMEM_BRANCH} \
|
||||
&& mkdir -p projects/rocshmem/build \
|
||||
&& cd projects/rocshmem/build \
|
||||
&& CC="ccache gcc" CXX="ccache g++" INSTALL_PREFIX=${ROCSHMEM_DIR} \
|
||||
bash ../scripts/build_configs/all_backends \
|
||||
-DROCM_PATH=${ROCM_PATH} \
|
||||
-DGPU_TARGETS="${DEEPEP_ROCM_ARCH}" \
|
||||
-DUSE_EXTERNAL_MPI=OFF
|
||||
|
||||
# DeepEP build stage - depends on ROCShmem, builds the HIP kernel wheel.
|
||||
FROM build_rocshmem AS build_deepep
|
||||
ARG DEEPEP_BRANCH="0f63d3e9"
|
||||
ARG DEEPEP_REPO="https://github.com/ROCm/DeepEP.git"
|
||||
ARG DEEPEP_NIC="cx7"
|
||||
@@ -571,8 +591,8 @@ RUN --mount=type=cache,target=/root/.cache/ccache \
|
||||
&& git checkout ${DEEPEP_BRANCH} \
|
||||
&& LDFLAGS="-fuse-ld=mold" MAX_JOBS="${MAX_JOBS:-$(nproc)}" python3 setup.py --variant rocm --rocm-explicit-ctx --nic ${DEEPEP_NIC} --rocm-gfx942-fp8-fnuz-max 224 bdist_wheel --dist-dir=/app/deep_install
|
||||
|
||||
# MoRI runtime dependencies live in Dockerfile.rocm so NIC backend changes do
|
||||
# not force users to rebuild the long-lived Dockerfile.rocm_base image.
|
||||
# MoRI runtime dependencies live in Dockerfile.rocm_72 so NIC backend changes do
|
||||
# not force users to rebuild the long-lived Dockerfile.rocm_72_base image.
|
||||
FROM base AS mori_base
|
||||
ARG NIC_BACKEND
|
||||
ARG AINIC_VERSION
|
||||
@@ -673,9 +693,8 @@ RUN if [ "$GIT_REPO_CHECK" != "0" ]; then \
|
||||
|
||||
# Install the packaging helpers before modifying requirements/rocm.txt.
|
||||
RUN --mount=type=cache,target=/root/.cache/uv \
|
||||
{ grep -Fq "set(TRITON_KERNELS_TAG \"${ROCM_TRITON_KERNELS_COMMIT}\")" \
|
||||
grep -Fq "set(TRITON_KERNELS_TAG \"${ROCM_TRITON_KERNELS_COMMIT}\")" \
|
||||
vllm/cmake/external_projects/triton_kernels.cmake \
|
||||
|| echo "WARNING: Dockerfile.rock ROCM_TRITON_KERNELS_COMMIT (${ROCM_TRITON_KERNELS_COMMIT}) does not match cmake TRITON_KERNELS_TAG; the Docker build uses the Dockerfile pin, cmake governs only non-Docker builds"; } \
|
||||
&& cd vllm \
|
||||
&& uv pip install --system setuptools_scm regex
|
||||
|
||||
@@ -731,12 +750,11 @@ COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/requirements /requir
|
||||
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/benchmarks /benchmarks
|
||||
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/tests /tests
|
||||
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/examples /examples
|
||||
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/tools/benchmark_helion_kernels.py /tools/benchmark_helion_kernels.py
|
||||
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/tools/install_torchcodec_rocm.sh /tools/install_torchcodec_rocm.sh
|
||||
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/tools /tools
|
||||
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile /docker/Dockerfile
|
||||
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile.cpu /docker/Dockerfile.cpu
|
||||
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm /docker/
|
||||
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm_base /docker/Dockerfile.rocm_base
|
||||
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm_72 /docker/
|
||||
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/Dockerfile.rocm_72_base /docker/
|
||||
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/ci-rocm.hcl /docker/ci-rocm.hcl
|
||||
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/docker-bake.hcl /docker/docker-bake.hcl
|
||||
COPY --from=build_vllm_wheel_release ${COMMON_WORKDIR}/vllm/docker/docker-bake-rocm.hcl /docker/docker-bake-rocm.hcl
|
||||
@@ -777,6 +795,9 @@ RUN --mount=type=bind,from=build_nixl,src=/app/install,target=/nixl_install \
|
||||
--mount=type=bind,from=build_deepep,src=/app/deep_install,target=/deep_install \
|
||||
uv pip install --system /nixl_install/*.whl /deep_install/*.whl
|
||||
|
||||
# Copy ROCShmem runtime libraries.
|
||||
COPY --from=build_rocshmem /opt/rocshmem /opt/rocshmem
|
||||
|
||||
# RDMA userspace libraries plus FFmpeg dev libs needed by torchcodec.
|
||||
RUN timeout --kill-after=30s 5m apt-get update -q -y --error-on=any \
|
||||
-o Acquire::Retries=3 \
|
||||
@@ -809,8 +830,7 @@ RUN --mount=type=cache,target=/root/.cache/uv \
|
||||
# Pre-install shared ROCm runtime dependencies.
|
||||
COPY requirements/common.txt requirements/rocm.txt /tmp/ci-base-requirements/
|
||||
RUN --mount=type=cache,target=/root/.cache/uv \
|
||||
uv pip install --system -c /etc/rocm-constraints.txt \
|
||||
-r /tmp/ci-base-requirements/rocm.txt \
|
||||
uv pip install --system -r /tmp/ci-base-requirements/rocm.txt \
|
||||
&& rm -rf /tmp/ci-base-requirements
|
||||
|
||||
# Enable fast and less brittle model downloads in tests.
|
||||
@@ -888,6 +908,8 @@ RUN mkdir src && mv vllm src/vllm
|
||||
# Catch GPU<->CPU syncs in execute_model/sample_tokens during tests.
|
||||
ENV VLLM_GPU_SYNC_CHECK=error
|
||||
|
||||
# Run the structural smoke test inside BuildKit, where the test image layers
|
||||
# are already available. Export only a build-scoped success marker.
|
||||
FROM test AS test_smoke
|
||||
ARG ROCM_SMOKE_ID
|
||||
RUN --network=none \
|
||||
@@ -934,37 +956,15 @@ RUN --mount=type=bind,from=export_vllm,src=/,target=/install \
|
||||
--mount=type=cache,target=/root/.cache/uv \
|
||||
cd /install \
|
||||
&& echo "opentelemetry-api<=1.40.0" > /tmp/otel-constraints.txt \
|
||||
&& uv pip install --system -c /etc/rocm-constraints.txt -c /tmp/otel-constraints.txt -r requirements/rocm.txt \
|
||||
&& uv pip install --system -c /tmp/otel-constraints.txt -r requirements/rocm.txt \
|
||||
&& pip uninstall -y vllm \
|
||||
&& uv pip install --system -c /etc/rocm-constraints.txt -c /tmp/otel-constraints.txt *.whl \
|
||||
&& uv pip install --system -c /tmp/otel-constraints.txt *.whl \
|
||||
&& rm /tmp/otel-constraints.txt
|
||||
|
||||
# Install NIXL ROCm wheel
|
||||
RUN --mount=type=bind,from=build_nixl,src=/app/install,target=/nixl_install \
|
||||
uv pip install --system /nixl_install/*.whl
|
||||
|
||||
# Recompute the base's triton fingerprint to detect resolver interference.
|
||||
RUN EXPECTED=$(grep '^TRITON_SHA256: ' /app/versions.txt | cut -d' ' -f2) \
|
||||
&& ACTUAL=$(find $(python3 -c 'import triton, os; print(os.path.dirname(triton.__file__))') \
|
||||
-type f \( -name '*.py' -o -name '*.so' \) | sort | xargs sha256sum | sha256sum | cut -d' ' -f1) \
|
||||
&& echo "triton sha256 expected=${EXPECTED} actual=${ACTUAL}" \
|
||||
&& test -n "${EXPECTED}" \
|
||||
&& test "${EXPECTED}" = "${ACTUAL}" \
|
||||
&& python3 -c "import torch; assert torch.version.hip, f'non-ROCm torch: {torch.__version__}'; print(f'torch={torch.__version__}')"
|
||||
|
||||
# Recompute the base's pybind11 fingerprint to detect resolver interference.
|
||||
RUN EXPECTED=$(grep '^PYBIND11_SHA256: ' /app/versions.txt | cut -d' ' -f2) \
|
||||
&& EXPECTED_ABI=$(grep '^PYBIND11_INTERNALS_VERSION: ' /app/versions.txt | cut -d' ' -f2) \
|
||||
&& INC=$(python3 -c 'import pybind11; print(pybind11.get_include())') \
|
||||
&& ACTUAL=$(find "${INC}" -type f -name '*.h' | sort | xargs sha256sum | sha256sum | cut -d' ' -f1) \
|
||||
&& ACTUAL_ABI=$(grep -rhE '^#[[:space:]]*define PYBIND11_INTERNALS_VERSION' "${INC}" | awk '{print $NF}') \
|
||||
&& echo "pybind11 $(pip show pybind11 | awk '/^Version:/ {print $2}')" \
|
||||
"sha256 expected=${EXPECTED} actual=${ACTUAL}" \
|
||||
"internals expected=${EXPECTED_ABI} actual=${ACTUAL_ABI}" \
|
||||
&& test -n "${EXPECTED}" \
|
||||
&& test "${EXPECTED}" = "${ACTUAL}" \
|
||||
&& test "${EXPECTED_ABI}" = "${ACTUAL_ABI}"
|
||||
|
||||
ARG COMMON_WORKDIR
|
||||
ARG BASE_IMAGE
|
||||
ARG NIC_BACKEND
|
||||
@@ -0,0 +1,479 @@
|
||||
ARG BASE_IMAGE=rocm/dev-ubuntu-22.04:7.2.3-complete
|
||||
ARG TRITON_BRANCH="669b31a" # release/internal/3.8.x as of 09/17
|
||||
ARG TRITON_REPO="https://github.com/ROCm/triton.git"
|
||||
ARG PYTORCH_BRANCH="733fca1" # release/2.13 as of 09/17
|
||||
ARG PYTORCH_REPO="https://github.com/ROCm/pytorch.git"
|
||||
ARG PYTORCH_VISION_BRANCH="v0.28.0"
|
||||
ARG PYTORCH_VISION_REPO="https://github.com/pytorch/vision.git"
|
||||
ARG PYTORCH_AUDIO_BRANCH="v2.11.0"
|
||||
ARG PYTORCH_AUDIO_REPO="https://github.com/pytorch/audio.git"
|
||||
ARG FA_BRANCH="0e60e394"
|
||||
ARG FA_REPO="https://github.com/Dao-AILab/flash-attention.git"
|
||||
ARG AITER_BRANCH="v0.1.25"
|
||||
ARG AITER_REPO="https://github.com/ROCm/aiter.git"
|
||||
ARG MORI_BRANCH="v1.2.3.post1"
|
||||
ARG MORI_REPO="https://github.com/ROCm/mori.git"
|
||||
ARG ROCM_SYSTEMS_REPO="https://github.com/ROCm/rocm-systems.git"
|
||||
ARG ROCPROFILER_SDK_COMMIT="2b22ab0195cc1461cd9abf3b969e9dd7c10af350"
|
||||
ARG ROCM_RUNTIME_COMMIT="af70edc91f1dc5d228fd4c076df99e65e49c4521"
|
||||
|
||||
# Sccache configuration (only used in release pipeline)
|
||||
ARG USE_SCCACHE
|
||||
ARG SCCACHE_DOWNLOAD_URL
|
||||
ARG SCCACHE_ENDPOINT
|
||||
ARG SCCACHE_BUCKET_NAME=vllm-build-sccache
|
||||
ARG SCCACHE_REGION_NAME=us-west-2
|
||||
ARG SCCACHE_S3_NO_CREDENTIALS=0
|
||||
|
||||
|
||||
FROM ${BASE_IMAGE} AS base
|
||||
|
||||
ENV PATH=/opt/rocm/llvm/bin:/opt/rocm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
|
||||
ENV ROCM_PATH=/opt/rocm
|
||||
ENV LD_LIBRARY_PATH=/opt/rocm/lib:/usr/local/lib:
|
||||
|
||||
ARG PYTORCH_ROCM_ARCH=gfx90a;gfx942;gfx950;gfx1100;gfx1101;gfx1200;gfx1201;gfx1150;gfx1151
|
||||
ENV PYTORCH_ROCM_ARCH=${PYTORCH_ROCM_ARCH}
|
||||
ENV AITER_ROCM_ARCH=gfx942;gfx950
|
||||
# Note: Do not set MORI_GPU_ARCHS here, it is automatically inferred at runtime
|
||||
|
||||
# Required for RCCL in ROCm7.1
|
||||
ENV HSA_NO_SCRATCH_RECLAIM=1
|
||||
|
||||
ARG PYTHON_VERSION=3.12
|
||||
ENV PYTHON_VERSION=${PYTHON_VERSION}
|
||||
|
||||
RUN mkdir -p /app
|
||||
WORKDIR /app
|
||||
ENV DEBIAN_FRONTEND=noninteractive
|
||||
|
||||
# Install Python and other dependencies (plus gcc-13, selected later in build_pytorch).
|
||||
RUN apt-get update -y \
|
||||
&& apt-get install -y software-properties-common git curl sudo vim less libgfortran5 libopenmpi-dev libpci-dev liblzma-dev pkg-config \
|
||||
&& for i in 1 2 3; do \
|
||||
add-apt-repository -y ppa:deadsnakes/ppa && add-apt-repository -y ppa:ubuntu-toolchain-r/test && break || \
|
||||
{ echo "Attempt $i failed, retrying in 5s..."; sleep 5; }; \
|
||||
done \
|
||||
&& apt-get update -y \
|
||||
&& apt-get install -y python${PYTHON_VERSION} python${PYTHON_VERSION}-dev python${PYTHON_VERSION}-venv \
|
||||
python${PYTHON_VERSION}-lib2to3 python-is-python3 gcc-13 g++-13 \
|
||||
&& update-alternatives --install /usr/bin/python3 python3 /usr/bin/python${PYTHON_VERSION} 1 \
|
||||
&& update-alternatives --set python3 /usr/bin/python${PYTHON_VERSION} \
|
||||
&& ln -sf /usr/bin/python${PYTHON_VERSION}-config /usr/bin/python3-config \
|
||||
&& curl -sS https://bootstrap.pypa.io/get-pip.py | python${PYTHON_VERSION} \
|
||||
&& python3 --version && python3 -m pip --version
|
||||
|
||||
RUN pip install -U packaging 'cmake<4' ninja wheel 'setuptools<80' 'pybind11<3.1.0' Cython
|
||||
RUN apt-get update && apt-get install -y libjpeg-dev libsox-dev libsox-fmt-all sox && rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install sccache if USE_SCCACHE is enabled (for release builds)
|
||||
ARG USE_SCCACHE
|
||||
ARG SCCACHE_DOWNLOAD_URL
|
||||
ARG SCCACHE_ENDPOINT
|
||||
ARG SCCACHE_BUCKET_NAME
|
||||
ARG SCCACHE_REGION_NAME
|
||||
ARG SCCACHE_S3_NO_CREDENTIALS
|
||||
RUN if [ "$USE_SCCACHE" = "1" ]; then \
|
||||
echo "Installing sccache..." \
|
||||
&& SCCACHE_ARCH="x86_64" \
|
||||
&& SCCACHE_VERSION="v0.8.1" \
|
||||
&& SCCACHE_DL_URL="${SCCACHE_DOWNLOAD_URL:-https://github.com/mozilla/sccache/releases/download/${SCCACHE_VERSION}/sccache-${SCCACHE_VERSION}-${SCCACHE_ARCH}-unknown-linux-musl.tar.gz}" \
|
||||
&& curl -L -o /tmp/sccache.tar.gz ${SCCACHE_DL_URL} \
|
||||
&& tar -xzf /tmp/sccache.tar.gz -C /tmp \
|
||||
&& mv /tmp/sccache-${SCCACHE_VERSION}-${SCCACHE_ARCH}-unknown-linux-musl/sccache /usr/bin/sccache \
|
||||
&& chmod +x /usr/bin/sccache \
|
||||
&& rm -rf /tmp/sccache.tar.gz /tmp/sccache-${SCCACHE_VERSION}-${SCCACHE_ARCH}-unknown-linux-musl \
|
||||
&& sccache --version; \
|
||||
fi
|
||||
|
||||
# Setup sccache for HIP compilation via HIP_CLANG_PATH
|
||||
# This creates wrapper scripts in a separate directory and points HIP to use them
|
||||
# This avoids modifying the original ROCm binaries which can break detection
|
||||
# NOTE: HIP_CLANG_PATH is NOT set as ENV to avoid affecting downstream images (Dockerfile.rocm_72)
|
||||
# Instead, each build stage should export HIP_CLANG_PATH=/opt/sccache-wrappers if USE_SCCACHE=1
|
||||
RUN if [ "$USE_SCCACHE" = "1" ]; then \
|
||||
echo "Setting up sccache wrappers for HIP compilation..." \
|
||||
&& mkdir -p /opt/sccache-wrappers \
|
||||
&& printf '#!/bin/bash\nexec sccache /opt/rocm/lib/llvm/bin/clang++ "$@"\n' > /opt/sccache-wrappers/clang++ \
|
||||
&& chmod +x /opt/sccache-wrappers/clang++ \
|
||||
&& printf '#!/bin/bash\nexec sccache /opt/rocm/lib/llvm/bin/clang "$@"\n' > /opt/sccache-wrappers/clang \
|
||||
&& chmod +x /opt/sccache-wrappers/clang \
|
||||
&& echo "sccache wrappers created in /opt/sccache-wrappers"; \
|
||||
fi
|
||||
|
||||
# Set sccache environment variables only when USE_SCCACHE=1
|
||||
# This prevents S3 config from leaking into images when sccache is not used
|
||||
ARG USE_SCCACHE
|
||||
ENV SCCACHE_BUCKET=${USE_SCCACHE:+${SCCACHE_BUCKET_NAME}}
|
||||
ENV SCCACHE_REGION=${USE_SCCACHE:+${SCCACHE_REGION_NAME}}
|
||||
ENV SCCACHE_S3_NO_CREDENTIALS=${USE_SCCACHE:+${SCCACHE_S3_NO_CREDENTIALS}}
|
||||
ENV SCCACHE_IDLE_TIMEOUT=${USE_SCCACHE:+0}
|
||||
|
||||
|
||||
###
|
||||
### Triton Build
|
||||
###
|
||||
FROM base AS build_triton
|
||||
ARG TRITON_BRANCH
|
||||
ARG TRITON_REPO
|
||||
RUN git clone ${TRITON_REPO}
|
||||
RUN cd triton \
|
||||
&& git checkout ${TRITON_BRANCH} \
|
||||
&& git config --global user.email "you@example.com" && git config --global user.name "Your Name" \
|
||||
&& if [ ! -f setup.py ]; then cd python; fi \
|
||||
&& MAX_JOBS=16 python3 setup.py bdist_wheel --dist-dir=dist \
|
||||
&& mkdir -p /app/install && cp dist/*.whl /app/install
|
||||
|
||||
|
||||
###
|
||||
### AMD SMI Build
|
||||
###
|
||||
FROM base AS build_amdsmi
|
||||
RUN cd /opt/rocm/share/amd_smi \
|
||||
&& pip wheel . --wheel-dir=dist
|
||||
RUN mkdir -p /app/install && cp /opt/rocm/share/amd_smi/dist/*.whl /app/install
|
||||
|
||||
|
||||
###
|
||||
### Pytorch build
|
||||
###
|
||||
FROM base AS build_pytorch
|
||||
ARG PYTORCH_BRANCH
|
||||
ARG PYTORCH_REPO
|
||||
ARG USE_SCCACHE
|
||||
|
||||
RUN apt-get update && apt-get install -y pkg-config liblzma-dev
|
||||
# torch 2.12 emits float->__bf16 libcalls (__truncsfbf2) absent from gcc-11 libgcc; build with gcc-13.
|
||||
RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-13 130 --slave /usr/bin/g++ g++ /usr/bin/g++-13
|
||||
RUN git clone ${PYTORCH_REPO} pytorch
|
||||
RUN cd pytorch && git checkout ${PYTORCH_BRANCH}
|
||||
RUN cd pytorch \
|
||||
&& pip install -r requirements.txt && git submodule update --init --recursive
|
||||
RUN cd pytorch && python3 tools/amd_build/build_amd.py \
|
||||
&& if [ "$USE_SCCACHE" = "1" ]; then \
|
||||
export HIP_CLANG_PATH=/opt/sccache-wrappers \
|
||||
&& export CMAKE_C_COMPILER_LAUNCHER=sccache \
|
||||
&& export CMAKE_CXX_COMPILER_LAUNCHER=sccache \
|
||||
&& sccache --show-stats; \
|
||||
fi \
|
||||
&& CMAKE_PREFIX_PATH=$(python3 -c 'import sys; print(sys.prefix)') python3 setup.py bdist_wheel --dist-dir=dist \
|
||||
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi
|
||||
RUN mkdir -p /app/install && cp /app/pytorch/dist/*.whl /app/install
|
||||
|
||||
|
||||
###
|
||||
### Shared PyTorch runtime for dependent builds
|
||||
###
|
||||
FROM base AS build_pytorch_runtime
|
||||
RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-13 130 --slave /usr/bin/g++ g++ /usr/bin/g++-13
|
||||
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install,ro \
|
||||
pip install /install/*.whl
|
||||
|
||||
|
||||
###
|
||||
### TorchVision build
|
||||
###
|
||||
FROM build_pytorch_runtime AS build_torchvision
|
||||
ARG PYTORCH_VISION_BRANCH
|
||||
ARG PYTORCH_VISION_REPO
|
||||
ARG USE_SCCACHE
|
||||
|
||||
RUN git clone ${PYTORCH_VISION_REPO} vision
|
||||
RUN cd vision && git checkout ${PYTORCH_VISION_BRANCH} \
|
||||
&& if [ "$USE_SCCACHE" = "1" ]; then \
|
||||
export HIP_CLANG_PATH=/opt/sccache-wrappers \
|
||||
&& export CMAKE_C_COMPILER_LAUNCHER=sccache \
|
||||
&& export CMAKE_CXX_COMPILER_LAUNCHER=sccache; \
|
||||
fi \
|
||||
&& python3 setup.py bdist_wheel --dist-dir=dist \
|
||||
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi
|
||||
RUN mkdir -p /app/install && cp /app/vision/dist/*.whl /app/install
|
||||
|
||||
|
||||
###
|
||||
### TorchAudio build
|
||||
###
|
||||
FROM build_pytorch_runtime AS build_torchaudio
|
||||
ARG PYTORCH_AUDIO_BRANCH
|
||||
ARG PYTORCH_AUDIO_REPO
|
||||
ARG USE_SCCACHE
|
||||
|
||||
RUN git clone ${PYTORCH_AUDIO_REPO} audio
|
||||
RUN cd audio && git checkout ${PYTORCH_AUDIO_BRANCH} \
|
||||
&& git submodule update --init --recursive
|
||||
RUN cd audio && pip install -r requirements.txt
|
||||
RUN cd audio \
|
||||
&& if [ "$USE_SCCACHE" = "1" ]; then \
|
||||
export HIP_CLANG_PATH=/opt/sccache-wrappers \
|
||||
&& export CMAKE_C_COMPILER_LAUNCHER=sccache \
|
||||
&& export CMAKE_CXX_COMPILER_LAUNCHER=sccache; \
|
||||
fi \
|
||||
&& python3 setup.py bdist_wheel --dist-dir=dist \
|
||||
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi
|
||||
RUN mkdir -p /app/install && cp /app/audio/dist/*.whl /app/install
|
||||
|
||||
|
||||
###
|
||||
### MORI Build
|
||||
###
|
||||
FROM base AS build_mori
|
||||
ARG MORI_BRANCH
|
||||
ARG MORI_REPO
|
||||
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
|
||||
pip install /install/*.whl
|
||||
RUN git clone ${MORI_REPO}
|
||||
RUN cd mori \
|
||||
&& git checkout ${MORI_BRANCH} \
|
||||
&& git submodule update --init --recursive \
|
||||
&& python3 setup.py bdist_wheel --dist-dir=dist && ls /app/mori/dist/*.whl
|
||||
RUN mkdir -p /app/install && cp /app/mori/dist/*.whl /app/install
|
||||
|
||||
|
||||
###
|
||||
### FlashAttention Build
|
||||
###
|
||||
FROM base AS build_fa
|
||||
ARG FA_BRANCH
|
||||
ARG FA_REPO
|
||||
ARG USE_SCCACHE
|
||||
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
|
||||
pip install /install/*.whl
|
||||
RUN git clone ${FA_REPO}
|
||||
RUN cd flash-attention \
|
||||
&& git checkout ${FA_BRANCH} \
|
||||
&& git submodule update --init \
|
||||
&& if [ "$USE_SCCACHE" = "1" ]; then \
|
||||
export HIP_CLANG_PATH=/opt/sccache-wrappers \
|
||||
&& sccache --show-stats; \
|
||||
fi \
|
||||
&& GPU_ARCHS=$(echo ${PYTORCH_ROCM_ARCH} | sed -e 's/;gfx1[0-9]\{3\}//g') python3 setup.py bdist_wheel --dist-dir=dist \
|
||||
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi
|
||||
RUN mkdir -p /app/install && cp /app/flash-attention/dist/*.whl /app/install
|
||||
|
||||
|
||||
###
|
||||
### AITER Build
|
||||
###
|
||||
FROM base AS build_aiter
|
||||
ARG AITER_BRANCH
|
||||
ARG AITER_REPO
|
||||
ARG USE_SCCACHE
|
||||
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
|
||||
pip install /install/*.whl
|
||||
RUN git clone --recursive --branch ${AITER_BRANCH} ${AITER_REPO}
|
||||
RUN cd aiter \
|
||||
&& git submodule update --init --recursive \
|
||||
&& pip install -r requirements.txt
|
||||
RUN pip install pyyaml && cd aiter \
|
||||
&& if [ "$USE_SCCACHE" = "1" ]; then \
|
||||
export HIP_CLANG_PATH=/opt/sccache-wrappers \
|
||||
&& sccache --show-stats; \
|
||||
fi \
|
||||
&& PREBUILD_KERNELS=1 AITER_USE_SYSTEM_TRITON=1 GPU_ARCHS=${AITER_ROCM_ARCH} python3 setup.py bdist_wheel --dist-dir=dist \
|
||||
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi \
|
||||
&& ls /app/aiter/dist/*.whl
|
||||
RUN mkdir -p /app/install && cp /app/aiter/dist/*.whl /app/install
|
||||
|
||||
|
||||
###
|
||||
### Final Build
|
||||
###
|
||||
|
||||
# Wheel release stage -
|
||||
# only includes dependencies used by wheel release pipeline
|
||||
FROM base AS debs_wheel_release
|
||||
RUN mkdir /app/debs
|
||||
RUN --mount=type=bind,from=build_triton,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
RUN --mount=type=bind,from=build_fa,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
RUN --mount=type=bind,from=build_amdsmi,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
RUN --mount=type=bind,from=build_torchvision,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
RUN --mount=type=bind,from=build_torchaudio,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
RUN --mount=type=bind,from=build_aiter,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
|
||||
# Full debs stage - includes Mori (used by Docker releases)
|
||||
FROM base AS debs
|
||||
RUN mkdir /app/debs
|
||||
RUN --mount=type=bind,from=build_triton,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
RUN --mount=type=bind,from=build_fa,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
RUN --mount=type=bind,from=build_amdsmi,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
RUN --mount=type=bind,from=build_torchvision,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
RUN --mount=type=bind,from=build_torchaudio,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
RUN --mount=type=bind,from=build_aiter,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
RUN --mount=type=bind,from=build_mori,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
|
||||
###
|
||||
### ROCr + CLR build
|
||||
###
|
||||
# Pull in updates to ROCr + clr from ROCm 7.2.4 and beyond.
|
||||
FROM base AS build_rocm_runtime
|
||||
ARG ROCM_SYSTEMS_REPO
|
||||
ARG ROCM_RUNTIME_COMMIT
|
||||
RUN apt-get update \
|
||||
&& apt-get install -y --no-install-recommends g++ \
|
||||
libelf-dev libdrm-dev libnuma-dev libdw-dev xxd \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
RUN pip install --no-cache-dir CppHeaderParser
|
||||
RUN git init -q /src && cd /src \
|
||||
&& git remote add origin ${ROCM_SYSTEMS_REPO} \
|
||||
&& git config core.sparseCheckout true \
|
||||
&& git config remote.origin.promisor true \
|
||||
&& git config remote.origin.partialclonefilter blob:none \
|
||||
&& git sparse-checkout init --cone \
|
||||
&& git sparse-checkout set projects/rocr-runtime projects/clr projects/hip shared cmake \
|
||||
&& git fetch --filter=blob:none --depth 1 origin ${ROCM_RUNTIME_COMMIT} \
|
||||
&& git checkout -q FETCH_HEAD
|
||||
# ROCr's trap_handler and blit_shaders call find_package(Clang/LLVM REQUIRED) only to get
|
||||
# IMPORTED executables. The ROCm dev image ships the binaries but no CMake package config, so
|
||||
# supply one pointing at them.
|
||||
RUN mkdir -p /opt/rocm-cmake-shim \
|
||||
&& printf 'if(NOT TARGET clang)\n add_executable(clang IMPORTED GLOBAL)\n set_target_properties(clang PROPERTIES IMPORTED_LOCATION "/opt/rocm/llvm/bin/clang")\nendif()\nset(Clang_PACKAGE_VERSION "rocm-image-shim")\n' > /opt/rocm-cmake-shim/ClangConfig.cmake \
|
||||
&& printf 'if(NOT TARGET llvm-objcopy)\n add_executable(llvm-objcopy IMPORTED GLOBAL)\n set_target_properties(llvm-objcopy PROPERTIES IMPORTED_LOCATION "/opt/rocm/llvm/bin/llvm-objcopy")\nendif()\nset(LLVM_FOUND TRUE)\nset(LLVM_PACKAGE_VERSION "rocm-image-shim")\n' > /opt/rocm-cmake-shim/LLVMConfig.cmake
|
||||
# ROCr first, and installed into /opt/rocm, because the clr build below needs its headers
|
||||
RUN cd /src/projects/rocr-runtime \
|
||||
&& cmake -B build -G Ninja \
|
||||
-DCMAKE_BUILD_TYPE=Release \
|
||||
-DCMAKE_PREFIX_PATH=/opt/rocm \
|
||||
-DCMAKE_INSTALL_PREFIX=/opt/rocm \
|
||||
-DClang_DIR=/opt/rocm-cmake-shim \
|
||||
-DLLVM_DIR=/opt/rocm-cmake-shim \
|
||||
&& cmake --build build --parallel 16 \
|
||||
&& cmake --install build \
|
||||
&& cmake --install build --prefix /rocr-install --strip
|
||||
RUN cd /src/projects/clr \
|
||||
&& cmake -B build -G Ninja \
|
||||
-DCMAKE_BUILD_TYPE=Release \
|
||||
-DCLR_BUILD_HIP=ON \
|
||||
-DCLR_BUILD_OCL=OFF \
|
||||
-DHIP_COMMON_DIR=/src/projects/hip \
|
||||
-DROCM_PATH=/opt/rocm \
|
||||
-DCMAKE_PREFIX_PATH=/opt/rocm \
|
||||
-DCMAKE_INSTALL_PREFIX=/opt/rocm \
|
||||
-DLLVM_DIR=/opt/rocm-cmake-shim \
|
||||
-DHIP_LLVM_ROOT=/opt/rocm/llvm \
|
||||
&& cmake --build build --parallel 16 \
|
||||
&& cmake --install build --prefix /clr-install --strip
|
||||
RUN mkdir -p /staging/lib \
|
||||
&& cp -P /rocr-install/lib/libhsa-runtime64.so* /staging/lib/ \
|
||||
&& cp -P /clr-install/lib/libamdhip64.so* /staging/lib/ \
|
||||
&& ls -l /staging/lib
|
||||
|
||||
# rocprofiler-sdk 1.3.2 fixes torch 2.12 profiler on ROCm 7.2.3
|
||||
FROM base AS build_rocprofiler_sdk
|
||||
ARG ROCM_SYSTEMS_REPO
|
||||
ARG ROCPROFILER_SDK_COMMIT
|
||||
ARG USE_SCCACHE
|
||||
RUN apt-get update \
|
||||
&& apt-get install -y --no-install-recommends libdw-dev libelf-dev libsqlite3-dev libzstd-dev \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
RUN git init -q /src && cd /src \
|
||||
&& git remote add origin ${ROCM_SYSTEMS_REPO} \
|
||||
&& git config core.sparseCheckout true \
|
||||
&& git config remote.origin.promisor true \
|
||||
&& git config remote.origin.partialclonefilter blob:none \
|
||||
&& git sparse-checkout init --cone \
|
||||
&& git sparse-checkout set projects/rocprofiler-sdk shared cmake \
|
||||
&& git fetch --filter=blob:none --depth 1 origin ${ROCPROFILER_SDK_COMMIT} \
|
||||
&& git checkout -q FETCH_HEAD
|
||||
# Patch the ROCm 7.2.3 build of rocprofiler-sdk to include queue-interposition fixes that are
|
||||
# absent from the pinned 7.14 build. This resolves hangs on torch.profiler on ROCm.
|
||||
# TODO: Drop these once ${ROCPROFILER_SDK_COMMIT} is bumped to a build that already
|
||||
# contains them, such as ROCm 10.0. See https://github.com/vllm-project/vllm/issues/54087.
|
||||
ARG ROCPROFILER_SDK_PR_7796=fae7fd57ffb8a76e811d6410b8a0cee1e94d710a
|
||||
ARG ROCPROFILER_SDK_PR_7924=f8db22adbe17686a9b05ae03367dd1ec73011653
|
||||
RUN cd /src \
|
||||
&& HSA_PATHS='projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/hsa/*' \
|
||||
&& curl -fsSL -o /tmp/qi-7796.patch "${ROCM_SYSTEMS_REPO%.git}/commit/${ROCPROFILER_SDK_PR_7796}.patch" \
|
||||
&& curl -fsSL -o /tmp/qi-7924.patch "${ROCM_SYSTEMS_REPO%.git}/commit/${ROCPROFILER_SDK_PR_7924}.patch" \
|
||||
&& git apply -p1 --include="$HSA_PATHS" --verbose /tmp/qi-7796.patch \
|
||||
&& git apply -p1 --include="$HSA_PATHS" --verbose /tmp/qi-7924.patch
|
||||
RUN cd /src/projects/rocprofiler-sdk \
|
||||
&& if [ "$USE_SCCACHE" = "1" ]; then \
|
||||
export CMAKE_C_COMPILER_LAUNCHER=sccache \
|
||||
&& export CMAKE_CXX_COMPILER_LAUNCHER=sccache; \
|
||||
fi \
|
||||
&& cmake -B build -G Ninja \
|
||||
-DCMAKE_BUILD_TYPE=Release \
|
||||
-DCMAKE_PREFIX_PATH=/opt/rocm \
|
||||
-DCMAKE_INSTALL_PREFIX=/opt/rocm \
|
||||
-DROCPROFILER_BUILD_TESTS=OFF \
|
||||
-DROCPROFILER_BUILD_SAMPLES=OFF \
|
||||
-DROCPROFILER_BUILD_DOCS=OFF \
|
||||
&& cmake --build build --parallel "$(nproc)" \
|
||||
&& cmake --install build --prefix /staging --strip
|
||||
|
||||
FROM base AS final
|
||||
RUN --mount=type=bind,from=debs,src=/app/debs,target=/install \
|
||||
pip install /install/*.whl
|
||||
|
||||
COPY --from=build_rocprofiler_sdk /staging/ /opt/rocm/
|
||||
ENV ROCPROFILER_QUEUE_INTERPOSITION=0
|
||||
|
||||
COPY --from=build_rocm_runtime /staging/ /opt/rocm/
|
||||
# Hard link the patched library onto every stock file name so that both the
|
||||
# SONAME lookup and the hard coded paths resolve to it.
|
||||
RUN set -eux; \
|
||||
cd /opt/rocm/lib; \
|
||||
ours_hsa="$(readlink -f libhsa-runtime64.so.1)"; \
|
||||
ours_hip="$(readlink -f libamdhip64.so.7)"; \
|
||||
for f in libhsa-runtime64.so.1.*; do \
|
||||
[ "$f" = "${ours_hsa##*/}" ] || ln -f "$ours_hsa" "$f"; \
|
||||
done; \
|
||||
for f in libamdhip64.so.7.*; do \
|
||||
[ "$f" = "${ours_hip##*/}" ] || ln -f "$ours_hip" "$f"; \
|
||||
done
|
||||
|
||||
RUN ldconfig
|
||||
|
||||
ARG BASE_IMAGE
|
||||
ARG TRITON_BRANCH
|
||||
ARG TRITON_REPO
|
||||
ARG PYTORCH_BRANCH
|
||||
ARG PYTORCH_VISION_BRANCH
|
||||
ARG PYTORCH_REPO
|
||||
ARG PYTORCH_VISION_REPO
|
||||
ARG PYTORCH_AUDIO_BRANCH
|
||||
ARG PYTORCH_AUDIO_REPO
|
||||
ARG FA_BRANCH
|
||||
ARG FA_REPO
|
||||
ARG AITER_BRANCH
|
||||
ARG AITER_REPO
|
||||
ARG MORI_BRANCH
|
||||
ARG MORI_REPO
|
||||
ARG ROCM_SYSTEMS_REPO
|
||||
ARG ROCPROFILER_SDK_COMMIT
|
||||
ARG ROCM_RUNTIME_COMMIT
|
||||
RUN echo "BASE_IMAGE: ${BASE_IMAGE}" > /app/versions.txt \
|
||||
&& echo "ROCM_SYSTEMS_REPO: ${ROCM_SYSTEMS_REPO}" >> /app/versions.txt \
|
||||
&& echo "ROCPROFILER_SDK_COMMIT: ${ROCPROFILER_SDK_COMMIT}" >> /app/versions.txt \
|
||||
&& echo "ROCM_RUNTIME_COMMIT: ${ROCM_RUNTIME_COMMIT}" >> /app/versions.txt \
|
||||
&& echo "TRITON_BRANCH: ${TRITON_BRANCH}" >> /app/versions.txt \
|
||||
&& echo "TRITON_REPO: ${TRITON_REPO}" >> /app/versions.txt \
|
||||
&& echo "PYTORCH_BRANCH: ${PYTORCH_BRANCH}" >> /app/versions.txt \
|
||||
&& echo "PYTORCH_VISION_BRANCH: ${PYTORCH_VISION_BRANCH}" >> /app/versions.txt \
|
||||
&& echo "PYTORCH_REPO: ${PYTORCH_REPO}" >> /app/versions.txt \
|
||||
&& echo "PYTORCH_VISION_REPO: ${PYTORCH_VISION_REPO}" >> /app/versions.txt \
|
||||
&& echo "PYTORCH_AUDIO_BRANCH: ${PYTORCH_AUDIO_BRANCH}" >> /app/versions.txt \
|
||||
&& echo "PYTORCH_AUDIO_REPO: ${PYTORCH_AUDIO_REPO}" >> /app/versions.txt \
|
||||
&& echo "FA_BRANCH: ${FA_BRANCH}" >> /app/versions.txt \
|
||||
&& echo "FA_REPO: ${FA_REPO}" >> /app/versions.txt \
|
||||
&& echo "AITER_BRANCH: ${AITER_BRANCH}" >> /app/versions.txt \
|
||||
&& echo "AITER_REPO: ${AITER_REPO}" >> /app/versions.txt \
|
||||
&& echo "MORI_BRANCH: ${MORI_BRANCH}" >> /app/versions.txt \
|
||||
&& echo "MORI_REPO: ${MORI_REPO}" >> /app/versions.txt
|
||||
+267
-270
@@ -1,12 +1,11 @@
|
||||
ARG BASE_IMAGE=rocm/dev-ubuntu-22.04:7.2.3-complete
|
||||
ARG TRITON_BRANCH="669b31a" # release/internal/3.8.x as of 09/17
|
||||
ARG BASE_IMAGE=ubuntu:22.04
|
||||
ARG PYTHON_VERSION=3.12
|
||||
ARG TORCH_VERSION=2.12.0+rocm10.0.0
|
||||
ARG TORCHVISION_VERSION=0.27.0+rocm10.0.0
|
||||
ARG TORCHAUDIO_VERSION=2.11.0+rocm10.0.0
|
||||
ARG TRITON_BRANCH="669b31ac" # release/internal/3.8.x as of 09/29/2026
|
||||
ARG TRITON_VERSION=3.8.0+git669b31ac
|
||||
ARG TRITON_REPO="https://github.com/ROCm/triton.git"
|
||||
ARG PYTORCH_BRANCH="733fca1" # release/2.13 as of 09/17
|
||||
ARG PYTORCH_REPO="https://github.com/ROCm/pytorch.git"
|
||||
ARG PYTORCH_VISION_BRANCH="v0.28.0"
|
||||
ARG PYTORCH_VISION_REPO="https://github.com/pytorch/vision.git"
|
||||
ARG PYTORCH_AUDIO_BRANCH="v2.11.0"
|
||||
ARG PYTORCH_AUDIO_REPO="https://github.com/pytorch/audio.git"
|
||||
ARG FA_BRANCH="0e60e394"
|
||||
ARG FA_REPO="https://github.com/Dao-AILab/flash-attention.git"
|
||||
ARG AITER_BRANCH="v0.1.25"
|
||||
@@ -14,8 +13,18 @@ ARG AITER_REPO="https://github.com/ROCm/aiter.git"
|
||||
ARG MORI_BRANCH="v1.2.3.post1"
|
||||
ARG MORI_REPO="https://github.com/ROCm/mori.git"
|
||||
ARG ROCM_SYSTEMS_REPO="https://github.com/ROCm/rocm-systems.git"
|
||||
ARG ROCPROFILER_SDK_COMMIT="2b22ab0195cc1461cd9abf3b969e9dd7c10af350"
|
||||
ARG ROCM_RUNTIME_COMMIT="af70edc91f1dc5d228fd4c076df99e65e49c4521"
|
||||
ARG ROCM_RUNTIME_COMMIT="25e14349f2606a6b95c4ccd45eb92a1835646cf2"
|
||||
|
||||
|
||||
# TheRock configuration
|
||||
ARG SITE_PACKAGES=/usr/local/lib/python${PYTHON_VERSION}/dist-packages
|
||||
ARG ROCM_RELEASE_WHEELS_MULTIARCH_URL="https://stable.repo.amd.com/rocm/whl-next/" # For pre-built GPU images, packages, etc.
|
||||
ARG ROCM_NIGHTLY_WHEELS_MULTIARCH_URL="https://nightly.repo.amd.com/rocm/whl-next/" # For torch, triton, etc. wheels with latest fixes.
|
||||
ARG ROCM_SDK_VERSION=10.0.0
|
||||
|
||||
# Build configuration
|
||||
ARG PYTORCH_ROCM_ARCH="gfx90a;gfx942;gfx950;gfx1100;gfx1101;gfx1200;gfx1201;gfx1150;gfx1151"
|
||||
ARG AITER_ROCM_ARCH="gfx942;gfx950"
|
||||
|
||||
# Sccache configuration (only used in release pipeline)
|
||||
ARG USE_SCCACHE
|
||||
@@ -25,47 +34,128 @@ ARG SCCACHE_BUCKET_NAME=vllm-build-sccache
|
||||
ARG SCCACHE_REGION_NAME=us-west-2
|
||||
ARG SCCACHE_S3_NO_CREDENTIALS=0
|
||||
|
||||
|
||||
FROM ${BASE_IMAGE} AS base
|
||||
|
||||
ENV PATH=/opt/rocm/llvm/bin:/opt/rocm/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
|
||||
ENV ROCM_PATH=/opt/rocm
|
||||
ENV LD_LIBRARY_PATH=/opt/rocm/lib:/usr/local/lib:
|
||||
ARG PYTORCH_ROCM_ARCH
|
||||
ENV PYTORCH_ROCM_ARCH="${PYTORCH_ROCM_ARCH}"
|
||||
ARG AITER_ROCM_ARCH
|
||||
ENV AITER_ROCM_ARCH="${AITER_ROCM_ARCH}"
|
||||
|
||||
ARG PYTORCH_ROCM_ARCH=gfx90a;gfx942;gfx950;gfx1100;gfx1101;gfx1200;gfx1201;gfx1150;gfx1151
|
||||
ENV PYTORCH_ROCM_ARCH=${PYTORCH_ROCM_ARCH}
|
||||
ENV AITER_ROCM_ARCH=gfx942;gfx950
|
||||
# Note: Do not set MORI_GPU_ARCHS here, it is automatically inferred at runtime
|
||||
ARG SITE_PACKAGES
|
||||
ENV SITE_PACKAGES=${SITE_PACKAGES}
|
||||
ENV SDK_DEV="$SITE_PACKAGES/_rocm_sdk_devel" \
|
||||
SDK_CORE="$SITE_PACKAGES/_rocm_sdk_core"
|
||||
ENV ROCM_PATH="$SDK_DEV" \
|
||||
ROCM_HOME="$SDK_DEV" \
|
||||
HIP_PATH="$SDK_DEV" \
|
||||
HIP_CLANG_PATH="$SDK_DEV/lib/llvm/bin" \
|
||||
HIP_DEVICE_LIB_PATH="$SDK_CORE/lib/llvm/amdgcn/bitcode"
|
||||
ENV LD_LIBRARY_PATH="$SDK_DEV/lib:$SDK_DEV/lib/rocm_sysdeps/lib:$SDK_CORE/lib:$SDK_CORE/lib/rocm_sysdeps/lib"
|
||||
ENV PATH="$SDK_DEV/bin:$SDK_DEV/lib/llvm/bin:$SDK_CORE/bin:$PATH"
|
||||
|
||||
# Required for RCCL in ROCm7.1
|
||||
ENV HSA_NO_SCRATCH_RECLAIM=1
|
||||
|
||||
ARG PYTHON_VERSION=3.12
|
||||
ARG PYTHON_VERSION
|
||||
ENV PYTHON_VERSION=${PYTHON_VERSION}
|
||||
|
||||
RUN mkdir -p /app
|
||||
WORKDIR /app
|
||||
ENV DEBIAN_FRONTEND=noninteractive
|
||||
|
||||
# Install Python and other dependencies (plus gcc-13, selected later in build_pytorch).
|
||||
# Install Python and other dependencies
|
||||
RUN apt-get update -y \
|
||||
&& apt-get install -y software-properties-common git curl sudo vim less libgfortran5 libopenmpi-dev libpci-dev liblzma-dev pkg-config \
|
||||
&& apt-get install -y software-properties-common build-essential git curl sudo vim less libgfortran5 libopenmpi-dev libpci-dev liblzma-dev pkg-config libdrm-dev libnuma-dev kmod \
|
||||
&& for i in 1 2 3; do \
|
||||
add-apt-repository -y ppa:deadsnakes/ppa && add-apt-repository -y ppa:ubuntu-toolchain-r/test && break || \
|
||||
add-apt-repository -y ppa:deadsnakes/ppa && break || \
|
||||
{ echo "Attempt $i failed, retrying in 5s..."; sleep 5; }; \
|
||||
done \
|
||||
&& apt-get update -y \
|
||||
&& apt-get install -y python${PYTHON_VERSION} python${PYTHON_VERSION}-dev python${PYTHON_VERSION}-venv \
|
||||
python${PYTHON_VERSION}-lib2to3 python-is-python3 gcc-13 g++-13 \
|
||||
python${PYTHON_VERSION}-lib2to3 python-is-python3 \
|
||||
&& update-alternatives --install /usr/bin/python3 python3 /usr/bin/python${PYTHON_VERSION} 1 \
|
||||
&& update-alternatives --set python3 /usr/bin/python${PYTHON_VERSION} \
|
||||
&& ln -sf /usr/bin/python${PYTHON_VERSION}-config /usr/bin/python3-config \
|
||||
&& curl -sS https://bootstrap.pypa.io/get-pip.py | python${PYTHON_VERSION} \
|
||||
&& python3 --version && python3 -m pip --version
|
||||
|
||||
RUN pip install -U packaging 'cmake<4' ninja wheel 'setuptools<80' 'pybind11<3.1.0' Cython
|
||||
RUN pip install -U packaging 'cmake<4' ninja wheel 'setuptools<80' pybind11==3.0.4 Cython
|
||||
RUN apt-get update && apt-get install -y libjpeg-dev libsox-dev libsox-fmt-all sox && rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Install ROCm SDK via TheRock
|
||||
ARG ROCM_RELEASE_WHEELS_MULTIARCH_URL
|
||||
ARG ROCM_NIGHTLY_WHEELS_MULTIARCH_URL
|
||||
ARG ROCM_SDK_VERSION
|
||||
RUN pip install --index-url ${ROCM_RELEASE_WHEELS_MULTIARCH_URL} \
|
||||
--extra-index-url ${ROCM_NIGHTLY_WHEELS_MULTIARCH_URL} \
|
||||
"rocm[libraries,devel]${ROCM_SDK_VERSION:+==${ROCM_SDK_VERSION}}" \
|
||||
&& rocm-sdk init
|
||||
|
||||
# Install PyTorch wheels (nightly for latest torch, release as fallback)
|
||||
ARG TORCH_VERSION
|
||||
ARG TORCHAUDIO_VERSION
|
||||
ARG TORCHVISION_VERSION
|
||||
RUN pip install \
|
||||
--index-url ${ROCM_RELEASE_WHEELS_MULTIARCH_URL} \
|
||||
--extra-index-url ${ROCM_NIGHTLY_WHEELS_MULTIARCH_URL} \
|
||||
"torch==${TORCH_VERSION}" \
|
||||
"torchaudio==${TORCHAUDIO_VERSION}" \
|
||||
"torchvision==${TORCHVISION_VERSION}"
|
||||
|
||||
# Install the AMD-compiled device kernels corresponding to PYTORCH_ROCM_ARCH:
|
||||
|
||||
# Possible values that can be in PYTORCH_ROCM_ARCH:
|
||||
#
|
||||
# gfx90a (MI210/250), gfx942 (MI300), gfx950 (MI350), gfx1100 (7900XTX),
|
||||
# gfx1101 (7800XT), gfx1200 (9070XT), gfx1201 (9070), gfx1150/1151 (Strix APU)
|
||||
|
||||
# For each arch in PYTORCH_ROCM_ARCH, install the precompiled GPU code that
|
||||
# PyTorch needs to run on that specific chip.
|
||||
SHELL ["/bin/bash", "-c"]
|
||||
RUN TORCH_VER=$(python3 -c "import torch; print(torch.__version__)") \
|
||||
&& for arch in $(echo "${PYTORCH_ROCM_ARCH}" | tr ';' ' '); do \
|
||||
echo "Installing device packages for ${arch}" \
|
||||
&& pip install --index-url ${ROCM_RELEASE_WHEELS_MULTIARCH_URL} \
|
||||
--extra-index-url ${ROCM_NIGHTLY_WHEELS_MULTIARCH_URL} \
|
||||
"torch[device-${arch}]==${TORCH_VER}" \
|
||||
"torchvision[device-${arch}]==${TORCHVISION_VERSION}" \
|
||||
"rocm[device-${arch}]==${ROCM_SDK_VERSION}"; \
|
||||
done
|
||||
|
||||
# Remove gfx90a xnack kpacks (ASAN artifacts that cause hipErrorInvalidImage on MI250)
|
||||
RUN rm -f "${SITE_PACKAGES}/torch/.kpack/torch_gfx90a:xnack+.kpack" \
|
||||
"${SITE_PACKAGES}/torch/.kpack/torch_gfx90a:xnack-.kpack"
|
||||
|
||||
RUN cd $SITE_PACKAGES/_rocm_sdk_core/share/amd_smi && pip install .
|
||||
ENV PYTHONPATH=$SITE_PACKAGES/_rocm_sdk_core/share/amd_smi
|
||||
|
||||
# TheRock runtime fixes
|
||||
RUN if [ -f "${SITE_PACKAGES}/rocm_sdk/__init__.py" ]; then \
|
||||
sed -i 's/rtld_global: bool = True/rtld_global: bool = False/g' "${SITE_PACKAGES}/rocm_sdk/__init__.py"; \
|
||||
fi \
|
||||
&& if [ -f "${SITE_PACKAGES}/_rocm_sdk_core/lib/librocm_smi64.so.1" ] \
|
||||
&& [ ! -e "${ROCM_PATH}/lib/librocm_smi64.so.7" ]; then \
|
||||
ln -sfn "${SITE_PACKAGES}/_rocm_sdk_core/lib/librocm_smi64.so.1" "${ROCM_PATH}/lib/librocm_smi64.so.7"; \
|
||||
fi \
|
||||
&& printf '%s\n%s\n' "${ROCM_PATH}/lib" "${SITE_PACKAGES}/_rocm_sdk_core/lib" > /etc/ld.so.conf.d/rocm-pip.conf \
|
||||
&& for f in ${SITE_PACKAGES}/_rocm_sdk_core/lib/libamdhip64.so*; do \
|
||||
bn=$(basename "$f"); \
|
||||
if [ -f "${ROCM_PATH}/lib/${bn}" ] && [ -f "$f" ] && [ ! -L "$f" ]; then \
|
||||
rm -f "$f"; \
|
||||
ln -sfn "${ROCM_PATH}/lib/${bn}" "$f"; \
|
||||
fi; \
|
||||
done \
|
||||
&& for f in librocprofiler-sdk.so.1 librocprofiler-register.so.0 \
|
||||
librocprofiler-sdk-roctx.so.1; do \
|
||||
if [ -e "${SITE_PACKAGES}/_rocm_sdk_core/lib/${f}" ] \
|
||||
&& [ -e "${SITE_PACKAGES}/_rocm_sdk_devel/lib/${f}" ]; then \
|
||||
rm -f "${SITE_PACKAGES}/_rocm_sdk_core/lib/${f}"; \
|
||||
ln -sfn "${SITE_PACKAGES}/_rocm_sdk_devel/lib/${f}" \
|
||||
"${SITE_PACKAGES}/_rocm_sdk_core/lib/${f}"; \
|
||||
fi; \
|
||||
done \
|
||||
&& ldconfig
|
||||
|
||||
# Install sccache if USE_SCCACHE is enabled (for release builds)
|
||||
ARG USE_SCCACHE
|
||||
ARG SCCACHE_DOWNLOAD_URL
|
||||
@@ -94,141 +184,35 @@ RUN if [ "$USE_SCCACHE" = "1" ]; then \
|
||||
RUN if [ "$USE_SCCACHE" = "1" ]; then \
|
||||
echo "Setting up sccache wrappers for HIP compilation..." \
|
||||
&& mkdir -p /opt/sccache-wrappers \
|
||||
&& printf '#!/bin/bash\nexec sccache /opt/rocm/lib/llvm/bin/clang++ "$@"\n' > /opt/sccache-wrappers/clang++ \
|
||||
&& printf '#!/bin/bash\nexec sccache %s/lib/llvm/bin/clang++ "$@"\n' "${ROCM_PATH}" > /opt/sccache-wrappers/clang++ \
|
||||
&& chmod +x /opt/sccache-wrappers/clang++ \
|
||||
&& printf '#!/bin/bash\nexec sccache /opt/rocm/lib/llvm/bin/clang "$@"\n' > /opt/sccache-wrappers/clang \
|
||||
&& printf '#!/bin/bash\nexec sccache %s/lib/llvm/bin/clang "$@"\n' "${ROCM_PATH}" > /opt/sccache-wrappers/clang \
|
||||
&& chmod +x /opt/sccache-wrappers/clang \
|
||||
&& echo "sccache wrappers created in /opt/sccache-wrappers"; \
|
||||
fi
|
||||
|
||||
# Set sccache environment variables only when USE_SCCACHE=1
|
||||
# This prevents S3 config from leaking into images when sccache is not used
|
||||
ARG USE_SCCACHE
|
||||
ENV SCCACHE_BUCKET=${USE_SCCACHE:+${SCCACHE_BUCKET_NAME}}
|
||||
ENV SCCACHE_REGION=${USE_SCCACHE:+${SCCACHE_REGION_NAME}}
|
||||
ENV SCCACHE_S3_NO_CREDENTIALS=${USE_SCCACHE:+${SCCACHE_S3_NO_CREDENTIALS}}
|
||||
ENV SCCACHE_IDLE_TIMEOUT=${USE_SCCACHE:+0}
|
||||
|
||||
|
||||
###
|
||||
### Triton Build
|
||||
###
|
||||
FROM base AS build_triton
|
||||
ARG TRITON_BRANCH
|
||||
ARG TRITON_REPO
|
||||
RUN git clone ${TRITON_REPO}
|
||||
RUN cd triton \
|
||||
&& git checkout ${TRITON_BRANCH} \
|
||||
&& git config --global user.email "you@example.com" && git config --global user.name "Your Name" \
|
||||
&& if [ ! -f setup.py ]; then cd python; fi \
|
||||
&& MAX_JOBS=16 python3 setup.py bdist_wheel --dist-dir=dist \
|
||||
&& mkdir -p /app/install && cp dist/*.whl /app/install
|
||||
|
||||
|
||||
###
|
||||
### AMD SMI Build
|
||||
###
|
||||
FROM base AS build_amdsmi
|
||||
RUN cd /opt/rocm/share/amd_smi \
|
||||
&& pip wheel . --wheel-dir=dist
|
||||
RUN mkdir -p /app/install && cp /opt/rocm/share/amd_smi/dist/*.whl /app/install
|
||||
|
||||
|
||||
###
|
||||
### Pytorch build
|
||||
###
|
||||
FROM base AS build_pytorch
|
||||
ARG PYTORCH_BRANCH
|
||||
ARG PYTORCH_REPO
|
||||
ARG USE_SCCACHE
|
||||
|
||||
RUN apt-get update && apt-get install -y pkg-config liblzma-dev
|
||||
# torch 2.12 emits float->__bf16 libcalls (__truncsfbf2) absent from gcc-11 libgcc; build with gcc-13.
|
||||
RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-13 130 --slave /usr/bin/g++ g++ /usr/bin/g++-13
|
||||
RUN git clone ${PYTORCH_REPO} pytorch
|
||||
RUN cd pytorch && git checkout ${PYTORCH_BRANCH}
|
||||
RUN cd pytorch \
|
||||
&& pip install -r requirements.txt && git submodule update --init --recursive
|
||||
RUN cd pytorch && python3 tools/amd_build/build_amd.py \
|
||||
&& if [ "$USE_SCCACHE" = "1" ]; then \
|
||||
export HIP_CLANG_PATH=/opt/sccache-wrappers \
|
||||
&& export CMAKE_C_COMPILER_LAUNCHER=sccache \
|
||||
&& export CMAKE_CXX_COMPILER_LAUNCHER=sccache \
|
||||
&& sccache --show-stats; \
|
||||
fi \
|
||||
&& CMAKE_PREFIX_PATH=$(python3 -c 'import sys; print(sys.prefix)') python3 setup.py bdist_wheel --dist-dir=dist \
|
||||
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi
|
||||
RUN mkdir -p /app/install && cp /app/pytorch/dist/*.whl /app/install
|
||||
|
||||
|
||||
###
|
||||
### Shared PyTorch runtime for dependent builds
|
||||
###
|
||||
FROM base AS build_pytorch_runtime
|
||||
RUN update-alternatives --install /usr/bin/gcc gcc /usr/bin/gcc-13 130 --slave /usr/bin/g++ g++ /usr/bin/g++-13
|
||||
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install,ro \
|
||||
pip install /install/*.whl
|
||||
|
||||
|
||||
###
|
||||
### TorchVision build
|
||||
###
|
||||
FROM build_pytorch_runtime AS build_torchvision
|
||||
ARG PYTORCH_VISION_BRANCH
|
||||
ARG PYTORCH_VISION_REPO
|
||||
ARG USE_SCCACHE
|
||||
|
||||
RUN git clone ${PYTORCH_VISION_REPO} vision
|
||||
RUN cd vision && git checkout ${PYTORCH_VISION_BRANCH} \
|
||||
&& if [ "$USE_SCCACHE" = "1" ]; then \
|
||||
export HIP_CLANG_PATH=/opt/sccache-wrappers \
|
||||
&& export CMAKE_C_COMPILER_LAUNCHER=sccache \
|
||||
&& export CMAKE_CXX_COMPILER_LAUNCHER=sccache; \
|
||||
fi \
|
||||
&& python3 setup.py bdist_wheel --dist-dir=dist \
|
||||
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi
|
||||
RUN mkdir -p /app/install && cp /app/vision/dist/*.whl /app/install
|
||||
|
||||
|
||||
###
|
||||
### TorchAudio build
|
||||
###
|
||||
FROM build_pytorch_runtime AS build_torchaudio
|
||||
ARG PYTORCH_AUDIO_BRANCH
|
||||
ARG PYTORCH_AUDIO_REPO
|
||||
ARG USE_SCCACHE
|
||||
|
||||
RUN git clone ${PYTORCH_AUDIO_REPO} audio
|
||||
RUN cd audio && git checkout ${PYTORCH_AUDIO_BRANCH} \
|
||||
&& git submodule update --init --recursive
|
||||
RUN cd audio && pip install -r requirements.txt
|
||||
RUN cd audio \
|
||||
&& if [ "$USE_SCCACHE" = "1" ]; then \
|
||||
export HIP_CLANG_PATH=/opt/sccache-wrappers \
|
||||
&& export CMAKE_C_COMPILER_LAUNCHER=sccache \
|
||||
&& export CMAKE_CXX_COMPILER_LAUNCHER=sccache; \
|
||||
fi \
|
||||
&& python3 setup.py bdist_wheel --dist-dir=dist \
|
||||
&& if [ "$USE_SCCACHE" = "1" ]; then sccache --show-stats; fi
|
||||
RUN mkdir -p /app/install && cp /app/audio/dist/*.whl /app/install
|
||||
|
||||
|
||||
###
|
||||
### MORI Build
|
||||
###
|
||||
FROM base AS build_mori
|
||||
ARG MORI_BRANCH
|
||||
ARG MORI_REPO
|
||||
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
|
||||
pip install /install/*.whl
|
||||
RUN git clone ${MORI_REPO}
|
||||
RUN cd mori \
|
||||
&& git checkout ${MORI_BRANCH} \
|
||||
&& git submodule update --init --recursive \
|
||||
&& python3 setup.py bdist_wheel --dist-dir=dist && ls /app/mori/dist/*.whl
|
||||
&& ln -sfn ${ROCM_PATH} /opt/rocm \
|
||||
&& pip install "setuptools_scm>=6.2" \
|
||||
&& CMAKE_PREFIX_PATH="${ROCM_PATH}:${ROCM_PATH}/lib/rocm_sysdeps" \
|
||||
python3 setup.py bdist_wheel --dist-dir=dist && ls /app/mori/dist/*.whl
|
||||
RUN mkdir -p /app/install && cp /app/mori/dist/*.whl /app/install
|
||||
|
||||
|
||||
###
|
||||
### FlashAttention Build
|
||||
###
|
||||
@@ -236,8 +220,6 @@ FROM base AS build_fa
|
||||
ARG FA_BRANCH
|
||||
ARG FA_REPO
|
||||
ARG USE_SCCACHE
|
||||
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
|
||||
pip install /install/*.whl
|
||||
RUN git clone ${FA_REPO}
|
||||
RUN cd flash-attention \
|
||||
&& git checkout ${FA_BRANCH} \
|
||||
@@ -258,12 +240,12 @@ FROM base AS build_aiter
|
||||
ARG AITER_BRANCH
|
||||
ARG AITER_REPO
|
||||
ARG USE_SCCACHE
|
||||
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
|
||||
pip install /install/*.whl
|
||||
RUN git clone --recursive --branch ${AITER_BRANCH} ${AITER_REPO}
|
||||
RUN cd aiter \
|
||||
&& git submodule update --init --recursive \
|
||||
&& pip install -r requirements.txt
|
||||
# AITER_USE_SYSTEM_TRITON keeps setup.py from resolving its own triton, which would
|
||||
# displace the pinned one already installed in base.
|
||||
RUN pip install pyyaml && cd aiter \
|
||||
&& if [ "$USE_SCCACHE" = "1" ]; then \
|
||||
export HIP_CLANG_PATH=/opt/sccache-wrappers \
|
||||
@@ -276,58 +258,30 @@ RUN mkdir -p /app/install && cp /app/aiter/dist/*.whl /app/install
|
||||
|
||||
|
||||
###
|
||||
### Final Build
|
||||
### Triton Build
|
||||
###
|
||||
FROM base AS build_triton
|
||||
ARG TRITON_BRANCH
|
||||
ARG TRITON_REPO
|
||||
RUN git clone ${TRITON_REPO}
|
||||
RUN cd triton \
|
||||
&& git checkout ${TRITON_BRANCH} \
|
||||
&& if [ ! -f setup.py ]; then cd python; fi \
|
||||
&& python3 setup.py bdist_wheel --dist-dir=dist \
|
||||
&& mkdir -p /app/install && cp dist/*.whl /app/install
|
||||
|
||||
# Wheel release stage -
|
||||
# only includes dependencies used by wheel release pipeline
|
||||
FROM base AS debs_wheel_release
|
||||
RUN mkdir /app/debs
|
||||
RUN --mount=type=bind,from=build_triton,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
RUN --mount=type=bind,from=build_fa,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
RUN --mount=type=bind,from=build_amdsmi,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
RUN --mount=type=bind,from=build_torchvision,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
RUN --mount=type=bind,from=build_torchaudio,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
RUN --mount=type=bind,from=build_aiter,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
|
||||
# Full debs stage - includes Mori (used by Docker releases)
|
||||
FROM base AS debs
|
||||
RUN mkdir /app/debs
|
||||
RUN --mount=type=bind,from=build_triton,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
RUN --mount=type=bind,from=build_fa,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
RUN --mount=type=bind,from=build_amdsmi,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
RUN --mount=type=bind,from=build_pytorch,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
RUN --mount=type=bind,from=build_torchvision,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
RUN --mount=type=bind,from=build_torchaudio,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
RUN --mount=type=bind,from=build_aiter,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
RUN --mount=type=bind,from=build_mori,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
|
||||
###
|
||||
### ROCr + CLR build
|
||||
###
|
||||
# Pull in updates to ROCr + clr from ROCm 7.2.4 and beyond.
|
||||
# Brings in multi-stream perf improvements; drop after these arrive in an official release
|
||||
FROM base AS build_rocm_runtime
|
||||
ARG ROCM_SYSTEMS_REPO
|
||||
ARG ROCM_RUNTIME_COMMIT
|
||||
# libgl-dev: rocclr does find_package(OpenGL REQUIRED) unconditionally and the wheel image has
|
||||
# no GL headers. The link must not pick up a libGL DT_NEEDED, which is asserted after the build.
|
||||
RUN apt-get update \
|
||||
&& apt-get install -y --no-install-recommends g++ \
|
||||
libelf-dev libdrm-dev libnuma-dev libdw-dev xxd \
|
||||
&& apt-get install -y --no-install-recommends g++ libdw-dev xxd libgl-dev libglx-dev \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
RUN pip install --no-cache-dir CppHeaderParser
|
||||
RUN git init -q /src && cd /src \
|
||||
@@ -339,20 +293,17 @@ RUN git init -q /src && cd /src \
|
||||
&& git sparse-checkout set projects/rocr-runtime projects/clr projects/hip shared cmake \
|
||||
&& git fetch --filter=blob:none --depth 1 origin ${ROCM_RUNTIME_COMMIT} \
|
||||
&& git checkout -q FETCH_HEAD
|
||||
# ROCr's trap_handler and blit_shaders call find_package(Clang/LLVM REQUIRED) only to get
|
||||
# IMPORTED executables. The ROCm dev image ships the binaries but no CMake package config, so
|
||||
# supply one pointing at them.
|
||||
RUN mkdir -p /opt/rocm-cmake-shim \
|
||||
&& printf 'if(NOT TARGET clang)\n add_executable(clang IMPORTED GLOBAL)\n set_target_properties(clang PROPERTIES IMPORTED_LOCATION "/opt/rocm/llvm/bin/clang")\nendif()\nset(Clang_PACKAGE_VERSION "rocm-image-shim")\n' > /opt/rocm-cmake-shim/ClangConfig.cmake \
|
||||
&& printf 'if(NOT TARGET llvm-objcopy)\n add_executable(llvm-objcopy IMPORTED GLOBAL)\n set_target_properties(llvm-objcopy PROPERTIES IMPORTED_LOCATION "/opt/rocm/llvm/bin/llvm-objcopy")\nendif()\nset(LLVM_FOUND TRUE)\nset(LLVM_PACKAGE_VERSION "rocm-image-shim")\n' > /opt/rocm-cmake-shim/LLVMConfig.cmake
|
||||
# ROCr first, and installed into /opt/rocm, because the clr build below needs its headers
|
||||
# libelf is taken from the SDK's own vendored sysdeps rather than apt -- the wheel image has no
|
||||
# /usr/include/libelf.h -- so ROCr links against the copy the wheel itself was built against.
|
||||
ENV SYSDEPS="$SDK_DEV/lib/rocm_sysdeps"
|
||||
# ROCr first, and installed into $SDK_DEV, because the clr build below needs its headers.
|
||||
RUN cd /src/projects/rocr-runtime \
|
||||
&& cmake -B build -G Ninja \
|
||||
-DCMAKE_BUILD_TYPE=Release \
|
||||
-DCMAKE_PREFIX_PATH=/opt/rocm \
|
||||
-DCMAKE_INSTALL_PREFIX=/opt/rocm \
|
||||
-DClang_DIR=/opt/rocm-cmake-shim \
|
||||
-DLLVM_DIR=/opt/rocm-cmake-shim \
|
||||
-DCMAKE_PREFIX_PATH="$SDK_DEV;$SYSDEPS" \
|
||||
-DCMAKE_INSTALL_PREFIX="$SDK_DEV" \
|
||||
-DLIBELF_INCLUDE_DIRS="$SYSDEPS/include" \
|
||||
-DLIBELF_LIBRARIES="$SYSDEPS/lib/libelf.so" \
|
||||
&& cmake --build build --parallel 16 \
|
||||
&& cmake --install build \
|
||||
&& cmake --install build --prefix /rocr-install --strip
|
||||
@@ -361,12 +312,12 @@ RUN cd /src/projects/clr \
|
||||
-DCMAKE_BUILD_TYPE=Release \
|
||||
-DCLR_BUILD_HIP=ON \
|
||||
-DCLR_BUILD_OCL=OFF \
|
||||
-DROCM_KPACK_ENABLED=ON \
|
||||
-DHIP_COMMON_DIR=/src/projects/hip \
|
||||
-DROCM_PATH=/opt/rocm \
|
||||
-DCMAKE_PREFIX_PATH=/opt/rocm \
|
||||
-DCMAKE_INSTALL_PREFIX=/opt/rocm \
|
||||
-DLLVM_DIR=/opt/rocm-cmake-shim \
|
||||
-DHIP_LLVM_ROOT=/opt/rocm/llvm \
|
||||
-DROCM_PATH="$SDK_DEV" \
|
||||
-DCMAKE_PREFIX_PATH="$SDK_DEV;$SYSDEPS" \
|
||||
-DCMAKE_INSTALL_PREFIX="$SDK_DEV" \
|
||||
-DHIP_LLVM_ROOT="$SDK_DEV/lib/llvm" \
|
||||
&& cmake --build build --parallel 16 \
|
||||
&& cmake --install build --prefix /clr-install --strip
|
||||
RUN mkdir -p /staging/lib \
|
||||
@@ -374,82 +325,115 @@ RUN mkdir -p /staging/lib \
|
||||
&& cp -P /clr-install/lib/libamdhip64.so* /staging/lib/ \
|
||||
&& ls -l /staging/lib
|
||||
|
||||
# rocprofiler-sdk 1.3.2 fixes torch 2.12 profiler on ROCm 7.2.3
|
||||
FROM base AS build_rocprofiler_sdk
|
||||
ARG ROCM_SYSTEMS_REPO
|
||||
ARG ROCPROFILER_SDK_COMMIT
|
||||
ARG USE_SCCACHE
|
||||
RUN apt-get update \
|
||||
&& apt-get install -y --no-install-recommends libdw-dev libelf-dev libsqlite3-dev libzstd-dev \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
RUN git init -q /src && cd /src \
|
||||
&& git remote add origin ${ROCM_SYSTEMS_REPO} \
|
||||
&& git config core.sparseCheckout true \
|
||||
&& git config remote.origin.promisor true \
|
||||
&& git config remote.origin.partialclonefilter blob:none \
|
||||
&& git sparse-checkout init --cone \
|
||||
&& git sparse-checkout set projects/rocprofiler-sdk shared cmake \
|
||||
&& git fetch --filter=blob:none --depth 1 origin ${ROCPROFILER_SDK_COMMIT} \
|
||||
&& git checkout -q FETCH_HEAD
|
||||
# Patch the ROCm 7.2.3 build of rocprofiler-sdk to include queue-interposition fixes that are
|
||||
# absent from the pinned 7.14 build. This resolves hangs on torch.profiler on ROCm.
|
||||
# TODO: Drop these once ${ROCPROFILER_SDK_COMMIT} is bumped to a build that already
|
||||
# contains them, such as ROCm 10.0. See https://github.com/vllm-project/vllm/issues/54087.
|
||||
ARG ROCPROFILER_SDK_PR_7796=fae7fd57ffb8a76e811d6410b8a0cee1e94d710a
|
||||
ARG ROCPROFILER_SDK_PR_7924=f8db22adbe17686a9b05ae03367dd1ec73011653
|
||||
RUN cd /src \
|
||||
&& HSA_PATHS='projects/rocprofiler-sdk/source/lib/rocprofiler-sdk/hsa/*' \
|
||||
&& curl -fsSL -o /tmp/qi-7796.patch "${ROCM_SYSTEMS_REPO%.git}/commit/${ROCPROFILER_SDK_PR_7796}.patch" \
|
||||
&& curl -fsSL -o /tmp/qi-7924.patch "${ROCM_SYSTEMS_REPO%.git}/commit/${ROCPROFILER_SDK_PR_7924}.patch" \
|
||||
&& git apply -p1 --include="$HSA_PATHS" --verbose /tmp/qi-7796.patch \
|
||||
&& git apply -p1 --include="$HSA_PATHS" --verbose /tmp/qi-7924.patch
|
||||
RUN cd /src/projects/rocprofiler-sdk \
|
||||
&& if [ "$USE_SCCACHE" = "1" ]; then \
|
||||
export CMAKE_C_COMPILER_LAUNCHER=sccache \
|
||||
&& export CMAKE_CXX_COMPILER_LAUNCHER=sccache; \
|
||||
fi \
|
||||
&& cmake -B build -G Ninja \
|
||||
-DCMAKE_BUILD_TYPE=Release \
|
||||
-DCMAKE_PREFIX_PATH=/opt/rocm \
|
||||
-DCMAKE_INSTALL_PREFIX=/opt/rocm \
|
||||
-DROCPROFILER_BUILD_TESTS=OFF \
|
||||
-DROCPROFILER_BUILD_SAMPLES=OFF \
|
||||
-DROCPROFILER_BUILD_DOCS=OFF \
|
||||
&& cmake --build build --parallel "$(nproc)" \
|
||||
&& cmake --install build --prefix /staging --strip
|
||||
|
||||
###
|
||||
### Final Build
|
||||
###
|
||||
|
||||
# Wheel release stage -
|
||||
# only includes dependencies used by wheel release pipeline
|
||||
FROM base AS debs_wheel_release
|
||||
RUN mkdir /app/debs
|
||||
RUN --mount=type=bind,from=build_triton,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
RUN --mount=type=bind,from=build_fa,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
RUN --mount=type=bind,from=build_aiter,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
# The wheel index also serves a torch repacked to require our triton instead of
|
||||
# the one TheRock built it with, and amdsmi, which TheRock ships only as sources
|
||||
# inside rocm-sdk-core.
|
||||
ARG TORCH_VERSION
|
||||
ARG ROCM_RELEASE_WHEELS_MULTIARCH_URL
|
||||
ARG ROCM_NIGHTLY_WHEELS_MULTIARCH_URL
|
||||
COPY tools/vllm-rocm/therock_wheels.py /tmp/therock_wheels.py
|
||||
# regex and pybase64 aren't installed here; fall back to the stdlib modules.
|
||||
RUN sed -i -e 's/import regex as re/import re/' \
|
||||
-e 's/import pybase64 as base64/import base64/' /tmp/therock_wheels.py \
|
||||
&& pip download --no-deps --dest /tmp/torch-dl \
|
||||
--index-url ${ROCM_RELEASE_WHEELS_MULTIARCH_URL} \
|
||||
--extra-index-url ${ROCM_NIGHTLY_WHEELS_MULTIARCH_URL} \
|
||||
"torch==${TORCH_VERSION}" \
|
||||
&& TRITON_VER=$(basename /app/debs/triton-*.whl | cut -d- -f2) \
|
||||
&& python3 /tmp/therock_wheels.py repack-torch /tmp/torch-dl/torch-*.whl \
|
||||
--triton-version "${TRITON_VER}" --out-dir /app/debs \
|
||||
&& python3 /tmp/therock_wheels.py patch-amdsmi "${SDK_CORE}/share/amd_smi" \
|
||||
--out-dir /tmp/amd_smi \
|
||||
&& pip wheel --no-deps --wheel-dir /app/debs /tmp/amd_smi \
|
||||
&& rm -rf /tmp/torch-dl /tmp/amd_smi /tmp/therock_wheels.py \
|
||||
&& ls -l /app/debs
|
||||
|
||||
FROM base AS debs
|
||||
RUN mkdir /app/debs
|
||||
RUN --mount=type=bind,from=build_triton,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
RUN --mount=type=bind,from=build_fa,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
RUN --mount=type=bind,from=build_aiter,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
RUN --mount=type=bind,from=build_mori,src=/app/install/,target=/install \
|
||||
cp /install/*.whl /app/debs
|
||||
|
||||
FROM base AS final
|
||||
RUN --mount=type=bind,from=debs,src=/app/debs,target=/install \
|
||||
pip install /install/*.whl
|
||||
ARG TRITON_VERSION
|
||||
|
||||
COPY --from=build_rocprofiler_sdk /staging/ /opt/rocm/
|
||||
# torch 2.12 implicitly attaches rocprofiler-sdk; its queue interposition hangs at HSA
|
||||
# teardown, so disable it. TODO: drop once ROCm's stock rocprofiler carries the fix.
|
||||
ENV ROCPROFILER_QUEUE_INTERPOSITION=0
|
||||
|
||||
COPY --from=build_rocm_runtime /staging/ /opt/rocm/
|
||||
# Hard link the patched library onto every stock file name so that both the
|
||||
# SONAME lookup and the hard coded paths resolve to it.
|
||||
COPY --from=build_rocm_runtime /staging/lib/ ${SDK_DEV}/lib/
|
||||
# Hard link the patched library onto every stock file name, in both SDK trees, so that the
|
||||
# SONAME lookup and the hard coded paths all resolve to it. Hard links rather than symlinks
|
||||
# because ldconfig rewrites SONAME symlinks but cannot re-point an inode. $SDK_CORE keeps its
|
||||
# own libhsa-runtime64 copy, which is on LD_LIBRARY_PATH and in ld.so.conf; relinking it is
|
||||
# belt and braces rather than strictly required, but it leaves no stock ROCr in the image.
|
||||
RUN set -eux; \
|
||||
cd /opt/rocm/lib; \
|
||||
cd "$SDK_DEV/lib"; \
|
||||
ours_hsa="$(readlink -f libhsa-runtime64.so.1)"; \
|
||||
ours_hip="$(readlink -f libamdhip64.so.7)"; \
|
||||
for f in libhsa-runtime64.so.1.*; do \
|
||||
[ "$f" = "${ours_hsa##*/}" ] || ln -f "$ours_hsa" "$f"; \
|
||||
done; \
|
||||
for f in libamdhip64.so.7.*; do \
|
||||
[ "$f" = "${ours_hip##*/}" ] || ln -f "$ours_hip" "$f"; \
|
||||
for f in "$SDK_DEV"/lib/libhsa-runtime64.so.1* "$SDK_CORE"/lib/libhsa-runtime64.so.1* \
|
||||
"$SDK_DEV"/lib/libamdhip64.so.7* "$SDK_CORE"/lib/libamdhip64.so.7*; do \
|
||||
[ -f "$f" ] && [ ! -L "$f" ] || continue; \
|
||||
case "${f##*/}" in libhsa*) ours="$ours_hsa";; *) ours="$ours_hip";; esac; \
|
||||
[ "$f" -ef "$ours" ] || ln -f "$ours" "$f"; \
|
||||
done
|
||||
|
||||
RUN ldconfig
|
||||
|
||||
# This is to prevent resolvers from checking METADATA and RECORD and clobbering
|
||||
# the Triton source build we installed. Constraints files are unable to help
|
||||
# with this. However, these entries are optional and can be blanked which
|
||||
# solves the problem.
|
||||
RUN TORCH_DIST_INFO=$(ls -d ${SITE_PACKAGES}/torch-*.dist-info) \
|
||||
&& grep -q '^Requires-Dist: triton' "$TORCH_DIST_INFO/METADATA" \
|
||||
&& sed -i '/^Requires-Dist: triton/d' "$TORCH_DIST_INFO/METADATA" \
|
||||
&& sed -i "s|^\(${TORCH_DIST_INFO##*/}/METADATA\),.*|\1,,|" "$TORCH_DIST_INFO/RECORD"
|
||||
|
||||
RUN --mount=type=bind,from=debs,src=/app/debs,target=/install \
|
||||
pip install /install/*.whl \
|
||||
&& ! pip check 2>&1 | grep -iE '^(torch|triton) ' \
|
||||
&& test "$(pip show triton | awk '/^Version:/ {print $2}')" = "${TRITON_VERSION}" \
|
||||
# Create a constraints file to protect the versions we need, especially
|
||||
# triton, requirements will try to reinstall repeatedly.
|
||||
&& pip list --format=freeze \
|
||||
| grep -E '^(torch|triton|torchaudio|torchvision|pybind11)==' > /etc/rocm-constraints.txt \
|
||||
&& echo "ROCm constraints:" && cat /etc/rocm-constraints.txt \
|
||||
&& test "$(wc -l < /etc/rocm-constraints.txt)" -eq 5 \
|
||||
&& grep -q '^pybind11==3\.0\.' /etc/rocm-constraints.txt \
|
||||
&& python3 -c "import torch; assert torch.version.hip, f'non-ROCm torch: {torch.__version__}'; print(f'torch={torch.__version__}')" \
|
||||
&& python3 -c "import triton; print(f'triton={triton.__version__}')" \
|
||||
&& pip list 2>/dev/null | grep -iE '^(torch|triton) '
|
||||
|
||||
ARG BASE_IMAGE
|
||||
ARG PYTHON_VERSION
|
||||
ARG ROCM_SDK_VERSION
|
||||
ARG TORCH_VERSION
|
||||
ARG TORCHAUDIO_VERSION
|
||||
ARG TORCHVISION_VERSION
|
||||
ARG PYTORCH_ROCM_ARCH
|
||||
ARG AITER_ROCM_ARCH
|
||||
ARG TRITON_BRANCH
|
||||
ARG TRITON_REPO
|
||||
ARG PYTORCH_BRANCH
|
||||
ARG PYTORCH_VISION_BRANCH
|
||||
ARG PYTORCH_REPO
|
||||
ARG PYTORCH_VISION_REPO
|
||||
ARG PYTORCH_AUDIO_BRANCH
|
||||
ARG PYTORCH_AUDIO_REPO
|
||||
ARG FA_BRANCH
|
||||
ARG FA_REPO
|
||||
ARG AITER_BRANCH
|
||||
@@ -457,23 +441,36 @@ ARG AITER_REPO
|
||||
ARG MORI_BRANCH
|
||||
ARG MORI_REPO
|
||||
ARG ROCM_SYSTEMS_REPO
|
||||
ARG ROCPROFILER_SDK_COMMIT
|
||||
ARG ROCM_RUNTIME_COMMIT
|
||||
ARG TRITON_VERSION
|
||||
# Record hashes for Triton and pybind11 so we can verify later.
|
||||
RUN echo "BASE_IMAGE: ${BASE_IMAGE}" > /app/versions.txt \
|
||||
&& echo "ROCM_SYSTEMS_REPO: ${ROCM_SYSTEMS_REPO}" >> /app/versions.txt \
|
||||
&& echo "ROCPROFILER_SDK_COMMIT: ${ROCPROFILER_SDK_COMMIT}" >> /app/versions.txt \
|
||||
&& echo "ROCM_RUNTIME_COMMIT: ${ROCM_RUNTIME_COMMIT}" >> /app/versions.txt \
|
||||
&& echo "PYTHON_VERSION: ${PYTHON_VERSION}" >> /app/versions.txt \
|
||||
&& echo "ROCM_SDK_VERSION: ${ROCM_SDK_VERSION}" >> /app/versions.txt \
|
||||
&& echo "TORCH_VERSION: ${TORCH_VERSION}" >> /app/versions.txt \
|
||||
&& echo "TORCHAUDIO_VERSION: ${TORCHAUDIO_VERSION}" >> /app/versions.txt \
|
||||
&& echo "TORCHVISION_VERSION: ${TORCHVISION_VERSION}" >> /app/versions.txt \
|
||||
&& echo "PYTORCH_ROCM_ARCH: ${PYTORCH_ROCM_ARCH}" >> /app/versions.txt \
|
||||
&& echo "AITER_ROCM_ARCH: ${AITER_ROCM_ARCH}" >> /app/versions.txt \
|
||||
&& echo "TRITON_BRANCH: ${TRITON_BRANCH}" >> /app/versions.txt \
|
||||
&& echo "TRITON_REPO: ${TRITON_REPO}" >> /app/versions.txt \
|
||||
&& echo "PYTORCH_BRANCH: ${PYTORCH_BRANCH}" >> /app/versions.txt \
|
||||
&& echo "PYTORCH_VISION_BRANCH: ${PYTORCH_VISION_BRANCH}" >> /app/versions.txt \
|
||||
&& echo "PYTORCH_REPO: ${PYTORCH_REPO}" >> /app/versions.txt \
|
||||
&& echo "PYTORCH_VISION_REPO: ${PYTORCH_VISION_REPO}" >> /app/versions.txt \
|
||||
&& echo "PYTORCH_AUDIO_BRANCH: ${PYTORCH_AUDIO_BRANCH}" >> /app/versions.txt \
|
||||
&& echo "PYTORCH_AUDIO_REPO: ${PYTORCH_AUDIO_REPO}" >> /app/versions.txt \
|
||||
&& echo "FA_BRANCH: ${FA_BRANCH}" >> /app/versions.txt \
|
||||
&& echo "FA_REPO: ${FA_REPO}" >> /app/versions.txt \
|
||||
&& echo "AITER_BRANCH: ${AITER_BRANCH}" >> /app/versions.txt \
|
||||
&& echo "AITER_REPO: ${AITER_REPO}" >> /app/versions.txt \
|
||||
&& echo "MORI_BRANCH: ${MORI_BRANCH}" >> /app/versions.txt \
|
||||
&& echo "MORI_REPO: ${MORI_REPO}" >> /app/versions.txt
|
||||
&& echo "MORI_REPO: ${MORI_REPO}" >> /app/versions.txt \
|
||||
&& echo "ROCM_SYSTEMS_REPO: ${ROCM_SYSTEMS_REPO}" >> /app/versions.txt \
|
||||
&& echo "ROCM_RUNTIME_COMMIT: ${ROCM_RUNTIME_COMMIT}" >> /app/versions.txt \
|
||||
# Fingerprint our triton so Dockerfile.rocm can prove it survived the build.
|
||||
&& echo "TRITON_VERSION: $(pip show triton | awk '/^Version:/ {print $2}')" >> /app/versions.txt \
|
||||
&& echo "TRITON_SHA256: $(find $(python3 -c 'import triton, os; print(os.path.dirname(triton.__file__))') \
|
||||
-type f \( -name '*.py' -o -name '*.so' \) | sort | xargs sha256sum | sha256sum | cut -d' ' -f1)" \
|
||||
>> /app/versions.txt \
|
||||
&& echo "PYBIND11_VERSION: $(pip show pybind11 | awk '/^Version:/ {print $2}')" >> /app/versions.txt \
|
||||
&& echo "PYBIND11_INTERNALS_VERSION: $(grep -rhE '^#[[:space:]]*define PYBIND11_INTERNALS_VERSION' \
|
||||
$(python3 -c 'import pybind11; print(pybind11.get_include())') | awk '{print $NF}')" \
|
||||
>> /app/versions.txt \
|
||||
&& echo "PYBIND11_SHA256: $(find $(python3 -c 'import pybind11; print(pybind11.get_include())') \
|
||||
-type f -name '*.h' | sort | xargs sha256sum | sha256sum | cut -d' ' -f1)" \
|
||||
>> /app/versions.txt
|
||||
|
||||
+5
-29
@@ -83,10 +83,6 @@ variable "UCX_BRANCH" {
|
||||
default = ""
|
||||
}
|
||||
|
||||
variable "ROCSHMEM_BRANCH" {
|
||||
default = ""
|
||||
}
|
||||
|
||||
variable "DEEPEP_BRANCH" {
|
||||
default = ""
|
||||
}
|
||||
@@ -95,10 +91,6 @@ variable "NIXL_CACHE_KEY" {
|
||||
default = ""
|
||||
}
|
||||
|
||||
variable "ROCSHMEM_CACHE_KEY" {
|
||||
default = ""
|
||||
}
|
||||
|
||||
variable "DEEPEP_CACHE_KEY" {
|
||||
default = ""
|
||||
}
|
||||
@@ -236,7 +228,7 @@ function "get_cache_to_rocm_rust" {
|
||||
])
|
||||
}
|
||||
|
||||
# Cache functions for upstream dependency stages (NIXL/UCX, ROCShmem, DeepEP).
|
||||
# Cache functions for upstream dependency stages (NIXL/UCX, DeepEP).
|
||||
# These stages are pinned to specific upstream commit hashes, so cache keys use
|
||||
# those hashes rather than the Buildkite commit. This means the cache persists
|
||||
# across all vLLM commits as long as the upstream dependency pins don't change.
|
||||
@@ -245,8 +237,7 @@ function "get_cache_from_rocm_deps" {
|
||||
params = []
|
||||
result = compact([
|
||||
NIXL_CACHE_KEY != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:nixl-rocm-${NIXL_CACHE_KEY}" : (NIXL_BRANCH != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:nixl-rocm-${NIXL_BRANCH}-ucx-${UCX_BRANCH}" : ""),
|
||||
ROCSHMEM_CACHE_KEY != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:rocshmem-rocm-${ROCSHMEM_CACHE_KEY}" : (ROCSHMEM_BRANCH != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:rocshmem-rocm-${ROCSHMEM_BRANCH}" : ""),
|
||||
DEEPEP_CACHE_KEY != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:deepep-rocm-${DEEPEP_CACHE_KEY}" : (DEEPEP_BRANCH != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:deepep-rocm-${DEEPEP_BRANCH}-rocshmem-${ROCSHMEM_BRANCH}" : ""),
|
||||
DEEPEP_CACHE_KEY != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:deepep-rocm-${DEEPEP_CACHE_KEY}" : (DEEPEP_BRANCH != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:deepep-rocm-${DEEPEP_BRANCH}" : ""),
|
||||
])
|
||||
}
|
||||
|
||||
@@ -257,17 +248,10 @@ function "get_cache_to_rocm_nixl" {
|
||||
])
|
||||
}
|
||||
|
||||
function "get_cache_to_rocm_rocshmem" {
|
||||
params = []
|
||||
result = compact([
|
||||
ROCSHMEM_CACHE_KEY != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:rocshmem-rocm-${ROCSHMEM_CACHE_KEY},mode=min" : (ROCSHMEM_BRANCH != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:rocshmem-rocm-${ROCSHMEM_BRANCH},mode=min" : ""),
|
||||
])
|
||||
}
|
||||
|
||||
function "get_cache_to_rocm_deepep" {
|
||||
params = []
|
||||
result = compact([
|
||||
DEEPEP_CACHE_KEY != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:deepep-rocm-${DEEPEP_CACHE_KEY},mode=min" : (DEEPEP_BRANCH != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:deepep-rocm-${DEEPEP_BRANCH}-rocshmem-${ROCSHMEM_BRANCH},mode=min" : ""),
|
||||
DEEPEP_CACHE_KEY != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:deepep-rocm-${DEEPEP_CACHE_KEY},mode=min" : (DEEPEP_BRANCH != "" ? "type=registry,ref=${DOCKERHUB_CACHE_REPO}:deepep-rocm-${DEEPEP_BRANCH},mode=min" : ""),
|
||||
])
|
||||
}
|
||||
|
||||
@@ -385,14 +369,6 @@ target "nixl-rocm-ci" {
|
||||
output = ["type=cacheonly"]
|
||||
}
|
||||
|
||||
target "rocshmem-rocm-ci" {
|
||||
inherits = ["_common-rocm", "_ci-rocm"]
|
||||
target = "build_rocshmem"
|
||||
cache-from = get_cache_from_rocm_deps()
|
||||
cache-to = get_cache_to_rocm_rocshmem()
|
||||
output = ["type=cacheonly"]
|
||||
}
|
||||
|
||||
target "deepep-rocm-ci" {
|
||||
inherits = ["_common-rocm", "_ci-rocm"]
|
||||
target = "build_deepep"
|
||||
@@ -418,7 +394,7 @@ target "ci-base-rocm-ci" {
|
||||
CI_BASE_IMAGE_TAG != "" ? "type=registry,ref=${CI_BASE_IMAGE_TAG}" : "",
|
||||
CI_BASE_TRUSTED_CONTENT_REF != "" ? "type=registry,ref=${CI_BASE_TRUSTED_CONTENT_REF}" : "",
|
||||
]),
|
||||
# Import upstream dependency caches so NIXL/ROCShmem/DeepEP stages
|
||||
# Import upstream dependency caches so NIXL/DeepEP stages
|
||||
# are cache hits even when ci_base itself needs rebuilding.
|
||||
get_cache_from_rocm_deps(),
|
||||
)
|
||||
@@ -431,5 +407,5 @@ target "ci-base-rocm-ci" {
|
||||
# Group for ci_base builds -- exports dependency stage caches alongside the
|
||||
# ci_base image so future rebuilds can reuse them independently.
|
||||
group "ci-base-rocm-ci-with-deps" {
|
||||
targets = ["nixl-rocm-ci", "rocshmem-rocm-ci", "deepep-rocm-ci", "ci-base-rocm-ci"]
|
||||
targets = ["nixl-rocm-ci", "deepep-rocm-ci", "ci-base-rocm-ci"]
|
||||
}
|
||||
|
||||
@@ -65,10 +65,6 @@ variable "UCX_BRANCH" {
|
||||
default = ""
|
||||
}
|
||||
|
||||
variable "ROCSHMEM_BRANCH" {
|
||||
default = ""
|
||||
}
|
||||
|
||||
variable "DEEPEP_BRANCH" {
|
||||
default = ""
|
||||
}
|
||||
|
||||
+1
-1
@@ -136,7 +136,7 @@ vllm bench {forward, latency, serve, throughput}
|
||||
|
||||
### forward
|
||||
|
||||
Benchmark the model forward of exact batches: each case lists its requests as `[N*]Q[@K][c][:D]` (query length Q, KV length K including the query, D draft tokens). A prefill's prefix is a prefix-cache hit unless `c` marks a chunked prefill; they differ only on models with SWA bounded replay, which recompute the last window of a hit. Each request's prefix length varies by ±50% by default (`--kv-len-jitter`), identically on every DP rank. Each forward is captured as a one-shot CUDA graph; sampling and CPU overhead aren't measured. Use `torchrun`, one process per GPU, for DP/TP/EP. CUDA only for now.
|
||||
Benchmark the model forward of exact batches: each case lists its requests as `[N*]Q[@K][c][:D]` (query length Q, KV length K including the query, D draft tokens). A prefill's prefix is a prefix-cache hit unless `c` marks a chunked prefill; they differ only on models with SWA bounded replay, which recompute the last window of a hit. Each request's prefix length varies by ±50% by default (`--kv-len-jitter`), identically on every DP rank. Each forward is captured as a one-shot CUDA graph; sampling and CPU overhead aren't measured. Use `torchrun`, one process per GPU, for DP/TP/EP. CUDA only for now. Add `--profile-dir DIR` to write a Perfetto trace and a per-kernel GPU time summary of each case; `python -m vllm.benchmarks.forward.profile A.kernels.json [B.kernels.json]` prints a summary or compares two. Under Nsight Systems, `nsys profile --capture-range=cudaProfilerApi --capture-range-end=repeat --cuda-graph-trace=node` records just those replays, one report per case.
|
||||
|
||||
```bash
|
||||
vllm bench forward \
|
||||
|
||||
+4
-37
@@ -83,7 +83,7 @@ directly from the Triton kernel through unified virtual addressing (UVA),
|
||||
prefetching on a side CUDA stream so transfers overlap with the compute of
|
||||
preceding layers. The freed GPU memory goes to the KV cache.
|
||||
|
||||
`cpu_offload` defaults to `true` and is set only through `--engram-config`.
|
||||
`cpu_offload` defaults to `true` (on)
|
||||
|
||||
To keep the tables resident on GPU instead (e.g. when host memory is
|
||||
scarce):
|
||||
@@ -93,40 +93,8 @@ vllm serve deepseek-ai/DeepSeek-V4.1-Flash \
|
||||
--engram-config.cpu_offload false
|
||||
```
|
||||
|
||||
CPU offload requires a GPU with UVA support; vLLM fails fast when it is
|
||||
unavailable.
|
||||
|
||||
## Host file gather (Qwen4Exp)
|
||||
|
||||
`host_file_gather` serves Qwen4Exp PLE rows from the loader's own file
|
||||
mapping of the safetensors shards instead of allocating the table. Set
|
||||
`--engram-config '{"host_file_gather": true}'`. It applies only to Qwen4Exp
|
||||
and takes precedence over `cpu_offload`.
|
||||
|
||||
Rows stay in the checkpoint dtype: an FP8 table stays FP8 and is dequantized
|
||||
on the GPU, and an unquantized table must match `--dtype`. CPU offload
|
||||
converts on load; this mode rejects the mismatch at load.
|
||||
|
||||
Faulted pages are reclaimable page cache, shared across co-located processes.
|
||||
They appear in each process's RSS as file-backed (`RssFile`), and PSS is
|
||||
where they are split across processes. A write to the mapping copies that
|
||||
page into anonymous RSS. Each step pays a host sync, and every TP rank
|
||||
gathers all rows. Co-located ranks share the cache; ranks on different
|
||||
machines do not.
|
||||
|
||||
Serving depends on the checkpoint files staying unchanged while the server
|
||||
runs: an in-place overwrite serves the new bytes, truncation kills the worker
|
||||
with SIGBUS, and deleting or renaming the files is safe.
|
||||
|
||||
The first touch of a row costs a disk read, and a page evicted under memory
|
||||
pressure is read again on the next miss. By default no prewarm is needed; for
|
||||
a fast first request, read the PLE files once after the server has started
|
||||
and allocated its KV cache.
|
||||
|
||||
Reload is unsupported and fails closed. ROCm and elastic EP reject the key.
|
||||
Load paths that do not hand over file-backed safetensors tensors fail at bind
|
||||
(`eager`, `.bin`, runai, `fastsafetensors`, `instanttensor`). The multithread
|
||||
loader is file-backed and is admitted. `--load-format dummy` serves zeros.
|
||||
CPU offload requires a GPU with UVA support;
|
||||
vLLM fails fast when it is unavailable.
|
||||
|
||||
## Data-parallel topologies (DeepSeek V4.1)
|
||||
|
||||
@@ -147,8 +115,7 @@ vllm serve deepseek-ai/DeepSeek-V4.1-Flash \
|
||||
```
|
||||
|
||||
`embedding_across_dp` is not supported with elastic expert parallelism yet.
|
||||
Qwen4Exp PLE tables are ETP-sharded instead and honor only `cpu_offload` and
|
||||
`host_file_gather`.
|
||||
Qwen4Exp PLE tables are ETP-sharded instead and honor `cpu_offload` only.
|
||||
|
||||
## Sharing host tables across DP replicas
|
||||
|
||||
|
||||
@@ -411,6 +411,35 @@ Example (OpenAI-compatible completions request):
|
||||
}
|
||||
```
|
||||
|
||||
## Explicit pool sizing for `SimpleCPUOffloadConnector`
|
||||
|
||||
`SimpleCPUOffloadConnector` accepts an optional `num_cpu_blocks` in
|
||||
`kv_connector_extra_config`. The scheduler and every worker use this exact
|
||||
number of offload blocks instead of automatic sizing from
|
||||
`cpu_bytes_to_use_per_rank` or `cpu_bytes_to_use / world_size`.
|
||||
Use the same positive integer on all ranks, chosen
|
||||
to fit the smallest per-worker capacity. This supports pipeline stages with
|
||||
different KV cache sizes without adding a collective during initialization.
|
||||
|
||||
```json
|
||||
{
|
||||
"kv_connector": "SimpleCPUOffloadConnector",
|
||||
"kv_role": "kv_both",
|
||||
"kv_connector_extra_config": {
|
||||
"cpu_bytes_to_use_per_rank": 1073741824,
|
||||
"num_cpu_blocks": 64,
|
||||
"lazy_offload": false
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The example count must be adjusted for the model, KV layout, and parallelism.
|
||||
The configured byte budget remains an upper bound: startup fails if the
|
||||
requested count exceeds a worker's capacity. For the `disk` backend the count
|
||||
selects disk slots and is checked against `disk_capacity_bytes`. Omitting
|
||||
`num_cpu_blocks` preserves automatic per-worker sizing. This option applies
|
||||
to `SimpleCPUOffloadConnector`, separately from `OffloadingConnector` above.
|
||||
|
||||
## Further Reading
|
||||
|
||||
- [vLLM blog: KV Offloading Connector](https://vllm.ai/blog/2026-01-08-kv-offloading-connector) — motivation, architecture (DMA-based async transfer), and benchmarks (TTFT and throughput).
|
||||
|
||||
@@ -136,6 +136,14 @@ WRITE mode is used by default. READ mode can be configured by setting `--kv-tran
|
||||
!!! note
|
||||
`notify_port` is used as a *base* port: each (DP rank, TP rank) pair within an instance uses `notify_port + offset` where the offset is based on the rank. Make sure the range starting at `notify_port` is free on the host.
|
||||
|
||||
### Hybrid attention and recurrent state
|
||||
|
||||
WRITE mode supports hybrid attention plus Mamba/KDA models, including DSpark, with equal producer and consumer TP sizes. Use matching cache layouts on both instances, `--mamba-cache-mode align`, and `read_mode: false` in each instance's `kv_connector_extra_config`. Both peers must support hybrid WRITE. Sliding-window WRITE and hybrid speculative methods other than DSpark remain unsupported.
|
||||
|
||||
For DSpark, configure the same draft model on both instances through `--speculative-config`. Target and draft attention groups retain their own block IDs and token spans; only the running recurrent state is transferred, not speculative scratch slots. The producer transfers the state after the first N-1 prompt tokens, and the consumer recomputes the last prompt token. Convolution and SSM transfers must both finish before the layer is complete.
|
||||
|
||||
Hybrid READ keeps its existing single-transferable-attention-group requirement. Plain-attention WRITE retains its existing notification format.
|
||||
|
||||
### Transport configuration
|
||||
|
||||
MoRI has two transport backends: RDMA and xGMI. You can select backend using `--kv-transfer-config.kv_connector_extra_config.backend $BACKEND`, with `$BACKEND` being `rdma` or `xgmi`. RDMA is the default backend and should be used in multi-node deployments.
|
||||
|
||||
@@ -187,7 +187,7 @@ Multi-image input can be extended to perform video captioning. We show this with
|
||||
|
||||
#### Custom RGBA Background Color
|
||||
|
||||
When loading RGBA images (images with transparency), vLLM converts them to RGB format. By default, transparent pixels are replaced with white background. You can customize this background color using the `rgba_background_color` parameter in `media_io_kwargs`.
|
||||
When loading images with transparency (including RGBA images and PNGs with a tRNS chunk), vLLM converts them to RGB format. By default, transparent pixels are replaced with white background. You can customize this background color using the `rgba_background_color` parameter in `media_io_kwargs`.
|
||||
|
||||
??? code
|
||||
|
||||
@@ -212,7 +212,7 @@ When loading RGBA images (images with transparency), vLLM converts them to RGB f
|
||||
|
||||
!!! note
|
||||
- The `rgba_background_color` accepts RGB values as a list `[R, G, B]` or tuple `(R, G, B)` where each value is 0-255
|
||||
- This setting only affects RGBA images with transparency; RGB images are unchanged
|
||||
- This setting affects images with transparency, including RGB PNGs with a transparent color; opaque RGB images are unchanged
|
||||
- If not specified, the default white background `(255, 255, 255)` is used for backward compatibility
|
||||
|
||||
#### Moondream3 Prompt Recipes { #moondream3-prompt-recipes }
|
||||
@@ -1089,7 +1089,7 @@ By passing this metadata, the model can better understand the temporal distribut
|
||||
|
||||
#### Custom RGBA Background Color
|
||||
|
||||
To use a custom background color for RGBA images, pass the `rgba_background_color` parameter via `--media-io-kwargs`:
|
||||
To use a custom background color for images with transparency, pass the `rgba_background_color` parameter via `--media-io-kwargs`:
|
||||
|
||||
```bash
|
||||
# Example: Black background for dark theme
|
||||
|
||||
@@ -201,6 +201,30 @@ checkpoint's output format.
|
||||
!!! note
|
||||
With `tool_choice="auto"`, structural-tag constraints require both `VLLM_ENFORCE_STRICT_TOOL_CALLING=true` (the default) and at least one tool with `strict: true`, or a server-side floor set via `--tool-strict-level`. When these conditions are met and the selected parser supports structural tags, vLLM constrains the tool-call envelope and pins the argument schema of each tool that sets `strict: true` (or of every tool under `--tool-strict-level parameter`). Otherwise, vLLM extracts tool calls from raw text, so arguments may occasionally be malformed or violate the function's parameter schema.
|
||||
|
||||
### Beam Models (`beam`)
|
||||
|
||||
Beam checkpoints describe their output format with `response_template` metadata, so the `beam` parser reads it like the `hf` parser does:
|
||||
|
||||
```bash
|
||||
vllm serve <model> \
|
||||
--enable-auto-tool-choice \
|
||||
--tool-call-parser beam \
|
||||
--reasoning-parser beam
|
||||
```
|
||||
|
||||
Unlike `hf`, the `beam` parser also constrains its output. For strict tools, required or named tool choice, and structured outputs, it derives a
|
||||
structural-tag grammar from the template. The grammar follows the template's tool call format, pins the arguments of strict tools when their
|
||||
encoding can be derived, allows a single call with `parallel_tool_calls=false`, and frames a structured output as the answer. Raw string arguments
|
||||
keep the `format` or `pattern` of their schema, as XGrammar keeps them in JSON strings, unless a match could end the value early. With
|
||||
`tool_choice="none"`, the grammar allows only reasoning and an answer, so a model that still sees the tools in its prompt cannot call them. When no
|
||||
grammar can be derived, required and named tool choice are rejected, and strict tools under `tool_choice="auto"` are left unconstrained with a
|
||||
warning.
|
||||
|
||||
It also constrains the output of requests without strict tools, forced tool choice, or structured outputs to the template's format: tool calls with
|
||||
free arguments or an answer, or just the rest of an answer the prompt already opened. This always-on grammar needs the `xgrammar` structured
|
||||
outputs backend, the default, and is disabled with `VLLM_BEAM_ALWAYS_CONSTRAIN_OUTPUT=0`. From the message start, every `beam` grammar opens with
|
||||
reasoning that holds visible text, unless `enable_thinking` is false.
|
||||
|
||||
### Hermes Models (`hermes`)
|
||||
|
||||
All Nous Research Hermes-series models newer than Hermes 2 Pro should be supported.
|
||||
|
||||
@@ -1,7 +1,7 @@
|
||||
<!-- markdownlint-disable MD041 MD051 -->
|
||||
--8<-- [start:installation]
|
||||
|
||||
vLLM supports AMD GPUs with ROCm 6.3 or above. Pre-built wheels are available for ROCm 7.0 and ROCm 7.2.1.
|
||||
vLLM supports AMD GPUs with ROCm 6.3 or above. Pre-built wheels are available for ROCm 10.0 (the default) and ROCm 7.2.
|
||||
|
||||
#### Prebuilt Wheels
|
||||
|
||||
@@ -9,6 +9,8 @@ vLLM supports AMD GPUs with ROCm 6.3 or above. Pre-built wheels are available fo
|
||||
| ------------ | -------------- | ------------ | ----------------- | ------------------ |
|
||||
| `rocm700` | 3.12 | 7.0 | >= 2.35 | `0.14.0` to `0.18.0` |
|
||||
| `rocm721` | 3.12 | 7.2.1 | >= 2.35 | Nightly releases after commit `171775f306a333a9cf105bfd533bf3e113d401d9` |
|
||||
| `rocm72` | 3.12 | 7.2 | >= 2.39 | Nightly releases on the legacy ROCm 7.2 stack |
|
||||
| `rocm100` | 3.12 | 10.0 | >= 2.35 | Nightly releases (default) |
|
||||
|
||||
--8<-- [end:installation]
|
||||
--8<-- [start:requirements]
|
||||
@@ -40,12 +42,17 @@ If you need a different ROCm version or want to use an existing PyTorch installa
|
||||
source .venv/bin/activate
|
||||
```
|
||||
|
||||
To install the latest version of vLLM for Python 3.12, ROCm 7.0 and `glibc >= 2.35`.
|
||||
To install the latest version of vLLM for Python 3.12, ROCm 10.0 and `glibc >= 2.35`:
|
||||
|
||||
```bash
|
||||
uv pip install vllm --extra-index-url https://wheels.vllm.ai/rocm/ --upgrade
|
||||
```
|
||||
|
||||
!!! tip
|
||||
The ROCm 10.0 wheels install the ROCm SDK from pip ([TheRock](https://github.com/ROCm/TheRock)), including the GPU kernels for every supported architecture.
|
||||
|
||||
Wheels for the legacy ROCm 7.2 stack, which use a system ROCm installation, remain available at `https://wheels.vllm.ai/rocm/${VLLM_VERSION}/rocm72`.
|
||||
|
||||
!!! tip
|
||||
You can find out about which ROCm version the latest vLLM supports by checking the `vllm` package in index in extra-index-url <https://wheels.vllm.ai/rocm/> at [https://wheels.vllm.ai/rocm/vllm](https://wheels.vllm.ai/rocm/vllm) .
|
||||
|
||||
@@ -195,9 +202,9 @@ install it without re-running the full installation:
|
||||
and place it in `vllm/vllm-rs`.
|
||||
|
||||
If you see an error about a wheel not being found, the wheel for your base
|
||||
commit and ROCm patch version might not be available. Check the available
|
||||
variants under `https://wheels.vllm.ai/rocm/<commit>/`. For example, ROCm 7.2.1
|
||||
uses the `rocm721` variant.
|
||||
commit and ROCm version might not be available. Check the available
|
||||
variants under `https://wheels.vllm.ai/rocm/<commit>/`. Variants are named by ROCm
|
||||
major and minor version, e.g. ROCm 7.2.3 uses the `rocm72` variant.
|
||||
|
||||
There are more environment variables to control the behavior of Python-only
|
||||
build:
|
||||
@@ -208,7 +215,7 @@ build:
|
||||
- `VLLM_PRECOMPILED_WHEEL_COMMIT`: override the full commit hash used to
|
||||
download the pre-compiled wheel.
|
||||
- `VLLM_PRECOMPILED_WHEEL_VARIANT`: specify the ROCm variant subdirectory, e.g.,
|
||||
`rocm700` or `rocm721`. If not specified, the variant is auto-detected based
|
||||
`rocm72` or `rocm100`. If not specified, the variant is auto-detected based
|
||||
on your system's ROCm version. An explicitly specified variant must match the
|
||||
detected environment.
|
||||
|
||||
@@ -226,23 +233,34 @@ You can find more information about vLLM's wheels in
|
||||
#### Full build (with compilation) {#full-build}
|
||||
|
||||
!!! tip
|
||||
- If you found that the following installation step does not work for you, please refer to [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base). Dockerfile is a form of installation steps.
|
||||
- The steps below follow [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base), which pins the validated version of every component. If a step does not work for you, refer to it; a Dockerfile is a form of installation steps.
|
||||
- The legacy ROCm 7.2 stack uses a system ROCm installation instead; its steps are in [docker/Dockerfile.rocm_72_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_72_base).
|
||||
|
||||
0. Install prerequisites (skip if you are already in an environment/docker with the following installed):
|
||||
0. Install the ROCm SDK and PyTorch (skip if you are already in an environment or docker image with them installed, e.g. `vllm/vllm-openai-rocm:base-nightly`).
|
||||
|
||||
- [ROCm](https://rocm.docs.amd.com/en/latest/deploy/linux/index.html)
|
||||
- [PyTorch](https://pytorch.org/)
|
||||
|
||||
For installing PyTorch, you can start from a fresh docker image, e.g, `rocm/pytorch:rocm7.0_ubuntu22.04_py3.10_pytorch_release_2.8.0`, `rocm/pytorch-nightly`. If you are using docker image, you can skip to Step 3.
|
||||
|
||||
Alternatively, you can install PyTorch using PyTorch wheels. You can check PyTorch installation guide in PyTorch [Getting Started](https://pytorch.org/get-started/locally/). Example:
|
||||
ROCm 10.0 is installed from pip ([TheRock](https://github.com/ROCm/TheRock)), together with PyTorch and the GPU kernels for your GPU's architecture. In a Python 3.12 environment:
|
||||
|
||||
```bash
|
||||
# Install PyTorch
|
||||
pip uninstall torch -y
|
||||
pip install --no-cache-dir torch torchvision --index-url https://download.pytorch.org/whl/nightly/rocm7.0
|
||||
# Your GPU's architecture, e.g. gfx942 for MI300 or gfx950 for MI350
|
||||
export GPU_ARCH=$(uvx --from rocm-bootstrap rocm-bootstrap-detect --unique)
|
||||
export ROCM_INDEX=https://stable.repo.amd.com/rocm/whl-next/
|
||||
|
||||
pip install --index-url ${ROCM_INDEX} \
|
||||
"rocm[libraries,devel,device-${GPU_ARCH}]==10.0.0" \
|
||||
"torch[device-${GPU_ARCH}]==2.12.0+rocm10.0.0" \
|
||||
"torchvision[device-${GPU_ARCH}]==0.27.0+rocm10.0.0"
|
||||
rocm-sdk init
|
||||
|
||||
# Point builds at the pip-installed ROCm SDK
|
||||
export ROCM_PATH=$(rocm-sdk path --root)
|
||||
export ROCM_HOME=${ROCM_PATH} HIP_PATH=${ROCM_PATH}
|
||||
export PATH=$(rocm-sdk path --bin):${PATH}
|
||||
```
|
||||
|
||||
!!! note
|
||||
- The validated `ROCM_SDK_VERSION`, `TORCH_VERSION` and `TORCHVISION_VERSION` can be found in [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base).
|
||||
- To support several GPU architectures, add a `device-<arch>` extra for each of them.
|
||||
|
||||
1. Install [Triton for ROCm](https://github.com/ROCm/triton.git)
|
||||
|
||||
Install ROCm's Triton following the instructions from [ROCm/triton](https://github.com/ROCm/triton.git)
|
||||
@@ -252,8 +270,7 @@ You can find more information about vLLM's wheels in
|
||||
pip uninstall -y triton
|
||||
git clone https://github.com/ROCm/triton.git
|
||||
cd triton
|
||||
# git checkout $TRITON_BRANCH
|
||||
git checkout f9e5bf54
|
||||
git checkout $TRITON_BRANCH
|
||||
if [ ! -f setup.py ]; then cd python; fi
|
||||
python3 setup.py install
|
||||
cd ../..
|
||||
@@ -265,17 +282,14 @@ You can find more information about vLLM's wheels in
|
||||
|
||||
2. Optionally, if you choose to use CK flash attention, you can install [flash attention for ROCm](https://github.com/Dao-AILab/flash-attention.git)
|
||||
|
||||
Install ROCm's flash attention (v2.8.0) following the instructions from [ROCm/flash-attention](https://github.com/Dao-AILab/flash-attention#amd-rocm-support)
|
||||
|
||||
For example, for ROCm 7.0, suppose your gfx arch is `gfx942`. To get your gfx architecture, run `rocminfo |grep gfx`.
|
||||
Install ROCm's flash attention following the instructions from [ROCm/flash-attention](https://github.com/Dao-AILab/flash-attention#amd-rocm-support)
|
||||
|
||||
```bash
|
||||
git clone https://github.com/Dao-AILab/flash-attention.git
|
||||
cd flash-attention
|
||||
# git checkout $FA_BRANCH
|
||||
git checkout 0e60e394
|
||||
git checkout $FA_BRANCH
|
||||
git submodule update --init
|
||||
GPU_ARCHS="gfx942" python3 setup.py install
|
||||
GPU_ARCHS="${GPU_ARCH}" python3 setup.py install
|
||||
cd ..
|
||||
```
|
||||
|
||||
@@ -290,7 +304,8 @@ You can find more information about vLLM's wheels in
|
||||
cd aiter
|
||||
git checkout $AITER_BRANCH_OR_COMMIT
|
||||
git submodule sync; git submodule update --init --recursive
|
||||
python3 setup.py develop
|
||||
# Keep the Triton installed in step 1
|
||||
AITER_USE_SYSTEM_TRITON=1 python3 setup.py develop
|
||||
```
|
||||
|
||||
!!! note
|
||||
@@ -300,32 +315,34 @@ You can find more information about vLLM's wheels in
|
||||
4. Optionally, if you want to use MORI for EP or PD disaggregation, you can install [MORI](https://github.com/ROCm/mori) using the following steps:
|
||||
|
||||
```bash
|
||||
pip install "setuptools_scm>=6.2"
|
||||
git clone https://github.com/ROCm/mori.git
|
||||
cd mori
|
||||
git checkout $MORI_BRANCH_OR_COMMIT
|
||||
git submodule sync; git submodule update --init --recursive
|
||||
MORI_GPU_ARCHS="gfx942;gfx950" python3 setup.py install
|
||||
# The ROCm SDK keeps its CMake configs for system dependencies (e.g. NUMA) under rocm_sysdeps
|
||||
CMAKE_PREFIX_PATH="${ROCM_PATH}:${ROCM_PATH}/lib/rocm_sysdeps" \
|
||||
MORI_GPU_ARCHS="gfx942;gfx950" python3 setup.py install
|
||||
```
|
||||
|
||||
!!! note
|
||||
- You will need to config the `$MORI_BRANCH_OR_COMMIT` for your purpose.
|
||||
- The validated `$MORI_BRANCH_OR_COMMIT` can be found in the [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base).
|
||||
|
||||
5. Build vLLM. For example, vLLM on ROCM 7.0 can be built with the following steps:
|
||||
5. Build vLLM. For example, vLLM on ROCm 10.0 can be built with the following steps:
|
||||
|
||||
???+ console "Commands"
|
||||
|
||||
```bash
|
||||
pip install --upgrade pip
|
||||
|
||||
# Build & install AMD SMI
|
||||
pip install /opt/rocm/share/amd_smi
|
||||
# Install AMD SMI, which the ROCm SDK ships as sources
|
||||
SDK_CORE=$(python3 -c "import _rocm_sdk_core, os; print(os.path.dirname(_rocm_sdk_core.__file__))")
|
||||
pip install ${SDK_CORE}/share/amd_smi
|
||||
# It loads the SDK's libamd_smi relative to its own directory
|
||||
export PYTHONPATH=${SDK_CORE}/share/amd_smi
|
||||
|
||||
# Install dependencies
|
||||
pip install --upgrade numba \
|
||||
scipy \
|
||||
huggingface-hub \
|
||||
setuptools_scm
|
||||
pip install -r requirements/rocm.txt
|
||||
|
||||
# To build for a single architecture (e.g., MI300) for faster installation (recommended):
|
||||
@@ -352,8 +369,9 @@ You can find more information about vLLM's wheels in
|
||||
vLLM offers official Docker images for deployment.
|
||||
The images can be used to run OpenAI compatible server and are available on Docker Hub as [vllm/vllm-openai-rocm](https://hub.docker.com/r/vllm/vllm-openai-rocm/tags).
|
||||
|
||||
- `vllm/vllm-openai-rocm:latest` — stable release
|
||||
- `vllm/vllm-openai-rocm:nightly` — preview build from the latest development branch, use this if you want the latest features and fixes
|
||||
- `vllm/vllm-openai-rocm:latest` — stable release, built on ROCm 10.0 (also tagged `latest-rocm100`)
|
||||
- `vllm/vllm-openai-rocm:nightly` — preview build from the latest development branch on ROCm 10.0 (also tagged `nightly-rocm100`), use this if you want the latest features and fixes
|
||||
- `vllm/vllm-openai-rocm:latest-rocm72` / `nightly-rocm72` — the same builds on the legacy ROCm 7.2 stack, kept for a transition period
|
||||
|
||||
```bash
|
||||
docker run --rm \
|
||||
@@ -409,7 +427,7 @@ You can build and run vLLM from source via the provided [docker/Dockerfile.rocm]
|
||||
??? info "(Optional) Build an image with ROCm software stack"
|
||||
|
||||
Build a docker image from [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base) which setup ROCm software stack needed by the vLLM.
|
||||
**This step is optional as this rocm_base image is usually prebuilt and store at [Docker Hub](https://hub.docker.com/r/rocm/vllm-dev) under tag `rocm/vllm-dev:base` to speed up user experience.**
|
||||
**This step is optional as this rocm_base image is usually prebuilt and store at [Docker Hub](https://hub.docker.com/r/rocm/vllm-dev) under tag `rocm/vllm-dev:base` to speed up user experience.** For the legacy ROCm 7.2 stack, the base image is built from [docker/Dockerfile.rocm_72_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_72_base) and published as `vllm/vllm-openai-rocm:base-nightly-rocm72`.
|
||||
If you choose to build this rocm_base image yourself, the steps are as follows.
|
||||
|
||||
It is important that the user kicks off the docker build using buildkit. Either the user put `DOCKER_BUILDKIT=1` as environment variable when calling docker build command, or the user needs to set up buildkit in the docker daemon configuration `/etc/docker/daemon.json` as follows and restart the daemon:
|
||||
@@ -422,7 +440,7 @@ You can build and run vLLM from source via the provided [docker/Dockerfile.rocm]
|
||||
}
|
||||
```
|
||||
|
||||
To build vllm on ROCm 7.0 for MI200 and MI300 series, you can use the default:
|
||||
To build vllm on ROCm 10.0 for all supported GPUs, you can use the default:
|
||||
|
||||
```bash
|
||||
DOCKER_BUILDKIT=1 docker build \
|
||||
@@ -441,15 +459,15 @@ It is important that the user kicks off the docker build using buildkit. Either
|
||||
}
|
||||
```
|
||||
|
||||
[docker/Dockerfile.rocm](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm) uses ROCm 7.0 by default, but also supports ROCm 5.7, 6.0, 6.1, 6.2, 6.3, and 6.4, in older vLLM branches.
|
||||
[docker/Dockerfile.rocm](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm) uses ROCm 10.0 (installed from TheRock wheels) by default. The legacy ROCm 7.2 stack is available as [docker/Dockerfile.rocm_72](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_72) with [docker/Dockerfile.rocm_72_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_72_base); older vLLM branches support ROCm 5.7 through 7.0.
|
||||
It provides flexibility to customize the build of docker image using the following arguments:
|
||||
|
||||
- `BASE_IMAGE`: specifies the base image used when running `docker build`. The default value `rocm/vllm-dev:base` is an image published and maintained by AMD. It is being built using [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base)
|
||||
- `BASE_IMAGE`: specifies the base image used when running `docker build`. The default value `rocm/vllm-dev:base` is an image published and maintained by AMD. It is being built using [docker/Dockerfile.rocm_base](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_base). For [docker/Dockerfile.rocm_72](https://github.com/vllm-project/vllm/blob/main/docker/Dockerfile.rocm_72), the default is `vllm/vllm-openai-rocm:base-nightly-rocm72`
|
||||
- `ARG_PYTORCH_ROCM_ARCH`: Allows to override the gfx architecture values from the base docker image
|
||||
|
||||
Their values can be passed in when running `docker build` with `--build-arg` options.
|
||||
|
||||
To build vllm on ROCm 7.0 for MI200 and MI300 series, you can use the default (which build a docker image with `vllm serve` as entrypoint):
|
||||
To build vllm on ROCm 10.0 for all supported GPUs, you can use the default (which build a docker image with `vllm serve` as entrypoint):
|
||||
|
||||
```bash
|
||||
DOCKER_BUILDKIT=1 docker build -f docker/Dockerfile.rocm -t vllm/vllm-openai-rocm .
|
||||
|
||||
@@ -455,6 +455,8 @@ Score templates are supported for **cross-encoder** models only. If you are usin
|
||||
|
||||
Some scoring models require a specific prompt format to work correctly. You can specify a custom score template using the `--chat-template` parameter (see [Chat Template](../../serving/online_serving/README.md#chat-template)).
|
||||
|
||||
For offline scoring, pass `chat_template` to the `LLM` constructor to use it for every `LLM.score` call. A `chat_template` passed to `LLM.score` overrides the constructor's template for that call.
|
||||
|
||||
Like chat templates, the score template receives a `messages` list. For scoring, each message has a `role` attribute—either `"query"` or `"document"`. For the usual kind of point-wise cross-encoder, you can expect exactly two messages: one query and one document. To access the query and document content, use Jinja's `selectattr` filter:
|
||||
|
||||
- **Query**: `{{ (messages | selectattr("role", "eq", "query") | first).content }}`
|
||||
@@ -464,6 +466,31 @@ This approach is more robust than index-based access (`messages[0]`, `messages[1
|
||||
|
||||
Example template file: [examples/pooling/score/template/nemotron-rerank.jinja](../../../examples/pooling/score/template/nemotron-rerank.jinja)
|
||||
|
||||
#### Templates declared by Sentence Transformers checkpoints
|
||||
|
||||
vLLM does not apply a checkpoint's chat template by default, since many rerankers inherit an unrelated template from their base LLM. The exception is a Sentence Transformers (v5.4+) `CrossEncoder` with a single `sequence-classification` Transformer module whose `sentence_bert_config.json` declares the `message` modality. Sentence Transformers formats the pairs of such checkpoints as `query`/`document` messages, so vLLM does the same with the saved template:
|
||||
|
||||
```json
|
||||
{
|
||||
"transformer_task": "sequence-classification",
|
||||
"modality_config": {
|
||||
"text": {"method": "forward", "method_output_name": "logits"},
|
||||
"message": {"method": "forward", "method_output_name": "logits", "format": "flat"}
|
||||
},
|
||||
"module_output_name": "scores",
|
||||
"processing_kwargs": {"chat_template": {"add_generation_prompt": true}}
|
||||
}
|
||||
```
|
||||
|
||||
- The default chat template is used, unless `processing_kwargs.chat_template.chat_template` names another template saved with the tokenizer (e.g. `additional_chat_templates/score.jinja` for `"score"`).
|
||||
- The other `processing_kwargs.chat_template` settings, such as `add_generation_prompt`, are passed to the template; the request's `chat_template_kwargs` override them.
|
||||
- Message contents are passed as strings, as with `--chat-template`.
|
||||
- vLLM does not restore the template suffix after truncation (`restore_suffix`). Use `max_tokens_per_doc` rather than `truncate_prompt_tokens` to truncate long documents.
|
||||
- `--chat-template` (or the `chat_template` argument of `LLM.score`) takes precedence over the saved template and its settings.
|
||||
- Models with a built-in score template keep using it.
|
||||
|
||||
The template in use is logged at startup.
|
||||
|
||||
### Enable/disable activation
|
||||
|
||||
You can enable or disable activation via `use_activation` only works for cross-encoder models.
|
||||
|
||||
@@ -313,14 +313,12 @@ Start the server:
|
||||
```shell
|
||||
vllm serve nvidia/llama-nemotron-rerank-vl-1b-v2 \
|
||||
--runner pooling \
|
||||
--trust-remote-code \
|
||||
--chat-template examples/pooling/score/template/nemotron-vl-rerank.jinja
|
||||
--trust-remote-code
|
||||
```
|
||||
|
||||
!!! note
|
||||
The chat template bundled with this checkpoint's tokenizer is not suitable
|
||||
for the Score/Rerank APIs. Use the provided override template when serving:
|
||||
`examples/pooling/score/template/nemotron-vl-rerank.jinja`.
|
||||
The checkpoint declares its score template in its Sentence Transformers config, so vLLM applies it automatically (see [Templates declared by Sentence Transformers checkpoints](scoring.md#templates-declared-by-sentence-transformers-checkpoints)).
|
||||
Revisions older than October 2026 need `--chat-template examples/pooling/score/template/nemotron-vl-rerank.jinja`.
|
||||
|
||||
Score a text query against an image document:
|
||||
|
||||
|
||||
@@ -105,7 +105,7 @@ The server keeps no state between calls. Everything the next call needs is in `s
|
||||
- **Send `prompt_token_ids` on the first chunk, even without a parser.** It seeds detokenization so the first token's leading space matches the coupled endpoint (see [Request format](#request-format)). Later chunks already carry that context in `stream_state`.
|
||||
- **Don't forward `[DONE]`.** It marks the end of the generate stream and isn't a chunk.
|
||||
|
||||
`/inference/v1/generate` returns `GenerateLogProbs` with integer `token_id`s (the generate server has no tokenizer), and derender turns those into `ChatCompletionLogProbs` / `CompletionLogProbs`, filling `token` and `bytes` from the tokenizer with the usual U+FFFD byte-fallback correction. Generate returns every top-k candidate the engine produced; derender keeps the ones the coupled endpoint would, with the same helpers: the first `top_logprobs` on chat (all of them for `top_logprobs=-1`) and the first `logprobs + 1` on completions (all of them for `logprobs=-1`). It reads those values from `chat_request` / `completion_request`, so without the original request, or with one that does not ask for logprobs, it returns every candidate. Requests that use `logprob_token_ids` get no output logprobs from generate (their `sampling_params.logprobs` is unset), so derender has none to return for them either. Streaming derender does the same per chunk on the plain detokenization path (see [Streaming state and logprobs](#streaming-state-and-logprobs)). The parser path doesn't resolve them yet and drops them.
|
||||
`/inference/v1/generate` returns `GenerateLogProbs` with integer `token_id`s (the generate server has no tokenizer), and derender turns those into `ChatCompletionLogProbs` / `CompletionLogProbs`, filling `token` and `bytes` from the tokenizer with the usual U+FFFD byte-fallback correction. Generate returns every top-k candidate the engine produced; derender keeps the ones the coupled endpoint would, with the same helpers: the first `top_logprobs` on chat (all of them for `top_logprobs=-1`) and the first `logprobs + 1` on completions (all of them for `logprobs=-1`). It reads those values from `chat_request` / `completion_request`, so without the original request, or with one that does not ask for logprobs, it returns every candidate. Requests that use `logprob_token_ids` get no output logprobs from generate (their `sampling_params.logprobs` is unset), so derender has none to return for them either. Streaming derender does the same per chunk on both the plain detokenization path and the parser path (see [Streaming state and logprobs](#streaming-state-and-logprobs)). As on `/v1/chat/completions`, a request with `include_reasoning: false` and a parser configured gets no logprobs at all, batch or streaming, since the decoded logprob tokens would reveal the hidden reasoning.
|
||||
|
||||
## Streaming cost
|
||||
|
||||
@@ -129,7 +129,7 @@ Streaming derender is stateless on the server side: all mutable state lives in t
|
||||
- `logprob_context_token_ids`: the trailing sampled token IDs (at most 4) from previous chunks, used to seed byte-fallback (U+FFFD) correction so multi-byte characters whose tokens split across chunk boundaries still resolve to real strings
|
||||
- `logprob_text_offset`: the cumulative emitted text length, so `text_offset` in completion streaming logprobs stays absolute across chunks instead of restarting at 0
|
||||
|
||||
When a streamed `GenerateTokensStreamChoice` carries `logprobs`, the integer `token_id`s are decoded per chunk and the resolved logprobs are attached to the corresponding streamed choice. For chat it is as `ChatCompletionLogProbs` and for completions it is converted to the flat `CompletionLogProbs` lists. Chunks without `logprobs` produce choices with `logprobs: null`.
|
||||
When a streamed `GenerateTokensStreamChoice` carries `logprobs`, the integer `token_id`s are decoded per chunk and the resolved logprobs are attached to the corresponding streamed choice. For chat it is as `ChatCompletionLogProbs` and for completions it is converted to the flat `CompletionLogProbs` lists. Chunks without `logprobs` produce choices with `logprobs: null`. On the parser path the chunk's logprobs are attached to whatever delta the parser emits for that chunk (reasoning, content or a tool call delta, or an empty delta while the parser is still buffering markup), which is what the coupled endpoint does; replay never touches them, since each chunk's `logprobs` cover exactly that chunk's tokens.
|
||||
|
||||
## Example
|
||||
|
||||
|
||||
@@ -1,8 +1,50 @@
|
||||
# Tokens In <> Tokens Out API
|
||||
|
||||
`/inference/v1/generate` takes prompt token IDs and returns generated token IDs. It is the generation step of the `render → generate → derender` pipeline used in disaggregated serving (see the [Renderer APIs](renderer.md) and [Derenderer APIs](derenderer.md)).
|
||||
`POST /inference/v1/generate` is the token-in/token-out (TITO) HTTP endpoint: it takes prompt token IDs and returns generated token IDs, logprobs keyed by integer token ID, and `finish_reason`/`stop_reason`, with no detokenization. Use it when the caller does its own tokenization and chat templating, such as an RL rollout worker. It is also the generation step of the `render → generate → derender` pipeline used in disaggregated serving (see the [Renderer APIs](renderer.md) and [Derenderer APIs](derenderer.md)).
|
||||
|
||||
The endpoint is registered on `vllm serve` with `--enable-scale-out`, and always with `--tokens-only`. `--tokens-only` also skips loading the tokenizer, so the server can't turn token IDs back into text.
|
||||
## Enabling the endpoint (`--enable-scale-out`, `--tokens-only`)
|
||||
|
||||
- `--enable-scale-out` registers it, with `/render`, `/derender` and `/inference/v1/abort_requests`.
|
||||
- `--tokens-only` always registers it, skips loading the tokenizer and forces `detokenize=False`. Stop strings (`sampling_params.stop`) are then rejected with a 400; use `stop_token_ids`.
|
||||
- With neither flag, it returns 403 `Scale-out endpoints are disabled. Restart vllm serve with --enable-scale-out to enable them.`
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-0.6B --enable-scale-out
|
||||
|
||||
curl -X POST http://localhost:8000/inference/v1/generate \
|
||||
-H "Content-Type: application/json" \
|
||||
-d '{"token_ids": [151644, 872, 198], "sampling_params": {"max_tokens": 32, "logprobs": 1}}'
|
||||
```
|
||||
|
||||
A full client is in [examples/scale_out/token_generation_client.py](../../../examples/scale_out/token_generation_client.py).
|
||||
|
||||
## Request fields
|
||||
|
||||
| Field | Notes |
|
||||
| --- | --- |
|
||||
| `token_ids` | Required `list[int]`: the already-templated prompt. Non-empty, no negative IDs. |
|
||||
| `sampling_params` | Required. Any [SamplingParams](../../api/README.md#inference-parameters) field, e.g. `max_tokens`, `temperature`, `logprobs`, `stop_token_ids`. Without `max_tokens`, the server applies its own default (as on the OpenAI endpoints), not 16. |
|
||||
| `output_mode` | `"tokens"` (default) or `"text"`, see [Output modes](#output-modes). |
|
||||
| `stream`, `stream_options` | SSE chunks ending with `data: [DONE]`. |
|
||||
| `return_token_ids` | Echo the final prompt IDs as `prompt_token_ids`. |
|
||||
| `features` / `content_parts` | Multimodal input, mutually exclusive. |
|
||||
| `request_id`, `model`, `cache_salt`, `priority`, `kv_transfer_params`, `ec_transfer_params` | Optional. |
|
||||
|
||||
## Response fields
|
||||
|
||||
| `choices[i]` field | Notes |
|
||||
| --- | --- |
|
||||
| `token_ids` | Generated token IDs. |
|
||||
| `finish_reason` | `stop`, `length` or `abort`. |
|
||||
| `stop_reason` | Stop token ID or string; `null` on EOS, length and abort. |
|
||||
| `logprobs` | `{"content": [{"token_id", "logprob", "rank", "top_logprobs": [...]}]}`, one entry per token, sampled token first. `token_id` is an integer, not the older `"token_id:<id>"` string. Needs `sampling_params.logprobs`. |
|
||||
| `weight_versions`, `routed_experts`, `sampling_mask` | Optional, see [Weight versions](#weight-versions); the last two need `--enable-return-routed-experts` / `--return-sampling-mask`. |
|
||||
|
||||
The top level adds `request_id`, `usage`, `prompt_token_ids` and `prompt_logprobs`. Stream chunks carry the same per-choice fields for the new tokens.
|
||||
|
||||
## Targeting a data-parallel rank (`X-data-parallel-rank`)
|
||||
|
||||
Send the header `X-data-parallel-rank: <int>` to pin a request to one data-parallel (DP) engine rank; there is no body field. Without the header, or with a non-integer value, the request is load balanced. A rank outside `[0, data_parallel_size)` is rejected.
|
||||
|
||||
## Output modes
|
||||
|
||||
|
||||
@@ -76,6 +76,10 @@ for span in completion.weight_versions:
|
||||
|
||||
The offsets cover the whole output, so with streamed delta outputs, collect the tokens before slicing. `/inference/v1/generate` returns the same list on each choice; see [Tokens In <> Tokens Out API](../serving/online_serving/token_in_token_out.md#weight-versions).
|
||||
|
||||
`/v1/completions` and `/v1/chat/completions` return it on each choice when the request sets `return_weight_versions: true`. The offsets count the choice's output tokens, which `return_token_ids` returns as `token_ids`, and a streamed choice gets the list on its last chunk.
|
||||
|
||||
An aborted request returns spans over its partial output. If you resubmit that output as part of a new prompt, the new request's spans start again at 0, so shift them by the number of tokens already generated before joining the two lists.
|
||||
|
||||
## Example
|
||||
|
||||
The [async RLHF example](../../examples/rl/rlhf_async_new_apis.py) demonstrates this pattern with `vllm.AsyncLLMEngine`, NCCL weight transfer, and mid-flight pause/resume with validation.
|
||||
|
||||
@@ -18,6 +18,8 @@ The following open-source RL libraries use vLLM for fast rollouts (sorted alphab
|
||||
|
||||
For weight synchronization between training and inference, see the [Weight Transfer](weight_transfer/README.md) documentation, which covers the pluggable backend system with [NCCL](weight_transfer/nccl.md) (multi-GPU) and [IPC](weight_transfer/ipc.md) (same-GPU) engines. To verify that an update landed on every rank, see the [Weight Checker](weight_checker.md).
|
||||
|
||||
For MoE policies, see [Routed Experts Return](routed_experts.md) (`--enable-return-routed-experts`) to get the per-token expert routing of each rollout for routing replay (R3) in the trainer.
|
||||
|
||||
For pipelining generation and training to improve GPU utilization and throughput, see the [Async Reinforcement Learning](async_rl.md) guide, which covers the pause/resume API for safely updating weights mid-flight.
|
||||
|
||||
See the following notebooks showing how to use vLLM for GRPO:
|
||||
|
||||
@@ -0,0 +1,54 @@
|
||||
# Routed Experts Return (MoE Routing Replay / R3)
|
||||
|
||||
`--enable-return-routed-experts` makes vLLM return the routed experts (the expert IDs each MoE layer's router selected) for every token of a request. RL trainers use it for MoE routing replay (R3): the training forward pass reuses the rollout's routing.
|
||||
|
||||
```bash
|
||||
vllm serve Qwen/Qwen3-30B-A3B --enable-return-routed-experts
|
||||
# offline: LLM("Qwen/Qwen3-30B-A3B", enable_return_routed_experts=True)
|
||||
```
|
||||
|
||||
The flag sets `AuxOutputConfig.enable_return_routed_experts` (full form: `--aux-output-config '{"enable_return_routed_experts": true, "max_bytes": N}'`). There is no per-request on/off switch.
|
||||
|
||||
## Requirements
|
||||
|
||||
Startup fails with a `ValueError` unless:
|
||||
|
||||
- **Model Runner V2** is used (`VLLM_USE_V2_MODEL_RUNNER=1` if vLLM falls back to V1);
|
||||
- the runner is `generate` and the model is **MoE**;
|
||||
- **prefix caching is enabled** (the default). `--no-enable-prefix-caching` fails; the old v0.21 advice to disable it is wrong.
|
||||
|
||||
Rejected at startup: adaptive speculative verification, PP > 1, DCP/PCP > 1, and the `NixlConnector`, `NixlPullConnector`, `NixlPushConnector`, `MoRIIOConnector`, `MooncakeConnector` KV connectors. Regular speculative decoding works.
|
||||
|
||||
### How it works: aux output connector
|
||||
|
||||
Routing is stored by the aux output connector (`vllm/distributed/aux_output_connector/`, configured via `--aux-output-config`), keyed by KV block hash and shared with prefix caching, which is why prefix caching is required. `max_bytes` is its host-memory LRU capacity (default: derived from KV cache capacity). If a prefix-cache hit needs evicted rows, the worker raises `BlockObjectStoreError`; increase `max_bytes`.
|
||||
|
||||
## Output format
|
||||
|
||||
| API | Field |
|
||||
| --- | --- |
|
||||
| Offline (`LLM`, `AsyncLLM`) | `CompletionOutput.routed_experts`: `np.ndarray` or `None` |
|
||||
| `/v1/completions`, `/v1/chat/completions` | `choices[i].routed_experts`: base64 `.npy`, **non-streaming only** |
|
||||
| `/inference/v1/generate` | `routed_experts` per choice (base64 `.npy`); with `stream: true`, on the final chunk |
|
||||
|
||||
- **Shape:** `(num_tokens - 1 - routed_experts_prompt_start, num_moe_layers, num_experts_per_tok)`, `num_tokens` = prompt + output. The last sampled token has no row.
|
||||
- **Rows:** row `i` is position `routed_experts_prompt_start + i` of `prompt_token_ids + output_token_ids`.
|
||||
- **dtype:** `uint8` for at most 256 experts, else `uint16`; logical expert IDs.
|
||||
- Filled only when the request finishes. `None` if the flag is off or the request aborted before any forward pass.
|
||||
|
||||
```python
|
||||
import base64, io
|
||||
import numpy as np
|
||||
|
||||
experts = np.load(io.BytesIO(base64.b64decode(choice["routed_experts"])))
|
||||
```
|
||||
|
||||
## `routed_experts_prompt_start`
|
||||
|
||||
Per-request, default `0`: drops the first N prompt-token rows (e.g. routing already received in earlier turns). Must be `0 <= value <= prompt_len`, else HTTP 400. Pass it top-level on `/v1/*completions`, inside `sampling_params` on `/inference/v1/generate`, or as `SamplingParams(routed_experts_prompt_start=...)`.
|
||||
|
||||
## Limitations
|
||||
|
||||
- The Rust HTTP frontend never returns `routed_experts` and always sends `routed_experts_prompt_start = 0`. Use the Python API server.
|
||||
|
||||
Example: [examples/rl/routed_experts_e2e.py](../../examples/rl/routed_experts_e2e.py).
|
||||
@@ -139,8 +139,9 @@ Each position then carries the sampled token's logprob and the top-128
|
||||
drop them. The finite entries are the sampling mask when it has at most `k`
|
||||
tokens.
|
||||
- The sampled token is always the first entry and may repeat inside the top-k.
|
||||
- `/inference/v1/generate` encodes IDs as `token_id:<id>`; the OpenAI
|
||||
endpoints need `return_tokens_as_token_ids`.
|
||||
- The `token_id:<id>` strings come from the OpenAI endpoints with
|
||||
`return_tokens_as_token_ids`; `/inference/v1/generate` returns an integer
|
||||
`token_id` instead.
|
||||
|
||||
## Limitations
|
||||
|
||||
|
||||
@@ -145,7 +145,7 @@ engine.send_weights() # called on every rank
|
||||
|
||||
## API Endpoints
|
||||
|
||||
When running vLLM as an HTTP server, the following endpoints are available for weight transfer. `HTTPVLLMWeightSyncClient` speaks the first four for you.
|
||||
When running vLLM as an HTTP server, the following endpoints are available for weight transfer. `HTTPVLLMWeightSyncClient` speaks the first five for you. A trainer may also implement the protocol directly without importing vLLM: see the wire contracts in [NCCL](nccl.md#trainer-without-vllm-over-http) and [IPC](ipc.md#trainer-without-vllm-over-http). This keeps the trainer decoupled from vLLM: it needs no vLLM install or version pin, only this documented HTTP and wire contract, so trainer and server can upgrade independently (non-PyTorch trainers such as JAX can use it too).
|
||||
|
||||
| Endpoint | Method | Description |
|
||||
| -------- | ------ | ----------- |
|
||||
@@ -153,6 +153,7 @@ When running vLLM as an HTTP server, the following endpoints are available for w
|
||||
| `/start_weight_update` | POST | Start a weight update |
|
||||
| `/update_weights` | POST | Transfer a batch of weights with backend-specific metadata |
|
||||
| `/finish_weight_update` | POST | Finish the update and optionally commit its `weight_version` |
|
||||
| `/shutdown_weight_transfer_engine` | POST | Release the transfer channel without stopping inference |
|
||||
| `/update_weight_version` | POST | Update `weight_version` without changing model weights |
|
||||
| `/weight_info` | GET | Get the latest committed weight version |
|
||||
| `/pause` | POST | Pause generation before weight sync to handle inflight requests |
|
||||
@@ -162,6 +163,12 @@ When running vLLM as an HTTP server, the following endpoints are available for w
|
||||
!!! note
|
||||
The HTTP weight transfer endpoints require `VLLM_SERVER_DEV_MODE=1` to be set.
|
||||
|
||||
!!! warning
|
||||
If `/update_weights` fails, the model is left mid-reload while `/health`
|
||||
stays 200, and `/finish_weight_update` is rejected. Before generating again,
|
||||
resend a full `/start_weight_update` → `/update_weights` →
|
||||
`/finish_weight_update` (see [#57854](https://github.com/vllm-project/vllm/pull/57854)).
|
||||
|
||||
The Rust frontend's optional gRPC `Control` service exposes the same pause, sleep, weight-transfer, and weight-version lifecycle for trusted sidecars. The `ServerInfo.rl_capabilities` response reports whether weight transfer and sleep mode were configured. Backend-specific `init_info` and `update_info` remain JSON metadata; model tensors continue to move over the configured NCCL, IPC, sparse-NCCL, or sharded-RDT transport.
|
||||
|
||||
## Extending the System
|
||||
|
||||
@@ -110,6 +110,43 @@ mode carries a per-chunk barrier across ranks: without it, a rank could overwrit
|
||||
its buffer while its colocated worker is still reading the current chunk. This is
|
||||
handled inside `send_weights()`.
|
||||
|
||||
## Trainer Without vLLM over HTTP
|
||||
|
||||
A colocated trainer can speak the [HTTP endpoints](README.md#api-endpoints)
|
||||
directly, with only PyTorch and an HTTP client. The server needs
|
||||
`VLLM_ALLOW_INSECURE_SERIALIZATION=1`; without it, init and start return 200 and
|
||||
the first `/update_weights` returns 500. Reference:
|
||||
[`rlhf_http_ipc_no_vllm.py`](../../../examples/rl/rlhf_http_ipc_no_vllm.py).
|
||||
|
||||
`POST /init_weight_transfer_engine` body: `{"init_info": {"packed": false}}`.
|
||||
|
||||
`POST /update_weights` body `{"update_info": {...}}`:
|
||||
|
||||
| Field | Meaning |
|
||||
| ----- | ------- |
|
||||
| `names`, `dtype_names`, `shapes` | Checkpoint names, `torch` dtype attribute names (`"bfloat16"`), full shapes |
|
||||
| `ipc_handles_pickled` | `base64(pickle.dumps(handles))`. Unpacked: a list with one dict per name. Packed: one dict for the shared buffer |
|
||||
| `tensor_sizes` | Packed only: byte size of each tensor in the chunk, in `names` order |
|
||||
|
||||
- A handle dict is `{gpu_uuid: reduce_tensor(t)[1]}` with one entry per GPU the
|
||||
server uses (`torch.multiprocessing.reductions.reduce_tensor`).
|
||||
`gpu_uuid = str(torch.cuda.get_device_properties(i).uuid)`, **without** the
|
||||
`GPU-` prefix that `nvidia-smi` prints. The worker overwrites `args[6]`
|
||||
(device index) with its own.
|
||||
- `ipc_handles` (unpickled) is not JSON-serializable; over HTTP use
|
||||
`ipc_handles_pickled`.
|
||||
- Packed: export one `uint8` buffer once and reuse it for every chunk. Copy
|
||||
`t.contiguous().view(-1).view(torch.uint8)` in `names` order and close a chunk
|
||||
**before** the next tensor would overflow it (unlike NCCL, which closes after).
|
||||
A tensor larger than the buffer is an error. Synchronize before each
|
||||
`/update_weights`; do not overwrite the buffer until the call returns.
|
||||
- Keep every exported tensor alive until `/finish_weight_update` returns, then
|
||||
drop the references and call `torch.cuda.ipc_collect()`.
|
||||
- Multi-rank trainers: each rank exports handles for its own GPU, the ranks
|
||||
gather the `{uuid: args}` dicts to rank 0 over their own `torch.distributed`
|
||||
group and merge them per index, and rank 0 sends all HTTP. In packed mode,
|
||||
barrier before and after each `/update_weights`.
|
||||
|
||||
## Rank-Local Updates
|
||||
|
||||
For different parameter subsets per worker, `update_info` may be a list indexed
|
||||
@@ -119,4 +156,5 @@ worker to participate.
|
||||
## Examples
|
||||
|
||||
- [RLHF with IPC weight syncing (`vllm serve`, HTTP)](../../../examples/rl/rlhf_http_ipc.py) - **Start here.** Server and training model share a single GPU; HTTP control plane, CUDA IPC data plane. Launches and tears down its own server
|
||||
- [IPC weight syncing from a trainer that does not import vLLM](../../../examples/rl/rlhf_http_ipc_no_vllm.py) - `torch` + `requests` only, `ipc_handles_pickled` over HTTP; see [Trainer Without vLLM over HTTP](#trainer-without-vllm-over-http)
|
||||
- [RLHF with IPC + FSDP2 and expert parallelism](../../../examples/rl/rlhf_ipc_fsdp_ep.py) - Multi-rank trainer colocated with a `--data-parallel-size 4` server on the same 4 GPUs: every FSDP rank builds an engine and joins the handle all-gather, with packed chunking and sleep/wake around the transfer
|
||||
|
||||
@@ -10,18 +10,14 @@ The `nccl_m2n` weight transfer engine uses [NCCL M2N](https://github.com/NVIDIA/
|
||||
|
||||
## Requirements
|
||||
|
||||
This backend depends on the `nccl-extensions` Python package (and its `nccl4py` dependency), which vLLM does not install. It needs NCCL 2.30.5 or newer, and `libnccl_m2n.so` is linked against a specific NCCL build, so vLLM must load the same library:
|
||||
|
||||
```bash
|
||||
export VLLM_NCCL_SO_PATH=/path/to/the/same/libnccl.so
|
||||
```
|
||||
This backend depends on the `nccl-extensions` Python package (and its `nccl4py` dependency), which vLLM does not install. Install both against a compatible NCCL 2.30.5 or newer runtime. The backend creates its data-plane communicator through `nccl4py` and passes that communicator directly to `nccl.m2n`.
|
||||
|
||||
The backend imports the runtime lazily, so vLLM is unaffected unless you select it.
|
||||
|
||||
## How It Works
|
||||
|
||||
1. The trainer ranks and all inference workers join **one** NCCL communicator via `StatelessProcessGroup`. Trainer ranks occupy `[0, T)` and workers `[T, T + N)` — m2n meshes are contiguous rank intervals, which is exactly what vLLM's existing `rank_offset` convention produces.
|
||||
2. At the init handshake the trainer ships the full transfer plan: both meshes — one per side, shared by every parameter — then each parameter's name, dtype, shape and placement. The workers resolve the matching **destination placements** and publish them back over the shared group; the meshes themselves are declared, not negotiated.
|
||||
1. A `StatelessProcessGroup` exchanges metadata and the NCCL unique ID. The trainer ranks and all inference workers then create **one** `nccl.core.Communicator`: trainer ranks occupy `[0, T)` and workers `[T, T + N)`. The communicator is passed directly to each M2N collective.
|
||||
2. At the init handshake the trainer ships the full transfer plan: the session-wide destination mesh, then each parameter's name, dtype, shape, source mesh, and source placements. The workers resolve the matching **destination placements** and publish them over the data-plane communicator; the meshes themselves are declared, not negotiated.
|
||||
3. Each round, both sides issue one `reshard` per parameter in the same order. Every rank in the communicator participates in every reshard, so — as with the NCCL engine — the worker must be inside `update_weights` while the trainer sends. The trainer engine handles that concurrency itself.
|
||||
|
||||
### Destination resolution
|
||||
@@ -83,19 +79,28 @@ engine.send_weights()
|
||||
|
||||
`send_weights()` drives the whole round — `start_weight_update`, `update_weights`, `finish_weight_update`. Every rank in the communicator must be inside the same reshard, so `update_weights` has to be in flight while the trainer sends; the engine runs it on a helper thread so the trainer loop does not have to open-code that.
|
||||
|
||||
Trainers with a custom producer (a Megatron export, MoE re-fusing) subclass `M2NWeightSource` and supply layouts explicitly.
|
||||
Trainers with a custom producer (a Megatron export, MoE re-fusing) subclass
|
||||
`M2NWeightSource` and supply layouts explicitly. A source may yield any
|
||||
nccl4py-compatible CUDA buffer: a torch tensor, a CUDA Array Interface object,
|
||||
or a raw device pointer. vLLM keeps buffer owners alive through stream
|
||||
completion, using allocator stream tracking when the buffer provides it.
|
||||
Sources must provide buffers on the communicator's current device; nccl4py
|
||||
resolves their pointers but does not verify device ownership. Raw-pointer
|
||||
sources are additionally responsible for pointer lifetime and matching their
|
||||
declared dtype and local shape.
|
||||
|
||||
See [`examples/rl/rlhf_m2n.py`](../../../examples/rl/rlhf_m2n.py) for a runnable FSDP → TP example.
|
||||
|
||||
## Limitations
|
||||
|
||||
- Tensor rank 1..3; 4-D parameters are rejected.
|
||||
- dtypes: int8/uint8, fp8 e4m3/e5m2, fp16, bf16, int32/uint32, fp32, int64/uint64, fp64. No fp4.
|
||||
- The trainer's device mesh must cover a contiguous rank interval starting at 0, and be 1-D or 2-D.
|
||||
- vLLM validates tensor-rank and dtype support while agreeing on the complete
|
||||
plan; the installed M2N runtime validates them again for each reshard.
|
||||
- Each parameter's source mesh must cover the same contiguous trainer-rank interval starting at 0, and be 1-D or 2-D.
|
||||
- `Partial` placements are not supported.
|
||||
- Models with known model-level checkpoint preprocessing use the full-tensor
|
||||
fallback for the affected module.
|
||||
- A reshard cannot be issued from inside a CUDA graph capture (weight updates run outside capture, so this only matters for custom callers).
|
||||
- The reshard plan is capped per *shard*, not per mesh: at most 16 source shards may feed one destination shard, and one source shard may feed at most 64 destination shards (`MAX_SOURCES` / `MAX_TARGETS` in the m2n build; raising either needs a rebuild).
|
||||
- A directly-resharded parameter spreads over `tp_size` destination shards, so each one is fed by fewer source shards. A parameter on the fallback path is replicated, i.e. a single destination shard fed by every source shard, so a trainer that **sharded** its parameters is still limited to 16 ranks whenever any parameter falls back — which fused parameters always do. A replicated trainer is unaffected.
|
||||
- vLLM checks every parameter's shard overlap against conservative 16-source
|
||||
and 64-target capacities before committing the plan. The installed M2N
|
||||
runtime remains authoritative and validates each reshard again.
|
||||
- Each communicator holds a cached staging pool (~256 MiB by default: 4 channels x 64 MiB).
|
||||
|
||||
@@ -130,6 +130,51 @@ pass — this is the invariant to test first.
|
||||
IPC does not read `metadata()` at all: it derives the update info from
|
||||
iteration as it goes, so it cannot observe a divergence.
|
||||
|
||||
## Trainer Without vLLM over HTTP
|
||||
|
||||
A trainer can speak the [HTTP endpoints](README.md#api-endpoints) directly, with
|
||||
only PyTorch and an HTTP client. It mints the `ncclUniqueId` itself and is NCCL
|
||||
rank 0. Reference: [`rlhf_http_nccl_no_vllm.py`](../../../examples/rl/rlhf_http_nccl_no_vllm.py).
|
||||
|
||||
`POST /init_weight_transfer_engine` body `{"init_info": {...}}`:
|
||||
|
||||
| Field | Default | Meaning |
|
||||
| ----- | ------- | ------- |
|
||||
| `nccl_unique_id_b64` | — | Standard (not URL-safe) base64 of the 128-byte `ncclUniqueId`. Replaces `master_address` / `master_port` |
|
||||
| `rank_offset` | — | First worker rank; `1` for one trainer rank |
|
||||
| `world_size` | — | Trainer ranks + `GET /get_world_size` |
|
||||
| `packed` | `false` | Packed broadcasting |
|
||||
| `packed_buffer_size_bytes` | 1 GiB | Must match the trainer's chunking |
|
||||
| `packed_num_buffers` | 2 | Worker-side receive buffers |
|
||||
|
||||
- Worker rank = `rank_offset + dp_index * (TP * PP) + rank`.
|
||||
- `/init_weight_transfer_engine` returns only after all ranks join: send it in
|
||||
the background while the trainer calls `ncclCommInitRank(world_size, uid, 0)`.
|
||||
There is no timeout; if the trainer never joins, the workers wait forever.
|
||||
- Right after init every worker issues a warm-up `all_reduce` (sum) of
|
||||
`torch.zeros(1)` (one float32 element). The trainer must issue the same one
|
||||
before any other collective.
|
||||
|
||||
Each sync: `POST /start_weight_update`, then `POST /update_weights` with
|
||||
`{"update_info": {"names", "dtype_names", "shapes"}}` **in the background**
|
||||
while rank 0 broadcasts in `names` order, then `POST /finish_weight_update`.
|
||||
`dtype_names` are `torch` attribute names (`"bfloat16"`); `shapes` are full
|
||||
checkpoint shapes.
|
||||
|
||||
- Unpacked: one broadcast per tensor.
|
||||
- Packed: concatenate `t.contiguous().view(-1).view(torch.uint8)` in `names`
|
||||
order; broadcast a chunk as soon as its size **exceeds**
|
||||
`packed_buffer_size_bytes` (the crossing tensor stays in the chunk), then the
|
||||
remainder.
|
||||
|
||||
Trainer-side communicators without vLLM:
|
||||
|
||||
| Communicator | Status |
|
||||
| ------------ | ------ |
|
||||
| `torch.distributed.ProcessGroupNCCL` on a private `TCPStore` + `eager_connect_single_device` | Works; the unique id is read from store key `"0"`, a torch implementation detail (used by the example) |
|
||||
| `ctypes` against torch's bundled `libnccl.so.2` | Works |
|
||||
| `torch.cuda.nccl.init_rank` | Fails on Python 3.10+ (`PY_SSIZE_T_CLEAN` `SystemError`) |
|
||||
|
||||
## Sparse NCCL
|
||||
|
||||
Sparse, flat-index weight patches use
|
||||
@@ -196,6 +241,7 @@ Sparse NCCL sends only `O(nnz)` indices and values. Checkpoint application still
|
||||
## Examples
|
||||
|
||||
- [RLHF with NCCL weight syncing (`vllm serve`, HTTP)](../../../examples/rl/rlhf_http_nccl.py) - **Start here.** Trainer on one GPU, 2x tensor-parallel fp8 server on two others; HTTP control plane, NCCL data plane. Launches and tears down its own server
|
||||
- [NCCL weight syncing from a trainer that does not import vLLM](../../../examples/rl/rlhf_http_nccl_no_vllm.py) - `torch` + `requests` only, `nccl_unique_id_b64` rendezvous; see [Trainer Without vLLM over HTTP](#trainer-without-vllm-over-http)
|
||||
- [RLHF with NCCL + FSDP2 and expert parallelism](../../../examples/rl/rlhf_nccl_fsdp_ep.py) - Multi-rank trainer: every FSDP rank builds an engine and joins the `full_tensor()` gather while only rank 0 touches the wire
|
||||
- [RLHF with sparse NCCL weight syncing (offline, Ray)](../../../examples/rl/rlhf_sparse_nccl.py) - Qwen3 MoE per-expert checkpoint updates with one trainer GPU and two TP2/EP2 inference GPUs
|
||||
- [RLHF with async weight syncing (offline, Ray)](../../../examples/rl/rlhf_async_new_apis.py) - Async generation with mid-flight pause, weight sync, resume, and validation against a fresh model; uses an in-process `AsyncLLMEngine` with `RayVLLMWeightSyncClient`
|
||||
|
||||
@@ -423,9 +423,9 @@ Most cache paths default to subdirectories under a single root. Changing `VLLM_C
|
||||
|
||||
| Environment Variable | Default | Description |
|
||||
| --- | --- | --- |
|
||||
| `VLLM_CACHE_ROOT` | `~/.cache/vllm` | Base cache directory. Respects `XDG_CACHE_HOME` if set. All paths below inherit from this unless explicitly overridden. |
|
||||
| `VLLM_CACHE_ROOT` | `~/.cache/vllm` | Base directory for vLLM-owned caches. Respects `XDG_CACHE_HOME` if set. |
|
||||
| *(torch.compile)* | `$VLLM_CACHE_ROOT/torch_compile_cache/` | Compilation cache for AOT-compiled models, Inductor graphs, and Triton kernels. Controlled by `VLLM_DISABLE_COMPILE_CACHE` (set to `1` to disable). |
|
||||
| `VLLM_FLASHINFER_AUTOTUNE_CACHE_DIR` | `$VLLM_CACHE_ROOT/flashinfer_autotune_cache/<flashinfer-version>/<arch>/<cache-hash>/` | FlashInfer autotune config cache. |
|
||||
| `VLLM_FLASHINFER_AUTOTUNE_CACHE_DIR` | Managed: `~/.cache/flashinfer/autotune/`; legacy: `$VLLM_CACHE_ROOT/flashinfer_autotune_cache/<flashinfer-version>/<arch>/<cache-hash>/` | Overrides cache placement for both FlashInfer autotune cache APIs. |
|
||||
| `VLLM_ASSETS_CACHE` | `$VLLM_CACHE_ROOT/assets/` | Downloaded assets (e.g., tokenizer files). |
|
||||
| `VLLM_XLA_CACHE_PATH` | `$VLLM_CACHE_ROOT/xla_cache/` | XLA/TPU compilation cache. |
|
||||
| `VLLM_MEDIA_CACHE` | *(disabled)* | Optional cache for downloaded media (images, video, audio). Not enabled unless explicitly set. |
|
||||
|
||||
@@ -473,9 +473,11 @@ async def stop_profile():
|
||||
if __name__ == "__main__":
|
||||
parser = argparse.ArgumentParser()
|
||||
parser.add_argument("--port", type=int, default=10001)
|
||||
# Port the prefill/decode servers register on (their proxy_ping_port).
|
||||
parser.add_argument("--service-discovery-port", type=int, default=36367)
|
||||
args = parser.parse_args()
|
||||
|
||||
t = start_service_discovery("0.0.0.0", 36367)
|
||||
t = start_service_discovery("0.0.0.0", args.service_discovery_port)
|
||||
# High-concurrency hardening. Quart's app.run() uses a shallow listen
|
||||
# backlog (100) and, with app.debug=True, adds per-request overhead that
|
||||
# starves the single accept loop. Under a burst of ~512 simultaneous client
|
||||
|
||||
@@ -340,7 +340,10 @@ async def _handle_completions(api: str, request: Request):
|
||||
):
|
||||
yield chunk
|
||||
|
||||
return StreamingResponse(generate_stream(), media_type="application/json")
|
||||
media_type = (
|
||||
"text/event-stream" if req_data.get("stream") else "application/json"
|
||||
)
|
||||
return StreamingResponse(generate_stream(), media_type=media_type)
|
||||
|
||||
except Exception as e:
|
||||
import sys
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user