MOVZXandHinata
1281bf9c03
The default sidecar pointed at qsa_kv_scales_swift.json, a name no checkpoint on this box carries. The guard in start.sh then stopped every start with "scales sidecar missing", and a start without the guard would run FP8 KV at scale 1.0. Point the default at qsa_kv_scales.json, the file both checkpoints ship, and rename the sidecar in the start.sh header, both READMEs, and the Dockerfile comment. Name SCALES in the guard message. Co-Authored-By: Hinata <hinata@idihore.id> Signed-off-by: MOVZX <movzx@yahoo.com>
104 lines
6.7 KiB
Docker
104 lines
6.7 KiB
Docker
# syntax=docker/dockerfile:1
|
|
# Three named build contexts feed this image. docker-compose.yml passes all three by itself:
|
|
#
|
|
# ops the compiled kernels from build.sh step 8 (this repo's artifacts/). The image then
|
|
# installs them instead of compiling: ~2 min.
|
|
# uvcache $HOME/.cache/uv ─┐ two wheel caches, so the build does not re-download the
|
|
# pipcache $HOME/.cache/pip ─┘ torch + CUDA wheel set (~6 GB). Read only.
|
|
#
|
|
# Plain `docker build -t vllm:latest .` passes none of them: the ops context is empty, so the
|
|
# image compiles every kernel from source, and the wheel caches start empty. BuildKit keeps the
|
|
# caches in its own volumes after the first build, so a rebuild downloads nothing again.
|
|
#
|
|
# docker compose up -d # kernels installed
|
|
# docker build -t vllm:latest . # compiles
|
|
# docker build --build-context ops="$PWD/artifacts" -t vllm:latest . # no compile
|
|
#
|
|
# A cache mount copies its context into the BuildKit cache volume the first time and reuses it
|
|
# after that. The caches on the host are only read: the build never writes to them.
|
|
#
|
|
# build.sh compares the tree object, the arch, CPython, CUDA and torch in the manifest with the
|
|
# tree it installs into, and hashes every unpacked file. A tarball that does not fit this tree
|
|
# stops the build. The image build runs with PACK=0: an image layer must not carry a 340 MB tarball.
|
|
FROM scratch AS uvcache
|
|
FROM scratch AS pipcache
|
|
FROM scratch AS ops
|
|
# trixie, not bookworm: the kernels of a host build on Debian 13 need glibc 2.38, bookworm has 2.36,
|
|
# so the loader would refuse them at serve time. trixie has 2.41.
|
|
FROM python:3.13-slim-trixie
|
|
# vllm: Qwen3.8-Flash-Next-W4A16 at 262K on 4x RTX 3090 (vLLM 0.30.0 + patches/).
|
|
#
|
|
# docker build -t vllm:latest .
|
|
# docker run -d --name vllm --gpus all --stop-timeout 20 --shm-size=8g \
|
|
# --ulimit memlock=-1 -p 8080:8000 \
|
|
# -v /data/models/halt95/Swift1.5-Qwen3.8-Flash-Next-W4A16-Merlin:/models/model:ro \
|
|
# -v "$PWD/.vllm-cache":/cache \
|
|
# -e MODEL_NAME=Hinata vllm:latest
|
|
#
|
|
# The checkpoint mount must contain qsa_kv_scales.json (the FP8 KV scales sidecar).
|
|
#
|
|
# The build fetches vLLM v0.30.0 and applies patches/ (build.sh step 2; needs network), then compiles
|
|
# every kernel of this tree from source (build.sh: cmake + ninja + nvcc 13.0 from the venv's
|
|
# CUDA wheels, arch 8.6). No prebuilt binaries go in. Budget 20-40 min and ~15 GB of build disk.
|
|
# Pass the ops context (see above) to install prebuilt kernels instead: ~2 min, same tree.
|
|
# trixie ships cmake 3.25, which is too old, so cmake and ninja come from pip.
|
|
#
|
|
# Layers are split so a rebuild stays cheap. The long compile only re-runs when a file it reads
|
|
# changes (build.sh, scripts/, patches/, start.sh). Docs, README, compose and
|
|
# the example env land in a later COPY, so editing them costs seconds. The apt, pip, uv and ccache
|
|
# caches are BuildKit cache mounts: they live outside the image, so a rebuild downloads nothing
|
|
# again and recompiles only the kernels whose source changed (build.sh prints the ccache stats).
|
|
ARG MAX_JOBS=32
|
|
ARG ARCH=8.6
|
|
ENV DEBIAN_FRONTEND=noninteractive PIP_NO_CACHE_DIR=1
|
|
RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \
|
|
--mount=type=cache,target=/var/lib/apt,sharing=locked \
|
|
apt-get update && apt-get install -y --no-install-recommends git curl ca-certificates tar gzip procps \
|
|
build-essential ccache binutils
|
|
WORKDIR /opt/vllm
|
|
# The build tools. These two mounts have their own ids: a cache mount with the same target path
|
|
# shares one volume, and a from= seed applies only when the volume is first created. Without the
|
|
# ids, this layer would create the pip/uv volumes empty and the seeds of the compile layer below
|
|
# would never be read. (PIP_NO_CACHE_DIR= (empty) makes pip use the cache.)
|
|
# build.sh uses uv for the venv when it is on PATH; it makes the 196-pin install several times faster.
|
|
# cmake<4: pip ships 4.x, the tree and its vendored projects are built and tested on 3.x.
|
|
RUN --mount=type=cache,id=tools-pip,target=/root/.cache/pip,sharing=locked \
|
|
--mount=type=cache,id=tools-uv,target=/root/.cache/uv,sharing=locked \
|
|
PIP_NO_CACHE_DIR= python -m pip install uv 'cmake<4' ninja
|
|
# The build inputs, in one layer. Nothing here changes on a docs edit.
|
|
COPY build.sh start.sh ./
|
|
COPY scripts/ scripts/
|
|
COPY patches/ patches/
|
|
RUN --mount=type=cache,id=compile-uv,target=/root/.cache/uv,from=uvcache,sharing=locked \
|
|
--mount=type=cache,id=compile-pip,target=/root/.cache/pip,from=pipcache,sharing=locked \
|
|
--mount=type=cache,id=compile-ccache,target=/root/.cache/ccache \
|
|
--mount=type=bind,from=ops,target=/ops,ro \
|
|
chmod +x start.sh build.sh scripts/*.sh scripts/*.py \
|
|
&& echo "==> caches at the start of the compile:" && (du -sh /root/.cache/uv /root/.cache/pip /root/.cache/ccache 2>/dev/null || true) \
|
|
&& ops_tar="$(ls -1 /ops/build-artifacts-*.tar.gz 2>/dev/null | head -1 || true)" \
|
|
&& if [ -n "$ops_tar" ]; then echo "==> ops source: $ops_tar (prebuilt, no compile)"; else echo "==> ops source: none passed, compiling from source"; fi \
|
|
&& PIP_NO_CACHE_DIR= MAX_JOBS="$MAX_JOBS" ARCH="$ARCH" PACK=0 ARTIFACTS="$ops_tar" bash build.sh \
|
|
&& rm -rf /opt/vllm/vllm/.deps /opt/vllm/vllm/CMakeUserPresets.json \
|
|
/opt/vllm/build-*.log
|
|
# The record build.sh writes (vllm/.build-products, vllm/.build-products.sha256) is KEPT on
|
|
# purpose: start.sh re-hashes every kernel against it before it starts the engine.
|
|
# Everything else: README, Dockerfile, compose, .env.example, systemd/. A copy, so it is cheap.
|
|
COPY . /opt/vllm/
|
|
RUN chmod +x start.sh build.sh scripts/*.sh scripts/*.py \
|
|
&& rm -rf /root/.cache
|
|
ENV TREE=/opt/vllm/vllm VENV=/opt/vllm/.venv HOST=0.0.0.0 PORT=8000 CACHE_ROOT=/cache \
|
|
MODEL_NAME=Hinata MODEL_MOUNT_PATH=/models/model LOG= \
|
|
CUDA_HOME=/opt/vllm/.venv/lib/python3.13/site-packages/nvidia/cu13 \
|
|
FLASHINFER_WORKSPACE_BASE=/cache/flashinfer TRITON_CACHE_DIR=/cache/triton
|
|
ENV PATH="/opt/vllm/.venv/lib/python3.13/site-packages/nvidia/cu13/bin:$PATH"
|
|
VOLUME /cache
|
|
EXPOSE 8000
|
|
# /v1/models answers 200 even when the engine is dead: probe a real one-token generation.
|
|
HEALTHCHECK --interval=60s --timeout=90s --start-period=30m --retries=3 \
|
|
CMD m="${MODEL_NAME%% *}"; m="${m:-Hinata}"; curl -fs -m 85 -X POST "http://127.0.0.1:${PORT}/v1/chat/completions" \
|
|
-H "Content-Type: application/json" ${VLLM_API_KEY:+-H "Authorization: Bearer $VLLM_API_KEY"} \
|
|
-d "{\"model\":\"$m\",\"messages\":[{\"role\":\"user\",\"content\":\"ok\"}],\"max_tokens\":1,\"chat_template_kwargs\":{\"enable_thinking\":false}}" \
|
|
| grep -q '"choices"' || exit 1
|
|
ENTRYPOINT ["/opt/vllm/start.sh"]
|
|
CMD ["/models/model"]
|