vLLM-4x-RTX-3090/README.md

iHoree-vLLM-4xRTX3090

Swift-1.5-Qwen3.8-Flash-Next-W4A16 served from four RTX 3090 cards. The model is a 125B MoE with 4-bit weights. It runs TP2xPP2 + EP with MTP K=3. The FP8 KV pool holds 924,993 tokens at a 262,144-token context. Single-stream output measured at 158-184 tok/s.

An Indonesian translation is in README-ID.md. This file is the source. When the two differ, this file wins.

1. Requirements

Item	Requirement
GPU	4 free cards of 24 GB, all the same model
RAM	96 GiB or more. The PLE table uses 69 GiB of host RAM.
/dev/shm	1 GB or more
Driver	CUDA 13.0 or newer
Disk	30 GB free for the build, 115 GB for the checkpoint
Python	3.13
Build tools	cmake 3.26 or newer, ninja, cc, c++

The checkpoint must contain qsa_kv_scales.json. That file holds the FP8 KV scales. Swift 1.5 changed k_proj/v_proj, so the scale file of one checkpoint does not work for another.

SCALES names another sidecar. Give a host path; Compose mounts it into the container. The default is $MODEL_DIR/qsa_kv_scales.json.

2. How to Build

sudo apt-get install -y cmake ninja-build ccache
./build.sh

build.sh runs eight steps: the tools, the serving tree, the venv, the pinned runtime, the CUDA toolchain, the compiled ops, the install, and the kernel pack. The first compile takes 20 minutes on 64 cores CPU.

Two variables control the compile:

1. MAX_JOBS — the number of compile jobs. The default is 32. Lower it when the build runs out of RAM.
2. ARCH — the GPU architecture. The default is 8.6 (Ampere).

The build prints every step. The installer runs with -v, so each compiler line is visible, and one status line appears every 60 seconds. Full output goes to build-<timestamp>.log in the repo root.

2.1. Re-runs

A re-run of an unchanged tree skips the compile and the install. It reads the tree commit and the two extension hashes from vllm/.build-products, re-hashes vllm/.build-products.sha256, and looks for the vllm install in the venv. Any mismatch starts the compile. ./build.sh --force always compiles. A re-run also re-checks artifacts/. When that folder is missing or incomplete, step 8 writes it again from the kernels that are already compiled (see 7).

2.2. The tree

Step 2 builds vllm/ from the upstream tag v0.30.0 plus patches/, then compares the result with the expected tree hash. A correct tree is left alone, so a re-run needs no network. FETCH=1 ./build.sh re-applies the series from the base. VLLM_UPSTREAM_URL= points at a mirror.

2.3. Tools on the build machine

build.sh uses uv for the venv when uv is on PATH, and CPython 3.13 in both cases. Set UV=0 to refuse uv, UV=/path/to/uv to select one. The wheel caches of the current user ($HOME/.cache/uv, $HOME/.cache/pip) are printed at step 3 and reused.

build.sh uses ccache when it is on PATH, as the CMake compiler launcher for C, C++, CUDA and HIP. Objects go to the ccache directory of the current user. Step 1 prints the directory, step 7 prints the hit and miss counts. Set CCACHE_DIR= to move them. Without ccache every kernel compiles from scratch.

cargo 1.95 is optional. With it, the build compiles the Rust frontend (vllm-rs, _rust_tool_parser). Without it, the Python tool parsers are used.

3. How to Run

sudo ./start.sh
tail -f /var/log/vllm.log

start.sh defaults: 127.0.0.1:8000, model name qwen38-flash-next, log /var/log/vllm.log. Compose sets MODEL_NAME=Hinata by default. The systemd install writes MODEL_NAME=Hinata to /etc/default/vllm. Use sudo for the log file. Set HOST=0.0.0.0 to serve on all interfaces.

start.sh keeps the compile caches in .vllm-cache/ in this directory.

CACHE_ROOT=/path/to/cache ./start.sh

The first start compiles the cudagraphs into $CACHE_ROOT and takes about 6 minutes. The first request builds the FlashInfer kernels. Each later start takes about 3 minutes.

4. Run as a systemd service

sudo scripts/install-as-service.sh install
sudo vi /etc/default/vllm
sudo scripts/install-as-service.sh start
sudo scripts/install-as-service.sh status

install rewrites /opt/vllm in systemd/vllm.service to the path of this repo, so the service runs this repo in place. update runs ./build.sh and restarts. logs tails the engine log. uninstall removes the unit, the environment file, and the logrotate file.

5. Logs

start.sh writes the engine and the PLE offload child process to /var/log/vllm.log. Set LOG=/other/path to move the file. Set LOG= (empty) to keep the output on the terminal. The image uses that mode, so docker compose logs works.

Install the rotation policy:

sudo install -m 0644 systemd/vllm.logrotate /etc/logrotate.d/vllm

That command replaces an existing /etc/logrotate.d/vllm on the host. The policy rotates the file daily, at 100 MB. It keeps 14 files, compresses them, and deletes logs older than 60 days. The container caps its own log the same way (json-file, 100 MB x 14).

vllm-logs.sh prints the engine stats lines as one colored line per update (throughput, KV usage, prefix cache, draft acceptance). Read the container log, or the file log with -f:

./vllm-logs.sh              # docker logs of the vllm container, last 10 minutes
./vllm-logs.sh -f -n 500    # tail /var/log/vllm.log, from the last 500 lines

6. Test the server

/v1/models returns 200 when the engine is dead. Do a one-token generation. Set model to the served name, the MODEL_NAME value:

curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
  -d '{"model":"qwen38-flash-next","messages":[{"role":"user","content":"hi"}],"max_tokens":1,\
       "chat_template_kwargs":{"enable_thinking":false}}'

A healthy start writes this line to the log:

GPU KV cache size: 924,993 tokens

The number changes with the KV memory setting.

7. Container (optional)

Use the container when the image is built on a box with no GPU and no driver, and runs on the GPU host.

Step 8 of build.sh writes the compiled kernels to artifacts/: a tarball, a per-file sha256 list, and a manifest. --no-pack (PACK=0) skips that step. A second run of the same tree does not pack again. Delete artifacts/ and the next run writes it again. That costs one pack (about 45 seconds), not a recompile.

Two build routes exist:

# kernels from artifacts/, no compile, about 2 minutes
docker build --build-context ops="$PWD/artifacts" -t vllm:latest .

# full compile, 20-40 minutes, 15 GB of build disk
docker build -t vllm:latest .

docker compose build passes the ops context from artifacts/ by itself.

The build checks the tarball before it installs it: the compiled inputs (see 7.5), the GPU architecture, the CPython minor, the CUDA major, the torch version, and the sha256 of every unpacked file. A tarball that does not fit the tree stops the build and names the item that does not match.

The image build needs network access: it downloads vLLM v0.30.0 and the pinned wheels. The base is Debian trixie. Kernels built on Debian 13 need glibc 2.38, and bookworm (glibc 2.36) cannot load them. cmake 3.25 in trixie is too old, so the image installs cmake and ninja from pip. The image carries a compiler toolchain, so it is large.

The image build takes the same two compile arguments as build.sh:

docker build -t vllm:latest --build-arg MAX_JOBS=16 --build-arg ARCH=8.6 .

7.1. The CUDA driver stub

The kernels link CUDA::cuda_driver (libcuda.so.1). The NVIDIA driver package provides it. A build container with no GPU does not, and the CUDA wheel has no stub for it, so step 5 runs scripts/gen-cuda-stub.sh. It writes a link-time stub next to the toolkit in the venv: 548 entry points, no bodies, 92 KB, soname libcuda.so.1. No loader search path includes that directory, so the extensions open the real driver at run time. Without the stub, cmake stops with Target "_C_stable_libtorch" links to: CUDA::cuda_driver but the target was not found. On the host the driver is present, so the step is a no-op after the first run.

The import check at the end of build.sh needs the driver too. Without a driver it checks the link layer and says so. The full import runs at serve time on the GPU host. libcuda.so.1 is expected only in vllm/_C_stable_libtorch.abi3.so. The MoE kernels get CUDA through libtorch_cuda, so it is not in their NEEDED list.

7.2. Build caches

The build reads three BuildKit cache mounts: the uv wheel cache, the pip cache, and the ccache object cache. They live outside the image, so a rebuild downloads nothing and recompiles only the kernels whose source changed. They start empty, so the first build downloads about 6 GB of wheels. Pass the caches of the current user to prevent that:

docker build -t vllm:latest \
  --build-context uvcache="$HOME/.cache/uv" \
  --build-context pipcache="$HOME/.cache/pip" .

The build reads those folders. It never writes to them.

7.3. Run with Docker Compose

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
cp .env.example .env
mkdir -p .vllm-cache "$HOME/.cache/uv" "$HOME/.cache/pip"
docker compose up -d
docker compose logs -f

docker compose up passes three contexts to the image build: artifacts (the kernels of ./build.sh), uvcache and pipcache (the wheel caches of this user). Each one must be a folder that exists. .vllm-cache is the run-time cache folder, mounted at /cache.

7.4. Run the image without Compose

docker run -d --name vllm --gpus all --stop-timeout 20 --shm-size=16g \
  --ulimit memlock=-1 -p 8080:8000 \
  -v /data/models/checkpoint:/models/model:ro \
  -v "$PWD/.vllm-cache":/cache \
  -e MODEL_NAME=Hinata vllm:latest

Without Compose, mount the sidecar yourself: -v <host>/scales.json:/opt/vllm/qsa_kv_scales.json:ro -e SCALES=/opt/vllm/qsa_kv_scales.json.

7.5. Add a patch without a recompile

The ops tarball carries compiled kernels. build.sh keys it on OPS_OBJ, the hash of the compiled inputs only: csrc, CMakeLists.txt, cmake, setup.py, pyproject.toml, MANIFEST.in, requirements, tools, rust-toolchain.toml, vllm/third_party, and vllm/vllm_flash_attn. The list is OPS_PATHS in build.sh.

A patch that touches Python only leaves OPS_OBJ alone. artifacts/ stays valid, so the image build keeps the ~2 minute route. A patch that moves any path in that list invalidates the tarball, and the build compiles.

To add a patch to the series:

1. Put the mail-format patch at the end of patches/, named NNNN-<slug>.patch. git format-patch writes that format.
2. Run ./build.sh. Step 2 applies the series and prints the tree it built when it does not match EXPECTED_TREE.
3. Write that tree hash into EXPECTED_TREE in build.sh, with the date you verified it.
4. Run ./build.sh again. It applies the series, matches the tree, and reuses the kernels.

A patch from a fork of upstream rarely applies to this series as-is. The series moves the same files. Test it on a clean tree before you add it: git -C vllm worktree add -q --detach /tmp/wt HEAD, then git -C /tmp/wt am --keep-non-patch <patch>, then git -C vllm worktree remove --force /tmp/wt. On a conflict, rebase the patch onto the series and re-export it with git format-patch.

8. Files

File	Use
build.sh	Fetch v0.30.0, apply patches/, build .venv/ (py3.13, 196 pins), compile, pack
start.sh	Build on first run, test the box, cache in .vllm-cache/, serve, write the log
vllm-logs.sh	Read the engine log as one stats line per update
scripts/install-as-service.sh	Install the systemd unit for this repo, start, update, log, uninstall
systemd/	vllm.service and the logrotate policy
Dockerfile, docker-compose.yml, .env.example	The container path
README.md, README-ID.md	This guide, and the Indonesian translation
scripts/	Topology selector, pinned requirements, CUDA driver stub
patches/	The 105 patches, in apply order
benchmarks/	bench.sh wrapper, and the power-limit benchmark (English, Indonesian)
abliteration/	The refusal-direction file for QWEN38_ABLITERATION

9. Variables

Variable	Default	Meaning
MODEL_DIR	none, required	Host directory of the checkpoint. Compose stops with a message when it is empty.
CKPT	/data/models/halt95/Swift1.5-Qwen3.8-Flash-Next-W4A16-Merlin	Checkpoint folder on the host. The first argument to start.sh sets it.
MODEL_MOUNT_PATH	/models/Qwen3.8-Flash-Next	Path of the checkpoint inside the container. Compose mounts MODEL_DIR here.
SCALES	$MODEL_DIR/qsa_kv_scales.json	Host path of the FP8 KV scale sidecar. Compose mounts it into the container.
MODEL_NAME	Hinata (compose), qwen38-flash-next (start.sh)	The name in the model field of a request. Spaces make aliases.
HOST_PORT	8080	The host port of the API.
VLLM_API_KEY	empty	Empty means no authentication.
CACHE_HOST_DIR	.vllm-cache in this directory	Where the compile caches live. The host path uses the same folder.
CUDA_VISIBLE_DEVICES	0,1,2,3	The GPU lane, in bus order.
PLE_HOME	0	The card that hosts the PLE offload. That card gets no CUDA context for it.
QWEN38_ABLITERATION	Qwen3.8-Flash-Next-Uncensored	Remove the refusal direction at run time. Empty disables it. See patches/0102.
ABLITERATION_DIR	./abliteration	Host folder with the direction file. Compose mounts it at /opt/qwen38/abliteration.
GUARD	warn	Bounds-guard telemetry: off, warn, or raise.
COUNTERS	0	1 adds the logging-only kernel counters.
VLLM_LOG_STATS_INTERVAL	1	Seconds between engine stats lines. The vLLM default is 10 s.
MAX_SEQS	4	Concurrent requests. CUDA graphs cover batches up to MAX_SEQS x 4 tokens.
MAX_BATCHED_TOKENS	2048	Prefill chunk in tokens. Must be >= MAX_SEQS. 2048 matches --block-size.
MAX_JOBS, ARCH	32, 8.6	Compile jobs and GPU architecture, for the build only.

10. Benchmarks

benchmarks/bench.sh runs llama-benchy against the local server. Install the tool first:

uv tool install git+https://github.com/eugr/llama-benchy

Run it from the repo root:

benchmarks/bench.sh [PP] [TG] [CONCURRENCY] [DEPTH] [extra llama-benchy args]

Argument	Default	Meaning
PP	32768	Prompt tokens. Use 1 for decode-only.
TG	2048	Generated tokens. Use 1 for prefill-only.
CONCURRENCY	1	One level, or a space-separated list, for example "1 2 4".
DEPTH	0	Prior-conversation tokens before the measured prompt. A list works too.

Extra arguments go to llama-benchy. The script fixes the server URL, the model name, and the tokenizer path. Override them with BASE_URL, MODEL, and TOKENIZER.

benchmarks/bench.sh 32768 1                  # prefill speed
benchmarks/bench.sh 1 4096                   # decode speed
benchmarks/bench.sh 32768 2048 "1 2 4"       # concurrency sweep
benchmarks/bench.sh 128 256 1 "0 8192 65536" # depth sweep, small turn

DEPTH changes what the run measures. --pp 8320 --depth 0 prices one cold 8K prompt. --pp 128 --depth 8192 prices a 128-token turn inside an 8K conversation. Decode speed falls with depth. One measured 128-token turn: 186 t/s at depth 0, then 156 t/s at depth 8192.

Add --enable-prefix-caching to measure both phases in one run. The tool then prints a cold row (it loads the prefix) and a warm row (the server reuses the cache). Add --runs N for more runs per cell, and --no-cache to skip the coherence test.

benchmarks/gpu-power-limit-effects-en.md and benchmarks/gpu-power-limit-effects-id.md record a power-limit sweep (220 W, 250 W, 285 W, PCIe Gen4 and Gen3) with the full method and the recommendation.