iHore/vLLM-4x-RTX-3090
8
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
main
vLLM-4x-RTX-3090/README.md
T
MOVZXandHinata b1fdbde895 build: Rewrite artifacts/ when it is missing, and stop failing re-runs 
Three faults ended a re-run before step 8, so a deleted artifacts/ was
never written again.

The import check read the first word of stdout, and vLLM logs to stdout.
On a box with mixed 3090 and 3090 Ti devices the device-mismatch WARNING
comes first, so the build stopped with "vllm reports WARNING, expected
0.30.0". Print a BUILD_SMOKE marker and read the version from that line.

Patch 0103 committed .build-products.sha256 into the serving tree, and
write_sums rewrites that file, so every re-run stopped at "modified
tracked files". Drop the file from the patch and exclude the two build
record paths from the check. The series now yields tree
a823fa36d3fd640f87b3de87153268d938130a47; OPS_OBJ is unchanged, so the
compiled kernels stay valid.

Step 8 skipped the pack whenever the kernels came from a tarball, and
judged an existing set by file presence alone. Require the tarball, the
per-file list, the manifest and the sidecar, verify the sidecar, and
pack from the products in the tree when any check fails.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
2026-10-18 08:38:14 +00:00

363 lines
19 KiB
Markdown
Raw Permalink Blame History

# iHoree-vLLM-4xRTX3090
Swift-1.5-Qwen3.8-Flash-Next-W4A16 served from four RTX 3090 cards. The model is a 125B
MoE with 4-bit weights. It runs TP2xPP2 + EP with MTP K=3. The FP8 KV pool holds 924,993
tokens at a 262,144-token context. Single-stream output measured at 158-184 tok/s.
An Indonesian translation is in `README-ID.md`. This file is the source. When the two
differ, this file wins.
## 1. Requirements
| Item | Requirement |
| ----------- | ------------------------------------------------------ |
| GPU | 4 free cards of 24 GB, all the same model |
| RAM | 96 GiB or more. The PLE table uses 69 GiB of host RAM. |
| `/dev/shm` | 1 GB or more |
| Driver | CUDA 13.0 or newer |
| Disk | 30 GB free for the build, 115 GB for the checkpoint |
| Python | 3.13 |
| Build tools | `cmake` 3.26 or newer, `ninja`, `cc`, `c++` |
The checkpoint must contain `qsa_kv_scales.json`. That file holds the FP8 KV scales.
Swift 1.5 changed `k_proj`/`v_proj`, so the scale file of one checkpoint does not work
for another.
`SCALES` names another sidecar. Give a host path; Compose mounts it into the container.
The default is `$MODEL_DIR/qsa_kv_scales.json`.
## 2. How to Build
```bash
sudo apt-get install -y cmake ninja-build ccache
./build.sh
```
`build.sh` runs eight steps: the tools, the serving tree, the venv, the pinned runtime,
the CUDA toolchain, the compiled ops, the install, and the kernel pack. The first compile
takes 20 minutes on 64 cores CPU.
Two variables control the compile:
1. `MAX_JOBS` — the number of compile jobs. The default is `32`. Lower it when the build
runs out of RAM.
2. `ARCH` — the GPU architecture. The default is `8.6` (Ampere).
The build prints every step. The installer runs with `-v`, so each compiler line is
visible, and one status line appears every 60 seconds. Full output goes to
`build-<timestamp>.log` in the repo root.
### 2.1. Re-runs
A re-run of an unchanged tree skips the compile and the install. It reads the tree commit
and the two extension hashes from `vllm/.build-products`, re-hashes
`vllm/.build-products.sha256`, and looks for the vllm install in the venv. Any mismatch
starts the compile. `./build.sh --force` always compiles. A re-run also re-checks
`artifacts/`. When that folder is missing or incomplete, step 8 writes it again from the kernels
that are already compiled (see 7).
### 2.2. The tree
Step 2 builds `vllm/` from the upstream tag v0.30.0 plus `patches/`, then compares the
result with the expected tree hash. A correct tree is left alone, so a re-run needs no
network. `FETCH=1 ./build.sh` re-applies the series from the base.
`VLLM_UPSTREAM_URL=` points at a mirror.
### 2.3. Tools on the build machine
`build.sh` uses `uv` for the venv when `uv` is on PATH, and CPython 3.13 in both cases.
Set `UV=0` to refuse `uv`, `UV=/path/to/uv` to select one. The wheel caches of the current
user (`$HOME/.cache/uv`, `$HOME/.cache/pip`) are printed at step 3 and reused.
`build.sh` uses `ccache` when it is on PATH, as the CMake compiler launcher for C, C++,
CUDA and HIP. Objects go to the ccache directory of the current user. Step 1 prints the
directory, step 7 prints the hit and miss counts. Set `CCACHE_DIR=` to move them. Without
`ccache` every kernel compiles from scratch.
`cargo` 1.95 is optional. With it, the build compiles the Rust frontend (`vllm-rs`,
`_rust_tool_parser`). Without it, the Python tool parsers are used.
## 3. How to Run
```bash
sudo ./start.sh
tail -f /var/log/vllm.log
```
`start.sh` defaults: `127.0.0.1:8000`, model name `qwen38-flash-next`, log
`/var/log/vllm.log`. Compose sets `MODEL_NAME=Hinata` by default. The systemd
install writes `MODEL_NAME=Hinata` to `/etc/default/vllm`. Use `sudo` for the
log file. Set `HOST=0.0.0.0` to serve on all interfaces.
`start.sh` keeps the compile caches in `.vllm-cache/` in this directory.
```bash
CACHE_ROOT=/path/to/cache ./start.sh
```
The first start compiles the cudagraphs into `$CACHE_ROOT` and takes about 6 minutes. The
first request builds the FlashInfer kernels. Each later start takes about 3 minutes.
## 4. Run as a systemd service
```bash
sudo scripts/install-as-service.sh install
sudo vi /etc/default/vllm
sudo scripts/install-as-service.sh start
sudo scripts/install-as-service.sh status
```
`install` rewrites `/opt/vllm` in `systemd/vllm.service` to the path of this repo, so the
service runs this repo in place. `update` runs `./build.sh` and restarts. `logs` tails the
engine log. `uninstall` removes the unit, the environment file, and the logrotate file.
## 5. Logs
`start.sh` writes the engine and the PLE offload child process to `/var/log/vllm.log`.
Set `LOG=/other/path` to move the file. Set `LOG=` (empty) to keep the output on the
terminal. The image uses that mode, so `docker compose logs` works.
Install the rotation policy:
```bash
sudo install -m 0644 systemd/vllm.logrotate /etc/logrotate.d/vllm
```
That command replaces an existing `/etc/logrotate.d/vllm` on the host. The policy rotates
the file daily, at 100 MB. It keeps 14 files, compresses them, and deletes logs older
than 60 days. The container caps its own log the same way (`json-file`, 100 MB x 14).
`vllm-logs.sh` prints the engine stats lines as one colored line per update
(throughput, KV usage, prefix cache, draft acceptance). Read the container log, or the
file log with `-f`:
```bash
./vllm-logs.sh # docker logs of the vllm container, last 10 minutes
./vllm-logs.sh -f -n 500 # tail /var/log/vllm.log, from the last 500 lines
```
## 6. Test the server
`/v1/models` returns 200 when the engine is dead. Do a one-token generation. Set
`model` to the served name, the `MODEL_NAME` value:
```bash
curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
-d '{"model":"qwen38-flash-next","messages":[{"role":"user","content":"hi"}],"max_tokens":1,\
"chat_template_kwargs":{"enable_thinking":false}}'
```
A healthy start writes this line to the log:
```
GPU KV cache size: 924,993 tokens
```
The number changes with the KV memory setting.
## 7. Container (optional)
Use the container when the image is built on a box with no GPU and no driver, and runs on
the GPU host.
Step 8 of `build.sh` writes the compiled kernels to `artifacts/`: a tarball, a per-file
sha256 list, and a manifest. `--no-pack` (`PACK=0`) skips that step. A second run of the
same tree does not pack again. Delete `artifacts/` and the next run writes it again. That costs
one pack (about 45 seconds), not a recompile.
Two build routes exist:
```bash
# kernels from artifacts/, no compile, about 2 minutes
docker build --build-context ops="$PWD/artifacts" -t vllm:latest .
# full compile, 20-40 minutes, 15 GB of build disk
docker build -t vllm:latest .
```
`docker compose build` passes the `ops` context from `artifacts/` by itself.
The build checks the tarball before it installs it: the compiled inputs (see 7.5), the GPU
architecture, the CPython minor, the CUDA major, the torch version, and the sha256 of every
unpacked file. A tarball that does not fit the tree stops the build and names the item that
does not match.
The image build needs network access: it downloads vLLM v0.30.0 and the pinned wheels. The
base is Debian trixie. Kernels built on Debian 13 need glibc 2.38, and bookworm (glibc
2.36) cannot load them. `cmake` 3.25 in trixie is too old, so the image installs `cmake`
and `ninja` from pip. The image carries a compiler toolchain, so it is large.
The image build takes the same two compile arguments as `build.sh`:
```bash
docker build -t vllm:latest --build-arg MAX_JOBS=16 --build-arg ARCH=8.6 .
```
### 7.1. The CUDA driver stub
The kernels link `CUDA::cuda_driver` (`libcuda.so.1`). The NVIDIA driver package provides
it. A build container with no GPU does not, and the CUDA wheel has no stub for it, so
step 5 runs `scripts/gen-cuda-stub.sh`. It writes a link-time stub next to the toolkit in
the venv: 548 entry points, no bodies, 92 KB, soname `libcuda.so.1`. No loader search path
includes that directory, so the extensions open the real driver at run time. Without the
stub, `cmake` stops with `Target "_C_stable_libtorch" links to: CUDA::cuda_driver but the
target was not found`. On the host the driver is present, so the step is a no-op after the
first run.
The import check at the end of `build.sh` needs the driver too. Without a driver it checks
the link layer and says so. The full import runs at serve time on the GPU host.
`libcuda.so.1` is expected only in `vllm/_C_stable_libtorch.abi3.so`. The MoE kernels get
CUDA through `libtorch_cuda`, so it is not in their `NEEDED` list.
### 7.2. Build caches
The build reads three BuildKit cache mounts: the `uv` wheel cache, the `pip` cache, and
the `ccache` object cache. They live outside the image, so a rebuild downloads nothing and
recompiles only the kernels whose source changed. They start empty, so the first build
downloads about 6 GB of wheels. Pass the caches of the current user to prevent that:
```bash
docker build -t vllm:latest \
--build-context uvcache="$HOME/.cache/uv" \
--build-context pipcache="$HOME/.cache/pip" .
```
The build reads those folders. It never writes to them.
### 7.3. Run with Docker Compose
```bash
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
cp .env.example .env
mkdir -p .vllm-cache "$HOME/.cache/uv" "$HOME/.cache/pip"
docker compose up -d
docker compose logs -f
```
`docker compose up` passes three contexts to the image build: `artifacts` (the kernels of
`./build.sh`), `uvcache` and `pipcache` (the wheel caches of this user). Each one must be a
folder that exists. `.vllm-cache` is the run-time cache folder, mounted at `/cache`.
### 7.4. Run the image without Compose
```bash
docker run -d --name vllm --gpus all --stop-timeout 20 --shm-size=16g \
--ulimit memlock=-1 -p 8080:8000 \
-v /data/models/checkpoint:/models/model:ro \
-v "$PWD/.vllm-cache":/cache \
-e MODEL_NAME=Hinata vllm:latest
```
Without Compose, mount the sidecar yourself: `-v <host>/scales.json:/opt/vllm/qsa_kv_scales.json:ro -e SCALES=/opt/vllm/qsa_kv_scales.json`.
### 7.5. Add a patch without a recompile
The ops tarball carries compiled kernels. `build.sh` keys it on `OPS_OBJ`, the hash of the
compiled inputs only: `csrc`, `CMakeLists.txt`, `cmake`, `setup.py`, `pyproject.toml`,
`MANIFEST.in`, `requirements`, `tools`, `rust-toolchain.toml`, `vllm/third_party`, and
`vllm/vllm_flash_attn`. The list is `OPS_PATHS` in `build.sh`.
A patch that touches Python only leaves `OPS_OBJ` alone. `artifacts/` stays valid, so the
image build keeps the ~2 minute route. A patch that moves any path in that list invalidates
the tarball, and the build compiles.
To add a patch to the series:
1. Put the mail-format patch at the end of `patches/`, named `NNNN-<slug>.patch`.
`git format-patch` writes that format.
2. Run `./build.sh`. Step 2 applies the series and prints the tree it built when it does not
match `EXPECTED_TREE`.
3. Write that tree hash into `EXPECTED_TREE` in `build.sh`, with the date you verified it.
4. Run `./build.sh` again. It applies the series, matches the tree, and reuses the kernels.
A patch from a fork of upstream rarely applies to this series as-is. The series moves the
same files. Test it on a clean tree before you add it:
`git -C vllm worktree add -q --detach /tmp/wt HEAD`, then `git -C /tmp/wt am --keep-non-patch
<patch>`, then `git -C vllm worktree remove --force /tmp/wt`. On a conflict, rebase the patch
onto the series and re-export it with `git format-patch`.
## 8. Files
| File | Use |
| -------------------------------------------------- | --------------------------------------------------------------------------------- |
| `build.sh` | Fetch v0.30.0, apply `patches/`, build `.venv/` (py3.13, 196 pins), compile, pack |
| `start.sh` | Build on first run, test the box, cache in `.vllm-cache/`, serve, write the log |
| `vllm-logs.sh` | Read the engine log as one stats line per update |
| `scripts/install-as-service.sh` | Install the systemd unit for this repo, start, update, log, uninstall |
| `systemd/` | `vllm.service` and the logrotate policy |
| `Dockerfile`, `docker-compose.yml`, `.env.example` | The container path |
| `README.md`, `README-ID.md` | This guide, and the Indonesian translation |
| `scripts/` | Topology selector, pinned requirements, CUDA driver stub |
| `patches/` | The 105 patches, in apply order |
| `benchmarks/` | `bench.sh` wrapper, and the power-limit benchmark (English, Indonesian) |
| `abliteration/` | The refusal-direction file for `QWEN38_ABLITERATION` |
## 9. Variables
| Variable | Default | Meaning |
| ------------------------- | -------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
| `MODEL_DIR` | none, required | Host directory of the checkpoint. Compose stops with a message when it is empty. |
| `CKPT` | `/data/models/halt95/Swift1.5-Qwen3.8-Flash-Next-W4A16-Merlin` | Checkpoint folder on the host. The first argument to `start.sh` sets it. |
| `MODEL_MOUNT_PATH` | `/models/Qwen3.8-Flash-Next` | Path of the checkpoint inside the container. Compose mounts `MODEL_DIR` here. |
| `SCALES` | `$MODEL_DIR/qsa_kv_scales.json` | Host path of the FP8 KV scale sidecar. Compose mounts it into the container. |
| `MODEL_NAME` | `Hinata` (compose), `qwen38-flash-next` (`start.sh`) | The name in the `model` field of a request. Spaces make aliases. |
| `HOST_PORT` | `8080` | The host port of the API. |
| `VLLM_API_KEY` | empty | Empty means no authentication. |
| `CACHE_HOST_DIR` | `.vllm-cache` in this directory | Where the compile caches live. The host path uses the same folder. |
| `CUDA_VISIBLE_DEVICES` | `0,1,2,3` | The GPU lane, in bus order. |
| `PLE_HOME` | `0` | The card that hosts the PLE offload. That card gets no CUDA context for it. |
| `QWEN38_ABLITERATION` | `Qwen3.8-Flash-Next-Uncensored` | Remove the refusal direction at run time. Empty disables it. See `patches/0102`. |
| `ABLITERATION_DIR` | `./abliteration` | Host folder with the direction file. Compose mounts it at `/opt/qwen38/abliteration`. |
| `GUARD` | `warn` | Bounds-guard telemetry: `off`, `warn`, or `raise`. |
| `COUNTERS` | `0` | `1` adds the logging-only kernel counters. |
| `VLLM_LOG_STATS_INTERVAL` | `1` | Seconds between engine stats lines. The vLLM default is 10 s. |
| `MAX_SEQS` | `4` | Concurrent requests. CUDA graphs cover batches up to `MAX_SEQS` x 4 tokens. |
| `MAX_BATCHED_TOKENS` | `2048` | Prefill chunk in tokens. Must be >= `MAX_SEQS`. 2048 matches `--block-size`. |
| `MAX_JOBS`, `ARCH` | `32`, `8.6` | Compile jobs and GPU architecture, for the build only. |
## 10. Benchmarks
`benchmarks/bench.sh` runs `llama-benchy` against the local server. Install the tool
first:
```bash
uv tool install git+https://github.com/eugr/llama-benchy
```
Run it from the repo root:
```bash
benchmarks/bench.sh [PP] [TG] [CONCURRENCY] [DEPTH] [extra llama-benchy args]
```
| Argument | Default | Meaning |
| ------------- | ------- | ----------------------------------------------------------------------- |
| `PP` | `32768` | Prompt tokens. Use `1` for decode-only. |
| `TG` | `2048` | Generated tokens. Use `1` for prefill-only. |
| `CONCURRENCY` | `1` | One level, or a space-separated list, for example `"1 2 4"`. |
| `DEPTH` | `0` | Prior-conversation tokens before the measured prompt. A list works too. |
Extra arguments go to `llama-benchy`. The script fixes the server URL, the model name,
and the tokenizer path. Override them with `BASE_URL`, `MODEL`, and `TOKENIZER`.
```bash
benchmarks/bench.sh 32768 1 # prefill speed
benchmarks/bench.sh 1 4096 # decode speed
benchmarks/bench.sh 32768 2048 "1 2 4" # concurrency sweep
benchmarks/bench.sh 128 256 1 "0 8192 65536" # depth sweep, small turn
```
`DEPTH` changes what the run measures. `--pp 8320 --depth 0` prices one cold 8K
prompt. `--pp 128 --depth 8192` prices a 128-token turn inside an 8K conversation.
Decode speed falls with depth. One measured 128-token turn: 186 t/s at depth 0, then
156 t/s at depth 8192.
Add `--enable-prefix-caching` to measure both phases in one run. The tool then prints a
cold row (it loads the prefix) and a warm row (the server reuses the cache). Add
`--runs N` for more runs per cell, and `--no-cache` to skip the coherence test.
`benchmarks/gpu-power-limit-effects-en.md` and `benchmarks/gpu-power-limit-effects-id.md`
record a power-limit sweep (220 W, 250 W, 285 W, PCIe Gen4 and Gen3) with the full
method and the recommendation.
Reference in New Issue View Git Blame Copy Permalink