# iHoree-vLLM-4xRTX3090 Swift-1.5-Qwen3.8-Flash-Next-W4A16 served from four RTX 3090 cards. The model is a 125B MoE with 4-bit weights. It runs TP2xPP2 + EP with MTP K=3. The FP8 KV pool holds 924,993 tokens at a 262,144-token context. Single-stream output measured at 158-184 tok/s. An Indonesian translation is in `README-ID.md`. This file is the source. When the two differ, this file wins. ## 1. Requirements | Item | Requirement | | ----------- | ------------------------------------------------------ | | GPU | 4 free cards of 24 GB, all the same model | | RAM | 96 GiB or more. The PLE table uses 69 GiB of host RAM. | | `/dev/shm` | 1 GB or more | | Driver | CUDA 13.0 or newer | | Disk | 30 GB free for the build, 115 GB for the checkpoint | | Python | 3.13 | | Build tools | `cmake` 3.26 or newer, `ninja`, `cc`, `c++` | The checkpoint must contain `qsa_kv_scales.json`. That file holds the FP8 KV scales. Swift 1.5 changed `k_proj`/`v_proj`, so the scale file of one checkpoint does not work for another. ## 2. How to Build ```bash sudo apt-get install -y cmake ninja-build ccache ./build.sh ``` `build.sh` runs eight steps: the tools, the serving tree, the venv, the pinned runtime, the CUDA toolchain, the compiled ops, the install, and the kernel pack. The first compile takes 20 minutes on 64 cores CPU. Two variables control the compile: 1. `MAX_JOBS` — the number of compile jobs. The default is `32`. Lower it when the build runs out of RAM. 2. `ARCH` — the GPU architecture. The default is `8.6` (Ampere). The build prints every step. The installer runs with `-v`, so each compiler line is visible, and one status line appears every 60 seconds. Full output goes to `build-.log` in the repo root. ### 2.1. Re-runs A re-run of an unchanged tree skips the compile and the install. It reads the tree commit and the two extension hashes from `vllm/.build-products`, re-hashes `vllm/.build-products.sha256`, and looks for the vllm install in the venv. Any mismatch starts the compile. `./build.sh --force` always compiles. ### 2.2. The tree Step 2 builds `vllm/` from the upstream tag v0.30.0 plus `patches/`, then compares the result with the expected tree hash. A correct tree is left alone, so a re-run needs no network. `FETCH=1 ./build.sh` re-applies the series from the base. `VLLM_UPSTREAM_URL=` points at a mirror. ### 2.3. Tools on the build machine `build.sh` uses `uv` for the venv when `uv` is on PATH, and CPython 3.13 in both cases. Set `UV=0` to refuse `uv`, `UV=/path/to/uv` to select one. The wheel caches of the current user (`$HOME/.cache/uv`, `$HOME/.cache/pip`) are printed at step 3 and reused. `build.sh` uses `ccache` when it is on PATH, as the CMake compiler launcher for C, C++, CUDA and HIP. Objects go to the ccache directory of the current user. Step 1 prints the directory, step 7 prints the hit and miss counts. Set `CCACHE_DIR=` to move them. Without `ccache` every kernel compiles from scratch. `cargo` 1.95 is optional. With it, the build compiles the Rust frontend (`vllm-rs`, `_rust_tool_parser`). Without it, the Python tool parsers are used. ## 3. How to Run ```bash sudo ./start.sh tail -f /var/log/vllm.log ``` `start.sh` defaults: `127.0.0.1:8000`, model name `qwen38-flash-next`, log `/var/log/vllm.log`. Compose sets `MODEL_NAME=Hinata` by default. The systemd install writes `MODEL_NAME=Hinata` to `/etc/default/vllm`. Use `sudo` for the log file. Set `HOST=0.0.0.0` to serve on all interfaces. `start.sh` keeps the compile caches in `.vllm-cache/` in this directory. ```bash CACHE_ROOT=/path/to/cache ./start.sh ``` The first start compiles the cudagraphs into `$CACHE_ROOT` and takes about 6 minutes. The first request builds the FlashInfer kernels. Each later start takes about 3 minutes. ## 4. Run as a systemd service ```bash sudo scripts/install-as-service.sh install sudo vi /etc/default/vllm sudo scripts/install-as-service.sh start sudo scripts/install-as-service.sh status ``` `install` rewrites `/opt/vllm` in `systemd/vllm.service` to the path of this repo, so the service runs this repo in place. `update` runs `./build.sh` and restarts. `logs` tails the engine log. `uninstall` removes the unit, the environment file, and the logrotate file. ## 5. Logs `start.sh` writes the engine and the PLE offload child process to `/var/log/vllm.log`. Set `LOG=/other/path` to move the file. Set `LOG=` (empty) to keep the output on the terminal. The image uses that mode, so `docker compose logs` works. Install the rotation policy: ```bash sudo install -m 0644 systemd/vllm.logrotate /etc/logrotate.d/vllm ``` That command replaces an existing `/etc/logrotate.d/vllm` on the host. The policy rotates the file daily, at 100 MB. It keeps 14 files, compresses them, and deletes logs older than 60 days. The container caps its own log the same way (`json-file`, 100 MB x 14). `vllm-logs.sh` prints the engine stats lines as one colored line per update (throughput, KV usage, prefix cache, draft acceptance). Read the container log, or the file log with `-f`: ```bash ./vllm-logs.sh # docker logs of the vllm container, last 10 minutes ./vllm-logs.sh -f -n 500 # tail /var/log/vllm.log, from the last 500 lines ``` ## 6. Test the server `/v1/models` returns 200 when the engine is dead. Do a one-token generation. Set `model` to the served name, the `MODEL_NAME` value: ```bash curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \ -d '{"model":"qwen38-flash-next","messages":[{"role":"user","content":"hi"}],"max_tokens":1,\ "chat_template_kwargs":{"enable_thinking":false}}' ``` A healthy start writes this line to the log: ``` GPU KV cache size: 924,993 tokens ``` The number changes with the KV memory setting. ## 7. Container (optional) Use the container when the image is built on a box with no GPU and no driver, and runs on the GPU host. Step 8 of `build.sh` writes the compiled kernels to `artifacts/`: a tarball, a per-file sha256 list, and a manifest. `--no-pack` (`PACK=0`) skips that step. A second run of the same tree does not pack again. Two build routes exist: ```bash # kernels from artifacts/, no compile, about 2 minutes docker build --build-context ops="$PWD/artifacts" -t vllm:latest . # full compile, 20-40 minutes, 15 GB of build disk docker build -t vllm:latest . ``` `docker compose build` passes the `ops` context from `artifacts/` by itself. The build checks the tarball before it installs it: the tree head, the GPU architecture, the CPython minor, the CUDA major, the torch version, and the sha256 of every unpacked file. A tarball that does not fit the tree stops the build and names the item that does not match. The image build needs network access: it downloads vLLM v0.30.0 and the pinned wheels. The base is Debian trixie. Kernels built on Debian 13 need glibc 2.38, and bookworm (glibc 2.36) cannot load them. `cmake` 3.25 in trixie is too old, so the image installs `cmake` and `ninja` from pip. The image carries a compiler toolchain, so it is large. The image build takes the same two compile arguments as `build.sh`: ```bash docker build -t vllm:latest --build-arg MAX_JOBS=16 --build-arg ARCH=8.6 . ``` ### 7.1. The CUDA driver stub The kernels link `CUDA::cuda_driver` (`libcuda.so.1`). The NVIDIA driver package provides it. A build container with no GPU does not, and the CUDA wheel has no stub for it, so step 5 runs `scripts/gen-cuda-stub.sh`. It writes a link-time stub next to the toolkit in the venv: 548 entry points, no bodies, 92 KB, soname `libcuda.so.1`. No loader search path includes that directory, so the extensions open the real driver at run time. Without the stub, `cmake` stops with `Target "_C_stable_libtorch" links to: CUDA::cuda_driver but the target was not found`. On the host the driver is present, so the step is a no-op after the first run. The import check at the end of `build.sh` needs the driver too. Without a driver it checks the link layer and says so. The full import runs at serve time on the GPU host. `libcuda.so.1` is expected only in `vllm/_C_stable_libtorch.abi3.so`. The MoE kernels get CUDA through `libtorch_cuda`, so it is not in their `NEEDED` list. ### 7.2. Build caches The build reads three BuildKit cache mounts: the `uv` wheel cache, the `pip` cache, and the `ccache` object cache. They live outside the image, so a rebuild downloads nothing and recompiles only the kernels whose source changed. They start empty, so the first build downloads about 6 GB of wheels. Pass the caches of the current user to prevent that: ```bash docker build -t vllm:latest \ --build-context uvcache="$HOME/.cache/uv" \ --build-context pipcache="$HOME/.cache/pip" . ``` The build reads those folders. It never writes to them. ### 7.3. Run with Docker Compose ```bash sudo nvidia-ctk runtime configure --runtime=docker sudo systemctl restart docker cp .env.example .env mkdir -p .vllm-cache "$HOME/.cache/uv" "$HOME/.cache/pip" docker compose up -d docker compose logs -f ``` `docker compose up` passes three contexts to the image build: `artifacts` (the kernels of `./build.sh`), `uvcache` and `pipcache` (the wheel caches of this user). Each one must be a folder that exists. `.vllm-cache` is the run-time cache folder, mounted at `/cache`. ### 7.4. Run the image without Compose ```bash docker run -d --name vllm --gpus all --stop-timeout 20 --shm-size=16g \ --ulimit memlock=-1 -p 8080:8000 \ -v /data/models/checkpoint:/models/model:ro \ -v "$PWD/.vllm-cache":/cache \ -e MODEL_NAME=Hinata vllm:latest ``` ## 8. Files | File | Use | | -------------------------------------------------- | --------------------------------------------------------------------------------- | | `build.sh` | Fetch v0.30.0, apply `patches/`, build `.venv/` (py3.13, 196 pins), compile, pack | | `start.sh` | Build on first run, test the box, cache in `.vllm-cache/`, serve, write the log | | `vllm-logs.sh` | Read the engine log as one stats line per update | | `scripts/install-as-service.sh` | Install the systemd unit for this repo, start, update, log, uninstall | | `systemd/` | `vllm.service` and the logrotate policy | | `Dockerfile`, `docker-compose.yml`, `.env.example` | The container path | | `README.md`, `README-ID.md` | This guide, and the Indonesian translation | | `scripts/` | Topology selector, pinned requirements, CUDA driver stub | | `patches/` | The 103 patches, in apply order | | `benchmarks/` | `bench.sh` wrapper, and the power-limit benchmark (English, Indonesian) | | `abliteration/` | The refusal-direction file for `QWEN38_ABLITERATION` | ## 9. Variables | Variable | Default | Meaning | | ---------------------- | ---------------------------------------------------- | ------------------------------------------------------------------------------------- | | `MODEL_DIR` | none, required | Host directory of the checkpoint. Compose stops with a message when it is empty. | | `MODEL_NAME` | `Hinata` (compose), `qwen38-flash-next` (`start.sh`) | The name in the `model` field of a request. Spaces make aliases. | | `HOST_PORT` | `8080` | The host port of the API. | | `VLLM_API_KEY` | empty | Empty means no authentication. | | `CACHE_HOST_DIR` | `.vllm-cache` in this directory | Where the compile caches live. The host path uses the same folder. | | `CUDA_VISIBLE_DEVICES` | `0,1,2,3` | The GPU lane, in bus order. | | `PLE_HOME` | `0` | The card that hosts the PLE offload. That card gets no CUDA context for it. | | `QWEN38_ABLITERATION` | `Qwen3.8-Flash-Next-Uncensored` | Remove the refusal direction at run time. Empty disables it. See `patches/0102`. | | `ABLITERATION_DIR` | `./abliteration` | Host folder with the direction file. Compose mounts it at `/opt/qwen38/abliteration`. | | `GUARD` | `warn` | Bounds-guard telemetry: `off`, `warn`, or `raise`. | | `COUNTERS` | `0` | `1` adds the logging-only kernel counters. | | `MAX_JOBS`, `ARCH` | `32`, `8.6` | Compile jobs and GPU architecture, for the build only. | ## 10. Benchmarks `benchmarks/bench.sh` runs `llama-benchy` against the local server. Install the tool first: ```bash uv tool install git+https://github.com/eugr/llama-benchy ``` Run it from the repo root: ```bash benchmarks/bench.sh [PP] [TG] [CONCURRENCY] [DEPTH] [extra llama-benchy args] ``` | Argument | Default | Meaning | | ------------- | ------- | ----------------------------------------------------------------------- | | `PP` | `32768` | Prompt tokens. Use `1` for decode-only. | | `TG` | `2048` | Generated tokens. Use `1` for prefill-only. | | `CONCURRENCY` | `1` | One level, or a space-separated list, for example `"1 2 4"`. | | `DEPTH` | `0` | Prior-conversation tokens before the measured prompt. A list works too. | Extra arguments go to `llama-benchy`. The script fixes the server URL, the model name, and the tokenizer path. Override them with `BASE_URL`, `MODEL`, and `TOKENIZER`. ```bash benchmarks/bench.sh 32768 1 # prefill speed benchmarks/bench.sh 1 4096 # decode speed benchmarks/bench.sh 32768 2048 "1 2 4" # concurrency sweep benchmarks/bench.sh 128 256 1 "0 8192 65536" # depth sweep, small turn ``` `DEPTH` changes what the run measures. `--pp 8320 --depth 0` prices one cold 8K prompt. `--pp 128 --depth 8192` prices a 128-token turn inside an 8K conversation. Decode speed falls with depth. One measured 128-token turn: 186 t/s at depth 0, then 156 t/s at depth 8192. Add `--enable-prefix-caching` to measure both phases in one run. The tool then prints a cold row (it loads the prefix) and a warm row (the server reuses the cache). Add `--runs N` for more runs per cell, and `--no-cache` to skip the coherence test. `benchmarks/gpu-power-limit-effects-en.md` and `benchmarks/gpu-power-limit-effects-id.md` record a power-limit sweep (220 W, 250 W, 285 W, PCIe Gen4 and Gen3) with the full method and the recommendation.