257 lines
12 KiB
Markdown
257 lines
12 KiB
Markdown
# iHoree-vLLM-4xRTX3090
|
|
|
|
Swift-1.5-Qwen3.8-Flash-Next-W4A16 served from four RTX 3090 cards. The model is a 125B
|
|
MoE with 4-bit weights. It runs TP2xPP2 + EP with MTP K=3. The FP8 KV pool holds 924,993
|
|
tokens at a 262,144-token context. Single-stream output measured at 158-184 tok/s.
|
|
|
|
## 1. Requirements
|
|
|
|
| Item | Requirement |
|
|
| ----------- | ------------------------------------------------------ |
|
|
| GPU | 4 free cards of 24 GB, all the same model |
|
|
| RAM | 96 GiB or more. The PLE table uses 69 GiB of host RAM. |
|
|
| `/dev/shm` | 1 GB or more |
|
|
| Driver | CUDA 13.0 or newer |
|
|
| Disk | 30 GB free for the build, 115 GB for the checkpoint |
|
|
| Python | 3.13 |
|
|
| Build tools | `cmake` 3.26 or newer, `ninja`, `cc`, `c++` |
|
|
|
|
The checkpoint must contain `qsa_kv_scales_swift.json`. That file holds the FP8 KV
|
|
scales. Swift 1.5 changed `k_proj`/`v_proj`, so the scale file of one checkpoint does not
|
|
work for another.
|
|
|
|
## 2. How to Build
|
|
|
|
```bash
|
|
sudo apt-get install -y cmake ninja-build ccache
|
|
./build.sh
|
|
```
|
|
|
|
`build.sh` runs eight steps: the tools, the serving tree, the venv, the pinned runtime,
|
|
the CUDA toolchain, the compiled ops, the install, and the kernel pack. The first compile
|
|
takes 20 minutes on 64 cores CPU.
|
|
|
|
Two variables control the compile:
|
|
|
|
1. `MAX_JOBS` — the number of compile jobs. The default is `32`. Lower it when the build
|
|
runs out of RAM.
|
|
2. `ARCH` — the GPU architecture. The default is `8.6` (Ampere).
|
|
|
|
The build prints every step. The installer runs with `-v`, so each compiler line is
|
|
visible, and one status line appears every 60 seconds. Full output goes to
|
|
`build-<timestamp>.log` in the repo root.
|
|
|
|
### 2.1. Re-runs
|
|
|
|
A re-run of an unchanged tree skips the compile and the install. It reads the tree commit
|
|
and the two extension hashes from `vllm/.build-products`, re-hashes
|
|
`vllm/.build-products.sha256`, and looks for the vllm install in the venv. Any mismatch
|
|
starts the compile. `./build.sh --force` always compiles.
|
|
|
|
### 2.2. The tree
|
|
|
|
Step 2 builds `vllm/` from the upstream tag v0.30.0 plus `patches/`, then compares the
|
|
result with the expected tree hash. A correct tree is left alone, so a re-run needs no
|
|
network. `FETCH=1 ./build.sh` re-applies the series from the base.
|
|
`VLLM_UPSTREAM_URL=` points at a mirror.
|
|
|
|
### 2.3. Tools on the build machine
|
|
|
|
`build.sh` uses `uv` for the venv when `uv` is on PATH, and CPython 3.13 in both cases.
|
|
Set `UV=0` to refuse `uv`, `UV=/path/to/uv` to select one. The wheel caches of the current
|
|
user (`$HOME/.cache/uv`, `$HOME/.cache/pip`) are printed at step 3 and reused.
|
|
|
|
`build.sh` uses `ccache` when it is on PATH, as the CMake compiler launcher for C, C++,
|
|
CUDA and HIP. Objects go to the ccache directory of the current user. Step 1 prints the
|
|
directory, step 7 prints the hit and miss counts. Set `CCACHE_DIR=` to move them. Without
|
|
`ccache` every kernel compiles from scratch.
|
|
|
|
`cargo` 1.95 is optional. With it, the build compiles the Rust frontend (`vllm-rs`,
|
|
`_rust_tool_parser`). Without it, the Python tool parsers are used.
|
|
|
|
## 3. How to Run
|
|
|
|
```bash
|
|
sudo ./start.sh
|
|
tail -f /var/log/vllm.log
|
|
```
|
|
|
|
`start.sh` defaults: `127.0.0.1:8000`, model name `qwen38-flash-next`, log
|
|
`/var/log/vllm.log`. The container and the systemd unit set `MODEL_NAME=Scarlet`. Use
|
|
`sudo` for the log file. Set `HOST=0.0.0.0` to serve on all interfaces.
|
|
|
|
`start.sh` keeps the compile caches in `.vllm-cache/` in this directory.
|
|
|
|
```bash
|
|
CACHE_ROOT=/path/to/cache ./start.sh
|
|
```
|
|
|
|
The first start compiles the cudagraphs into `$CACHE_ROOT` and takes about 6 minutes. The
|
|
first request builds the FlashInfer kernels. Each later start takes about 3 minutes.
|
|
|
|
## 4. Run as a systemd service
|
|
|
|
```bash
|
|
sudo scripts/install-as-service.sh install
|
|
sudo vi /etc/default/vllm
|
|
sudo scripts/install-as-service.sh start
|
|
sudo scripts/install-as-service.sh status
|
|
```
|
|
|
|
`install` rewrites `/opt/vllm` in `systemd/vllm.service` to the path of this repo, so the
|
|
service runs this repo in place. `update` runs `./build.sh` and restarts. `logs` tails the
|
|
engine log. `uninstall` removes the unit, the environment file, and the logrotate file.
|
|
|
|
## 5. Logs
|
|
|
|
`start.sh` writes the engine and the PLE offload child process to `/var/log/vllm.log`.
|
|
Set `LOG=/other/path` to move the file. Set `LOG=` (empty) to keep the output on the
|
|
terminal. The image uses that mode, so `docker compose logs` works.
|
|
|
|
Install the rotation policy:
|
|
|
|
```bash
|
|
sudo install -m 0644 systemd/vllm.logrotate /etc/logrotate.d/vllm
|
|
```
|
|
|
|
That command replaces an existing `/etc/logrotate.d/vllm` on the host. The policy rotates
|
|
the file daily, at 100 MB. It keeps 14 files, compresses them, and deletes logs older
|
|
than 60 days. The container caps its own log the same way (`json-file`, 100 MB x 14).
|
|
|
|
## 6. Test the server
|
|
|
|
`/v1/models` returns 200 when the engine is dead. Do a one-token generation:
|
|
|
|
```bash
|
|
curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
|
|
-d '{"model":"Scarlet","messages":[{"role":"user","content":"hi"}],"max_tokens":1,\
|
|
"chat_template_kwargs":{"enable_thinking":false}}'
|
|
```
|
|
|
|
A healthy start writes this line to the log:
|
|
|
|
```
|
|
GPU KV cache size: 924,993 tokens
|
|
```
|
|
|
|
The number changes with the KV memory setting.
|
|
|
|
## 7. Container (optional)
|
|
|
|
Use the container when the image is built on a box with no GPU and no driver, and runs on
|
|
the GPU host.
|
|
|
|
Step 8 of `build.sh` writes the compiled kernels to `artifacts/`: a tarball, a per-file
|
|
sha256 list, and a manifest. `--no-pack` (`PACK=0`) skips that step. A second run of the
|
|
same tree does not pack again.
|
|
|
|
Two build routes exist:
|
|
|
|
```bash
|
|
# kernels from artifacts/, no compile, about 2 minutes
|
|
docker build --build-context ops="$PWD/artifacts" -t vllm:latest .
|
|
|
|
# full compile, 20-40 minutes, 15 GB of build disk
|
|
docker build -t vllm:latest .
|
|
```
|
|
|
|
`docker compose build` passes the `ops` context from `artifacts/` by itself.
|
|
|
|
The build checks the tarball before it installs it: the tree head, the GPU architecture,
|
|
the CPython minor, the CUDA major, the torch version, and the sha256 of every unpacked
|
|
file. A tarball that does not fit the tree stops the build and names the item that does
|
|
not match.
|
|
|
|
The image build needs network access: it downloads vLLM v0.30.0 and the pinned wheels. The
|
|
base is Debian trixie. Kernels built on Debian 13 need glibc 2.38, and bookworm (glibc
|
|
2.36) cannot load them. `cmake` 3.25 in trixie is too old, so the image installs `cmake`
|
|
and `ninja` from pip. The image carries a compiler toolchain, so it is large.
|
|
|
|
The image build takes the same two compile arguments as `build.sh`:
|
|
|
|
```bash
|
|
docker build -t vllm:latest --build-arg MAX_JOBS=16 --build-arg ARCH=8.6 .
|
|
```
|
|
|
|
### 7.1. The CUDA driver stub
|
|
|
|
The kernels link `CUDA::cuda_driver` (`libcuda.so.1`). The NVIDIA driver package provides
|
|
it. A build container with no GPU does not, and the CUDA wheel has no stub for it, so
|
|
step 5 runs `scripts/gen-cuda-stub.sh`. It writes a link-time stub next to the toolkit in
|
|
the venv: 548 entry points, no bodies, 92 KB, soname `libcuda.so.1`. No loader search path
|
|
includes that directory, so the extensions open the real driver at run time. Without the
|
|
stub, `cmake` stops with `Target "_C_stable_libtorch" links to: CUDA::cuda_driver but the
|
|
target was not found`. On the host the driver is present, so the step is a no-op after the
|
|
first run.
|
|
|
|
The import check at the end of `build.sh` needs the driver too. Without a driver it checks
|
|
the link layer and says so. The full import runs at serve time on the GPU host.
|
|
`libcuda.so.1` is expected only in `vllm/_C_stable_libtorch.abi3.so`. The MoE kernels get
|
|
CUDA through `libtorch_cuda`, so it is not in their `NEEDED` list.
|
|
|
|
### 7.2. Build caches
|
|
|
|
The build reads three BuildKit cache mounts: the `uv` wheel cache, the `pip` cache, and
|
|
the `ccache` object cache. They live outside the image, so a rebuild downloads nothing and
|
|
recompiles only the kernels whose source changed. They start empty, so the first build
|
|
downloads about 6 GB of wheels. Pass the caches of the current user to prevent that:
|
|
|
|
```bash
|
|
docker build -t vllm:latest \
|
|
--build-context uvcache="$HOME/.cache/uv" \
|
|
--build-context pipcache="$HOME/.cache/pip" .
|
|
```
|
|
|
|
The build reads those folders. It never writes to them.
|
|
|
|
### 7.3. Run with Docker Compose
|
|
|
|
```bash
|
|
sudo nvidia-ctk runtime configure --runtime=docker
|
|
sudo systemctl restart docker
|
|
cp .env.example .env
|
|
mkdir -p .vllm-cache "$HOME/.cache/uv" "$HOME/.cache/pip"
|
|
docker compose up -d
|
|
docker compose logs -f
|
|
```
|
|
|
|
`docker compose up` passes three contexts to the image build: `artifacts` (the kernels of
|
|
`./build.sh`), `uvcache` and `pipcache` (the wheel caches of this user). Each one must be a
|
|
folder that exists. `.vllm-cache` is the run-time cache folder, mounted at `/cache`.
|
|
|
|
### 7.4. Run the image without Compose
|
|
|
|
```bash
|
|
docker run -d --name vllm --gpus all --stop-timeout 20 --shm-size=16g \
|
|
--ulimit memlock=-1 -p 8080:8000 \
|
|
-v /data/models/checkpoint:/models/model:ro \
|
|
-v "$PWD/.vllm-cache":/cache \
|
|
-e MODEL_NAME=Scarlet vllm:latest
|
|
```
|
|
|
|
## 8. Files
|
|
|
|
| File | Use |
|
|
| -------------------------------------------------- | --------------------------------------------------------------------------------- |
|
|
| `build.sh` | Fetch v0.30.0, apply `patches/`, build `.venv/` (py3.13, 196 pins), compile, pack |
|
|
| `start.sh` | Build on first run, test the box, cache in `.vllm-cache/`, serve, write the log |
|
|
| `scripts/install-as-service.sh` | Install the systemd unit for this repo, start, update, log, uninstall |
|
|
| `systemd/` | `vllm.service` and the logrotate policy |
|
|
| `Dockerfile`, `docker-compose.yml`, `.env.example` | The container path |
|
|
| `scripts/` | Topology selector, pinned requirements, CUDA driver stub |
|
|
| `patches/` | The 100 patches, in apply order |
|
|
|
|
## 9. Variables
|
|
|
|
| Variable | Default | Meaning |
|
|
| ------------------------- | ----------------------------------------------------- | -------------------------------------------------------------------------------- |
|
|
| `MODEL_DIR` | none, required | Host directory of the checkpoint. Compose stops with a message when it is empty. |
|
|
| `MODEL_NAME` | `Scarlet` (compose), `qwen38-flash-next` (`start.sh`) | The name in the `model` field of a request. Spaces make aliases. |
|
|
| `HOST_PORT` | `8080` | The host port of the API. |
|
|
| `VLLM_API_KEY` | empty | Empty means no authentication. |
|
|
| `CACHE_HOST_DIR` | `.vllm-cache` in this directory | Where the compile caches live. The host path uses the same folder. |
|
|
| `CUDA_VISIBLE_DEVICES` | `0,1,2,3` | The GPU lane, in bus order. |
|
|
| `PLE_HOME` | `0` | The card that hosts the PLE offload. That card gets no CUDA context for it. |
|
|
| `VLLM_LOG_STATS_INTERVAL` | `1` | Seconds between engine stats lines (throughput, KV use, queue). |
|
|
| `MAX_JOBS`, `ARCH` | `32`, `8.6` | Compile jobs and GPU architecture, for the build only. |
|