From d6062059b4bc534650746796731749000d54708d Mon Sep 17 00:00:00 2001 From: MOVZX Date: Thu, 15 Oct 2026 07:52:20 +0000 Subject: [PATCH] docs: Add the Indonesian README and sync the guide with the code Rename the default model from Scarlet to Hinata in the Dockerfile environment, the healthcheck, and the file that scripts/install-as-service.sh writes to /etc/default/vllm. Compose already defaults to Hinata. Sync README.md with the code. Section 6 paired port 8000 with the Compose model name, so that request fails. Document vllm-logs.sh, the abliteration, guard, and counter variables, and the new folders, so the patch count reaches 103. Add section 10 for benchmarks/bench.sh, with the defaults, the example runs, and what DEPTH changes. Drop the duplicate CUDA_VISIBLE_DEVICES and PLE_HOME rows. Add README-ID.md, the full Indonesian translation. README.md stays the source, and both files say so. Co-Authored-By: Hinata Signed-off-by: MOVZX --- Dockerfile | 6 +- README-ID.md | 334 ++++++++++++++++++++++++++++++++++ README.md | 99 ++++++++-- scripts/install-as-service.sh | 2 +- 4 files changed, 421 insertions(+), 20 deletions(-) create mode 100644 README-ID.md diff --git a/Dockerfile b/Dockerfile index 774ba76..5eed3ce 100644 --- a/Dockerfile +++ b/Dockerfile @@ -33,7 +33,7 @@ FROM python:3.13-slim-trixie # --ulimit memlock=-1 -p 8080:8000 \ # -v /data/models/halt95/Swift1.5-Qwen3.8-Flash-Next-W4A16-Merlin:/models/model:ro \ # -v "$PWD/.vllm-cache":/cache \ -# -e MODEL_NAME=Scarlet vllm:latest +# -e MODEL_NAME=Hinata vllm:latest # # The checkpoint mount must contain qsa_kv_scales_swift.json (the FP8 KV scales sidecar). # @@ -87,7 +87,7 @@ COPY . /opt/vllm/ RUN chmod +x start.sh build.sh scripts/*.sh scripts/*.py \ && rm -rf /root/.cache ENV TREE=/opt/vllm/vllm VENV=/opt/vllm/.venv HOST=0.0.0.0 PORT=8000 CACHE_ROOT=/cache \ - MODEL_NAME=Scarlet MODEL_MOUNT_PATH=/models/model LOG= \ + MODEL_NAME=Hinata MODEL_MOUNT_PATH=/models/model LOG= \ CUDA_HOME=/opt/vllm/.venv/lib/python3.13/site-packages/nvidia/cu13 \ FLASHINFER_WORKSPACE_BASE=/cache/flashinfer TRITON_CACHE_DIR=/cache/triton ENV PATH="/opt/vllm/.venv/lib/python3.13/site-packages/nvidia/cu13/bin:$PATH" @@ -95,7 +95,7 @@ VOLUME /cache EXPOSE 8000 # /v1/models answers 200 even when the engine is dead: probe a real one-token generation. HEALTHCHECK --interval=60s --timeout=90s --start-period=30m --retries=3 \ - CMD m="${MODEL_NAME%% *}"; m="${m:-Scarlet}"; curl -fs -m 85 -X POST "http://127.0.0.1:${PORT}/v1/chat/completions" \ + CMD m="${MODEL_NAME%% *}"; m="${m:-Hinata}"; curl -fs -m 85 -X POST "http://127.0.0.1:${PORT}/v1/chat/completions" \ -H "Content-Type: application/json" ${VLLM_API_KEY:+-H "Authorization: Bearer $VLLM_API_KEY"} \ -d "{\"model\":\"$m\",\"messages\":[{\"role\":\"user\",\"content\":\"ok\"}],\"max_tokens\":1,\"chat_template_kwargs\":{\"enable_thinking\":false}}" \ | grep -q '"choices"' || exit 1 diff --git a/README-ID.md b/README-ID.md new file mode 100644 index 0000000..2d64683 --- /dev/null +++ b/README-ID.md @@ -0,0 +1,334 @@ +# iHoree-vLLM-4xRTX3090 + +Dokumen ini adalah terjemahan dari `README.md`. Bahasa Inggris tetap acuan. +Jika isinya berbeda, ikuti `README.md`. + +Swift-1.5-Qwen3.8-Flash-Next-W4A16 berjalan dari empat kartu RTX 3090. Modelnya +adalah MoE 125B dengan bobot 4-bit. Modelnya berjalan TP2xPP2 + EP dengan MTP +K=3. Pool KV FP8 menampung 924.993 token pada konteks 262.144 token. Keluaran +satu aliran diukur pada 158-184 tok/s. + +## 1. Kebutuhan + +| Item | Kebutuhan | +| ----------- | ----------------------------------------------------- | +| GPU | 4 kartu bebas 24 GB, semua model yang sama | +| RAM | 96 GiB atau lebih. Tabel PLE memakai 69 GiB RAM host. | +| `/dev/shm` | 1 GB atau lebih | +| Driver | CUDA 13.0 atau lebih baru | +| Disk | 30 GB bebas untuk build, 115 GB untuk checkpoint | +| Python | 3.13 | +| Build tools | `cmake` 3.26 atau lebih baru, `ninja`, `cc`, `c++` | + +Checkpoint harus berisi `qsa_kv_scales_swift.json`. File itu menyimpan scale KV +FP8. Swift 1.5 mengubah `k_proj`/`v_proj`, jadi file scale satu checkpoint tidak +bekerja untuk checkpoint lain. + +## 2. Cara Build + +```bash +sudo apt-get install -y cmake ninja-build ccache +./build.sh +``` + +`build.sh` menjalankan delapan langkah: tools, tree serving, venv, runtime +terkunci, toolchain CUDA, ops yang di-compile, install, dan pack kernel. Compile +pertama butuh 20 menit pada CPU 64 core. + +Dua variabel mengontrol compile: + +1. `MAX_JOBS` — jumlah job compile. Bawaannya `32`. Turunkan nilainya ketika + build kehabisan RAM. +2. `ARCH` — arsitektur GPU. Bawaannya `8.6` (Ampere). + +Build mencetak setiap langkah. Installer berjalan dengan `-v`, jadi setiap baris +compiler terlihat, dan satu baris status muncul setiap 60 detik. Output lengkap +masuk ke `build-.log` di root repo. + +### 2.1. Jalankan Ulang + +Jalankan ulang pada tree yang tidak berubah melewati compile dan install. Ia +membaca commit tree dan dua hash ekstensi dari `vllm/.build-products`, +meng-hash ulang `vllm/.build-products.sha256`, dan mencari install vllm di venv. +Setiap ketidakcocokan memulai compile. `./build.sh --force` selalu meng-compile. + +### 2.2. Tree + +Langkah 2 membangun `vllm/` dari tag upstream v0.30.0 plus `patches/`, lalu +membandingkan hasilnya dengan hash tree yang diharapkan. Tree yang benar dibiarkan, +jadi menjalankan ulang tidak butuh jaringan. `FETCH=1 ./build.sh` menerapkan ulang +seri patch dari base. `VLLM_UPSTREAM_URL=` menunjuk ke mirror. + +### 2.3. Tools di Mesin Build + +`build.sh` memakai `uv` untuk venv ketika `uv` ada di PATH, dan CPython 3.13 pada +kedua kasus. Set `UV=0` untuk menolak `uv`, `UV=/path/to/uv` untuk memilih satu. +Cache wheel milik user saat ini (`$HOME/.cache/uv`, `$HOME/.cache/pip`) dicetak +pada langkah 3 dan dipakai ulang. + +`build.sh` memakai `ccache` ketika ada di PATH, sebagai launcher compiler CMake +untuk C, C++, CUDA, dan HIP. Objek masuk ke direktori ccache milik user saat ini. +Langkah 1 mencetak direktorinya, langkah 7 mencetak jumlah hit dan miss. Set +`CCACHE_DIR=` untuk memindahkannya. Tanpa `ccache`, setiap kernel di-compile dari +nol. + +`cargo` 1.95 bersifat opsional. Jika ada, build meng-compile frontend Rust +(`vllm-rs`, `_rust_tool_parser`). Jika tidak ada, tool parser Python yang dipakai. + +## 3. Cara Menjalankan + +```bash +sudo ./start.sh +tail -f /var/log/vllm.log +``` + +Bawaan `start.sh`: `127.0.0.1:8000`, nama model `qwen38-flash-next`, log +`/var/log/vllm.log`. Compose menyetel `MODEL_NAME=Hinata` sebagai bawaan. Install +systemd menulis `MODEL_NAME=Hinata` ke `/etc/default/vllm`. Pakai `sudo` untuk +file log. Set `HOST=0.0.0.0` untuk melayani semua interface. + +`start.sh` menyimpan cache compile di `.vllm-cache/` di direktori ini. + +```bash +CACHE_ROOT=/path/to/cache ./start.sh +``` + +Start pertama meng-compile cudagraphs ke `$CACHE_ROOT` dan butuh sekitar 6 menit. +Request pertama membangun kernel FlashInfer. Setiap start berikutnya butuh +sekitar 3 menit. + +## 4. Jalankan sebagai Service systemd + +```bash +sudo scripts/install-as-service.sh install +sudo vi /etc/default/vllm +sudo scripts/install-as-service.sh start +sudo scripts/install-as-service.sh status +``` + +`install` menulis ulang `/opt/vllm` di `systemd/vllm.service` ke path repo ini, +jadi service menjalankan repo ini di tempat. `update` menjalankan `./build.sh` +dan restart. `logs` menampilkan ekor log engine. `uninstall` menghapus unit, +file environment, dan file logrotate. + +## 5. Log + +`start.sh` menulis engine dan proses anak PLE offload ke `/var/log/vllm.log`. +Set `LOG=/other/path` untuk memindahkan file. Set `LOG=` (kosong) untuk +menampilkan output di terminal. Image memakai mode itu, jadi +`docker compose logs` bekerja. + +Pasang kebijakan rotasi: + +```bash +sudo install -m 0644 systemd/vllm.logrotate /etc/logrotate.d/vllm +``` + +Perintah itu menggantikan `/etc/logrotate.d/vllm` yang sudah ada di host. +Kebijakan itu merotasi file setiap hari, pada 100 MB. Ia menyimpan 14 file, +mengompresnya, dan menghapus log lebih tua dari 60 hari. Container membatasi +log-nya sendiri dengan cara yang sama (`json-file`, 100 MB x 14). + +`vllm-logs.sh` mencetak baris statistik engine sebagai satu baris berwarna per +pembaruan (throughput, pemakaian KV, prefix cache, penerimaan draft). Baca log +container, atau log file dengan `-f`: + +```bash +./vllm-logs.sh # docker logs dari container vllm, 10 menit terakhir +./vllm-logs.sh -f -n 500 # ekor /var/log/vllm.log, dari 500 baris terakhir +``` + +## 6. Uji Server + +`/v1/models` mengembalikan 200 ketika engine mati. Lakukan generate satu token. +Set `model` ke nama yang dilayani, nilai `MODEL_NAME`: + +```bash +curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \ + -d '{"model":"qwen38-flash-next","messages":[{"role":"user","content":"hi"}],"max_tokens":1,\ + "chat_template_kwargs":{"enable_thinking":false}}' +``` + +Start yang sehat menulis baris ini ke log: + +``` +GPU KV cache size: 924,993 tokens +``` + +Angka itu berubah bersama setting memori KV. + +## 7. Container (opsional) + +Pakai container ketika image dibangun di mesin tanpa GPU dan tanpa driver, lalu +berjalan di host GPU. + +Langkah 8 dari `build.sh` menulis kernel yang sudah di-compile ke `artifacts/`: +satu tarball, daftar sha256 per file, dan satu manifest. `--no-pack` (`PACK=0`) +melewati langkah itu. Jalan kedua pada tree yang sama tidak mem-pack ulang. + +Ada dua route build: + +```bash +# kernel dari artifacts/, tanpa compile, sekitar 2 menit +docker build --build-context ops="$PWD/artifacts" -t vllm:latest . + +# compile penuh, 20-40 menit, disk build 15 GB +docker build -t vllm:latest . +``` + +`docker compose build` mengirim context `ops` dari `artifacts/` sendiri. + +Build memeriksa tarball sebelum ia memasangnya: head tree, arsitektur GPU, +CPython minor, CUDA major, versi torch, dan sha256 setiap file hasil ekstrak. +Tarball yang tidak cocok dengan tree menghentikan build dan menyebut item yang +tidak cocok. + +Build image butuh akses jaringan: ia mengunduh vLLM v0.30.0 dan wheel terkunci. +Base-nya Debian trixie. Kernel yang dibangun di Debian 13 butuh glibc 2.38, dan +bookworm (glibc 2.36) tidak dapat memuatnya. `cmake` 3.25 di trixie terlalu tua, +jadi image memasang `cmake` dan `ninja` dari pip. Image membawa toolchain +compiler, jadi ukurannya besar. + +Build image menerima dua argumen compile yang sama dengan `build.sh`: + +```bash +docker build -t vllm:latest --build-arg MAX_JOBS=16 --build-arg ARCH=8.6 . +``` + +### 7.1. Stub Driver CUDA + +Kernel men-link `CUDA::cuda_driver` (`libcuda.so.1`). Paket driver NVIDIA +menyediakan itu. Container build tanpa GPU tidak punya, dan wheel CUDA tidak +punya stub untuk itu, jadi langkah 5 menjalankan `scripts/gen-cuda-stub.sh`. Ia +menulis stub link-time di samping toolkit di venv: 548 entry point, tanpa body, +92 KB, soname `libcuda.so.1`. Tidak ada loader search path yang memasukkan +direktori itu, jadi ekstensi membuka driver asli saat jalan. Tanpa stub, `cmake` +berhenti dengan `Target "_C_stable_libtorch" links to: CUDA::cuda_driver but the +target was not found`. Di host driver ada, jadi langkah itu tidak melakukan apa +pun setelah pertama kali berjalan. + +Pemeriksaan import di akhir `build.sh` butuh driver juga. Tanpa driver, ia +memeriksa layer link dan mengatakannya. Import penuh berjalan saat serve di host +GPU. `libcuda.so.1` hanya diharapkan ada di +`vllm/_C_stable_libtorch.abi3.so`. Kernel MoE mendapat CUDA melalui +`libtorch_cuda`, jadi ia tidak ada di daftar `NEEDED` mereka. + +### 7.2. Cache Build + +Build membaca tiga BuildKit cache mounts: cache wheel `uv`, cache `pip`, dan +cache objek `ccache`. Ketiganya hidup di luar image, jadi rebuild tidak +mengunduh apa pun dan hanya meng-compile kernel yang sumbernya berubah. +Ketiganya mulai dalam keadaan kosong, jadi build pertama mengunduh sekitar 6 GB +wheel. Kirim cache milik user saat ini untuk mencegah itu: + +```bash +docker build -t vllm:latest \ + --build-context uvcache="$HOME/.cache/uv" \ + --build-context pipcache="$HOME/.cache/pip" . +``` + +Build membaca folder itu. Build tidak pernah menulis ke dalamnya. + +### 7.3. Jalankan dengan Docker Compose + +```bash +sudo nvidia-ctk runtime configure --runtime=docker +sudo systemctl restart docker +cp .env.example .env +mkdir -p .vllm-cache "$HOME/.cache/uv" "$HOME/.cache/pip" +docker compose up -d +docker compose logs -f +``` + +`docker compose up` mengirim tiga context ke build image: `artifacts` (kernel +dari `./build.sh`), `uvcache` dan `pipcache` (cache wheel user ini). Masing-masing +harus folder yang ada. `.vllm-cache` adalah folder cache saat jalan, di-mount ke +`/cache`. + +### 7.4. Jalankan Image Tanpa Compose + +```bash +docker run -d --name vllm --gpus all --stop-timeout 20 --shm-size=16g \ + --ulimit memlock=-1 -p 8080:8000 \ + -v /data/models/checkpoint:/models/model:ro \ + -v "$PWD/.vllm-cache":/cache \ + -e MODEL_NAME=Hinata vllm:latest +``` + +## 8. File + +| File | Guna | +| -------------------------------------------------- | --------------------------------------------------------------------------------- | +| `build.sh` | Ambil v0.30.0, pasang `patches/`, build `.venv/` (py3.13, 196 pin), compile, pack | +| `start.sh` | Build pada jalan pertama, uji mesin, cache di `.vllm-cache/`, serve, tulis log | +| `vllm-logs.sh` | Baca log engine sebagai satu baris statistik per pembaruan | +| `scripts/install-as-service.sh` | Pasang unit systemd untuk repo ini, start, update, log, uninstall | +| `systemd/` | `vllm.service` dan kebijakan logrotate | +| `Dockerfile`, `docker-compose.yml`, `.env.example` | Jalur container | +| `scripts/` | Pemilih topologi, requirements terkunci, stub driver CUDA | +| `patches/` | 103 patch, dalam urutan pasang | +| `benchmarks/` | Wrapper `bench.sh`, dan benchmark power limit (Inggris, Indonesia) | +| `abliteration/` | File arah refusal untuk `QWEN38_ABLITERATION` | + +## 9. Variabel + +| Variabel | Bawaan | Arti | +| ---------------------- | ---------------------------------------------------- | -------------------------------------------------------------------------------------- | +| `MODEL_DIR` | tidak ada, wajib | Direktori host tempat checkpoint. Compose berhenti dengan pesan ketika kosong. | +| `MODEL_NAME` | `Hinata` (compose), `qwen38-flash-next` (`start.sh`) | Nama di field `model` pada request. Spasi membuat alias. | +| `HOST_PORT` | `8080` | Port host untuk API. | +| `VLLM_API_KEY` | kosong | Kosong berarti tanpa autentikasi. | +| `CACHE_HOST_DIR` | `.vllm-cache` di direktori ini | Tempat cache compile berada. Jalur host memakai folder yang sama. | +| `CUDA_VISIBLE_DEVICES` | `0,1,2,3` | Jalur GPU, dalam urutan bus. | +| `PLE_HOME` | `0` | Kartu yang menjadi inang PLE offload. Kartu itu tidak mendapat CUDA context untuk itu. | +| `QWEN38_ABLITERATION` | `Qwen3.8-Flash-Next-Uncensored` | Hapus arah refusal saat jalan. Kosong menonaktifkannya. Lihat `patches/0102`. | +| `ABLITERATION_DIR` | `./abliteration` | Folder host dengan file arah itu. Compose mem-mount ke `/opt/qwen38/abliteration`. | +| `GUARD` | `warn` | Telemetri bounds-guard: `off`, `warn`, atau `raise`. | +| `COUNTERS` | `0` | `1` menambah counter kernel khusus log. | +| `MAX_JOBS`, `ARCH` | `32`, `8.6` | Job compile dan arsitektur GPU, hanya untuk build. | + +## 10. Benchmark + +`benchmarks/bench.sh` menjalankan `llama-benchy` terhadap server lokal. Pasang +tool-nya lebih dulu: + +```bash +uv tool install git+https://github.com/eugr/llama-benchy +``` + +Jalankan dari root repo: + +```bash +benchmarks/bench.sh [PP] [TG] [CONCURRENCY] [DEPTH] [arg llama-benchy tambahan] +``` + +| Argumen | Bawaan | Arti | +| ------------- | ------- | ------------------------------------------------------------------------- | +| `PP` | `32768` | Token prompt. Pakai `1` untuk hanya decode. | +| `TG` | `2048` | Token yang dihasilkan. Pakai `1` untuk hanya prefill. | +| `CONCURRENCY` | `1` | Satu tingkat, atau daftar dengan pemisah spasi, contoh `"1 2 4"`. | +| `DEPTH` | `0` | Token percakapan sebelumnya sebelum prompt yang diukur. Daftar juga bisa. | + +Argumen tambahan diteruskan ke `llama-benchy`. Script mengunci URL server, nama +model, dan path tokenizer. Ganti dengan `BASE_URL`, `MODEL`, dan `TOKENIZER`. + +```bash +benchmarks/bench.sh 32768 1 # kecepatan prefill +benchmarks/bench.sh 1 4096 # kecepatan decode +benchmarks/bench.sh 32768 2048 "1 2 4" # sweep concurrency +benchmarks/bench.sh 128 256 1 "0 8192 65536" # sweep depth, turn kecil +``` + +`DEPTH` mengubah apa yang diukur satu run. `--pp 8320 --depth 0` mengukur satu +prompt 8K yang dingin. `--pp 128 --depth 8192` mengukur turn 128 token di dalam +percakapan 8K. Kecepatan decode turun seiring depth. Satu turn 128 token yang +diukur: 186 t/s pada depth 0, lalu 156 t/s pada depth 8192. + +Tambah `--enable-prefix-caching` untuk mengukur dua fase dalam satu run. Tool +itu lalu mencetak baris dingin (ia memuat prefix) dan baris hangat (server +memakai ulang cache). Tambah `--runs N` untuk lebih banyak run per sel, dan +`--no-cache` untuk melewati uji koherensi. + +`benchmarks/gpu-power-limit-effects-en.md` dan `benchmarks/gpu-power-limit-effects-id.md` +mencatat sweep power limit (220 W, 250 W, 285 W, PCIe Gen4 dan Gen3) dengan +metode lengkap dan rekomendasinya. diff --git a/README.md b/README.md index 0f9572b..067bb0e 100644 --- a/README.md +++ b/README.md @@ -4,6 +4,9 @@ Swift-1.5-Qwen3.8-Flash-Next-W4A16 served from four RTX 3090 cards. The model is MoE with 4-bit weights. It runs TP2xPP2 + EP with MTP K=3. The FP8 KV pool holds 924,993 tokens at a 262,144-token context. Single-stream output measured at 158-184 tok/s. +An Indonesian translation is in `README-ID.md`. This file is the source. When the two +differ, this file wins. + ## 1. Requirements | Item | Requirement | @@ -77,8 +80,9 @@ tail -f /var/log/vllm.log ``` `start.sh` defaults: `127.0.0.1:8000`, model name `qwen38-flash-next`, log -`/var/log/vllm.log`. The container and the systemd unit set `MODEL_NAME=Scarlet`. Use -`sudo` for the log file. Set `HOST=0.0.0.0` to serve on all interfaces. +`/var/log/vllm.log`. Compose sets `MODEL_NAME=Hinata` by default. The systemd +install writes `MODEL_NAME=Hinata` to `/etc/default/vllm`. Use `sudo` for the +log file. Set `HOST=0.0.0.0` to serve on all interfaces. `start.sh` keeps the compile caches in `.vllm-cache/` in this directory. @@ -118,13 +122,23 @@ That command replaces an existing `/etc/logrotate.d/vllm` on the host. The polic the file daily, at 100 MB. It keeps 14 files, compresses them, and deletes logs older than 60 days. The container caps its own log the same way (`json-file`, 100 MB x 14). +`vllm-logs.sh` prints the engine stats lines as one colored line per update +(throughput, KV usage, prefix cache, draft acceptance). Read the container log, or the +file log with `-f`: + +```bash +./vllm-logs.sh # docker logs of the vllm container, last 10 minutes +./vllm-logs.sh -f -n 500 # tail /var/log/vllm.log, from the last 500 lines +``` + ## 6. Test the server -`/v1/models` returns 200 when the engine is dead. Do a one-token generation: +`/v1/models` returns 200 when the engine is dead. Do a one-token generation. Set +`model` to the served name, the `MODEL_NAME` value: ```bash curl -s localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \ - -d '{"model":"Scarlet","messages":[{"role":"user","content":"hi"}],"max_tokens":1,\ + -d '{"model":"qwen38-flash-next","messages":[{"role":"user","content":"hi"}],"max_tokens":1,\ "chat_template_kwargs":{"enable_thinking":false}}' ``` @@ -226,7 +240,7 @@ docker run -d --name vllm --gpus all --stop-timeout 20 --shm-size=16g \ --ulimit memlock=-1 -p 8080:8000 \ -v /data/models/checkpoint:/models/model:ro \ -v "$PWD/.vllm-cache":/cache \ - -e MODEL_NAME=Scarlet vllm:latest + -e MODEL_NAME=Hinata vllm:latest ``` ## 8. Files @@ -235,21 +249,74 @@ docker run -d --name vllm --gpus all --stop-timeout 20 --shm-size=16g \ | -------------------------------------------------- | --------------------------------------------------------------------------------- | | `build.sh` | Fetch v0.30.0, apply `patches/`, build `.venv/` (py3.13, 196 pins), compile, pack | | `start.sh` | Build on first run, test the box, cache in `.vllm-cache/`, serve, write the log | +| `vllm-logs.sh` | Read the engine log as one stats line per update | | `scripts/install-as-service.sh` | Install the systemd unit for this repo, start, update, log, uninstall | | `systemd/` | `vllm.service` and the logrotate policy | | `Dockerfile`, `docker-compose.yml`, `.env.example` | The container path | +| `README.md`, `README-ID.md` | This guide, and the Indonesian translation | | `scripts/` | Topology selector, pinned requirements, CUDA driver stub | -| `patches/` | The 100 patches, in apply order | +| `patches/` | The 103 patches, in apply order | +| `benchmarks/` | `bench.sh` wrapper, and the power-limit benchmark (English, Indonesian) | +| `abliteration/` | The refusal-direction file for `QWEN38_ABLITERATION` | ## 9. Variables -| Variable | Default | Meaning | -| ------------------------- | ----------------------------------------------------- | -------------------------------------------------------------------------------- | -| `MODEL_DIR` | none, required | Host directory of the checkpoint. Compose stops with a message when it is empty. | -| `MODEL_NAME` | `Scarlet` (compose), `qwen38-flash-next` (`start.sh`) | The name in the `model` field of a request. Spaces make aliases. | -| `HOST_PORT` | `8080` | The host port of the API. | -| `VLLM_API_KEY` | empty | Empty means no authentication. | -| `CACHE_HOST_DIR` | `.vllm-cache` in this directory | Where the compile caches live. The host path uses the same folder. | -| `CUDA_VISIBLE_DEVICES` | `0,1,2,3` | The GPU lane, in bus order. | -| `PLE_HOME` | `0` | The card that hosts the PLE offload. That card gets no CUDA context for it. | -| `MAX_JOBS`, `ARCH` | `32`, `8.6` | Compile jobs and GPU architecture, for the build only. | +| Variable | Default | Meaning | +| ---------------------- | ---------------------------------------------------- | ------------------------------------------------------------------------------------- | +| `MODEL_DIR` | none, required | Host directory of the checkpoint. Compose stops with a message when it is empty. | +| `MODEL_NAME` | `Hinata` (compose), `qwen38-flash-next` (`start.sh`) | The name in the `model` field of a request. Spaces make aliases. | +| `HOST_PORT` | `8080` | The host port of the API. | +| `VLLM_API_KEY` | empty | Empty means no authentication. | +| `CACHE_HOST_DIR` | `.vllm-cache` in this directory | Where the compile caches live. The host path uses the same folder. | +| `CUDA_VISIBLE_DEVICES` | `0,1,2,3` | The GPU lane, in bus order. | +| `PLE_HOME` | `0` | The card that hosts the PLE offload. That card gets no CUDA context for it. | +| `QWEN38_ABLITERATION` | `Qwen3.8-Flash-Next-Uncensored` | Remove the refusal direction at run time. Empty disables it. See `patches/0102`. | +| `ABLITERATION_DIR` | `./abliteration` | Host folder with the direction file. Compose mounts it at `/opt/qwen38/abliteration`. | +| `GUARD` | `warn` | Bounds-guard telemetry: `off`, `warn`, or `raise`. | +| `COUNTERS` | `0` | `1` adds the logging-only kernel counters. | +| `MAX_JOBS`, `ARCH` | `32`, `8.6` | Compile jobs and GPU architecture, for the build only. | + +## 10. Benchmarks + +`benchmarks/bench.sh` runs `llama-benchy` against the local server. Install the tool +first: + +```bash +uv tool install git+https://github.com/eugr/llama-benchy +``` + +Run it from the repo root: + +```bash +benchmarks/bench.sh [PP] [TG] [CONCURRENCY] [DEPTH] [extra llama-benchy args] +``` + +| Argument | Default | Meaning | +| ------------- | ------- | ----------------------------------------------------------------------- | +| `PP` | `32768` | Prompt tokens. Use `1` for decode-only. | +| `TG` | `2048` | Generated tokens. Use `1` for prefill-only. | +| `CONCURRENCY` | `1` | One level, or a space-separated list, for example `"1 2 4"`. | +| `DEPTH` | `0` | Prior-conversation tokens before the measured prompt. A list works too. | + +Extra arguments go to `llama-benchy`. The script fixes the server URL, the model name, +and the tokenizer path. Override them with `BASE_URL`, `MODEL`, and `TOKENIZER`. + +```bash +benchmarks/bench.sh 32768 1 # prefill speed +benchmarks/bench.sh 1 4096 # decode speed +benchmarks/bench.sh 32768 2048 "1 2 4" # concurrency sweep +benchmarks/bench.sh 128 256 1 "0 8192 65536" # depth sweep, small turn +``` + +`DEPTH` changes what the run measures. `--pp 8320 --depth 0` prices one cold 8K +prompt. `--pp 128 --depth 8192` prices a 128-token turn inside an 8K conversation. +Decode speed falls with depth. One measured 128-token turn: 186 t/s at depth 0, then +156 t/s at depth 8192. + +Add `--enable-prefix-caching` to measure both phases in one run. The tool then prints a +cold row (it loads the prefix) and a warm row (the server reuses the cache). Add +`--runs N` for more runs per cell, and `--no-cache` to skip the coherence test. + +`benchmarks/gpu-power-limit-effects-en.md` and `benchmarks/gpu-power-limit-effects-id.md` +record a power-limit sweep (220 W, 250 W, 285 W, PCIe Gen4 and Gen3) with the full +method and the recommendation. diff --git a/scripts/install-as-service.sh b/scripts/install-as-service.sh index fced26e..1725d15 100755 --- a/scripts/install-as-service.sh +++ b/scripts/install-as-service.sh @@ -41,7 +41,7 @@ case "$cmd" in if [ ! -f "$DEFAULT" ]; then { echo "# Read by $UNIT.service. Uncomment and edit, then: systemctl restart $UNIT" echo "CKPT=$ckpt" - echo "MODEL_NAME=Scarlet" + echo "MODEL_NAME=Hinata" echo "# LOG=/var/log/vllm.log" echo "# CACHE_ROOT=$HERE/.vllm-cache" echo "# CUDA_VISIBLE_DEVICES=0,1,2,3"