iHore/vLLM-4x-RTX-3090
8
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: Document the KV scale sidecar and the model variables

Browse Source 
SCALES had no row in the Variables table, and CKPT and MODEL_MOUNT_PATH
had none, so a reader had to open start.sh and docker-compose.yml to
place a scale file outside the model folder.

Add the three rows to the Variables table, say in section 1 that SCALES
takes a host path and that Compose mounts it, and give the -v and -e
pair for a start without Compose in section 7.4. Sync README-ID.md.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
This commit is contained in:
MOVZX
2026-10-15 18:52:18 +00:00
co-authored by Hinata
parent b89e85e99f
commit af239c4229
2 changed files with 50 additions and 34 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README-ID.md
+25 -17
View File
@@ -20,9 +20,12 @@ satu aliran diukur pada 158-184 tok/s.
| Python | 3.13 |
| Build tools | `cmake` 3.26 atau lebih baru, `ninja`, `cc`, `c++` |
Checkpoint harus berisi `qsa_kv_scales.json`. File itu menyimpan scale KV
FP8. Swift 1.5 mengubah `k_proj`/`v_proj`, jadi file scale satu checkpoint tidak
bekerja untuk checkpoint lain.
Checkpoint harus berisi `qsa_kv_scales.json`. File itu menyimpan scale KV FP8.
Swift 1.5 mengubah `k_proj`/`v_proj`, jadi file scale satu checkpoint tidak bekerja
untuk checkpoint lain.
`SCALES` menunjuk ke sidecar lain. Beri path host; Compose mem-mount-nya ke container.
Bawaannya `$MODEL_DIR/qsa_kv_scales.json`.
## 2. Cara Build
@@ -255,6 +258,8 @@ docker run -d --name vllm --gpus all --stop-timeout 20 --shm-size=16g \
-e MODEL_NAME=Hinata vllm:latest
```
Tanpa Compose, mount sendiri sidecar-nya: `-v <host>/scales.json:/opt/vllm/qsa_kv_scales.json:ro -e SCALES=/opt/vllm/qsa_kv_scales.json`.
## 8. File
| File | Guna |
@@ -272,20 +277,23 @@ docker run -d --name vllm --gpus all --stop-timeout 20 --shm-size=16g \
## 9. Variabel
| Variabel | Bawaan | Arti |
| ---------------------- | ---------------------------------------------------- | -------------------------------------------------------------------------------------- |
| `MODEL_DIR` | tidak ada, wajib | Direktori host tempat checkpoint. Compose berhenti dengan pesan ketika kosong. |
| `MODEL_NAME` | `Hinata` (compose), `qwen38-flash-next` (`start.sh`) | Nama di field `model` pada request. Spasi membuat alias. |
| `HOST_PORT` | `8080` | Port host untuk API. |
| `VLLM_API_KEY` | kosong | Kosong berarti tanpa autentikasi. |
| `CACHE_HOST_DIR` | `.vllm-cache` di direktori ini | Tempat cache compile berada. Jalur host memakai folder yang sama. |
| `CUDA_VISIBLE_DEVICES` | `0,1,2,3` | Jalur GPU, dalam urutan bus. |
| `PLE_HOME` | `0` | Kartu yang menjadi inang PLE offload. Kartu itu tidak mendapat CUDA context untuk itu. |
| `QWEN38_ABLITERATION` | `Qwen3.8-Flash-Next-Uncensored` | Hapus arah refusal saat jalan. Kosong menonaktifkannya. Lihat `patches/0102`. |
| `ABLITERATION_DIR` | `./abliteration` | Folder host dengan file arah itu. Compose mem-mount ke `/opt/qwen38/abliteration`. |
| `GUARD` | `warn` | Telemetri bounds-guard: `off`, `warn`, atau `raise`. |
| `COUNTERS` | `0` | `1` menambah counter kernel khusus log. |
| `MAX_JOBS`, `ARCH` | `32`, `8.6` | Job compile dan arsitektur GPU, hanya untuk build. |
| Variabel | Bawaan | Arti |
| ---------------------- | -------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
| `MODEL_DIR` | tidak ada, wajib | Direktori host tempat checkpoint. Compose berhenti dengan pesan ketika kosong. |
| `CKPT` | `/data/models/halt95/Swift1.5-Qwen3.8-Flash-Next-W4A16-Merlin` | Folder checkpoint di host. Argumen pertama `start.sh` mengaturnya. |
| `MODEL_MOUNT_PATH` | `/models/Qwen3.8-Flash-Next` | Path checkpoint di dalam container. Compose mem-mount `MODEL_DIR` ke sini. |
| `SCALES` | `$MODEL_DIR/qsa_kv_scales.json` | Path host sidecar scale KV FP8. Compose mem-mount-nya ke container. |
| `MODEL_NAME` | `Hinata` (compose), `qwen38-flash-next` (`start.sh`) | Nama di field `model` pada request. Spasi membuat alias. |
| `HOST_PORT` | `8080` | Port host untuk API. |
| `VLLM_API_KEY` | kosong | Kosong berarti tanpa autentikasi. |
| `CACHE_HOST_DIR` | `.vllm-cache` di direktori ini | Tempat cache compile berada. Jalur host memakai folder yang sama. |
| `CUDA_VISIBLE_DEVICES` | `0,1,2,3` | Jalur GPU, dalam urutan bus. |
| `PLE_HOME` | `0` | Kartu yang menjadi inang PLE offload. Kartu itu tidak mendapat CUDA context untuk itu. |
| `QWEN38_ABLITERATION` | `Qwen3.8-Flash-Next-Uncensored` | Hapus arah refusal saat jalan. Kosong menonaktifkannya. Lihat `patches/0102`. |
| `ABLITERATION_DIR` | `./abliteration` | Folder host dengan file arah itu. Compose mem-mount ke `/opt/qwen38/abliteration`. |
| `GUARD` | `warn` | Telemetri bounds-guard: `off`, `warn`, atau `raise`. |
| `COUNTERS` | `0` | `1` menambah counter kernel khusus log. |
| `MAX_JOBS`, `ARCH` | `32`, `8.6` | Job compile dan arsitektur GPU, hanya untuk build. |
## 10. Benchmark
README.md
+25 -17
View File
@@ -19,9 +19,12 @@ differ, this file wins.
| Python | 3.13 |
| Build tools | `cmake` 3.26 or newer, `ninja`, `cc`, `c++` |
The checkpoint must contain `qsa_kv_scales.json`. That file holds the FP8 KV
scales. Swift 1.5 changed `k_proj`/`v_proj`, so the scale file of one checkpoint does not
work for another.
The checkpoint must contain `qsa_kv_scales.json`. That file holds the FP8 KV scales.
Swift 1.5 changed `k_proj`/`v_proj`, so the scale file of one checkpoint does not work
for another.
`SCALES` names another sidecar. Give a host path; Compose mounts it into the container.
The default is `$MODEL_DIR/qsa_kv_scales.json`.
## 2. How to Build
@@ -243,6 +246,8 @@ docker run -d --name vllm --gpus all --stop-timeout 20 --shm-size=16g \
-e MODEL_NAME=Hinata vllm:latest
```
Without Compose, mount the sidecar yourself: `-v <host>/scales.json:/opt/vllm/qsa_kv_scales.json:ro -e SCALES=/opt/vllm/qsa_kv_scales.json`.
## 8. Files
| File | Use |
@@ -261,20 +266,23 @@ docker run -d --name vllm --gpus all --stop-timeout 20 --shm-size=16g \
## 9. Variables
| Variable | Default | Meaning |
| ---------------------- | ---------------------------------------------------- | ------------------------------------------------------------------------------------- |
| `MODEL_DIR` | none, required | Host directory of the checkpoint. Compose stops with a message when it is empty. |
| `MODEL_NAME` | `Hinata` (compose), `qwen38-flash-next` (`start.sh`) | The name in the `model` field of a request. Spaces make aliases. |
| `HOST_PORT` | `8080` | The host port of the API. |
| `VLLM_API_KEY` | empty | Empty means no authentication. |
| `CACHE_HOST_DIR` | `.vllm-cache` in this directory | Where the compile caches live. The host path uses the same folder. |
| `CUDA_VISIBLE_DEVICES` | `0,1,2,3` | The GPU lane, in bus order. |
| `PLE_HOME` | `0` | The card that hosts the PLE offload. That card gets no CUDA context for it. |
| `QWEN38_ABLITERATION` | `Qwen3.8-Flash-Next-Uncensored` | Remove the refusal direction at run time. Empty disables it. See `patches/0102`. |
| `ABLITERATION_DIR` | `./abliteration` | Host folder with the direction file. Compose mounts it at `/opt/qwen38/abliteration`. |
| `GUARD` | `warn` | Bounds-guard telemetry: `off`, `warn`, or `raise`. |
| `COUNTERS` | `0` | `1` adds the logging-only kernel counters. |
| `MAX_JOBS`, `ARCH` | `32`, `8.6` | Compile jobs and GPU architecture, for the build only. |
| Variable | Default | Meaning |
| ---------------------- | -------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
| `MODEL_DIR` | none, required | Host directory of the checkpoint. Compose stops with a message when it is empty. |
| `CKPT` | `/data/models/halt95/Swift1.5-Qwen3.8-Flash-Next-W4A16-Merlin` | Checkpoint folder on the host. The first argument to `start.sh` sets it. |
| `MODEL_MOUNT_PATH` | `/models/Qwen3.8-Flash-Next` | Path of the checkpoint inside the container. Compose mounts `MODEL_DIR` here. |
| `SCALES` | `$MODEL_DIR/qsa_kv_scales.json` | Host path of the FP8 KV scale sidecar. Compose mounts it into the container. |
| `MODEL_NAME` | `Hinata` (compose), `qwen38-flash-next` (`start.sh`) | The name in the `model` field of a request. Spaces make aliases. |
| `HOST_PORT` | `8080` | The host port of the API. |
| `VLLM_API_KEY` | empty | Empty means no authentication. |
| `CACHE_HOST_DIR` | `.vllm-cache` in this directory | Where the compile caches live. The host path uses the same folder. |
| `CUDA_VISIBLE_DEVICES` | `0,1,2,3` | The GPU lane, in bus order. |
| `PLE_HOME` | `0` | The card that hosts the PLE offload. That card gets no CUDA context for it. |
| `QWEN38_ABLITERATION` | `Qwen3.8-Flash-Next-Uncensored` | Remove the refusal direction at run time. Empty disables it. See `patches/0102`. |
| `ABLITERATION_DIR` | `./abliteration` | Host folder with the direction file. Compose mounts it at `/opt/qwen38/abliteration`. |
| `GUARD` | `warn` | Bounds-guard telemetry: `off`, `warn`, or `raise`. |
| `COUNTERS` | `0` | `1` adds the logging-only kernel counters. |
| `MAX_JOBS`, `ARCH` | `32`, `8.6` | Compile jobs and GPU architecture, for the build only. |
## 10. Benchmarks