docs: Document the KV scale sidecar and the model variables
SCALES had no row in the Variables table, and CKPT and MODEL_MOUNT_PATH had none, so a reader had to open start.sh and docker-compose.yml to place a scale file outside the model folder. Add the three rows to the Variables table, say in section 1 that SCALES takes a host path and that Compose mounts it, and give the -v and -e pair for a start without Compose in section 7.4. Sync README-ID.md. Co-Authored-By: Hinata <hinata@idihore.id> Signed-off-by: MOVZX <movzx@yahoo.com>
This commit is contained in:
+25
-17
@@ -20,9 +20,12 @@ satu aliran diukur pada 158-184 tok/s.
|
|||||||
| Python | 3.13 |
|
| Python | 3.13 |
|
||||||
| Build tools | `cmake` 3.26 atau lebih baru, `ninja`, `cc`, `c++` |
|
| Build tools | `cmake` 3.26 atau lebih baru, `ninja`, `cc`, `c++` |
|
||||||
|
|
||||||
Checkpoint harus berisi `qsa_kv_scales.json`. File itu menyimpan scale KV
|
Checkpoint harus berisi `qsa_kv_scales.json`. File itu menyimpan scale KV FP8.
|
||||||
FP8. Swift 1.5 mengubah `k_proj`/`v_proj`, jadi file scale satu checkpoint tidak
|
Swift 1.5 mengubah `k_proj`/`v_proj`, jadi file scale satu checkpoint tidak bekerja
|
||||||
bekerja untuk checkpoint lain.
|
untuk checkpoint lain.
|
||||||
|
|
||||||
|
`SCALES` menunjuk ke sidecar lain. Beri path host; Compose mem-mount-nya ke container.
|
||||||
|
Bawaannya `$MODEL_DIR/qsa_kv_scales.json`.
|
||||||
|
|
||||||
## 2. Cara Build
|
## 2. Cara Build
|
||||||
|
|
||||||
@@ -255,6 +258,8 @@ docker run -d --name vllm --gpus all --stop-timeout 20 --shm-size=16g \
|
|||||||
-e MODEL_NAME=Hinata vllm:latest
|
-e MODEL_NAME=Hinata vllm:latest
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Tanpa Compose, mount sendiri sidecar-nya: `-v <host>/scales.json:/opt/vllm/qsa_kv_scales.json:ro -e SCALES=/opt/vllm/qsa_kv_scales.json`.
|
||||||
|
|
||||||
## 8. File
|
## 8. File
|
||||||
|
|
||||||
| File | Guna |
|
| File | Guna |
|
||||||
@@ -272,20 +277,23 @@ docker run -d --name vllm --gpus all --stop-timeout 20 --shm-size=16g \
|
|||||||
|
|
||||||
## 9. Variabel
|
## 9. Variabel
|
||||||
|
|
||||||
| Variabel | Bawaan | Arti |
|
| Variabel | Bawaan | Arti |
|
||||||
| ---------------------- | ---------------------------------------------------- | -------------------------------------------------------------------------------------- |
|
| ---------------------- | -------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
|
||||||
| `MODEL_DIR` | tidak ada, wajib | Direktori host tempat checkpoint. Compose berhenti dengan pesan ketika kosong. |
|
| `MODEL_DIR` | tidak ada, wajib | Direktori host tempat checkpoint. Compose berhenti dengan pesan ketika kosong. |
|
||||||
| `MODEL_NAME` | `Hinata` (compose), `qwen38-flash-next` (`start.sh`) | Nama di field `model` pada request. Spasi membuat alias. |
|
| `CKPT` | `/data/models/halt95/Swift1.5-Qwen3.8-Flash-Next-W4A16-Merlin` | Folder checkpoint di host. Argumen pertama `start.sh` mengaturnya. |
|
||||||
| `HOST_PORT` | `8080` | Port host untuk API. |
|
| `MODEL_MOUNT_PATH` | `/models/Qwen3.8-Flash-Next` | Path checkpoint di dalam container. Compose mem-mount `MODEL_DIR` ke sini. |
|
||||||
| `VLLM_API_KEY` | kosong | Kosong berarti tanpa autentikasi. |
|
| `SCALES` | `$MODEL_DIR/qsa_kv_scales.json` | Path host sidecar scale KV FP8. Compose mem-mount-nya ke container. |
|
||||||
| `CACHE_HOST_DIR` | `.vllm-cache` di direktori ini | Tempat cache compile berada. Jalur host memakai folder yang sama. |
|
| `MODEL_NAME` | `Hinata` (compose), `qwen38-flash-next` (`start.sh`) | Nama di field `model` pada request. Spasi membuat alias. |
|
||||||
| `CUDA_VISIBLE_DEVICES` | `0,1,2,3` | Jalur GPU, dalam urutan bus. |
|
| `HOST_PORT` | `8080` | Port host untuk API. |
|
||||||
| `PLE_HOME` | `0` | Kartu yang menjadi inang PLE offload. Kartu itu tidak mendapat CUDA context untuk itu. |
|
| `VLLM_API_KEY` | kosong | Kosong berarti tanpa autentikasi. |
|
||||||
| `QWEN38_ABLITERATION` | `Qwen3.8-Flash-Next-Uncensored` | Hapus arah refusal saat jalan. Kosong menonaktifkannya. Lihat `patches/0102`. |
|
| `CACHE_HOST_DIR` | `.vllm-cache` di direktori ini | Tempat cache compile berada. Jalur host memakai folder yang sama. |
|
||||||
| `ABLITERATION_DIR` | `./abliteration` | Folder host dengan file arah itu. Compose mem-mount ke `/opt/qwen38/abliteration`. |
|
| `CUDA_VISIBLE_DEVICES` | `0,1,2,3` | Jalur GPU, dalam urutan bus. |
|
||||||
| `GUARD` | `warn` | Telemetri bounds-guard: `off`, `warn`, atau `raise`. |
|
| `PLE_HOME` | `0` | Kartu yang menjadi inang PLE offload. Kartu itu tidak mendapat CUDA context untuk itu. |
|
||||||
| `COUNTERS` | `0` | `1` menambah counter kernel khusus log. |
|
| `QWEN38_ABLITERATION` | `Qwen3.8-Flash-Next-Uncensored` | Hapus arah refusal saat jalan. Kosong menonaktifkannya. Lihat `patches/0102`. |
|
||||||
| `MAX_JOBS`, `ARCH` | `32`, `8.6` | Job compile dan arsitektur GPU, hanya untuk build. |
|
| `ABLITERATION_DIR` | `./abliteration` | Folder host dengan file arah itu. Compose mem-mount ke `/opt/qwen38/abliteration`. |
|
||||||
|
| `GUARD` | `warn` | Telemetri bounds-guard: `off`, `warn`, atau `raise`. |
|
||||||
|
| `COUNTERS` | `0` | `1` menambah counter kernel khusus log. |
|
||||||
|
| `MAX_JOBS`, `ARCH` | `32`, `8.6` | Job compile dan arsitektur GPU, hanya untuk build. |
|
||||||
|
|
||||||
## 10. Benchmark
|
## 10. Benchmark
|
||||||
|
|
||||||
|
|||||||
@@ -19,9 +19,12 @@ differ, this file wins.
|
|||||||
| Python | 3.13 |
|
| Python | 3.13 |
|
||||||
| Build tools | `cmake` 3.26 or newer, `ninja`, `cc`, `c++` |
|
| Build tools | `cmake` 3.26 or newer, `ninja`, `cc`, `c++` |
|
||||||
|
|
||||||
The checkpoint must contain `qsa_kv_scales.json`. That file holds the FP8 KV
|
The checkpoint must contain `qsa_kv_scales.json`. That file holds the FP8 KV scales.
|
||||||
scales. Swift 1.5 changed `k_proj`/`v_proj`, so the scale file of one checkpoint does not
|
Swift 1.5 changed `k_proj`/`v_proj`, so the scale file of one checkpoint does not work
|
||||||
work for another.
|
for another.
|
||||||
|
|
||||||
|
`SCALES` names another sidecar. Give a host path; Compose mounts it into the container.
|
||||||
|
The default is `$MODEL_DIR/qsa_kv_scales.json`.
|
||||||
|
|
||||||
## 2. How to Build
|
## 2. How to Build
|
||||||
|
|
||||||
@@ -243,6 +246,8 @@ docker run -d --name vllm --gpus all --stop-timeout 20 --shm-size=16g \
|
|||||||
-e MODEL_NAME=Hinata vllm:latest
|
-e MODEL_NAME=Hinata vllm:latest
|
||||||
```
|
```
|
||||||
|
|
||||||
|
Without Compose, mount the sidecar yourself: `-v <host>/scales.json:/opt/vllm/qsa_kv_scales.json:ro -e SCALES=/opt/vllm/qsa_kv_scales.json`.
|
||||||
|
|
||||||
## 8. Files
|
## 8. Files
|
||||||
|
|
||||||
| File | Use |
|
| File | Use |
|
||||||
@@ -261,20 +266,23 @@ docker run -d --name vllm --gpus all --stop-timeout 20 --shm-size=16g \
|
|||||||
|
|
||||||
## 9. Variables
|
## 9. Variables
|
||||||
|
|
||||||
| Variable | Default | Meaning |
|
| Variable | Default | Meaning |
|
||||||
| ---------------------- | ---------------------------------------------------- | ------------------------------------------------------------------------------------- |
|
| ---------------------- | -------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
|
||||||
| `MODEL_DIR` | none, required | Host directory of the checkpoint. Compose stops with a message when it is empty. |
|
| `MODEL_DIR` | none, required | Host directory of the checkpoint. Compose stops with a message when it is empty. |
|
||||||
| `MODEL_NAME` | `Hinata` (compose), `qwen38-flash-next` (`start.sh`) | The name in the `model` field of a request. Spaces make aliases. |
|
| `CKPT` | `/data/models/halt95/Swift1.5-Qwen3.8-Flash-Next-W4A16-Merlin` | Checkpoint folder on the host. The first argument to `start.sh` sets it. |
|
||||||
| `HOST_PORT` | `8080` | The host port of the API. |
|
| `MODEL_MOUNT_PATH` | `/models/Qwen3.8-Flash-Next` | Path of the checkpoint inside the container. Compose mounts `MODEL_DIR` here. |
|
||||||
| `VLLM_API_KEY` | empty | Empty means no authentication. |
|
| `SCALES` | `$MODEL_DIR/qsa_kv_scales.json` | Host path of the FP8 KV scale sidecar. Compose mounts it into the container. |
|
||||||
| `CACHE_HOST_DIR` | `.vllm-cache` in this directory | Where the compile caches live. The host path uses the same folder. |
|
| `MODEL_NAME` | `Hinata` (compose), `qwen38-flash-next` (`start.sh`) | The name in the `model` field of a request. Spaces make aliases. |
|
||||||
| `CUDA_VISIBLE_DEVICES` | `0,1,2,3` | The GPU lane, in bus order. |
|
| `HOST_PORT` | `8080` | The host port of the API. |
|
||||||
| `PLE_HOME` | `0` | The card that hosts the PLE offload. That card gets no CUDA context for it. |
|
| `VLLM_API_KEY` | empty | Empty means no authentication. |
|
||||||
| `QWEN38_ABLITERATION` | `Qwen3.8-Flash-Next-Uncensored` | Remove the refusal direction at run time. Empty disables it. See `patches/0102`. |
|
| `CACHE_HOST_DIR` | `.vllm-cache` in this directory | Where the compile caches live. The host path uses the same folder. |
|
||||||
| `ABLITERATION_DIR` | `./abliteration` | Host folder with the direction file. Compose mounts it at `/opt/qwen38/abliteration`. |
|
| `CUDA_VISIBLE_DEVICES` | `0,1,2,3` | The GPU lane, in bus order. |
|
||||||
| `GUARD` | `warn` | Bounds-guard telemetry: `off`, `warn`, or `raise`. |
|
| `PLE_HOME` | `0` | The card that hosts the PLE offload. That card gets no CUDA context for it. |
|
||||||
| `COUNTERS` | `0` | `1` adds the logging-only kernel counters. |
|
| `QWEN38_ABLITERATION` | `Qwen3.8-Flash-Next-Uncensored` | Remove the refusal direction at run time. Empty disables it. See `patches/0102`. |
|
||||||
| `MAX_JOBS`, `ARCH` | `32`, `8.6` | Compile jobs and GPU architecture, for the build only. |
|
| `ABLITERATION_DIR` | `./abliteration` | Host folder with the direction file. Compose mounts it at `/opt/qwen38/abliteration`. |
|
||||||
|
| `GUARD` | `warn` | Bounds-guard telemetry: `off`, `warn`, or `raise`. |
|
||||||
|
| `COUNTERS` | `0` | `1` adds the logging-only kernel counters. |
|
||||||
|
| `MAX_JOBS`, `ARCH` | `32`, `8.6` | Compile jobs and GPU architecture, for the build only. |
|
||||||
|
|
||||||
## 10. Benchmarks
|
## 10. Benchmarks
|
||||||
|
|
||||||
|
|||||||
Reference in New Issue
Block a user