diff --git a/README-ID.md b/README-ID.md index 6809fe3..978bc64 100644 --- a/README-ID.md +++ b/README-ID.md @@ -20,9 +20,12 @@ satu aliran diukur pada 158-184 tok/s. | Python | 3.13 | | Build tools | `cmake` 3.26 atau lebih baru, `ninja`, `cc`, `c++` | -Checkpoint harus berisi `qsa_kv_scales.json`. File itu menyimpan scale KV -FP8. Swift 1.5 mengubah `k_proj`/`v_proj`, jadi file scale satu checkpoint tidak -bekerja untuk checkpoint lain. +Checkpoint harus berisi `qsa_kv_scales.json`. File itu menyimpan scale KV FP8. +Swift 1.5 mengubah `k_proj`/`v_proj`, jadi file scale satu checkpoint tidak bekerja +untuk checkpoint lain. + +`SCALES` menunjuk ke sidecar lain. Beri path host; Compose mem-mount-nya ke container. +Bawaannya `$MODEL_DIR/qsa_kv_scales.json`. ## 2. Cara Build @@ -255,6 +258,8 @@ docker run -d --name vllm --gpus all --stop-timeout 20 --shm-size=16g \ -e MODEL_NAME=Hinata vllm:latest ``` +Tanpa Compose, mount sendiri sidecar-nya: `-v /scales.json:/opt/vllm/qsa_kv_scales.json:ro -e SCALES=/opt/vllm/qsa_kv_scales.json`. + ## 8. File | File | Guna | @@ -272,20 +277,23 @@ docker run -d --name vllm --gpus all --stop-timeout 20 --shm-size=16g \ ## 9. Variabel -| Variabel | Bawaan | Arti | -| ---------------------- | ---------------------------------------------------- | -------------------------------------------------------------------------------------- | -| `MODEL_DIR` | tidak ada, wajib | Direktori host tempat checkpoint. Compose berhenti dengan pesan ketika kosong. | -| `MODEL_NAME` | `Hinata` (compose), `qwen38-flash-next` (`start.sh`) | Nama di field `model` pada request. Spasi membuat alias. | -| `HOST_PORT` | `8080` | Port host untuk API. | -| `VLLM_API_KEY` | kosong | Kosong berarti tanpa autentikasi. | -| `CACHE_HOST_DIR` | `.vllm-cache` di direktori ini | Tempat cache compile berada. Jalur host memakai folder yang sama. | -| `CUDA_VISIBLE_DEVICES` | `0,1,2,3` | Jalur GPU, dalam urutan bus. | -| `PLE_HOME` | `0` | Kartu yang menjadi inang PLE offload. Kartu itu tidak mendapat CUDA context untuk itu. | -| `QWEN38_ABLITERATION` | `Qwen3.8-Flash-Next-Uncensored` | Hapus arah refusal saat jalan. Kosong menonaktifkannya. Lihat `patches/0102`. | -| `ABLITERATION_DIR` | `./abliteration` | Folder host dengan file arah itu. Compose mem-mount ke `/opt/qwen38/abliteration`. | -| `GUARD` | `warn` | Telemetri bounds-guard: `off`, `warn`, atau `raise`. | -| `COUNTERS` | `0` | `1` menambah counter kernel khusus log. | -| `MAX_JOBS`, `ARCH` | `32`, `8.6` | Job compile dan arsitektur GPU, hanya untuk build. | +| Variabel | Bawaan | Arti | +| ---------------------- | -------------------------------------------------------------- | -------------------------------------------------------------------------------------- | +| `MODEL_DIR` | tidak ada, wajib | Direktori host tempat checkpoint. Compose berhenti dengan pesan ketika kosong. | +| `CKPT` | `/data/models/halt95/Swift1.5-Qwen3.8-Flash-Next-W4A16-Merlin` | Folder checkpoint di host. Argumen pertama `start.sh` mengaturnya. | +| `MODEL_MOUNT_PATH` | `/models/Qwen3.8-Flash-Next` | Path checkpoint di dalam container. Compose mem-mount `MODEL_DIR` ke sini. | +| `SCALES` | `$MODEL_DIR/qsa_kv_scales.json` | Path host sidecar scale KV FP8. Compose mem-mount-nya ke container. | +| `MODEL_NAME` | `Hinata` (compose), `qwen38-flash-next` (`start.sh`) | Nama di field `model` pada request. Spasi membuat alias. | +| `HOST_PORT` | `8080` | Port host untuk API. | +| `VLLM_API_KEY` | kosong | Kosong berarti tanpa autentikasi. | +| `CACHE_HOST_DIR` | `.vllm-cache` di direktori ini | Tempat cache compile berada. Jalur host memakai folder yang sama. | +| `CUDA_VISIBLE_DEVICES` | `0,1,2,3` | Jalur GPU, dalam urutan bus. | +| `PLE_HOME` | `0` | Kartu yang menjadi inang PLE offload. Kartu itu tidak mendapat CUDA context untuk itu. | +| `QWEN38_ABLITERATION` | `Qwen3.8-Flash-Next-Uncensored` | Hapus arah refusal saat jalan. Kosong menonaktifkannya. Lihat `patches/0102`. | +| `ABLITERATION_DIR` | `./abliteration` | Folder host dengan file arah itu. Compose mem-mount ke `/opt/qwen38/abliteration`. | +| `GUARD` | `warn` | Telemetri bounds-guard: `off`, `warn`, atau `raise`. | +| `COUNTERS` | `0` | `1` menambah counter kernel khusus log. | +| `MAX_JOBS`, `ARCH` | `32`, `8.6` | Job compile dan arsitektur GPU, hanya untuk build. | ## 10. Benchmark diff --git a/README.md b/README.md index d271e33..22ea9f2 100644 --- a/README.md +++ b/README.md @@ -19,9 +19,12 @@ differ, this file wins. | Python | 3.13 | | Build tools | `cmake` 3.26 or newer, `ninja`, `cc`, `c++` | -The checkpoint must contain `qsa_kv_scales.json`. That file holds the FP8 KV -scales. Swift 1.5 changed `k_proj`/`v_proj`, so the scale file of one checkpoint does not -work for another. +The checkpoint must contain `qsa_kv_scales.json`. That file holds the FP8 KV scales. +Swift 1.5 changed `k_proj`/`v_proj`, so the scale file of one checkpoint does not work +for another. + +`SCALES` names another sidecar. Give a host path; Compose mounts it into the container. +The default is `$MODEL_DIR/qsa_kv_scales.json`. ## 2. How to Build @@ -243,6 +246,8 @@ docker run -d --name vllm --gpus all --stop-timeout 20 --shm-size=16g \ -e MODEL_NAME=Hinata vllm:latest ``` +Without Compose, mount the sidecar yourself: `-v /scales.json:/opt/vllm/qsa_kv_scales.json:ro -e SCALES=/opt/vllm/qsa_kv_scales.json`. + ## 8. Files | File | Use | @@ -261,20 +266,23 @@ docker run -d --name vllm --gpus all --stop-timeout 20 --shm-size=16g \ ## 9. Variables -| Variable | Default | Meaning | -| ---------------------- | ---------------------------------------------------- | ------------------------------------------------------------------------------------- | -| `MODEL_DIR` | none, required | Host directory of the checkpoint. Compose stops with a message when it is empty. | -| `MODEL_NAME` | `Hinata` (compose), `qwen38-flash-next` (`start.sh`) | The name in the `model` field of a request. Spaces make aliases. | -| `HOST_PORT` | `8080` | The host port of the API. | -| `VLLM_API_KEY` | empty | Empty means no authentication. | -| `CACHE_HOST_DIR` | `.vllm-cache` in this directory | Where the compile caches live. The host path uses the same folder. | -| `CUDA_VISIBLE_DEVICES` | `0,1,2,3` | The GPU lane, in bus order. | -| `PLE_HOME` | `0` | The card that hosts the PLE offload. That card gets no CUDA context for it. | -| `QWEN38_ABLITERATION` | `Qwen3.8-Flash-Next-Uncensored` | Remove the refusal direction at run time. Empty disables it. See `patches/0102`. | -| `ABLITERATION_DIR` | `./abliteration` | Host folder with the direction file. Compose mounts it at `/opt/qwen38/abliteration`. | -| `GUARD` | `warn` | Bounds-guard telemetry: `off`, `warn`, or `raise`. | -| `COUNTERS` | `0` | `1` adds the logging-only kernel counters. | -| `MAX_JOBS`, `ARCH` | `32`, `8.6` | Compile jobs and GPU architecture, for the build only. | +| Variable | Default | Meaning | +| ---------------------- | -------------------------------------------------------------- | ------------------------------------------------------------------------------------- | +| `MODEL_DIR` | none, required | Host directory of the checkpoint. Compose stops with a message when it is empty. | +| `CKPT` | `/data/models/halt95/Swift1.5-Qwen3.8-Flash-Next-W4A16-Merlin` | Checkpoint folder on the host. The first argument to `start.sh` sets it. | +| `MODEL_MOUNT_PATH` | `/models/Qwen3.8-Flash-Next` | Path of the checkpoint inside the container. Compose mounts `MODEL_DIR` here. | +| `SCALES` | `$MODEL_DIR/qsa_kv_scales.json` | Host path of the FP8 KV scale sidecar. Compose mounts it into the container. | +| `MODEL_NAME` | `Hinata` (compose), `qwen38-flash-next` (`start.sh`) | The name in the `model` field of a request. Spaces make aliases. | +| `HOST_PORT` | `8080` | The host port of the API. | +| `VLLM_API_KEY` | empty | Empty means no authentication. | +| `CACHE_HOST_DIR` | `.vllm-cache` in this directory | Where the compile caches live. The host path uses the same folder. | +| `CUDA_VISIBLE_DEVICES` | `0,1,2,3` | The GPU lane, in bus order. | +| `PLE_HOME` | `0` | The card that hosts the PLE offload. That card gets no CUDA context for it. | +| `QWEN38_ABLITERATION` | `Qwen3.8-Flash-Next-Uncensored` | Remove the refusal direction at run time. Empty disables it. See `patches/0102`. | +| `ABLITERATION_DIR` | `./abliteration` | Host folder with the direction file. Compose mounts it at `/opt/qwen38/abliteration`. | +| `GUARD` | `warn` | Bounds-guard telemetry: `off`, `warn`, or `raise`. | +| `COUNTERS` | `0` | `1` adds the logging-only kernel counters. | +| `MAX_JOBS`, `ARCH` | `32`, `8.6` | Compile jobs and GPU architecture, for the build only. | ## 10. Benchmarks