diff --git a/.env.example b/.env.example index 8ade668..a76abbf 100644 --- a/.env.example +++ b/.env.example @@ -33,6 +33,8 @@ TOPO_TIMEOUT=20 GUARD=warn # 1 adds the two logging-only counters (exact-restart counters /30 s, FP8 KV clips /300 s). COUNTERS=0 +# Seconds between engine stats lines (throughput, KV usage, queue depth). The vLLM default is 10 s. +VLLM_LOG_STATS_INTERVAL=1 # Extracted from orcarouter/Qwen3.8-Flash-Next-Uncensored, hidden 2560 QWEN38_ABLITERATION=Qwen3.8-Flash-Next-Uncensored diff --git a/README-ID.md b/README-ID.md index 978bc64..67f50e1 100644 --- a/README-ID.md +++ b/README-ID.md @@ -277,23 +277,24 @@ Tanpa Compose, mount sendiri sidecar-nya: `-v /scales.json:/opt/vllm/qsa_k ## 9. Variabel -| Variabel | Bawaan | Arti | -| ---------------------- | -------------------------------------------------------------- | -------------------------------------------------------------------------------------- | -| `MODEL_DIR` | tidak ada, wajib | Direktori host tempat checkpoint. Compose berhenti dengan pesan ketika kosong. | -| `CKPT` | `/data/models/halt95/Swift1.5-Qwen3.8-Flash-Next-W4A16-Merlin` | Folder checkpoint di host. Argumen pertama `start.sh` mengaturnya. | -| `MODEL_MOUNT_PATH` | `/models/Qwen3.8-Flash-Next` | Path checkpoint di dalam container. Compose mem-mount `MODEL_DIR` ke sini. | -| `SCALES` | `$MODEL_DIR/qsa_kv_scales.json` | Path host sidecar scale KV FP8. Compose mem-mount-nya ke container. | -| `MODEL_NAME` | `Hinata` (compose), `qwen38-flash-next` (`start.sh`) | Nama di field `model` pada request. Spasi membuat alias. | -| `HOST_PORT` | `8080` | Port host untuk API. | -| `VLLM_API_KEY` | kosong | Kosong berarti tanpa autentikasi. | -| `CACHE_HOST_DIR` | `.vllm-cache` di direktori ini | Tempat cache compile berada. Jalur host memakai folder yang sama. | -| `CUDA_VISIBLE_DEVICES` | `0,1,2,3` | Jalur GPU, dalam urutan bus. | -| `PLE_HOME` | `0` | Kartu yang menjadi inang PLE offload. Kartu itu tidak mendapat CUDA context untuk itu. | -| `QWEN38_ABLITERATION` | `Qwen3.8-Flash-Next-Uncensored` | Hapus arah refusal saat jalan. Kosong menonaktifkannya. Lihat `patches/0102`. | -| `ABLITERATION_DIR` | `./abliteration` | Folder host dengan file arah itu. Compose mem-mount ke `/opt/qwen38/abliteration`. | -| `GUARD` | `warn` | Telemetri bounds-guard: `off`, `warn`, atau `raise`. | -| `COUNTERS` | `0` | `1` menambah counter kernel khusus log. | -| `MAX_JOBS`, `ARCH` | `32`, `8.6` | Job compile dan arsitektur GPU, hanya untuk build. | +| Variabel | Bawaan | Arti | +| ------------------------- | -------------------------------------------------------------- | -------------------------------------------------------------------------------------- | +| `MODEL_DIR` | tidak ada, wajib | Direktori host tempat checkpoint. Compose berhenti dengan pesan ketika kosong. | +| `CKPT` | `/data/models/halt95/Swift1.5-Qwen3.8-Flash-Next-W4A16-Merlin` | Folder checkpoint di host. Argumen pertama `start.sh` mengaturnya. | +| `MODEL_MOUNT_PATH` | `/models/Qwen3.8-Flash-Next` | Path checkpoint di dalam container. Compose mem-mount `MODEL_DIR` ke sini. | +| `SCALES` | `$MODEL_DIR/qsa_kv_scales.json` | Path host sidecar scale KV FP8. Compose mem-mount-nya ke container. | +| `MODEL_NAME` | `Hinata` (compose), `qwen38-flash-next` (`start.sh`) | Nama di field `model` pada request. Spasi membuat alias. | +| `HOST_PORT` | `8080` | Port host untuk API. | +| `VLLM_API_KEY` | kosong | Kosong berarti tanpa autentikasi. | +| `CACHE_HOST_DIR` | `.vllm-cache` di direktori ini | Tempat cache compile berada. Jalur host memakai folder yang sama. | +| `CUDA_VISIBLE_DEVICES` | `0,1,2,3` | Jalur GPU, dalam urutan bus. | +| `PLE_HOME` | `0` | Kartu yang menjadi inang PLE offload. Kartu itu tidak mendapat CUDA context untuk itu. | +| `QWEN38_ABLITERATION` | `Qwen3.8-Flash-Next-Uncensored` | Hapus arah refusal saat jalan. Kosong menonaktifkannya. Lihat `patches/0102`. | +| `ABLITERATION_DIR` | `./abliteration` | Folder host dengan file arah itu. Compose mem-mount ke `/opt/qwen38/abliteration`. | +| `GUARD` | `warn` | Telemetri bounds-guard: `off`, `warn`, atau `raise`. | +| `COUNTERS` | `0` | `1` menambah counter kernel khusus log. | +| `VLLM_LOG_STATS_INTERVAL` | `1` | Jeda antar baris stats engine, dalam detik. Default vLLM: 10 s. | +| `MAX_JOBS`, `ARCH` | `32`, `8.6` | Job compile dan arsitektur GPU, hanya untuk build. | ## 10. Benchmark diff --git a/README.md b/README.md index 22ea9f2..4d40494 100644 --- a/README.md +++ b/README.md @@ -266,23 +266,24 @@ Without Compose, mount the sidecar yourself: `-v /scales.json:/opt/vllm/qs ## 9. Variables -| Variable | Default | Meaning | -| ---------------------- | -------------------------------------------------------------- | ------------------------------------------------------------------------------------- | -| `MODEL_DIR` | none, required | Host directory of the checkpoint. Compose stops with a message when it is empty. | -| `CKPT` | `/data/models/halt95/Swift1.5-Qwen3.8-Flash-Next-W4A16-Merlin` | Checkpoint folder on the host. The first argument to `start.sh` sets it. | -| `MODEL_MOUNT_PATH` | `/models/Qwen3.8-Flash-Next` | Path of the checkpoint inside the container. Compose mounts `MODEL_DIR` here. | -| `SCALES` | `$MODEL_DIR/qsa_kv_scales.json` | Host path of the FP8 KV scale sidecar. Compose mounts it into the container. | -| `MODEL_NAME` | `Hinata` (compose), `qwen38-flash-next` (`start.sh`) | The name in the `model` field of a request. Spaces make aliases. | -| `HOST_PORT` | `8080` | The host port of the API. | -| `VLLM_API_KEY` | empty | Empty means no authentication. | -| `CACHE_HOST_DIR` | `.vllm-cache` in this directory | Where the compile caches live. The host path uses the same folder. | -| `CUDA_VISIBLE_DEVICES` | `0,1,2,3` | The GPU lane, in bus order. | -| `PLE_HOME` | `0` | The card that hosts the PLE offload. That card gets no CUDA context for it. | -| `QWEN38_ABLITERATION` | `Qwen3.8-Flash-Next-Uncensored` | Remove the refusal direction at run time. Empty disables it. See `patches/0102`. | -| `ABLITERATION_DIR` | `./abliteration` | Host folder with the direction file. Compose mounts it at `/opt/qwen38/abliteration`. | -| `GUARD` | `warn` | Bounds-guard telemetry: `off`, `warn`, or `raise`. | -| `COUNTERS` | `0` | `1` adds the logging-only kernel counters. | -| `MAX_JOBS`, `ARCH` | `32`, `8.6` | Compile jobs and GPU architecture, for the build only. | +| Variable | Default | Meaning | +| ------------------------- | -------------------------------------------------------------- | ------------------------------------------------------------------------------------- | +| `MODEL_DIR` | none, required | Host directory of the checkpoint. Compose stops with a message when it is empty. | +| `CKPT` | `/data/models/halt95/Swift1.5-Qwen3.8-Flash-Next-W4A16-Merlin` | Checkpoint folder on the host. The first argument to `start.sh` sets it. | +| `MODEL_MOUNT_PATH` | `/models/Qwen3.8-Flash-Next` | Path of the checkpoint inside the container. Compose mounts `MODEL_DIR` here. | +| `SCALES` | `$MODEL_DIR/qsa_kv_scales.json` | Host path of the FP8 KV scale sidecar. Compose mounts it into the container. | +| `MODEL_NAME` | `Hinata` (compose), `qwen38-flash-next` (`start.sh`) | The name in the `model` field of a request. Spaces make aliases. | +| `HOST_PORT` | `8080` | The host port of the API. | +| `VLLM_API_KEY` | empty | Empty means no authentication. | +| `CACHE_HOST_DIR` | `.vllm-cache` in this directory | Where the compile caches live. The host path uses the same folder. | +| `CUDA_VISIBLE_DEVICES` | `0,1,2,3` | The GPU lane, in bus order. | +| `PLE_HOME` | `0` | The card that hosts the PLE offload. That card gets no CUDA context for it. | +| `QWEN38_ABLITERATION` | `Qwen3.8-Flash-Next-Uncensored` | Remove the refusal direction at run time. Empty disables it. See `patches/0102`. | +| `ABLITERATION_DIR` | `./abliteration` | Host folder with the direction file. Compose mounts it at `/opt/qwen38/abliteration`. | +| `GUARD` | `warn` | Bounds-guard telemetry: `off`, `warn`, or `raise`. | +| `COUNTERS` | `0` | `1` adds the logging-only kernel counters. | +| `VLLM_LOG_STATS_INTERVAL` | `1` | Seconds between engine stats lines. The vLLM default is 10 s. | +| `MAX_JOBS`, `ARCH` | `32`, `8.6` | Compile jobs and GPU architecture, for the build only. | ## 10. Benchmarks diff --git a/docker-compose.yml b/docker-compose.yml index 2a18dd2..d754687 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -40,6 +40,7 @@ services: TOPO_TIMEOUT: ${TOPO_TIMEOUT:-20} GUARD: ${GUARD:-warn} COUNTERS: ${COUNTERS:-0} + VLLM_LOG_STATS_INTERVAL: ${VLLM_LOG_STATS_INTERVAL:-1} QWEN38_ABLITERATION: ${QWEN38_ABLITERATION:-Qwen3.8-Flash-Next-Uncensored} volumes: - "${MODEL_DIR:?Set MODEL_DIR in .env}:${MODEL_MOUNT_PATH:-/models/Qwen3.8-Flash-Next}:ro" diff --git a/start.sh b/start.sh index 1955ee6..69565fc 100755 --- a/start.sh +++ b/start.sh @@ -22,6 +22,7 @@ # PLE_HOME GPU that hosts the PLE offload home (0) # GUARD bounds-guard telemetry off|warn|raise (warn) # COUNTERS=1 logging-only counters (top-k exact restarts /30s, FP8 KV clips /300s) +# VLLM_LOG_STATS_INTERVAL seconds between engine stats lines (1; the vLLM default is 10 s) # CUDA_HOME toolkit for first-serve JIT; default: the venv's CUDA 13.0 wheels # AUTO_TOPO 0 (default, fixed shipped settings) | 1 | strict -> scripts/topo_select.py # AUTO_TOPO_NCCL=default AUTO_TOPO_BIND=1 LANE_GPUS=auto LANE_CPUS=... TOPO_TIMEOUT=20 @@ -202,7 +203,9 @@ export VLLM_PLE_OFFLOAD_HOME_DEVICE="${PLE_HOME:-0}" export VLLM_HOST_EMBED_TABLE=1 export VLLM_USE_BREAKABLE_CUDAGRAPH=0 export VLLM_QSA_KV_SCALES="${SCALES:-$CKPT/qsa_kv_scales.json}" -# engine stats line (throughput, KV usage, queue). Set empty to keep vLLM's default. +# Engine stats line (throughput, KV usage, queue). 1 s matches how often the numbers are useful. +# Unset, vLLM waits 10 s (vllm/envs.py:47), so a quiet run looks idle for ten seconds. +export VLLM_LOG_STATS_INTERVAL="${VLLM_LOG_STATS_INTERVAL:-1}" export HF_HUB_OFFLINE=1 HF_DATASETS_OFFLINE=1 # the kernel toggles (the shipped values). These names are the tree's own: envs.py in the # serving tree registers MERLIN_* for the kernel toggles and VLLM_E2_* for the runtime knobs.