From 3a0a5d15104623ad974159e0c1e44ce7eefe1be0 Mon Sep 17 00:00:00 2001 From: MOVZX Date: Fri, 16 Oct 2026 06:01:27 +0000 Subject: [PATCH] feat: Set the engine stats interval to 1 s The stats line carried throughput, KV usage and queue depth, but the interval was never set. Unset, vLLM waits 10 s (vllm/envs.py:47), so a quiet run looked idle for ten seconds and a short one printed nothing. Export VLLM_LOG_STATS_INTERVAL in start.sh with a default of 1, and pass it through Compose so .env can raise it back to 10. Add the variable to .env.example, to the start.sh header, and to the Variables table in both READMEs. Co-Authored-By: Hinata Signed-off-by: MOVZX --- .env.example | 2 ++ README-ID.md | 35 ++++++++++++++++++----------------- README.md | 35 ++++++++++++++++++----------------- docker-compose.yml | 1 + start.sh | 5 ++++- 5 files changed, 43 insertions(+), 35 deletions(-) diff --git a/.env.example b/.env.example index 8ade668..a76abbf 100644 --- a/.env.example +++ b/.env.example @@ -33,6 +33,8 @@ TOPO_TIMEOUT=20 GUARD=warn # 1 adds the two logging-only counters (exact-restart counters /30 s, FP8 KV clips /300 s). COUNTERS=0 +# Seconds between engine stats lines (throughput, KV usage, queue depth). The vLLM default is 10 s. +VLLM_LOG_STATS_INTERVAL=1 # Extracted from orcarouter/Qwen3.8-Flash-Next-Uncensored, hidden 2560 QWEN38_ABLITERATION=Qwen3.8-Flash-Next-Uncensored diff --git a/README-ID.md b/README-ID.md index 978bc64..67f50e1 100644 --- a/README-ID.md +++ b/README-ID.md @@ -277,23 +277,24 @@ Tanpa Compose, mount sendiri sidecar-nya: `-v /scales.json:/opt/vllm/qsa_k ## 9. Variabel -| Variabel | Bawaan | Arti | -| ---------------------- | -------------------------------------------------------------- | -------------------------------------------------------------------------------------- | -| `MODEL_DIR` | tidak ada, wajib | Direktori host tempat checkpoint. Compose berhenti dengan pesan ketika kosong. | -| `CKPT` | `/data/models/halt95/Swift1.5-Qwen3.8-Flash-Next-W4A16-Merlin` | Folder checkpoint di host. Argumen pertama `start.sh` mengaturnya. | -| `MODEL_MOUNT_PATH` | `/models/Qwen3.8-Flash-Next` | Path checkpoint di dalam container. Compose mem-mount `MODEL_DIR` ke sini. | -| `SCALES` | `$MODEL_DIR/qsa_kv_scales.json` | Path host sidecar scale KV FP8. Compose mem-mount-nya ke container. | -| `MODEL_NAME` | `Hinata` (compose), `qwen38-flash-next` (`start.sh`) | Nama di field `model` pada request. Spasi membuat alias. | -| `HOST_PORT` | `8080` | Port host untuk API. | -| `VLLM_API_KEY` | kosong | Kosong berarti tanpa autentikasi. | -| `CACHE_HOST_DIR` | `.vllm-cache` di direktori ini | Tempat cache compile berada. Jalur host memakai folder yang sama. | -| `CUDA_VISIBLE_DEVICES` | `0,1,2,3` | Jalur GPU, dalam urutan bus. | -| `PLE_HOME` | `0` | Kartu yang menjadi inang PLE offload. Kartu itu tidak mendapat CUDA context untuk itu. | -| `QWEN38_ABLITERATION` | `Qwen3.8-Flash-Next-Uncensored` | Hapus arah refusal saat jalan. Kosong menonaktifkannya. Lihat `patches/0102`. | -| `ABLITERATION_DIR` | `./abliteration` | Folder host dengan file arah itu. Compose mem-mount ke `/opt/qwen38/abliteration`. | -| `GUARD` | `warn` | Telemetri bounds-guard: `off`, `warn`, atau `raise`. | -| `COUNTERS` | `0` | `1` menambah counter kernel khusus log. | -| `MAX_JOBS`, `ARCH` | `32`, `8.6` | Job compile dan arsitektur GPU, hanya untuk build. | +| Variabel | Bawaan | Arti | +| ------------------------- | -------------------------------------------------------------- | -------------------------------------------------------------------------------------- | +| `MODEL_DIR` | tidak ada, wajib | Direktori host tempat checkpoint. Compose berhenti dengan pesan ketika kosong. | +| `CKPT` | `/data/models/halt95/Swift1.5-Qwen3.8-Flash-Next-W4A16-Merlin` | Folder checkpoint di host. Argumen pertama `start.sh` mengaturnya. | +| `MODEL_MOUNT_PATH` | `/models/Qwen3.8-Flash-Next` | Path checkpoint di dalam container. Compose mem-mount `MODEL_DIR` ke sini. | +| `SCALES` | `$MODEL_DIR/qsa_kv_scales.json` | Path host sidecar scale KV FP8. Compose mem-mount-nya ke container. | +| `MODEL_NAME` | `Hinata` (compose), `qwen38-flash-next` (`start.sh`) | Nama di field `model` pada request. Spasi membuat alias. | +| `HOST_PORT` | `8080` | Port host untuk API. | +| `VLLM_API_KEY` | kosong | Kosong berarti tanpa autentikasi. | +| `CACHE_HOST_DIR` | `.vllm-cache` di direktori ini | Tempat cache compile berada. Jalur host memakai folder yang sama. | +| `CUDA_VISIBLE_DEVICES` | `0,1,2,3` | Jalur GPU, dalam urutan bus. | +| `PLE_HOME` | `0` | Kartu yang menjadi inang PLE offload. Kartu itu tidak mendapat CUDA context untuk itu. | +| `QWEN38_ABLITERATION` | `Qwen3.8-Flash-Next-Uncensored` | Hapus arah refusal saat jalan. Kosong menonaktifkannya. Lihat `patches/0102`. | +| `ABLITERATION_DIR` | `./abliteration` | Folder host dengan file arah itu. Compose mem-mount ke `/opt/qwen38/abliteration`. | +| `GUARD` | `warn` | Telemetri bounds-guard: `off`, `warn`, atau `raise`. | +| `COUNTERS` | `0` | `1` menambah counter kernel khusus log. | +| `VLLM_LOG_STATS_INTERVAL` | `1` | Jeda antar baris stats engine, dalam detik. Default vLLM: 10 s. | +| `MAX_JOBS`, `ARCH` | `32`, `8.6` | Job compile dan arsitektur GPU, hanya untuk build. | ## 10. Benchmark diff --git a/README.md b/README.md index 22ea9f2..4d40494 100644 --- a/README.md +++ b/README.md @@ -266,23 +266,24 @@ Without Compose, mount the sidecar yourself: `-v /scales.json:/opt/vllm/qs ## 9. Variables -| Variable | Default | Meaning | -| ---------------------- | -------------------------------------------------------------- | ------------------------------------------------------------------------------------- | -| `MODEL_DIR` | none, required | Host directory of the checkpoint. Compose stops with a message when it is empty. | -| `CKPT` | `/data/models/halt95/Swift1.5-Qwen3.8-Flash-Next-W4A16-Merlin` | Checkpoint folder on the host. The first argument to `start.sh` sets it. | -| `MODEL_MOUNT_PATH` | `/models/Qwen3.8-Flash-Next` | Path of the checkpoint inside the container. Compose mounts `MODEL_DIR` here. | -| `SCALES` | `$MODEL_DIR/qsa_kv_scales.json` | Host path of the FP8 KV scale sidecar. Compose mounts it into the container. | -| `MODEL_NAME` | `Hinata` (compose), `qwen38-flash-next` (`start.sh`) | The name in the `model` field of a request. Spaces make aliases. | -| `HOST_PORT` | `8080` | The host port of the API. | -| `VLLM_API_KEY` | empty | Empty means no authentication. | -| `CACHE_HOST_DIR` | `.vllm-cache` in this directory | Where the compile caches live. The host path uses the same folder. | -| `CUDA_VISIBLE_DEVICES` | `0,1,2,3` | The GPU lane, in bus order. | -| `PLE_HOME` | `0` | The card that hosts the PLE offload. That card gets no CUDA context for it. | -| `QWEN38_ABLITERATION` | `Qwen3.8-Flash-Next-Uncensored` | Remove the refusal direction at run time. Empty disables it. See `patches/0102`. | -| `ABLITERATION_DIR` | `./abliteration` | Host folder with the direction file. Compose mounts it at `/opt/qwen38/abliteration`. | -| `GUARD` | `warn` | Bounds-guard telemetry: `off`, `warn`, or `raise`. | -| `COUNTERS` | `0` | `1` adds the logging-only kernel counters. | -| `MAX_JOBS`, `ARCH` | `32`, `8.6` | Compile jobs and GPU architecture, for the build only. | +| Variable | Default | Meaning | +| ------------------------- | -------------------------------------------------------------- | ------------------------------------------------------------------------------------- | +| `MODEL_DIR` | none, required | Host directory of the checkpoint. Compose stops with a message when it is empty. | +| `CKPT` | `/data/models/halt95/Swift1.5-Qwen3.8-Flash-Next-W4A16-Merlin` | Checkpoint folder on the host. The first argument to `start.sh` sets it. | +| `MODEL_MOUNT_PATH` | `/models/Qwen3.8-Flash-Next` | Path of the checkpoint inside the container. Compose mounts `MODEL_DIR` here. | +| `SCALES` | `$MODEL_DIR/qsa_kv_scales.json` | Host path of the FP8 KV scale sidecar. Compose mounts it into the container. | +| `MODEL_NAME` | `Hinata` (compose), `qwen38-flash-next` (`start.sh`) | The name in the `model` field of a request. Spaces make aliases. | +| `HOST_PORT` | `8080` | The host port of the API. | +| `VLLM_API_KEY` | empty | Empty means no authentication. | +| `CACHE_HOST_DIR` | `.vllm-cache` in this directory | Where the compile caches live. The host path uses the same folder. | +| `CUDA_VISIBLE_DEVICES` | `0,1,2,3` | The GPU lane, in bus order. | +| `PLE_HOME` | `0` | The card that hosts the PLE offload. That card gets no CUDA context for it. | +| `QWEN38_ABLITERATION` | `Qwen3.8-Flash-Next-Uncensored` | Remove the refusal direction at run time. Empty disables it. See `patches/0102`. | +| `ABLITERATION_DIR` | `./abliteration` | Host folder with the direction file. Compose mounts it at `/opt/qwen38/abliteration`. | +| `GUARD` | `warn` | Bounds-guard telemetry: `off`, `warn`, or `raise`. | +| `COUNTERS` | `0` | `1` adds the logging-only kernel counters. | +| `VLLM_LOG_STATS_INTERVAL` | `1` | Seconds between engine stats lines. The vLLM default is 10 s. | +| `MAX_JOBS`, `ARCH` | `32`, `8.6` | Compile jobs and GPU architecture, for the build only. | ## 10. Benchmarks diff --git a/docker-compose.yml b/docker-compose.yml index 2a18dd2..d754687 100644 --- a/docker-compose.yml +++ b/docker-compose.yml @@ -40,6 +40,7 @@ services: TOPO_TIMEOUT: ${TOPO_TIMEOUT:-20} GUARD: ${GUARD:-warn} COUNTERS: ${COUNTERS:-0} + VLLM_LOG_STATS_INTERVAL: ${VLLM_LOG_STATS_INTERVAL:-1} QWEN38_ABLITERATION: ${QWEN38_ABLITERATION:-Qwen3.8-Flash-Next-Uncensored} volumes: - "${MODEL_DIR:?Set MODEL_DIR in .env}:${MODEL_MOUNT_PATH:-/models/Qwen3.8-Flash-Next}:ro" diff --git a/start.sh b/start.sh index 1955ee6..69565fc 100755 --- a/start.sh +++ b/start.sh @@ -22,6 +22,7 @@ # PLE_HOME GPU that hosts the PLE offload home (0) # GUARD bounds-guard telemetry off|warn|raise (warn) # COUNTERS=1 logging-only counters (top-k exact restarts /30s, FP8 KV clips /300s) +# VLLM_LOG_STATS_INTERVAL seconds between engine stats lines (1; the vLLM default is 10 s) # CUDA_HOME toolkit for first-serve JIT; default: the venv's CUDA 13.0 wheels # AUTO_TOPO 0 (default, fixed shipped settings) | 1 | strict -> scripts/topo_select.py # AUTO_TOPO_NCCL=default AUTO_TOPO_BIND=1 LANE_GPUS=auto LANE_CPUS=... TOPO_TIMEOUT=20 @@ -202,7 +203,9 @@ export VLLM_PLE_OFFLOAD_HOME_DEVICE="${PLE_HOME:-0}" export VLLM_HOST_EMBED_TABLE=1 export VLLM_USE_BREAKABLE_CUDAGRAPH=0 export VLLM_QSA_KV_SCALES="${SCALES:-$CKPT/qsa_kv_scales.json}" -# engine stats line (throughput, KV usage, queue). Set empty to keep vLLM's default. +# Engine stats line (throughput, KV usage, queue). 1 s matches how often the numbers are useful. +# Unset, vLLM waits 10 s (vllm/envs.py:47), so a quiet run looks idle for ten seconds. +export VLLM_LOG_STATS_INTERVAL="${VLLM_LOG_STATS_INTERVAL:-1}" export HF_HUB_OFFLINE=1 HF_DATASETS_OFFLINE=1 # the kernel toggles (the shipped values). These names are the tree's own: envs.py in the # serving tree registers MERLIN_* for the kernel toggles and VLLM_E2_* for the runtime knobs.