iHore/vLLM-4x-RTX-3090
8
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: Set the engine stats interval to 1 s

Browse Source 
The stats line carried throughput, KV usage and queue depth, but the
interval was never set. Unset, vLLM waits 10 s (vllm/envs.py:47), so a
quiet run looked idle for ten seconds and a short one printed nothing.

Export VLLM_LOG_STATS_INTERVAL in start.sh with a default of 1, and pass
it through Compose so .env can raise it back to 10. Add the variable to
.env.example, to the start.sh header, and to the Variables table in both
READMEs.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
This commit is contained in:
MOVZX
2026-10-16 06:01:27 +00:00
co-authored by Hinata
parent c4d88acb4d
commit 3a0a5d1510
5 changed files with 43 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.env.example
+2
View File
@@ -33,6 +33,8 @@ TOPO_TIMEOUT=20
GUARD=warn
# 1 adds the two logging-only counters (exact-restart counters /30 s, FP8 KV clips /300 s).
COUNTERS=0
# Seconds between engine stats lines (throughput, KV usage, queue depth). The vLLM default is 10 s.
VLLM_LOG_STATS_INTERVAL=1
# Extracted from orcarouter/Qwen3.8-Flash-Next-Uncensored, hidden 2560
QWEN38_ABLITERATION=Qwen3.8-Flash-Next-Uncensored
README-ID.md
+18 -17
View File
@@ -277,23 +277,24 @@ Tanpa Compose, mount sendiri sidecar-nya: `-v <host>/scales.json:/opt/vllm/qsa_k
## 9. Variabel
| Variabel | Bawaan | Arti |
| ---------------------- | -------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
| `MODEL_DIR` | tidak ada, wajib | Direktori host tempat checkpoint. Compose berhenti dengan pesan ketika kosong. |
| `CKPT` | `/data/models/halt95/Swift1.5-Qwen3.8-Flash-Next-W4A16-Merlin` | Folder checkpoint di host. Argumen pertama `start.sh` mengaturnya. |
| `MODEL_MOUNT_PATH` | `/models/Qwen3.8-Flash-Next` | Path checkpoint di dalam container. Compose mem-mount `MODEL_DIR` ke sini. |
| `SCALES` | `$MODEL_DIR/qsa_kv_scales.json` | Path host sidecar scale KV FP8. Compose mem-mount-nya ke container. |
| `MODEL_NAME` | `Hinata` (compose), `qwen38-flash-next` (`start.sh`) | Nama di field `model` pada request. Spasi membuat alias. |
| `HOST_PORT` | `8080` | Port host untuk API. |
| `VLLM_API_KEY` | kosong | Kosong berarti tanpa autentikasi. |
| `CACHE_HOST_DIR` | `.vllm-cache` di direktori ini | Tempat cache compile berada. Jalur host memakai folder yang sama. |
| `CUDA_VISIBLE_DEVICES` | `0,1,2,3` | Jalur GPU, dalam urutan bus. |
| `PLE_HOME` | `0` | Kartu yang menjadi inang PLE offload. Kartu itu tidak mendapat CUDA context untuk itu. |
| `QWEN38_ABLITERATION` | `Qwen3.8-Flash-Next-Uncensored` | Hapus arah refusal saat jalan. Kosong menonaktifkannya. Lihat `patches/0102`. |
| `ABLITERATION_DIR` | `./abliteration` | Folder host dengan file arah itu. Compose mem-mount ke `/opt/qwen38/abliteration`. |
| `GUARD` | `warn` | Telemetri bounds-guard: `off`, `warn`, atau `raise`. |
| `COUNTERS` | `0` | `1` menambah counter kernel khusus log. |
| `MAX_JOBS`, `ARCH` | `32`, `8.6` | Job compile dan arsitektur GPU, hanya untuk build. |
| Variabel | Bawaan | Arti |
| ------------------------- | -------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
| `MODEL_DIR` | tidak ada, wajib | Direktori host tempat checkpoint. Compose berhenti dengan pesan ketika kosong. |
| `CKPT` | `/data/models/halt95/Swift1.5-Qwen3.8-Flash-Next-W4A16-Merlin` | Folder checkpoint di host. Argumen pertama `start.sh` mengaturnya. |
| `MODEL_MOUNT_PATH` | `/models/Qwen3.8-Flash-Next` | Path checkpoint di dalam container. Compose mem-mount `MODEL_DIR` ke sini. |
| `SCALES` | `$MODEL_DIR/qsa_kv_scales.json` | Path host sidecar scale KV FP8. Compose mem-mount-nya ke container. |
| `MODEL_NAME` | `Hinata` (compose), `qwen38-flash-next` (`start.sh`) | Nama di field `model` pada request. Spasi membuat alias. |
| `HOST_PORT` | `8080` | Port host untuk API. |
| `VLLM_API_KEY` | kosong | Kosong berarti tanpa autentikasi. |
| `CACHE_HOST_DIR` | `.vllm-cache` di direktori ini | Tempat cache compile berada. Jalur host memakai folder yang sama. |
| `CUDA_VISIBLE_DEVICES` | `0,1,2,3` | Jalur GPU, dalam urutan bus. |
| `PLE_HOME` | `0` | Kartu yang menjadi inang PLE offload. Kartu itu tidak mendapat CUDA context untuk itu. |
| `QWEN38_ABLITERATION` | `Qwen3.8-Flash-Next-Uncensored` | Hapus arah refusal saat jalan. Kosong menonaktifkannya. Lihat `patches/0102`. |
| `ABLITERATION_DIR` | `./abliteration` | Folder host dengan file arah itu. Compose mem-mount ke `/opt/qwen38/abliteration`. |
| `GUARD` | `warn` | Telemetri bounds-guard: `off`, `warn`, atau `raise`. |
| `COUNTERS` | `0` | `1` menambah counter kernel khusus log. |
| `VLLM_LOG_STATS_INTERVAL` | `1` | Jeda antar baris stats engine, dalam detik. Default vLLM: 10 s. |
| `MAX_JOBS`, `ARCH` | `32`, `8.6` | Job compile dan arsitektur GPU, hanya untuk build. |
## 10. Benchmark
README.md
+18 -17
View File
@@ -266,23 +266,24 @@ Without Compose, mount the sidecar yourself: `-v <host>/scales.json:/opt/vllm/qs
## 9. Variables
| Variable | Default | Meaning |
| ---------------------- | -------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
| `MODEL_DIR` | none, required | Host directory of the checkpoint. Compose stops with a message when it is empty. |
| `CKPT` | `/data/models/halt95/Swift1.5-Qwen3.8-Flash-Next-W4A16-Merlin` | Checkpoint folder on the host. The first argument to `start.sh` sets it. |
| `MODEL_MOUNT_PATH` | `/models/Qwen3.8-Flash-Next` | Path of the checkpoint inside the container. Compose mounts `MODEL_DIR` here. |
| `SCALES` | `$MODEL_DIR/qsa_kv_scales.json` | Host path of the FP8 KV scale sidecar. Compose mounts it into the container. |
| `MODEL_NAME` | `Hinata` (compose), `qwen38-flash-next` (`start.sh`) | The name in the `model` field of a request. Spaces make aliases. |
| `HOST_PORT` | `8080` | The host port of the API. |
| `VLLM_API_KEY` | empty | Empty means no authentication. |
| `CACHE_HOST_DIR` | `.vllm-cache` in this directory | Where the compile caches live. The host path uses the same folder. |
| `CUDA_VISIBLE_DEVICES` | `0,1,2,3` | The GPU lane, in bus order. |
| `PLE_HOME` | `0` | The card that hosts the PLE offload. That card gets no CUDA context for it. |
| `QWEN38_ABLITERATION` | `Qwen3.8-Flash-Next-Uncensored` | Remove the refusal direction at run time. Empty disables it. See `patches/0102`. |
| `ABLITERATION_DIR` | `./abliteration` | Host folder with the direction file. Compose mounts it at `/opt/qwen38/abliteration`. |
| `GUARD` | `warn` | Bounds-guard telemetry: `off`, `warn`, or `raise`. |
| `COUNTERS` | `0` | `1` adds the logging-only kernel counters. |
| `MAX_JOBS`, `ARCH` | `32`, `8.6` | Compile jobs and GPU architecture, for the build only. |
| Variable | Default | Meaning |
| ------------------------- | -------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
| `MODEL_DIR` | none, required | Host directory of the checkpoint. Compose stops with a message when it is empty. |
| `CKPT` | `/data/models/halt95/Swift1.5-Qwen3.8-Flash-Next-W4A16-Merlin` | Checkpoint folder on the host. The first argument to `start.sh` sets it. |
| `MODEL_MOUNT_PATH` | `/models/Qwen3.8-Flash-Next` | Path of the checkpoint inside the container. Compose mounts `MODEL_DIR` here. |
| `SCALES` | `$MODEL_DIR/qsa_kv_scales.json` | Host path of the FP8 KV scale sidecar. Compose mounts it into the container. |
| `MODEL_NAME` | `Hinata` (compose), `qwen38-flash-next` (`start.sh`) | The name in the `model` field of a request. Spaces make aliases. |
| `HOST_PORT` | `8080` | The host port of the API. |
| `VLLM_API_KEY` | empty | Empty means no authentication. |
| `CACHE_HOST_DIR` | `.vllm-cache` in this directory | Where the compile caches live. The host path uses the same folder. |
| `CUDA_VISIBLE_DEVICES` | `0,1,2,3` | The GPU lane, in bus order. |
| `PLE_HOME` | `0` | The card that hosts the PLE offload. That card gets no CUDA context for it. |
| `QWEN38_ABLITERATION` | `Qwen3.8-Flash-Next-Uncensored` | Remove the refusal direction at run time. Empty disables it. See `patches/0102`. |
| `ABLITERATION_DIR` | `./abliteration` | Host folder with the direction file. Compose mounts it at `/opt/qwen38/abliteration`. |
| `GUARD` | `warn` | Bounds-guard telemetry: `off`, `warn`, or `raise`. |
| `COUNTERS` | `0` | `1` adds the logging-only kernel counters. |
| `VLLM_LOG_STATS_INTERVAL` | `1` | Seconds between engine stats lines. The vLLM default is 10 s. |
| `MAX_JOBS`, `ARCH` | `32`, `8.6` | Compile jobs and GPU architecture, for the build only. |
## 10. Benchmarks
docker-compose.yml
+1
View File
@@ -40,6 +40,7 @@ services:
TOPO_TIMEOUT: ${TOPO_TIMEOUT:-20}
GUARD: ${GUARD:-warn}
COUNTERS: ${COUNTERS:-0}
VLLM_LOG_STATS_INTERVAL: ${VLLM_LOG_STATS_INTERVAL:-1}
QWEN38_ABLITERATION: ${QWEN38_ABLITERATION:-Qwen3.8-Flash-Next-Uncensored}
volumes:
- "${MODEL_DIR:?Set MODEL_DIR in .env}:${MODEL_MOUNT_PATH:-/models/Qwen3.8-Flash-Next}:ro"
start.sh
+4 -1
View File
@@ -22,6 +22,7 @@
# PLE_HOME GPU that hosts the PLE offload home (0)
# GUARD bounds-guard telemetry off|warn|raise (warn)
# COUNTERS=1 logging-only counters (top-k exact restarts /30s, FP8 KV clips /300s)
# VLLM_LOG_STATS_INTERVAL seconds between engine stats lines (1; the vLLM default is 10 s)
# CUDA_HOME toolkit for first-serve JIT; default: the venv's CUDA 13.0 wheels
# AUTO_TOPO 0 (default, fixed shipped settings) | 1 | strict -> scripts/topo_select.py
# AUTO_TOPO_NCCL=default AUTO_TOPO_BIND=1 LANE_GPUS=auto LANE_CPUS=... TOPO_TIMEOUT=20
@@ -202,7 +203,9 @@ export VLLM_PLE_OFFLOAD_HOME_DEVICE="${PLE_HOME:-0}"
export VLLM_HOST_EMBED_TABLE=1
export VLLM_USE_BREAKABLE_CUDAGRAPH=0
export VLLM_QSA_KV_SCALES="${SCALES:-$CKPT/qsa_kv_scales.json}"
# engine stats line (throughput, KV usage, queue). Set empty to keep vLLM's default.
# Engine stats line (throughput, KV usage, queue). 1 s matches how often the numbers are useful.
# Unset, vLLM waits 10 s (vllm/envs.py:47), so a quiet run looks idle for ten seconds.
export VLLM_LOG_STATS_INTERVAL="${VLLM_LOG_STATS_INTERVAL:-1}"
export HF_HUB_OFFLINE=1 HF_DATASETS_OFFLINE=1
# the kernel toggles (the shipped values). These names are the tree's own: envs.py in the
# serving tree registers MERLIN_* for the kernel toggles and VLLM_E2_* for the runtime knobs.