MOVZXandHinata
3a0a5d1510
The stats line carried throughput, KV usage and queue depth, but the interval was never set. Unset, vLLM waits 10 s (vllm/envs.py:47), so a quiet run looked idle for ten seconds and a short one printed nothing. Export VLLM_LOG_STATS_INTERVAL in start.sh with a default of 1, and pass it through Compose so .env can raise it back to 10. Add the variable to .env.example, to the start.sh header, and to the Variables table in both READMEs. Co-Authored-By: Hinata <hinata@idihore.id> Signed-off-by: MOVZX <movzx@yahoo.com>
58 lines
2.7 KiB
Bash
58 lines
2.7 KiB
Bash
# ── network ──────────────────────────────────────────────────────────────────
|
|
HOST_BIND_ADDR=0.0.0.0
|
|
HOST_PORT=8080
|
|
VLLM_API_KEY=
|
|
|
|
# ── model ────────────────────────────────────────────────────────────────────
|
|
MODEL_DIR=/data/models/halt95/Swift1.5-Qwen3.8-Flash-Next-W4A16-Merlin
|
|
MODEL_MOUNT_PATH=/models/Qwen3.8-Flash-Next
|
|
MODEL_NAME=Hinata
|
|
# The FP8 KV scale sidecar, a host path. Default: $MODEL_DIR/qsa_kv_scales.json.
|
|
# SCALES=
|
|
|
|
# ── image and container ──────────────────────────────────────────────────────
|
|
IMAGE=qwen38-flash-next
|
|
IMAGE_TAG=latest
|
|
CONTAINER_NAME=vllm
|
|
# CACHE_HOST_DIR=/srv/vllm-cache
|
|
|
|
# ── engine ───────────────────────────────────────────────────────────────────
|
|
# Card hosting the PLE offload home (no CUDA context there).
|
|
PLE_HOME=0
|
|
# GPU lane, in bus order. The shipped shape uses all four cards.
|
|
CUDA_VISIBLE_DEVICES=0,1,2,3
|
|
# Topology selector (opt-in): 0 keeps the fixed shipped settings | 1 | strict.
|
|
# The image has no numactl, so AUTO_TOPO_BIND=1 is skipped.
|
|
AUTO_TOPO=0
|
|
AUTO_TOPO_NCCL=legacy
|
|
AUTO_TOPO_BIND=0
|
|
# LANE_GPUS=auto
|
|
# LANE_CPUS=0-15,16-31
|
|
TOPO_TIMEOUT=20
|
|
# Bounds-guard telemetry: off | warn | fail.
|
|
GUARD=warn
|
|
# 1 adds the two logging-only counters (exact-restart counters /30 s, FP8 KV clips /300 s).
|
|
COUNTERS=0
|
|
# Seconds between engine stats lines (throughput, KV usage, queue depth). The vLLM default is 10 s.
|
|
VLLM_LOG_STATS_INTERVAL=1
|
|
|
|
# Extracted from orcarouter/Qwen3.8-Flash-Next-Uncensored, hidden 2560
|
|
QWEN38_ABLITERATION=Qwen3.8-Flash-Next-Uncensored
|
|
ABLITERATION_DIR=./abliteration
|
|
|
|
# ── lifecycle, logs, health ──────────────────────────────────────────────────
|
|
LOG_MAX_SIZE=100m
|
|
LOG_MAX_FILE=14
|
|
|
|
# The image healthcheck is a real one-token generation, not /v1/models. Only the tuning is here.
|
|
HEALTHCHECK_INTERVAL=60s
|
|
HEALTHCHECK_TIMEOUT=90s
|
|
HEALTHCHECK_START_PERIOD=30m
|
|
HEALTHCHECK_RETRIES=3
|
|
|
|
# ── build (docker compose build reads these) ──────────────────────────────────────
|
|
UV_CACHE_DIR=/root/.cache/uv
|
|
PIP_CACHE_DIR=/root/.cache/pip
|
|
MAX_JOBS=32
|
|
ARCH=8.6
|