MOVZXandHinata
b89e85e99f
The container read the sidecar only inside the MODEL_DIR mount, so a scale file in another folder needed a manual -v and a container path. Bind-mount the file named by SCALES to /opt/vllm/qsa_kv_scales.json, and pass that fixed path to the engine. The default stays $MODEL_DIR/qsa_kv_scales.json, so the shipped setup does not change. Add a commented SCALES entry to .env.example. Co-Authored-By: Hinata <hinata@idihore.id> Signed-off-by: MOVZX <movzx@yahoo.com>
56 lines
2.5 KiB
Bash
56 lines
2.5 KiB
Bash
# ── network ──────────────────────────────────────────────────────────────────
|
|
HOST_BIND_ADDR=0.0.0.0
|
|
HOST_PORT=8080
|
|
VLLM_API_KEY=
|
|
|
|
# ── model ────────────────────────────────────────────────────────────────────
|
|
MODEL_DIR=/data/models/halt95/Swift1.5-Qwen3.8-Flash-Next-W4A16-Merlin
|
|
MODEL_MOUNT_PATH=/models/Qwen3.8-Flash-Next
|
|
MODEL_NAME=Hinata
|
|
# The FP8 KV scale sidecar, a host path. Default: $MODEL_DIR/qsa_kv_scales.json.
|
|
# SCALES=
|
|
|
|
# ── image and container ──────────────────────────────────────────────────────
|
|
IMAGE=qwen38-flash-next
|
|
IMAGE_TAG=latest
|
|
CONTAINER_NAME=vllm
|
|
# CACHE_HOST_DIR=/srv/vllm-cache
|
|
|
|
# ── engine ───────────────────────────────────────────────────────────────────
|
|
# Card hosting the PLE offload home (no CUDA context there).
|
|
PLE_HOME=0
|
|
# GPU lane, in bus order. The shipped shape uses all four cards.
|
|
CUDA_VISIBLE_DEVICES=0,1,2,3
|
|
# Topology selector (opt-in): 0 keeps the fixed shipped settings | 1 | strict.
|
|
# The image has no numactl, so AUTO_TOPO_BIND=1 is skipped.
|
|
AUTO_TOPO=0
|
|
AUTO_TOPO_NCCL=legacy
|
|
AUTO_TOPO_BIND=0
|
|
# LANE_GPUS=auto
|
|
# LANE_CPUS=0-15,16-31
|
|
TOPO_TIMEOUT=20
|
|
# Bounds-guard telemetry: off | warn | fail.
|
|
GUARD=warn
|
|
# 1 adds the two logging-only counters (exact-restart counters /30 s, FP8 KV clips /300 s).
|
|
COUNTERS=0
|
|
|
|
# Extracted from orcarouter/Qwen3.8-Flash-Next-Uncensored, hidden 2560
|
|
QWEN38_ABLITERATION=Qwen3.8-Flash-Next-Uncensored
|
|
ABLITERATION_DIR=./abliteration
|
|
|
|
# ── lifecycle, logs, health ──────────────────────────────────────────────────
|
|
LOG_MAX_SIZE=100m
|
|
LOG_MAX_FILE=14
|
|
|
|
# The image healthcheck is a real one-token generation, not /v1/models. Only the tuning is here.
|
|
HEALTHCHECK_INTERVAL=60s
|
|
HEALTHCHECK_TIMEOUT=90s
|
|
HEALTHCHECK_START_PERIOD=30m
|
|
HEALTHCHECK_RETRIES=3
|
|
|
|
# ── build (docker compose build reads these) ──────────────────────────────────────
|
|
UV_CACHE_DIR=/root/.cache/uv
|
|
PIP_CACHE_DIR=/root/.cache/pip
|
|
MAX_JOBS=32
|
|
ARCH=8.6
|