iHore/vLLM-4x-RTX-3090
8
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat: Mount the KV scale sidecar from any host path

Browse Source 
The container read the sidecar only inside the MODEL_DIR mount, so a
scale file in another folder needed a manual -v and a container path.

Bind-mount the file named by SCALES to /opt/vllm/qsa_kv_scales.json, and
pass that fixed path to the engine. The default stays
$MODEL_DIR/qsa_kv_scales.json, so the shipped setup does not change. Add
a commented SCALES entry to .env.example.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
This commit is contained in:
MOVZX
2026-10-15 18:52:18 +00:00
co-authored by Hinata
parent 1281bf9c03
commit b89e85e99f
2 changed files with 4 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.env.example
+2
View File
@@ -7,6 +7,8 @@ VLLM_API_KEY=
MODEL_DIR=/data/models/halt95/Swift1.5-Qwen3.8-Flash-Next-W4A16-Merlin MODEL_DIR=/data/models/halt95/Swift1.5-Qwen3.8-Flash-Next-W4A16-Merlin
MODEL_MOUNT_PATH=/models/Qwen3.8-Flash-Next MODEL_MOUNT_PATH=/models/Qwen3.8-Flash-Next
MODEL_NAME=Hinata MODEL_NAME=Hinata
# The FP8 KV scale sidecar, a host path. Default: $MODEL_DIR/qsa_kv_scales.json.
# SCALES=
# ── image and container ────────────────────────────────────────────────────── # ── image and container ──────────────────────────────────────────────────────
IMAGE=qwen38-flash-next IMAGE=qwen38-flash-next
docker-compose.yml
+2 -1
View File
@@ -28,7 +28,7 @@ services:
environment: environment:
MODEL_NAME: ${MODEL_NAME:-Hinata} MODEL_NAME: ${MODEL_NAME:-Hinata}
PORT: 8000 PORT: 8000
SCALES: ${SCALES:-} SCALES: /opt/vllm/qsa_kv_scales.json
VLLM_API_KEY: ${VLLM_API_KEY:-} VLLM_API_KEY: ${VLLM_API_KEY:-}
PLE_HOME: ${PLE_HOME:-0} PLE_HOME: ${PLE_HOME:-0}
CUDA_VISIBLE_DEVICES: ${CUDA_VISIBLE_DEVICES:-0,1,2,3} CUDA_VISIBLE_DEVICES: ${CUDA_VISIBLE_DEVICES:-0,1,2,3}
@@ -45,6 +45,7 @@ services:
- "${MODEL_DIR:?Set MODEL_DIR in .env}:${MODEL_MOUNT_PATH:-/models/Qwen3.8-Flash-Next}:ro" - "${MODEL_DIR:?Set MODEL_DIR in .env}:${MODEL_MOUNT_PATH:-/models/Qwen3.8-Flash-Next}:ro"
- ${ABLITERATION_DIR:-./abliteration}:/opt/qwen38/abliteration:ro - ${ABLITERATION_DIR:-./abliteration}:/opt/qwen38/abliteration:ro
- ${CACHE_HOST_DIR:-./.vllm-cache}:/cache - ${CACHE_HOST_DIR:-./.vllm-cache}:/cache
- ${SCALES:-${MODEL_DIR}/qsa_kv_scales.json}:/opt/vllm/qsa_kv_scales.json:ro
- ./start.sh:/opt/vllm/start.sh:ro - ./start.sh:/opt/vllm/start.sh:ro
restart: unless-stopped restart: unless-stopped
stop_grace_period: 20s stop_grace_period: 20s