feat: Mount the KV scale sidecar from any host path
The container read the sidecar only inside the MODEL_DIR mount, so a scale file in another folder needed a manual -v and a container path. Bind-mount the file named by SCALES to /opt/vllm/qsa_kv_scales.json, and pass that fixed path to the engine. The default stays $MODEL_DIR/qsa_kv_scales.json, so the shipped setup does not change. Add a commented SCALES entry to .env.example. Co-Authored-By: Hinata <hinata@idihore.id> Signed-off-by: MOVZX <movzx@yahoo.com>
This commit is contained in:
+2
-1
@@ -28,7 +28,7 @@ services:
|
||||
environment:
|
||||
MODEL_NAME: ${MODEL_NAME:-Hinata}
|
||||
PORT: 8000
|
||||
SCALES: ${SCALES:-}
|
||||
SCALES: /opt/vllm/qsa_kv_scales.json
|
||||
VLLM_API_KEY: ${VLLM_API_KEY:-}
|
||||
PLE_HOME: ${PLE_HOME:-0}
|
||||
CUDA_VISIBLE_DEVICES: ${CUDA_VISIBLE_DEVICES:-0,1,2,3}
|
||||
@@ -45,6 +45,7 @@ services:
|
||||
- "${MODEL_DIR:?Set MODEL_DIR in .env}:${MODEL_MOUNT_PATH:-/models/Qwen3.8-Flash-Next}:ro"
|
||||
- ${ABLITERATION_DIR:-./abliteration}:/opt/qwen38/abliteration:ro
|
||||
- ${CACHE_HOST_DIR:-./.vllm-cache}:/cache
|
||||
- ${SCALES:-${MODEL_DIR}/qsa_kv_scales.json}:/opt/vllm/qsa_kv_scales.json:ro
|
||||
- ./start.sh:/opt/vllm/start.sh:ro
|
||||
restart: unless-stopped
|
||||
stop_grace_period: 20s
|
||||
|
||||
Reference in New Issue
Block a user