MOVZXandHinata
d9ca3bc4f2
MAX_SEQS and the prefill chunk were fixed in the start profile, so any change to concurrency needed an edit to the script. Read both from .env, and validate them before the engine starts. The chunk must be at least the sequence count, the rule vLLM applies at scheduler.py:309. A MAX_SEQS whose full context exceeds the 924,993-token pool prints a note, because that shape preempts and recomputes instead of failing. The captured graph list was a fixed [1,2,4,8,12,16], which covers a decode batch of 4 requests and nothing wider. One decode step is K+1 = 4 tokens per request, so the list now follows MAX_SEQS: 1, 2, 4, then every 4 tokens up to MAX_SEQS x 4. At the shipped 4 it produces the old list exactly. Above 8 graphs a note prints, since each size costs capture time and graph VRAM. Drop --mm-processor-cache-gb. This workload sends each image once, so the cache never had a hit to keep, and vLLM's own 4 GiB default is the better base. Co-Authored-By: Hinata <hinata@idihore.id> Signed-off-by: MOVZX <movzx@yahoo.com>
64 lines
3.0 KiB
Bash
64 lines
3.0 KiB
Bash
# ── network ──────────────────────────────────────────────────────────────────
|
|
HOST_BIND_ADDR=0.0.0.0
|
|
HOST_PORT=8080
|
|
VLLM_API_KEY=
|
|
|
|
# ── model ────────────────────────────────────────────────────────────────────
|
|
MODEL_DIR=/data/models/halt95/Swift1.5-Qwen3.8-Flash-Next-W4A16-Merlin
|
|
MODEL_MOUNT_PATH=/models/Qwen3.8-Flash-Next
|
|
MODEL_NAME=Hinata
|
|
# The FP8 KV scale sidecar, a host path. Default: $MODEL_DIR/qsa_kv_scales.json.
|
|
# SCALES=
|
|
|
|
# ── image and container ──────────────────────────────────────────────────────
|
|
IMAGE=qwen38-flash-next
|
|
IMAGE_TAG=latest
|
|
CONTAINER_NAME=vllm
|
|
# CACHE_HOST_DIR=/srv/vllm-cache
|
|
|
|
# ── engine ───────────────────────────────────────────────────────────────────
|
|
# Card hosting the PLE offload home (no CUDA context there).
|
|
PLE_HOME=0
|
|
# GPU lane, in bus order. The shipped shape uses all four cards.
|
|
CUDA_VISIBLE_DEVICES=0,1,2,3
|
|
# Topology selector (opt-in): 0 keeps the fixed shipped settings | 1 | strict.
|
|
# The image has no numactl, so AUTO_TOPO_BIND=1 is skipped.
|
|
AUTO_TOPO=0
|
|
AUTO_TOPO_NCCL=legacy
|
|
AUTO_TOPO_BIND=0
|
|
# LANE_GPUS=auto
|
|
# LANE_CPUS=0-15,16-31
|
|
TOPO_TIMEOUT=20
|
|
# Bounds-guard telemetry: off | warn | fail.
|
|
GUARD=warn
|
|
# 1 adds the two logging-only counters (exact-restart counters /30 s, FP8 KV clips /300 s).
|
|
COUNTERS=0
|
|
# Concurrent requests. CUDA graphs are captured up to MAX_SEQS x 4 tokens (MTP K=3), so a raise
|
|
# adds capture time and graph VRAM. N x 262144 tokens above the 924,993-token KV pool preempts.
|
|
MAX_SEQS=4
|
|
# Prefill chunk in tokens. Must be >= MAX_SEQS. 2048 matches --block-size; 4096 prefills one
|
|
# max-size image in one chunk.
|
|
MAX_BATCHED_TOKENS=2048
|
|
# Seconds between engine stats lines (throughput, KV usage, queue depth). The vLLM default is 10 s.
|
|
VLLM_LOG_STATS_INTERVAL=1
|
|
|
|
# Extracted from orcarouter/Qwen3.8-Flash-Next-Uncensored, hidden 2560
|
|
QWEN38_ABLITERATION=Qwen3.8-Flash-Next-Uncensored
|
|
ABLITERATION_DIR=./abliteration
|
|
|
|
# ── lifecycle, logs, health ──────────────────────────────────────────────────
|
|
LOG_MAX_SIZE=100m
|
|
LOG_MAX_FILE=14
|
|
|
|
# The image healthcheck is a real one-token generation, not /v1/models. Only the tuning is here.
|
|
HEALTHCHECK_INTERVAL=60s
|
|
HEALTHCHECK_TIMEOUT=90s
|
|
HEALTHCHECK_START_PERIOD=30m
|
|
HEALTHCHECK_RETRIES=3
|
|
|
|
# ── build (docker compose build reads these) ──────────────────────────────────────
|
|
UV_CACHE_DIR=/root/.cache/uv
|
|
PIP_CACHE_DIR=/root/.cache/pip
|
|
MAX_JOBS=32
|
|
ARCH=8.6
|