The vision tower sat on GPU 0 and 1 all the time, and did work only
when a request carried an image. It now streams from pinned host RAM
and encodes over H2D, which frees 432 MiB on GPU 0 and 1. Stage 1
carries no tower, so GPU 2 and 3 gain nothing, and the KV cache stays
at 924,993 tokens because --kv-cache-memory pins it. The freed memory
is headroom, not capacity. An image request passes: 2223 image tokens
in 11.77 s, with no VRAM growth.
Key the prebuilt ops cache on the compiled inputs, so a Python-only
patch installs from the shipped tarball instead of a 20-40 minute
compile.
Raise --mm-processor-cache-gb to 16. That cache is duplicated for each
API process and engine core process, so 16 GiB costs 32 GiB of host
RAM and no VRAM, and repeated images stop re-processing.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
The stats line carried throughput, KV usage and queue depth, but the
interval was never set. Unset, vLLM waits 10 s (vllm/envs.py:47), so a
quiet run looked idle for ten seconds and a short one printed nothing.
Export VLLM_LOG_STATS_INTERVAL in start.sh with a default of 1, and pass
it through Compose so .env can raise it back to 10. Add the variable to
.env.example, to the start.sh header, and to the Variables table in both
READMEs.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
A request that named another model failed with 404 "The model `X` does
not exist.", and the model name changes often here. vLLM already has
the switch: _is_model_supported() accepts every name when
VLLM_SKIP_MODEL_NAME_VALIDATION is set, and _check_model is the only
gate, on the chat, completion, responses, tokenize, pooling and
realtime paths alike.
Turn it on in the serve profile, under the upstream name, so 0 still
restores the strict check. The served name does not change: /v1/models
and the model field of every reply still report MODEL_NAME. No patch is
needed, and Compose bind-mounts start.sh, so a restart applies this.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
The default sidecar pointed at qsa_kv_scales_swift.json, a name no
checkpoint on this box carries. The guard in start.sh then stopped every
start with "scales sidecar missing", and a start without the guard would
run FP8 KV at scale 1.0.
Point the default at qsa_kv_scales.json, the file both checkpoints ship,
and rename the sidecar in the start.sh header, both READMEs, and the
Dockerfile comment. Name SCALES in the guard message.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
MOVZXandHinata