The vision tower sat on GPU 0 and 1 all the time, and did work only
when a request carried an image. It now streams from pinned host RAM
and encodes over H2D, which frees 432 MiB on GPU 0 and 1. Stage 1
carries no tower, so GPU 2 and 3 gain nothing, and the KV cache stays
at 924,993 tokens because --kv-cache-memory pins it. The freed memory
is headroom, not capacity. An image request passes: 2223 image tokens
in 11.77 s, with no VRAM growth.
Key the prebuilt ops cache on the compiled inputs, so a Python-only
patch installs from the shipped tarball instead of a 20-40 minute
compile.
Raise --mm-processor-cache-gb to 16. That cache is duplicated for each
API process and engine core process, so 16 GiB costs 32 GiB of host
RAM and no VRAM, and repeated images stop re-processing.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
The stats line carried throughput, KV usage and queue depth, but the
interval was never set. Unset, vLLM waits 10 s (vllm/envs.py:47), so a
quiet run looked idle for ten seconds and a short one printed nothing.
Export VLLM_LOG_STATS_INTERVAL in start.sh with a default of 1, and pass
it through Compose so .env can raise it back to 10. Add the variable to
.env.example, to the start.sh header, and to the Variables table in both
READMEs.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
A request that named another model failed with 404 "The model `X` does
not exist.", and the model name changes often here. vLLM already has
the switch: _is_model_supported() accepts every name when
VLLM_SKIP_MODEL_NAME_VALIDATION is set, and _check_model is the only
gate, on the chat, completion, responses, tokenize, pooling and
realtime paths alike.
Turn it on in the serve profile, under the upstream name, so 0 still
restores the strict check. The served name does not change: /v1/models
and the model field of every reply still report MODEL_NAME. No patch is
needed, and Compose bind-mounts start.sh, so a restart applies this.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
SCALES had no row in the Variables table, and CKPT and MODEL_MOUNT_PATH
had none, so a reader had to open start.sh and docker-compose.yml to
place a scale file outside the model folder.
Add the three rows to the Variables table, say in section 1 that SCALES
takes a host path and that Compose mounts it, and give the -v and -e
pair for a start without Compose in section 7.4. Sync README-ID.md.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
The container read the sidecar only inside the MODEL_DIR mount, so a
scale file in another folder needed a manual -v and a container path.
Bind-mount the file named by SCALES to /opt/vllm/qsa_kv_scales.json, and
pass that fixed path to the engine. The default stays
$MODEL_DIR/qsa_kv_scales.json, so the shipped setup does not change. Add
a commented SCALES entry to .env.example.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
The default sidecar pointed at qsa_kv_scales_swift.json, a name no
checkpoint on this box carries. The guard in start.sh then stopped every
start with "scales sidecar missing", and a start without the guard would
run FP8 KV at scale 1.0.
Point the default at qsa_kv_scales.json, the file both checkpoints ship,
and rename the sidecar in the start.sh header, both READMEs, and the
Dockerfile comment. Name SCALES in the guard message.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Rename the default model from Scarlet to Hinata in the Dockerfile
environment, the healthcheck, and the file that
scripts/install-as-service.sh writes to /etc/default/vllm. Compose
already defaults to Hinata.
Sync README.md with the code. Section 6 paired port 8000 with the
Compose model name, so that request fails. Document vllm-logs.sh, the
abliteration, guard, and counter variables, and the new folders, so the
patch count reaches 103. Add section 10 for benchmarks/bench.sh, with
the defaults, the example runs, and what DEPTH changes. Drop the
duplicate CUDA_VISIBLE_DEVICES and PLE_HOME rows.
Add README-ID.md, the full Indonesian translation. README.md stays the
source, and both files say so.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Wrap llama-benchy for this box. Take prompt length, generation length,
concurrency, and context depth as arguments, and pass the rest to
llama-benchy. Defaults: pp=32768, tg=2048, concurrency 1, depth 0.
Validate the arguments, report a missing llama-benchy, and allow
BASE_URL, MODEL, and TOKENIZER overrides from the environment.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Record the 220 W, 250 W, and 285 W sweep on PCIe Gen4 x16 and Gen3 x16.
State the testbed, the reproduce commands, and the results. Add the
findings: prefill scales sub-linearly, +12.9% from 220 W to 285 W on
Gen4, while decode stays inside the measured spread. Recommend Gen4 at
250 W.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
MOVZXandHinata