A request that named another model failed with 404 "The model `X` does
not exist.", and the model name changes often here. vLLM already has
the switch: _is_model_supported() accepts every name when
VLLM_SKIP_MODEL_NAME_VALIDATION is set, and _check_model is the only
gate, on the chat, completion, responses, tokenize, pooling and
realtime paths alike.
Turn it on in the serve profile, under the upstream name, so 0 still
restores the strict check. The served name does not change: /v1/models
and the model field of every reply still report MODEL_NAME. No patch is
needed, and Compose bind-mounts start.sh, so a restart applies this.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
SCALES had no row in the Variables table, and CKPT and MODEL_MOUNT_PATH
had none, so a reader had to open start.sh and docker-compose.yml to
place a scale file outside the model folder.
Add the three rows to the Variables table, say in section 1 that SCALES
takes a host path and that Compose mounts it, and give the -v and -e
pair for a start without Compose in section 7.4. Sync README-ID.md.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
The container read the sidecar only inside the MODEL_DIR mount, so a
scale file in another folder needed a manual -v and a container path.
Bind-mount the file named by SCALES to /opt/vllm/qsa_kv_scales.json, and
pass that fixed path to the engine. The default stays
$MODEL_DIR/qsa_kv_scales.json, so the shipped setup does not change. Add
a commented SCALES entry to .env.example.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
The default sidecar pointed at qsa_kv_scales_swift.json, a name no
checkpoint on this box carries. The guard in start.sh then stopped every
start with "scales sidecar missing", and a start without the guard would
run FP8 KV at scale 1.0.
Point the default at qsa_kv_scales.json, the file both checkpoints ship,
and rename the sidecar in the start.sh header, both READMEs, and the
Dockerfile comment. Name SCALES in the guard message.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Rename the default model from Scarlet to Hinata in the Dockerfile
environment, the healthcheck, and the file that
scripts/install-as-service.sh writes to /etc/default/vllm. Compose
already defaults to Hinata.
Sync README.md with the code. Section 6 paired port 8000 with the
Compose model name, so that request fails. Document vllm-logs.sh, the
abliteration, guard, and counter variables, and the new folders, so the
patch count reaches 103. Add section 10 for benchmarks/bench.sh, with
the defaults, the example runs, and what DEPTH changes. Drop the
duplicate CUDA_VISIBLE_DEVICES and PLE_HOME rows.
Add README-ID.md, the full Indonesian translation. README.md stays the
source, and both files say so.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Wrap llama-benchy for this box. Take prompt length, generation length,
concurrency, and context depth as arguments, and pass the rest to
llama-benchy. Defaults: pp=32768, tg=2048, concurrency 1, depth 0.
Validate the arguments, report a missing llama-benchy, and allow
BASE_URL, MODEL, and TOKENIZER overrides from the environment.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Record the 220 W, 250 W, and 285 W sweep on PCIe Gen4 x16 and Gen3 x16.
State the testbed, the reproduce commands, and the results. Add the
findings: prefill scales sub-linearly, +12.9% from 220 W to 285 W on
Gen4, while decode stays inside the measured spread. Recommend Gen4 at
250 W.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
MOVZXandHinata