Three faults ended a re-run before step 8, so a deleted artifacts/ was
never written again.
The import check read the first word of stdout, and vLLM logs to stdout.
On a box with mixed 3090 and 3090 Ti devices the device-mismatch WARNING
comes first, so the build stopped with "vllm reports WARNING, expected
0.30.0". Print a BUILD_SMOKE marker and read the version from that line.
Patch 0103 committed .build-products.sha256 into the serving tree, and
write_sums rewrites that file, so every re-run stopped at "modified
tracked files". Drop the file from the patch and exclude the two build
record paths from the check. The series now yields tree
a823fa36d3fd640f87b3de87153268d938130a47; OPS_OBJ is unchanged, so the
compiled kernels stay valid.
Step 8 skipped the pack whenever the kernels came from a tarball, and
judged an existing set by file presence alone. Require the tarball, the
per-file list, the manifest and the sidecar, verify the sidecar, and
pack from the products in the tree when any check fails.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
MAX_SEQS and the prefill chunk were fixed in the start profile, so any
change to concurrency needed an edit to the script. Read both from .env,
and validate them before the engine starts. The chunk must be at least
the sequence count, the rule vLLM applies at scheduler.py:309. A
MAX_SEQS whose full context exceeds the 924,993-token pool prints a
note, because that shape preempts and recomputes instead of failing.
The captured graph list was a fixed [1,2,4,8,12,16], which covers a
decode batch of 4 requests and nothing wider. One decode step is K+1 = 4
tokens per request, so the list now follows MAX_SEQS: 1, 2, 4, then
every 4 tokens up to MAX_SEQS x 4. At the shipped 4 it produces the old
list exactly. Above 8 graphs a note prints, since each size costs
capture time and graph VRAM.
Drop --mm-processor-cache-gb. This workload sends each image once, so
the cache never had a hit to keep, and vLLM's own 4 GiB default is the
better base.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
The vision tower sat on GPU 0 and 1 all the time, and did work only
when a request carried an image. It now streams from pinned host RAM
and encodes over H2D, which frees 432 MiB on GPU 0 and 1. Stage 1
carries no tower, so GPU 2 and 3 gain nothing, and the KV cache stays
at 924,993 tokens because --kv-cache-memory pins it. The freed memory
is headroom, not capacity. An image request passes: 2223 image tokens
in 11.77 s, with no VRAM growth.
Key the prebuilt ops cache on the compiled inputs, so a Python-only
patch installs from the shipped tarball instead of a 20-40 minute
compile.
Raise --mm-processor-cache-gb to 16. That cache is duplicated for each
API process and engine core process, so 16 GiB costs 32 GiB of host
RAM and no VRAM, and repeated images stop re-processing.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
The stats line carried throughput, KV usage and queue depth, but the
interval was never set. Unset, vLLM waits 10 s (vllm/envs.py:47), so a
quiet run looked idle for ten seconds and a short one printed nothing.
Export VLLM_LOG_STATS_INTERVAL in start.sh with a default of 1, and pass
it through Compose so .env can raise it back to 10. Add the variable to
.env.example, to the start.sh header, and to the Variables table in both
READMEs.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
SCALES had no row in the Variables table, and CKPT and MODEL_MOUNT_PATH
had none, so a reader had to open start.sh and docker-compose.yml to
place a scale file outside the model folder.
Add the three rows to the Variables table, say in section 1 that SCALES
takes a host path and that Compose mounts it, and give the -v and -e
pair for a start without Compose in section 7.4. Sync README-ID.md.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
The default sidecar pointed at qsa_kv_scales_swift.json, a name no
checkpoint on this box carries. The guard in start.sh then stopped every
start with "scales sidecar missing", and a start without the guard would
run FP8 KV at scale 1.0.
Point the default at qsa_kv_scales.json, the file both checkpoints ship,
and rename the sidecar in the start.sh header, both READMEs, and the
Dockerfile comment. Name SCALES in the guard message.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Rename the default model from Scarlet to Hinata in the Dockerfile
environment, the healthcheck, and the file that
scripts/install-as-service.sh writes to /etc/default/vllm. Compose
already defaults to Hinata.
Sync README.md with the code. Section 6 paired port 8000 with the
Compose model name, so that request fails. Document vllm-logs.sh, the
abliteration, guard, and counter variables, and the new folders, so the
patch count reaches 103. Add section 10 for benchmarks/bench.sh, with
the defaults, the example runs, and what DEPTH changes. Drop the
duplicate CUDA_VISIBLE_DEVICES and PLE_HOME rows.
Add README-ID.md, the full Indonesian translation. README.md stays the
source, and both files say so.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
MOVZXandHinata