Three faults ended a re-run before step 8, so a deleted artifacts/ was
never written again.
The import check read the first word of stdout, and vLLM logs to stdout.
On a box with mixed 3090 and 3090 Ti devices the device-mismatch WARNING
comes first, so the build stopped with "vllm reports WARNING, expected
0.30.0". Print a BUILD_SMOKE marker and read the version from that line.
Patch 0103 committed .build-products.sha256 into the serving tree, and
write_sums rewrites that file, so every re-run stopped at "modified
tracked files". Drop the file from the patch and exclude the two build
record paths from the check. The series now yields tree
a823fa36d3fd640f87b3de87153268d938130a47; OPS_OBJ is unchanged, so the
compiled kernels stay valid.
Step 8 skipped the pack whenever the kernels came from a tarball, and
judged an existing set by file presence alone. Require the tarball, the
per-file list, the manifest and the sidecar, verify the sidecar, and
pack from the products in the tree when any check fails.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
git am refreshes every file timestamp, so a Python-only patch series
forced a full extension rebuild and ninja saw everything as changed.
Diff csrc/ and the build files between the recorded tree and HEAD, and
rebuild only when a C++/CUDA source moved; otherwise the shipped .so
stays.
Update the expected tree to the 106-patch series
(c9c526755364d44d8d9923d9acc9d6d1a3d58718).
Add ipc: host, SYS_PTRACE, and seccomp=unconfined to the container so
a profiler or debugger can attach.
Raise --block-size from 2048 to 4096 and drop
--safetensors-load-strategy prefetch.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
attach_deferred_state raised 'deferred model state for a step without
sampled outputs' when a non-last PP rank produced a PLE deferred state but
receive() left the step slot None (no row samples). With num_running_reqs
> 1 a prefill-only step appears (a 2048-token chunk for one request,
num_output_tokens=0, scheduled_spec_decode_tokens={}), and the raise
killed the engine every time. Confirmed in the container log: the same
PLE fatal at sample_tokens on sequence 101, 1766, 3150, 3264.
The deferred state has no consumer for a None slot (get_prev_sampled_outputs
returns None) and references the KV cache it was built for, so drop it to
release the reference. This is a prefill-only step under concurrent load,
not a fault.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
MAX_SEQS and the prefill chunk were fixed in the start profile, so any
change to concurrency needed an edit to the script. Read both from .env,
and validate them before the engine starts. The chunk must be at least
the sequence count, the rule vLLM applies at scheduler.py:309. A
MAX_SEQS whose full context exceeds the 924,993-token pool prints a
note, because that shape preempts and recomputes instead of failing.
The captured graph list was a fixed [1,2,4,8,12,16], which covers a
decode batch of 4 requests and nothing wider. One decode step is K+1 = 4
tokens per request, so the list now follows MAX_SEQS: 1, 2, 4, then
every 4 tokens up to MAX_SEQS x 4. At the shipped 4 it produces the old
list exactly. Above 8 graphs a note prints, since each size costs
capture time and graph VRAM.
Drop --mm-processor-cache-gb. This workload sends each image once, so
the cache never had a hit to keep, and vLLM's own 4 GiB default is the
better base.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
The vision tower sat on GPU 0 and 1 all the time, and did work only
when a request carried an image. It now streams from pinned host RAM
and encodes over H2D, which frees 432 MiB on GPU 0 and 1. Stage 1
carries no tower, so GPU 2 and 3 gain nothing, and the KV cache stays
at 924,993 tokens because --kv-cache-memory pins it. The freed memory
is headroom, not capacity. An image request passes: 2223 image tokens
in 11.77 s, with no VRAM growth.
Key the prebuilt ops cache on the compiled inputs, so a Python-only
patch installs from the shipped tarball instead of a 20-40 minute
compile.
Raise --mm-processor-cache-gb to 16. That cache is duplicated for each
API process and engine core process, so 16 GiB costs 32 GiB of host
RAM and no VRAM, and repeated images stop re-processing.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
The stats line carried throughput, KV usage and queue depth, but the
interval was never set. Unset, vLLM waits 10 s (vllm/envs.py:47), so a
quiet run looked idle for ten seconds and a short one printed nothing.
Export VLLM_LOG_STATS_INTERVAL in start.sh with a default of 1, and pass
it through Compose so .env can raise it back to 10. Add the variable to
.env.example, to the start.sh header, and to the Variables table in both
READMEs.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
A request that named another model failed with 404 "The model `X` does
not exist.", and the model name changes often here. vLLM already has
the switch: _is_model_supported() accepts every name when
VLLM_SKIP_MODEL_NAME_VALIDATION is set, and _check_model is the only
gate, on the chat, completion, responses, tokenize, pooling and
realtime paths alike.
Turn it on in the serve profile, under the upstream name, so 0 still
restores the strict check. The served name does not change: /v1/models
and the model field of every reply still report MODEL_NAME. No patch is
needed, and Compose bind-mounts start.sh, so a restart applies this.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
SCALES had no row in the Variables table, and CKPT and MODEL_MOUNT_PATH
had none, so a reader had to open start.sh and docker-compose.yml to
place a scale file outside the model folder.
Add the three rows to the Variables table, say in section 1 that SCALES
takes a host path and that Compose mounts it, and give the -v and -e
pair for a start without Compose in section 7.4. Sync README-ID.md.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
The container read the sidecar only inside the MODEL_DIR mount, so a
scale file in another folder needed a manual -v and a container path.
Bind-mount the file named by SCALES to /opt/vllm/qsa_kv_scales.json, and
pass that fixed path to the engine. The default stays
$MODEL_DIR/qsa_kv_scales.json, so the shipped setup does not change. Add
a commented SCALES entry to .env.example.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
The default sidecar pointed at qsa_kv_scales_swift.json, a name no
checkpoint on this box carries. The guard in start.sh then stopped every
start with "scales sidecar missing", and a start without the guard would
run FP8 KV at scale 1.0.
Point the default at qsa_kv_scales.json, the file both checkpoints ship,
and rename the sidecar in the start.sh header, both READMEs, and the
Dockerfile comment. Name SCALES in the guard message.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Rename the default model from Scarlet to Hinata in the Dockerfile
environment, the healthcheck, and the file that
scripts/install-as-service.sh writes to /etc/default/vllm. Compose
already defaults to Hinata.
Sync README.md with the code. Section 6 paired port 8000 with the
Compose model name, so that request fails. Document vllm-logs.sh, the
abliteration, guard, and counter variables, and the new folders, so the
patch count reaches 103. Add section 10 for benchmarks/bench.sh, with
the defaults, the example runs, and what DEPTH changes. Drop the
duplicate CUDA_VISIBLE_DEVICES and PLE_HOME rows.
Add README-ID.md, the full Indonesian translation. README.md stays the
source, and both files say so.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Wrap llama-benchy for this box. Take prompt length, generation length,
concurrency, and context depth as arguments, and pass the rest to
llama-benchy. Defaults: pp=32768, tg=2048, concurrency 1, depth 0.
Validate the arguments, report a missing llama-benchy, and allow
BASE_URL, MODEL, and TOKENIZER overrides from the environment.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
Record the 220 W, 250 W, and 285 W sweep on PCIe Gen4 x16 and Gen3 x16.
State the testbed, the reproduce commands, and the results. Add the
findings: prefill scales sub-linearly, +12.9% from 220 W to 285 W on
Gen4, while decode stays inside the measured spread. Recommend Gen4 at
250 W.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
MOVZXandHinata