Three faults ended a re-run before step 8, so a deleted artifacts/ was
never written again.
The import check read the first word of stdout, and vLLM logs to stdout.
On a box with mixed 3090 and 3090 Ti devices the device-mismatch WARNING
comes first, so the build stopped with "vllm reports WARNING, expected
0.30.0". Print a BUILD_SMOKE marker and read the version from that line.
Patch 0103 committed .build-products.sha256 into the serving tree, and
write_sums rewrites that file, so every re-run stopped at "modified
tracked files". Drop the file from the patch and exclude the two build
record paths from the check. The series now yields tree
a823fa36d3fd640f87b3de87153268d938130a47; OPS_OBJ is unchanged, so the
compiled kernels stay valid.
Step 8 skipped the pack whenever the kernels came from a tarball, and
judged an existing set by file presence alone. Require the tarball, the
per-file list, the manifest and the sidecar, verify the sidecar, and
pack from the products in the tree when any check fails.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
attach_deferred_state raised 'deferred model state for a step without
sampled outputs' when a non-last PP rank produced a PLE deferred state but
receive() left the step slot None (no row samples). With num_running_reqs
> 1 a prefill-only step appears (a 2048-token chunk for one request,
num_output_tokens=0, scheduled_spec_decode_tokens={}), and the raise
killed the engine every time. Confirmed in the container log: the same
PLE fatal at sample_tokens on sequence 101, 1766, 3150, 3264.
The deferred state has no consumer for a None slot (get_prev_sampled_outputs
returns None) and references the KV cache it was built for, so drop it to
release the reference. This is a prefill-only step under concurrent load,
not a fault.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
The vision tower sat on GPU 0 and 1 all the time, and did work only
when a request carried an image. It now streams from pinned host RAM
and encodes over H2D, which frees 432 MiB on GPU 0 and 1. Stage 1
carries no tower, so GPU 2 and 3 gain nothing, and the KV cache stays
at 924,993 tokens because --kv-cache-memory pins it. The freed memory
is headroom, not capacity. An image request passes: 2223 image tokens
in 11.77 s, with no VRAM growth.
Key the prebuilt ops cache on the compiled inputs, so a Python-only
patch installs from the shipped tarball instead of a 20-40 minute
compile.
Raise --mm-processor-cache-gb to 16. That cache is duplicated for each
API process and engine core process, so 16 GiB costs 32 GiB of host
RAM and no VRAM, and repeated images stop re-processing.
Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>
MOVZXandHinata