build: Rewrite artifacts/ when it is missing, and stop failing re-runs

Three faults ended a re-run before step 8, so a deleted artifacts/ was
never written again.

The import check read the first word of stdout, and vLLM logs to stdout.
On a box with mixed 3090 and 3090 Ti devices the device-mismatch WARNING
comes first, so the build stopped with "vllm reports WARNING, expected
0.30.0". Print a BUILD_SMOKE marker and read the version from that line.

Patch 0103 committed .build-products.sha256 into the serving tree, and
write_sums rewrites that file, so every re-run stopped at "modified
tracked files". Drop the file from the patch and exclude the two build
record paths from the check. The series now yields tree
a823fa36d3fd640f87b3de87153268d938130a47; OPS_OBJ is unchanged, so the
compiled kernels stay valid.

Step 8 skipped the pack whenever the kernels came from a tarball, and
judged an existing set by file presence alone. Require the tarball, the
per-file list, the manifest and the sidecar, verify the sidecar, and
pack from the products in the tree when any check fails.

Co-Authored-By: Hinata <hinata@idihore.id>
Signed-off-by: MOVZX <movzx@yahoo.com>

README-ID.md

@@ -54,6 +54,8 @@ Jalankan ulang pada tree yang tidak berubah melewati compile dan install. Ia
 membaca commit tree dan dua hash ekstensi dari `vllm/.build-products`,
 meng-hash ulang `vllm/.build-products.sha256`, dan mencari install vllm di venv.
 Setiap ketidakcocokan memulai compile. `./build.sh --force` selalu meng-compile.
+Jalankan ulang juga memeriksa ulang `artifacts/`. Kalau foldernya hilang atau tidak
+lengkap, langkah 8 menulisnya lagi dari kernel yang sudah di-compile (lihat 7).
 
 ### 2.2. Tree
 
@@ -167,7 +169,9 @@ berjalan di host GPU.
 
 Langkah 8 dari `build.sh` menulis kernel yang sudah di-compile ke `artifacts/`:
 satu tarball, daftar sha256 per file, dan satu manifest. `--no-pack` (`PACK=0`)
-melewati langkah itu. Jalan kedua pada tree yang sama tidak mem-pack ulang.
+melewati langkah itu. Jalan kedua pada tree yang sama tidak mem-pack ulang. Hapus isi
+`artifacts/` dan jalan berikutnya menulisnya lagi. Biayanya satu kali packing (sekitar 45
+detik), bukan compile ulang.
 
 Ada dua route build:
 

README.md

@@ -52,7 +52,9 @@ visible, and one status line appears every 60 seconds. Full output goes to
 A re-run of an unchanged tree skips the compile and the install. It reads the tree commit
 and the two extension hashes from `vllm/.build-products`, re-hashes
 `vllm/.build-products.sha256`, and looks for the vllm install in the venv. Any mismatch
-starts the compile. `./build.sh --force` always compiles.
+starts the compile. `./build.sh --force` always compiles. A re-run also re-checks
+`artifacts/`. When that folder is missing or incomplete, step 8 writes it again from the kernels
+that are already compiled (see 7).
 
 ### 2.2. The tree
 
@@ -160,7 +162,8 @@ the GPU host.
 
 Step 8 of `build.sh` writes the compiled kernels to `artifacts/`: a tarball, a per-file
 sha256 list, and a manifest. `--no-pack` (`PACK=0`) skips that step. A second run of the
-same tree does not pack again.
+same tree does not pack again. Delete `artifacts/` and the next run writes it again. That costs
+one pack (about 45 seconds), not a recompile.
 
 Two build routes exist:
 

build.sh

@@ -22,7 +22,8 @@
 # that tarball, so the Dockerfile uses this route when you pass --build-context ops=$PWD/artifacts.
 #
 # step 8 packs the kernels into artifacts/ for the image build (PACK=0 or --no-pack skips it; the
-# Dockerfile sets it, an image build must not grow a 340 MB tarball inside a layer).
+# Dockerfile sets it, an image build must not grow a 340 MB tarball inside a layer). Delete artifacts/
+# and the next run writes it again from the compiled products, with no recompile.
 #
 # Needs: git curl tar gzip sha256sum cmake >= 3.26 ninja cc c++, and uv (UV=<path>, UV=0 to refuse)
 # or python3.13 with the venv module. Optional: cargo 1.95 for the Rust frontend.
@@ -102,7 +103,9 @@ step 2 "serving tree (upstream v0.30.0 + $(ls "$HERE"/patches/*.patch | wc -l) p
 # The tree is not in this repo: this repo ships the patches, not the tree. Fetch applies the series
 # to the upstream tag and asserts the result, so a patch series that drifts fails here, not at serve.
 BASE=ced6857afa0ea7b2e3f0846a62e1394e90f15607              # tag v0.30.0
-EXPECTED_TREE=c9c526755364d44d8d9923d9acc9d6d1a3d58718     # BASE + patches/*.patch (106), verified 2026-10-17
+EXPECTED_TREE=a823fa36d3fd640f87b3de87153268d938130a47     # BASE + patches/*.patch (106), verified 2026-10-18
+# The tree carries no build record: the series of patch 0103 once committed .build-products.sha256,
+# which build.sh rewrites, so every re-run failed the "modified tracked files" check below.
 URL="${VLLM_UPSTREAM_URL:-https://github.com/vllm-project/vllm.git}"
 # git am makes a commit, so git needs a committer identity: the host's ~/.gitconfig is not there (a
 # container, a build user, a fresh box). The committer is not in the tree hash, so the assertion
@@ -131,7 +134,10 @@ fi
 tree="$(git -C "$SRC" rev-parse 'HEAD^{tree}' 2>/dev/null || echo none)"
 [ "$tree" = "$EXPECTED_TREE" ] || { echo "tree $tree != expected $EXPECTED_TREE: the series drifted, or the tree is older than the patches. FETCH=1 re-applies it"; exit 1; }
 echo "    tree ready: $(git -C "$SRC" rev-parse --short HEAD) tree $tree"
-[ -z "$(git -C "$SRC" status --porcelain --untracked-files=no 2>/dev/null)" ] || { echo "$SRC has modified tracked files"; exit 1; }
+# build.sh writes the build record (MARK/SUMS below) inside the tree, so those two paths are never a
+# change to the series. Excluding them keeps this check for real drift only, and names the file.
+dirty="$(git -C "$SRC" status --porcelain --untracked-files=no -- . ':(exclude).build-products' ':(exclude).build-products.sha256' 2>/dev/null)"
+[ -n "$dirty" ] && { echo "$SRC has modified tracked files:"; printf '%s\n' "$dirty" | sed 's/^/      /'; exit 1; }
 TREE_HEAD="$(git -C "$SRC" rev-parse --short HEAD 2>/dev/null || echo unknown)"
 # The commit SHA of the same tree differs between machines: git am stamps the committer date of the
 # box that applied the patches, so the host commit and the container commit of one patch set are
@@ -405,10 +411,14 @@ import vllm._C_stable_libtorch, vllm._moe_C_stable_libtorch
 import vllm.vllm_flash_attn._vllm_fa2_C, vllm.vllm_flash_attn._vllm_fa3_C
 from vllm.tool_parsers import ToolParserManager
 ToolParserManager.get_tool_parser("qwen3_coder")
-print(vllm.__version__, vllm.__file__)')" \
+print("BUILD_SMOKE", vllm.__version__, vllm.__file__)' | tee /dev/stderr)" \
     || { echo "the installed tree does not import; see $build_log"; exit 1; }
-  [ "${smoke%% *}" = 0.30.0 ] || { echo "vllm reports ${smoke%% *}, expected 0.30.0"; exit 1; }
-  echo "    import check: vllm $smoke"
+  # vLLM writes its log lines to stdout too: a box with mixed GPUs (3090 + 3090 Ti) prints a WARNING
+  # there before the version. Read the version from the marker line, never from the first word.
+  line="$(printf '%s\n' "$smoke" | grep '^BUILD_SMOKE ' | tail -1)"
+  ver="${line#BUILD_SMOKE }"; ver="${ver%% *}"
+  [ "$ver" = 0.30.0 ] || { echo "the installed tree does not report 0.30.0, it printed:"; printf '%s\n' "$smoke" | tail -3 | sed 's/^/      /'; exit 1; }
+  echo "    import check: vllm $ver  $(printf '%s\n' "$line" | cut -d' ' -f3-)"
 else
   echo "    no CUDA driver here: the import check runs at serve time, on the GPU box"
   if command -v readelf >/dev/null; then
@@ -442,10 +452,17 @@ fi
 # so this tarball is the fast route for Docker: ARTIFACTS=artifacts/<name>.tar.gz, no compile.
 # The tag carries what the products were built for: sm86 (arch 8.6) + the CPython minor + the CUDA
 # major. A 3090 tarball does not work on a 5090; CUDA 13 products do not work with CUDA 12.
+# An artifacts set is current when the tarball, its per-file list and its manifest are all there, the
+# manifest names this tree, and the tarball still matches its own sha256 sidecar. Delete any of them
+# (or the whole folder) and this step writes the set again from the products already in the tree, so a
+# lost artifacts/ costs a pack (~45s), never a recompile.
+artifacts_current(){ local d="$1" n="$2" o
+  { [ -s "$d/$n.tar.gz" ] && [ -s "$d/$n.sha256" ] && [ -s "$d/$n.manifest" ] && [ -s "$d/$n.tar.gz.sha256" ]; } || return 1
+  [ "$(sed -n 's/^tree_obj=//p' "$d/$n.manifest" | head -1)" = "$TREE_OBJ" ] || return 1
+  o="$(sed -n 's/^ops_obj=//p' "$d/$n.manifest" | head -1)"; [ -z "$o" ] || [ "$o" = "$OPS_OBJ" ] || return 1
+  ( cd "$d" && sha256sum -c --quiet "$n.tar.gz.sha256" ) ; }
 if [ "$PACK" = 0 ]; then
   step 8 "pack the kernels for the image build (skipped: PACK=0 / --no-pack)"
-elif [ "$prebuilt" = 1 ]; then
-  step 8 "pack the kernels for the image build (skipped: these kernels came from a tarball)"
 else
   step 8 "pack the kernels for the image build"
   OUT="${OUT:-$HERE/artifacts}"; mkdir -p "$OUT"
@@ -453,10 +470,10 @@ else
   tv="$("$VENV/bin/python" -c 'import torch; print(torch.__version__.split("+")[0])')"
   tag="sm${ARCH/./}-py${pyv/./}-cu${nv_rel%%.*}"
   name="build-artifacts-$tag"
-  have_obj=""; [ -s "$OUT/$name.manifest" ] && have_obj="$(sed -n 's/^tree_obj=//p' "$OUT/$name.manifest" | head -1)"
-  if [ -s "$OUT/$name.tar.gz" ] && [ -s "$OUT/$name.sha256" ] && [ "$have_obj" = "$TREE_OBJ" ]; then
+  if artifacts_current "$OUT" "$name"; then
     echo "    artifacts/$name.tar.gz already holds tree ${TREE_OBJ:0:7}: not packed again"
   else
+    [ "$prebuilt" = 1 ] && echo "    these kernels came from a tarball, and artifacts/ holds no set for this tree: packing them back"
     files="$( cd "$SRC" && while read -r p; do case "$p" in ''|'#'*) continue ;; esac
              find "${p%/}" -type f; done < "$LIST" | LC_ALL=C sort )"
     n="$(printf '%s\n' "$files" | grep -c . || true)"