ci: pin the smoke image with a running container during the test
Lint / hadolint (push) Successful in 12s
Lint / skill-floor (push) Successful in 13s
Lint / doc-drift (push) Successful in 17s
Lint / actionlint (push) Successful in 26s

The label only protects against reclaims that honour it. The runner
hosts also run jobs we do not control: runner-2 has a second act_runner
registered with gitea.egl.lan on the same Docker daemon, and
opencode-devbox's workflows run raw `docker system prune -af` in 7
places. No prune removes an image a running container uses, so the
smoke step now starts a holder (`ci-reclaim-docker.sh hold`: --rm,
--network none, sleep 3600) right after the preflight, and the
always() cleanup step runs `release` (holder + image). Unprotected now:
only the seconds between the build step's load and the holder starting.

Verified on an isolated docker:29-dind: an unlabelled held image survives
raw system prune and image prune -af (control without holder: deleted);
hold fails on a missing image; holder is AutoRemove + network none;
release removes holder and image; reclaim behaviour unchanged.
This commit is contained in:
2026-10-23 12:46:39 +02:00
parent e0611c787f
commit 779efbf4fe
5 changed files with 86 additions and 27 deletions
+18 -8
View File
@@ -658,13 +658,18 @@ jobs:
EXPECTED_NODE_MAJOR=$(sed -n 's/^ARG NODE_VERSION=\([0-9][0-9]*\).*/\1/p' Dockerfile.base) EXPECTED_NODE_MAJOR=$(sed -n 's/^ARG NODE_VERSION=\([0-9][0-9]*\).*/\1/p' Dockerfile.base)
export EXPECTED_NODE_MAJOR export EXPECTED_NODE_MAJOR
bash scripts/ci-reclaim-docker.sh assert-protected "$SMOKE_IMAGE" bash scripts/ci-reclaim-docker.sh assert-protected "$SMOKE_IMAGE"
# Pin the image with a running container for the whole test, so a
# raw prune from a job we do not control (other repos, the other
# Gitea instance's runner on the same VM) cannot remove it either.
bash scripts/ci-reclaim-docker.sh hold "$SMOKE_IMAGE"
bash scripts/smoke-test.sh "$SMOKE_IMAGE" bash scripts/smoke-test.sh "$SMOKE_IMAGE"
# The label keeps this image from every job's reclaim, so it is removed # The label keeps this image from every job's reclaim and the holder pins
# here instead. A job killed before this step leaves an orphan, which the # it, so both are removed here. A job killed before this step leaves an
# reclaim's age sweep deletes once it is older than SMOKE_IMAGE_MAX_AGE. # orphan: the holder exits after 1 h, and the reclaim's age sweep deletes
# the image once it is older than SMOKE_IMAGE_MAX_AGE.
- name: Remove smoke image - name: Remove smoke image
if: always() if: always()
run: docker image rm "$SMOKE_IMAGE" || true run: bash scripts/ci-reclaim-docker.sh release "$SMOKE_IMAGE"
# ── Phase 3b: amd64 smoke for the studio variant ──────────────────── # ── Phase 3b: amd64 smoke for the studio variant ────────────────────
# Additive + independent of the core `smoke` job: gates ONLY # Additive + independent of the core `smoke` job: gates ONLY
@@ -737,13 +742,18 @@ jobs:
EXPECTED_NODE_MAJOR=$(sed -n 's/^ARG NODE_VERSION=\([0-9][0-9]*\).*/\1/p' Dockerfile.base) EXPECTED_NODE_MAJOR=$(sed -n 's/^ARG NODE_VERSION=\([0-9][0-9]*\).*/\1/p' Dockerfile.base)
export EXPECTED_NODE_MAJOR export EXPECTED_NODE_MAJOR
bash scripts/ci-reclaim-docker.sh assert-protected "$SMOKE_IMAGE" bash scripts/ci-reclaim-docker.sh assert-protected "$SMOKE_IMAGE"
# Pin the image with a running container for the whole test, so a
# raw prune from a job we do not control (other repos, the other
# Gitea instance's runner on the same VM) cannot remove it either.
bash scripts/ci-reclaim-docker.sh hold "$SMOKE_IMAGE"
bash scripts/smoke-test.sh "$SMOKE_IMAGE" bash scripts/smoke-test.sh "$SMOKE_IMAGE"
# The label keeps this image from every job's reclaim, so it is removed # The label keeps this image from every job's reclaim and the holder pins
# here instead. A job killed before this step leaves an orphan, which the # it, so both are removed here. A job killed before this step leaves an
# reclaim's age sweep deletes once it is older than SMOKE_IMAGE_MAX_AGE. # orphan: the holder exits after 1 h, and the reclaim's age sweep deletes
# the image once it is older than SMOKE_IMAGE_MAX_AGE.
- name: Remove smoke image - name: Remove smoke image
if: always() if: always()
run: docker image rm "$SMOKE_IMAGE" || true run: bash scripts/ci-reclaim-docker.sh release "$SMOKE_IMAGE"
# ── Phase 4: multi-arch publish ───────────────────────────────────── # ── Phase 4: multi-arch publish ─────────────────────────────────────
build-variant: build-variant:
+2 -2
View File
@@ -91,8 +91,8 @@ jobs:
run: bash scripts/check-workflow-shell.sh .gitea/workflows run: bash scripts/check-workflow-shell.sh .gitea/workflows
- name: No raw docker prune in workflows (shared runner daemon) - name: No raw docker prune in workflows (shared runner daemon)
# Jobs on the same runner share its Docker daemon (runner-a1 runs # Everything on one runner host shares its Docker daemon (runner-a1
# several docker-publish.yml jobs at once), and # runs several jobs at once; runner-2 also serves gitea.egl.lan), and
# a bare `docker system prune -a` in one job deletes the smoke image # a bare `docker system prune -a` in one job deletes the smoke image
# another job is testing (v1.12.2 run 740 shipped without its -studio # another job is testing (v1.12.2 run 740 shipped without its -studio
# tags that way). Reclaim goes through scripts/ci-reclaim-docker.sh, # tags that way). Reclaim goes through scripts/ci-reclaim-docker.sh,
+6 -3
View File
@@ -45,12 +45,15 @@ re-brand of opencode-devbox's `pi-only` variant.
update-description). The `-studio` variant adds independent update-description). The `-studio` variant adds independent
`smoke-studio` + `build-variant-studio` jobs that gate only the `smoke-studio` + `build-variant-studio` jobs that gate only the
`-studio` tags (never the core `:latest` release). `-studio` tags (never the core `:latest` release).
- `scripts/ci-reclaim-docker.sh` — the ONLY way a CI job frees disk. Jobs on - `scripts/ci-reclaim-docker.sh` — the ONLY way a CI job frees disk. Everything
the same runner share its Docker daemon (runner-a1 runs several at once), so a bare `docker system prune -a` in one on one runner host shares its Docker daemon (runner-a1 runs several jobs at
once; runner-2 also hosts a runner for gitea.egl.lan), so a bare `docker system prune -a` in one
job deletes the smoke image another job is mid-test on (v1.12.2 run 740 lost job deletes the smoke image another job is mid-test on (v1.12.2 run 740 lost
its `-studio` tags that way). The script spares images labelled its `-studio` tags that way). The script spares images labelled
`se.jordbo.pi-devbox.ci-smoke-image`; the smoke jobs apply that label, `se.jordbo.pi-devbox.ci-smoke-image`; the smoke jobs apply that label,
assert it (`assert-protected`) and remove their own image at the end. assert it (`assert-protected`), pin the image with a running container for
the test (`hold`, against raw prunes from jobs we do not control) and
`release` it at the end.
`lint.yml` fails on any raw `docker system|image prune` in a workflow. `lint.yml` fails on any raw `docker system|image prune` in a workflow.
## Versioning scheme ## Versioning scheme
+17 -6
View File
@@ -17,9 +17,11 @@ Pre-v1.0.0 tags followed the pi npm version (`v{pi_version}[letter]`).
- **CI: a smoke job's image could be deleted mid-test by another job's disk - **CI: a smoke job's image could be deleted mid-test by another job's disk
reclaim.** This is why v1.12.2's `-studio` tags were not published by its reclaim.** This is why v1.12.2's `-studio` tags were not published by its
tag run. Jobs that land on the same runner share its Docker daemon (in run tag run. Everything on one runner host shares its Docker daemon: concurrent
740, runner-a1 ran `smoke-studio` and `build-variant` at the same time), and jobs of a runner with capacity > 1 (in run 740, runner-a1 ran `smoke-studio`
each job began with `docker system prune -af --volumes`, which removes every and `build-variant` at the same time), and the jobs of other act_runner
containers on the same VM (runner-2 has one for gitea.jordbo.se and one for
gitea.egl.lan). Each pi-devbox job began with `docker system prune -af --volumes`, which removes every
image that no container is using at that moment. A smoke image is unused image that no container is using at that moment. A smoke image is unused
between `smoke-test.sh`'s one-shot `docker run`s. In run 740, `smoke` between `smoke-test.sh`'s one-shot `docker run`s. In run 740, `smoke`
passed and `build-variant` started. Its prune logged passed and `build-variant` started. Its prune logged
@@ -33,8 +35,14 @@ Pre-v1.0.0 tags followed the pi npm version (`v{pi_version}[letter]`).
volumes, build cache) but never prunes an image labelled volumes, build cache) but never prunes an image labelled
`se.jordbo.pi-devbox.ci-smoke-image`. The smoke builds apply that label, a `se.jordbo.pi-devbox.ci-smoke-image`. The smoke builds apply that label, a
preflight fails if it is missing, and each smoke job removes its own image at preflight fails if it is missing, and each smoke job removes its own image at
the end. Smoke images left by a killed job are swept once they are older the end. Jobs we do not control can still run a raw prune on the same
than 6 h. Smoke tags carry the run id (`pi-devbox:smoke-<run_id>`), so two daemon: opencode-devbox's workflows do, in 7 places, and so may whatever
gitea.egl.lan runs. So the smoke step also pins its image with a running
container (`ci-reclaim-docker.sh hold`) for the whole test, because no prune
removes an image a running container uses. Only the few seconds between the
build step loading the image and the pin starting are left unprotected. The
holder exits by itself after 1 h, and smoke images left by a killed job are
swept once they are older than 6 h. Smoke tags carry the run id (`pi-devbox:smoke-<run_id>`), so two
runs on different refs cannot test or delete each other's image. `lint.yml` runs on different refs cannot test or delete each other's image. `lint.yml`
fails on any raw `docker system|image prune` in a workflow. Job order is fails on any raw `docker system|image prune` in a workflow. Job order is
unchanged: the studio jobs still gate only the `-studio` tags. Reordering unchanged: the studio jobs still gate only the `-studio` tags. Reordering
@@ -45,7 +53,10 @@ Pre-v1.0.0 tags followed the pi npm version (`v{pi_version}[letter]`).
fresh labelled image, deletes an unlabelled one and a labelled one created fresh labelled image, deletes an unlabelled one and a labelled one created
in 1970, keeps an image a running container uses, and prints no errors. in 1970, keeps an image a running container uses, and prints no errors.
`assert-protected` passes the labelled image and fails both an unlabelled `assert-protected` passes the labelled image and fails both an unlabelled
and a missing one. and a missing one. With `hold`, an unlabelled image survives both a raw
`docker system prune -af --volumes` and `docker image prune -af` (control:
without the holder, the same prune deletes it). `hold` fails on a missing
image, and `release` removes the holder and the image.
## v1.12.2 — 2026-10-22 ## v1.12.2 — 2026-10-22
+43 -8
View File
@@ -7,13 +7,17 @@
# #
# bash scripts/ci-reclaim-docker.sh # reclaim (default) # bash scripts/ci-reclaim-docker.sh # reclaim (default)
# bash scripts/ci-reclaim-docker.sh assert-protected IMG # smoke preflight # bash scripts/ci-reclaim-docker.sh assert-protected IMG # smoke preflight
# bash scripts/ci-reclaim-docker.sh hold IMG # pin IMG while testing
# bash scripts/ci-reclaim-docker.sh release IMG # drop the pin + IMG
# bash scripts/ci-reclaim-docker.sh lint WORKFLOW... # lint.yml guard # bash scripts/ci-reclaim-docker.sh lint WORKFLOW... # lint.yml guard
# #
# WHY THIS EXISTS. Jobs that land on the same runner share its Docker daemon # WHY THIS EXISTS. Everything that runs on one runner host shares its Docker
# (runner-a1 runs several at once: in run 740 it had smoke-studio and # daemon: concurrent jobs of a runner with capacity > 1 (runner-a1 had
# build-variant going together), and each job used to start with # smoke-studio and build-variant going together in run 740), and the jobs of
# `docker system prune -af --volumes`. That removes # other act_runner containers on the same VM (runner-2 has one registered with
# every image no container is using at that instant, and a smoke image is # gitea.jordbo.se and one with gitea.egl.lan). Each pi-devbox job used to start
# with `docker system prune -af --volumes`. That removes every image no
# container is using at that instant, and a smoke image is
# exactly that for most of its life: smoke-test.sh runs one `docker run --rm` # exactly that for most of its life: smoke-test.sh runs one `docker run --rm`
# per assertion, so between assertions nothing holds the image. v1.12.2 run 740 # per assertion, so between assertions nothing holds the image. v1.12.2 run 740
# (2026-10-22): smoke passed, build-variant started and pruned, and its log # (2026-10-22): smoke passed, build-variant started and pruned, and its log
@@ -23,8 +27,14 @@
# locally", build-variant-studio was skipped, and the release went out without # locally", build-variant-studio was skipped, and the release went out without
# its -studio tags. Job ORDER cannot close this: smoke and smoke-studio prune # its -studio tags. Job ORDER cannot close this: smoke and smoke-studio prune
# each other too whenever their start times drift apart, and other runs share # each other too whenever their start times drift apart, and other runs share
# the daemon. So the smoke images carry a label, and this script never prunes # the daemon. Two defences, because we only control our own jobs:
# an image that has it. # - the smoke images carry a label, and this script never prunes an image
# that has it (covers pi-devbox's own jobs, before and during the test);
# - `hold` starts a container from the image for the whole smoke step. No
# prune removes an image a running container uses, so this also covers
# jobs that never call this script (opencode-devbox's raw prunes,
# anything gitea.egl.lan runs). What is left is the few seconds between
# the build step loading the image and `hold` starting.
# #
# WHAT IT DOES. The same reclaim as `docker system prune -af --volumes` + # WHAT IT DOES. The same reclaim as `docker system prune -af --volumes` +
# `docker builder prune -af`, done per object type so the image step can take # `docker builder prune -af`, done per object type so the image step can take
@@ -86,6 +96,29 @@ assert_protected() {
echo "OK: ${img} is present and carries ${SMOKE_IMAGE_LABEL}=true" echo "OK: ${img} is present and carries ${SMOKE_IMAGE_LABEL}=true"
} }
holder_name() { printf 'ci-hold-%s' "$(printf '%s' "$1" | tr -c 'A-Za-z0-9_.-' '_')"; }
# The holder exits on its own after an hour (--rm), so a job killed before
# `release` does not pin its image forever; the age sweep then removes it.
hold() {
local img="${1:?usage: $0 hold <image>}" name
name=$(holder_name "$img")
docker rm -f "$name" >/dev/null 2>&1 || true
if ! docker run -d --rm --name "$name" --network none --entrypoint "" \
"$img" sleep 3600 >/dev/null; then
echo "::error::could not start holder ${name} for ${img}; is the image still in the daemon?"
return 1
fi
echo "OK: ${img} pinned by running container ${name} (exits by itself after 1 h)"
}
release() {
local img="${1:?usage: $0 release <image>}"
docker rm -f "$(holder_name "$img")" >/dev/null 2>&1 || true
docker image rm "$img" >/dev/null 2>&1 || true
echo "released ${img}"
}
lint_workflows() { lint_workflows() {
[ "$#" -gt 0 ] || { echo "usage: $0 lint <workflow.yml>..." >&2; return 2; } [ "$#" -gt 0 ] || { echo "usage: $0 lint <workflow.yml>..." >&2; return 2; }
local hits local hits
@@ -101,6 +134,8 @@ lint_workflows() {
case "${1:-reclaim}" in case "${1:-reclaim}" in
reclaim) reclaim ;; reclaim) reclaim ;;
assert-protected) shift; assert_protected "$@" ;; assert-protected) shift; assert_protected "$@" ;;
hold) shift; hold "$@" ;;
release) shift; release "$@" ;;
lint) shift; lint_workflows "$@" ;; lint) shift; lint_workflows "$@" ;;
*) echo "usage: $0 [reclaim | assert-protected <image> | lint <workflow.yml>...]" >&2; exit 2 ;; *) echo "usage: $0 [reclaim | assert-protected <image> | hold <image> | release <image> | lint <workflow.yml>...]" >&2; exit 2 ;;
esac esac