ci: stop disk reclaim from deleting a smoke image mid-test
Lint / skill-floor (push) Successful in 9s
Lint / hadolint (push) Successful in 15s
Lint / actionlint (push) Successful in 22s
Lint / doc-drift (push) Successful in 12s

v1.12.2 run 740: build-variant's `docker system prune -af --volumes`
untagged pi-devbox:smoke-studio 30 ms after smoke-studio loaded it, so
every studio assertion failed and the -studio tags were not published.
All jobs share the runner's Docker daemon, and a smoke image is unused
between smoke-test.sh's one-shot `docker run`s.

- scripts/ci-reclaim-docker.sh: the same reclaim, per object type, but
  never prunes an image labelled se.jordbo.pi-devbox.ci-smoke-image;
  labelled orphans are swept after 6h. `assert-protected IMG` is the
  smoke preflight; `lint WORKFLOW...` rejects raw system/image prunes.
- smoke jobs: label their image, assert it, tag it per run
  (pi-devbox:smoke[-studio]-<run_id>), remove it in an always() step.
- all five reclaim sites call the script; lint.yml runs the guard.
- Job order unchanged: studio jobs still gate only the -studio tags.

Verified on an isolated docker:29-dind (old prune deletes a labelled
image; new reclaim keeps it, removes unlabelled + 1970-created labelled
images, spares an in-use image, no errors); actionlint, shellcheck,
lint-shell, check-workflow-shell, check-doc-drift all clean.
This commit is contained in:
2026-10-23 08:34:39 +02:00
parent fb2cb2305c
commit 12e66aa52e
5 changed files with 193 additions and 14 deletions
+104
View File
@@ -0,0 +1,104 @@
#!/usr/bin/env bash
# ci-reclaim-docker.sh — free disk on the CI runner's Docker daemon WITHOUT
# deleting a smoke image that another job is still testing.
#
# Used by every docker-publish.yml job that reclaims disk (build-base, smoke,
# smoke-studio, build-variant, build-variant-studio). Not shipped in the image.
#
# bash scripts/ci-reclaim-docker.sh # reclaim (default)
# bash scripts/ci-reclaim-docker.sh assert-protected IMG # smoke preflight
# bash scripts/ci-reclaim-docker.sh lint WORKFLOW... # lint.yml guard
#
# WHY THIS EXISTS. The jobs share ONE Docker daemon (the runner's socket), and
# each used to start with `docker system prune -af --volumes`. That removes
# every image no container is using at that instant, and a smoke image is
# exactly that for most of its life: smoke-test.sh runs one `docker run --rm`
# per assertion, so between assertions nothing holds the image. v1.12.2 run 740
# (2026-10-22): smoke passed, build-variant started and pruned, and its log
# shows `untagged: pi-devbox:smoke-studio` at 22:58:49.11Z, 30 ms after
# smoke-studio had imported that image (22:58:49.08Z). Every smoke-studio
# assertion then failed with "Unable to find image 'pi-devbox:smoke-studio'
# locally", build-variant-studio was skipped, and the release went out without
# its -studio tags. Job ORDER cannot close this: smoke and smoke-studio prune
# each other too whenever their start times drift apart, and other runs share
# the daemon. So the smoke images carry a label, and this script never prunes
# an image that has it.
#
# WHAT IT DOES. The same reclaim as `docker system prune -af --volumes` +
# `docker builder prune -af`, done per object type so the image step can take
# a filter:
# 1. stopped containers (running ones are never touched)
# 2. unused images WITHOUT the label (the old `system prune -a` image step)
# 3. unused images WITH the label, older than SMOKE_IMAGE_MAX_AGE
# — orphans of a smoke job that died before its own cleanup step
# 4. unused networks, anonymous volumes (what `system prune --volumes` did)
# 5. the build cache
# Each step is `|| true`, as before: failing to reclaim disk must not fail the
# job. A filter the daemon rejected would show up as an error line in the log.
#
# `assert-protected IMG` is the smoke jobs' preflight: it fails unless IMG is in
# the daemon AND carries the label. Without it, a build that silently lost the
# label would look fine until the next time a sibling job's prune landed
# mid-test; and a missing image gets one clear error line instead of every
# assertion failing with "Unable to find image ... locally".
#
# `lint WORKFLOW...` fails if any workflow runs `docker system prune` or
# `docker image prune` itself. One such line in any job reopens the race, so
# lint.yml runs this on every push rather than relying on review.
#
# Environment:
# SMOKE_IMAGE_LABEL label key that protects an image (default below; the
# workflow sets it once in its top-level env)
# SMOKE_IMAGE_MAX_AGE `until=` value for the orphan sweep (default 6h; a
# smoke job takes ~10 min). Compared against the image's
# Created time, which is the build time: nothing here
# sets SOURCE_DATE_EPOCH.
set -u
SMOKE_IMAGE_LABEL="${SMOKE_IMAGE_LABEL:-se.jordbo.pi-devbox.ci-smoke-image}"
SMOKE_IMAGE_MAX_AGE="${SMOKE_IMAGE_MAX_AGE:-6h}"
step() { printf '+ %s\n' "$*"; "$@" || true; }
reclaim() {
step docker container prune -f
step docker image prune -af --filter "label!=${SMOKE_IMAGE_LABEL}"
step docker image prune -af --filter "label=${SMOKE_IMAGE_LABEL}" \
--filter "until=${SMOKE_IMAGE_MAX_AGE}"
step docker network prune -f
step docker volume prune -f
step docker builder prune -af
}
assert_protected() {
local img="${1:?usage: $0 assert-protected <image>}" val
if ! val=$(docker image inspect \
--format "{{ index .Config.Labels \"${SMOKE_IMAGE_LABEL}\" }}" "$img" 2>&1); then
echo "::error::${img} is not in the runner's Docker daemon (${val}). Was it removed by a concurrent prune that does not honour ${SMOKE_IMAGE_LABEL}?"
return 1
fi
if [ "$val" != "true" ]; then
echo "::error::${img} lacks the label ${SMOKE_IMAGE_LABEL}=true (got '${val}'), so a sibling job's reclaim could delete it mid-test. Check the build step's labels: input."
return 1
fi
echo "OK: ${img} is present and carries ${SMOKE_IMAGE_LABEL}=true"
}
lint_workflows() {
[ "$#" -gt 0 ] || { echo "usage: $0 lint <workflow.yml>..." >&2; return 2; }
local hits
hits=$(grep -nE '^[^#]*docker[[:space:]]+(system|image)[[:space:]]+prune' "$@" || true)
if [ -n "$hits" ]; then
echo "::error::raw docker prune in a workflow. It deletes smoke images other jobs are testing on the shared runner daemon; call 'bash scripts/ci-reclaim-docker.sh' instead."
printf '%s\n' "$hits"
return 1
fi
echo "OK: no raw 'docker system|image prune' in $# workflow file(s)"
}
case "${1:-reclaim}" in
reclaim) reclaim ;;
assert-protected) shift; assert_protected "$@" ;;
lint) shift; lint_workflows "$@" ;;
*) echo "usage: $0 [reclaim | assert-protected <image> | lint <workflow.yml>...]" >&2; exit 2 ;;
esac