ci: pin the smoke image with a running container during the test
The label only protects against reclaims that honour it. The runner hosts also run jobs we do not control: runner-2 has a second act_runner registered with gitea.egl.lan on the same Docker daemon, and opencode-devbox's workflows run raw `docker system prune -af` in 7 places. No prune removes an image a running container uses, so the smoke step now starts a holder (`ci-reclaim-docker.sh hold`: --rm, --network none, sleep 3600) right after the preflight, and the always() cleanup step runs `release` (holder + image). Unprotected now: only the seconds between the build step's load and the holder starting. Verified on an isolated docker:29-dind: an unlabelled held image survives raw system prune and image prune -af (control without holder: deleted); hold fails on a missing image; holder is AutoRemove + network none; release removes holder and image; reclaim behaviour unchanged.
This commit is contained in:
@@ -7,13 +7,17 @@
|
||||
#
|
||||
# bash scripts/ci-reclaim-docker.sh # reclaim (default)
|
||||
# bash scripts/ci-reclaim-docker.sh assert-protected IMG # smoke preflight
|
||||
# bash scripts/ci-reclaim-docker.sh hold IMG # pin IMG while testing
|
||||
# bash scripts/ci-reclaim-docker.sh release IMG # drop the pin + IMG
|
||||
# bash scripts/ci-reclaim-docker.sh lint WORKFLOW... # lint.yml guard
|
||||
#
|
||||
# WHY THIS EXISTS. Jobs that land on the same runner share its Docker daemon
|
||||
# (runner-a1 runs several at once: in run 740 it had smoke-studio and
|
||||
# build-variant going together), and each job used to start with
|
||||
# `docker system prune -af --volumes`. That removes
|
||||
# every image no container is using at that instant, and a smoke image is
|
||||
# WHY THIS EXISTS. Everything that runs on one runner host shares its Docker
|
||||
# daemon: concurrent jobs of a runner with capacity > 1 (runner-a1 had
|
||||
# smoke-studio and build-variant going together in run 740), and the jobs of
|
||||
# other act_runner containers on the same VM (runner-2 has one registered with
|
||||
# gitea.jordbo.se and one with gitea.egl.lan). Each pi-devbox job used to start
|
||||
# with `docker system prune -af --volumes`. That removes every image no
|
||||
# container is using at that instant, and a smoke image is
|
||||
# exactly that for most of its life: smoke-test.sh runs one `docker run --rm`
|
||||
# per assertion, so between assertions nothing holds the image. v1.12.2 run 740
|
||||
# (2026-10-22): smoke passed, build-variant started and pruned, and its log
|
||||
@@ -23,8 +27,14 @@
|
||||
# locally", build-variant-studio was skipped, and the release went out without
|
||||
# its -studio tags. Job ORDER cannot close this: smoke and smoke-studio prune
|
||||
# each other too whenever their start times drift apart, and other runs share
|
||||
# the daemon. So the smoke images carry a label, and this script never prunes
|
||||
# an image that has it.
|
||||
# the daemon. Two defences, because we only control our own jobs:
|
||||
# - the smoke images carry a label, and this script never prunes an image
|
||||
# that has it (covers pi-devbox's own jobs, before and during the test);
|
||||
# - `hold` starts a container from the image for the whole smoke step. No
|
||||
# prune removes an image a running container uses, so this also covers
|
||||
# jobs that never call this script (opencode-devbox's raw prunes,
|
||||
# anything gitea.egl.lan runs). What is left is the few seconds between
|
||||
# the build step loading the image and `hold` starting.
|
||||
#
|
||||
# WHAT IT DOES. The same reclaim as `docker system prune -af --volumes` +
|
||||
# `docker builder prune -af`, done per object type so the image step can take
|
||||
@@ -86,6 +96,29 @@ assert_protected() {
|
||||
echo "OK: ${img} is present and carries ${SMOKE_IMAGE_LABEL}=true"
|
||||
}
|
||||
|
||||
holder_name() { printf 'ci-hold-%s' "$(printf '%s' "$1" | tr -c 'A-Za-z0-9_.-' '_')"; }
|
||||
|
||||
# The holder exits on its own after an hour (--rm), so a job killed before
|
||||
# `release` does not pin its image forever; the age sweep then removes it.
|
||||
hold() {
|
||||
local img="${1:?usage: $0 hold <image>}" name
|
||||
name=$(holder_name "$img")
|
||||
docker rm -f "$name" >/dev/null 2>&1 || true
|
||||
if ! docker run -d --rm --name "$name" --network none --entrypoint "" \
|
||||
"$img" sleep 3600 >/dev/null; then
|
||||
echo "::error::could not start holder ${name} for ${img}; is the image still in the daemon?"
|
||||
return 1
|
||||
fi
|
||||
echo "OK: ${img} pinned by running container ${name} (exits by itself after 1 h)"
|
||||
}
|
||||
|
||||
release() {
|
||||
local img="${1:?usage: $0 release <image>}"
|
||||
docker rm -f "$(holder_name "$img")" >/dev/null 2>&1 || true
|
||||
docker image rm "$img" >/dev/null 2>&1 || true
|
||||
echo "released ${img}"
|
||||
}
|
||||
|
||||
lint_workflows() {
|
||||
[ "$#" -gt 0 ] || { echo "usage: $0 lint <workflow.yml>..." >&2; return 2; }
|
||||
local hits
|
||||
@@ -101,6 +134,8 @@ lint_workflows() {
|
||||
case "${1:-reclaim}" in
|
||||
reclaim) reclaim ;;
|
||||
assert-protected) shift; assert_protected "$@" ;;
|
||||
hold) shift; hold "$@" ;;
|
||||
release) shift; release "$@" ;;
|
||||
lint) shift; lint_workflows "$@" ;;
|
||||
*) echo "usage: $0 [reclaim | assert-protected <image> | lint <workflow.yml>...]" >&2; exit 2 ;;
|
||||
*) echo "usage: $0 [reclaim | assert-protected <image> | hold <image> | release <image> | lint <workflow.yml>...]" >&2; exit 2 ;;
|
||||
esac
|
||||
|
||||
Reference in New Issue
Block a user