Files
pi-devbox/scripts/check-doc-drift.sh
T
joakimp b8d818ed99
Lint / hadolint (push) Successful in 9s
Lint / doc-drift (push) Successful in 13s
Lint / skill-floor (push) Successful in 15s
Lint / actionlint (push) Successful in 18s
fix(docs): correct the Hub size claims, and gate them so they cannot rot again
DOCKER_HUB.md claimed ~1.1 GB for :latest while Docker Hub served 1.37 GB --
20% wrong, drifting quietly across eight releases, and POSTed to Docker Hub by
update-description every time. It is the first number a stranger reads about
this image.

Root cause is structural, not carelessness: every OTHER claim check-doc-drift.sh
guards is anchored to a build file (a pin, an ARG, a placeholder), so it cannot
rot without someone editing the thing it describes. Nothing in this repo states
the image size, so nothing measured it.

Corrected against measured full_size after v1.9.2 published (amd64/arm64):
  :latest         ~1.1  -> ~1.23 GB   (1.228 / 1.211)
  :latest-studio  ~1.15 -> ~1.25 GB   (1.255 / 1.238)
  :base-latest    ~1.0  -> ~1.17 GB   (1.167 / 1.151)
full_size tracks the FIRST manifest entry (amd64), not the sum across arches --
measured: full_size=1.228, amd64=1.228, arm64=1.211, sum=2.439 -- which is what
the table's per-arch "Size (compressed)" column claims.

New check 8 gates those claims against Hub. Fails on drift; SKIPS LOUDLY via a
new skip() helper (counted, named in the summary) when curl/python3 are absent,
the API is unreachable, or SKIP_SIZE_CHECK=1. A skip is deliberately neither OK
nor a failure: printing an unverified claim as OK is the habit this file exists
to break, and failing on Docker Hub's uptime would hold releases hostage to a
third party. Not wired into hooks/pre-push, so pushes stay offline.

Two bugs caught by writing the expected exit code down before running the check:

  1. The tolerance would have missed its own motivating case. The percentage is
     computed against the MEASURED size, but the 20% first chosen came from the
     claim-relative figure; the real drift was |1.1-1.37|/1.37 = 19.7% and would
     have passed. Now 15%, inside a window with both bounds measured: above the
     largest legitimate skew (11.4%, a claim describing the published release
     while the next tag changes the size) and below the rot it must catch.

  2. A `|| true` on the python invocation made the gate FAIL OPEN -- it printed
     DRIFT and exited 0. Removed; the outer `|| SIZE_RC=$?` satisfies set -e
     without swallowing the code. Verified two-sided: 19% drift exits 1 at the
     default tolerance and 0 at SIZE_TOLERANCE_PCT=25, so the threshold does the
     work rather than the ordering.

NOT covered, and the script says so where a reader will see it: README.md's
~3.2 GB figures are UNCOMPRESSED and the registry exposes compressed sizes only,
so measuring them needs a real pull. A green check 8 says nothing about them.

Verified: check-doc-drift.sh rc=0 on the corrected tree, rc=1 on injected drift,
rc=0 with --warn-only, SKIP+rc=0 on the offline path (proxy to a dead port);
shellcheck clean at default severity (the sed single-quote SC2016 carries a
disable with its reason); lint-shell.sh 16 files clean.
2026-09-14 20:33:40 +02:00

394 lines
18 KiB
Bash
Executable File

#!/usr/bin/env bash
# check-doc-drift.sh — fail when a hand-maintained doc claim contradicts the
# build files it describes.
#
# THE DEFECT CLASS THIS EXISTS TO CATCH, measured 2026-09-10 while preparing
# v1.9.0. Five separate claims had rotted, all of them the same shape: a fact
# written once by hand, in a file nothing verifies, about a value that lives
# somewhere else and moved.
#
# 1..3. README.md's "Version pins" table was wrong on EVERY row — pi `0.84.4`
# vs ARG PI_VERSION=0.85.1, pi-atelier `v0.10.0` vs v0.10.1, mempalace
# `3.8.0` vs 3.9.0. That table is the worst possible place for this: it
# exists precisely to be the reviewable record of what is deliberately
# frozen, so when it lies, the review it enables is worthless.
# 4. README.md carried a "Planned for an upcoming minor release" section
# listing typst PDF export, which had ALREADY SHIPPED, tagged with a
# self-contradicting "(shipped in Unreleased/base)" marker. The
# CHANGELOG had already documented three earlier instances of exactly
# this stale-"Unreleased"-pointer class (see its v1.8.7 notes).
# 5. DOCKER_HUB.md claimed "Node.js v22" while this release ships Node 24.
# This one is the reason the gate exists at all: DOCKER_HUB.md is
# PUBLISHED. `update-description` in docker-publish.yml POSTs it to Hub
# as full_description on every tag, so unlike README.md — which no
# workflow or gate reads — a stale claim here is what users see.
#
# WHY A GATE AND NOT "REMEMBER TO CHECK". DOCKER_HUB.md had gone eight releases
# (v1.8.6 → v1.9.0) without a touch. Nothing generates it and nothing verifies
# it; the only mechanism keeping it true was whoever remembered. That is the
# same failure mode check-skill-floor.sh was written for, and the same fix:
# convert "someone remembers" into "CI refuses".
#
# WHY THESE FIVE CHECKS AND NOT MORE. Every check here compares a doc string to
# a value that EXISTS IN THIS REPO, so it can never be wrong about the world and
# needs no network, no token, and no built image. Claims that require a running
# container to verify (image sizes, the "N mempalace_* tools" count) are
# deliberately NOT gated: a check that cannot be evaluated honestly at lint time
# would either be skipped or guessed, and a guessing gate is worse than none.
# If you want those, assert them in scripts/smoke-test.sh where a real image is
# available.
#
# DELIBERATELY NOT GATED: Dockerfile.base's `# BASE_REBUILD_DATE:` comment, which
# is also stale (2026-07-13, three base rebuilds ago). base_tag is a hash of
# Dockerfile.base's CONTENT plus rootfs/, comments included, so a gate that
# demanded that comment be current would force a ~60 min base rebuild on any
# release that touched no base files at all. Fix it when you are already
# rebuilding the base — then it is free. This is a real cost asymmetry, not
# laziness.
#
# EXIT CODES (same contract as lint-shell.sh and check-skill-floor.sh):
# 0 every checked claim matches
# 1 at least one claim has drifted
# 2 cannot run (a file or ARG this gate reads is missing/unparseable)
# A gate that cannot run must not pass, so a missing input is 2, never 0.
set -euo pipefail
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$REPO_ROOT"
README="README.md"
HUB="DOCKER_HUB.md"
DF_VARIANT="Dockerfile.variant"
DF_BASE="Dockerfile.base"
# Docker Hub rejects a full_description longer than this. docker-publish.yml has
# no size check of its own; it only notices via a non-200 from the API, i.e.
# after paying the whole build. Catching it here makes it a 2-second failure.
HUB_MAX_CHARS=25000
WARN_ONLY=0
FAILURES=0
SKIPS=0
# Tolerance for the published size claims (check 8), as a percentage OF THE
# MEASURED SIZE. The denominator matters: against the claim instead, the same
# drift reads as a different number, and an early draft of this gate took 20%
# from the claim-relative figure and would therefore have MISSED its own
# motivating case. Both bounds are measured, not guessed:
# - the rot that motivated this check: claimed 1.1 GB vs measured 1.37 GB
# = 19.7% off, so the threshold must sit BELOW that or the gate is theatre.
# - the largest legitimate skew, i.e. a claim describing the currently-published
# release while the next tag changes the size: v1.9.1's 1.37 GB against
# v1.9.2's measured 1.23 GB = 11.4% off, so the threshold must sit ABOVE that
# or every size-changing release trips it.
# 15% sits in that 11.4%-19.7% window. Widen it only with a measured reason, and
# re-derive both bounds if you do.
SIZE_TOLERANCE_PCT="${SIZE_TOLERANCE_PCT:-15}"
usage() {
cat <<'EOF'
Usage: check-doc-drift.sh [--warn-only] [-h|--help]
Compares hand-written claims in README.md and DOCKER_HUB.md against the build
files they describe (Dockerfile.base, Dockerfile.variant).
--warn-only Report drift but exit 0 (advisory use, e.g. a local pre-push hook).
Exit: 0 = in sync, 1 = drift, 2 = cannot run.
EOF
}
while [ $# -gt 0 ]; do
case "$1" in
--warn-only) WARN_ONLY=1; shift ;;
-h|--help) usage; exit 0 ;;
*) echo "::error::unknown argument: $1" >&2; usage >&2; exit 2 ;;
esac
done
for f in "$README" "$HUB" "$DF_VARIANT" "$DF_BASE"; do
if [ ! -f "$f" ]; then
echo "::error::$f not found (cwd $PWD). Cannot evaluate doc drift, so this is exit 2, not a pass."
exit 2
fi
done
# Read `ARG NAME=value` from a Dockerfile. Exit 2 when absent: if the ARG this
# gate is built around has been renamed, the gate is measuring nothing and must
# say so rather than silently comparing against an empty string.
read_arg() {
local file="$1" name="$2" value
value="$(sed -n "s/^ARG ${name}=\\(.*\\)\$/\\1/p" "$file" | head -1)"
if [ -z "$value" ]; then
echo "::error::ARG ${name} not found in ${file}. It was probably renamed;" >&2
echo "::error::update check-doc-drift.sh to match, because this gate is now blind." >&2
exit 2
fi
printf '%s' "$value"
}
# One row of README's "Version pins" table: `| pi | `0.85.1` | ... |`
read_pin_row() {
sed -n "s/^| $1 | \`\\([^\`]*\`*\\)\` |.*/\\1/p" "$README" | head -1
}
fail() {
FAILURES=$((FAILURES + 1))
echo "::error::$1"
}
ok() { printf ' OK %s\n' "$1"; }
# A check that could not be EVALUATED, as distinct from one that passed.
# Deliberately neither ok() nor fail(): printing it as OK would launder an
# unmeasured claim into a passing one (the exact habit this file exists to
# break), while failing on a third party's uptime would make every release
# hostage to Docker Hub's API. Loud, counted, and surfaced in the summary.
skip() { SKIPS=$((SKIPS + 1)); printf ' SKIP %s\n' "$1"; }
echo "Checking hand-maintained doc claims against the build files they describe."
echo
# ---------------------------------------------------------------------------
# 1-3. README's version-pin table vs the ARGs it names by name.
# ---------------------------------------------------------------------------
check_pin() {
local label="$1" documented="$2" actual="$3" where="$4"
if [ -z "$documented" ]; then
fail "README.md: no '| $label |' row found in the version-pin table. Either the
table was restructured (update this gate) or the row was dropped (restore it)."
return
fi
if [ "$documented" != "$actual" ]; then
fail "README.md version-pin table is stale for $label: says '$documented',
$where says '$actual'. Fix the table — it is the reviewable record of what
this repo deliberately freezes, so a wrong row defeats its only purpose."
return
fi
ok "README pin $label = $actual"
}
PI_ACTUAL="$(read_arg "$DF_VARIANT" PI_VERSION)"
ATELIER_ACTUAL="$(read_arg "$DF_VARIANT" PI_ATELIER_REF)"
MEMPALACE_ACTUAL="$(read_arg "$DF_BASE" MEMPALACE_VERSION)"
check_pin pi "$(read_pin_row pi)" "$PI_ACTUAL" "ARG PI_VERSION in $DF_VARIANT"
check_pin pi-atelier "$(read_pin_row pi-atelier)" "$ATELIER_ACTUAL" "ARG PI_ATELIER_REF in $DF_VARIANT"
check_pin mempalace "$(read_pin_row mempalace)" "$MEMPALACE_ACTUAL" "ARG MEMPALACE_VERSION in $DF_BASE"
# ---------------------------------------------------------------------------
# 4. DOCKER_HUB.md's Node claim vs ARG NODE_VERSION. This is the published page,
# so it is the one whose staleness reaches users.
# ---------------------------------------------------------------------------
NODE_ACTUAL="$(read_arg "$DF_BASE" NODE_VERSION)"
NODE_DOCUMENTED="$(sed -n 's/.*\*\*Node\.js\*\* v\([0-9][0-9]*\).*/\1/p' "$HUB" | head -1)"
if [ -z "$NODE_DOCUMENTED" ]; then
fail "$HUB: could not find a '**Node.js** vNN' claim. If the wording changed,
update this gate; do not leave the published page unverified."
elif [ "$NODE_DOCUMENTED" != "$NODE_ACTUAL" ]; then
fail "$HUB claims Node v$NODE_DOCUMENTED but ARG NODE_VERSION=$NODE_ACTUAL.
This file is PUBLISHED to Docker Hub by update-description on every tag,
and it is read from the TAG — so fix it before tagging, not after."
else
ok "$HUB Node claim = v$NODE_ACTUAL"
fi
# ---------------------------------------------------------------------------
# 5. Placeholders CI will not substitute. docker-publish.yml substitutes exactly
# {{PI_VERSION}} and then greps for leftovers of that ONE token, so any other
# {{...}} sails through the guard and is published literally.
# ---------------------------------------------------------------------------
UNKNOWN_PLACEHOLDERS="$(grep -o '{{[A-Za-z0-9_]*}}' "$HUB" | sort -u | grep -v '^{{PI_VERSION}}$' || true)"
if [ -n "$UNKNOWN_PLACEHOLDERS" ]; then
fail "$HUB contains placeholders CI does not substitute, which would be
published verbatim: $(echo "$UNKNOWN_PLACEHOLDERS" | tr '\n' ' ')
docker-publish.yml only fills {{PI_VERSION}}; add substitution there first."
else
ok "$HUB has no placeholders beyond {{PI_VERSION}}"
fi
# Match only the UPPER_SNAKE placeholder convention CI uses. A bare '{{' search
# is WRONG here, and the first version of this check proved it by failing on
# README.md:900 — `docker inspect --format '{{json .Config.Labels}}'`, a Go
# template in a legitimate example, not a placeholder. The gate was wrong, not
# the doc. Keep this anchored to [A-Z] so Go/Jinja/Handlebars examples pass.
README_PLACEHOLDERS="$(grep -o '{{[A-Z][A-Z0-9_]*}}' "$README" | sort -u || true)"
if [ -n "$README_PLACEHOLDERS" ]; then
fail "$README contains placeholder(s) nothing substitutes, so they would render
literally for every reader: $(echo "$README_PLACEHOLDERS" | tr '\n' ' ')
Only DOCKER_HUB.md gets substitution, and only for {{PI_VERSION}}."
else
ok "$README has no unsubstituted placeholders"
fi
# ---------------------------------------------------------------------------
# 6. Hub full_description length.
# ---------------------------------------------------------------------------
HUB_CHARS="$(wc -c < "$HUB" | tr -d ' ')"
if [ "$HUB_CHARS" -gt "$HUB_MAX_CHARS" ]; then
fail "$HUB is $HUB_CHARS chars, over Docker Hub's $HUB_MAX_CHARS-char
full_description limit. update-description would fail with a non-200 AFTER
the full build. Trim it — this file is the essentials-only page, and
README.md is the long form on purpose."
else
ok "$HUB is $HUB_CHARS chars (limit $HUB_MAX_CHARS)"
fi
# ---------------------------------------------------------------------------
# 7. Stale "Unreleased" pointers. "Unreleased" is a CHANGELOG-only concept; in
# a user-facing doc it is always a pointer that outlived what it pointed at.
# This class has now bitten five times, hence a gate rather than vigilance.
# ---------------------------------------------------------------------------
STALE_MARKERS="$(grep -n 'Unreleased' "$README" "$HUB" || true)"
if [ -n "$STALE_MARKERS" ]; then
fail "'Unreleased' appears in a user-facing doc, which is always a stale
pointer once the thing ships (it has happened five times here):
${STALE_MARKERS//$'\n'/$'\n' }
State the fact directly, or move it to CHANGELOG.md where 'Unreleased' means something."
else
ok "no stale 'Unreleased' pointers in $README or $HUB"
fi
# ---------------------------------------------------------------------------
# 8. Published size claims vs Docker Hub's MEASURED full_size.
#
# Why this exists: every other claim in these docs is checked against a file
# in this repo, so it cannot rot without someone editing the thing it
# describes. The size claims had no such anchor -- nothing in the repo states
# the image size -- so they quietly went 24% wrong across eight releases
# (DOCKER_HUB.md said ~1.1 GB; :latest measured 1.37 GB on 2026-09-14).
# DOCKER_HUB.md is POSTed to Docker Hub by update-description, so that number
# is the first thing a stranger reads about this image.
#
# Hub's `full_size` tracks the FIRST manifest entry (amd64 here), NOT the sum
# across architectures -- measured: v1.9.2 full_size=1.228 GB, amd64=1.228,
# arm64=1.211, sum=2.439. That matches the table's per-arch "Size
# (compressed)" column, which is why full_size is the right field.
#
# NOT COVERED, deliberately: README.md's ~3.2 GB figures are UNCOMPRESSED
# on-disk sizes, and the registry API exposes compressed sizes only (layer
# sizes in a manifest are compressed; the config blob carries no uncompressed
# totals). Measuring them needs a real pull, so they are out of scope here --
# do not read a green check 8 as covering them.
# ---------------------------------------------------------------------------
if [ "${SKIP_SIZE_CHECK:-0}" = "1" ]; then
skip "size claims -- SKIP_SIZE_CHECK=1 was set"
elif ! command -v curl >/dev/null 2>&1 || ! command -v python3 >/dev/null 2>&1; then
skip "size claims -- need both curl and python3 to measure them"
else
# Derive the repo from the doc's own rows rather than hardcoding it, so a
# rename cannot leave this check silently probing a repo nobody publishes to.
# shellcheck disable=SC2016 # single quotes are deliberate: this is a sed
# script, and its \( \) groups and \1 backreference must reach sed unexpanded.
HUB_REPO_PATH="$(sed -n 's/^| `\([^:`]*\):[^`]*`.*/\1/p' "$HUB" | head -1)"
if [ -z "$HUB_REPO_PATH" ]; then
skip "size claims -- found no \`repo:tag\` image rows in $HUB to check"
else
HUB_TAGS_JSON="$(curl -sS -m 20 \
"https://hub.docker.com/v2/repositories/${HUB_REPO_PATH}/tags/?page_size=100" \
2>/dev/null || true)"
if [ -z "$HUB_TAGS_JSON" ]; then
skip "size claims -- Docker Hub API unreachable (offline?); NOT verified"
else
SIZE_RC=0
# NO `|| true` on the python invocation: an early draft had one, and it
# swallowed the exit code so a printed DRIFT line still exited 0 -- a gate
# that reports the defect and passes anyway. The outer `|| SIZE_RC=$?` is
# what keeps `set -e` happy while preserving the code.
SIZE_OUT="$(HUB_MD="$HUB" HUB_JSON="$HUB_TAGS_JSON" TOL="$SIZE_TOLERANCE_PCT" \
python3 <<'PYEOF'
import json, os, re, sys
try:
data = json.loads(os.environ["HUB_JSON"])
except (ValueError, KeyError) as exc:
print(" SKIP size claims -- Hub API returned unparseable JSON (%s)" % exc)
sys.exit(3)
# full_size == first manifest entry (amd64), which is the per-arch number the
# table's "Size (compressed)" column claims. Verified against .images[] sizes.
sizes = {
r["name"]: r["full_size"] / 1e9
for r in data.get("results", [])
if isinstance(r.get("full_size"), int) and r.get("name")
}
if not sizes:
print(" SKIP size claims -- Hub API returned no usable tags")
sys.exit(3)
tol = float(os.environ["TOL"])
row = re.compile(r"^\|\s*`([^`:]+):([^`]+)`\s*\|[^|]*\|\s*~?([0-9]+(?:\.[0-9]+)?)\s*GB\s*\|")
checked = drift = 0
with open(os.environ["HUB_MD"], encoding="utf-8") as fh:
for line in fh:
m = row.match(line)
if not m:
continue # rows saying "same", and every non-image row
_repo, tag, claimed = m.group(1), m.group(2), float(m.group(3))
if "X.Y.Z" in tag:
continue # placeholder row; the concrete tag is checked instead
# base-<hash> is content-addressed and immutable, so its size is
# base-latest's by construction -- probe the alias that always exists.
probe = "base-latest" if tag.startswith("base-") else tag
actual = sizes.get(probe)
if actual is None:
print(" SKIP size %s -- tag '%s' not present on Hub" % (tag, probe))
continue
checked += 1
off = abs(claimed - actual) / actual * 100
if off <= tol:
print(" OK size %s claims ~%.2f GB, Hub measures %.2f GB (%.0f%% off)"
% (tag, claimed, actual, off))
else:
drift += 1
print(" DRIFT size %s claims ~%.2f GB but Hub measures %.2f GB"
" (%.0f%% off, tolerance %.0f%%)" % (tag, claimed, actual, off, tol))
if checked == 0:
print(" SKIP size claims -- no checkable rows resolved to a published tag")
sys.exit(3)
sys.exit(1 if drift else 0)
PYEOF
)" || SIZE_RC=$?
printf '%s\n' "$SIZE_OUT"
case "$SIZE_RC" in
0) : ;;
3) SKIPS=$((SKIPS + 1)) ;;
*)
fail "a published size claim in $HUB has drifted from what Docker Hub
actually serves (see DRIFT above). This page is POSTed to Docker Hub by
update-description, so it is the first size a stranger sees. Re-measure and
update the table:
curl -sS 'https://hub.docker.com/v2/repositories/${HUB_REPO_PATH}/tags/?page_size=100' |
jq -r '.results[] | \"\\(.name) \\(.full_size/1e9)\"'"
;;
esac
fi
fi
fi
echo
if [ "$FAILURES" -eq 0 ]; then
if [ "$SKIPS" -gt 0 ]; then
echo "OK: every checked doc claim matches the build files" \
"($SKIPS check(s) SKIPPED and therefore NOT verified -- see SKIP above)."
else
echo "OK: every checked doc claim matches the build files."
fi
exit 0
fi
echo "::error::$FAILURES doc claim(s) have drifted from the build files."
echo
echo "Docs are read from the TAG, not from main: docker-publish.yml checks out"
echo "github.ref, so a fix pushed after tagging does not reach the release or the"
echo "Hub page. Update the docs BEFORE you tag."
if [ "$WARN_ONLY" -eq 1 ]; then
echo "(--warn-only: exiting 0 anyway)"
exit 0
fi
exit 1