feat(ci): gate documentation drift, and make docs a pre-tag release step
Lint / hadolint (push) Successful in 13s
Lint / actionlint (push) Successful in 17s
Lint / doc-drift (push) Successful in 7s
Lint / skill-floor (push) Successful in 10s

Five doc claims had rotted by v1.9.0, all the same shape: a value written once
by hand, in a file nothing verifies, about a number that lives elsewhere and
moved. README pin table wrong on all three rows; a "Planned" section describing
something already shipped; DOCKER_HUB.md claiming Node v22 against Node 24.

DOCKER_HUB.md is why this is a gate and not a resolution to be careful: it is
PUBLISHED (update-description POSTs it as Docker Hub full_description on every
tag), it had gone eight releases untouched, nothing generates it, and it is read
from the TAG -- so the stale page shipped with v1.9.0 regardless.

scripts/check-doc-drift.sh: seven checks, all repo-local (no network, token,
image, or sibling clone). Exit 0/1/2 matching lint-shell.sh; a renamed ARG is a
red 2, not a green tick. Wired as a fourth lint.yml job so "the docs lie" is its
own red name.

Verified with 15 controls, including two false-positive controls: the first
placeholder check flagged README.md:900, a Go template in a legitimate
`docker inspect --format` example. The gate was wrong, not the doc, so the
pattern is now anchored to the UPPER_SNAKE convention CI substitutes.

Not gated, deliberately: counts/sizes needing a running image (they belong in
smoke-test.sh -- a guessing gate is worse than none), and Dockerfile.base
BASE_REBUILD_DATE, because base_tag hashes that file content-wise and demanding
it be current would force a ~60 min rebuild on releases that touch no base
files. Free during a rebuild, expensive otherwise.

AGENTS.md step 3 rewritten around the mechanism: checkout@v4 with no ref: means
every job reads github.ref, the tag. Docs must be right BEFORE tagging.
This commit is contained in:
Joakim Persson
2026-09-10 22:01:49 +02:00
parent 35964abd01
commit 3a44e81cad
4 changed files with 410 additions and 1 deletions
+43
View File
@@ -172,3 +172,46 @@ jobs:
- name: Vendored pi-extensions skill floor matches the package - name: Vendored pi-extensions skill floor matches the package
run: bash scripts/check-skill-floor.sh run: bash scripts/check-skill-floor.sh
doc-drift:
# Gate hand-maintained doc claims against the build files they describe.
# Its own job for the same reason as skill-floor: "the docs lie" should be a
# distinct red name, not a line buried in a job about workflow syntax.
#
# The gap it closes, measured 2026-09-10 while preparing v1.9.0 — five
# claims had rotted, every one of them a fact written by hand in a file
# nothing verified:
# * README.md's "Version pins" table was wrong on ALL THREE rows (pi
# 0.84.4 vs 0.85.1, pi-atelier v0.10.0 vs v0.10.1, mempalace 3.8.0 vs
# 3.9.0) — and that table exists specifically to be the reviewable
# record of what the repo freezes on purpose, so a wrong row destroys
# the only thing it is for.
# * README.md listed already-shipped typst PDF export under "Planned for
# an upcoming minor release", marked "(shipped in Unreleased/base)".
# * DOCKER_HUB.md claimed "Node.js v22" while v1.9.0 ships Node 24.
#
# DOCKER_HUB.md is why this is a gate and not a habit. It is PUBLISHED —
# update-description POSTs it to Docker Hub as full_description on every tag
# — and it had gone eight releases (v1.8.6 -> v1.9.0) untouched. Nothing
# generates it and nothing checked it, so the only thing keeping it true was
# someone remembering. It is also read from the TAG, so a fix pushed to main
# after tagging never reaches the published page.
#
# Cheap and hermetic on purpose: every check compares a doc string against a
# value that exists in this repo, so no network, no token, no built image,
# and no sibling clone. Claims that genuinely need a running container (image
# sizes, the "N mempalace_* tools" count) are deliberately left out — a gate
# that cannot evaluate a claim honestly would have to guess, and a guessing
# gate is worse than none. Assert those in scripts/smoke-test.sh instead.
#
# Exit codes 0 in sync / 1 drift / 2 cannot-run, matching lint-shell.sh and
# check-skill-floor.sh. A renamed ARG makes the gate blind, so that is a red
# 2, not a green tick.
runs-on: ubuntu-latest
container:
image: catthehacker/ubuntu:act-latest
steps:
- uses: actions/checkout@v4
- name: Doc claims match the build files
run: bash scripts/check-doc-drift.sh
+38 -1
View File
@@ -92,7 +92,44 @@ re-brand of opencode-devbox's `pi-only` variant.
is hashed into `base_tag`, so it costs a base rebuild (~67 min); and if the is hashed into `base_tag`, so it costs a base rebuild (~67 min); and if the
section the phrase canary names has changed, re-pin it in section the phrase canary names has changed, re-pin it in
`scripts/smoke-test.sh`. `scripts/smoke-test.sh`.
3. Update `CHANGELOG.md` Unreleased → vX.Y.Z section. 3. **Update the docs this release makes stale — BEFORE you tag.** Rename
`CHANGELOG.md`'s `## Unreleased` to `## vX.Y.Z — YYYY-MM-DD` (em dash, as
every prior release heading uses), then run the gate:
```bash
bash scripts/check-doc-drift.sh # 0 in sync / 1 drift / 2 cannot run
```
It compares README.md's version-pin table against the ARGs it names, and
DOCKER_HUB.md's Node claim against `ARG NODE_VERSION`, plus Hub's
25 000-char limit, unsubstituted `{{PLACEHOLDERS}}`, and stale `Unreleased`
pointers in user-facing docs.
**Why before and not after:** `docker-publish.yml` runs `actions/checkout@v4`
with no `ref:`, so every job reads `github.ref` — the **tag**. A doc fix
pushed to `main` after tagging does not reach the release, and for
`DOCKER_HUB.md` it does not reach the published Hub page either, because
`update-description` POSTs that file as Docker Hub's `full_description` from
the tag's tree. Getting it in afterwards means re-pointing the tag, which is
its own hazard (v1.8.14 went `601fc98` → `361babd` and broke deploy
verification until `git fetch --tags --force`).
The gate is deliberately narrow — it only checks claims verifiable from files
in this repo. Still eyeball, because these are NOT gated:
- counts and sizes (`~1.1 GB`, "N `mempalace_*` tools", "7 extensions") —
they need a running image; assert them in `scripts/smoke-test.sh` instead
- feature prose that quietly became false, e.g. a "Planned for an upcoming
release" section describing something that already shipped
- `Dockerfile.base`'s `# BASE_REBUILD_DATE:` marker. Ungated on purpose:
`base_tag` hashes Dockerfile.base's content, comments included, so
demanding it be current would force a ~60 min base rebuild on a release
that touched no base files. **Fix it when the base is already rebuilding —
then it is free.**
Measured cost of skipping this, 2026-09-10 (v1.9.0): five stale claims, one
of them published. README's pin table was wrong on all three rows, and
DOCKER_HUB.md — untouched for eight releases — still said Node v22 while the
image shipped Node 24.
4. Verify `docker compose up` works locally with the current `latest` image 4. Verify `docker compose up` works locally with the current `latest` image
if you're upgrading users from a previous version. Then run the if you're upgrading users from a previous version. Then run the
**post-recreate sanity check** inside the running container to confirm **post-recreate sanity check** inside the running container to confirm
+83
View File
@@ -11,6 +11,89 @@ Pre-v1.0.0 tags followed the pi npm version (`v{pi_version}[letter]`).
--- ---
## Unreleased
**A gate for documentation drift, because five claims rotted in one release and
one of them was published.** Preparing v1.9.0 turned up a cluster of stale
facts, all the same shape — a value written once by hand, in a file nothing
verifies, about a number that lives somewhere else and moved:
- `README.md`'s "Version pins" table was wrong on **all three rows**: pi
`0.84.4` vs `ARG PI_VERSION=0.85.1`, pi-atelier `v0.10.0` vs `v0.10.1`,
mempalace `3.8.0` vs `3.9.0`. That table is the worst possible place for this,
because it exists *specifically* to be the reviewable record of what the repo
freezes deliberately — so a wrong row destroys the only thing it is for.
- `README.md` listed already-shipped typst PDF export under "Planned for an
upcoming minor release", carrying the self-contradicting marker "(shipped in
Unreleased/base)". The **fourth** instance of the stale-`Unreleased`-pointer
class this changelog already documented three of.
- `DOCKER_HUB.md` claimed "Node.js v22" while v1.9.0 ships Node 24.
The last one is why this became a gate rather than a resolution to be careful.
`DOCKER_HUB.md` is **published**: `update-description` POSTs it to Docker Hub as
`full_description` on every tag. It had gone **eight releases** (v1.8.6 →
v1.9.0) without a touch. Nothing generates it — CI only substitutes
`{{PI_VERSION}}` — and nothing checked it, so the sole mechanism keeping it true
was whoever remembered. Worse, it is read from the **tag**, so the stale page
published with v1.9.0 anyway and the fix could only ride the next release.
**New: `scripts/check-doc-drift.sh` + a `doc-drift` job in `lint.yml`.** Seven
checks, all comparing a doc string to a value that exists in this repo, so it
needs no network, no token, no built image, and no sibling clone:
- README's three pin-table rows vs the ARGs they name *by name*
- `DOCKER_HUB.md`'s Node claim vs `ARG NODE_VERSION`
- placeholders CI will not substitute — the publish step greps for leftovers of
`{{PI_VERSION}}` only, so any *second* token sails through and publishes
literally
- `DOCKER_HUB.md` under Docker Hub's 25 000-char `full_description` limit
(previously discoverable only as a non-200 *after* the full build)
- `Unreleased` appearing in a user-facing doc, which is always a pointer that
outlived what it pointed at
Exit codes match `lint-shell.sh` and `check-skill-floor.sh`: `0` in sync, `1`
drift, `2` cannot run — a renamed ARG makes the gate blind, which is a red `2`,
never a green tick. Verified with **15 controls**: every check fails when its
claim is broken, the real v1.9.0 Node bug is caught, and two false-positive
controls pass — the first version of the placeholder check wrongly flagged
`README.md:900`'s `docker inspect --format '{{json .Config.Labels}}'`, a Go
template in a legitimate example, so the pattern is now anchored to the
UPPER_SNAKE convention CI actually substitutes. **The gate was wrong, not the
doc** — which is the whole reason a gate gets negative controls.
Deliberately **not** gated, and the reasons matter more than the list:
- Counts and sizes (`~1.1 GB`, "N `mempalace_*` tools", "7 extensions") need a
running image. A gate that cannot evaluate a claim honestly would have to
guess, and a guessing gate is worse than none — assert these in
`scripts/smoke-test.sh`, where a real image exists.
- `Dockerfile.base`'s `# BASE_REBUILD_DATE:` marker, itself stale (2026-07-13,
three base rebuilds ago). `base_tag` hashes Dockerfile.base's *content*,
comments included, so demanding it be current would force a ~60 min base
rebuild on a release that touched no base files at all. It is free to fix
while the base is *already* rebuilding, and expensive at any other moment.
That cost asymmetry is now written into the release checklist rather than
enforced.
**Release checklist step 3 rewritten** (`AGENTS.md`) around the mechanism that
made this expensive: `docker-publish.yml` runs `actions/checkout@v4` with no
`ref:`, so every job reads `github.ref` — the tag. Docs must be correct *before*
tagging; afterwards the only routes are re-pointing the tag (its own hazard —
v1.8.14 went `601fc98` → `361babd` and broke deploy verification until
`git fetch --tags --force`) or waiting for the next release. The step now also
names what the gate cannot see, so "gate is green" is not mistaken for "docs are
true". The same reflex went into the `ci-release-watcher` skill, as the first
correctness rule — it is the only one that expires once the tag exists.
Also fixed in passing: README's `pi-devbox-version` sample was v1.5.0-era and
structurally outdated (it predated the `palace:` line the surrounding prose
advertises, the `pi-atelier` component, and the whole `skills:` block). Replaced
with real observed output rather than hand-written text. `DOCKER_HUB.md`'s "7
user-facing extensions" was **verified correct**; its "29 `mempalace_*` tools"
is stale (a live client shows 45) but left alone rather than corrected on a
guess, since that count cannot be attributed to the baked 3.9.0 server without
measuring it.
## v1.9.0 — 2026-09-10 ## v1.9.0 — 2026-09-10
**`shellcheck` is now in the image, because the release gate it depends on could **`shellcheck` is now in the image, because the release gate it depends on could
+246
View File
@@ -0,0 +1,246 @@
#!/usr/bin/env bash
# check-doc-drift.sh — fail when a hand-maintained doc claim contradicts the
# build files it describes.
#
# THE DEFECT CLASS THIS EXISTS TO CATCH, measured 2026-09-10 while preparing
# v1.9.0. Five separate claims had rotted, all of them the same shape: a fact
# written once by hand, in a file nothing verifies, about a value that lives
# somewhere else and moved.
#
# 1..3. README.md's "Version pins" table was wrong on EVERY row — pi `0.84.4`
# vs ARG PI_VERSION=0.85.1, pi-atelier `v0.10.0` vs v0.10.1, mempalace
# `3.8.0` vs 3.9.0. That table is the worst possible place for this: it
# exists precisely to be the reviewable record of what is deliberately
# frozen, so when it lies, the review it enables is worthless.
# 4. README.md carried a "Planned for an upcoming minor release" section
# listing typst PDF export, which had ALREADY SHIPPED, tagged with a
# self-contradicting "(shipped in Unreleased/base)" marker. The
# CHANGELOG had already documented three earlier instances of exactly
# this stale-"Unreleased"-pointer class (see its v1.8.7 notes).
# 5. DOCKER_HUB.md claimed "Node.js v22" while this release ships Node 24.
# This one is the reason the gate exists at all: DOCKER_HUB.md is
# PUBLISHED. `update-description` in docker-publish.yml POSTs it to Hub
# as full_description on every tag, so unlike README.md — which no
# workflow or gate reads — a stale claim here is what users see.
#
# WHY A GATE AND NOT "REMEMBER TO CHECK". DOCKER_HUB.md had gone eight releases
# (v1.8.6 → v1.9.0) without a touch. Nothing generates it and nothing verifies
# it; the only mechanism keeping it true was whoever remembered. That is the
# same failure mode check-skill-floor.sh was written for, and the same fix:
# convert "someone remembers" into "CI refuses".
#
# WHY THESE FIVE CHECKS AND NOT MORE. Every check here compares a doc string to
# a value that EXISTS IN THIS REPO, so it can never be wrong about the world and
# needs no network, no token, and no built image. Claims that require a running
# container to verify (image sizes, the "N mempalace_* tools" count) are
# deliberately NOT gated: a check that cannot be evaluated honestly at lint time
# would either be skipped or guessed, and a guessing gate is worse than none.
# If you want those, assert them in scripts/smoke-test.sh where a real image is
# available.
#
# DELIBERATELY NOT GATED: Dockerfile.base's `# BASE_REBUILD_DATE:` comment, which
# is also stale (2026-07-13, three base rebuilds ago). base_tag is a hash of
# Dockerfile.base's CONTENT plus rootfs/, comments included, so a gate that
# demanded that comment be current would force a ~60 min base rebuild on any
# release that touched no base files at all. Fix it when you are already
# rebuilding the base — then it is free. This is a real cost asymmetry, not
# laziness.
#
# EXIT CODES (same contract as lint-shell.sh and check-skill-floor.sh):
# 0 every checked claim matches
# 1 at least one claim has drifted
# 2 cannot run (a file or ARG this gate reads is missing/unparseable)
# A gate that cannot run must not pass, so a missing input is 2, never 0.
set -euo pipefail
REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$REPO_ROOT"
README="README.md"
HUB="DOCKER_HUB.md"
DF_VARIANT="Dockerfile.variant"
DF_BASE="Dockerfile.base"
# Docker Hub rejects a full_description longer than this. docker-publish.yml has
# no size check of its own; it only notices via a non-200 from the API, i.e.
# after paying the whole build. Catching it here makes it a 2-second failure.
HUB_MAX_CHARS=25000
WARN_ONLY=0
FAILURES=0
usage() {
cat <<'EOF'
Usage: check-doc-drift.sh [--warn-only] [-h|--help]
Compares hand-written claims in README.md and DOCKER_HUB.md against the build
files they describe (Dockerfile.base, Dockerfile.variant).
--warn-only Report drift but exit 0 (advisory use, e.g. a local pre-push hook).
Exit: 0 = in sync, 1 = drift, 2 = cannot run.
EOF
}
while [ $# -gt 0 ]; do
case "$1" in
--warn-only) WARN_ONLY=1; shift ;;
-h|--help) usage; exit 0 ;;
*) echo "::error::unknown argument: $1" >&2; usage >&2; exit 2 ;;
esac
done
for f in "$README" "$HUB" "$DF_VARIANT" "$DF_BASE"; do
if [ ! -f "$f" ]; then
echo "::error::$f not found (cwd $PWD). Cannot evaluate doc drift, so this is exit 2, not a pass."
exit 2
fi
done
# Read `ARG NAME=value` from a Dockerfile. Exit 2 when absent: if the ARG this
# gate is built around has been renamed, the gate is measuring nothing and must
# say so rather than silently comparing against an empty string.
read_arg() {
local file="$1" name="$2" value
value="$(sed -n "s/^ARG ${name}=\\(.*\\)\$/\\1/p" "$file" | head -1)"
if [ -z "$value" ]; then
echo "::error::ARG ${name} not found in ${file}. It was probably renamed;" >&2
echo "::error::update check-doc-drift.sh to match, because this gate is now blind." >&2
exit 2
fi
printf '%s' "$value"
}
# One row of README's "Version pins" table: `| pi | `0.85.1` | ... |`
read_pin_row() {
sed -n "s/^| $1 | \`\\([^\`]*\`*\\)\` |.*/\\1/p" "$README" | head -1
}
fail() {
FAILURES=$((FAILURES + 1))
echo "::error::$1"
}
ok() { printf ' OK %s\n' "$1"; }
echo "Checking hand-maintained doc claims against the build files they describe."
echo
# ---------------------------------------------------------------------------
# 1-3. README's version-pin table vs the ARGs it names by name.
# ---------------------------------------------------------------------------
check_pin() {
local label="$1" documented="$2" actual="$3" where="$4"
if [ -z "$documented" ]; then
fail "README.md: no '| $label |' row found in the version-pin table. Either the
table was restructured (update this gate) or the row was dropped (restore it)."
return
fi
if [ "$documented" != "$actual" ]; then
fail "README.md version-pin table is stale for $label: says '$documented',
$where says '$actual'. Fix the table — it is the reviewable record of what
this repo deliberately freezes, so a wrong row defeats its only purpose."
return
fi
ok "README pin $label = $actual"
}
PI_ACTUAL="$(read_arg "$DF_VARIANT" PI_VERSION)"
ATELIER_ACTUAL="$(read_arg "$DF_VARIANT" PI_ATELIER_REF)"
MEMPALACE_ACTUAL="$(read_arg "$DF_BASE" MEMPALACE_VERSION)"
check_pin pi "$(read_pin_row pi)" "$PI_ACTUAL" "ARG PI_VERSION in $DF_VARIANT"
check_pin pi-atelier "$(read_pin_row pi-atelier)" "$ATELIER_ACTUAL" "ARG PI_ATELIER_REF in $DF_VARIANT"
check_pin mempalace "$(read_pin_row mempalace)" "$MEMPALACE_ACTUAL" "ARG MEMPALACE_VERSION in $DF_BASE"
# ---------------------------------------------------------------------------
# 4. DOCKER_HUB.md's Node claim vs ARG NODE_VERSION. This is the published page,
# so it is the one whose staleness reaches users.
# ---------------------------------------------------------------------------
NODE_ACTUAL="$(read_arg "$DF_BASE" NODE_VERSION)"
NODE_DOCUMENTED="$(sed -n 's/.*\*\*Node\.js\*\* v\([0-9][0-9]*\).*/\1/p' "$HUB" | head -1)"
if [ -z "$NODE_DOCUMENTED" ]; then
fail "$HUB: could not find a '**Node.js** vNN' claim. If the wording changed,
update this gate; do not leave the published page unverified."
elif [ "$NODE_DOCUMENTED" != "$NODE_ACTUAL" ]; then
fail "$HUB claims Node v$NODE_DOCUMENTED but ARG NODE_VERSION=$NODE_ACTUAL.
This file is PUBLISHED to Docker Hub by update-description on every tag,
and it is read from the TAG — so fix it before tagging, not after."
else
ok "$HUB Node claim = v$NODE_ACTUAL"
fi
# ---------------------------------------------------------------------------
# 5. Placeholders CI will not substitute. docker-publish.yml substitutes exactly
# {{PI_VERSION}} and then greps for leftovers of that ONE token, so any other
# {{...}} sails through the guard and is published literally.
# ---------------------------------------------------------------------------
UNKNOWN_PLACEHOLDERS="$(grep -o '{{[A-Za-z0-9_]*}}' "$HUB" | sort -u | grep -v '^{{PI_VERSION}}$' || true)"
if [ -n "$UNKNOWN_PLACEHOLDERS" ]; then
fail "$HUB contains placeholders CI does not substitute, which would be
published verbatim: $(echo "$UNKNOWN_PLACEHOLDERS" | tr '\n' ' ')
docker-publish.yml only fills {{PI_VERSION}}; add substitution there first."
else
ok "$HUB has no placeholders beyond {{PI_VERSION}}"
fi
# Match only the UPPER_SNAKE placeholder convention CI uses. A bare '{{' search
# is WRONG here, and the first version of this check proved it by failing on
# README.md:900 — `docker inspect --format '{{json .Config.Labels}}'`, a Go
# template in a legitimate example, not a placeholder. The gate was wrong, not
# the doc. Keep this anchored to [A-Z] so Go/Jinja/Handlebars examples pass.
README_PLACEHOLDERS="$(grep -o '{{[A-Z][A-Z0-9_]*}}' "$README" | sort -u || true)"
if [ -n "$README_PLACEHOLDERS" ]; then
fail "$README contains placeholder(s) nothing substitutes, so they would render
literally for every reader: $(echo "$README_PLACEHOLDERS" | tr '\n' ' ')
Only DOCKER_HUB.md gets substitution, and only for {{PI_VERSION}}."
else
ok "$README has no unsubstituted placeholders"
fi
# ---------------------------------------------------------------------------
# 6. Hub full_description length.
# ---------------------------------------------------------------------------
HUB_CHARS="$(wc -c < "$HUB" | tr -d ' ')"
if [ "$HUB_CHARS" -gt "$HUB_MAX_CHARS" ]; then
fail "$HUB is $HUB_CHARS chars, over Docker Hub's $HUB_MAX_CHARS-char
full_description limit. update-description would fail with a non-200 AFTER
the full build. Trim it — this file is the essentials-only page, and
README.md is the long form on purpose."
else
ok "$HUB is $HUB_CHARS chars (limit $HUB_MAX_CHARS)"
fi
# ---------------------------------------------------------------------------
# 7. Stale "Unreleased" pointers. "Unreleased" is a CHANGELOG-only concept; in
# a user-facing doc it is always a pointer that outlived what it pointed at.
# This class has now bitten five times, hence a gate rather than vigilance.
# ---------------------------------------------------------------------------
STALE_MARKERS="$(grep -n 'Unreleased' "$README" "$HUB" || true)"
if [ -n "$STALE_MARKERS" ]; then
fail "'Unreleased' appears in a user-facing doc, which is always a stale
pointer once the thing ships (it has happened five times here):
${STALE_MARKERS//$'\n'/$'\n' }
State the fact directly, or move it to CHANGELOG.md where 'Unreleased' means something."
else
ok "no stale 'Unreleased' pointers in $README or $HUB"
fi
echo
if [ "$FAILURES" -eq 0 ]; then
echo "OK: every checked doc claim matches the build files."
exit 0
fi
echo "::error::$FAILURES doc claim(s) have drifted from the build files."
echo
echo "Docs are read from the TAG, not from main: docker-publish.yml checks out"
echo "github.ref, so a fix pushed after tagging does not reach the release or the"
echo "Hub page. Update the docs BEFORE you tag."
if [ "$WARN_ONLY" -eq 1 ]; then
echo "(--warn-only: exiting 0 anyway)"
exit 0
fi
exit 1