skills: record the vendored snapshot's provenance, and report which copy wins
Lint / actionlint (push) Successful in 16s
Lint / hadolint (push) Successful in 16s

Found while verifying v1.8.7 from inside a fresh container: the baked mempalace
snapshot is read by no host on this fleet. devbox-skill-reconcile repoints
~/.agents/skills/mempalace at the mounted live clone (the v1.8.5 fix working as
designed), and all four compose stacks mount a workspace containing the
skillset. So the phrase canary that blocked v1.8.7's first tag polices a file
nobody opens, while the drift that could actually mislead an agent — a git pull
nobody ran in /workspace/skillset — was invisible from inside the container and
is invisible to CI by construction.

Record provenance instead of policing it, and move the check to where the
skillset actually is:

- Dockerfile.variant: ARG SKILLSET_SNAPSHOT_REF (the claim) + a sha256 of the
  shipped bytes measured in the manifest layer (the fact), as manifest siblings
  rather than components{} members, plus an OCI label. An ARG default, not a
  CI-resolved output: no credential for the private skillset, no change at any
  of the four variant build call sites, and a local docker build records what CI
  does. Variant-only, so no base rebuild — check-base-hash.sh scans
  Dockerfile.base alone, verified by running it.
- pi-devbox-version: a skills: section naming baked vs live <repo> @ <sha> per
  vendored skill, and for mempalace whether the live copy is identical to the
  baked fingerprint, at the same commit with uncommitted edits, or divergent.
  entrypoint-user.sh passes the new --no-skills, because the banner prints
  before the links exist and long before the reconcile runs.
- scripts/vendor-mempalace-skill.sh: refresh the file and rewrite the ref
  together (a cp without an ARG bump makes the manifest lie, which is worse than
  anonymity); --check verifies the claim against a real clone.
- 5 new smoke assertions (78 -> 83), mutation-tested through the real sh -c
  path: 6 fabricated manifests, where a well-formed hash of the wrong file
  proves the two manifest assertions are not redundant; the all-baked reporting
  test verified to FAIL against a live-skillset environment.

Reviewed mid-flight by pi@emb-7kj4vr4g over the logstream (correlation
skillset-vendor-drift), which retracted its own earlier recommendation of a
build-time byte-compare against skillset HEAD and supplied the better framing:
the invariant is NON-CONTRADICTION, not currency. Byte parity on a fallback
would have cost a resync commit plus a ~67-min base rebuild for each of the four
skillset commits pushed in one evening. Its warning also found a real bug here:
the script now CONSTRUCTS the snapshot from `git show HEAD:<path>` instead of
copying the working tree, because a clean `git diff` says nothing about an
untracked file — the one input the first draft would have recorded a false ref
for. Tested: untracked, unstaged and staged-but-uncommitted all refuse, atomically.

Also fixes three stale in-repo markers of the same class the canary belongs to
(true when written, silently false at release): two dangling "Unreleased"
pointers and a typst line still marked Unreleased five releases after v1.4.0.
This commit is contained in:
2026-08-26 10:27:00 +02:00
parent dbb78798fb
commit e070e0bcbf
7 changed files with 604 additions and 7 deletions
+71 -3
View File
@@ -7,7 +7,7 @@
# - pi binary present and (if EXPECTED_PI_VERSION set) matches CI's resolved version
# - mempalace core matches the audited pin (if EXPECTED_MEMPALACE_VERSION set)
# - new v1.0.0 base additions (pandoc, graphviz, imagemagick, yq, tealdeer)
# - typst PDF engine for pandoc (Unreleased) — `pandoc --pdf-engine=typst`
# - typst PDF engine for pandoc (v1.4.0) — `pandoc --pdf-engine=typst`
# - non-modal editors nano + micro (alongside nvim)
# - terminfo for modern emulators: xterm-kitty, xterm-ghostty, wezterm,
# alacritty, foot (kitty-terminfo + ncurses-term + compiled ghostty alias)
@@ -463,6 +463,38 @@ run "pi-devbox-version --json round-trips the manifest byte-for-byte" '
'
run_expect "pi-devbox-version --quiet is a compact one-liner" \
"pi-devbox-version --quiet | wc -l" "1"
# ── Vendored skill snapshot provenance ─────────────────────────────────
# The vendored mempalace skill is the one baked artefact with no /opt clone
# behind it (private upstream — see VENDORED.md), so until now the manifest
# could not say which skillset commit it came from. Two fields now travel with
# it: the CLAIMED ref (ARG default in Dockerfile.variant) and the MEASURED
# sha256 of the shipped bytes. Assert both are well-formed, and — separately —
# that the measurement still describes the file in the image.
#
# Kept as two assertions for the same reason the component checks are: one
# proves the fields are not empty/garbage, the other proves they are not merely
# self-consistent. A single combined check could pass on a manifest whose hash
# was computed from a file that was later overwritten (the pi-extensions skill
# copy at Dockerfile.variant:165 does exactly that kind of overwrite, one stage
# earlier), which is the failure this second one exists to catch.
run "manifest records the vendored skill snapshot provenance" '
j=/etc/pi-devbox/build-manifest.json
r=$(jq -r ".skillset_snapshot_ref // empty" $j)
s=$(jq -r ".skillset_snapshot_sha256 // empty" $j)
echo "ref=[$r] sha256=[$s]" >&2
printf "%s" "$r" | grep -qxE "[0-9a-f]{40}" \
|| { echo "skillset_snapshot_ref is not a 40-hex commit" >&2; exit 1; }
printf "%s" "$s" | grep -qxE "[0-9a-f]{64}" \
|| { echo "skillset_snapshot_sha256 is not a 64-hex digest" >&2; exit 1; }
'
run "manifest skill fingerprint matches the baked snapshot" '
j=/etc/pi-devbox/build-manifest.json
f=/usr/local/share/pi-devbox/skills/mempalace/SKILL.md
m=$(jq -r ".skillset_snapshot_sha256 // empty" $j)
a=$(sha256sum "$f" | cut -d" " -f1)
echo "manifest=[$m] actual=[$a]" >&2
[ -n "$m" ] && [ "$m" = "$a" ]
'
# OCI labels live in the image config, not the container fs — inspect them
# from the host docker rather than via `docker run`.
LBL=$(docker inspect --format '{{ index .Config.Labels "se.jordbo.pi-devbox.pi-extensions-ref" }}' "$IMAGE" 2>/dev/null || true)
@@ -539,8 +571,24 @@ exec_test "mempalace skill linked (fallback)" 'test -L $HOME/.agents/skills
# one absent, so a re-vendored stale snapshot fails just as loudly as a
# forgotten bump. A one-way canary only catches half the drift.
# * a phrase canary can only ever detect "older than what I remembered to pin",
# never "older than skillset main". The real fix is a CI job diffing this
# file against the skillset repo — see the Unreleased changelog note.
# never "older than skillset main".
#
# That structural limit is now addressed, but NOT by the "CI job diffing this
# file against the skillset repo" this comment used to point at (that pointer
# also dangled: it referenced an Unreleased changelog note that had become the
# v1.8.7 heading). A CI diff cannot be done without granting CI a credential
# for the PRIVATE skillset repo, and it would guard a file that on this fleet
# NO host reads — all four compose stacks mount a workspace containing the
# skillset, so devbox-skill-reconcile repoints this link at the live clone and
# the baked copy is a CI/no-mount fallback only. Instead the snapshot now
# carries its provenance (skillset_snapshot_ref + a measured
# skillset_snapshot_sha256 in build-manifest.json, written by
# scripts/vendor-mempalace-skill.sh), which moves the check to where the
# skillset actually IS: `scripts/vendor-mempalace-skill.sh --check` for a
# maintainer, and `pi-devbox-version` for an agent inside any container.
# This assertion is kept because it is orthogonal and free: it pins content,
# not provenance, so it still catches a re-vendored snapshot whose ref was
# bumped correctly but whose bytes came from the wrong place.
exec_test "mempalace skill snapshot is current" 'f=$HOME/.agents/skills/mempalace/SKILL.md; grep -q "Provenance is stamped for you" "$f" && ! grep -q "Attribute what you file yourself" "$f" && echo ok'
# Link TARGETS, not just link existence: with no skillset mounted (as here) the
# baked tree must be what resolves, for all three vendored skills.
@@ -551,6 +599,26 @@ exec_test "vendored skills resolve to the baked tree (no skillset mounted)" \
*) echo "$s resolves to $(readlink -f $HOME/.agents/skills/$s)" >&2; exit 1 ;;
esac
done; echo ok'
# ... and that the tool REPORTS that resolution, which is the half that was
# missing: a stale baked snapshot and a current live clone were
# indistinguishable from inside the container. CI mounts no skillset, so every
# vendored skill must report "baked" here — which also makes this a real test of
# the fallback path rather than of the environment it happens to run in.
exec_test "pi-devbox-version reports skill sources (all baked, no skillset here)" \
'out=$(pi-devbox-version)
echo "$out" | grep -q "skills:" || { echo "no skills section" >&2; exit 1; }
for s in mempalace pi-extensions pi-devbox-environment; do
echo "$out" | grep -qE "^ $s +baked$" \
|| { echo "$s not reported as baked" >&2; exit 1; }
done; echo ok'
# The boot banner must NOT carry the section: entrypoint-user.sh prints the
# version FIRST, before the baked links exist and long before the skillset
# deploy + reconcile run last, so anything it said about skill sources would be
# a pre-reconcile state that is about to change.
exec_test "pi-devbox-version --no-skills omits the skills section" \
'! pi-devbox-version --no-skills | grep -q "skills:"'
exec_test "entrypoint prints the version banner with --no-skills" \
'grep -q "pi-devbox-version --no-skills" /usr/local/bin/entrypoint-user.sh'
# The handover path itself. CI never mounts a skillset, so without this the
# v1.8.5 fix would ship untested: fabricate a skillset + a skills dir holding
# baked-style links, run the reconciler, and assert all three outcomes —