diff --git a/.gitea/workflows/lint.yml b/.gitea/workflows/lint.yml index 6ca7f32..dfedbc0 100644 --- a/.gitea/workflows/lint.yml +++ b/.gitea/workflows/lint.yml @@ -137,3 +137,38 @@ jobs: - name: Run hadolint run: hadolint Dockerfile.base Dockerfile.variant + + skill-floor: + # Gate the VENDORED pi-extensions skill snapshot in rootfs/ against the + # package repo it is a snapshot of. Its own job rather than a step in + # `actionlint`, so "the floor is stale" is a distinct red name in the runs + # list instead of being buried in a lint job that is about something else. + # + # The gap it closes, measured 2026-09-10: the floor sat at 34284 B, untouched + # since fa04d20 (2026-07-30), while the package copy was 38973 B. + # Dockerfile.variant copies the fresh package copy over the SERVED path but + # never writes back to the floor, so nothing in the repo ever noticed. That + # matters because the floor is a FALLBACK: the copy is guarded by + # `if [ -f /opt/pi-extensions/skill/SKILL.md ]`, so a build whose clone + # yields no skill/ ships the vendored snapshot and still goes green, with no + # manifest flag or label saying which copy was served. + # + # Gating on another repo is normally a smell; it is proportionate here + # because the check compares the skill DIRECTORY hash, so it can only fire + # when that directory actually changed — which is exactly when the floor has + # gone stale. pi-extensions commits that leave skill/ alone cannot turn this + # red. No secret is needed either: the repo is anonymously clonable (verified + # 2026-09-10 with `git ls-remote` and no credentials), so this cannot start + # failing when a token expires. + # + # Exit codes are 0 in sync / 1 drift / 2 cannot-run, matching + # scripts/lint-shell.sh: a gate that cannot run must not pass, so an + # unreachable package repo is a red 2 rather than a green tick. + runs-on: ubuntu-latest + container: + image: catthehacker/ubuntu:act-latest + steps: + - uses: actions/checkout@v4 + + - name: Vendored pi-extensions skill floor matches the package + run: bash scripts/check-skill-floor.sh diff --git a/CHANGELOG.md b/CHANGELOG.md index 14bca59..6372f53 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -92,6 +92,66 @@ derivation's `mine` query at the newest end (`order: "desc"`); with the previous default `asc` + `limit: 100`, a device passing 100 authored events would have its recent replies fall out of the join window and see answered asks resurface. +**Four small packages, each chosen from a gap that was measured rather than +imagined.** All four were picked by looking back at a real session — the +`gitea.egl.lan`/FreeIPA debugging of 2026-09-09..10 — and asking which absences +actually cost time, not which tools sound useful. `bind9-dnsutils` (~6.1 MB, 10 +packages): `dig`, `host` **and** `nslookup` were all absent, so the container +could resolve names but had no way to interrogate a *specific* nameserver — +`getent hosts` only follows the resolver's default path, so diagnosing "gateway +`172.16.88.1` NXDOMAINs the `egl.lan` zone while `10.20.253.1` is authoritative +for it" had to be hand-rolled in `python3`. Note the package name: plain +`dnsutils` is transitional in trixie. `ldap-utils` (1244 KB, **zero** extra deps): +the fleet authenticates against FreeIPA, yet every LDAP probe had to be run by +SSHing to an already-enrolled host; this gives simple binds only, since GSSAPI +would additionally need `krb5-user` + `libsasl2-modules-gssapi-mit`, which is a +Kerberos-client decision rather than a tool. `xxd` (198 KB) is frank convenience +— `od -c` already does the job. `python3-yaml` (552 KB, zero extra deps) is the +shellcheck story repeating exactly: `scripts/check-workflow-shell.sh`, the guard +against the Gitea `sh`/dash footgun that broke `resolve-versions` (`ed49b8d`) and +`promote-base-latest` (`b7197e8`), hard-exits with "python3 yaml module missing" +without it — and `lint.yml` installing it explicitly in CI was the evidence the +image lacked it. **`netcat-openbsd` was proposed and deliberately rejected**: +measured redundant, because `socat` is already baked and bash's `/dev/tcp` does +reachability checks with zero packages. The reason is recorded in +`Dockerfile.base` so the omission reads as a decision rather than an oversight. + +**The vendored `pi-extensions` skill floor was 41 days stale, and is now gated so +it cannot silently rot again.** `rootfs/usr/local/share/pi-devbox/skills/pi-extensions/` +sat at 34284 B, untouched since `fa04d20` (2026-07-30), while the package copy +was 38973 B — four copies of one skill existed across the fleet with three +different sizes. `Dockerfile.variant` copies the freshly-cloned package copy over +the **served** path but never writes back to the repo floor, so nothing in the +repo ever noticed. That is worse than ordinary staleness because the floor is a +**fallback**: the copy is guarded by `if [ -f /opt/pi-extensions/skill/SKILL.md ]`, +so a build whose clone yields no `skill/` keeps the vendored snapshot and still +goes **green**, with no manifest flag and no label recording which copy was +served — the image would ship a July skill and nothing would say so. The floor is +refreshed here from `pi-extensions@c64c122`, and the new `skill-floor` job in +`lint.yml` runs `scripts/check-skill-floor.sh` to keep it that way. + +The check compares the **directory** hash, using the same `tree_sha256` pipeline +`Dockerfile.variant` uses for `skillset_snapshot_tree_sha256` and for the same +documented reason: a `sha256sum SKILL.md` answers "did this one file change", not +"is this the same skill", and `pi-extensions` ships two files. That is not +hypothetical — it was **verified by negative control**: with `SKILL.md` left +byte-identical and only `evaluate-extension-usage.py` edited, the directory check +correctly fails while a file-only compare would have passed. Exit codes are `0` +in sync / `1` drift / `2` cannot-run, matching `scripts/lint-shell.sh`, so an +unreachable package repo is a red `2` rather than a green tick. Gating on another +repo is normally a smell; it is proportionate here because the check can only +fire when `skill/` itself changed — which is exactly when the floor has gone +stale — and it needs no secret, since `pi-extensions` is anonymously clonable +(verified with `git ls-remote` and no credentials). + +> **What this does *not* fix, stated so nobody reads more into it than is there.** +> The floor is now fresh and guarded, but the *silent-fallback* half remains: +> if the build-time copy is ever absent, the build still succeeds with no +> manifest flag or OCI label recording that the vendored snapshot was served +> instead of the package copy. The durable fix for that is a manifest field +> alongside the existing `skillset_snapshot_tree_sha256`, which this change does +> not add. + --- ## v1.8.14 — 2026-09-08 diff --git a/Dockerfile.base b/Dockerfile.base index a5cf161..af24267 100644 --- a/Dockerfile.base +++ b/Dockerfile.base @@ -162,6 +162,19 @@ ENV DEBIAN_FRONTEND=noninteractive # /dev/tcp does reachability checks with zero packages # (verified against gitea.egl.lan:3000). Recorded here so the # omission reads as a decision rather than an oversight. +# python3-yaml — PyYAML. Added 2026-09-10 for precisely the same reason as +# shellcheck above: a gate this repo ALREADY OWNS could not be +# run locally by anyone. scripts/check-workflow-shell.sh — the +# guard that catches the "bash-only syntax under Gitea's default +# sh/dash shell" footgun that broke resolve-versions (ed49b8d) +# and promote-base-latest (b7197e8) — hard-exits with "ERROR: +# python3 yaml module missing" without it. lint.yml installs it +# explicitly in CI (`shellcheck python3-yaml`), which is itself +# the evidence that the image lacked it. Measured 2026-09-10 +# while wiring the skill-floor job: the guard could not be run +# before pushing — the same write → push → wait-for-CI loop that +# shellcheck was baked to shorten. 552 KB, and pulls ZERO extra +# packages under --no-install-recommends. RUN apt-get update && \ apt-get upgrade -y --no-install-recommends && \ apt-get install -y --no-install-recommends \ @@ -206,6 +219,7 @@ RUN apt-get update && \ bind9-dnsutils \ ldap-utils \ xxd \ + python3-yaml \ && ln -s /usr/bin/fdfind /usr/local/bin/fd \ && apt-get clean \ && rm -rf /var/lib/apt/lists/* diff --git a/scripts/check-skill-floor.sh b/scripts/check-skill-floor.sh new file mode 100755 index 0000000..71f9549 --- /dev/null +++ b/scripts/check-skill-floor.sh @@ -0,0 +1,166 @@ +#!/usr/bin/env bash +# check-skill-floor.sh — fail when the vendored pi-extensions skill snapshot in +# rootfs/ ("the floor") has drifted from the package repo it is a snapshot of. +# +# THE DEFECT THIS EXISTS TO CATCH, measured 2026-09-10. +# rootfs/usr/local/share/pi-devbox/skills/pi-extensions/ ships a vendored copy +# of the pi-extensions skill so the skill is ALWAYS present in the image. +# Dockerfile.variant then copies the freshly-cloned package copy OVER the served +# path at /usr/local/share/... — but it never writes back to the repo floor. So +# the floor only silently rots, and it had: 34284 B, untouched since fa04d20 +# (2026-07-30), while the package copy was 38973 B. Four copies existed with +# three different sizes. +# +# Why that is worse than ordinary staleness: the floor is a FALLBACK. The copy +# step is guarded by `if [ -f /opt/pi-extensions/skill/SKILL.md ]`, so a build +# where the package clone yields no skill/ keeps the vendored snapshot and still +# succeeds — green, with no manifest flag and no label saying which copy was +# served. The image would ship a July skill and nothing would say so. Keeping +# the floor fresh means that fallback is harmless instead of a silent regression. +# +# WHY A DIRECTORY HASH AND NOT `sha256sum SKILL.md`. +# The same pipeline Dockerfile.variant uses for skillset_snapshot_tree_sha256, +# and for the same documented reason: a file-only compare answers "did this one +# file change", not "is this the same skill". pi-extensions ships TWO files +# (SKILL.md + evaluate-extension-usage.py), so a sibling-file edit would pass a +# file-only check. If you change the pipeline here, change it there too. +# +# WHY GATING ON ANOTHER REPO IS PROPORTIONATE HERE, since that is normally a +# smell: this fires only when the package's skill/ DIRECTORY HASH changes, which +# is exactly and only when the floor has genuinely gone stale. pi-extensions +# commits that do not touch skill/ leave the hash alone and cannot turn this red. +# The repo is also anonymously clonable (verified 2026-09-10 with `git ls-remote` +# and no credentials), so this needs no secret and cannot break on token expiry. +# +# Exit codes — deliberately three, matching scripts/lint-shell.sh's philosophy +# that a gate which cannot run must not pass: +# 0 in sync (or the package legitimately has no skill/ at this ref) +# 1 DRIFT — the floor differs from the package +# 2 cannot run — no package copy could be obtained +set -euo pipefail + +REPO_ROOT=$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd) +FLOOR_DIR="${REPO_ROOT}/rootfs/usr/local/share/pi-devbox/skills/pi-extensions" + +# Defaults mirror Dockerfile.variant's ARGs so this checks what the build builds. +PI_EXTENSIONS_REPO="${PI_EXTENSIONS_REPO:-https://gitea.jordbo.se/joakimp/pi-extensions.git}" +PI_EXTENSIONS_REF="${PI_EXTENSIONS_REF:-main}" + +PACKAGE_DIR="" +WARN_ONLY=0 +TMPDIR_CLONE="" + +usage() { + cat <<'EOF' +Usage: scripts/check-skill-floor.sh [options] + + --package-dir DIR Compare against an existing skill directory instead of + cloning. In a devbox container use /opt/pi-extensions/skill + for a fully offline run. + --warn-only Report drift but exit 0 (advisory use, e.g. a local hook). + -h, --help This text. + +Environment: PI_EXTENSIONS_REPO, PI_EXTENSIONS_REF (default main) — both mirror +the Dockerfile.variant ARGs of the same name. +EOF +} + +while [ $# -gt 0 ]; do + case "$1" in + --package-dir) PACKAGE_DIR="${2:-}"; shift 2 ;; + --warn-only) WARN_ONLY=1; shift ;; + -h|--help) usage; exit 0 ;; + *) echo "::error::unknown argument: $1" >&2; usage >&2; exit 2 ;; + esac +done + +cleanup() { + if [ -n "$TMPDIR_CLONE" ]; then rm -rf "$TMPDIR_CLONE"; fi +} +trap cleanup EXIT + +# Identical to Dockerfile.variant's tree_sha256(): relative paths + per-file +# sha256 over a sorted `find`, folded into one digest. Deterministic, never +# readdir order. +tree_sha256() { + ( cd "$1" && find . -type f -print | LC_ALL=C sort | xargs -r sha256sum ) \ + 2>/dev/null | sha256sum | cut -d' ' -f1 +} + +if [ ! -d "$FLOOR_DIR" ]; then + echo "::error::floor directory is missing: ${FLOOR_DIR}" + echo "::error::rootfs/ is supposed to guarantee the skill is always in the image." + exit 2 +fi + +SOURCE_DESC="" +if [ -n "$PACKAGE_DIR" ]; then + if [ ! -d "$PACKAGE_DIR" ]; then + echo "::error::--package-dir does not exist: ${PACKAGE_DIR}" + exit 2 + fi + SOURCE_DESC="local directory ${PACKAGE_DIR}" +else + command -v git >/dev/null 2>&1 || { echo "::error::git not found; cannot obtain the package copy."; exit 2; } + TMPDIR_CLONE=$(mktemp -d) + # Fetch the single ref shallowly. `git fetch ` accepts a branch, a tag + # and (on Gitea) a reachable commit, which is why this is not `clone --branch` + # — CI resolves PI_EXTENSIONS_REF to a 40-hex SHA before the build. + if ! ( cd "$TMPDIR_CLONE" \ + && git init -q . \ + && git remote add origin "$PI_EXTENSIONS_REPO" \ + && git fetch -q --depth 1 origin "$PI_EXTENSIONS_REF" \ + && git checkout -q FETCH_HEAD ) 2>/dev/null; then + echo "::error::could not fetch ${PI_EXTENSIONS_REF} from ${PI_EXTENSIONS_REPO}" + echo "::error::Cannot determine whether the floor is stale, so this is exit 2, not a pass." + echo "::error::For an offline run, pass --package-dir /opt/pi-extensions/skill" + exit 2 + fi + PACKAGE_SHA=$( cd "$TMPDIR_CLONE" && git rev-parse --short HEAD ) + PACKAGE_DIR="${TMPDIR_CLONE}/skill" + SOURCE_DESC="${PI_EXTENSIONS_REPO} @ ${PI_EXTENSIONS_REF} (${PACKAGE_SHA})" +fi + +# A ref with no skill/ is the documented fallback case: Dockerfile.variant keeps +# the vendored snapshot and the build succeeds. Nothing to compare, so this is +# not drift — but it IS the exact condition under which the floor ships, so say +# so loudly rather than printing a silent green tick. +if [ ! -d "$PACKAGE_DIR" ]; then + echo "::warning::package has no skill/ at this ref — the vendored floor is what will ship." + echo " source : ${SOURCE_DESC}" + echo " floor : $(tree_sha256 "$FLOOR_DIR")" + exit 0 +fi + +FLOOR_HASH=$(tree_sha256 "$FLOOR_DIR") +PKG_HASH=$(tree_sha256 "$PACKAGE_DIR") + +if [ "$FLOOR_HASH" = "$PKG_HASH" ]; then + echo "OK: vendored pi-extensions floor matches the package." + echo " source : ${SOURCE_DESC}" + echo " tree_sha256: ${FLOOR_HASH}" + exit 0 +fi + +# `set -e` interacts badly with `[ … ] && x` as a bare statement, so both of +# these are explicit if-blocks rather than AND-lists. +LEVEL="error" +if [ "$WARN_ONLY" -eq 1 ]; then LEVEL="warning"; fi + +echo "::${LEVEL}::vendored pi-extensions skill floor has DRIFTED from the package." +echo " source : ${SOURCE_DESC}" +echo " floor tree_sha256 : ${FLOOR_HASH}" +echo " pkg tree_sha256 : ${PKG_HASH}" +echo "" +echo " per-file differences:" +diff -rq "$FLOOR_DIR" "$PACKAGE_DIR" 2>&1 | sed 's/^/ /' || true +echo "" +echo " Remedy — re-sync the floor and commit it:" +echo " cp -a /skill/. ${FLOOR_DIR}/" +echo " git add ${FLOOR_DIR#"${REPO_ROOT}/"} && git commit" +echo "" +echo " NOTE this forces one full base rebuild: base_tag hashes Dockerfile.base" +echo " + rootfs/, and that rebuild is what re-bakes the refreshed floor." + +if [ "$WARN_ONLY" -eq 1 ]; then exit 0; fi +exit 1