skills: add credential-incident-response, and assert it stays baked

Carries the facts a two-day credential incident produced, not the discipline:
probe the issuer FIRST (11 of 13 "exposed" credentials were already dead at the
provider, which cost five HTTP requests to learn and was never checked), the
403-vs-401 trap that scoped tokens introduce into liveness probes, revocation
beats deletion for anything already replicated, the three places a secret hides
in a Chroma palace (FTS content, metadata, raw bytes) in coverage order, scope
derivation from measured consumers, and this fleet's age store with its
single-recipient weakness.

Facts transfer between sessions; exhortations do not — hence a separate skill
for the domain knowledge and a one-line pointer in the always-loaded block.

Authored here, so baked is canonical and it is NOT added to skillset-owned.txt.
Skill dirs are picked up by a glob in entrypoint-user.sh, so no registration is
needed — verified rather than assumed, since an enumerated list would have left
the skill inert, a fitting failure given its subject. Three smoke assertions
extended so a future rebuild cannot silently drop it.

.env.example

@@ -146,6 +146,14 @@ GIT_USER_EMAIL=
 # Detection is automatic if the skillset lives at WORKSPACE_PATH/skillset.
 # SKILLSET_CONTAINER_PATH=
 
+# ── cli_utils (standalone commands from a mounted checkout) ──────────
+# If a cli_utils repo is mounted, the entrypoint symlinks its bin/ commands
+# into ~/.local/bin on every start, so they survive container recreate and
+# resolve in non-interactive shells too (docker exec, agent tool shells).
+# Detection is automatic at WORKSPACE_PATH/cli_utils (or one level below).
+# CLI_UTILS_CONTAINER_PATH=
+# CLI_UTILS_LINK=0                    # disable the linking entirely
+
 # ── Locale ───────────────────────────────────────────────────────────
 # LANG=sv_SE.UTF-8
 # LANGUAGE=sv_SE:sv

CHANGELOG.md

@@ -11,6 +11,173 @@ Pre-v1.0.0 tags followed the pi npm version (`v{pi_version}[letter]`).
 
 ---
 
+## v1.8.11 — 2026-08-27
+
+**Shell state that the writable layer eats on every recreate now gets rebuilt at
+start.** Two additions to `entrypoint-user.sh`, both idempotent, both silent
+no-ops when the thing they wire up is absent.
+
+**`cli_utils` commands are linked onto `PATH`.** If a `cli_utils` checkout is
+mounted, every executable in its `bin/` is symlinked into `~/.local/bin` at
+container start — `git-status-all`, `git-pull-all`, `devbox-sanity`,
+`pi-devbox-sanity`, `pi-session-repair`, `docker-clean`, `vpn-status`. Detection:
+`CLI_UTILS_CONTAINER_PATH` → `/workspace/cli_utils` → `$HOME/cli_utils` →
+`/workspace/*/cli_utils`; `CLI_UTILS_LINK=0` disables it.
+
+The reason this is an *image* concern and not the user's problem to re-solve: on a
+host, `cli_utils/install.sh` puts those commands on `PATH` by symlinking them into
+`~/.local/bin`, which is persistent there — and **ephemeral here**. Same installer,
+same repo, opposite durability, so the fix died on every `--force-recreate` and the
+next session was back to typing `/workspace/cli_utils/bin/git-status-all`. Running
+`install.sh` *inside* a container is the trap rather than the fix: it re-creates
+the same disposable state.
+
+**Symlinks rather than a `PATH` edit in an rc file, deliberately.** `~/.local/bin`
+is already ahead of `/usr/local/bin` in `ENV PATH`, so links resolve in
+**non-interactive** shells too — `docker exec <c> git-status-all`, agent tool
+shells, scripts. An rc-file `PATH` edit cannot reach those: `~/.bashrc` returns
+early when the shell is not interactive. Measured on tor-ms22 2026-08-27,
+`command -v git-status-all` failed in a non-interactive shell while succeeding in
+an interactive one, from exactly that asymmetry. Guards, because `~/.local/bin` is
+shared with other tooling: a real file is never clobbered, a symlink pointing
+somewhere else is never stolen, our own links are refreshed, and links into a
+`cli_utils/bin` whose target vanished are pruned — a dangling link on `PATH`
+reports "No such file or directory" and reads as a broken container rather than a
+removed script.
+
+**A per-device boot hook: `~/.config/devbox-shell/init.sh`.** If the host provides
+one, it runs once at start with output to `~/.pi/agent/devbox-init.log`. That
+directory is the host-owned bind-mount already sourced into every interactive
+shell by `/etc/skel-devbox/.bash_aliases`, so this is its boot-time twin — the
+same ownership and the same persistence, but running *before any shell*, which is
+what non-interactive fixups (symlinks, directories, one-off migrations) need. **It
+introduces no new trust boundary**: that path is already arbitrary code from the
+same owner; only *when* it runs is new. Invoked as `bash <file>`, never sourced,
+and its exit status is ignored — a hook must not be able to mutate the
+entrypoint's own shell state or stop a container from starting.
+
+With the hook in place, the next "can this run on every recreate?" question needs
+no image change at all — which is the point, given what the next paragraph costs.
+
+**This moves the base hash.** `base-decide` folds `cat entrypoint.sh
+entrypoint-user.sh` into it, so this change forces the ~40-minute base rebuild at
+the next tag whether or not anything else in the base moved. It is a rider, not a
+reason to tag.
+
+**How it was validated, since CI cannot.** `docker-publish.yml` runs only on
+`push: tags: v*`, and `lint.yml` runs `actionlint` over workflow `run:` steps —
+neither one executes `entrypoint-user.sh`. So both sections were extracted and run
+against fixtures in a throwaway `$HOME` before commit: real file not clobbered,
+foreign symlink respected, stale link pruned, new command picked up, second run
+byte-identical, `CLI_UTILS_LINK=0` honoured, and "no `cli_utils` anywhere" a silent
+`exit 0`. Then run for real in a live v1.8.10 container, after which
+`command -v git-status-all` resolved in a *non-interactive* shell. No
+`smoke-test.sh` assertion was added on purpose: the positive path needs a
+`/workspace` mount that smoke does not have, and asserting it there would repeat
+the v1.8.0 mistake of a smoke assertion written against a stage that does not
+exist at run time. `workflow_dispatch` with `smoke_only` remains the way to
+exercise this against `HEAD` before a tag.
+
+**Also carried by the floating `mempalace-toolkit` main ref** (resolved at build
+time, not by a pi-devbox commit — `MEMPALACE_TOOLKIT_REF=main`):
+
+**A scrubbed re-export of a dormant session could silently never reach the
+palace host.** `bin/mempalace-pi-session` ships to the palace with
+`rsync -a --update`, and the stage file's mtime is deliberately the SOURCE
+transcript's mtime (`os.utime()`, "preserve session mtime for dedup
+stability"). Re-exporting a session that has not been appended to since its
+last ship therefore produces a mtime that is *not newer* than the receiver's —
+exactly the case a redactor upgrade needs to ship, since content differs while
+mtime does not. `--update` reported success and sent nothing. Found and
+patched by `pi@mbp-m1-2020` (mempalace-toolkit `a361b71`): `--update` →
+`--checksum`, which compares content and ignores size/mtime entirely.
+Dropping `--update` outright was considered and rejected — rsync's default
+quick check already transfers on a size difference alone, which would have
+masked the *next* instance of this (a redaction whose placeholder happens to
+match the secret's length) as fixed. `os.utime()` is untouched; its backdating
+is a separate, load-bearing design call for dedup stability. New regression
+test, `scripts/test-rsync-ship-idempotency.sh`, runs fully offline (a local
+rsync destination exercises the same size/mtime/checksum comparison as the ssh
+transfer) and is built to *discriminate*: it must fail against `--update` and
+pass against `--checksum,` not merely exercise the code path — the first draft
+of the test used fixture strings of different lengths and passed for the wrong
+reason (rsync's quick check transfers on size difference alone regardless of
+`--update`), which is the same trap the patch itself was written to avoid.
+**Acceptance line for this class of change going forward:** "receiver sha256
+matches sender for every staged file", not "local stage is clean" — a clean
+local stage says nothing about what a dormant session already sent.
+
+**An event addressed to an identity no session runs as is delivered to
+nobody, and this fleet has now hit it three separate ways.** RFC 003 gains
+§7.13 and open-decision 10 (mempalace-toolkit `21023e7`, docs only, no image
+behaviour change): the owed-set derivation — the log's only push channel — is
+keyed on `to_agent`, and a reply is always addressed back to whatever string
+the *original writer* put in `from_agent`. Nothing validates that string
+against a live session identity, so authoring under a synthetic or foreign
+name makes every reply to that event write-only. Measured cost this cycle: a
+directed ask planted under a synthetic sender drew a correct reply containing
+an urgent security finding, and it sat unread for ~2h20m, found only because a
+human asked whether mail had arrived. Permitted exception, unchanged: a
+synthetic sender is fine for a deliberate control experiment, provided the
+body names the real identity to reply to.
+
+**Also carried by the live `skillset` mount** (each device's own clone, not
+baked — except the `mempalace` skill's fallback snapshot, re-vendored below):
+
+**The mermaid-diagrams checker's cut gate moved from client pixels to a
+per-SVG user-space unit.** `CUT_PX` was calibrated against one live page at
+one render scale; sweeping `--viewport` 500→1600 on an *unchanged* document
+moved the worst overflow −1.0px → −3.0px, near-proportional to the viewport,
+i.e. a constant geometric overflow viewed through a changing scale. `cutU =
+cutPx / scale` (scale taken per-SVG, never a page average — one page mixes
+scales 0.643–0.988) recovers that invariant: the sweep now collapses to
+exactly −3.0u at every width. Re-deriving the threshold against the live host
+surfaced a real false negative the old pixel gate had: a label at
+`cutPx=0.4, scale=0.678` read as healthy under `CUT_PX=0.5` but is `0.59u` —
+a genuine cut hiding behind a compressed render scale. `CUT_U` stays `0.5`;
+`cutPx` and `scale` are still printed on every issue so a devtools ruler still
+confirms the number on the actual page. A new, explicitly-deferred finding
+from the same review: `cut` only measures vertically, so an unbreakable token
+wider than its box (a long URL, a `snake_case` identifier) is invisible to
+soft-wrap, tall, *and* cut simultaneously — filed as a backlog item, not
+implemented, pending a fifth acceptance control.
+
+**The `from_agent`-identity finding above is also now in the `mempalace`
+skill itself** ("Writing to another machine", and Anti-Patterns), and the
+baked fallback snapshot of that skill was refreshed to match
+(`vendor-mempalace-skill.sh`, `6eb20af` → `a12fe5e`) — sanctioned to skip on
+its own (`--check` reported stale-but-truthful), done anyway because this
+release's point is getting today's fixes live, and the base rebuild below was
+already forced regardless.
+
+### Dependency audit (2026-08-27)
+
+Every component checked against upstream by direct command, not assumed:
+
+| Component | Baked in v1.8.10 | Upstream now | Action |
+|---|---|---|---|
+| **mempalace-toolkit** | `b2b50af` | **`21023e7`** | ships the rsync ship-fix + RFC 003 §7.13 (both above) |
+| **skillset** (mempalace fallback snapshot) | `6eb20af` | **`a12fe5e`** | re-vendored (above); live-mounted devices already had it |
+| pi | `0.84.3` (pinned) | `0.84.3` is npm latest | none |
+| mempalace | `3.8.0` (pinned) | `3.8.0` is PyPI latest | none |
+| pi-atelier | `v0.8.2` (pinned) | `v0.8.2` highest tag | none |
+| pi-studio (studio variant) | `v0.9.52` | `v0.9.52` — `main`'s commit and the tag's commit are identical (0 either direction) | none |
+| pi-toolkit | `0e1369e` | `0e1369e` (local clone HEAD == `origin/main`) | none |
+| pi-extensions | `2022887` | `2022887` (local clone HEAD == `origin/main`) | none |
+| pi-fork | `bf702b4` | `bf702b4` | none |
+| pi-observational-memory | `ce9fc98` | `ce9fc98` | none |
+
+pi-toolkit / pi-extensions checked against their actual Gitea origin (the
+Dockerfile's `PI_TOOLKIT_REPO` / `PI_EXTENSIONS_REPO`), not a GitHub mirror —
+querying `api.github.com` for those two returned nothing (rate-limited or
+blocked; not investigated, the local clones are the source of truth anyway).
+No SHA above is fork-supplied; a fork inventing plausible-looking upstream SHAs
+is a recorded failure mode (v1.8.9), so every value here came from
+`git ls-remote`, a local clone's own `origin/HEAD`, `npm view`/registry JSON,
+or the PyPI JSON API, run directly.
+
+---
+
 ## v1.8.10 — 2026-08-27
 
 **This tag exists to deploy a fix and a safety net that are currently running on

Dockerfile.variant

@@ -305,7 +305,7 @@ ARG MEMPALACE_TOOLKIT_REF=main
 # no ~67-minute base rebuild. (scripts/check-base-hash.sh scans only
 # Dockerfile.base, so no folding into the base hash is required — nor would
 # it be correct, since this ARG changes nothing about the base's contents.)
-ARG SKILLSET_SNAPSHOT_REF=6eb20af181f0147cb8c1377f6e36a6a47a68e8e5
+ARG SKILLSET_SNAPSHOT_REF=a12fe5ecc71e60feb24791e3e33571105f1afba7
 
 # Dockerfile.base sets description="pi-devbox — base image (variant-independent)"
 # and every variant INHERITS it, so both published images used to advertise

README.md

@@ -538,6 +538,35 @@ to refresh.
 Anything not on a volume is on the writable layer and is lost on
 container recreate.
 
+### Rebuilding ephemeral shell state at start
+
+Two entrypoint steps put back the kind of state that the writable layer eats, so a
+recreate does not cost you a manual re-install:
+
+- **`cli_utils` commands.** If a `cli_utils` checkout is mounted, every
+  executable in its `bin/` is symlinked into `~/.local/bin` on start, so
+  `git-status-all` and friends are on `PATH` without a path prefix. Detection:
+  `CLI_UTILS_CONTAINER_PATH` → `/workspace/cli_utils` → `$HOME/cli_utils` →
+  `/workspace/*/cli_utils`. Set `CLI_UTILS_LINK=0` to disable. Existing real files
+  in `~/.local/bin` and symlinks pointing elsewhere are left alone, so a
+  deliberate override still wins; links whose target disappeared are pruned.
+  Do **not** run a host installer's `install.sh` inside the container to achieve
+  this — it writes to the ephemeral home and dies on the next recreate.
+- **A per-device boot hook.** If `~/.config/devbox-shell/init.sh` exists it is run
+  once at start (`bash`, never sourced, exit status ignored), with output in
+  `~/.pi/agent/devbox-init.log`. `~/.config/devbox-shell/` is the host-owned
+  bind-mount whose `bash_aliases` is already sourced into every interactive shell,
+  so a hook there persists across recreates with no image change. Use it for
+  fixups that must exist *before any shell* — symlinks, directories, one-off
+  migrations.
+
+The distinction that decides which mechanism you want: `~/.local/bin` is on `ENV
+PATH`, so symlinks there work in **non-interactive** shells too (`docker exec <c>
+<cmd>`, agent tool shells, scripts). A `PATH` edit in `bash_aliases` reaches only
+*interactive* shells, because `~/.bashrc` returns early when non-interactive —
+which is also why shell **functions** (fzf helpers and the like) can only come
+from the sourced file, never from a symlink.
+
 ## MemPalace integration
 
 MemPalace is installed in the base image and pre-warmed with the

entrypoint-user.sh

@@ -188,6 +188,111 @@ if [ "${MEMPALACE_FEED:-1}" != "0" ] && [ -n "$MEMPALACE_FEEDER" ]; then
   fi
 fi
 
+# ── cli_utils: link workspace bin/ commands onto PATH ────────────────
+# Standalone commands from a mounted cli_utils checkout (git-status-all,
+# git-pull-all, devbox-sanity, pi-session-repair, ...) live in <repo>/bin. On a
+# host they reach PATH via cli_utils' own install.sh, whose install_bin step
+# symlinks them into ~/.local/bin — but that home is on the container's WRITABLE
+# LAYER, so every recreate loses them and the human is back to typing
+# /workspace/cli_utils/bin/git-status-all. This is the container equivalent of
+# that install step, re-run at every start.
+#
+# WHY SYMLINKS RATHER THAN A PATH EDIT IN AN rc FILE: ~/.local/bin is already
+# ahead of /usr/local/bin in ENV PATH (Dockerfile.base), so links here resolve in
+# NON-interactive shells too — `docker exec <c> git-status-all`, agent tool
+# shells, scripts. An rc-file PATH edit cannot reach those, because ~/.bashrc
+# returns early when the shell is not interactive. Measured 2026-08-27 on
+# tor-ms22: `command -v git-status-all` failed in a non-interactive shell while
+# working in an interactive one, from exactly that asymmetry.
+#
+# Detection order (first hit wins):
+#   1. CLI_UTILS_CONTAINER_PATH   explicit, for non-standard layouts
+#   2. /workspace/cli_utils       repo directly in the workspace root
+#   3. $HOME/cli_utils           dedicated mount
+#   4. /workspace/*/cli_utils     workspace root holds several repo groups
+# CLI_UTILS_LINK=0 disables. Absent repo = silent no-op, which is the common
+# case for anyone who does not use cli_utils.
+if [ "${CLI_UTILS_LINK:-1}" != "0" ]; then
+  CLI_UTILS_BIN=""
+  if [ -n "${CLI_UTILS_CONTAINER_PATH:-}" ] && [ -d "${CLI_UTILS_CONTAINER_PATH}/bin" ]; then
+    CLI_UTILS_BIN="${CLI_UTILS_CONTAINER_PATH}/bin"
+  elif [ -d /workspace/cli_utils/bin ]; then
+    CLI_UTILS_BIN=/workspace/cli_utils/bin
+  elif [ -d "$HOME/cli_utils/bin" ]; then
+    CLI_UTILS_BIN="$HOME/cli_utils/bin"
+  else
+    # `if` bodies, not `&&` chains: under `set -e` a loop whose LAST command is a
+    # false test exits non-zero and would abort the entrypoint. With no match the
+    # glob stays literal, so that is the normal case on any machine without this
+    # repo — i.e. the bug would have been "container will not start", not "links
+    # missing".
+    for _cu in /workspace/*/cli_utils/bin; do
+      if [ -d "$_cu" ]; then
+        CLI_UTILS_BIN="$_cu"
+        break
+      fi
+    done
+    unset _cu
+  fi
+
+  if [ -n "$CLI_UTILS_BIN" ]; then
+    mkdir -p "$HOME/.local/bin" 2>/dev/null || true
+    # Never clobber a real file, and never steal a link that points elsewhere: a
+    # deliberate user override in ~/.local/bin must win, and silently shadowing
+    # an image-provided command is worse than the missing command.
+    for _f in "$CLI_UTILS_BIN"/*; do
+      if [ ! -f "$_f" ] || [ ! -x "$_f" ]; then
+        continue
+      fi
+      _link="$HOME/.local/bin/$(basename "$_f")"
+      if [ -e "$_link" ] && [ ! -L "$_link" ]; then
+        continue
+      fi
+      if [ -L "$_link" ]; then
+        case "$(readlink "$_link")" in
+          "$CLI_UTILS_BIN"/*) ;;
+          *) continue ;;
+        esac
+      fi
+      ln -sf "$_f" "$_link" 2>/dev/null || true
+    done
+    # Prune links we own whose target vanished (command renamed, repo moved),
+    # mirroring the skillset deploy's --prune-stale. A dangling link on PATH
+    # reports "No such file or directory" for a command that simply no longer
+    # exists, which reads as a broken container rather than a removed script.
+    for _link in "$HOME/.local/bin"/*; do
+      [ -L "$_link" ] || continue
+      case "$(readlink "$_link")" in
+        */cli_utils/bin/*) [ -e "$_link" ] || rm -f "$_link" ;;
+      esac
+    done
+    unset _f _link
+  fi
+  unset CLI_UTILS_BIN
+fi
+
+# ── Per-device boot hook ─────────────────────────────────────────────
+# Runs ~/.config/devbox-shell/init.sh if the host provides one. That directory is
+# the host-owned, bind-mounted shell-sharing dir (see "Volumes and persistence"),
+# so a hook placed there survives every recreate WITHOUT an image change — the
+# boot-time twin of the interactive bridge in /etc/skel-devbox/.bash_aliases,
+# which sources ~/.config/devbox-shell/bash_aliases for every interactive shell.
+#
+# NO NEW TRUST BOUNDARY: that same directory is already sourced into every
+# interactive shell, i.e. it is already arbitrary code from the same owner. What
+# is new is only WHEN it runs — once at start, before any shell — which is what
+# non-interactive fixups (symlinks, dirs, one-off migrations) need.
+#
+# Deliberately `bash <file>`, not `.` — a hook must not be able to mutate this
+# entrypoint's own shell state, and its exit status must not matter. Output goes
+# to a log rather than the container's start output, so a chatty hook cannot
+# masquerade as a startup error.
+if [ -r "$HOME/.config/devbox-shell/init.sh" ]; then
+  mkdir -p "$HOME/.pi/agent" 2>/dev/null || true
+  bash "$HOME/.config/devbox-shell/init.sh" \
+    >"$HOME/.pi/agent/devbox-init.log" 2>&1 || true
+fi
+
 # ── Git config defaults ──────────────────────────────────────────────
 if [ -n "${GIT_USER_NAME:-}" ] && ! git config --global user.name &>/dev/null; then
   git config --global user.name "$GIT_USER_NAME"

rootfs/usr/local/share/pi-devbox/pi-global-AGENTS.append.md

@@ -70,3 +70,41 @@ rather than merely confusing you:
   local disk, so `mempalace search` can return older and different results than
   the MCP tools while both look correct. Use the MCP tools for the central
   palace; the CLI only for a local one.
+
+## Before you file a finding: second measurement, different route
+
+This is here rather than in a skill because it has to fire *without* a matching
+task description, and because the version of it that lived only in a skill was
+violated five times in one session by an agent that had the skill available.
+
+**Any claim you are about to record as fact — in a drawer, a diary entry, a
+coordination event, or a report to the user — needs a second measurement taken
+by a different route.** Not a re-read of your reasoning: re-reading has caught
+zero of these. A disagreeing measurement has caught all of them.
+
+The two shapes that get filed as fact and are not:
+
+- **A negative result** (`401`, connection refused, zero rows, "not found") is
+  first a claim about *your filter*, not about the world. Wrong host, wrong port,
+  wrong table, capped output.
+- **A positive result** proves only what your command *actually asked*. An SSH
+  handshake can succeed against the wrong host (`ssh -G` tells you which rule
+  captured the name); a `401` can be a real answer from an issuer that never
+  minted the credential.
+
+Cheapest habit that works: **write the expected result next to each check before
+running it**, then diff. Expectations declared up front turn a silent wrong
+assumption into a visible mismatch. And if you cannot think of a second route to
+the same fact, you do not have a finding — you have a hypothesis, so label it as
+one.
+
+## Handling an exposed credential
+
+If a task touches a leaked secret, a token rotation, "is this credential still
+live?", whether to delete stored content, or which scopes a new token needs:
+**read `~/.agents/skills/credential-incident-response/SKILL.md` first.** One rule
+is load-bearing enough to state here: **probe the issuing provider before doing
+anything else** — most "exposed" credentials in a long-lived fleet are already
+dead, and the ones that are live are often far more privileged than assumed.
+Severity first, cleanup second, and prefer **revocation over deletion** for
+anything already replicated.

rootfs/usr/local/share/pi-devbox/skills/VENDORED.md

@@ -9,6 +9,7 @@ one", which was a bug).
 | skill | owner | how it gets here |
 |-------|-------|------------------|
 | `pi-devbox-environment` | pi-devbox (this repo) | authored here; the canonical copy |
+| `credential-incident-response` | pi-devbox (this repo) | authored here; the canonical copy |
 | `pi-extensions` | the `pi-extensions` package repo (`skill/`) | **vendored fallback** + refreshed at build |
 | `mempalace` | the `skillset` repo | **vendored fallback** (snapshot only) |
 

rootfs/usr/local/share/pi-devbox/skills/credential-incident-response/SKILL.md

@@ -0,0 +1,157 @@
+---
+name: credential-incident-response
+description: >-
+  Respond correctly when a live credential is found somewhere it should not be —
+  in a chat transcript, a MemPalace drawer, a log, a git-tracked config, or an
+  agent-authored note. Load this whenever a task involves a leaked/exposed
+  secret, a token rotation, a "is this credential still live?" question, deciding
+  whether to delete or scrub stored content, or choosing scopes for a new API
+  token. Covers the mandatory order of operations (probe the issuer FIRST —
+  severity before cleanliness), leak-free credential identity via sha256[:8]
+  fingerprints, why revocation beats deletion for anything already replicated,
+  deriving least-privilege scopes from measured consumers instead of guessing,
+  where this fleet's secrets actually live (age-encrypted .env.age in
+  docker-compose-repo, plus gitignored plaintext .env drift), the three places a
+  secret hides in a Chroma palace, and the exposures that rotation does NOT fix.
+---
+
+# Credential incident response
+
+A leaked credential is a **severity** question before it is a cleanliness
+question. Two days of scrubbing, redaction plumbing and deletion planning were
+once spent on a set of 13 credentials of which **11 were already dead at the
+provider** — a fact that cost five HTTP requests to establish and was never
+checked. Meanwhile the two live ones turned out to be instance-owner **admin**
+tokens, which nobody had looked at either.
+
+## 1. Order of operations — do not reorder this
+
+1. **Is it still accepted?** Probe the issuing provider. Dead credential →
+   hygiene item, stop panicking. Live → incident, continue.
+2. **What can it do?** Read the identity back. `is_admin`, `id=1`, scopes,
+   which account. A read-only repo token and an instance-owner admin token are
+   not the same finding.
+3. **What consumes it?** Grep for real consumers before assuming breakage.
+4. **Where does it live?** Enumerate copies (store, palace, transcripts, git).
+5. **Then** rotate/revoke, and only then consider cleanup.
+
+Doing 4→3→1 in reverse produces confident, wrong severity calls and wasted
+cleanup. If you only have time for one step, do step 1.
+
+## 2. Leak-free identity: fingerprint, never the value
+
+Publishing an 8-hex fingerprint lets you compare a credential across machines,
+files, drawers and peers without ever materialising the secret. Same formula as
+`mempalace_redact.py`:
+
+```sh
+printf '%s' "$SECRET" | sha256sum | cut -c1-8   # printf, NOT echo (no newline)
+printf '%s' 'test' | sha256sum | cut -c1-8      # self-test -> 9f86d081
+```
+
+Report as `(variable, fp, length)`. Equal fingerprints across hosts prove a
+shared credential; that is usually the important part. **Never** paste a live
+value into a search query, a palace drawer, an event body, or a chat message —
+in an agent context your own tool output is itself captured and re-filed.
+
+## 3. Liveness probes, and the trap that scoping creates
+
+```sh
+# Gitea
+curl -sS -m 10 -o /dev/null -w '%{http_code}\n' -H "Authorization: token $T" \
+  "$GITEA_HOST/api/v1/repos/<owner>/<repo>/actions/runs?limit=1"
+# GitHub
+curl -sS -m 10 -o /dev/null -w '%{http_code}\n' -H "Authorization: token $T" \
+  https://api.github.com/user
+```
+
+- `200` live · `401` revoked/invalid · **`403` = wrong question, not a dead token**
+- **Probe the issuer that minted it.** A 401 from an unrelated instance says
+  nothing. Resolve the host from config (`GITEA_EGL_HOST` etc.), do not assume.
+- **Under scoped tokens, `/api/v1/user` returns 403 for a perfectly live token**
+  unless `user` scope was granted. So it cannot distinguish *revoked* from
+  *merely scoped*. Use a **repository route the token is authorised for**.
+- Verify **both directions** after a rotation: old → 401, new → 200. The second
+  check is what catches "deleted the wrong token".
+- Port/scheme come from config, not habit: one instance here is
+  `http://gitea.egl.lan:3000` — plain HTTP, with 443 refused.
+
+## 4. Revocation beats deletion — the load-bearing rule
+
+Once revoked, stored copies are **inert**; you may leave them. Deleting them is
+best-effort over an *unbounded* copy set: FTS shadow rows, feed inbox `.jsonl`
+files on every host, sqlite free pages after the delete, mesh replicas that
+already synced, and backups. **Revocation invalidates every copy everywhere at
+once, including copies nobody enumerated.**
+
+So: **rotate + revoke first.** Treat drawer deletion as optional hygiene, never
+as the remedy. Then record the retired fingerprints as *known-dead* so the next
+census recognises them instead of reopening the investigation.
+
+Corollary: never reach for `mempalace_sync` or a bulk `delete_by_source` on a
+shared palace as incident response. High blast radius, low actual benefit.
+
+## 5. Finding a secret in a Chroma palace — three targets, in this order
+
+1. `embedding_fulltext_search_content.c0` — **where document text actually is**
+2. `embedding_metadata.string_value` — metadata fields only
+3. raw byte scan of every `*.sqlite3` — backstop, covers FTS pages and free space
+
+Scanning only (2) is the classic false clean: hundreds of thousands of rows,
+zero hits, and the secret sitting in (1) the whole time. Semantic search proves
+nothing about absence — it returns top-k. For completeness, enumerate by filing
+window (`list_drawers(since=T, before=T+1m)`), since one mine shares a minute.
+
+Value-agnostic sweeps (uuid / 40-hex / `NAME=VALUE`) drown in false positives at
+fleet scale — 608 candidates, mostly session UUIDs and git SHAs. Name-anchoring
+plus entropy plus provenance, applied to **document text**, is what works.
+
+## 6. Choosing scopes: derive them from measured consumers
+
+Before creating a replacement token, find out what actually uses it:
+
+```sh
+git -C <repo> remote get-url origin        # ssh:// ? then git needs NO token
+git config --global --list | grep -iE 'credential|insteadof'   # and no helper?
+grep -rhoE 'api/v1/[A-Za-z0-9/{}$_.-]+' <consumers>  | sort -u  # exact routes
+grep -rhoE '\-X [A-Z]+' <consumers>                             # any writes?
+```
+
+Real outcome here: git used SSH keys throughout, and the token's only consumer
+read three CI-run routes with `GET`. So `repository: Read` and nothing else
+replaced two admin tokens. **Scoping shrinks the blast radius of the next leak
+far more than any redaction pipeline does** — a read-only token in a transcript
+is a hygiene event, not an instance compromise.
+
+Then prove the scope with an acceptance suite that declares expectations first:
+must-work routes → `200`; `/admin/*`, `/user`, `/user/repos` → `403`.
+
+## 7. What rotation does *not* fix
+
+- **A cleartext channel.** If the endpoint is `http://`, the *new* token is
+  exposed identically from first use. Raise TLS separately.
+- **Git history.** A secret committed and pushed cannot be fixed by any store or
+  palace operation — it needs rotation *and* history surgery.
+- **Agent-authored content.** Stage-write redactors see transcripts only, never
+  `add_drawer` / `checkpoint` / `diary_write` output. Never type a secret into
+  the palace yourself; nothing downstream will catch it.
+- **Plaintext/encrypted drift.** Gitignored plaintext `.env` files go stale while
+  `.env.age` moves on, so old values linger on disk (and in backups) long after
+  rotation. They are a common source of "mystery" fingerprints in a census.
+
+## 8. This fleet's secret store (verify, do not assume)
+
+- All `*.env.age` live in **one** repo: `joakimp/docker-compose-repo`. `myconfigs`
+  has none.
+- Every `.age` file has **one X25519 recipient** — a single key tracked in
+  `myconfigs` under git-crypt. Unlocking git-crypt therefore decrypts the entire
+  fleet's secrets, including hosts you have no access to. The age layer adds no
+  isolation beyond git-crypt.
+- Flow: `./fetch-secrets.sh <host>` (decrypt → `.env`) → edit → `./encrypt-secrets.sh <host>`
+  → commit → push → `docker compose up -d --force-recreate`.
+- **Always pass the host argument** to `encrypt-secrets.sh`. Bare, it walks the
+  whole tree and re-encrypts every `.env` it finds, re-nonced, including stale
+  ones — silently rolling back other hosts' secrets.
+- After any re-encrypt, check the header still shows exactly **one X25519
+  recipient**; a hand-rolled `age -r` locks the rest of the fleet out, and the
+  failure only appears on another machine, later.

rootfs/usr/local/share/pi-devbox/skills/mempalace/SKILL.md

@@ -428,10 +428,56 @@ An obligation you never agreed to is noise, so the sender states it:
 | `to_agent="*"` (any status) | broadcast FYI | nothing |
 | any other status (`ready`, `applied`, `blocked`, …) | a statement of fact | nothing |
 
+**That table says what you *owe*. Delivery is stricter, and the difference bites:
+the mailbox is an obligation channel, not a news channel.** Mailbox candidates are
+drawn with `status="open"`, so an event carrying any **terminal** status
+(`applied`, `superseded`, `failed`, `blocked`) is never a candidate — *whoever it
+is addressed to*. A `task.reply` written to a named machine to share a finding is
+delivered to nobody, ever, and neither is any `event_ack`. It sits in the log
+until somebody reads the log.
+
+So the most natural inter-machine message — *"here is something you should
+know"* — is exactly the shape that gets no delivery. Pick deliberately:
+
+| You want the peer to… | Write |
+|---|---|
+| **do something**, and you need it tracked until done | directed `status="open"` ask, with a `correlation_id` |
+| **know something**, no response needed | terminal-status event **plus a drawer** — the drawer is what actually reaches them, via search |
+
+What does **not** work is a terminal report plus an expectation of attention.
+Measured 2026-08-26: a detailed report addressed to `pi@<peer>` with
+`status="applied"` went unread for two and a half hours until the operator quoted
+the event id by hand, with the mailbox working correctly the whole time. Full
+mechanism in the toolkit's `docs/rfc-003-coordination-log.md` §7.12.
+
+One more timing fact, because it looks like negligence and is not: a delivered
+ask is queued into the agent's **next turn** (`deliverAs: "steer"`, deliberately
+no `triggerTurn`), and the poll fires when the agent is *idle*. Between delivery
+and the next turn no inference runs, so **a human starting a turn is the
+trigger** (§7.11). An agent that "has not reacted" has usually not been running.
+
 Ack with `mempalace_event_ack(event_id=…, from_agent="<you>", status=…)`. It
 **appends a new event** and never mutates the original; the correlation id is
 copied for you, and `metadata.ack_of` is set to the event you answered.
 
+**Claiming, and what it does not do.** `status="claimed"` announces that you have
+picked work up. Nothing requires it — a directed open ask owes "an ack *or* a
+reply", and finishing the work is a complete answer. Do it anyway when the work is
+long or the machine is unreliable, because it is the only thing that later
+distinguishes *nobody started this* from *someone started and their container
+died mid-task*. Be clear about its limits, both of which follow from candidacy
+requiring exactly `status="open"`:
+
+- **It does not notify the requester.** `claimed` is not `open`, so a claim is no
+  more deliverable than a finished report is (see the delivery table above). Its
+  reader is whoever pulls the log.
+- **It does not quiet your own mailbox.** The ask stays owed until a *terminal*
+  event of yours joins it, so a claimed-then-silent thread keeps resurfacing —
+  correctly.
+
+Prefer a prompt terminal reply over a claim plus a long silence; claim *in
+addition*, when the gap between pickup and finish is where a machine might die.
+
 #### What you actually owe — derive it, do not read it off `status`
 
 The log is append-only and `status` is written **once**, so it is an honest
@@ -502,6 +548,17 @@ Two consequences worth internalising:
 - **Address the stamped name you actually saw** in a `from_agent` field, e.g.
   `pi@tor-ms22`. A bare `pi` reaches nobody's mailbox once stamping is live, and
   older events in the log still carry bare names — do not copy them.
+- **The rule runs in reverse too: what you put in YOUR OWN `from_agent` decides
+  where every reply to your event goes.** Nothing stops you writing a synthetic
+  or borrowed identity there, and a reply is always addressed back to exactly
+  that string — so if no live session ever runs as it, the reply is stored,
+  searchable, and delivered to no one. Measured cost: a directed ask sent under
+  a synthetic sender got two correct replies, one of them an urgent security
+  finding, and both sat unread for ~2h20m because nobody's mailbox was that
+  identity (RFC 003 §7.13). Authoring under a synthetic name is fine for a
+  deliberate control experiment — this fleet does it on purpose — but then
+  **name the real identity to reply to inside the body**, because the address
+  line is not a safe place to also carry provenance.
 - **Use `status="open"` only when you truly need an answer.** It places an
   obligation on another machine.
 - **Never broadcast an ask.** `to_agent="*"` + `status="open"` obliges everyone
@@ -513,7 +570,8 @@ Two consequences worth internalising:
   be matched to it at all.
 - **Corrections are new events, never edits.** Say explicitly what you retract
   and name the id — drawer or event — that carried the withdrawn claim.
-- **Put a retraction where the reader will look.** An event reaches a live agent;
+- **Put a retraction where the reader will look.** A *directed open ask* reaches a
+  live agent's mailbox; a **terminal-status event reaches no mailbox at all**, and
   a *drawer* is what a future semantic search finds. If you filed advice as a
   drawer and later withdraw it, file the withdrawal as a drawer too — otherwise
   the next agent finds your original confident advice and no trace of the
@@ -600,4 +658,5 @@ Entity-relationship triples with temporal validity. Query with `mempalace_kg_que
 - **Don't treat the palace as a task list.** It's for knowledge and context, not todos.
 - **Don't broadcast an ask, and don't leave one unanswered.** On a shared palace, `to_agent="*"` + `status="open"` obliges every machine and therefore none of them. And don't expect acking to tidy your mailbox: `status` is immutable, so the event keeps matching either way — what a terminal reply buys you is that the *derived* owed set (see *What you actually owe*) stops counting it. Leave asks unanswered and that set only grows, until everyone learns to stop looking. "Seen, not doing it" is a complete answer — silence is not.
 - **Don't assume you would have heard.** Nothing pushes another machine's message into your session. If you did not run the mailbox query at wake-up, a correction addressed to you by name can sit unread while you confidently rebuild the thing it warned you about.
+- **Don't author an ask under an identity nobody runs as, including your own throwaway labels.** The failure is symmetric to the one above: it is not that you missed a message, it is that nothing could ever have delivered the reply to you, because you addressed it at a name instead of an agent. If you must use a synthetic sender for a control or an experiment, say inside the body who should actually receive the reply.
 - **Don't invent provenance metadata, and don't hand-stamp it either.** An earlier version of this list told you to set `added_by="<harness>@<device>"` by hand; that instruction has been withdrawn, because RFC 001 §7.3.2 places provenance at the client/server boundary and the pi bridge now does it uniformly (see *Provenance is stamped for you* above) — but the withdrawal only holds where the bridge is live, so run the one-line check in that bullet first; on an older image hand-stamping is still the only signal a hand-filed drawer gets. DO NOT invent values for the palace's own metadata fields (`device`, `agent_kind`, `origin_device`): those are stamped by infrastructure that also records *how* each was determined, and a fabricated value is worse than none because it silently corrupts a future merge. DO pass `source_drawer_id` on `kg_add`. And never put a machine name in a diary's `agent_name` — it becomes the wing name and hides your entries from `diary_read`.

rootfs/usr/local/share/pi-devbox/skills/pi-devbox-environment/SKILL.md

@@ -143,6 +143,9 @@ mine:
 | "`tor-ms22` is not in the SSH config" | `grep … \| head -20` — the entry was at **line 454**. `~/.ssh/config` here is ~500 lines. |
 | "the Docker host has no `docker`" | non-interactive SSH `PATH` lacks `/usr/local/bin` (§2, §3). It was at `/usr/local/bin/docker`. |
 | "no ControlMaster is running" | pattern `ssh ` (trailing space) cannot match a master: those processes **rename themselves** to `ssh: <controlpath> [mux]`. |
+| "the credential is not in the palace" | scanned `embedding_metadata.string_value` only. Drawer **text** lives in `embedding_fulltext_search_content.c0`; 554k metadata rows proved nothing. |
+| "this token is dead — 401" | probed it against the **wrong issuer**. A 401 from an instance that never issued the credential is not evidence about the credential. |
+| "that host is unreachable, can't test" | tried ports 443 and 80. It was on **3000**, and the env var I already held (`GITEA_EGL_HOST`) stated the scheme and port. |
 
 Habits that would have caught all three:
 
@@ -157,8 +160,48 @@ ssh -F "$HOME/.ssh-local/config" mac 'command -v docker || ls /usr/local/bin/doc
 ps -eo pid,etime,args | grep -Ei 'mux|mosh|ssh'
 ```
 
-A positive result needs no such scepticism — it carries its own evidence. Only
+Absence has to be *earned*, so spend the extra command there.
-absence has to be *earned*, so spend the extra command there.
+
+### …and a positive result only proves what you *actually asked*
+
+An earlier version of this section claimed "a positive result needs no such
+scepticism — it carries its own evidence." **That is false, and believing it
+cost a later session three more wrong findings.** A positive result is evidence
+about the question your command really posed, which may not be the question you
+meant. The failure is invisible precisely *because* the command succeeded.
+
+| Claim | The command succeeded — at answering something else |
+|---|---|
+| "EGL git over SSH works" | `ssh git@gitea.egl.lan` greeted me as `joakimp`. `~/.ssh/config` had `Host gitea*` → `HostName gitea.jordbo.se`, so I authenticated **to the wrong instance**. The real EGL account is `ecsjper`. |
+| "the port config regressed" | compared `ssh -G` output against `2222` — a value produced by **my own earlier `-p 2222` flag**, not by the config. I reported the user's edit as a regression it never caused. |
+| "the CI runners authenticate with this token" | pure fabrication, contradicted by my own scan output already on screen. The runners use per-runner `REGISTRATION_TOKEN`. |
+
+Two habits that actually catch this class, both cheap:
+
+```sh
+# 1. ask which RULE captured your hostname before trusting any ssh result.
+#    ssh_config is first-obtained-value-wins PER KEYWORD, not per block: a
+#    specific block only wins the keywords it declares, so a later `Host gitea*`
+#    still supplies HostName unless the specific block restates it.
+ssh -G git@thehost | grep -E '^(hostname|port|user|identityfile)'
+
+# 2. state the expected result BEFORE running the check, and diff against it.
+#    This is the single technique that separated the one verification that went
+#    right (10/10, expectations declared per probe) from five that went wrong
+#    (results interpreted after the fact, each time in the direction I expected).
+probe "/repos/.../actions/runs" 200   # must work
+probe "/admin/users"            403   # must be denied
+```
+
+And the meta-observation, which is the reason this subsection exists: across all
+five errors, **not one was caught by re-reading my own reasoning.** Every one was
+caught by a second measurement that disagreed — the SSH lie surfaced only because
+the greeting said `joakimp` while a token probe minutes earlier had said
+`ecsjper`; the fabrication surfaced only because the user read my own output back
+to me. So the operational rule is not "be careful". It is: **for a load-bearing
+claim, produce a second measurement by a different route, and expect it to
+disagree.** If you cannot think of a second route, you do not yet have a finding
+— you have a hypothesis.
 
 **`dscp`/`scp` with accented filenames on a macOS host.** macOS stores filenames
 in Unicode **NFD** (decomposed — e.g. `ä` is `a` + combining U+0308), while the

scripts/smoke-test.sh

@@ -245,6 +245,8 @@ run "socat"           "socat -V"
 run "studio-expose helper" "test -x /usr/local/bin/studio-expose"
 run "image-baked pi-devbox-environment skill" \
     "test -f /usr/local/share/pi-devbox/skills/pi-devbox-environment/SKILL.md"
+run "image-baked credential-incident-response skill" \
+    "test -f /usr/local/share/pi-devbox/skills/credential-incident-response/SKILL.md"
 run "global-AGENTS append snippet present" \
     "test -f /usr/local/share/pi-devbox/pi-global-AGENTS.append.md"
 run "pi-devbox block merged into pi-global-AGENTS.md" \
@@ -596,9 +598,9 @@ exec_test "mempalace skill linked (fallback)"      'test -L $HOME/.agents/skills
 # bumped correctly but whose bytes came from the wrong place.
 exec_test "mempalace skill snapshot is current"    'f=$HOME/.agents/skills/mempalace/SKILL.md; grep -q "Provenance is stamped for you" "$f" && ! grep -q "Attribute what you file yourself" "$f" && echo ok'
 # Link TARGETS, not just link existence: with no skillset mounted (as here) the
-# baked tree must be what resolves, for all three vendored skills.
+# baked tree must be what resolves, for all four vendored skills.
 exec_test "vendored skills resolve to the baked tree (no skillset mounted)" \
-    'for s in mempalace pi-extensions pi-devbox-environment; do
+    'for s in mempalace pi-extensions pi-devbox-environment credential-incident-response; do
        case "$(readlink -f $HOME/.agents/skills/$s)" in
          /usr/local/share/pi-devbox/skills/$s) ;;
          *) echo "$s resolves to $(readlink -f $HOME/.agents/skills/$s)" >&2; exit 1 ;;
@@ -612,7 +614,7 @@ exec_test "vendored skills resolve to the baked tree (no skillset mounted)" \
 exec_test "pi-devbox-version reports skill sources (all baked, no skillset here)" \
     'out=$(pi-devbox-version)
      echo "$out" | grep -q "skills:" || { echo "no skills section" >&2; exit 1; }
-     for s in mempalace pi-extensions pi-devbox-environment; do
+     for s in mempalace pi-extensions pi-devbox-environment credential-incident-response; do
        echo "$out" | grep -qE "^    $s +baked$" \
          || { echo "$s not reported as baked" >&2; exit 1; }
      done; echo ok'