skills: add credential-incident-response, and assert it stays baked

Carries the facts a two-day credential incident produced, not the discipline:
probe the issuer FIRST (11 of 13 "exposed" credentials were already dead at the
provider, which cost five HTTP requests to learn and was never checked), the
403-vs-401 trap that scoped tokens introduce into liveness probes, revocation
beats deletion for anything already replicated, the three places a secret hides
in a Chroma palace (FTS content, metadata, raw bytes) in coverage order, scope
derivation from measured consumers, and this fleet's age store with its
single-recipient weakness.

Facts transfer between sessions; exhortations do not — hence a separate skill
for the domain knowledge and a one-line pointer in the always-loaded block.

Authored here, so baked is canonical and it is NOT added to skillset-owned.txt.
Skill dirs are picked up by a glob in entrypoint-user.sh, so no registration is
needed — verified rather than assumed, since an enumerated list would have left
the skill inert, a fitting failure given its subject. Three smoke assertions
extended so a future rebuild cannot silently drop it.

rootfs/usr/local/share/pi-devbox/skills/VENDORED.md

@@ -9,6 +9,7 @@ one", which was a bug).
 | skill | owner | how it gets here |
 |-------|-------|------------------|
 | `pi-devbox-environment` | pi-devbox (this repo) | authored here; the canonical copy |
+| `credential-incident-response` | pi-devbox (this repo) | authored here; the canonical copy |
 | `pi-extensions` | the `pi-extensions` package repo (`skill/`) | **vendored fallback** + refreshed at build |
 | `mempalace` | the `skillset` repo | **vendored fallback** (snapshot only) |
 

rootfs/usr/local/share/pi-devbox/skills/credential-incident-response/SKILL.md

@@ -0,0 +1,157 @@
+---
+name: credential-incident-response
+description: >-
+  Respond correctly when a live credential is found somewhere it should not be —
+  in a chat transcript, a MemPalace drawer, a log, a git-tracked config, or an
+  agent-authored note. Load this whenever a task involves a leaked/exposed
+  secret, a token rotation, a "is this credential still live?" question, deciding
+  whether to delete or scrub stored content, or choosing scopes for a new API
+  token. Covers the mandatory order of operations (probe the issuer FIRST —
+  severity before cleanliness), leak-free credential identity via sha256[:8]
+  fingerprints, why revocation beats deletion for anything already replicated,
+  deriving least-privilege scopes from measured consumers instead of guessing,
+  where this fleet's secrets actually live (age-encrypted .env.age in
+  docker-compose-repo, plus gitignored plaintext .env drift), the three places a
+  secret hides in a Chroma palace, and the exposures that rotation does NOT fix.
+---
+
+# Credential incident response
+
+A leaked credential is a **severity** question before it is a cleanliness
+question. Two days of scrubbing, redaction plumbing and deletion planning were
+once spent on a set of 13 credentials of which **11 were already dead at the
+provider** — a fact that cost five HTTP requests to establish and was never
+checked. Meanwhile the two live ones turned out to be instance-owner **admin**
+tokens, which nobody had looked at either.
+
+## 1. Order of operations — do not reorder this
+
+1. **Is it still accepted?** Probe the issuing provider. Dead credential →
+   hygiene item, stop panicking. Live → incident, continue.
+2. **What can it do?** Read the identity back. `is_admin`, `id=1`, scopes,
+   which account. A read-only repo token and an instance-owner admin token are
+   not the same finding.
+3. **What consumes it?** Grep for real consumers before assuming breakage.
+4. **Where does it live?** Enumerate copies (store, palace, transcripts, git).
+5. **Then** rotate/revoke, and only then consider cleanup.
+
+Doing 4→3→1 in reverse produces confident, wrong severity calls and wasted
+cleanup. If you only have time for one step, do step 1.
+
+## 2. Leak-free identity: fingerprint, never the value
+
+Publishing an 8-hex fingerprint lets you compare a credential across machines,
+files, drawers and peers without ever materialising the secret. Same formula as
+`mempalace_redact.py`:
+
+```sh
+printf '%s' "$SECRET" | sha256sum | cut -c1-8   # printf, NOT echo (no newline)
+printf '%s' 'test' | sha256sum | cut -c1-8      # self-test -> 9f86d081
+```
+
+Report as `(variable, fp, length)`. Equal fingerprints across hosts prove a
+shared credential; that is usually the important part. **Never** paste a live
+value into a search query, a palace drawer, an event body, or a chat message —
+in an agent context your own tool output is itself captured and re-filed.
+
+## 3. Liveness probes, and the trap that scoping creates
+
+```sh
+# Gitea
+curl -sS -m 10 -o /dev/null -w '%{http_code}\n' -H "Authorization: token $T" \
+  "$GITEA_HOST/api/v1/repos/<owner>/<repo>/actions/runs?limit=1"
+# GitHub
+curl -sS -m 10 -o /dev/null -w '%{http_code}\n' -H "Authorization: token $T" \
+  https://api.github.com/user
+```
+
+- `200` live · `401` revoked/invalid · **`403` = wrong question, not a dead token**
+- **Probe the issuer that minted it.** A 401 from an unrelated instance says
+  nothing. Resolve the host from config (`GITEA_EGL_HOST` etc.), do not assume.
+- **Under scoped tokens, `/api/v1/user` returns 403 for a perfectly live token**
+  unless `user` scope was granted. So it cannot distinguish *revoked* from
+  *merely scoped*. Use a **repository route the token is authorised for**.
+- Verify **both directions** after a rotation: old → 401, new → 200. The second
+  check is what catches "deleted the wrong token".
+- Port/scheme come from config, not habit: one instance here is
+  `http://gitea.egl.lan:3000` — plain HTTP, with 443 refused.
+
+## 4. Revocation beats deletion — the load-bearing rule
+
+Once revoked, stored copies are **inert**; you may leave them. Deleting them is
+best-effort over an *unbounded* copy set: FTS shadow rows, feed inbox `.jsonl`
+files on every host, sqlite free pages after the delete, mesh replicas that
+already synced, and backups. **Revocation invalidates every copy everywhere at
+once, including copies nobody enumerated.**
+
+So: **rotate + revoke first.** Treat drawer deletion as optional hygiene, never
+as the remedy. Then record the retired fingerprints as *known-dead* so the next
+census recognises them instead of reopening the investigation.
+
+Corollary: never reach for `mempalace_sync` or a bulk `delete_by_source` on a
+shared palace as incident response. High blast radius, low actual benefit.
+
+## 5. Finding a secret in a Chroma palace — three targets, in this order
+
+1. `embedding_fulltext_search_content.c0` — **where document text actually is**
+2. `embedding_metadata.string_value` — metadata fields only
+3. raw byte scan of every `*.sqlite3` — backstop, covers FTS pages and free space
+
+Scanning only (2) is the classic false clean: hundreds of thousands of rows,
+zero hits, and the secret sitting in (1) the whole time. Semantic search proves
+nothing about absence — it returns top-k. For completeness, enumerate by filing
+window (`list_drawers(since=T, before=T+1m)`), since one mine shares a minute.
+
+Value-agnostic sweeps (uuid / 40-hex / `NAME=VALUE`) drown in false positives at
+fleet scale — 608 candidates, mostly session UUIDs and git SHAs. Name-anchoring
+plus entropy plus provenance, applied to **document text**, is what works.
+
+## 6. Choosing scopes: derive them from measured consumers
+
+Before creating a replacement token, find out what actually uses it:
+
+```sh
+git -C <repo> remote get-url origin        # ssh:// ? then git needs NO token
+git config --global --list | grep -iE 'credential|insteadof'   # and no helper?
+grep -rhoE 'api/v1/[A-Za-z0-9/{}$_.-]+' <consumers>  | sort -u  # exact routes
+grep -rhoE '\-X [A-Z]+' <consumers>                             # any writes?
+```
+
+Real outcome here: git used SSH keys throughout, and the token's only consumer
+read three CI-run routes with `GET`. So `repository: Read` and nothing else
+replaced two admin tokens. **Scoping shrinks the blast radius of the next leak
+far more than any redaction pipeline does** — a read-only token in a transcript
+is a hygiene event, not an instance compromise.
+
+Then prove the scope with an acceptance suite that declares expectations first:
+must-work routes → `200`; `/admin/*`, `/user`, `/user/repos` → `403`.
+
+## 7. What rotation does *not* fix
+
+- **A cleartext channel.** If the endpoint is `http://`, the *new* token is
+  exposed identically from first use. Raise TLS separately.
+- **Git history.** A secret committed and pushed cannot be fixed by any store or
+  palace operation — it needs rotation *and* history surgery.
+- **Agent-authored content.** Stage-write redactors see transcripts only, never
+  `add_drawer` / `checkpoint` / `diary_write` output. Never type a secret into
+  the palace yourself; nothing downstream will catch it.
+- **Plaintext/encrypted drift.** Gitignored plaintext `.env` files go stale while
+  `.env.age` moves on, so old values linger on disk (and in backups) long after
+  rotation. They are a common source of "mystery" fingerprints in a census.
+
+## 8. This fleet's secret store (verify, do not assume)
+
+- All `*.env.age` live in **one** repo: `joakimp/docker-compose-repo`. `myconfigs`
+  has none.
+- Every `.age` file has **one X25519 recipient** — a single key tracked in
+  `myconfigs` under git-crypt. Unlocking git-crypt therefore decrypts the entire
+  fleet's secrets, including hosts you have no access to. The age layer adds no
+  isolation beyond git-crypt.
+- Flow: `./fetch-secrets.sh <host>` (decrypt → `.env`) → edit → `./encrypt-secrets.sh <host>`
+  → commit → push → `docker compose up -d --force-recreate`.
+- **Always pass the host argument** to `encrypt-secrets.sh`. Bare, it walks the
+  whole tree and re-encrypts every `.env` it finds, re-nonced, including stale
+  ones — silently rolling back other hosts' secrets.
+- After any re-encrypt, check the header still shows exactly **one X25519
+  recipient**; a hand-rolled `age -r` locks the rest of the fleet out, and the
+  failure only appears on another machine, later.

scripts/smoke-test.sh

@@ -245,6 +245,8 @@ run "socat"           "socat -V"
 run "studio-expose helper" "test -x /usr/local/bin/studio-expose"
 run "image-baked pi-devbox-environment skill" \
     "test -f /usr/local/share/pi-devbox/skills/pi-devbox-environment/SKILL.md"
+run "image-baked credential-incident-response skill" \
+    "test -f /usr/local/share/pi-devbox/skills/credential-incident-response/SKILL.md"
 run "global-AGENTS append snippet present" \
     "test -f /usr/local/share/pi-devbox/pi-global-AGENTS.append.md"
 run "pi-devbox block merged into pi-global-AGENTS.md" \
@@ -596,9 +598,9 @@ exec_test "mempalace skill linked (fallback)"      'test -L $HOME/.agents/skills
 # bumped correctly but whose bytes came from the wrong place.
 exec_test "mempalace skill snapshot is current"    'f=$HOME/.agents/skills/mempalace/SKILL.md; grep -q "Provenance is stamped for you" "$f" && ! grep -q "Attribute what you file yourself" "$f" && echo ok'
 # Link TARGETS, not just link existence: with no skillset mounted (as here) the
-# baked tree must be what resolves, for all three vendored skills.
+# baked tree must be what resolves, for all four vendored skills.
 exec_test "vendored skills resolve to the baked tree (no skillset mounted)" \
-    'for s in mempalace pi-extensions pi-devbox-environment; do
+    'for s in mempalace pi-extensions pi-devbox-environment credential-incident-response; do
        case "$(readlink -f $HOME/.agents/skills/$s)" in
          /usr/local/share/pi-devbox/skills/$s) ;;
          *) echo "$s resolves to $(readlink -f $HOME/.agents/skills/$s)" >&2; exit 1 ;;
@@ -612,7 +614,7 @@ exec_test "vendored skills resolve to the baked tree (no skillset mounted)" \
 exec_test "pi-devbox-version reports skill sources (all baked, no skillset here)" \
     'out=$(pi-devbox-version)
      echo "$out" | grep -q "skills:" || { echo "no skills section" >&2; exit 1; }
-     for s in mempalace pi-extensions pi-devbox-environment; do
+     for s in mempalace pi-extensions pi-devbox-environment credential-incident-response; do
        echo "$out" | grep -qE "^    $s +baked$" \
          || { echo "$s not reported as baked" >&2; exit 1; }
      done; echo ok'