skills: add credential-incident-response, and assert it stays baked

Carries the facts a two-day credential incident produced, not the discipline: probe the issuer FIRST (11 of 13 "exposed" credentials were already dead at the provider, which cost five HTTP requests to learn and was never checked), the 403-vs-401 trap that scoped tokens introduce into liveness probes, revocation beats deletion for anything already replicated, the three places a secret hides in a Chroma palace (FTS content, metadata, raw bytes) in coverage order, scope derivation from measured consumers, and this fleet's age store with its single-recipient weakness. Facts transfer between sessions; exhortations do not — hence a separate skill for the domain knowledge and a one-line pointer in the always-loaded block. Authored here, so baked is canonical and it is NOT added to skillset-owned.txt. Skill dirs are picked up by a glob in entrypoint-user.sh, so no registration is needed — verified rather than assumed, since an enumerated list would have left the skill inert, a fitting failure given its subject. Three smoke assertions extended so a future rebuild cannot silently drop it.
skills: fix the half of the negative-result rule that was wrong
2026-08-30 00:50:11 +02:00 · 2026-08-30 00:50:11 +02:00
5 changed files with 246 additions and 5 deletions
@@ -70,3 +70,41 @@ rather than merely confusing you:
  local disk, so `mempalace search` can return older and different results than
  the MCP tools while both look correct. Use the MCP tools for the central
  palace; the CLI only for a local one.
+
+## Before you file a finding: second measurement, different route
+
+This is here rather than in a skill because it has to fire *without* a matching
+task description, and because the version of it that lived only in a skill was
+violated five times in one session by an agent that had the skill available.
+
+**Any claim you are about to record as fact — in a drawer, a diary entry, a
+coordination event, or a report to the user — needs a second measurement taken
+by a different route.** Not a re-read of your reasoning: re-reading has caught
+zero of these. A disagreeing measurement has caught all of them.
+
+The two shapes that get filed as fact and are not:
+
+- **A negative result** (`401`, connection refused, zero rows, "not found") is
+  first a claim about *your filter*, not about the world. Wrong host, wrong port,
+  wrong table, capped output.
+- **A positive result** proves only what your command *actually asked*. An SSH
+  handshake can succeed against the wrong host (`ssh -G` tells you which rule
+  captured the name); a `401` can be a real answer from an issuer that never
+  minted the credential.
+
+Cheapest habit that works: **write the expected result next to each check before
+running it**, then diff. Expectations declared up front turn a silent wrong
+assumption into a visible mismatch. And if you cannot think of a second route to
+the same fact, you do not have a finding — you have a hypothesis, so label it as
+one.
+
+## Handling an exposed credential
+
+If a task touches a leaked secret, a token rotation, "is this credential still
+live?", whether to delete stored content, or which scopes a new token needs:
+**read `~/.agents/skills/credential-incident-response/SKILL.md` first.** One rule
+is load-bearing enough to state here: **probe the issuing provider before doing
+anything else** — most "exposed" credentials in a long-lived fleet are already
+dead, and the ones that are live are often far more privileged than assumed.
+Severity first, cleanup second, and prefer **revocation over deletion** for
+anything already replicated.
@@ -9,6 +9,7 @@ one", which was a bug).
 | skill | owner | how it gets here |
 |-------|-------|------------------|
 | `pi-devbox-environment` | pi-devbox (this repo) | authored here; the canonical copy |
+| `credential-incident-response` | pi-devbox (this repo) | authored here; the canonical copy |
 | `pi-extensions` | the `pi-extensions` package repo (`skill/`) | **vendored fallback** + refreshed at build |
 | `mempalace` | the `skillset` repo | **vendored fallback** (snapshot only) |

@@ -0,0 +1,157 @@
+---
+name: credential-incident-response
+description: >-
+  Respond correctly when a live credential is found somewhere it should not be —
+  in a chat transcript, a MemPalace drawer, a log, a git-tracked config, or an
+  agent-authored note. Load this whenever a task involves a leaked/exposed
+  secret, a token rotation, a "is this credential still live?" question, deciding
+  whether to delete or scrub stored content, or choosing scopes for a new API
+  token. Covers the mandatory order of operations (probe the issuer FIRST —
+  severity before cleanliness), leak-free credential identity via sha256[:8]
+  fingerprints, why revocation beats deletion for anything already replicated,
+  deriving least-privilege scopes from measured consumers instead of guessing,
+  where this fleet's secrets actually live (age-encrypted .env.age in
+  docker-compose-repo, plus gitignored plaintext .env drift), the three places a
+  secret hides in a Chroma palace, and the exposures that rotation does NOT fix.
+---
+
+# Credential incident response
+
+A leaked credential is a **severity** question before it is a cleanliness
+question. Two days of scrubbing, redaction plumbing and deletion planning were
+once spent on a set of 13 credentials of which **11 were already dead at the
+provider** — a fact that cost five HTTP requests to establish and was never
+checked. Meanwhile the two live ones turned out to be instance-owner **admin**
+tokens, which nobody had looked at either.
+
+## 1. Order of operations — do not reorder this
+
+1. **Is it still accepted?** Probe the issuing provider. Dead credential →
+   hygiene item, stop panicking. Live → incident, continue.
+2. **What can it do?** Read the identity back. `is_admin`, `id=1`, scopes,
+   which account. A read-only repo token and an instance-owner admin token are
+   not the same finding.
+3. **What consumes it?** Grep for real consumers before assuming breakage.
+4. **Where does it live?** Enumerate copies (store, palace, transcripts, git).
+5. **Then** rotate/revoke, and only then consider cleanup.
+
+Doing 4→3→1 in reverse produces confident, wrong severity calls and wasted
+cleanup. If you only have time for one step, do step 1.
+
+## 2. Leak-free identity: fingerprint, never the value
+
+Publishing an 8-hex fingerprint lets you compare a credential across machines,
+files, drawers and peers without ever materialising the secret. Same formula as
+`mempalace_redact.py`:
+
+```sh
+printf '%s' "$SECRET" | sha256sum | cut -c1-8   # printf, NOT echo (no newline)
+printf '%s' 'test' | sha256sum | cut -c1-8      # self-test -> 9f86d081
+```
+
+Report as `(variable, fp, length)`. Equal fingerprints across hosts prove a
+shared credential; that is usually the important part. **Never** paste a live
+value into a search query, a palace drawer, an event body, or a chat message —
+in an agent context your own tool output is itself captured and re-filed.
+
+## 3. Liveness probes, and the trap that scoping creates
+
+```sh
+# Gitea
+curl -sS -m 10 -o /dev/null -w '%{http_code}\n' -H "Authorization: token $T" \
+  "$GITEA_HOST/api/v1/repos/<owner>/<repo>/actions/runs?limit=1"
+# GitHub
+curl -sS -m 10 -o /dev/null -w '%{http_code}\n' -H "Authorization: token $T" \
+  https://api.github.com/user
+```
+
+- `200` live · `401` revoked/invalid · **`403` = wrong question, not a dead token**
+- **Probe the issuer that minted it.** A 401 from an unrelated instance says
+  nothing. Resolve the host from config (`GITEA_EGL_HOST` etc.), do not assume.
+- **Under scoped tokens, `/api/v1/user` returns 403 for a perfectly live token**
+  unless `user` scope was granted. So it cannot distinguish *revoked* from
+  *merely scoped*. Use a **repository route the token is authorised for**.
+- Verify **both directions** after a rotation: old → 401, new → 200. The second
+  check is what catches "deleted the wrong token".
+- Port/scheme come from config, not habit: one instance here is
+  `http://gitea.egl.lan:3000` — plain HTTP, with 443 refused.
+
+## 4. Revocation beats deletion — the load-bearing rule
+
+Once revoked, stored copies are **inert**; you may leave them. Deleting them is
+best-effort over an *unbounded* copy set: FTS shadow rows, feed inbox `.jsonl`
+files on every host, sqlite free pages after the delete, mesh replicas that
+already synced, and backups. **Revocation invalidates every copy everywhere at
+once, including copies nobody enumerated.**
+
+So: **rotate + revoke first.** Treat drawer deletion as optional hygiene, never
+as the remedy. Then record the retired fingerprints as *known-dead* so the next
+census recognises them instead of reopening the investigation.
+
+Corollary: never reach for `mempalace_sync` or a bulk `delete_by_source` on a
+shared palace as incident response. High blast radius, low actual benefit.
+
+## 5. Finding a secret in a Chroma palace — three targets, in this order
+
+1. `embedding_fulltext_search_content.c0` — **where document text actually is**
+2. `embedding_metadata.string_value` — metadata fields only
+3. raw byte scan of every `*.sqlite3` — backstop, covers FTS pages and free space
+
+Scanning only (2) is the classic false clean: hundreds of thousands of rows,
+zero hits, and the secret sitting in (1) the whole time. Semantic search proves
+nothing about absence — it returns top-k. For completeness, enumerate by filing
+window (`list_drawers(since=T, before=T+1m)`), since one mine shares a minute.
+
+Value-agnostic sweeps (uuid / 40-hex / `NAME=VALUE`) drown in false positives at
+fleet scale — 608 candidates, mostly session UUIDs and git SHAs. Name-anchoring
+plus entropy plus provenance, applied to **document text**, is what works.
+
+## 6. Choosing scopes: derive them from measured consumers
+
+Before creating a replacement token, find out what actually uses it:
+
+```sh
+git -C <repo> remote get-url origin        # ssh:// ? then git needs NO token
+git config --global --list | grep -iE 'credential|insteadof'   # and no helper?
+grep -rhoE 'api/v1/[A-Za-z0-9/{}$_.-]+' <consumers>  | sort -u  # exact routes
+grep -rhoE '\-X [A-Z]+' <consumers>                             # any writes?
+```
+
+Real outcome here: git used SSH keys throughout, and the token's only consumer
+read three CI-run routes with `GET`. So `repository: Read` and nothing else
+replaced two admin tokens. **Scoping shrinks the blast radius of the next leak
+far more than any redaction pipeline does** — a read-only token in a transcript
+is a hygiene event, not an instance compromise.
+
+Then prove the scope with an acceptance suite that declares expectations first:
+must-work routes → `200`; `/admin/*`, `/user`, `/user/repos` → `403`.
+
+## 7. What rotation does *not* fix
+
+- **A cleartext channel.** If the endpoint is `http://`, the *new* token is
+  exposed identically from first use. Raise TLS separately.
+- **Git history.** A secret committed and pushed cannot be fixed by any store or
+  palace operation — it needs rotation *and* history surgery.
+- **Agent-authored content.** Stage-write redactors see transcripts only, never
+  `add_drawer` / `checkpoint` / `diary_write` output. Never type a secret into
+  the palace yourself; nothing downstream will catch it.
+- **Plaintext/encrypted drift.** Gitignored plaintext `.env` files go stale while
+  `.env.age` moves on, so old values linger on disk (and in backups) long after
+  rotation. They are a common source of "mystery" fingerprints in a census.
+
+## 8. This fleet's secret store (verify, do not assume)
+
+- All `*.env.age` live in **one** repo: `joakimp/docker-compose-repo`. `myconfigs`
+  has none.
+- Every `.age` file has **one X25519 recipient** — a single key tracked in
+  `myconfigs` under git-crypt. Unlocking git-crypt therefore decrypts the entire
+  fleet's secrets, including hosts you have no access to. The age layer adds no
+  isolation beyond git-crypt.
+- Flow: `./fetch-secrets.sh <host>` (decrypt → `.env`) → edit → `./encrypt-secrets.sh <host>`
+  → commit → push → `docker compose up -d --force-recreate`.
+- **Always pass the host argument** to `encrypt-secrets.sh`. Bare, it walks the
+  whole tree and re-encrypts every `.env` it finds, re-nonced, including stale
+  ones — silently rolling back other hosts' secrets.
+- After any re-encrypt, check the header still shows exactly **one X25519
+  recipient**; a hand-rolled `age -r` locks the rest of the fleet out, and the
+  failure only appears on another machine, later.
@@ -143,6 +143,9 @@ mine:
 | "`tor-ms22` is not in the SSH config" | `grep … \| head -20` — the entry was at **line 454**. `~/.ssh/config` here is ~500 lines. |
 | "the Docker host has no `docker`" | non-interactive SSH `PATH` lacks `/usr/local/bin` (§2, §3). It was at `/usr/local/bin/docker`. |
 | "no ControlMaster is running" | pattern `ssh ` (trailing space) cannot match a master: those processes **rename themselves** to `ssh: <controlpath> [mux]`. |
+| "the credential is not in the palace" | scanned `embedding_metadata.string_value` only. Drawer **text** lives in `embedding_fulltext_search_content.c0`; 554k metadata rows proved nothing. |
+| "this token is dead — 401" | probed it against the **wrong issuer**. A 401 from an instance that never issued the credential is not evidence about the credential. |
+| "that host is unreachable, can't test" | tried ports 443 and 80. It was on **3000**, and the env var I already held (`GITEA_EGL_HOST`) stated the scheme and port. |

 Habits that would have caught all three:

@@ -157,8 +160,48 @@ ssh -F "$HOME/.ssh-local/config" mac 'command -v docker || ls /usr/local/bin/doc
 ps -eo pid,etime,args | grep -Ei 'mux|mosh|ssh'
 ```

-A positive result needs no such scepticism — it carries its own evidence. Only
-absence has to be *earned*, so spend the extra command there.
+Absence has to be *earned*, so spend the extra command there.
+
+### …and a positive result only proves what you *actually asked*
+
+An earlier version of this section claimed "a positive result needs no such
+scepticism — it carries its own evidence." **That is false, and believing it
+cost a later session three more wrong findings.** A positive result is evidence
+about the question your command really posed, which may not be the question you
+meant. The failure is invisible precisely *because* the command succeeded.
+
+| Claim | The command succeeded — at answering something else |
+|---|---|
+| "EGL git over SSH works" | `ssh git@gitea.egl.lan` greeted me as `joakimp`. `~/.ssh/config` had `Host gitea*` → `HostName gitea.jordbo.se`, so I authenticated **to the wrong instance**. The real EGL account is `ecsjper`. |
+| "the port config regressed" | compared `ssh -G` output against `2222` — a value produced by **my own earlier `-p 2222` flag**, not by the config. I reported the user's edit as a regression it never caused. |
+| "the CI runners authenticate with this token" | pure fabrication, contradicted by my own scan output already on screen. The runners use per-runner `REGISTRATION_TOKEN`. |
+
+Two habits that actually catch this class, both cheap:
+
+```sh
+# 1. ask which RULE captured your hostname before trusting any ssh result.
+#    ssh_config is first-obtained-value-wins PER KEYWORD, not per block: a
+#    specific block only wins the keywords it declares, so a later `Host gitea*`
+#    still supplies HostName unless the specific block restates it.
+ssh -G git@thehost | grep -E '^(hostname|port|user|identityfile)'
+
+# 2. state the expected result BEFORE running the check, and diff against it.
+#    This is the single technique that separated the one verification that went
+#    right (10/10, expectations declared per probe) from five that went wrong
+#    (results interpreted after the fact, each time in the direction I expected).
+probe "/repos/.../actions/runs" 200   # must work
+probe "/admin/users"            403   # must be denied
+```
+
+And the meta-observation, which is the reason this subsection exists: across all
+five errors, **not one was caught by re-reading my own reasoning.** Every one was
+caught by a second measurement that disagreed — the SSH lie surfaced only because
+the greeting said `joakimp` while a token probe minutes earlier had said
+`ecsjper`; the fabrication surfaced only because the user read my own output back
+to me. So the operational rule is not "be careful". It is: **for a load-bearing
+claim, produce a second measurement by a different route, and expect it to
+disagree.** If you cannot think of a second route, you do not yet have a finding
+— you have a hypothesis.

 **`dscp`/`scp` with accented filenames on a macOS host.** macOS stores filenames
 in Unicode **NFD** (decomposed — e.g. `ä` is `a` + combining U+0308), while the
@@ -245,6 +245,8 @@ run "socat"           "socat -V"
 run "studio-expose helper" "test -x /usr/local/bin/studio-expose"
 run "image-baked pi-devbox-environment skill" \
    "test -f /usr/local/share/pi-devbox/skills/pi-devbox-environment/SKILL.md"
+run "image-baked credential-incident-response skill" \
+    "test -f /usr/local/share/pi-devbox/skills/credential-incident-response/SKILL.md"
 run "global-AGENTS append snippet present" \
    "test -f /usr/local/share/pi-devbox/pi-global-AGENTS.append.md"
 run "pi-devbox block merged into pi-global-AGENTS.md" \
@@ -596,9 +598,9 @@ exec_test "mempalace skill linked (fallback)"      'test -L $HOME/.agents/skills
 # bumped correctly but whose bytes came from the wrong place.
 exec_test "mempalace skill snapshot is current"    'f=$HOME/.agents/skills/mempalace/SKILL.md; grep -q "Provenance is stamped for you" "$f" && ! grep -q "Attribute what you file yourself" "$f" && echo ok'
 # Link TARGETS, not just link existence: with no skillset mounted (as here) the
-# baked tree must be what resolves, for all three vendored skills.
+# baked tree must be what resolves, for all four vendored skills.
 exec_test "vendored skills resolve to the baked tree (no skillset mounted)" \
-    'for s in mempalace pi-extensions pi-devbox-environment; do
+    'for s in mempalace pi-extensions pi-devbox-environment credential-incident-response; do
       case "$(readlink -f $HOME/.agents/skills/$s)" in
         /usr/local/share/pi-devbox/skills/$s) ;;
         *) echo "$s resolves to $(readlink -f $HOME/.agents/skills/$s)" >&2; exit 1 ;;
@@ -612,7 +614,7 @@ exec_test "vendored skills resolve to the baked tree (no skillset mounted)" \
 exec_test "pi-devbox-version reports skill sources (all baked, no skillset here)" \
    'out=$(pi-devbox-version)
     echo "$out" | grep -q "skills:" || { echo "no skills section" >&2; exit 1; }
-     for s in mempalace pi-extensions pi-devbox-environment; do
+     for s in mempalace pi-extensions pi-devbox-environment credential-incident-response; do
       echo "$out" | grep -qE "^    $s +baked$" \
         || { echo "$s not reported as baked" >&2; exit 1; }
     done; echo ok'