From 36e65fe657654d11ddc6487c261e332931e5a129 Mon Sep 17 00:00:00 2001 From: Joakim Persson Date: Sun, 30 Aug 2026 00:50:11 +0200 Subject: [PATCH] skills: add credential-incident-response, and assert it stays baked MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Carries the facts a two-day credential incident produced, not the discipline: probe the issuer FIRST (11 of 13 "exposed" credentials were already dead at the provider, which cost five HTTP requests to learn and was never checked), the 403-vs-401 trap that scoped tokens introduce into liveness probes, revocation beats deletion for anything already replicated, the three places a secret hides in a Chroma palace (FTS content, metadata, raw bytes) in coverage order, scope derivation from measured consumers, and this fleet's age store with its single-recipient weakness. Facts transfer between sessions; exhortations do not — hence a separate skill for the domain knowledge and a one-line pointer in the always-loaded block. Authored here, so baked is canonical and it is NOT added to skillset-owned.txt. Skill dirs are picked up by a glob in entrypoint-user.sh, so no registration is needed — verified rather than assumed, since an enumerated list would have left the skill inert, a fitting failure given its subject. Three smoke assertions extended so a future rebuild cannot silently drop it. --- .../local/share/pi-devbox/skills/VENDORED.md | 1 + .../credential-incident-response/SKILL.md | 157 ++++++++++++++++++ scripts/smoke-test.sh | 8 +- 3 files changed, 163 insertions(+), 3 deletions(-) create mode 100644 rootfs/usr/local/share/pi-devbox/skills/credential-incident-response/SKILL.md diff --git a/rootfs/usr/local/share/pi-devbox/skills/VENDORED.md b/rootfs/usr/local/share/pi-devbox/skills/VENDORED.md index 10d25f2..90fe4b2 100644 --- a/rootfs/usr/local/share/pi-devbox/skills/VENDORED.md +++ b/rootfs/usr/local/share/pi-devbox/skills/VENDORED.md @@ -9,6 +9,7 @@ one", which was a bug). | skill | owner | how it gets here | |-------|-------|------------------| | `pi-devbox-environment` | pi-devbox (this repo) | authored here; the canonical copy | +| `credential-incident-response` | pi-devbox (this repo) | authored here; the canonical copy | | `pi-extensions` | the `pi-extensions` package repo (`skill/`) | **vendored fallback** + refreshed at build | | `mempalace` | the `skillset` repo | **vendored fallback** (snapshot only) | diff --git a/rootfs/usr/local/share/pi-devbox/skills/credential-incident-response/SKILL.md b/rootfs/usr/local/share/pi-devbox/skills/credential-incident-response/SKILL.md new file mode 100644 index 0000000..5eb986b --- /dev/null +++ b/rootfs/usr/local/share/pi-devbox/skills/credential-incident-response/SKILL.md @@ -0,0 +1,157 @@ +--- +name: credential-incident-response +description: >- + Respond correctly when a live credential is found somewhere it should not be — + in a chat transcript, a MemPalace drawer, a log, a git-tracked config, or an + agent-authored note. Load this whenever a task involves a leaked/exposed + secret, a token rotation, a "is this credential still live?" question, deciding + whether to delete or scrub stored content, or choosing scopes for a new API + token. Covers the mandatory order of operations (probe the issuer FIRST — + severity before cleanliness), leak-free credential identity via sha256[:8] + fingerprints, why revocation beats deletion for anything already replicated, + deriving least-privilege scopes from measured consumers instead of guessing, + where this fleet's secrets actually live (age-encrypted .env.age in + docker-compose-repo, plus gitignored plaintext .env drift), the three places a + secret hides in a Chroma palace, and the exposures that rotation does NOT fix. +--- + +# Credential incident response + +A leaked credential is a **severity** question before it is a cleanliness +question. Two days of scrubbing, redaction plumbing and deletion planning were +once spent on a set of 13 credentials of which **11 were already dead at the +provider** — a fact that cost five HTTP requests to establish and was never +checked. Meanwhile the two live ones turned out to be instance-owner **admin** +tokens, which nobody had looked at either. + +## 1. Order of operations — do not reorder this + +1. **Is it still accepted?** Probe the issuing provider. Dead credential → + hygiene item, stop panicking. Live → incident, continue. +2. **What can it do?** Read the identity back. `is_admin`, `id=1`, scopes, + which account. A read-only repo token and an instance-owner admin token are + not the same finding. +3. **What consumes it?** Grep for real consumers before assuming breakage. +4. **Where does it live?** Enumerate copies (store, palace, transcripts, git). +5. **Then** rotate/revoke, and only then consider cleanup. + +Doing 4→3→1 in reverse produces confident, wrong severity calls and wasted +cleanup. If you only have time for one step, do step 1. + +## 2. Leak-free identity: fingerprint, never the value + +Publishing an 8-hex fingerprint lets you compare a credential across machines, +files, drawers and peers without ever materialising the secret. Same formula as +`mempalace_redact.py`: + +```sh +printf '%s' "$SECRET" | sha256sum | cut -c1-8 # printf, NOT echo (no newline) +printf '%s' 'test' | sha256sum | cut -c1-8 # self-test -> 9f86d081 +``` + +Report as `(variable, fp, length)`. Equal fingerprints across hosts prove a +shared credential; that is usually the important part. **Never** paste a live +value into a search query, a palace drawer, an event body, or a chat message — +in an agent context your own tool output is itself captured and re-filed. + +## 3. Liveness probes, and the trap that scoping creates + +```sh +# Gitea +curl -sS -m 10 -o /dev/null -w '%{http_code}\n' -H "Authorization: token $T" \ + "$GITEA_HOST/api/v1/repos///actions/runs?limit=1" +# GitHub +curl -sS -m 10 -o /dev/null -w '%{http_code}\n' -H "Authorization: token $T" \ + https://api.github.com/user +``` + +- `200` live · `401` revoked/invalid · **`403` = wrong question, not a dead token** +- **Probe the issuer that minted it.** A 401 from an unrelated instance says + nothing. Resolve the host from config (`GITEA_EGL_HOST` etc.), do not assume. +- **Under scoped tokens, `/api/v1/user` returns 403 for a perfectly live token** + unless `user` scope was granted. So it cannot distinguish *revoked* from + *merely scoped*. Use a **repository route the token is authorised for**. +- Verify **both directions** after a rotation: old → 401, new → 200. The second + check is what catches "deleted the wrong token". +- Port/scheme come from config, not habit: one instance here is + `http://gitea.egl.lan:3000` — plain HTTP, with 443 refused. + +## 4. Revocation beats deletion — the load-bearing rule + +Once revoked, stored copies are **inert**; you may leave them. Deleting them is +best-effort over an *unbounded* copy set: FTS shadow rows, feed inbox `.jsonl` +files on every host, sqlite free pages after the delete, mesh replicas that +already synced, and backups. **Revocation invalidates every copy everywhere at +once, including copies nobody enumerated.** + +So: **rotate + revoke first.** Treat drawer deletion as optional hygiene, never +as the remedy. Then record the retired fingerprints as *known-dead* so the next +census recognises them instead of reopening the investigation. + +Corollary: never reach for `mempalace_sync` or a bulk `delete_by_source` on a +shared palace as incident response. High blast radius, low actual benefit. + +## 5. Finding a secret in a Chroma palace — three targets, in this order + +1. `embedding_fulltext_search_content.c0` — **where document text actually is** +2. `embedding_metadata.string_value` — metadata fields only +3. raw byte scan of every `*.sqlite3` — backstop, covers FTS pages and free space + +Scanning only (2) is the classic false clean: hundreds of thousands of rows, +zero hits, and the secret sitting in (1) the whole time. Semantic search proves +nothing about absence — it returns top-k. For completeness, enumerate by filing +window (`list_drawers(since=T, before=T+1m)`), since one mine shares a minute. + +Value-agnostic sweeps (uuid / 40-hex / `NAME=VALUE`) drown in false positives at +fleet scale — 608 candidates, mostly session UUIDs and git SHAs. Name-anchoring +plus entropy plus provenance, applied to **document text**, is what works. + +## 6. Choosing scopes: derive them from measured consumers + +Before creating a replacement token, find out what actually uses it: + +```sh +git -C remote get-url origin # ssh:// ? then git needs NO token +git config --global --list | grep -iE 'credential|insteadof' # and no helper? +grep -rhoE 'api/v1/[A-Za-z0-9/{}$_.-]+' | sort -u # exact routes +grep -rhoE '\-X [A-Z]+' # any writes? +``` + +Real outcome here: git used SSH keys throughout, and the token's only consumer +read three CI-run routes with `GET`. So `repository: Read` and nothing else +replaced two admin tokens. **Scoping shrinks the blast radius of the next leak +far more than any redaction pipeline does** — a read-only token in a transcript +is a hygiene event, not an instance compromise. + +Then prove the scope with an acceptance suite that declares expectations first: +must-work routes → `200`; `/admin/*`, `/user`, `/user/repos` → `403`. + +## 7. What rotation does *not* fix + +- **A cleartext channel.** If the endpoint is `http://`, the *new* token is + exposed identically from first use. Raise TLS separately. +- **Git history.** A secret committed and pushed cannot be fixed by any store or + palace operation — it needs rotation *and* history surgery. +- **Agent-authored content.** Stage-write redactors see transcripts only, never + `add_drawer` / `checkpoint` / `diary_write` output. Never type a secret into + the palace yourself; nothing downstream will catch it. +- **Plaintext/encrypted drift.** Gitignored plaintext `.env` files go stale while + `.env.age` moves on, so old values linger on disk (and in backups) long after + rotation. They are a common source of "mystery" fingerprints in a census. + +## 8. This fleet's secret store (verify, do not assume) + +- All `*.env.age` live in **one** repo: `joakimp/docker-compose-repo`. `myconfigs` + has none. +- Every `.age` file has **one X25519 recipient** — a single key tracked in + `myconfigs` under git-crypt. Unlocking git-crypt therefore decrypts the entire + fleet's secrets, including hosts you have no access to. The age layer adds no + isolation beyond git-crypt. +- Flow: `./fetch-secrets.sh ` (decrypt → `.env`) → edit → `./encrypt-secrets.sh ` + → commit → push → `docker compose up -d --force-recreate`. +- **Always pass the host argument** to `encrypt-secrets.sh`. Bare, it walks the + whole tree and re-encrypts every `.env` it finds, re-nonced, including stale + ones — silently rolling back other hosts' secrets. +- After any re-encrypt, check the header still shows exactly **one X25519 + recipient**; a hand-rolled `age -r` locks the rest of the fleet out, and the + failure only appears on another machine, later. diff --git a/scripts/smoke-test.sh b/scripts/smoke-test.sh index b86c016..4c3d57a 100755 --- a/scripts/smoke-test.sh +++ b/scripts/smoke-test.sh @@ -245,6 +245,8 @@ run "socat" "socat -V" run "studio-expose helper" "test -x /usr/local/bin/studio-expose" run "image-baked pi-devbox-environment skill" \ "test -f /usr/local/share/pi-devbox/skills/pi-devbox-environment/SKILL.md" +run "image-baked credential-incident-response skill" \ + "test -f /usr/local/share/pi-devbox/skills/credential-incident-response/SKILL.md" run "global-AGENTS append snippet present" \ "test -f /usr/local/share/pi-devbox/pi-global-AGENTS.append.md" run "pi-devbox block merged into pi-global-AGENTS.md" \ @@ -596,9 +598,9 @@ exec_test "mempalace skill linked (fallback)" 'test -L $HOME/.agents/skills # bumped correctly but whose bytes came from the wrong place. exec_test "mempalace skill snapshot is current" 'f=$HOME/.agents/skills/mempalace/SKILL.md; grep -q "Provenance is stamped for you" "$f" && ! grep -q "Attribute what you file yourself" "$f" && echo ok' # Link TARGETS, not just link existence: with no skillset mounted (as here) the -# baked tree must be what resolves, for all three vendored skills. +# baked tree must be what resolves, for all four vendored skills. exec_test "vendored skills resolve to the baked tree (no skillset mounted)" \ - 'for s in mempalace pi-extensions pi-devbox-environment; do + 'for s in mempalace pi-extensions pi-devbox-environment credential-incident-response; do case "$(readlink -f $HOME/.agents/skills/$s)" in /usr/local/share/pi-devbox/skills/$s) ;; *) echo "$s resolves to $(readlink -f $HOME/.agents/skills/$s)" >&2; exit 1 ;; @@ -612,7 +614,7 @@ exec_test "vendored skills resolve to the baked tree (no skillset mounted)" \ exec_test "pi-devbox-version reports skill sources (all baked, no skillset here)" \ 'out=$(pi-devbox-version) echo "$out" | grep -q "skills:" || { echo "no skills section" >&2; exit 1; } - for s in mempalace pi-extensions pi-devbox-environment; do + for s in mempalace pi-extensions pi-devbox-environment credential-incident-response; do echo "$out" | grep -qE "^ $s +baked$" \ || { echo "$s not reported as baked" >&2; exit 1; } done; echo ok'