# Secret hygiene: keeping credentials out of the palace

A palace is mined from **transcripts**, and transcripts contain whatever the
terminal printed. Print an environment, `cat` a `.env`, paste a `curl -H
"Authorization: …"`, and the secret becomes a drawer — searchable by every agent
on the fleet, on every machine, indefinitely. This document describes the
scrubbing that exists, what it deliberately does **not** try to do, and the two
layers it belongs in.

Measured on this fleet, 2026-08-27: a single leaked bearer token had reached
**3 drawers, 13 feeder inbox files across all three devices, and 10 local files
dating back 10 days**. Nobody was careless; an agent inspected an environment
variable while debugging. That frequency is the design premise — this is a
routine event to be handled by the pipeline, not an incident to be handled by
discipline.

## 1. Where the hook goes, and why there are two of them

| Layer | Implemented in | Sees | Catches |
|---|---|---|---|
| **Client, pre-staging** | `bin/mempalace_redact.py`, called from `bin/mempalace-pi-session` at the point the staged transcript is written | the machine's own environment **and** the transcript | the real-world case: a secret this machine holds, printed into a session, before it is uploaded anywhere |
| **Server, pre-persist** | *not yet implemented* — see §5 | only the text it is handed | secrets from clients that predate the scrubber, from non-pi clients, and secrets an agent types straight into `add_drawer` |

The client hook is the load-bearing one, because it is the only layer that can
compare text against **the actual secret values it holds** (see tier 1 below) and
because it stops the leak *before transmission*. The server hook is defence in
depth: pattern-only, but it covers write paths the feeder never sees.

Client placement matters for one specific reason: the pi feeder writes the staged
transcript from an in-memory structure, and the remote path then `rsync`s **that
same file** byte-for-byte. Scrubbing at the write point therefore covers local
mining and remote upload with a single hook — there is no second serialization to
forget.

## 2. Detection is anchored on meaning, never on entropy

The tempting design — "redact long random-looking strings" — is actively
destructive here, because a palace is *full* of high-entropy strings that are its
own primary keys:

```
drawer_pi-devbox_gotchas_f7b1c8d4c7b590196351ab9b      drawer id
evt_20260826T213924_397b1f710c2d                       event id
rep_d344e349ba276d6fc11997cd552f6937                   replica id
1787780364792-000000-rep_d344e349ba276d6f…             hybrid logical clock
ecc2a9c574f4156403cf4e90aea8b0d088c75700               git commit sha
```

An entropy detector fires on every one of those, and the resulting redaction is
silent, permanent, and destroys traceability. So detection uses three anchors
that carry meaning instead:

Each tier is a different *kind of evidence* that a string is a credential. The
tier is not a severity ranking of the secret — it is how much to trust the
detection. Rule names appear in the output; the tier tells you how to read them.

| Tier | Anchor | Default | False-positive risk |
|---|---|---|---|
| **1 — known values** | literal values from this process's env, for variables whose *name* says secret (`…TOKEN`, `…SECRET`, `…PASSWORD`, `…API_KEY`) | **redact** | none by construction: the value *is* the secret |
| **2 — known shapes** | vendor-prefixed credentials: `ghp_…`, `github_pat_…`, `glpat-…`, `xox[abprs]-…`, `sk-…`, `AKIA…`, `AIza…`, `hf_…`, JWTs, PEM private-key blocks, credentials inside URLs, `Authorization:` headers | **redact** | very low: the prefix is meaningful, not random |
| **3 — name=value** | an assignment whose *key* says secret | **report only** | measured **high** — see §3 |

### Worked examples, one line each

```
MEMPALACE_REMOTE_TOKEN=pfrDBfak…            tier 1 — value matches this env's secret
the token is pfrDBfak… apparently           tier 1 — same value, bare in prose, still caught
git clone https://joakim:hunter2@git/x      tier 2 — credentials in a URL
Authorization: Bearer abcdefghijklmnop      tier 2 — header shape
ghp_AbCdEf… / glpat-… / AKIA… / sk-ant-…    tier 2 — vendor prefix
db_password = s3cr3t-p4ssw0rd-xyz           tier 3 — only the KEY suggests it (reported)
GITEA_TOKEN=0123456789abcdef… (40 hex)      tier 3 — indistinguishable from a commit sha
```

### Which tier fired? Read it off the output

The tool prints **rule names**, not tier numbers, because the rule says *what*
matched. The feeder prefixes them with the tier so both are visible:

```
[REDACTED] 2026-06-27T23-13-49.jsonl  T1:env-value=1, T2:github-fine-grained=1  fp=ad78c7d4,fcd95ab5
[scrub] 29 redaction(s) applied, 141 name-anchored candidate(s) reported only
```

`RULE_TIERS` in `mempalace_redact.py` is the authoritative mapping, `tier_of()`
reads it, and a self-test fails if any rule is left unclassified — so the two
vocabularies cannot drift apart silently.

Tier 1 catches any presentation of a secret — env dump, JSON, error message,
URL, prose — because it matches the value itself. It is also, **in the default
configuration, the only tier that resolves a shape collision**: a 40-hex Gitea
PAT is byte-identical to a git commit sha, so only a known value (tier 1,
redacts) or a naming key (tier 3, reports) can tell them apart. A sha-shaped PAT
is therefore caught if and only if it belongs to this machine — an accepted gap,
since the alternative is redacting every commit sha in the palace.

## 3. The false-positive measurement, which changed the design

Tier 3 was originally going to redact. Measured against **52 MB of real fleet
transcripts (45 sessions)** it produced **403 hits**, and inspection of them (with
values masked) showed the overwhelming majority were not secrets:

- `GITEA_ACCESS_TOKEN=${GITEA_ACCESS_TOKEN:-}` — docker-compose interpolation
- `const tokens = countTokensForModel` — TypeScript source stored as memory
- `refreshToken(credentials: Credentials …)` — a *type annotation*
- `--setattr=krbPasswordExpiration=20260529…` — an IPA attribute holding a date
- `|1.FORGE.TOKENS:…` — AAAK diary shorthand
- `(root@host) Password:` followed by unrelated terminal output

Redacting those would corrupt code, configuration and documentation held as
memory, in order to catch secrets tier 1 already catches by value. After adding
guards for interpolation (`${VAR}`, `$VAR`, `%VAR%`, `{{tpl}}`), code context,
non-secret key suffixes (`…Expiration`, `…Count`, `…Type`) and all-digit values,
the enforced count on the same corpus fell from **403 to 29** — and those 29 are
tier-1 and tier-2 hits, i.e. real credential shapes.

So tier 3 **reports and does not rewrite** by default. Set
`MEMPALACE_REDACT_STRICT=1` to make it enforce, e.g. on a machine whose
transcripts are configuration-heavy rather than code-heavy.

## 4. Known false negatives — stated, not hidden

The scrubber will not catch a novel credential format pasted bare into prose
with no name nearby, a secret belonging to a machine whose environment this
process cannot see, a base64-of-a-secret, or a secret split across lines.

**"The scrubber ran" must never be read as "there are no secrets in here."** To
keep that measurable rather than assumed, two things are reported:

- every scrub prints a count — including `0 redaction(s)`, because printing
  nothing is indistinguishable from a scrubber that never ran;
- `suspicions()` reports high-entropy strings it did **not** redact, as
  `(length, fingerprint)` pairs rather than values, so the miss rate can be
  tracked over time and a recurring fingerprint can be investigated by hand.

Findings never carry the secret. A `Finding` holds the rule, the label, the
length and `sha256(value)[:8]` — enough to recognise the same leak recurring,
not enough to recover it.

**Fail closed.** If the redactor cannot be imported, the feeder refuses to stage
rather than staging unscrubbed (`exit 3`). Override deliberately with
`MEMPALACE_FEED_ALLOW_UNSCRUBBED=1`.

## 5. The server-side layer (not yet implemented)

Three call sites, because the server has three near-duplicate validators rather
than one:

| Path | Function | Covers |
|---|---|---|
| drawers | `config.py` → `sanitize_content()` | `add_drawer`, `update_drawer`, `diary_write`, `checkpoint` — all four route through it |
| events | `logstream.py` → `_sanitize_body()` | `event_append`, and `event_ack` transitively |
| artifacts | `logstream.py` → `put_artifact()` | inlines its own checks; needs its own edit |

Server-side scrubbing is **tier 2 only** (plus optional tier-3 reporting): the
hub cannot see a client's environment, so tier 1 is structurally unavailable
there. That asymmetry is the reason the client hook is not redundant.

## 6. Cleaning up a leak that already landed

1. **Never re-echo the value while hunting it.** Pass it via stdin, never argv
   (visible in `ps` on a shared host). Note `get_drawer` is a trap: reading a
   drawer in order to redact it prints the secret back into the live transcript.
2. **Redact, don't delete.** Replacing the value in place keeps the mined
   transcript's memory value; a stale embedding vector is a cheap price.
3. **Scrub the feeder inbox too**, or the next mine re-files it.
4. **A live session file needs an equal-length in-place overwrite** — the agent
   holds it open, so temp-file-plus-rename loses everything appended afterwards.
5. **Expect residue.** SQLite keeps old page content in freed pages until
   `VACUUM`, so a raw byte scan still matches after a successful `UPDATE`. Decide
   explicitly whether that matters: if the credential store on the same host is
   plaintext anyway, it usually does not.

## 7. Usage

```bash
# unit tests, including the must-not-redact corpus of palace id shapes
python3 bin/mempalace_redact.py --self-test

# scrub anything on stdin; report goes to stderr
some-command | python3 bin/mempalace_redact.py > clean.txt

# also list high-entropy strings that were NOT redacted, as fingerprints
python3 bin/mempalace_redact.py --suspicions < transcript.jsonl > clean.jsonl
```