mempalace-toolkit/docs/secret-hygiene.md

Secret hygiene: keeping credentials out of the palace

A palace is mined from transcripts, and transcripts contain whatever the terminal printed. Print an environment, cat a .env, paste a curl -H "Authorization: …", and the secret becomes a drawer — searchable by every agent on the fleet, on every machine, indefinitely. This document describes the scrubbing that exists, what it deliberately does not try to do, and the two layers it belongs in.

Measured on this fleet, 2026-08-27: a single leaked bearer token had reached 3 drawers, 13 feeder inbox files across all three devices, and 10 local files dating back 10 days. Nobody was careless; an agent inspected an environment variable while debugging. That frequency is the design premise — this is a routine event to be handled by the pipeline, not an incident to be handled by discipline.

1. Where the hook goes, and why there are two of them

Layer	Implemented in	Sees	Catches
Client, pre-staging	bin/mempalace_redact.py, called from bin/mempalace-pi-session at the point the staged transcript is written	the machine's own environment and the transcript	the real-world case: a secret this machine holds, printed into a session, before it is uploaded anywhere
Server, pre-persist	not yet implemented — see §5	only the text it is handed	secrets from clients that predate the scrubber, from non-pi clients, and secrets an agent types straight into add_drawer

The client hook is the load-bearing one, because it is the only layer that can compare text against the actual secret values it holds (see tier 1 below) and because it stops the leak before transmission. The server hook is defence in depth: pattern-only, but it covers write paths the feeder never sees.

Client placement matters for one specific reason: the pi feeder writes the staged transcript from an in-memory structure, and the remote path then rsyncs that same file byte-for-byte. Scrubbing at the write point therefore covers local mining and remote upload with a single hook — there is no second serialization to forget.

2. Detection is anchored on meaning, never on entropy

The tempting design — "redact long random-looking strings" — is actively destructive here, because a palace is full of high-entropy strings that are its own primary keys:

drawer_pi-devbox_gotchas_f7b1c8d4c7b590196351ab9b      drawer id
evt_20260826T213924_397b1f710c2d                       event id
rep_d344e349ba276d6fc11997cd552f6937                   replica id
1787780364792-000000-rep_d344e349ba276d6f…             hybrid logical clock
ecc2a9c574f4156403cf4e90aea8b0d088c75700               git commit sha

An entropy detector fires on every one of those, and the resulting redaction is silent, permanent, and destroys traceability. So detection uses three anchors that carry meaning instead:

Each tier is a different kind of evidence that a string is a credential. The tier is not a severity ranking of the secret — it is how much to trust the detection. Rule names appear in the output; the tier tells you how to read them.

Tier	Anchor	Default	False-positive risk
1 — known values	literal values from this process's env, for variables whose name says secret (…TOKEN, …SECRET, …PASSWORD, …API_KEY)	redact	none by construction: the value is the secret
2 — known shapes	vendor-prefixed credentials: ghp_…, github_pat_…, glpat-…, xox[abprs]-…, sk-…, AKIA…, AIza…, hf_…, JWTs, PEM private-key blocks, credentials inside URLs, Authorization: headers	redact	very low: the prefix is meaningful, not random
3 — name=value	an assignment whose key says secret	report only	measured high — see §3

Worked examples, one line each

MEMPALACE_REMOTE_TOKEN=pfrDBfak…            tier 1 — value matches this env's secret
the token is pfrDBfak… apparently           tier 1 — same value, bare in prose, still caught
git clone https://joakim:hunter2@git/x      tier 2 — credentials in a URL
Authorization: Bearer abcdefghijklmnop      tier 2 — header shape
ghp_AbCdEf… / glpat-… / AKIA… / sk-ant-…    tier 2 — vendor prefix
db_password = s3cr3t-p4ssw0rd-xyz           tier 3 — only the KEY suggests it (reported)
GITEA_TOKEN=0123456789abcdef… (40 hex)      tier 3 — indistinguishable from a commit sha

Which tier fired? Read it off the output

The tool prints rule names, not tier numbers, because the rule says what matched. The feeder prefixes them with the tier so both are visible:

[REDACTED] 2026-06-27T23-13-49.jsonl  T1:env-value=1, T2:github-fine-grained=1  fp=ad78c7d4,fcd95ab5
[scrub] 29 redaction(s) applied, 141 name-anchored candidate(s) reported only

RULE_TIERS in mempalace_redact.py is the authoritative mapping, tier_of() reads it, and a self-test fails if any rule is left unclassified — so the two vocabularies cannot drift apart silently.

Tier 1 catches any presentation of a secret — env dump, JSON, error message, URL, prose — because it matches the value itself. It is also, in the default configuration, the only tier that resolves a shape collision: a 40-hex Gitea PAT is byte-identical to a git commit sha, so only a known value (tier 1, redacts) or a naming key (tier 3, reports) can tell them apart. A sha-shaped PAT is therefore caught if and only if it belongs to this machine — an accepted gap, since the alternative is redacting every commit sha in the palace.

3. The false-positive measurement, which changed the design

Tier 3 was originally going to redact. Measured against 52 MB of real fleet transcripts (45 sessions) it produced 403 hits, and inspection of them (with values masked) showed the overwhelming majority were not secrets:

• GITEA_ACCESS_TOKEN=${GITEA_ACCESS_TOKEN:-} — docker-compose interpolation
• const tokens = countTokensForModel — TypeScript source stored as memory
• refreshToken(credentials: Credentials …) — a type annotation
• --setattr=krbPasswordExpiration=20260529… — an IPA attribute holding a date
• |1.FORGE.TOKENS:… — AAAK diary shorthand
• (root@host) Password: followed by unrelated terminal output

Redacting those would corrupt code, configuration and documentation held as memory, in order to catch secrets tier 1 already catches by value. After adding guards for interpolation (${VAR}, $VAR, %VAR%, {{tpl}}), code context, non-secret key suffixes (…Expiration, …Count, …Type) and all-digit values, the enforced count on the same corpus fell from 403 to 29 — and those 29 are tier-1 and tier-2 hits, i.e. real credential shapes.

So tier 3 reports and does not rewrite by default. Set MEMPALACE_REDACT_STRICT=1 to make it enforce, e.g. on a machine whose transcripts are configuration-heavy rather than code-heavy.

4. Known false negatives — stated, not hidden

The scrubber will not catch a novel credential format pasted bare into prose with no name nearby, a secret belonging to a machine whose environment this process cannot see, a base64-of-a-secret, or a secret split across lines.

"The scrubber ran" must never be read as "there are no secrets in here." To keep that measurable rather than assumed, two things are reported:

• every scrub prints a count — including 0 redaction(s), because printing nothing is indistinguishable from a scrubber that never ran;
• suspicions() reports high-entropy strings it did not redact, as (length, fingerprint) pairs rather than values, so the miss rate can be tracked over time and a recurring fingerprint can be investigated by hand.

Findings never carry the secret. A Finding holds the rule, the label, the length and sha256(value)[:8] — enough to recognise the same leak recurring, not enough to recover it.

Fail closed. If the redactor cannot be imported, the feeder refuses to stage rather than staging unscrubbed (exit 3). Override deliberately with MEMPALACE_FEED_ALLOW_UNSCRUBBED=1.

5. The server-side layer (not yet implemented)

Three call sites, because the server has three near-duplicate validators rather than one:

Path	Function	Covers
drawers	config.py → sanitize_content()	add_drawer, update_drawer, diary_write, checkpoint — all four route through it
events	logstream.py → _sanitize_body()	event_append, and event_ack transitively
artifacts	logstream.py → put_artifact()	inlines its own checks; needs its own edit

Server-side scrubbing is tier 2 only (plus optional tier-3 reporting): the hub cannot see a client's environment, so tier 1 is structurally unavailable there. That asymmetry is the reason the client hook is not redundant.

6. Cleaning up a leak that already landed

1. Never re-echo the value while hunting it. Pass it via stdin, never argv (visible in ps on a shared host). Note get_drawer is a trap: reading a drawer in order to redact it prints the secret back into the live transcript.
2. Redact, don't delete. Replacing the value in place keeps the mined transcript's memory value; a stale embedding vector is a cheap price.
3. Scrub the feeder inbox too, or the next mine re-files it.
4. A live session file needs an equal-length in-place overwrite — the agent holds it open, so temp-file-plus-rename loses everything appended afterwards.
5. Expect residue. SQLite keeps old page content in freed pages until VACUUM, so a raw byte scan still matches after a successful UPDATE. Decide explicitly whether that matters: if the credential store on the same host is plaintext anyway, it usually does not.

7. Usage

# unit tests, including the must-not-redact corpus of palace id shapes
python3 bin/mempalace_redact.py --self-test

# scrub anything on stdin; report goes to stderr
some-command | python3 bin/mempalace_redact.py > clean.txt

# also list high-entropy strings that were NOT redacted, as fingerprints
python3 bin/mempalace_redact.py --suspicions < transcript.jsonl > clean.jsonl