Compare commits

...

7 Commits

Author SHA1 Message Date
joakimp a2846a5f7e release: adopt pi 0.84.4 + pi-atelier v0.10.0, and fix the doc claim the pi bump invalidates
Lint / hadolint (push) Successful in 11s
Lint / actionlint (push) Successful in 21s
Publish Docker Image / resolve-versions (push) Successful in 19s
Publish Docker Image / base-decide (push) Successful in 16s
Publish Docker Image / build-base (push) Successful in 59m39s
Publish Docker Image / smoke-studio (push) Successful in 5m22s
Publish Docker Image / smoke (push) Successful in 17m53s
Publish Docker Image / build-variant-studio (push) Successful in 17m14s
Publish Docker Image / build-variant (push) Successful in 18m22s
Publish Docker Image / promote-base-latest (push) Successful in 12s
Publish Docker Image / update-description (push) Successful in 20s
pi 0.84.3 -> 0.84.4 (no Breaking Changes / Removed heading in that section,
grepped). Adopted for three fixes that land on machinery this fleet runs:
#6879 (large tool results crossing the auto-compaction threshold were sent to
the provider before compacting), #8345 (a resumed session corrupted its next
appended entry when the JSONL lacked a trailing newline -- that file is the
memory feeder's input; measured 49/49 clean here beforehand), and #8537
(triggerTurn:false messages sent mid-run were inserted between a tool call and
its result). The mempalace mailbox is outside #8537's precondition: it delivers
at agent_settled with deliverAs:"steer" and no triggerTurn, and 0.84.4 leaves
the documented steer semantics unchanged.

pi-atelier v0.8.2 -> v0.10.0: two minor releases, both UI-only, no BREAKING
notice. v0.9.0 raises its minimum pi to 0.84.0 and, unlike the
0.7.1-under-pi-0.84 startup-hang precedent, encodes it in peerDependencies
(>=0.84.0). Satisfied by PI_VERSION=0.84.4. Both executable floors compare with
sort -V, so 0.10.0 >= 0.7.1 evaluates correctly.

docs/observational-memory.md: pi's own compaction.md gained one paragraph in
0.84.4 -- autoCompact is now also checked mid-run, after a tool batch's results
are appended. Our text said compaction is checked only when pi goes idle and so
"never interrupts a turn"; that was only ever true of the OM trigger. The
section now states both entry points into session_before_compact and the
diagram carries the second edge (mermaid checker re-run: 6 blocks, 44 labels,
0 soft-wrapped, no cut glyphs at 1280px and 800px).

README: the version-pin table had been wrong since v1.8.6 -- 93f986e moved
ARG PI_VERSION to 0.84.3 and MEMPALACE_VERSION to 3.8.0 and neither table row,
so it advertised pi 0.84.2 / mempalace 3.7.1. Corrected, plus the
--expected-version example that would now fail against a 0.84.4 image.

CHANGELOG: Unreleased retitled v1.8.12 (2026-08-31) with the audits above and a
dependency-audit table -- every other component measured SAME (skillset
snapshot --check OK at a12fe5e, 0 commits since baked).
2026-08-31 07:08:41 +02:00
joakimp 58c22afb04 skills: the fingerprint advice was missing its precondition, and the skill had no section on proving absence
Lint / hadolint (push) Successful in 10s
Lint / actionlint (push) Successful in 23s
Docs only; no image behaviour changes.

WHY THIS AND NOT A PRIVATE NOTE. pi@emb-7kj4vr4g reported itself for printing
sha256[:8] fingerprints of GIT_USER_EMAIL, GIT_USER_NAME and HOST_SSH_USER, and
wrote a private rule forbidding it. It had not broken a rule. It followed §2 of
this skill as written, and §2 is incomplete: it says a fingerprint lets you
compare a credential "without ever materialising the secret" with no condition
attached. When two agents independently make the same mistake, the artifact that
taught them both is the bug.

§2 NOW CARRIES THE PRECONDITION. A fingerprint is 32 bits over its INPUT SPACE,
so publishing fp8(x) hands anyone a MEMBERSHIP ORACLE: they can test x == v for
every candidate v they can generate. Safe for a 40-char random token; a wordlist
for a hostname, username, e-mail, port, path, commit SHA or weak password. "High
entropy" is the usual sufficient condition, NOT the test — a commit SHA is
160-bit and still fully enumerable from the repo. Operationally: if you can
imagine writing the wordlist, you cannot publish the fingerprint. Also added:
candidate fingerprints are working memory and never output (an extractor hashes
hostnames and paths too, so the tempting "print what the scanner saw" debug step
leaks low-entropy fingerprints wholesale), and a plain statement that a
fingerprint register is a CONFIRMATION ORACLE for anyone already holding a
candidate corpus — which is exactly how a retired token is identified in old
transcripts, and works identically for someone else holding those same files.

NEW §6, "Proving absence: instrument strength, and four ways a scan lies clean",
placed next to §5 on purpose: §5 optimises against false POSITIVES, and every
failure in §6 is a false NEGATIVE. Triage optimises precision, a gate optimises
recall, and conflating them is what produced three clean reports over secrets
that were really there. Contents: instrument ranking (exact-byte value search >
class/structure pass > fingerprint census) with the instruction to state which
one produced your zero; census vs class passes as different questions, both
failure modes measured on this fleet; the tokenisation trap where quoting alone
decided detectability; scan the index or pushed tree, never the working tree;
git filters never run on symlinks while check-attr claims they do; two-sided
self-tests that abort, incl. the fixture-interaction artifact; row-gone is not
bytes-gone.

Attribution kept per finding: the census/class split and the instrument
ranking's provenance are pi@emb-7kj4vr4g's; exact-byte search over index blobs
is pi@tor-ms22's. The credential sense of "census" originated in this skill, not
with either agent.

TRAP FOR THE NEXT EDITOR, also in the CHANGELOG: the frontmatter description is
now 1022 of 1024 characters. Trim before adding, or the skill silently fails to
load. Verified by parsing the frontmatter (1022 chars, name intact, every prior
trigger phrase retained).

Deployment: baked skill -> needs an image rebuild AND a container recreate to
reach a running container.
2026-08-30 23:32:53 +02:00
joakimp 30094782df shell: source cli_utils' functions, and install the iproute2 that one of them needs
Lint / hadolint (push) Successful in 8s
Lint / actionlint (push) Successful in 30s
v1.8.11 linked cli_utils' bin/ COMMANDS onto PATH and stopped there. Nothing ever
sourced cli_utils.sh, so its 14 FUNCTIONS were missing from every interactive shell
whose $HOME has no zsh rc -- which is the normal case, not an edge case: the
container's interactive shell is bash and zsh is not installed in the image. A
symlink cannot carry a shell function and a function cannot be reached from a
non-interactive shell, so the two mechanisms are disjoint and both are required.
The image was already paying this layer's dependency cost (fzf, bat, fd, rg, jq are
baked partly FOR these functions) while delivering none of its benefit.

The two changes ship together because they are coupled: portcheck is one of the 14,
and it was a hard stub in every image up to v1.8.11 -- neither ss nor ip nor lsof
nor netstat was present, so it printed "portcheck requires at least one of: ss,
lsof, netstat" and exited. Wiring the functions in without iproute2 would have
shipped a visibly broken one.

MEASURED, not assumed:
  - the loader is bash-safe despite the *.zsh filenames: `bash --noprofile --norc`
    exits 0, defines all 14, and they run (pathls, mkcd, up, extract, agents-sync,
    fhist verified). The tree's one zsh-only construct (print -z in fzf/fhist.zsh)
    is already guarded by [[ -n $ZSH_VERSION ]] with a bash fallback.
  - fresh-$HOME seeding resolves 14/14; CLI_UTILS_SOURCE=0 is honoured; an absent
    checkout is a genuinely silent no-op (no output, no leaked _cu).
  - interactive shell startup 12 ms -> 17 ms.
  - iproute2 is ~5.5 MB (4.2 MB itself + 6 libs under --no-install-recommends;
    libpam-cap is a Recommends and correctly dropped). ss lands at /usr/bin/ss,
    ip at /usr/sbin/ip, both already on the developer PATH, and `portcheck --all`
    then correctly identifies the socat listener on 8765.
  - hadolint clean on both Dockerfiles; repo-wide shellcheck -S error and bash -n
    clean. .bash_aliases is outside CI's discovery (no shebang, not *.sh), so it
    was checked by hand with -s bash at error AND warning level.

Named explicitly per this repo's floating-ref rule: /workspace/cli_utils is a HOST
BIND MOUNT, not a pinned ref, so the image now executes unpinned content in every
interactive shell. Errors are left visible rather than sent to /dev/null so that a
future zsh-only file in that repo is diagnosable rather than mysterious, and
CLI_UTILS_SOURCE=0 is the documented escape hatch. It is deliberately independent
of CLI_UTILS_LINK=0: the two disable independent mechanisms.

Deployment: needs a rebuild AND a recreate. $HOME is the container's writable layer
rather than a named volume (verified -- ~/.bash_aliases carries the container start
mtime while ~/.bashrc carries the image's), so the skel file is re-seeded on every
recreate; a host-bind-mounted ~/.bash_aliases is still never overwritten.
2026-08-30 11:48:03 +02:00
joakimp 9b5783f9dd skills: a sixth instance, found by the repo owner within the hour
Lint / actionlint (push) Successful in 16s
Lint / hadolint (push) Successful in 56s
Claimed "## Unreleased is a new convention in this repo" after reading
CHANGELOG.md once — minutes after b615571 had renamed that very section to
`## v1.8.11`. 33 commits touch the heading; the convention is that new entries
land under Unreleased and the heading is renamed at tag time, exactly as it
was renamed out from under my snapshot.

Same root cause as the five instances the section above already records: an
absence observed in one frame, promoted to a fact about the world, with no
second measurement. `git log -S'## Unreleased' -- CHANGELOG.md` was the oracle
and costs one command.

The generalisation is worth more than the instance, so it goes in the habits
block: to learn a repeating PROCESS, read history, not the file. A file's
current content is one frame of a cycle, and the frame you catch may be the one
where the thing you are looking for has just been consumed.
2026-08-30 10:50:20 +02:00
joakimp d9a7fe101b changelog: an Unreleased section for a rule that was already there
Lint / actionlint (push) Successful in 16s
Lint / hadolint (push) Successful in 1m22s
Records the two skill commits ahead of tomorrow's build, and states the finding
that shaped them: the "a negative result is usually your own filter" rule was
already baked, already symlinked in at every container start, and already
survived every recreate — then was violated five times by a session that had it
available. The gap was activation, not persistence, which is why the
cross-cutting form went into the always-appended AGENTS block instead of into a
skill that only loads when a task description matches.

Also notes what the entry's own subject implies for the reader: neither change
reaches a running container until the image is rebuilt AND the container
recreated, since ~/.agents/skills and the global AGENTS.md both live in the
image rather than in a volume or a mount.
2026-08-30 00:52:36 +02:00
joakimp 36e65fe657 skills: add credential-incident-response, and assert it stays baked
Lint / hadolint (push) Successful in 8s
Lint / actionlint (push) Successful in 18s
Carries the facts a two-day credential incident produced, not the discipline:
probe the issuer FIRST (11 of 13 "exposed" credentials were already dead at the
provider, which cost five HTTP requests to learn and was never checked), the
403-vs-401 trap that scoped tokens introduce into liveness probes, revocation
beats deletion for anything already replicated, the three places a secret hides
in a Chroma palace (FTS content, metadata, raw bytes) in coverage order, scope
derivation from measured consumers, and this fleet's age store with its
single-recipient weakness.

Facts transfer between sessions; exhortations do not — hence a separate skill
for the domain knowledge and a one-line pointer in the always-loaded block.

Authored here, so baked is canonical and it is NOT added to skillset-owned.txt.
Skill dirs are picked up by a glob in entrypoint-user.sh, so no registration is
needed — verified rather than assumed, since an enumerated list would have left
the skill inert, a fitting failure given its subject. Three smoke assertions
extended so a future rebuild cannot silently drop it.
2026-08-30 00:50:11 +02:00
joakimp f0ebea2d98 skills: fix the half of the negative-result rule that was wrong
pi-devbox-environment already warned that "a negative result is usually your
own filter" — baked, symlinked in at every container start, authored by an
earlier session. It survived every recreate, was available all of a later
session, and was violated five times anyway. So the gap was never persistence.

That section also closed with "a positive result needs no such scepticism — it
carries its own evidence." That is false, and it aimed the scepticism budget
one way only. Three of those five errors were positives:

  - an SSH handshake SUCCEEDED and greeted me as joakimp while I believed I was
    probing gitea.egl.lan — `Host gitea*` had rewritten HostName
  - a 401 that was a genuine answer from an issuer which never minted the token
  - a "regression" produced by diffing against a value my own -p 2222 flag set

Adds the three missing false-negative rows, replaces the false claim with the
"a positive result only proves what you actually asked" subsection, and records
the two habits that actually caught these: `ssh -G` to learn which rule
captured a hostname, and declaring the expected result before running a check.

The cross-cutting form goes in pi-global-AGENTS.append.md rather than in a
skill, because it has to fire without a task description matching it — being
loadable on demand is exactly what failed. Across all five errors, none was
caught by re-reading my reasoning; every one was caught by a second measurement
that disagreed.
2026-08-30 00:50:11 +02:00
11 changed files with 778 additions and 20 deletions
+302
View File
@@ -11,6 +11,308 @@ Pre-v1.0.0 tags followed the pi npm version (`v{pi_version}[letter]`).
---
## v1.8.12 — 2026-08-31
**`pi` `0.84.3` → `0.84.4`, and `pi-atelier` `v0.8.2` → `v0.10.0`.** Both audited
by the routine in `Dockerfile.variant` rather than adopted on sight, and the
audit notes live next to the pins where the next reader will meet them.
**pi 0.84.4 (published 2026-08-28) carries no `Breaking Changes` and no
`Removed` heading** — checked by grepping the section, 0 matches, which is worth
stating because 0.84.3 *did* have one. It was adopted for three fixes that land
on machinery this fleet runs every day, not for the feature list:
- **#6879** — a large tool result crossing the auto-compaction threshold used to
be sent to the provider *before* compaction. Pi now compacts between tool
execution and the next assistant response inside the same run. That is the
shape of nearly every session on these boxes, where a single `event_list` or
palace search returns hundreds of KB.
- **#8345** — a resumed session corrupted its next appended entry when the JSONL
file lacked a trailing newline. That file is the memory feeder's *input*, so
the failure would have surfaced as unexplained gaps in `wing_conversations`
rather than as an error. Measured on tor-ms22 before bumping: 49/49
transcripts end in a newline and 0 lines fail `json.loads` — this corpus was
never bitten, and we now know that rather than hope it.
- **#8537** — extension messages sent with `triggerTurn: false` *while the agent
is running* were inserted between a tool call and its result, so
order-validating providers rejected the replayed history. **The mempalace
mailbox is outside that precondition**: it delivers at `agent_settled`, when
no inference is in flight, with `{deliverAs: "steer"}` and deliberately no
`triggerTurn`. 0.84.4 also leaves the documented steer semantics untouched
("delivered after the current assistant turn finishes executing its tool
calls, before the next LLM call"), so RFC 003 §7.11 stands as written. Recorded
because this fix is precisely what would make a *mid-run* delivery safe, which
is the only reason we would ever change that call.
Also new and relevant, though nothing here uses them yet: `ui_prompt_start` /
`ui_prompt_end` extension events (the `docs/extensions.md` diff is add-only — no
steer or `triggerTurn` semantics moved), and an RPC `clear_queue` that returns
and removes queued steering messages. The second one can discard an
already-delivered but unconsumed mailbox steer; that is survivable because the
mailbox re-delivers on `MEMPALACE_MAILBOX_RESURFACE_MS` (default 3600000), and it
is written down here so a future "the mailbox lost a message" report has a
candidate cause. The three new `PI_HYPERLINKS` / `PI_IMAGE_PROTOCOL` /
`PI_TRUE_COLOR` environment variables were grepped against this whole repo: no
collisions with anything the image sets.
**The bump moved one documented mechanism, so `docs/observational-memory.md` §3
moved with it.** Pi's own `docs/compaction.md` gained exactly one paragraph in
0.84.4: the `autoCompact` threshold is now *also* checked mid-run, after a tool
batch's results are appended and before the next assistant response, skipped only
when that batch ends the run and no queued message needs another response. Our
doc said compaction is "checked when pi goes idle, so it never interrupts a
turn". That was only ever true of observational-memory's **own** trigger
(`compaction-trigger.ts` hooks `agent_settled`); read as a statement about pi it
is now false. `session_before_compact` (`compaction-hook.ts`) therefore has
**two** entry points and the second can fire inside a turn — harmless for the
ledger fold, which makes no model call, but a doc that ships a false promise
about when a hook runs is worse than one that admits two paths. The §3 mermaid
diagram gained the second edge, and the whole file re-passes the bundled mermaid
checker (6 blocks, 44 labels, 0 soft-wrapped, no cut glyphs at 1280px and
800px).
**pi-atelier `v0.8.2` → `v0.10.0` is two minor releases and both are UI-only** —
Sidebar kept calm during an active Turn, composer frame and Status Rail polish,
fullscreen-copy-safe Sidebar, Windows path normalisation, Workspace Pulse
deferred until pi trusts the project. Neither release carries a BREAKING notice.
The coupling that matters runs the *opposite* way to this pin's hard-earned
floor: v0.9.0 renders the Sidebar as a separate split-layout child and therefore
"raises the minimum supported Pi version to 0.84.0", and — unlike the
0.7.1-under-pi-0.84 startup-hang precedent, which its metadata never encoded —
this time `peerDependencies` says so (`>=0.84.0`, up from `>=0.80.7`). Satisfied
with room to spare by `PI_VERSION=0.84.4`. It also pairs deliberately with a
0.84.4 feature: atelier keeps Sidebar content out of the fullscreen transcript
selection while pi adds `fullscreenCopyOnSelect` and Ctrl+X for the selection
itself. Both executable floors (`scripts/smoke-test.sh`,
`scripts/recreate-sanity-check.sh`) compare with `sort -V`, so `0.10.0 >= 0.7.1`
is evaluated correctly — verified by running the comparison, because the string
form of that test reads `0.10.0` as *older* than `0.7.1`.
**While bumping the pins, the README's own pin table turned out to have been
wrong since v1.8.6.** It advertised pi `0.84.2` and mempalace `3.7.1` in the very
table whose purpose is to tell a reader what is pinned and where. Both rows went
stale in the *same* commit — `93f986e` (v1.8.6, "adopt pi 0.84.3 + mempalace
3.8.0") moved both `ARG`s and neither table row; the rows themselves date from
`29b6209` (v1.8.0) and `2ebf00d` (v1.8.4). Only atelier's row was still true.
All three corrected now, and the `--expected-version 0.84.3` example in the
recreate-sanity section updated too, since that one is a copy-pasteable command
that would now fail against a 0.84.4 image. Worth noting how it survived two
releases: nothing checks prose against the `ARG`s, so this table has to be
remembered by hand on every pin bump, and once it was not.
**`credential-incident-response` gained the section its own guidance had been
missing, and §2 gained a precondition it should always have carried.** Docs only;
no image behaviour moves. Both changes came out of a session where three separate
detectors reported *clean* over secrets that were really there — the skill was
the artifact that had taught two agents the pattern, so the fix belongs here
rather than in either operator's private notes.
**§2 previously said an 8-hex fingerprint lets you compare a credential "without
ever materialising the secret", with no condition attached.** That is true only
when the *input space* is unreachable. A fingerprint is 32 bits over whatever it
was computed from, so publishing `fp8(x)` hands anyone a **membership oracle**:
they can test `x == v` for every candidate `v` they can generate. For a 40-char
random token, fine. For a hostname, username, e-mail, port, path, commit SHA or
weak password, that candidate set is a wordlist — and note that "high entropy" is
the usual sufficient condition, not the test: a commit SHA is 160-bit and still
fully enumerable from the repo. Two agents on this fleet published fingerprints of
`GIT_USER_EMAIL`-class values while following this section as written; harmless in
that instance, because those values sit in every commit trailer already, but the
guidance licensed it. §2 now states the precondition, adds that candidate
fingerprints are working memory and never output (a scanner hashes hostnames and
paths too, so "print what it saw" leaks wholesale), and names what a fingerprint
register *is* — a confirmation oracle for anyone already holding a candidate
corpus, which is exactly how a retired token gets identified in old transcripts,
and works the same way for someone else holding those files.
**New §6, "Proving absence: instrument strength, and four ways a scan lies
clean".** Deliberately placed next to §5, because §5 optimises against false
*positives* (name-anchoring, provenance — what stops a triage sweep drowning in
session UUIDs) and every failure in §6 is a false *negative*. Triage optimises
precision; a gate optimises recall, and conflating the two is what produced the
clean reports. It carries: an instrument-strength ranking (exact-byte value search
> class/structure pass > fingerprint census) with the standing instruction to say
which one produced your zero; census and class passes answering different
questions, with both failure modes measured here — a class-only pre-commit hook
passed plaintext UUID API credentials to a shared repo twice because a UUID has no
key header, while a census-only gate reported 0 hits with freshly-synced SSH
private keys in the tree because no key is in the census; the tokenisation trap,
where maximal-run extraction swallows an unquoted `VAR=<uuid>` so the value is
never hashed alone while a *quoted* one is found, meaning quoting alone decided
detectability; scan the index or the pushed tree, never the working tree, plus why
a repo-only fix on an rsync-published mirror is temporary rather than weaker; git
filters never running on symlinks, where `check-attr` answers `git-crypt` for a
path it can never encrypt, so a coverage audit must join the attribute against the
file mode and verify the blob magic; two-sided self-tests that abort, including
the fixture-interaction artifact where a quoted and unquoted probe share one
buffer and make the weak extractor look as strong as the union; and row-gone is
not bytes-gone, since a correct sqlite DELETE leaves the payload in freelist pages
until VACUUM.
Findings contributed by `pi@emb-7kj4vr4g` (the census/class split, and the
instrument ranking's provenance) and `pi@tor-ms22` (exact-byte value search over
index blobs). The description's trigger list grew accordingly and is 1022/1024
characters — **it has almost no headroom, so trim before adding to it**, or the
skill silently fails to load.
**Deployment:** the skill is baked at
`/usr/local/share/pi-devbox/skills/credential-incident-response/`, so this needs
an image rebuild **and** a container recreate to reach any running container.
**Two vendored skills changed, and one of the changes is a correction rather than
an addition.** Nothing about the image's behaviour moves; this is entirely about
what the next agent reads before it acts.
**`pi-devbox-environment` §2 had a rule that was half wrong, and the wrong half
cost five findings in one session.** The section "A negative result is usually
your own filter" closed with *"a positive result needs no such scepticism — it
carries its own evidence."* That sentence is false. A positive result is evidence
about the question your command *actually posed*, which may not be the question
you meant — and the failure is invisible precisely because the command succeeded.
Three measured instances, all from 2026-08-29, all filed as fact before being
caught: an SSH handshake that succeeded and greeted the agent as `joakimp` while
it believed it was probing `gitea.egl.lan` (a `Host gitea*` block had rewritten
`HostName`, so it authenticated to the wrong Gitea instance); a `401` that was a
genuine answer from an issuer which had never minted the credential being tested;
and a "regression" produced by diffing `ssh -G` output against a `2222` that the
agent's own earlier `-p 2222` flag had supplied. The section now carries a
counterpart, *"…and a positive result only proves what you actually asked"*, plus
the three false-negative rows that session added (a palace scan that queried
`embedding_metadata` while documents live in `embedding_fulltext_search_content`;
a token declared dead on a 401 from the wrong issuer; a host declared unreachable
after trying two of its three open ports, with the port written in an environment
variable the agent already held).
**The cross-cutting form of that rule went into `pi-global-AGENTS.append.md`, not
into the skill — deliberately, and this is the whole point of the change.** The
rule *already existed* in the baked skill, authored by an earlier session,
symlinked into `~/.agents/skills/` at every container start. It survived every
recreate, was available for the entire session that broke it, and was violated
five times anyway. So the gap was never persistence; it was **activation**.
A reasoning rule that only loads when a task description happens to match it
cannot fire on the occasions that need it, because "I am about to state something
false" is not a recognisable task type. The always-appended block is read by every
agent in every container without being asked for, which is the only property that
matters here. Writing a sixth document restating the rule would have felt like
progress and changed nothing.
**New baked skill: `credential-incident-response`.** Authored here, so the baked
copy is canonical and it is *not* listed in `skillset-owned.txt`. It carries the
*facts* a two-day credential incident produced, on the theory that facts transfer
between sessions where exhortations do not: probe the issuing provider **first**
(11 of 13 "exposed" credentials in that sweep turned out to be already dead at the
provider — five HTTP requests would have established it, and nobody asked);
`sha256[:8]` fingerprints as leak-free credential identity; the `403`-vs-`401`
trap that scoped tokens introduce into liveness probes, where a live token looks
revoked on `/api/v1/user`; **revocation beats deletion** for anything already
replicated, because deletion is best-effort over an unbounded copy set (FTS shadow
rows, per-host feed inboxes, sqlite free pages, mesh replicas, backups) while
revocation invalidates copies nobody enumerated; the three places a secret hides
in a Chroma palace, in coverage order; deriving least-privilege scopes from
*measured* consumers; and the exposures rotation does not fix (cleartext channels,
git history, agent-authored drawers).
**Three smoke assertions extended** so a rebuild cannot silently drop the new
skill: baked-file existence, resolves-to-the-baked-tree, and reported as `baked`
by `pi-devbox-version`. Skill directories are picked up by a glob in
`entrypoint-user.sh`, so no registration was needed — verified rather than
assumed, since an enumerated list would have left the skill inert, which would
have been a fitting way for *this* skill to fail.
Neither skills change reaches a running container until the image is rebuilt **and**
the container recreated: `~/.agents/skills/` and the global `AGENTS.md` both live in
the image, not in a volume or a mount.
**`cli_utils`' shell *functions* are now sourced, closing the half of that wiring
the image never did.** v1.8.11 linked the repo's `bin/` **commands** into
`~/.local/bin` so they resolve in non-interactive shells; nothing ever sourced
`cli_utils.sh`, so its 14 **functions** (`fgit`, `fhist`, `fssh`, `fdocker`,
`fmark`, `fproc`, `fex`, `fenv`, `extract`, `mkcd`, `pathls`, `portcheck`,
`agents-sync`, `up`) were missing from every interactive shell whose `$HOME` had
no zsh rc. That is the normal case, not an edge case: the container's interactive
shell is bash and **zsh is not installed in the image**. A symlink cannot carry a
shell function and a function cannot be reached from a non-interactive shell, so
the two mechanisms are disjoint and both are required — the image had been paying
this layer's dependency cost (`fzf`, `bat`, `fd`, `rg`, `jq` are baked partly *for*
these functions) while delivering none of its benefit. Now sourced from
`/etc/skel-devbox/.bash_aliases`, with the same detection order as the symlink
block so commands and functions can never come from two different clones.
`CLI_UTILS_SOURCE=0` opts out, deliberately independent of `CLI_UTILS_LINK=0`
because the two disable independent mechanisms. Measured: all 14 resolve in a
freshly-seeded `$HOME`, the opt-out is honoured, an absent checkout is a genuinely
silent no-op (no output, no leaked `_cu` variable), and interactive shell startup
goes from 12 ms to 17 ms.
**Named explicitly, per this repo's own floating-ref rule: `/workspace/cli_utils`
is a host bind mount, not a pinned ref.** Sourcing it means the image now executes
content it does not pin, on every interactive shell, on every device. It is
bash-safe today and that was measured rather than assumed — sourcing under
`bash --noprofile --norc` exits 0 and defines all 14 despite the `*.zsh`
filenames, the functions run, and the tree's single zsh-only construct (`print -z`
in `fzf/fhist.zsh`) is already guarded by `[[ -n $ZSH_VERSION ]]` with a bash
fallback. The residual risk is future content: a cli_utils commit adding a
genuinely zsh-only file would surface as parse errors at every prompt, fleet-wide.
Errors are therefore left visible rather than sent to `/dev/null`, so the failure
is diagnosable, and `CLI_UTILS_SOURCE=0` is the one-line escape hatch.
**`iproute2` is installed, so the container can answer "what is listening in
here".** Neither `ss` nor `ip` was present in any image up to and including
v1.8.11 — nor `lsof`, nor `netstat` — which made `cli_utils`' `portcheck` a hard
stub that printed `portcheck requires at least one of: ss, lsof, netstat` and
exited. `ss` satisfies its preferred branch (`ss -tlnp`), which is also the only
branch that reports the owning PID. `net-tools` is deliberately **not** added
(`netstat` is deprecated and only a fallback path) and neither is `lsof` (~500 KB
for a third route to the same answer). Cost measured, not estimated: ~5.5 MB total
— `iproute2` is 4.2 MB and pulls six libs under `--no-install-recommends`
(`libbpf1`, `libmnl0`, `libtirpc-common`, `libtirpc3t64`, `libxtables12`,
`libcap2-bin`; `libpam-cap` is a Recommends and is correctly dropped). Verified in
a live container: `ss` at `/usr/bin/ss`, `ip` at `/usr/sbin/ip`, both already on
the developer `PATH`, and `portcheck --all` then correctly identifies the `socat`
listener on 8765.
The two changes above also need a rebuild **and** a recreate, for a different
reason than the skills: `$HOME` is the container's writable layer rather than a
named volume (verified — `~/.bash_aliases` carries the container's start mtime
while `~/.bashrc` carries the image's), so the skel file is re-seeded on every
recreate. A `$HOME/.bash_aliases` that is bind-mounted from the host is still
never overwritten, which is the existing contract.
### Dependency audit (2026-08-31)
Every component checked against upstream by direct command, not assumed:
| Component | Baked in v1.8.11 | Upstream now | Action |
|---|---|---|---|
| **pi** | `0.84.3` (pinned) | **`0.84.4`** is npm latest | bumped + audited (above) |
| **pi-atelier** | `v0.8.2` (pinned) | **`v0.10.0`** highest tag | bumped + audited (above) |
| mempalace | `3.8.0` (pinned) | `3.8.0` is PyPI latest | none |
| skillset (mempalace fallback snapshot) | `a12fe5e` | `a12fe5e` == `origin/main`, 0 commits since | none — `--check` reports OK, no NOTICE |
| mempalace-toolkit | `21023e7` | `21023e7` | none |
| pi-toolkit | `0e1369e` | `0e1369e` | none |
| pi-extensions | `2022887` | `2022887` | none |
| pi-fork | `bf702b4` | `bf702b4` | none |
| pi-observational-memory | `ce9fc98` | `ce9fc98` (v3.0.4, peerDeps `*` → no pi floor to clear) | none |
| pi-studio (studio variant) | `3328b3d` | `3328b3d` | none |
| floating `*_VERSION=latest` tools (16) | — | 14 already at latest; `git-lfs` `3.7.1`→`3.8.0` (feature, no breaking section), `uv` `0.12.6`→`0.12.7` (patch) | adopted implicitly by the rebuild; named here per this repo's floating-ref rule |
| node | major pin `22`, installed `v22.23.2` | `v22.23.2` is the newest 22.x | none — a newer LTS *line* (24.x) exists and is deliberately not tracked |
Two method notes, because both would have produced a confident wrong answer:
- **An annotated tag's `ls-remote` SHA is the tag object, not the commit.**
`refs/tags/v0.8.2` is `6e07bf85` while `refs/tags/v0.8.2^{}` is `159f34cf` —
the value actually baked. Comparing the un-dereferenced form reported
`pi-atelier` as *drifted from its own pin*, which would have been a false
integrity alarm about the one component whose pin is load-bearing. Always
deref with `^{}` before calling a pin broken.
- **`git ls-remote --tags | sort -V | tail` is not a "latest release" proxy.**
`typst/typst` carries date-style tags (`v23-03-28`) and `mikefarah/yq` carries
`vTestA`/`vTestB`; both sort *after* the real releases. `Dockerfile.base`
itself resolves `latest` by reading the `Location` of
`curl -sI …/releases/latest`, so replaying that exact step is both
noise-immune and the same source of truth the build will see.
---
## v1.8.11 — 2026-08-27
**Shell state that the writable layer eats on every recreate now gets rebuilt at
+19
View File
@@ -83,6 +83,24 @@ ENV DEBIAN_FRONTEND=noninteractive
# above); TERM=xterm-ghostty is compiled from an alias further
# down (ncurses ships `ghostty`, not `xterm-ghostty`). iTerm2
# defaults to xterm-256color (ncurses-base), so needs nothing.
# iproute2 — `ss` (socket statistics) and `ip`. Measured 2026-08-30 on
# v1.8.11: NEITHER was present, so the container could not
# answer "what is listening in here" by any means, and
# cli_utils' `portcheck` was a hard stub — it prints
# "portcheck requires at least one of: ss, lsof, netstat" and
# all three were absent. `ss` satisfies its preferred branch
# (`ss -tlnp`), which is also the branch that reports the
# owning PID, so nothing further is needed: net-tools is
# deliberately NOT added (`netstat` is deprecated and only a
# fallback branch) and neither is lsof (~500 KB for a third
# path to the same answer). ~5.5 MB total: iproute2 itself is
# 4.2 MB and pulls 6 libs under --no-install-recommends
# (libbpf1, libmnl0, libtirpc-common, libtirpc3t64,
# libxtables12, libcap2-bin — libpam-cap is a Recommends and
# is correctly dropped). Verified end-to-end in a live
# container: `ss` lands at /usr/bin/ss, `ip` at /usr/sbin/ip
# (both already on the developer PATH), and `portcheck --all`
# then correctly identifies the socat listener on 8765.
RUN apt-get update && \
apt-get upgrade -y --no-install-recommends && \
apt-get install -y --no-install-recommends \
@@ -122,6 +140,7 @@ RUN apt-get update && \
nano \
kitty-terminfo \
ncurses-term \
iproute2 \
&& ln -s /usr/bin/fdfind /usr/local/bin/fd \
&& apt-get clean \
&& rm -rf /var/lib/apt/lists/*
+45 -5
View File
@@ -57,6 +57,32 @@ ARG USER_NAME=developer
# v0.74.0..v0.75.5; discovered + fixed in v0.75.5b, 2026-05-23). The `latest`
# branch below is kept only for a deliberate local `docker build` override.
#
# AUDITED AT 0.84.4 (2026-08-31, was 0.84.3): NO "Breaking Changes" and no
# "Removed" heading in the 0.84.4 section (grepped, 0 matches) — unlike 0.84.3,
# whose heading is described in the paragraph below and stays audited. Adopted
# for three fixes that land on machinery this fleet actually runs:
# - #6879 large tool results crossing the auto-compaction threshold were sent
# to the provider BEFORE compacting; pi now compacts between tool execution
# and the next assistant response in the same run. This is the shape of
# nearly every session here (multi-hundred-KB logstream/palace tool output).
# - #8345 a resumed session corrupted its next appended entry when the JSONL
# lacked a trailing newline. That file is the memory feeder's own input.
# Measured on tor-ms22 before the bump: 49/49 transcripts end in a newline,
# 0 lines fail json.loads — the bug had not bitten this corpus.
# - #8537 extension messages sent with `triggerTurn: false` WHILE THE AGENT IS
# RUNNING were inserted between a tool call and its result, so
# order-validating providers rejected the replayed history. The mempalace
# mailbox is outside that precondition — it delivers at `agent_settled`
# (idle) with `{deliverAs:"steer"}` and deliberately no `triggerTurn` — and
# 0.84.4 leaves the documented steer semantics unchanged, so RFC 003 §7.11
# still holds. Recorded because the fix is what would make a future mid-run
# delivery safe, which is the only reason we would ever change that call.
# One doc consequence, fixed in this same release: pi's own docs/compaction.md
# gained exactly one paragraph — the autoCompact threshold is now ALSO checked
# mid-run, after a tool batch's results are appended. See
# docs/observational-memory.md §3, which had said compaction is only checked
# when pi goes idle.
#
# AUDITED AT 0.84.3 (2026-08-25, was 0.84.2): upstream's notes carry a
# "Breaking Changes" heading — `GoogleThinkingLevel` renamed to
# `GoogleApiThinkingLevel`. INERT FOR THIS IMAGE: all four vendored companions
@@ -69,9 +95,7 @@ ARG USER_NAME=developer
# `.agents/skills/<group>/` directories were not discovered, and root Markdown
# files such as README.md / AGENTS.md inside a skill dir were reported as
# broken skills unless they declared valid skill frontmatter.
# pi-atelier needs no companion bump: v0.8.2 clears the >=0.7.1 floor that
# pi >= 0.84 requires (see PI_ATELIER_REF below).
ARG PI_VERSION=0.84.3
ARG PI_VERSION=0.84.4
ARG PI_TOOLKIT_REF=main
ARG PI_EXTENSIONS_REF=main
# Repo URLs default to the canonical gitea origin but are overridable so a
@@ -101,15 +125,31 @@ ARG PI_OBSMEM_REF=master
# pin and PI_VERSION together, checking atelier's CHANGELOG for the pi
# version it claims to track.
#
# AUDITED AT v0.10.0 (2026-08-31, was v0.8.2 — two minor releases): no
# BREAKING notice in either release, and both are UI-only (Sidebar calm during
# an active Turn, composer frame + Status Rail, fullscreen-copy-safe Sidebar,
# Windows path normalisation, Workspace Pulse deferred until pi trusts the
# project). The one coupling that matters runs the OPPOSITE way to the floor
# above: v0.9.0 renders the Sidebar as a separate split-layout child and
# therefore "raises the minimum supported Pi version to 0.84.0", which its
# peerDependencies do encode this time (`>=0.84.0`, up from `>=0.80.7`).
# Satisfied with room to spare by PI_VERSION 0.84.4 above — and note that both
# executable floors (scripts/smoke-test.sh, scripts/recreate-sanity-check.sh)
# compare with `sort -V`, so 0.10.0 >= 0.7.1 is evaluated correctly rather than
# as the string comparison that would read 0.10.0 as older than 0.7.1.
# Pairs deliberately with pi 0.84.4's own fullscreen selection-copy controls:
# atelier keeps Sidebar content out of the transcript selection, pi adds
# `fullscreenCopyOnSelect` + Ctrl+X for the selection itself.
#
# No `npm install` step, unlike pi-fork/pi-observational-memory/pi-studio:
# pi-atelier declares ZERO runtime dependencies (only peerDeps, satisfied by
# the baked pi) and has no build step — pi loads its TypeScript directly from
# the /opt checkout. Adding an install here would be a no-op that only costs
# build time.
ARG PI_ATELIER_REPO=https://github.com/michaelmjhhhh/pi-atelier.git
ARG PI_ATELIER_REF=v0.8.2
ARG PI_ATELIER_REF=v0.10.0
# Human-readable tag PI_ATELIER_REF was resolved from; recorded as a label.
ARG PI_ATELIER_VERSION=v0.8.2
ARG PI_ATELIER_VERSION=v0.10.0
RUN set -e && \
# git_fetch_ref: clone-equivalent helper that accepts EITHER a branch name
+4 -4
View File
@@ -1093,7 +1093,7 @@ persisted volumes survived, and pi runtime wiring is intact:
```bash
./scripts/recreate-sanity-check.sh # auto-detects variant
./scripts/recreate-sanity-check.sh --expected-image-version 1.8.9 # assert the pi-devbox release tag
./scripts/recreate-sanity-check.sh --expected-version 0.84.3 # assert the pi coding agent version
./scripts/recreate-sanity-check.sh --expected-version 0.84.4 # assert the pi coding agent version
```
Those are **two different versions**, and the flags are not interchangeable:
@@ -1132,9 +1132,9 @@ resolved to `latest` at build time:
| Component | Pin | Where |
|---|---|---|
| pi | `0.84.2` | `ARG PI_VERSION` — `Dockerfile.variant` |
| pi-atelier | `v0.8.2` | `ARG PI_ATELIER_REF` — `Dockerfile.variant` |
| mempalace | `3.7.1` | `ARG MEMPALACE_VERSION` — `Dockerfile.base` |
| pi | `0.84.4` | `ARG PI_VERSION` — `Dockerfile.variant` |
| pi-atelier | `v0.10.0` | `ARG PI_ATELIER_REF` — `Dockerfile.variant` |
| mempalace | `3.8.0` | `ARG MEMPALACE_VERSION` — `Dockerfile.base` |
The objective is **not** to freeze versions. Bumping is routine — usually one
line plus a changelog note. The objective is that adopting a new upstream
+14 -6
View File
@@ -21,8 +21,9 @@ palace, see
> Verified on pi-devbox **v1.8.9** (`release_tag v1.8.9`, source `aac4a1c`),
> which bakes pi-observational-memory **v3.0.4** at commit `ce9fc98` — the value
> in `/etc/pi-devbox/build-manifest.json` → `components.pi-observational-memory`.
> Every number below was read from that tree, from pi 0.84.3's own docs, or from
> the live container.
> Every number below was read from that tree, from pi's own docs, or from the
> live container. The pi-side mechanics were first read at pi **0.84.3** and
> re-checked at **0.84.4** (v1.8.12), which moved one of them — see §3.
---
@@ -93,6 +94,7 @@ flowchart TD
S(["agent_settled"]) --> C{"81k tokens<br/>since compacting?"}
C -- yes --> CP["ctx.compact()"]
CP --> H(["session_before_compact"])
A(["pi autoCompact<br/>idle, or mid-run<br/>after a tool batch"]) --> H
H --> F["fold the ledger<br/>no model call"]
F --> VIS["compacted memory"]
```
@@ -105,10 +107,16 @@ flowchart TD
a *successful same-turn* reflection **and** an active pool above
`observationsPoolTargetTokens` [10000]. Not a third worker on a third
threshold.
- **compaction** — `compactAfterTokens` [81000], checked when pi goes idle, so it
never interrupts a turn. Pi will also compact on its own when the context is
nearly full (`contextTokens > contextWindow - reserveTokens`, `reserveTokens`
[16384]).
- **compaction** — `compactAfterTokens` [81000], checked at `agent_settled`, so
*this* trigger never interrupts a turn. Pi will also compact on its own when
the context is nearly full (`contextTokens > contextWindow - reserveTokens`,
`reserveTokens` [16384]), and **from pi 0.84.4 that check also runs mid-run** —
after a tool batch's results are appended, before the next assistant response,
skipped only when the batch ends the run and no queued message needs another
response. So `session_before_compact` has **two** entry points and the second
one can fire *inside* a turn. Harmless for the fold itself, which makes no
model call, but worth stating plainly: "never interrupts a turn" was only ever
true of the observational-memory trigger, and reads as a promise about pi's.
## 4. What compaction actually does to your context
+44
View File
@@ -116,6 +116,50 @@ if command -v fzf >/dev/null 2>&1; then
eval "$(fzf --bash)" 2>/dev/null || true
fi
# cli_utils — shell FUNCTIONS (fgit, fhist, fssh, portcheck, up, mkcd, extract,
# agents-sync, …). This is the OTHER HALF of the cli_utils wiring, and until
# v1.8.11 the image shipped only one half. entrypoint-user.sh symlinks the repo's
# bin/ COMMANDS into ~/.local/bin, which is what makes them resolve in
# NON-interactive shells (docker exec, agent tool shells, scripts). A symlink
# cannot carry a shell function, and a function cannot be reached from a
# non-interactive shell, so the two mechanisms are disjoint and both are
# required. Nothing sourced the loader: measured 2026-08-30 on v1.8.11, all 14
# functions were simply missing on a device whose $HOME has no zsh rc — which is
# the normal case, since the container's interactive shell is bash and zsh is not
# installed in the image. The image was already paying this layer's dependency
# cost (fzf, bat, fd, rg, jq are all baked partly FOR these functions) while
# delivering none of its benefit.
#
# Detection order deliberately mirrors the symlink block in entrypoint-user.sh so
# that commands and functions can never come from two different clones.
# CLI_UTILS_SOURCE=0 opts out. That is independent of CLI_UTILS_LINK=0 on purpose:
# they disable independent mechanisms, and someone who wants PATH commands
# without 14 extra functions in every prompt (or vice versa) should be able to
# say so.
#
# THE LOADER IS BASH-SAFE, MEASURED, NOT ASSUMED: despite every function file
# being named *.zsh, sourcing cli_utils.sh under `bash --noprofile --norc` exits
# 0 with no errors and defines all 14, and they run (pathls, mkcd, up, extract,
# agents-sync, fhist all verified). The single zsh-only construct in the tree
# (`print -z` in fzf/fhist.zsh) is already guarded by [[ -n $ZSH_VERSION ]] with
# a bash fallback, and the loader's own header states "bash & zsh compatible".
# ACCEPTED RISK, stated plainly: /workspace/cli_utils is a HOST BIND MOUNT, so
# unlike a pinned git ref this content floats outside the image's control. A
# future cli_utils commit that adds a genuinely zsh-only file would surface as
# parse errors at every prompt on every device. Errors are left VISIBLE rather
# than sent to /dev/null so that failure is diagnosable instead of mysterious,
# and CLI_UTILS_SOURCE=0 is the documented one-line escape hatch.
if [ "${CLI_UTILS_SOURCE:-1}" != "0" ]; then
for _cu in "${CLI_UTILS_CONTAINER_PATH:-}" /workspace/cli_utils "$HOME/cli_utils" /workspace/*/cli_utils; do
[ -n "$_cu" ] || continue
if [ -r "$_cu/cli_utils.sh" ]; then
. "$_cu/cli_utils.sh" || true
break
fi
done
unset _cu
fi
# ── PROMPT_COMMAND: flush history every prompt ───────────────────────
# Installed AFTER zoxide init so zoxide's hook is already in place;
# we append with a newline separator to avoid the ';;' parse error
@@ -70,3 +70,41 @@ rather than merely confusing you:
local disk, so `mempalace search` can return older and different results than
the MCP tools while both look correct. Use the MCP tools for the central
palace; the CLI only for a local one.
## Before you file a finding: second measurement, different route
This is here rather than in a skill because it has to fire *without* a matching
task description, and because the version of it that lived only in a skill was
violated five times in one session by an agent that had the skill available.
**Any claim you are about to record as fact — in a drawer, a diary entry, a
coordination event, or a report to the user — needs a second measurement taken
by a different route.** Not a re-read of your reasoning: re-reading has caught
zero of these. A disagreeing measurement has caught all of them.
The two shapes that get filed as fact and are not:
- **A negative result** (`401`, connection refused, zero rows, "not found") is
first a claim about *your filter*, not about the world. Wrong host, wrong port,
wrong table, capped output.
- **A positive result** proves only what your command *actually asked*. An SSH
handshake can succeed against the wrong host (`ssh -G` tells you which rule
captured the name); a `401` can be a real answer from an issuer that never
minted the credential.
Cheapest habit that works: **write the expected result next to each check before
running it**, then diff. Expectations declared up front turn a silent wrong
assumption into a visible mismatch. And if you cannot think of a second route to
the same fact, you do not have a finding — you have a hypothesis, so label it as
one.
## Handling an exposed credential
If a task touches a leaked secret, a token rotation, "is this credential still
live?", whether to delete stored content, or which scopes a new token needs:
**read `~/.agents/skills/credential-incident-response/SKILL.md` first.** One rule
is load-bearing enough to state here: **probe the issuing provider before doing
anything else** — most "exposed" credentials in a long-lived fleet are already
dead, and the ones that are live are often far more privileged than assumed.
Severity first, cleanup second, and prefer **revocation over deletion** for
anything already replicated.
@@ -9,6 +9,7 @@ one", which was a bug).
| skill | owner | how it gets here |
|-------|-------|------------------|
| `pi-devbox-environment` | pi-devbox (this repo) | authored here; the canonical copy |
| `credential-incident-response` | pi-devbox (this repo) | authored here; the canonical copy |
| `pi-extensions` | the `pi-extensions` package repo (`skill/`) | **vendored fallback** + refreshed at build |
| `mempalace` | the `skillset` repo | **vendored fallback** (snapshot only) |
@@ -0,0 +1,255 @@
---
name: credential-incident-response
description: >-
Respond correctly when a live credential is found where it should not be —
in a chat transcript, a MemPalace drawer, a log, a git-tracked config, or an
agent-authored note. Load this whenever a task involves a leaked/exposed
secret, a token rotation, a "is this credential still live?" question, deciding
whether to delete or scrub stored content, proving a corpus is clean, or
choosing scopes for a new API token. Covers the mandatory order of operations
(probe the issuer FIRST — severity before cleanliness), leak-free identity via
sha256[:8] fingerprints and when publishing one is safe,
why revocation beats deletion for anything already replicated, scopes derived
from measured consumers, the three places a secret hides in a Chroma palace, how to prove ABSENCE rather than assume it (instrument strength,
census vs class passes, the tokenisation trap where quoting decides detectability, why git filters never run on symlinks, self-tests that abort),
where this fleet's secrets live, and what rotation does NOT fix.
---
# Credential incident response
A leaked credential is a **severity** question before it is a cleanliness
question. Two days of scrubbing, redaction plumbing and deletion planning were
once spent on a set of 13 credentials of which **11 were already dead at the
provider** — a fact that cost five HTTP requests to establish and was never
checked. Meanwhile the two live ones turned out to be instance-owner **admin**
tokens, which nobody had looked at either.
## 1. Order of operations — do not reorder this
1. **Is it still accepted?** Probe the issuing provider. Dead credential →
hygiene item, stop panicking. Live → incident, continue.
2. **What can it do?** Read the identity back. `is_admin`, `id=1`, scopes,
which account. A read-only repo token and an instance-owner admin token are
not the same finding.
3. **What consumes it?** Grep for real consumers before assuming breakage.
4. **Where does it live?** Enumerate copies (store, palace, transcripts, git).
5. **Then** rotate/revoke, and only then consider cleanup.
Doing 4→3→1 in reverse produces confident, wrong severity calls and wasted
cleanup. If you only have time for one step, do step 1.
## 2. Leak-free identity: fingerprint, never the value
Publishing an 8-hex fingerprint lets you compare a credential across machines,
files, drawers and peers without ever materialising the secret. Same formula as
`mempalace_redact.py`:
```sh
printf '%s' "$SECRET" | sha256sum | cut -c1-8 # printf, NOT echo (no newline)
printf '%s' 'test' | sha256sum | cut -c1-8 # self-test -> 9f86d081
```
Report as `(variable, fp, length)`. Equal fingerprints across hosts prove a
shared credential; that is usually the important part. **Never** paste a live
value into a search query, a palace drawer, an event body, or a chat message —
in an agent context your own tool output is itself captured and re-filed.
**Precondition — only fingerprint what an adversary cannot enumerate.** An 8-hex
fingerprint is 32 bits over its *input space*, so publishing `fp8(x)` hands
anyone a **membership oracle**: they can test `x == v` for every candidate `v`
they can generate. For a 40-char random token that space is unreachable. For a
hostname, username, e-mail, port, path, commit SHA or weak password it is a
wordlist. **If you can imagine writing the wordlist, you cannot publish the
fingerprint** — reference those by name and location instead. "High entropy" is
the usual *sufficient condition*, not the test: a commit SHA is 160-bit and still
fully enumerable from the repo. `sha256("")` = `e3b0c442` is the degenerate case,
recognisable on sight precisely because its input space has one member.
**Candidate fingerprints are working memory, never output.** A scanner that hashes
every token in a file also hashes hostnames, paths and e-mails. Print only
fingerprints that *matched* a known entry — the tempting debug step when a scan
returns zero ("print what it saw") publishes low-entropy fingerprints wholesale.
And say plainly what a fingerprint register *is*, so nobody rediscovers it later
as an alarm: even for an unguessable secret, a published fingerprint is a
**confirmation oracle** for anyone who already holds a candidate corpus. That is
exactly how a long-retired token gets identified in old transcripts — and it works
identically for someone else holding those same files. Net positive, since they
would already hold the value; state it rather than leaving it implicit.
## 3. Liveness probes, and the trap that scoping creates
```sh
# Gitea
curl -sS -m 10 -o /dev/null -w '%{http_code}\n' -H "Authorization: token $T" \
"$GITEA_HOST/api/v1/repos/<owner>/<repo>/actions/runs?limit=1"
# GitHub
curl -sS -m 10 -o /dev/null -w '%{http_code}\n' -H "Authorization: token $T" \
https://api.github.com/user
```
- `200` live · `401` revoked/invalid · **`403` = wrong question, not a dead token**
- **Probe the issuer that minted it.** A 401 from an unrelated instance says
nothing. Resolve the host from config (`GITEA_EGL_HOST` etc.), do not assume.
- **Under scoped tokens, `/api/v1/user` returns 403 for a perfectly live token**
unless `user` scope was granted. So it cannot distinguish *revoked* from
*merely scoped*. Use a **repository route the token is authorised for**.
- Verify **both directions** after a rotation: old → 401, new → 200. The second
check is what catches "deleted the wrong token".
- Port/scheme come from config, not habit: one instance here is
`http://gitea.egl.lan:3000` — plain HTTP, with 443 refused.
## 4. Revocation beats deletion — the load-bearing rule
Once revoked, stored copies are **inert**; you may leave them. Deleting them is
best-effort over an *unbounded* copy set: FTS shadow rows, feed inbox `.jsonl`
files on every host, sqlite free pages after the delete, mesh replicas that
already synced, and backups. **Revocation invalidates every copy everywhere at
once, including copies nobody enumerated.**
So: **rotate + revoke first.** Treat drawer deletion as optional hygiene, never
as the remedy. Then record the retired fingerprints as *known-dead* so the next
census recognises them instead of reopening the investigation.
Corollary: never reach for `mempalace_sync` or a bulk `delete_by_source` on a
shared palace as incident response. High blast radius, low actual benefit.
## 5. Finding a secret in a Chroma palace — three targets, in this order
1. `embedding_fulltext_search_content.c0` — **where document text actually is**
2. `embedding_metadata.string_value` — metadata fields only
3. raw byte scan of every `*.sqlite3` — backstop, covers FTS pages and free space
Scanning only (2) is the classic false clean: hundreds of thousands of rows,
zero hits, and the secret sitting in (1) the whole time. Semantic search proves
nothing about absence — it returns top-k. For completeness, enumerate by filing
window (`list_drawers(since=T, before=T+1m)`), since one mine shares a minute.
Value-agnostic sweeps (uuid / 40-hex / `NAME=VALUE`) drown in false positives at
fleet scale — 608 candidates, mostly session UUIDs and git SHAs. Name-anchoring
plus entropy plus provenance, applied to **document text**, is what works.
## 6. Proving absence: instrument strength, and four ways a scan lies clean
Section 5's warning is about false *positives* — name-anchoring and provenance are
what stop a triage sweep drowning in session UUIDs. **A gate is the opposite job.**
Triage optimises precision; proving absence optimises recall. Every failure below
reported a reassuring zero over a secret that was really there.
**Rank the instrument, and state which one produced your zero.**
| Instrument | Needs | Blind to |
|---|---|---|
| exact-byte value search | you hold the value | nothing — no tokeniser to fool |
| class/structure pass | a header pattern | anything without a recognisable shape |
| fingerprint census | a fingerprint list | any secret not listed; tokenisation |
A census is deliberately value-free, so it must *extract candidates and hash them*
— which makes its sensitivity a property of the tokeniser, not of the corpus. If
you hold the value, search the bytes instead, and search the value's JSON-escaped
rendering too when the corpus is `.jsonl`.
**1. Census and class answer different questions; neither substitutes.** A census
answers *"has a KNOWN secret leaked?"*, a class pass *"is there secret-SHAPED
material here?"* Both failure modes were measured on this fleet: a class-only
pre-commit hook passed plaintext UUID API credentials to a shared repo twice,
because a UUID carries no key header — while a census-only gate reported 0 hits
with freshly-synced SSH private keys and an age identity in the tree, because no
key is in the census. Run both passes.
**2. Tokenisation — quoting alone can decide detectability.** Maximal-run
extraction swallows the value of an *unquoted* assignment:
```
PROXMOX_SECRET=<uuid> # ONE run; the uuid is never hashed alone -> MISS
export SECRET="<uuid>" # the quote ends the run; bare uuid hashed -> HIT
```
Take the **union** of three strategies, because each fails in a different
direction — (2) is the one that recovers the unquoted case:
~~~python
runs = re.findall(r'[^\s"\'`]{12,}', text) # 1. maximal runs
split = [p for r in runs for p in re.split(r'[=!,;:@|()\[\]{}<>]', r) if len(p) >= 12]
shape = re.findall(UUID_RE, text) + re.findall(r'[0-9a-f]{32,64}', text)
candidates = set(runs) | set(split) | set(shape)
~~~
**3. Scan the index or the pushed tree, never the working tree.** The working tree
is not what gets published. And for an rsync-published mirror a repo-only fix is
not weaker, it is *temporary*: the next sync re-publishes the live disk. Fix the
live file first, verify it clean **by fingerprint**, then sync. Read blobs with
`git ls-tree -r <sha>` plus one `git cat-file --batch` (thousands of `git show`
calls is the slow way).
**4. Git filters never run on symlinks — and `check-attr` will not tell you.** A
symlink's blob is the *target path*, so `filter=git-crypt` can never encrypt it,
yet `git check-attr filter` cheerfully answers `git-crypt` for that path. **A
symlinked secret stays plaintext no matter what `.gitattributes` says.** Join the
attribute against the **file mode** (`git ls-files -s`, mode `120000`) and verify
the index blob really begins `\0GITCRYPT\0`. Report encrypted / symlinked /
scanned as three separate numbers and assert they sum — encrypted and symlinked
blobs are *skipped*, not certified clean.
**Self-test two-sided, and abort if it cannot discriminate.** Require a synthetic
positive to fire AND a negative to stay silent before trusting any zero. Keep the
fixtures in *structurally separate buffers*: put a quoted and an unquoted probe in
one buffer and the quote terminates the run, handing the bare token to the weak
extractor and making it look as strong as the union — a self-test artifact that
has already fooled an agent here. And never gate on `$?` when the tool has a
lock-skip or no-op path that also exits 0; judge the reported line.
**Row-gone is not bytes-gone.** A correct sqlite `DELETE` leaves the payload in
freelist pages until `VACUUM`, so deletion effectiveness is *two* numbers: rows
removed, and a raw byte scan of the `.sqlite3`. One aggregate figure reported as
"erased" has only measured "unretrievable".
## 7. Choosing scopes: derive them from measured consumers
Before creating a replacement token, find out what actually uses it:
```sh
git -C <repo> remote get-url origin # ssh:// ? then git needs NO token
git config --global --list | grep -iE 'credential|insteadof' # and no helper?
grep -rhoE 'api/v1/[A-Za-z0-9/{}$_.-]+' <consumers> | sort -u # exact routes
grep -rhoE '\-X [A-Z]+' <consumers> # any writes?
```
Real outcome here: git used SSH keys throughout, and the token's only consumer
read three CI-run routes with `GET`. So `repository: Read` and nothing else
replaced two admin tokens. **Scoping shrinks the blast radius of the next leak
far more than any redaction pipeline does** — a read-only token in a transcript
is a hygiene event, not an instance compromise.
Then prove the scope with an acceptance suite that declares expectations first:
must-work routes → `200`; `/admin/*`, `/user`, `/user/repos` → `403`.
## 8. What rotation does *not* fix
- **A cleartext channel.** If the endpoint is `http://`, the *new* token is
exposed identically from first use. Raise TLS separately.
- **Git history.** A secret committed and pushed cannot be fixed by any store or
palace operation — it needs rotation *and* history surgery.
- **Agent-authored content.** Stage-write redactors see transcripts only, never
`add_drawer` / `checkpoint` / `diary_write` output. Never type a secret into
the palace yourself; nothing downstream will catch it.
- **Plaintext/encrypted drift.** Gitignored plaintext `.env` files go stale while
`.env.age` moves on, so old values linger on disk (and in backups) long after
rotation. They are a common source of "mystery" fingerprints in a census.
## 9. This fleet's secret store (verify, do not assume)
- All `*.env.age` live in **one** repo: `joakimp/docker-compose-repo`. `myconfigs`
has none.
- Every `.age` file has **one X25519 recipient** — a single key tracked in
`myconfigs` under git-crypt. Unlocking git-crypt therefore decrypts the entire
fleet's secrets, including hosts you have no access to. The age layer adds no
isolation beyond git-crypt.
- Flow: `./fetch-secrets.sh <host>` (decrypt → `.env`) → edit → `./encrypt-secrets.sh <host>`
→ commit → push → `docker compose up -d --force-recreate`.
- **Always pass the host argument** to `encrypt-secrets.sh`. Bare, it walks the
whole tree and re-encrypts every `.env` it finds, re-nonced, including stale
ones — silently rolling back other hosts' secrets.
- After any re-encrypt, check the header still shows exactly **one X25519
recipient**; a hand-rolled `age -r` locks the rest of the fleet out, and the
failure only appears on another machine, later.
@@ -143,6 +143,10 @@ mine:
| "`tor-ms22` is not in the SSH config" | `grep … \| head -20` — the entry was at **line 454**. `~/.ssh/config` here is ~500 lines. |
| "the Docker host has no `docker`" | non-interactive SSH `PATH` lacks `/usr/local/bin` (§2, §3). It was at `/usr/local/bin/docker`. |
| "no ControlMaster is running" | pattern `ssh ` (trailing space) cannot match a master: those processes **rename themselves** to `ssh: <controlpath> [mux]`. |
| "the credential is not in the palace" | scanned `embedding_metadata.string_value` only. Drawer **text** lives in `embedding_fulltext_search_content.c0`; 554k metadata rows proved nothing. |
| "this token is dead — 401" | probed it against the **wrong issuer**. A 401 from an instance that never issued the credential is not evidence about the credential. |
| "that host is unreachable, can't test" | tried ports 443 and 80. It was on **3000**, and the env var I already held (`GITEA_EGL_HOST`) stated the scheme and port. |
| "this repo has no `## Unreleased` convention" | read `CHANGELOG.md` **once**, minutes after a release commit had renamed that section to a version heading. 33 commits touch `## Unreleased`. A snapshot cannot show you a cycle. |
Habits that would have caught all three:
@@ -155,10 +159,55 @@ ssh -F "$HOME/.ssh-local/config" mac 'command -v docker || ls /usr/local/bin/doc
# match a process's ACTUAL argv, not the name you imagine
ps -eo pid,etime,args | grep -Ei 'mux|mosh|ssh'
# to learn a repeating PROCESS or convention, read history, not the file. A
# file's current content is one frame of a cycle, and the frame you happen to
# catch may be the one where the thing you are looking for was just consumed.
git log -S'## Unreleased' -- CHANGELOG.md # not `head -60 CHANGELOG.md`
```
A positive result needs no such scepticism — it carries its own evidence. Only
absence has to be *earned*, so spend the extra command there.
Absence has to be *earned*, so spend the extra command there.
### …and a positive result only proves what you *actually asked*
An earlier version of this section claimed "a positive result needs no such
scepticism — it carries its own evidence." **That is false, and believing it
cost a later session three more wrong findings.** A positive result is evidence
about the question your command really posed, which may not be the question you
meant. The failure is invisible precisely *because* the command succeeded.
| Claim | The command succeeded — at answering something else |
|---|---|
| "EGL git over SSH works" | `ssh git@gitea.egl.lan` greeted me as `joakimp`. `~/.ssh/config` had `Host gitea*` → `HostName gitea.jordbo.se`, so I authenticated **to the wrong instance**. The real EGL account is `ecsjper`. |
| "the port config regressed" | compared `ssh -G` output against `2222` — a value produced by **my own earlier `-p 2222` flag**, not by the config. I reported the user's edit as a regression it never caused. |
| "the CI runners authenticate with this token" | pure fabrication, contradicted by my own scan output already on screen. The runners use per-runner `REGISTRATION_TOKEN`. |
Two habits that actually catch this class, both cheap:
```sh
# 1. ask which RULE captured your hostname before trusting any ssh result.
# ssh_config is first-obtained-value-wins PER KEYWORD, not per block: a
# specific block only wins the keywords it declares, so a later `Host gitea*`
# still supplies HostName unless the specific block restates it.
ssh -G git@thehost | grep -E '^(hostname|port|user|identityfile)'
# 2. state the expected result BEFORE running the check, and diff against it.
# This is the single technique that separated the one verification that went
# right (10/10, expectations declared per probe) from five that went wrong
# (results interpreted after the fact, each time in the direction I expected).
probe "/repos/.../actions/runs" 200 # must work
probe "/admin/users" 403 # must be denied
```
And the meta-observation, which is the reason this subsection exists: across all
five errors, **not one was caught by re-reading my own reasoning.** Every one was
caught by a second measurement that disagreed — the SSH lie surfaced only because
the greeting said `joakimp` while a token probe minutes earlier had said
`ecsjper`; the fabrication surfaced only because the user read my own output back
to me. So the operational rule is not "be careful". It is: **for a load-bearing
claim, produce a second measurement by a different route, and expect it to
disagree.** If you cannot think of a second route, you do not yet have a finding
— you have a hypothesis.
**`dscp`/`scp` with accented filenames on a macOS host.** macOS stores filenames
in Unicode **NFD** (decomposed — e.g. `ä` is `a` + combining U+0308), while the
+5 -3
View File
@@ -245,6 +245,8 @@ run "socat" "socat -V"
run "studio-expose helper" "test -x /usr/local/bin/studio-expose"
run "image-baked pi-devbox-environment skill" \
"test -f /usr/local/share/pi-devbox/skills/pi-devbox-environment/SKILL.md"
run "image-baked credential-incident-response skill" \
"test -f /usr/local/share/pi-devbox/skills/credential-incident-response/SKILL.md"
run "global-AGENTS append snippet present" \
"test -f /usr/local/share/pi-devbox/pi-global-AGENTS.append.md"
run "pi-devbox block merged into pi-global-AGENTS.md" \
@@ -596,9 +598,9 @@ exec_test "mempalace skill linked (fallback)" 'test -L $HOME/.agents/skills
# bumped correctly but whose bytes came from the wrong place.
exec_test "mempalace skill snapshot is current" 'f=$HOME/.agents/skills/mempalace/SKILL.md; grep -q "Provenance is stamped for you" "$f" && ! grep -q "Attribute what you file yourself" "$f" && echo ok'
# Link TARGETS, not just link existence: with no skillset mounted (as here) the
# baked tree must be what resolves, for all three vendored skills.
# baked tree must be what resolves, for all four vendored skills.
exec_test "vendored skills resolve to the baked tree (no skillset mounted)" \
'for s in mempalace pi-extensions pi-devbox-environment; do
'for s in mempalace pi-extensions pi-devbox-environment credential-incident-response; do
case "$(readlink -f $HOME/.agents/skills/$s)" in
/usr/local/share/pi-devbox/skills/$s) ;;
*) echo "$s resolves to $(readlink -f $HOME/.agents/skills/$s)" >&2; exit 1 ;;
@@ -612,7 +614,7 @@ exec_test "vendored skills resolve to the baked tree (no skillset mounted)" \
exec_test "pi-devbox-version reports skill sources (all baked, no skillset here)" \
'out=$(pi-devbox-version)
echo "$out" | grep -q "skills:" || { echo "no skills section" >&2; exit 1; }
for s in mempalace pi-extensions pi-devbox-environment; do
for s in mempalace pi-extensions pi-devbox-environment credential-incident-response; do
echo "$out" | grep -qE "^ $s +baked$" \
|| { echo "$s not reported as baked" >&2; exit 1; }
done; echo ok'