Compare commits

...

6 Commits

Author SHA1 Message Date
joakimp 6891dc32b8 changelog: release v1.8.10 — deploy the scrubber, and name the check that must not be skipped
Publish Docker Image / build-variant-studio (push) Successful in 17m4s
Lint / actionlint (push) Successful in 17s
Publish Docker Image / resolve-versions (push) Successful in 14s
Publish Docker Image / base-decide (push) Successful in 12s
Publish Docker Image / build-variant (push) Successful in 18m45s
Publish Docker Image / build-base (push) Successful in 41m59s
Publish Docker Image / update-description (push) Successful in 6s
Publish Docker Image / promote-base-latest (push) Successful in 11s
Publish Docker Image / smoke (push) Successful in 4m55s
Lint / hadolint (push) Successful in 8s
Publish Docker Image / smoke-studio (push) Successful in 5m9s
Audited every component against upstream rather than assuming. Only two moved
since v1.8.9:

- mempalace-toolkit 5b8d78f -> b2b50af (ours): feeder scrubber, the symlink fix
  that stops it refusing to stage, hlc owed-set join, queued-delivery note,
  explicit MEMPALACE_MAILBOX_NOTIFY protocol modes, RFC 003 + fleet-memory docs.
- pi-studio v0.9.48 -> v0.9.52 (upstream, studio variant only): 22 commits, all
  additive — PDF previews, hideable header, contextual side questions. No removals
  or renames. Its pi floor is >=0.84.3 against our exact 0.84.3 pin: satisfied,
  zero headroom, named as a watch item because the next floor bump breaks the
  studio job only, after core has already published.

Verified unchanged, with dates proving they predate the v1.8.9 bake: pi 0.84.3 (=
npm latest), mempalace 3.8.0 (= PyPI latest), pi-atelier v0.8.2, playwright 1.62.1
(no drift this cycle despite floating on latest), pi-fork bf702b4, pi-obsmem
ce9fc98, pi-toolkit 0e1369e, pi-extensions 2022887. No breaking changes anywhere,
so everything can ship in one tag.

The reason for tagging now is not features. The scrubber has been live on exactly
ONE device since this morning; every other device has kept staging unscrubbed
transcripts into a shared palace, and cleanup after the fact is manual redaction
(done twice today, with freed-page residue left behind by choice).

The entry leads with the acceptance check instead of burying it, because this
release's worst failure is silent: the feeder is fail-closed, so a packaging or
path mistake stops the fleet's memory feed and nothing complains — refusing to
stage looks exactly like a quiet session. f0bffd1 exists because that very bug was
real (BASH_SOURCE reports the symlink path, not the target). First client on the
new image must confirm a [scrub] summary line appears, the drawer count moves, and
exit 3 did not fire. Silence is the failure signal, not success.
2026-08-27 17:51:46 +02:00
Joakim Persson 8a673ec143 docs: unclip the diagrams, and answer what compaction leaves behind
Lint / hadolint (push) Successful in 8s
Lint / actionlint (push) Successful in 15s
Two problems reported against docs/observational-memory.md, one cosmetic and one
substantive. Both turned out to be worth more than the fix.

Clipping. Several boxes lost their bottom line of text in the viewer, and neither
the source nor my own render showed it. Cause: Mermaid measures a node label with
its own font metrics, commits to a box size, then renders that label as real HTML
in a <foreignObject> — so any host stylesheet touching line-height or font-size
inflates the text past a box that is already fixed, and the overflow is clipped.
The error accumulates per line, which is why it always eats the last line of the
tallest labels.

Rejected the obvious fix after testing it rather than assuming it:
%%{init: {'flowchart': {'htmlLabels': false}}}%% *is* honoured (labels switch from
16 foreignObject to 7 tspan) and clips identically, because inflated font-size
inherits into SVG text too. The fix that works is a hard limit of two short lines
per node, with the detail moved into prose under each diagram — one- and two-line
boxes have the vertical slack to absorb inflation, three- and four-line boxes do
not. The nodes were carrying paragraph-sized text; the diagrams are better for
losing it.

Regression harness: render every block with line-height: 1.7 !important forced
onto the label HTML, screenshot, read it. That caught two survivors of the rewrite
that looked fine in the clean render — a long unbreakable /opt path wrapping to a
third line, and a cylinder shape whose curved bottom leaves less room than a
rectangle for the same two lines.

New §4, because the document explained that compaction folds the ledger and never
said what that leaves in the context. Asked directly: is the session back to
knowing nothing? No. A verbatim tail survives, sized by keepRecentTokens (20k) and
cut only at turn boundaries; the system prompt and AGENTS.md were never in the
compacted region because they are rebuilt from disk each request; nothing is
deleted from disk, since compaction appends a compaction entry rather than
rewriting lines; and recall keeps resolving ids whose sources left the context
because it reads the full branch via sessionManager.getBranch() and never consults
the context window. Repeated compaction renders from live records, not from the
previous summary's prose, so there is no generation-loss spiral.

Also corrects this repo's own "compaction calls no model" to the steady-state
claim it actually is: with an empty ledger the hook returns nothing and explicitly
declines ownership, and pi's native model-based summariser runs. Snippet quoted in
the doc.

And a bug shipped in the first version: §9 said the ledger entries are
custom_message and specifically not custom. Exactly backwards, so the one grep
that section existed to get right was the one it got wrong. Verified empirically
against the live session file — 11 om.observations.recorded and 6
om.reflections.recorded, all "type":"custom", next to "type":"custom_message"
entries whose customType is mempalace-mailbox and mempalace-wakeup. That is where
the confusion came from, and the distinction is load-bearing rather than
cosmetic: the mailbox uses the context-visible append API, om's ledger uses the
invisible one. Which makes it a feature the doc now advertises — the ledger costs
zero context until it is folded.
2026-08-27 14:24:19 +02:00
joakimp cdb6fc0950 changelog: name what the floating toolkit ref will pull into the next tag
Lint / hadolint (push) Successful in 8s
Lint / actionlint (push) Successful in 16s
MEMPALACE_TOOLKIT_REF=main floats and docker-publish.yml resolves it to a SHA at
build time, so toolkit main moving 5b8d78f -> f0bffd1 (10 commits) ships in the
next tagged image whether or not this repo has a commit. v1.8.9 adopted the rule
after that shape bit twice (553d865, 5b8d78f): name the behaviour change BEFORE
tagging. This is that rule obeyed rather than re-learned — the work was pushed to
toolkit main earlier today and this entry was missing, which is exactly the gap
that caused a cross-host misattribution in v1.8.7.

Contents: the feeder-side secret scrubber (three tiers, T3 report-only after a
measured 403 -> 29 false-positive calibration on 52 MB of real transcripts, fail
closed); the symlink near-miss that fail-closed would have turned into a
fleet-wide silent memory outage at bake time, caught before tagging; the mailbox
work (hlc owed-set join, queued-until-next-turn note, explicit notify protocol
modes, tmux path documented unverified); and the documentation set (RFC 003,
fleet-memory.md, secret-hygiene.md). Also states what remains unscrubbed: the
opencode bridge write path and the unbuilt server-side layer.

No tag pushed — per the release protocol, no tag means no build.
2026-08-27 14:20:35 +02:00
Joakim Persson 14371e2da6 docs: explain the memory that runs itself, and fix a claim the mailbox falsified
Lint / hadolint (push) Successful in 12s
Lint / actionlint (push) Successful in 16s
`pi-observational-memory` is baked, registered in the seeded settings.json, and
handed a cheaper model than the session it serves — and the only description in
this repo was five words in a feature list. Someone meeting `/om:status` or a
"compacted memory" block had nothing to read that said whether to leave any of it
on. New docs/observational-memory.md, 272 lines and five diagrams.

Scoped by who is authoritative, so there is one copy of each claim:

- Upstream (/opt/pi-observational-memory/docs/) already documents the mechanism
  well — concepts.md, how-it-works.md, configuration.md, including a v3 lifecycle
  diagram that matches the deployed code. Linked, not re-derived.
- This document takes the four facts pi-devbox owns and can change: the pinned
  commit it bakes (v3.0.4 ce9fc98, matching build-manifest.json), the packages[]
  entry that decides which copy loads, the Haiku-workers-vs-Opus-session split,
  and the devbox-pi-config volume that makes the ledger outlive the container.
- Plus the confusion this image creates by shipping two things called memory: a
  section contrasting it with MemPalace, on the line "observational memory keeps
  a session coherent, the palace keeps the fleet coherent".

Placement follows the split fleet-ops states for itself — reusable mechanism is
not deployment data — so a "why is this in my container" document belongs in the
repo that pins and wires the component. Linked twice from the README, because
until this commit the README referenced docs/ zero times and the file already
sitting there was reachable only by listing the directory.

Numbers were read out of the live container and the baked tree rather than out of
release notes, which caught one thing the pi-extensions skill still has wrong:
the dropper is gated on a successful same-turn reflection, not on a token
threshold of its own.

Also fixes a README sentence that v1.8.9 made false. § Cross-machine agent
coordination ended with "Nothing in this image polls the log on the agent's
behalf"; the mailbox has shipped since aac4a1c. Replaced with the three knobs and
their defaults, derived-not-read owed-ness, and the queued-into-the-next-turn
delivery measured on two devices — and dated to "as baked in v1.8.9
(mempalace-toolkit 5b8d78f)", pointing at RFC 003 §7.11–§7.12 for the mechanism,
because toolkit main is already ahead (a92c75d pings the human who is not
looking) and describing that here would trade a stale-behind claim for a
stale-ahead one.

The cause is worth more than the fix: the behaviour arrived through the floating
MEMPALACE_TOOLKIT_REF, so no diff in this repo ever touched the paragraph making
the claim. v1.8.9's own rule fired for the CHANGELOG and nobody swept the README.
The CHANGELOG records what changed, the README asserts what is true, and only the
first is reviewed at release time — so the rule now extends to grepping the README
for absolute claims (nothing, never, does not, only) about a component whose SHA
moved.

Diagrams were verified by rendering, not by parsing. Both comparison diagrams
parsed clean and rendered with their meaning reversed: Mermaid laid the second
declared subgraph out first, putting "with observational memory" before "without"
and MemPalace before observational memory in the diagram whose entire job was
that contrast. Rebuilt as declaration-ordered chains and re-rendered at mermaid@11
— the version pi-studio pins — in the baked headless browser, then read back as an
image. mermaid.parse() proves syntax and says nothing about layout.
2026-08-27 13:55:55 +02:00
joakimp aac4a1c323 release: v1.8.9 — the version flag that blamed the wrong component
Lint / hadolint (push) Successful in 15s
Lint / actionlint (push) Successful in 18s
Publish Docker Image / resolve-versions (push) Successful in 9s
Publish Docker Image / base-decide (push) Successful in 9s
Publish Docker Image / build-base (push) Successful in 41m49s
Publish Docker Image / smoke (push) Successful in 4m51s
Publish Docker Image / smoke-studio (push) Successful in 4m59s
Publish Docker Image / build-variant-studio (push) Successful in 16m58s
Publish Docker Image / build-variant (push) Successful in 28m35s
Publish Docker Image / update-description (push) Successful in 7s
Publish Docker Image / promote-base-latest (push) Successful in 17s
Two versions, two flags. `--expected-version` has only ever asserted
`pi --version`, but AGENTS.md step 4 spelled it `X.Y.Z` inside a checklist where
every other X.Y.Z is the pi-devbox tag. Run as documented for v1.8.8 the final
runtime gate of the release printed

    ✗ pi version mismatch: expected 1.8.8, got 0.84.3

and exited 1 — a red accusing the image of being the wrong version. Not one
reader's slip: the v1.8.8 release-readiness handoff from pi@emb-7kj4vr4g
propagated the same wrong spelling twice while correctly calling step 4 "not
ceremonial", so two independent readers converged on it. README.md had it right
all along, which means the two documents disagreed.

- new --expected-image-version asserts the pi-devbox release tag, read from
  release_tag in /etc/pi-devbox/build-manifest.json (no checkout, no network);
  leading `v` optional on either side
- both flags detect being handed the other one's value, and the test is exact
  rather than heuristic: the value is compared against the other quantity the
  image itself reports, so it can only fire on a real mix-up
- neither flag is required now. With none, live `pi --version` is asserted
  against the manifest's pi_version — not a tautology, since a stale pi in the
  ~/.pi/npm-global volume can shadow the baked one, exactly as a stale
  npm:pi-atelier can in packages[]
- the header note replaced was stale and load-bearing: it claimed pi is resolved
  from 'latest' and cannot be self-derived, while Dockerfile.variant pins
  ARG PI_VERSION=0.84.3 and docker-publish.yml reads that ARG as its source of
  truth. The same withdrawn claim also sat in cli_utils' pi-devbox-sanity --help
- argument parsing: a missing value, or a value that is another flag, is a usage
  error instead of silently consuming the next argument; --help works

All fourteen flag combinations exercised by execution, including the two
manifest-absent branches and the shadowing branch a healthy container cannot
reach — mutation-tested with a doctored manifest so each failure branch was
observed firing rather than assumed present.

CHANGELOG also names what no commit here causes: mempalace-toolkit main moved
e70bef2 -> 5b8d78f, so this tag ships the auto-delivered logstream mailbox
because base_tag folds the resolved toolkit SHA. It would have landed either
way; going unnamed is the 553d865 shape that already caused one cross-host
misattribution. Component audit found nothing else to bump — pi, mempalace,
pi-atelier all equal their upstream latest, and every other floating ref
resolves to the commit already baked.
2026-08-26 18:47:03 +02:00
joakimp 34cf1e3810 release: v1.8.8, and a notice that named the wrong remedy
Lint / hadolint (push) Successful in 14s
Lint / actionlint (push) Successful in 17s
Publish Docker Image / resolve-versions (push) Successful in 10s
Publish Docker Image / base-decide (push) Successful in 19s
Publish Docker Image / build-base (push) Successful in 1h3m31s
Publish Docker Image / smoke (push) Successful in 4m47s
Publish Docker Image / smoke-studio (push) Successful in 8m8s
Publish Docker Image / build-variant-studio (push) Successful in 20m4s
Publish Docker Image / build-variant (push) Successful in 26m9s
Publish Docker Image / promote-base-latest (push) Successful in 9s
Publish Docker Image / update-description (push) Successful in 14s
Freezes the v1.8.8 section and clears the two non-code checklist items
pi@emb-7kj4vr4g handed over (evt_20260826T134919_a614ecfc2d4f), plus the two
carried nits from its round-2 verification (evt_20260826T133356_d56792791a49).
Every claim below was re-measured here rather than taken from the handoff.

THE STALENESS NOTICE ASSERTED A DIRECTION IT NEVER TESTED — Blocker 1's shape,
one layer down, in the message I added to replace the message that named the
wrong cause. The notice fired on "recorded != HEAD" and then announced HEAD as
the newer side without testing ancestry, so a clone that was merely BEHIND got
"has moved to 82a8d3c; the snapshot describes the older 5fd0d5c" when 82a8d3c is
5fd0d5c's ANCESTOR. Found by EMB against the real state of its own host, not a
fabrication. The verdict was never wrong (rc 0, nothing mis-verified) but the
remedy it implies is a ~67-minute base rebuild when the actual fix is `git pull`
— the only one of the two carried nits with a price tag, which is why it went
first. Now tests ancestry with the merge-base --is-ancestor primitive the refresh
path 60 lines below already used, and reports three verdicts: stale (refresh),
clone behind (pull, do NOT refresh), diverged (reconcile). All three verified by
execution; only the first was correct before. --help no longer errors on the one
script whose argument order was itself a landmine. The `-s "$VENDORED"` guard is
now commented as load-bearing: it makes the empty-stdin collision unreachable by
construction, which also means no test below exercises it any more, so deleting
it as "redundant with the probes" would silently restore the false OK.

CHANGELOG: retitled, and three stale spots fixed in what becomes the permanent
record. It cited the skill at 82a8d3c (twice superseded); it RE-ASSERTED the
retracted mailbox measurement as live evidence 200 lines after withdrawing it,
which is the exact non-contradiction failure this release exists to fix; and its
warning block still described the pre-e8ddeaf world ("still records c04cd15",
"now exits 1") while quoting as exemplary the very notice whose direction was
unverified. Per EMB's steer the conclusion was kept and only the evidence
replaced: the status filter does drop broadcast noise, it just never computed
owed-ness. Honest replacement, measured on both machines: raw filter returns 2
here and 1 there, EVERY ONE already answered, derivation returns 0 for both.

SNAPSHOT RESYNCED AGAIN, 5fd0d5c -> 6eb20af, because skillset 6eb20af adds the
limit of my own seq ordering test: seq is REPLICA-LOCAL, equal to origin_seq only
because one replica authors for all four machines, so use hlc once mesh_peers
reports a peer. Recorded as reasoning not measurement — a second replica cannot
be stood up here. The durable half is the asymmetry: seq skew makes an ANSWERED
item resurface (noise, visible, self-correcting) while created_at SUPPRESSES AN
UNANSWERED ask forever (silent, permanent), so the skill now says outright that
"fixing" a resurfacing item with a timestamp trades the safe failure for the
dangerous one. The resync was free: rootfs/ was already changing, so the base
rebuild was forced regardless — the ordering warning about accidental staleness
does not apply to a deliberate refresh before the tag.

--check is a clean OK at 6eb20af with no notice, canary re-verified bidirectionally
(present 3, withdrawn 0), baked snapshot 0644, tree hash recomputed at build time
and re-verified in-container. bash -n clean; shellcheck/hadolint/actionlint remain
absent locally, so CI is still the only evidence for those.
2026-08-26 15:56:03 +02:00
8 changed files with 1154 additions and 45 deletions
+17 -3
View File
@@ -98,9 +98,23 @@ re-brand of opencode-devbox's `pi-only` variant.
**post-recreate sanity check** inside the running container to confirm **post-recreate sanity check** inside the running container to confirm
persisted volumes survived and the pi runtime wiring re-deployed (not just persisted volumes survived and the pi runtime wiring re-deployed (not just
that the container booted): that the container booted):
`docker compose exec devbox bash scripts/recreate-sanity-check.sh --expected-version X.Y.Z` `docker compose exec devbox bash scripts/recreate-sanity-check.sh --expected-image-version X.Y.Z`
(or just `pi-devbox-sanity --expected-version X.Y.Z` if `cli_utils/bin` is (or just `pi-devbox-sanity --expected-image-version X.Y.Z` if
on PATH). This is the runtime peer of the build-time `smoke-test.sh` gate. `cli_utils/bin` is on PATH). This is the runtime peer of the build-time
`smoke-test.sh` gate.
**`X.Y.Z` here is the pi-devbox release tag** you are shipping (e.g.
`1.8.9`), which is what the rest of this checklist means by `vX.Y.Z`.
`--expected-image-version` is the flag that asserts it. There is also an
`--expected-version`, and it means something else — the **pi coding agent**
version (e.g. `0.84.3`, the `ARG PI_VERSION` pin). Handing the release tag
to that one used to report *"pi version mismatch: expected 1.8.8, got
0.84.3"*, i.e. a red on the final gate of the release accusing the wrong
component; it now tells you to use `--expected-image-version` instead, and
the reverse mix-up is caught too. Both flags are optional — with neither,
the live pi version is asserted against the version recorded in the image's
own build manifest (which catches a stale `pi` in the `~/.pi/npm-global`
volume shadowing the baked one) and the image tag is reported
informationally.
5. Push tag: `git tag vX.Y.Z && git push origin vX.Y.Z`. 5. Push tag: `git tag vX.Y.Z && git push origin vX.Y.Z`.
6. Watch CI: smoke job builds amd64 only and asserts size + extensions + 6. Watch CI: smoke job builds amd64 only and asserts size + extensions +
pi version + new-base-tooling presence. Variant build is multi-arch pi version + new-base-tooling presence. Variant build is multi-arch
+535 -19
View File
@@ -11,7 +11,486 @@ Pre-v1.0.0 tags followed the pi npm version (`v{pi_version}[letter]`).
--- ---
## Unreleased ## v1.8.10 — 2026-08-27
**This tag exists to deploy a fix and a safety net that are currently running on
exactly one machine.** The feeder scrubber has been hand-copied to `/opt` on one
device since this morning; every other device has kept staging unscrubbed
transcripts into the shared palace. Nothing here is a new capability for its own
sake.
`MEMPALACE_TOOLKIT_REF=main` floats: `docker-publish.yml` resolves it to a
concrete SHA at build time, so whatever is on toolkit `main` when the tag is
pushed ships in that image whether or not this repo has a commit. That is the
rule v1.8.9 adopted after `553d865`/`5b8d78f` shipped undocumented twice — *name
the behaviour change before tagging, not after* — and this entry is that rule
being obeyed rather than re-learned.
**`mempalace-toolkit` main moves `5b8d78f` → `b2b50af`** (13 commits, ~2100
insertions / ~520 deletions). No pi-devbox commit implements any of it.
### ⚠️ THE ONE CHECK THIS RELEASE MUST NOT SKIP
**On the first client that runs this image, verify the memory feed still stages.**
The feeder is *fail-closed* by design: no redactor module, no staging (`exit 3`).
That is correct behaviour and it is also the failure mode with no alarm — a
packaging or path mistake stops the fleet's entire transcript feed and nothing
complains loudly, because refusing to stage looks exactly like a quiet session.
This is not hypothetical. `f0bffd1` exists because the feeder is installed as a
symlink (`/usr/local/bin/mempalace-pi-session` → `/opt/mempalace-toolkit/bin/…`)
and `${BASH_SOURCE[0]}` reports the *symlink* path, so the module lookup landed in
a directory where it does not exist. Had that shipped, every device would have
refused to stage on first boot. It was caught by execution, not by review.
Acceptance, in order, on the first recreated client:
1. Run a session, then confirm the feeder logged a scrub summary — a
`[scrub]` line with tier-tagged counts (`T1:env-value=…`, `T2:github-pat=…`),
or an explicit "zero redactions". **Silence is the failure signal**, not success.
2. Confirm the palace drawer count *moved* for that session (the feed reached the
server, not just the stager).
3. Confirm `exit 3` did **not** fire: `mempalace-pi-session` invoked through the
`/usr/local/bin` symlink must find `mempalace_redact.py`.
4. Only then trust the rest of this release.
If step 1 or 3 fails, the memory feed is down fleet-wide until it is fixed, and
sessions that ran in the meantime are not recoverable from the palace — they were
never staged. `MEMPALACE_FEED_ALLOW_UNSCRUBBED=1` is the loud escape hatch, and
using it means accepting unscrubbed transcripts until the packaging is repaired.
### Dependency audit (2026-08-27)
Every component checked against upstream, not assumed:
| Component | Baked in v1.8.9 | Upstream now | Action |
|---|---|---|---|
| **mempalace-toolkit** | `5b8d78f` | **`b2b50af`** | ships the scrubber + symlink fix + `hlc` join |
| **pi-studio** (studio variant) | `v0.9.48` | **`v0.9.52`** | 22 commits, additive only — see below |
| pi | `0.84.3` (pinned) | `0.84.3` is npm latest | none |
| mempalace | `3.8.0` (pinned) | `3.8.0` is PyPI latest | none |
| pi-atelier | `v0.8.2` (pinned) | `v0.8.2` highest tag | none |
| playwright | `1.62.1` (floats `latest`) | `1.62.1` | none — no drift this cycle |
| pi-fork | `bf702b4` | `bf702b4` (2026-08-24) | none |
| pi-observational-memory | `ce9fc98` (v3.0.4) | `ce9fc98` | none |
| pi-toolkit | `0e1369e` | `0e1369e` (2026-08-07) | none |
| pi-extensions | `2022887` | `2022887` (2026-08-17) | none |
**`pi-studio` `v0.9.48` → `v0.9.52`** — four releases, 22 commits, all additive:
PDFs open directly in Studio with watched previews, the header can hide, and
contextual *side questions* arrive (selected-tool use, frozen git context, export,
keyboard shortcuts). No removals or renames in the diff; the changes are
concentrated in `client/studio-client.js`, `index.ts` and three new `shared/`
helpers.
**Its `pi` floor is `>=0.84.3` and we pin exactly `0.84.3` — satisfied with zero
headroom.** Worth naming as a watch item rather than a problem: the next studio
release that raises the floor breaks the studio variant until `PI_VERSION` moves,
and that failure surfaces at build time in the studio job only, after the core
variant has already published.
### Also pulled in by the floating toolkit ref (documentation only)
RFC 003 gains **§9.2**, a proposed direction for the one open decision this
fleet keeps tripping over — that a report addressed to a device is never
delivered, because mailbox candidacy requires exactly `status="open"`. It records
a negative result worth keeping: widening the owed set to include terminal events
cannot work, since the asserting shape and the clearing shape must be disjoint or
every closure mints a fresh obligation. No code implements §9.2 in this release.
The skillset snapshot also moves, so this image bakes the mermaid-diagrams skill's
Playwright driver and the honest note that a `claimed` ack notifies nobody.
### Transcripts get scrubbed before they are staged (`3d47937`, `836e35b`, `f0bffd1`)
`bin/mempalace_redact.py`, called from `mempalace-pi-session` at the moment the
staged transcript is written — one hook covering both transports, because local
mode mines that file and remote mode rsyncs the same bytes.
- **Why it exists, measured rather than argued.** One leaked bearer token had
reached 3 drawers, 13 feeder inbox files across all three devices, and 10 local
files spanning 10 days, from an agent printing an env var while debugging. A
second sweep then found `GITEA_ACCESS_TOKEN` in 2 more drawers and
`GITEA_EGL_ACCESS_TOKEN` in 3. This is routine agent behaviour, so the fix
belongs in the pipeline, not in discipline.
- **Detection is name-anchored, never entropy-anchored.** A palace's own primary
keys — drawer ids, chunk ids, event ids, replica ids, HLCs, commit SHAs — *are*
its high-entropy strings, so an entropy detector eats the memory it protects,
silently and unrecoverably. Three tiers instead: T1 literal values from this
process's env whose name says secret (zero false positives by construction);
T2 vendor shapes (`ghp_`, `glpat-`, `xox*-`, `sk-`, `AKIA`, JWT, PEM, URL
credentials, `Authorization:`); T3 key-name-says-secret.
- **T3 is report-only, because the false-positive rate was measured.** On 52 MB
of real fleet transcripts T3 fired 403 times, mostly `${VAR}` interpolation in
compose files, TypeScript identifiers, a *type annotation*
(`credentials: Credentials`), an IPA attribute holding a date
(`krbPasswordExpiration`), AAAK diary shorthand, and terminal output following
an ssh `Password:` prompt. With interpolation/code-context/key-suffix guards the
enforced count fell **403 → 29** on the same corpus. `MEMPALACE_REDACT_STRICT=1`
makes T3 enforce.
- **Operational shape.** Fail closed — no redactor, no staging (`exit 3`),
overridable with `MEMPALACE_FEED_ALLOW_UNSCRUBBED=1`. Every run prints a count
*including* `0 redaction(s)`, because silence is indistinguishable from a
scrubber that never ran. Findings carry rule, label, length and `sha256[:8]` —
never the value.
**Near-miss this image would have shipped, caught before tagging (`f0bffd1`).**
The image installs `/usr/local/bin/mempalace-pi-session` as a **symlink** into
`/opt/mempalace-toolkit/bin`, and `${BASH_SOURCE[0]}` reports the invoked path,
not the target — so the sibling-module lookup resolved to `/usr/local/bin`, the
redactor was absent, and fail-closed did as instructed: `[FATAL] ... refusing to
stage`. Measured side by side, the symlinked invocation FATALed while the direct
one scrubbed 40 findings. **At the next bake that would have stopped every
feeder tick on every device — a silent fleet-wide memory outage, worse than the
leak the scrubber prevents.** Fixed by chasing the symlink chain in portable
shell (`readlink -f` avoided: GNU/newer-BSD only, and this script also runs
directly on macOS hosts) with colon-separated fallback candidates. Verified via
the symlink, the direct path, and a second-hop symlink. General lesson: fail-closed
converts "module not found" into an outage, which makes the module lookup
load-bearing infrastructure that must be tested through the invocation path the
fleet actually uses — not the convenient one from a checkout.
### The mailbox becomes explainable and mesh-safe (`bfe9c5c`, `a92c75d`, `e917662`, `ecc2a9c`)
- **Owed-set derivation joins on `hlc`, not `seq`** (`bfe9c5c`). `seq` is a
replica-local arrival counter — the same event is `#7` in one database and `#12`
in another — so a second replica would let already-answered asks resurrect.
`hlc` is immutable and replicated, fixed-width, so string comparison *is* causal
comparison. A safe no-op on today's single replica (verified: the positive-control
pair orders identically under both keys), correct once a mesh exists.
- **Delivered text now says it is queued** (`a92c75d`). Delivery uses `steer`
with no `triggerTurn`, and the poll fires on `agent_settled`, so nothing wakes
the model — a delivered ask sits until a human starts the next turn. Measured
case: a directed report sat unread for 2.5 hours. The note explains the agent is
not ignoring the ask, it is not running.
- **`MEMPALACE_MAILBOX_NOTIFY` gains explicit `=kitty` / `=osc777` modes**
(`e917662`). Terminal autodetection inside a container is not unreliable, it is
*blind*: `docker exec` forwards neither `KITTY_WINDOW_ID` nor `TERM_PROGRAM`, and
`TMUX` is unset because tmux runs on the host. Verified on a live process:
`TERM=xterm-256color` and nothing else.
- **The terminal path through tmux is documented as UNVERIFIED** (`ecc2a9c`).
Test sequences written to the pty produced no notification on a remote client;
tmux likely drops unknown OSC types without `allow-passthrough`, and multi-client
routing (one ask pinging every attached client) is an open question.
### Documentation (`e1cc759`, `982b001`, `d4d8bb6`, `d2764bf`)
- **RFC 003, the coordination-log spec the code had been citing all along** — it
did not exist anywhere. 11 sections, retrospective against mempalace 3.8.0
`logstream.py`, incl. owed-set derivation, ten dogfooded landmines and seven open
decisions. Non-obvious findings: `event_append` has **no** idempotency guard on
the write path (verify-before-retry; the replication path *is* guarded),
coordination tools are exempt from both palace locks by design,
`GET /logstream/events` **never existed** in 3.8.0 (not proxy-blocked), and
`mempalace sync` never touches the logstream — the log is permanent and unbounded.
- **`docs/fleet-memory.md`**, operator-facing: five storage types, a decision tree,
latency expectations (~2–5 min live session; next session while offline),
broadcast exclusion by design, fan-out, and the search-before-answer /
diary-at-session-end / verify-don't-retry habits.
- **`docs/secret-hygiene.md`**, incl. the tier definitions, the measured FP data,
stated false negatives, and the three server-side call sites (specified, not
built — tier 2 only there, since the hub cannot see a client's env).
- Phase 1 exposure record moved to the private fleet repo with a moved-note stub;
retention direction for the unbounded log (logrotate-style: never rotate
still-owed events, rotation invalidates held cursors, archive-verify-delete).
### The other memory system finally gets explained — `docs/observational-memory.md`
`pi-observational-memory` has been baked for several releases and described in
one line of the feature list (*"the `recall` tool for session compaction"*),
which is enough to name it and not nearly enough to use it. New 272-line
explainer with five diagrams, aimed at someone who has seen `/om:status` or a
"compacted memory" block and wondered whether to leave any of it switched on.
**Scoped to what this repo is authoritative for, because upstream already
documents the mechanism well.** `/opt/pi-observational-memory/docs/` ships
`concepts.md`, `how-it-works.md` and `configuration.md`, including a correct v3
lifecycle diagram — so the new document links those for depth and spends its own
words on the four facts pi-devbox owns and can change: the pinned commit it bakes
(v3.0.4 `ce9fc98`, the value in `build-manifest.json`), the `packages[]` entry
that decides which copy loads, the Haiku-workers-vs-Opus-session split seeded
into `~/.pi/agent/settings.json`, and the `devbox-pi-config` volume that makes
the ledger survive `--force-recreate`. Plus the confusion this image creates by
shipping two things called memory: a section contrasting it with MemPalace, on
the line *observational memory keeps a session coherent, the palace keeps the
fleet coherent*.
Every stated number was read out of the live container or the baked tree rather
than copied from release notes — including the correction that the dropper is
gated on a **successful same-turn reflection** and not on a token threshold of
its own, which is the one detail `pi-extensions/SKILL.md` still gets wrong.
Placement follows the audience split fleet-ops states for itself: reusable
mechanism is not deployment data, so a "why is this in my container" document
belongs in the repo that **pins and wires** the component, pointing upstream for
depth. Linked twice from the README, because before this commit the README
referenced `docs/` zero times and the one file already there
(`mempalace-broker-design.md`) was reachable only by listing the directory.
### A README claim that v1.8.9 made false, and how it got there
**§ Cross-machine agent coordination ended with "Nothing in this image polls the
log on the agent's behalf." The mailbox shipped in v1.8.9, so that sentence has
been wrong since `aac4a1c`.** Replaced with the three knobs and their defaults
(`MEMPALACE_MAILBOX`, `MEMPALACE_MAILBOX_POLL_MS` 300000,
`MEMPALACE_MAILBOX_RESURFACE_MS` 3600000), the fact that owed-ness is *derived*
rather than read off `status`, and the queued-into-the-next-turn delivery
semantics measured on two devices.
The cause is the one v1.8.9 wrote a rule about: the behaviour arrived through the
floating `MEMPALACE_TOOLKIT_REF`, so **no diff in this repo ever touched the
paragraph that made the claim**. v1.8.9's rule ("name a floating-ref behaviour
change in the CHANGELOG before tagging") fired for the CHANGELOG and nobody swept
the README. The CHANGELOG records what *changed*; the README asserts what is
*true*, and only the first is reviewed at release time. Extending the rule
accordingly: grep the README for absolute claims — *nothing*, *never*, *does
not*, *only* — about any component whose SHA moved.
**The replacement is dated on purpose.** It says it describes the bridge *as baked
in v1.8.9* (`mempalace-toolkit` `5b8d78f`) and points at that repo's
`docs/rfc-003-coordination-log.md` §7.11–§7.12 for the mechanism, because toolkit
main is already ahead of the baked copy (`a92c75d` makes delivery say it is queued
and ping the human who is not looking; `e917662` and `ecc2a9c` refine that notify
path) and none of it reaches a container until a base rebuild. Documenting those
here would have swapped a stale-behind claim for a stale-ahead one — the same
defect with the sign flipped.
### Diagrams verified by rendering, not by parsing
Both comparison diagrams **parsed clean and rendered with their meaning
reversed**: Mermaid laid the second declared `subgraph` out first, so "with
observational memory" appeared before "without", and MemPalace before
observational memory in the diagram whose entire job was that contrast. A third
was legible only at 1280px. Rebuilt as declaration-ordered node chains, then
re-rendered at mermaid@11 — the version `pi-studio` pins — in the baked headless
browser and read back as an image. Recorded because it generalises:
`mermaid.parse()` proves syntax and says nothing about layout, so a diagram is
unverified until someone has looked at it.
### … and rendering it in *my* browser was still not enough
Reported from a real viewer: several boxes had their bottom line of text sliced
off. Reproduced and root-caused rather than nudged — **Mermaid measures a node
label with its own font metrics, computes the box, then renders the label as real
HTML inside a `<foreignObject>`.** Any host stylesheet that touches the
`line-height` or `font-size` of that HTML makes the text taller than the box
already committed to, and the overflow is clipped at the box edge. Error
accumulates per line, so the loss always lands on the last line of the tallest
labels — which is exactly what was reported.
Two fixes were tried and only the second works:
- `%%{init: {'flowchart': {'htmlLabels': false}}}%%` — **rejected, and verified
ineffective rather than assumed so.** The directive *is* honoured (label
elements switch from 16 `foreignObject` to 7 `tspan`), and the clipping is
identical, because the inflated font-size still inherits into SVG text.
- **A hard limit of two short lines per node, with the detail moved into the prose
under each diagram.** One- and two-line boxes have enough vertical slack to
absorb the inflation; three- and four-line boxes do not. This is also better
documentation — the old nodes were carrying paragraph-sized text.
The regression harness is now the interesting artefact: render every block with a
deliberately inflated `line-height: 1.7 !important` on the label HTML, screenshot,
and read it. Two survivors of the rewrite were caught only by that harness — a
long unbreakable `/opt/pi-observational-memory` path silently wrapping to a third
line, and a cylinder (`[( )]`) shape, whose curved bottom leaves less room than a
rectangle for the same two lines.
### §4 answers the question the document left hanging: what compaction does to your context
Asked directly and worth writing down: *if the old conversation is folded away, is
the session back to knowing nothing?* No — and the specifics are all checkable
against pi 0.84.3's own `docs/compaction.md` and the extension's source:
- **A verbatim tail survives, sized by a token budget rather than a message
count.** Pi walks back from the newest entry until `keepRecentTokens` [20000],
and everything from that `firstKeptEntryId` onward is kept **unchanged**. Cut
points land on turn boundaries, never mid-tool-call.
- **The system prompt and `AGENTS.md` are not in the compacted region at all** —
they are rebuilt from disk on every request, so compaction cannot lose them.
- **Nothing is deleted from disk.** Compaction *appends* a `compaction` entry
carrying the summary and the cut pointer; no session line is rewritten in place.
- **`recall` therefore still resolves ids whose sources left the context**, because
it reads the full branch via `sessionManager.getBranch()` and never consults the
context window.
- **Repeated compaction does not summarise the summary.** The text is always
rendered from live observation/reflection records, so there is no
generation-loss spiral; the projection is incremental against the last full-fold
boundary and escalates to a true re-fold from the branch root at
`observationsPoolMaxTokens` [20000].
And one correction to this repo's own earlier claim: **"compaction calls no model"
is a steady-state property, not an absolute.** If the ledger is empty — compaction
firing before the observer has ever run — the hook returns nothing and explicitly
declines ownership (`// Decline ownership so Pi's native summarizer preserves the
pre-cut context.`), and pi's own model-based summariser runs. The doc now says so,
with the snippet.
### A shipped doc bug: the ledger entry type was stated exactly backwards
§9 told readers the entries are `custom_message` and specifically *not* `custom`.
It is the other way round, so the one grep the section existed to get right was
the one it got wrong. Corrected against the live session file — 11
`om.observations.recorded` and 6 `om.reflections.recorded` entries, all
`"type":"custom"`, alongside `"type":"custom_message"` entries whose `customType`
is `mempalace-mailbox` and `mempalace-wakeup`, which is precisely where the
confusion came from: **the mailbox uses the context-visible API, om's ledger uses
the invisible one.**
That is not a typo but a load-bearing distinction, and the fix turns it into a
feature the doc now advertises: `custom` entries *"do not participate in LLM
context"* (pi `docs/session-format.md`), so **the ledger costs zero context until
it is folded** — now a row in the cost table.
### Not covered by any of this
The opencode bridge is a separate write path the feeder hook never sees, and the
server-side layer is unbuilt — so a secret typed straight into `add_drawer`, or
staged by a non-pi client, still lands unscrubbed.
---
## v1.8.9 — 2026-08-26
The coordination log gets a reader, and the release checklist's last gate stops
accusing the wrong component.
### The mailbox arrives — named here *because nothing in this repo caused it*
**`mempalace-toolkit` main moves `e70bef2` → `5b8d78f` (exactly one commit, 281
insertions / 9 deletions across `extensions/pi/mempalace.ts` and
`extensions/pi/README.md`), and that is what actually ships the auto-delivered
logstream mailbox.** No pi-devbox commit implements it. `docker-publish.yml`
resolves `MEMPALACE_TOOLKIT_REF=main` to a concrete SHA at build time and folds
that SHA into `base_tag`, so the mailbox would have landed in the next tagged
image **whether or not this section existed** — which is precisely why it exists.
That is the same shipped-undocumented shape as `553d865` in v1.8.7, and that one
caused a cross-host misattribution: an agent on another machine reasoned about
which image contained which behaviour from a CHANGELOG that never mentioned it.
The rule this release adopts: **if a floating ref will pull a behaviour change
into the image, name it in the CHANGELOG before tagging, not after.**
What the mailbox does, from the shipped code rather than from the design
discussion:
- **The bridge was write-only.** It stamped provenance on the way *out* and never
read the log back, so a directed ask reached an agent only if that agent
happened to run `mempalace_event_list` itself. The channel carried real
cross-machine traffic from 2026-08-18 onward with **zero readers** — every
delivery in that window happened because a human said "check your mailbox".
- **Doubly gated, exactly like the provenance stamper:** inert unless *both*
`MEMPALACE_PI_DEVICE` and `MEMPALACE_REMOTE_URL` are set. An unstamped client
has no address to be reached at, so there is nothing for it to read.
- **On by default, opt out with `MEMPALACE_MAILBOX=0`.** Deliberate: an opt-in
fix for a nobody-remembers-to-do-it problem only relocates the forgetting.
Tunables: `MEMPALACE_MAILBOX_POLL_MS` (min gap between mid-session polls,
default 300000) and `MEMPALACE_MAILBOX_RESURFACE_MS` (re-announce a still-owed
ask after, default 3600000).
- **Owed-ness is derived, never read off `status`.** `event_ack` appends and never
mutates, and `status` is written once, so a directed `open` keeps matching the
mailbox query forever — answered or not. A candidate counts as answered only
when one of this device's own events has a **higher `seq`**, joins via
`metadata.ack_of` or a shared `correlation_id`, and carries a terminal status
(`applied`, `superseded`, `failed`, `blocked`). `claimed` and `ready` are
deliberately **not** terminal — that is how "taken, but not finished" keeps
resurfacing.
- **`*` broadcasts are excluded from the owed set.** `to_agent: <me>` also matches
broadcasts per the tool contract, so without this a broadcast written with
`status="open"` would make every machine believe it personally owed the same
answer — and the code would contradict the skill that documents it.
- **The dedup map is in memory on purpose.** A restart forgets, so an already-seen
ask can resurface: visible noise a human corrects in one turn. The opposite
failure — suppressing an unanswered ask — is silent and permanent. Do not
"fix" the noise by persisting it.
- **Delivery queues, it never interrupts.** A sections push at
`before_agent_start` plus a second `agent_settled` handler behind the 300 s
floor, using `steer` and *not* `triggerTurn`: `agent_settled` means idle, so
nothing wakes a model on inbound fleet traffic.
Measured on v1.8.8 (which bakes `e70bef2`, i.e. no mailbox) immediately before
this release: the wake-up mailbox query had to be run by hand, returned **3**
directed asks with `status="open"`, and the derivation above resolved **all
three** as already answered — the third independent confirmation that the raw
`status` filter never shrinks, and the first taken on a fresh container with no
memory of having answered them.
### `--expected-image-version`: two versions, two flags
**`scripts/recreate-sanity-check.sh --expected-version 1.8.8` reported
`✗ pi version mismatch: expected 1.8.8, got 0.84.3` and exit 1** — a red on the
final runtime gate of a release, accusing the image of being the wrong version,
when the flag had only ever asserted `pi --version`. `AGENTS.md` step 4 spelled
it `--expected-version X.Y.Z` inside a checklist where every *other* `X.Y.Z` is
the pi-devbox tag; `README.md` got it right, so the two documents disagreed.
Not hypothetical, and not one reader's slip: the v1.8.8 release-readiness handoff
from `pi@emb-7kj4vr4g` (`evt_20260826T134919_a614ecfc2d4f`) propagated
`--expected-version 1.8.8` twice, in its body and in
`metadata.cannot_check_here`, while correctly calling step 4 "the runtime peer of
the smoke gate, so it is not ceremonial". Two independent readers, one on another
machine, converged on the wrong meaning. Left alone it puts a spurious red on
every release, and the intuitive remedy — re-pull, re-recreate — is pure waste.
- **New `--expected-image-version X.Y.Z`** asserts the pi-devbox release tag,
read from `release_tag` in `/etc/pi-devbox/build-manifest.json` (the image's
own build-time ground truth — no checkout, no network, no Docker socket). A
leading `v` is optional on either side, so `1.8.9` and `v1.8.9` both work.
- **Both flags now detect being handed the other one's value**, and the test is
exact rather than heuristic: the value is compared against the *other*
quantity this image actually reports, so it can only fire when the mix-up is
real. `--expected-version 1.8.9` now says *"is the pi-devbox IMAGE version,
not the pi version — use `--expected-image-version`"*, and the reverse mix-up
is caught the same way.
- **Neither flag is required any more.** With none, the live `pi --version` is
asserted against `pi_version` in the build manifest. That is not a tautology:
`pi` resolves through `PATH`, and a stale install in the `~/.pi/npm-global`
volume can shadow the baked one — the same shadowing this script already
guards against for `npm:pi-atelier` in `packages[]`. Verified by mutating the
manifest to a different version, which made the new check fail as intended.
- **The header note it replaced was stale and load-bearing.** It claimed pi "is
resolved from `latest` at CI build time and is NOT pinned … cannot self-derive
an expected version". `Dockerfile.variant` pins `ARG PI_VERSION=0.84.3`, and
`docker-publish.yml` *reads that ARG* as its source of truth (refusing to
build on a floating value, checking it is published on npm, warning when npm
is ahead). The same withdrawn claim also sat in `cli_utils`'s
`pi-devbox-sanity --help`, the third place this confusion lived; fixed there
too, in that repo.
- Argument parsing hardened while in there: a flag whose value is missing — or
is another flag — is now a usage error (exit 2) instead of silently consuming
the next argument, and `--help` works.
All fourteen flag combinations were exercised by execution, including the two
manifest-absent branches and the shadowing branch, which a healthy container
cannot reach naturally — mutation-tested with a doctored manifest path so that
each failure branch was observed *firing* rather than assumed present.
### Component audit: no bumps, and that is the finding
Checked before tagging, since a base rebuild was already forced:
| Component | In v1.8.8 | Upstream now | Action |
|---|---|---|---|
| pi (npm) | `0.84.3` (pinned) | `0.84.3` is `latest` | none |
| mempalace (PyPI) | `3.8.0` (pinned) | `3.8.0` | none |
| pi-atelier | `v0.8.2` (pinned) | `v0.8.2` highest tag | none |
| pi-toolkit, pi-extensions, pi-fork, pi-observational-memory, pi-studio | floating | **identical to baked** | none |
| skillset snapshot | `6eb20af` | `6eb20af` | none |
| **mempalace-toolkit** | `e70bef2` | **`5b8d78f`** | ships the mailbox |
So the whole ~67-minute base rebuild this tag pays for is attributable to the
toolkit SHA alone — `base_tag` folds it, and it moved. Every other floating ref
resolved to the commit already baked (verified with `git ls-remote` per repo, not
by reading a cached clone).
One claim in this audit came from a fork that had fabricated its findings — six
plausible-looking toolkit commits with five nonexistent SHAs, a pi `0.84.4` that
npm has never published, a pi-studio commit `ls-remote` says does not exist, and
a compatibility floor of `0.8.2` where the code says `0.7.1`. Every row above was
therefore re-measured directly. Recorded because the failure mode is specific:
none of it looked wrong, and `git cat-file -e` is what caught it.
---
## v1.8.8 — 2026-08-26
The vendored `mempalace` skill snapshot stops being anonymous, and the The vendored `mempalace` skill snapshot stops being anonymous, and the
container starts saying which copy of each skill it is actually reading. container starts saying which copy of each skill it is actually reading.
@@ -45,6 +524,23 @@ being fixed.**
only, `2` cannot determine (ref absent from this clone). `AGENTS.md` step 2 only, `2` cannot determine (ref absent from this clone). `AGENTS.md` step 2
rewritten to state all three, since its promise that "the message rewritten to state all three, since its promise that "the message
distinguishes the two" was exactly what the branch was breaking. distinguishes the two" was exactly what the branch was breaking.
- **The staleness `NOTICE` then asserted a direction it had never tested** — the
same defect one layer down, found by pi@emb-7kj4vr4g against the real state of
its own host. The branch fired the notice on "recorded ≠ HEAD" and announced
that HEAD was the newer side, so a clone that was merely *behind* was told
*"has moved to 82a8d3c; the snapshot describes the older 5fd0d5c"* when
`82a8d3c` is `5fd0d5c`'s **ancestor**. Harmless to the verdict (`rc` stayed 0,
nothing was mis-verified) but it points the operator at a refresh — a
~67-minute base rebuild — when the real remedy is `git pull`. It now tests
ancestry with the `merge-base --is-ancestor` primitive the refresh path two
sections above already used, and reports three distinct verdicts: **stale**
(recorded is an ancestor — refresh), **your clone is behind** (HEAD is an
ancestor — pull, do not refresh), **diverged** (neither). All three verified by
execution; only the first was right before.
- **`--help` died with `unknown option: --help`.** The strict argument loop that
closed the silent-ignore hole never added a `--help` case, so the one script
whose argument *order* was itself a landmine had an erroring discoverability
path. It now prints its own header block.
- **`VENDORED.md` contradicted itself, in the release whose stated invariant is - **`VENDORED.md` contradicted itself, in the release whose stated invariant is
non-contradiction.** Its hand-maintained "Snapshot provenance at last refresh" non-contradiction.** Its hand-maintained "Snapshot provenance at last refresh"
line named skillset `670f7f1` — seven commits behind the ARG, and *the very line named skillset `670f7f1` — seven commits behind the ARG, and *the very
@@ -88,8 +584,22 @@ earlier. Fixed in skillset `5fd0d5c`, which derives owed-ness by joining on
terminal reply suppresses every later ask on the same thread forever. Because terminal reply suppresses every later ask on the same thread forever. Because
the skillset is mounted live on every enrolled host, that correction was already the skillset is mounted live on every enrolled host, that correction was already
deployed fleet-wide before this image was built; the vendored snapshot is deployed fleet-wide before this image was built; the vendored snapshot is
resynced to it (`c04cd15` → `5fd0d5c`) so the no-clone fallback does not ship resynced to it (`c04cd15` → `5fd0d5c` → `6eb20af`) so the no-clone fallback does
the withdrawn rule. Canary re-verified bidirectionally against the new bytes. not ship the withdrawn rule. Canary re-verified bidirectionally against the new
bytes.
`6eb20af` adds the limit of that ordering test, found when pi@emb-7kj4vr4g
verified it rather than adopting it: **`seq` is replica-local.** It equals
`origin_seq` today only because one replica authors events for all four machines,
so a second replica could order the same pair differently and derive a different
owed-set from the same log — use `hlc` (already on every event, total and
causally consistent) once `mesh_peers` reports any peer. Documented as reasoning,
not measurement, since a second replica cannot be stood up to test it. The part
worth keeping is the **asymmetry**: local-`seq` skew makes an answered item
*resurface* (noise, self-correcting, visible), while a timestamp comparison
*suppresses an unanswered ask forever* (silent, permanent) — so anyone tempted to
"fix" a resurfacing item with `created_at` would be trading the safe failure for
the dangerous one.
**Also carried, previously undocumented:** `dbb7879` resynced the vendored **Also carried, previously undocumented:** `dbb7879` resynced the vendored
`mempalace` snapshot to skillset `c04cd15` ("the withdrawal only holds where the `mempalace` snapshot to skillset `c04cd15` ("the withdrawal only holds where the
@@ -275,14 +785,17 @@ the skillset actually is — a maintainer's clone, or any running container.
because when every host is a thin client of one palace the stamped agent name because when every host is a thin client of one palace the stamped agent name
is the only thing distinguishing them. Set both or neither: a container is the only thing distinguishing them. Set both or neither: a container
without the device var can read the log but is addressable by nobody. without the device var can read the log but is addressable by nobody.
- the skillset's `mempalace` skill (`82a8d3c`, live on every host that mounts - the skillset's `mempalace` skill (`6eb20af`, live on every host that mounts
the skillset, no rebuild needed) — the **norms**: a mailbox query at wake-up, the skillset, no rebuild needed) — the **norms**: a mailbox query at wake-up,
and the sender-declared ack contract, where a *directed* event with and the sender-declared ack contract, where a *directed* event with
`status="open"` is owed a reply and a `*` broadcast owes nothing. Measured `status="open"` is owed a reply and a `*` broadcast owes nothing. The
while designing it: an unfiltered mailbox returned 5 events, 4 of them `status` filter earns its place by dropping broadcast noise — measured, an
finished broadcasts from eight days earlier, where `status="open"` returned unfiltered mailbox returned 5 events, 4 of them finished broadcasts from
exactly the 1 that needed an answer — an unfiltered mailbox trains you to eight days earlier — but that is **all** it does; it does not compute
ignore it, so the filter is the feature. owed-ness, and the version of this entry that claimed otherwise is withdrawn
above. Measured today, both machines: the raw filter returns 2 asks here and
1 there, **every one already answered**, while the derivation returns 0 for
both. Dropping noise and deciding what is owed are two different jobs.
- mempalace-toolkit `extensions/pi/README.md` (`e70bef2`) — the **mechanism**, - mempalace-toolkit `extensions/pi/README.md` (`e70bef2`) — the **mechanism**,
including that the bridge is *write-only* today (it stamps events going out including that the bridge is *write-only* today (it stamps events going out
and never reads the log, so nothing in this image polls on the agent's and never reads the log, so nothing in this image polls on the agent's
@@ -290,16 +803,19 @@ the skillset actually is — a maintainer's clone, or any running container.
implements `GET /logstream/stream`, but a reverse proxy exposing only `/mcp` implements `GET /logstream/stream`, but a reverse proxy exposing only `/mcp`
makes it unreachable — verified by 404s against the real endpoint. makes it unreachable — verified by 404s against the real endpoint.
⚠️ **This makes the vendored snapshot stale on purpose.** The skill edit is in ⚠️ **The snapshot was refreshed rather than left stale.** The skill edits landed
the skillset (`82a8d3c`), so `SKILLSET_SNAPSHOT_REF` still records `c04cd15` in the skillset (`5fd0d5c`, then `6eb20af`), so `SKILLSET_SNAPSHOT_REF` was
and `scripts/vendor-mempalace-skill.sh --check` now exits 1 with resynced to match and `scripts/vendor-mempalace-skill.sh --check` is a clean
*"has moved to 82a8d3c; the snapshot describes the older c04cd15"*. Refreshing `OK` with no notice: the no-clone fallback carries the **corrected** protocol,
it is a deliberate release-day decision, not an oversight — hence the new not the withdrawn one. That mattered more than currency usually does, because
step 2 in `AGENTS.md` § *Release-day checklist*, which states both that the the superseded copy contained an instruction — the `mesh_peers` gate — that
refresh costs a base rebuild and that skipping it is legitimate because every actively told a reader to skip the feature. Refreshing remains a deliberate
enrolled host reads its live clone. What is not legitimate is skipping it release-day decision rather than an automatic one: it costs a base rebuild, and
*silently*, which is precisely what the new manifest fields and skipping it is legitimate because every enrolled host reads its live clone.
`pi-devbox-version` output make impossible. What is not legitimate is skipping it *silently*, which is what the new manifest
fields and `pi-devbox-version` output make impossible — hence step 2 in
`AGENTS.md` § *Release-day checklist*. In this release the refresh was free:
`rootfs/` was already changing, so the base rebuild was forced anyway.
--- ---
+1 -1
View File
@@ -305,7 +305,7 @@ ARG MEMPALACE_TOOLKIT_REF=main
# no ~67-minute base rebuild. (scripts/check-base-hash.sh scans only # no ~67-minute base rebuild. (scripts/check-base-hash.sh scans only
# Dockerfile.base, so no folding into the base hash is required — nor would # Dockerfile.base, so no folding into the base hash is required — nor would
# it be correct, since this ARG changes nothing about the base's contents.) # it be correct, since this ARG changes nothing about the base's contents.)
ARG SKILLSET_SNAPSHOT_REF=5fd0d5c406df506fd0e70977b6bd87f5fbc3b803 ARG SKILLSET_SNAPSHOT_REF=6eb20af181f0147cb8c1377f6e36a6a47a68e8e5
# Dockerfile.base sets description="pi-devbox — base image (variant-independent)" # Dockerfile.base sets description="pi-devbox — base image (variant-independent)"
# and every variant INHERITS it, so both published images used to advertise # and every variant INHERITS it, so both published images used to advertise
+60 -4
View File
@@ -20,7 +20,9 @@ on the host.
- `pi-extensions` — TypeScript extensions for pi (preview, MCP bridges, - `pi-extensions` — TypeScript extensions for pi (preview, MCP bridges,
mempalace integration, etc.) mempalace integration, etc.)
- `pi-fork` — the `fork` tool for spawning sub-agents - `pi-fork` — the `fork` tool for spawning sub-agents
- `pi-observational-memory` — the `recall` tool for session compaction - `pi-observational-memory` — durable session memory: the ledger that makes
compaction cheap, plus the `recall` tool. See
[`docs/observational-memory.md`](docs/observational-memory.md)
- `pi-atelier` — TUI sidebar: ordered panels, split-pane, themes. Pinned to an - `pi-atelier` — TUI sidebar: ordered panels, split-pane, themes. Pinned to an
audited tag; see [Version pins](#version-pins-pi-pi-atelier-mempalace) audited tag; see [Version pins](#version-pins-pi-pi-atelier-mempalace)
@@ -579,7 +581,50 @@ convention that a directed event with `status="open"` is a request owed a reply
while a `*` broadcast owes nothing. The mechanism side (what the bridge stamps, while a `*` broadcast owes nothing. The mechanism side (what the bridge stamps,
and why live SSE push depends on the palace deployment's reverse proxy rather and why live SSE push depends on the palace deployment's reverse proxy rather
than on this image) is documented in the toolkit's `extensions/pi/README.md`. than on this image) is documented in the toolkit's `extensions/pi/README.md`.
Nothing in this image polls the log on the agent's behalf.
**Since v1.8.9 the bridge reads the log for you.** Earlier images were write-only
— they stamped provenance on the way out and never read back, so a directed ask
reached an agent only if that agent happened to run `mempalace_event_list`
itself. The mailbox is gated on the same two variables as the stamper, is on by
default, and derives what is *owed* rather than trusting `status` (an acked event
keeps matching a `status="open"` query forever, because the log is append-only):
| Variable | Default | Effect |
|---|---|---|
| `MEMPALACE_MAILBOX` | unset (on) | `0` disables mailbox reads entirely |
| `MEMPALACE_MAILBOX_POLL_MS` | `300000` | minimum gap between mid-session polls |
| `MEMPALACE_MAILBOX_RESURFACE_MS` | `3600000` | re-announce a still-owed ask after this long |
Delivery **queues, it never interrupts**: the poll runs when pi goes idle and the
message is steered into the *next* turn, so nothing wakes the model on inbound
fleet traffic. The practical consequence, measured on two devices: the message
appears in your session window and the agent acts on it when the next turn
starts — you are the trigger. (That describes the bridge **as baked in v1.8.9**,
`mempalace-toolkit` `5b8d78f`; the mailbox's own mechanism and landmines live in
the toolkit's `docs/rfc-003-coordination-log.md` §7.11–§7.12, which moves ahead of
whatever this image has baked.)
## Observational memory (in-session memory)
The image also bakes [pi-observational-memory](https://github.com/elpapi42/pi-observational-memory),
which is memory of a *different kind* from the palace and is easy to confuse with
it. It keeps a small branch-local ledger of observations and reflections while a
session runs, so when pi compacts the conversation the summary is a
**deterministic fold of that ledger rather than a model call**, and every item
keeps a 12-character id that `recall(<id>)` resolves back to the exact source.
In one line: **observational memory keeps a session coherent; the palace keeps
the fleet coherent.**
It is on by default, needs no habit from you, and sends its background work to a
cheaper model than your session (Haiku while the session runs Opus, in the seeded
`~/.pi/agent/settings.json`). Inspect it from inside pi with `/om:status` and
`/om:view`; turn all proactive work off for one run with
`PI_OBSERVATIONAL_MEMORY_PASSIVE=1 pi`.
What it is for, how the lifecycle works, what it costs, every setting and its
default, and how it differs from MemPalace:
[`docs/observational-memory.md`](docs/observational-memory.md).
## Agent skills ## Agent skills
@@ -1017,10 +1062,21 @@ After `docker compose up -d --force-recreate`, run the **runtime** peer of
persisted volumes survived, and pi runtime wiring is intact: persisted volumes survived, and pi runtime wiring is intact:
```bash ```bash
./scripts/recreate-sanity-check.sh # auto-detects variant ./scripts/recreate-sanity-check.sh # auto-detects variant
./scripts/recreate-sanity-check.sh --expected-version 0.79.4 # assert pi version ./scripts/recreate-sanity-check.sh --expected-image-version 1.8.9 # assert the pi-devbox release tag
./scripts/recreate-sanity-check.sh --expected-version 0.84.3 # assert the pi coding agent version
``` ```
Those are **two different versions**, and the flags are not interchangeable:
`--expected-image-version` takes the pi-devbox release tag (`v` optional),
`--expected-version` takes `pi --version`. Hand one the other's value and it
says so by name instead of reporting a mismatch against the wrong component.
With neither flag, both values are read from the image's own build manifest
(`/etc/pi-devbox/build-manifest.json`): the live pi version is asserted against
the one recorded at build time — which catches a stale `pi` in the
`~/.pi/npm-global` volume shadowing the baked one — and the release tag is
reported informationally.
If `cli_utils` is on your PATH, the `pi-devbox-sanity` wrapper runs the same If `cli_utils` is on your PATH, the `pi-devbox-sanity` wrapper runs the same
check by short name and locates the repo automatically (override with check by short name and locates the repo automatically (override with
`PI_DEVBOX_REPO=/path/to/pi-devbox`). Like `smoke-test.sh`, this script is `PI_DEVBOX_REPO=/path/to/pi-devbox`). Like `smoke-test.sh`, this script is
+371
View File
@@ -0,0 +1,371 @@
# Observational memory — why this image has it, and what it does for you
**Audience:** anyone using this container for long pi sessions who has wondered
what `recall`, `/om:status` and "compacted memory" are, or whether they should
leave any of it switched on.
**Companion documents:** the extension ships its own reference docs at
`/opt/pi-observational-memory/docs/` —
[`concepts.md`](https://github.com/elpapi42/pi-observational-memory/blob/main/docs/concepts.md)
(the model),
[`how-it-works.md`](https://github.com/elpapi42/pi-observational-memory/blob/main/docs/how-it-works.md)
(hooks and internals) and
[`configuration.md`](https://github.com/elpapi42/pi-observational-memory/blob/main/docs/configuration.md)
(every setting). Pi's own compaction mechanics are in
`/usr/lib/node_modules/@earendil-works/pi-coding-agent/docs/compaction.md`.
Those are normative; this document is the **deployment** view — what is pinned
here, how it is wired, what it costs, and how it differs from MemPalace. For the
palace, see
[`mempalace-toolkit/docs/fleet-memory.md`](https://gitea.jordbo.se/joakimp/mempalace-toolkit/src/branch/main/docs/fleet-memory.md).
> Verified on pi-devbox **v1.8.9** (`release_tag v1.8.9`, source `aac4a1c`),
> which bakes pi-observational-memory **v3.0.4** at commit `ce9fc98` — the value
> in `/etc/pi-devbox/build-manifest.json` → `components.pi-observational-memory`.
> Every number below was read from that tree, from pi 0.84.3's own docs, or from
> the live container.
---
## 1. The problem it solves
A long pi session outgrows the model's context window. Pi's answer is
**compaction**: fold the older part of the conversation into a summary and keep
recent messages verbatim. That is unavoidable, and it is where sessions go
wrong — the summary is produced *at the moment of pressure*, by a model, about a
transcript that is about to leave the context.
Observational memory changes *when* the remembering happens. Instead of
summarising in a panic at the end, it keeps a small **ledger** up to date while
the session runs, and compaction then just folds that ledger.
```mermaid
flowchart LR
A0["plain compaction"] --> A1["context fills"]
A1 --> A2["a model summarises<br/>under pressure"]
A2 --> A3["prose summary,<br/>no way back"]
B0["with observational<br/>memory"] --> B1["context fills"]
B1 --> B2["ledger written<br/>as you work"]
B2 --> B3["compaction folds<br/>the ledger"]
B3 --> B4["ids you can<br/>recall"]
```
Top row is pi on its own: one model call at the worst possible moment, detail
chosen in a hurry, and the original wording gone from view. Bottom row is this
image's default: the thinking happened earlier on a cheap model, the fold is
deterministic, and every line in the result carries an id that resolves back to
the exact source.
## 2. The mental model: three layers and a ledger
| Layer | What it is | Example |
|---|---|---|
| **Observation** | a timestamped, source-backed event from the conversation | "user rejected option B because it needs a base rebuild" |
| **Reflection** | a durable conclusion *backed by* observations | "the user optimises for avoiding 67-minute rebuilds" |
| **Drop** | a tombstone retiring an observation from active memory | the superseded detail of a bug that is now fixed |
These are appended to the session as silent ledger entries
(`om.observations.recorded`, `om.reflections.recorded`,
`om.observations.dropped`) and **folded** — replayed in order — to produce the
memory state. The ledger is the source of truth; what you see in a compacted
session is a rendering of it.
Two properties follow, and both matter later:
- **The ledger itself costs no context.** Those entries are pi `custom` entries,
which *"do not participate in LLM context"* (pi `docs/session-format.md`). They
sit in the session file and reach the model only via the fold at compaction.
- **Memory is branch-local.** A pi session is a tree (resume, fork), and the fold
follows the current branch only, so a forked branch does not inherit another
branch's view.
## 3. The lifecycle
Three background workers and one compaction hook, driven by *raw token
progress* rather than wall-clock time. Defaults in brackets.
```mermaid
flowchart TD
T(["turn_end"]) --> O{"10k raw tokens<br/>since observing?"}
O -- yes --> OBS["<b>observer</b> runs"]
O -- "no" --> R{"20k tokens<br/>since reflecting?"}
R -- yes --> REF["<b>reflector</b> runs"]
REF -- "if pool over 10k" --> DR["<b>dropper</b> prunes"]
S(["agent_settled"]) --> C{"81k tokens<br/>since compacting?"}
C -- yes --> CP["ctx.compact()"]
CP --> H(["session_before_compact"])
H --> F["fold the ledger<br/>no model call"]
F --> VIS["compacted memory"]
```
- **observer** — `observeAfterTokens` [10000]: writes observations for the
conversation it has not covered yet.
- **reflector** — `reflectAfterTokens` [20000]: promotes patterns across
observations into durable reflections.
- **dropper** — no clock of its own. It is post-reflection maintenance, gated on
a *successful same-turn* reflection **and** an active pool above
`observationsPoolTargetTokens` [10000]. Not a third worker on a third
threshold.
- **compaction** — `compactAfterTokens` [81000], checked when pi goes idle, so it
never interrupts a turn. Pi will also compact on its own when the context is
nearly full (`contextTokens > contextWindow - reserveTokens`, `reserveTokens`
[16384]).
## 4. What compaction actually does to your context
This is the question the rest of the document used to leave hanging: if the old
conversation is folded away, is the session back to knowing nothing?
**No.** Compaction replaces *part* of the context, not all of it, and it deletes
nothing at all from disk.
```mermaid
flowchart LR
SYS["system prompt<br/>+ AGENTS.md"] --> CTX["what the model sees<br/>on the next turn"]
SUM["folded memory:<br/>reflections + observations"] --> CTX
TAIL["recent turns,<br/>verbatim"] --> CTX
DISK[("session .jsonl: all of it")] -. "recall(id)" .-> CTX
```
Where each piece comes from:
- **System prompt and `AGENTS.md` — never compacted, because they were never
conversation.** Pi rebuilds them from disk on every request
(`loadContextFileFromDir`), so they cannot be lost by compaction.
- **The verbatim tail — sized by a token budget, not a message count.** Pi walks
backwards from the newest entry accumulating token estimates until
`keepRecentTokens` [20000] is reached; that entry becomes `firstKeptEntryId`,
and *everything from there on is kept unchanged*. Cut points land on turn
boundaries, never mid-tool-call. So the most recent ~20k tokens of real work —
your last instructions, the diffs, the test output — survive word for word.
- **The folded memory — replaces only what came before that cut.** Rendered from
the ledger's records: reflections and observations, each with its 12-hex id.
- **The session file — untouched.** Compaction *appends* a `compaction` entry
(`{"type":"compaction", summary, firstKeptEntryId, tokensBefore, …}`) and
rebuilds context from it on later turns. Nothing is rewritten in place; the
only documented way to remove session content is deleting the whole `.jsonl`.
That last point is what makes the answer to "is the detail gone?" *no* rather
than *mostly*: `recall` does not read the context window at all. It calls
`sessionManager.getBranch()` — the full branch from the root — and resolves an
observation id back to the original entries. Detail that left the model's view
an hour ago is still one `recall` away.
**Repeated compaction does not summarise the summary.** The rendered text is
always built from live observation/reflection *records*, never from the previous
compaction's prose, so there is no generation-loss spiral. (Mechanically the
projection is incremental — it re-derives back to the last full-fold boundary and
carries the rest forward, escalating to a genuine re-fold from the branch root
when the observation pool reaches `observationsPoolMaxTokens` [20000].)
So the honest summary of the state after compaction: **the model keeps its
instructions, keeps recent work verbatim, trades older turns for a dense
id-carrying digest of them, and can pull any of it back on demand.** Not a fresh
start — a smaller, cheaper, still-navigable one.
### One caveat about "no model call"
If the ledger is empty — compaction fires before the observer has ever run — the
hook returns nothing and *declines ownership*, and pi's own model-based
summariser runs instead:
```ts
const summary = renderSummary(projection.reflections, projection.observations);
if (summary.length === 0) {
// Decline ownership so Pi's native summarizer preserves the pre-cut context.
return;
}
```
In steady state (any session old enough to have produced one observation) om's
hook wins and compaction is model-free. "Never calls a model" is true in practice
and false in principle; the fallback is deliberate, so an empty ledger degrades
to normal pi rather than to no summary at all.
## 5. What you actually get
- **Compaction stops being a stall.** In steady state the latency path is
deterministic work over ledger entries, not a summarisation call.
- **Nothing important vanishes silently.** Compaction is lossy by design, but
every item keeps a 12-character id, and `recall(<id>)` returns the exact
evidence — original wording, reasoning, file path, error text.
- **The bookkeeping runs on a cheaper model than your session.** In this image
that is deliberate and visible (§7): background workers on Haiku, session on
Opus.
- **It is automatic.** No habit to maintain, unlike the palace protocol — which is
exactly why the two complement each other (§11).
- **Forks stay clean.** Branch-local memory means a `fork` sub-agent's noise does
not leak into the parent's folded memory.
## 6. `recall` is not a search tool
`recall` takes **one specific 12-hex id** that already appears in compacted
memory or in `/om:view`. It cannot be given a topic. It can return an observation
(marked `active` or `dropped`), or a reflection together with the observations
supporting it.
```mermaid
sequenceDiagram
participant M as compacted memory
participant A as agent
participant L as ledger
M->>A: "[high] user rejected option B (a1b2c3d4e5f6)"
A->>L: recall("a1b2c3d4e5f6")
L-->>A: exact observation + source ids
Note over A: acts on the original wording
```
The rule of thumb the agent skill uses: recall **before a load-bearing action**
that rests on a compressed memory — shipping a change, asserting a fact,
answering "why do you believe that". One recall is cheap; redoing finished work
is not.
## 7. How it is wired in this image
```mermaid
flowchart TB
IMG["baked in the image:<br/>v3.0.4 @ ce9fc98"] --> REG["settings.json<br/>packages[]"]
REG --> SESS["your pi session"]
SESS -- "your turns" --> SM["session model:<br/>Opus"]
SESS -- "observer, reflector,<br/>dropper" --> WM["memory model:<br/>Haiku"]
SESS -- "ledger entries" --> JL["session .jsonl"]
JL --> VOL[("devbox-pi-config<br/>volume")]
```
Four consequences of that wiring:
1. **There is no separate database.** Memory *is* entries inside the ordinary pi
session file (`~/.pi/agent/sessions/<project>/<timestamp>_<uuid>.jsonl`).
Nothing extra to back up, nothing to migrate.
2. **It survives container recreate**, because `~/.pi` is the `devbox-pi-config`
named volume (`docker-compose.yml`) — the same one holding your pi config and
session history.
3. **`packages[]` is the only source of truth for which copy is loaded.** A clone
at `/workspace/pi-observational-memory` may exist (and today matches `/opt`
byte-for-byte at `ce9fc98`) — its presence proves nothing. To run a patched
build you point `packages[]` at it explicitly and start a new session.
4. **The worker model is a deliberate choice, and it is yours to change.** The
seeded config sends background work to Haiku while your session runs Opus:
```json
"observational-memory": {
"model": { "provider": "amazon-bedrock", "id": "eu.anthropic.claude-haiku-4-5-20251001-v1:0" },
"debugLog": false
}
```
## 8. What it costs
| Resource | Cost |
|---|---|
| Model calls | up to **three** background calls per consolidation pass (observer, reflector, dropper), each capped at `agentMaxTurns` [16], on the configured memory model — not your session model |
| Latency in your turns | none by construction: workers run from `turn_end`, compaction runs when pi is idle, and the fold itself does no model work |
| Disk | negligible — JSON lines inside a session file that would exist anyway (measured here: `~/.pi/agent/sessions` = 30 MB total, tens of `om.*` entries per session) |
| Context window | **zero until compaction.** `custom` entries do not enter LLM context; only the folded summary does |
| Attention | none once configured; there is no protocol for you or the agent to remember |
If that is still more than you want on a given run, §9's `passive` switch turns
off all proactive work while keeping `recall` and `/om:*` usable.
## 9. Configuration
Global: `~/.pi/agent/settings.json` (persisted in the volume). Per project:
`<project>/.pi/settings.json`, which overrides global. Precedence is
project → global → environment, and the environment can only override `passive`.
| Key | Default | What it changes |
|---|---|---|
| `observeAfterTokens` | `10000` | observer cadence — lower means smaller chunks and more calls |
| `reflectAfterTokens` | `20000` | reflector cadence (and thereby dropper opportunities) |
| `observerChunkMaxTokens` | 20% of the memory model's context window, else `60000` | cap on one observer run's input |
| `compactAfterTokens` | `81000` | when proactive auto-compaction fires |
| `observationsPoolMaxTokens` | `20000` | pool size at which compaction does a full re-fold from the branch root |
| `observationsPoolTargetTokens` | half of max (`10000`) | what the dropper aims back down to |
| `agentMaxTurns` | `16` | shared turn cap for the three workers |
| `model` | unset → session model | send background work to a cheaper/faster model |
| `showWorkerNotifications` | `true` | routine "observer ran" notices |
| `passive` | `false` | **kill switch** for all proactive background work; `recall` and `/om:*` still work |
| `debugLog` | `false` | per-session NDJSON trace at `~/.pi/agent/observational-memory/debug/<session-id>.ndjson` |
Pi's own compaction knobs live under a separate `compaction` key —
`keepRecentTokens` [20000] sets the verbatim tail from §4, `reserveTokens`
[16384] the headroom that triggers pi's own compaction.
One-off passive run, no config edit:
```bash
PI_OBSERVATIONAL_MEMORY_PASSIVE=1 pi
```
Invalid values are ignored rather than fatal, so a typo degrades to the default
instead of breaking your session — which also means a typo is silent. Check with
`/om:status`.
## 10. Confirming it is actually working
Do not infer health from the absence of a warning; look:
```bash
# 1. inside pi — the authoritative view
/om:status # visible-vs-full drift, thresholds, worker state
/om:view # what the agent currently sees
/om:view full # full ledger truth at the branch tip
# 2. from a shell — are ledger entries being written, and has it compacted?
grep -o '"customType":"om\.[a-z.]*"' \
"$(ls -t ~/.pi/agent/sessions/*/*.jsonl | head -1)" | sort | uniq -c
grep -c '"type":"compaction"' "$(ls -t ~/.pi/agent/sessions/*/*.jsonl | head -1)"
# 3. which copy is loaded, and at what commit
python3 -c "import json;print(json.load(open('$HOME/.pi/agent/settings.json'))['packages'])"
git -c safe.directory=/opt/pi-observational-memory -C /opt/pi-observational-memory rev-parse HEAD
```
Ledger entries are `"type":"custom"` with `"customType":"om.…"`. Do not grep for
`custom_message` — that is a *different* pi API for entries that **do** enter LLM
context, used here by the MemPalace mailbox (`customType: "mempalace-mailbox"`),
not by om.
## 11. It is not the same thing as MemPalace
Both are called "memory" and they solve different problems. Nothing is wrong with
running both — this image does, and they cover each other's failure modes.
```mermaid
flowchart LR
O0["observational<br/>memory"] --> O1["horizon:<br/>this session"]
O1 --> O2["scope: one branch,<br/>one machine"]
O2 --> O3["automatic"]
O3 --> O4["retrieval:<br/>recall(id)"]
P0["MemPalace"] --> P1["horizon: months,<br/>machines"]
P1 --> P2["scope:<br/>the fleet"]
P2 --> P3["protocol-driven"]
P3 --> P4["retrieval:<br/>search, KG, mailbox"]
```
| Question | Answer |
|---|---|
| "What did we decide 200 turns ago in *this* session?" | observational memory (and `recall` for the exact wording) |
| "What did we decide last month, or on another machine?" | MemPalace (`mempalace_search`, diaries) |
| "What is true *right now* about version X?" | MemPalace knowledge graph |
| "Does another machine need something from me?" | MemPalace coordination log — see [Cross-machine agent coordination](../README.md#cross-machine-agent-coordination) |
| "Why is compaction not losing my session?" | observational memory |
The crisp version: **observational memory keeps a session coherent; the palace
keeps the fleet coherent.** A container recreate wipes neither — but only because
`~/.pi` and the palace both live outside the container filesystem.
## 12. Gotchas
- **Branch-local means branch-local.** Resuming or forking changes which ledger
is folded. Memory that "disappeared" is usually on another branch.
- **`recall` needs an id, not a topic.** If you only have a topic, that is a
palace search, not a recall.
- **A `/workspace` clone is not evidence of what is loaded** — see §7.3.
- **`showWorkerNotifications: true` is not proof of work**; it reports runs, and
an observer that deliberately emits nothing writes no ledger entry and simply
retries after another `observeAfterTokens`.
- **A turn bigger than `keepRecentTokens` splits.** The cut then lands mid-turn at
an assistant message and pi merges two summaries — rare, but it is why a very
large single turn can lose more verbatim detail than you would expect.
- **`git log` in the baked tree needs `safe.directory`** (`/opt` is root-owned):
`git -c safe.directory=/opt/pi-observational-memory -C /opt/pi-observational-memory log`.
@@ -462,6 +462,24 @@ every later ask on the same `correlation_id` for good; verified on a live thread
where a `ready` reply at `seq` 16 sits *before* the request at `seq` 17 that it where a `ready` reply at `seq` 16 sits *before* the request at `seq` 17 that it
obviously cannot have answered. obviously cannot have answered.
**On a real mesh, compare `hlc` instead.** `seq` is *replica-local*: it equals
`origin_seq` today only because a single replica authors events for every
machine. Enrol a second replica and a late-syncing peer event gets a late local
`seq`, so two replicas can order the same pair differently and derive different
owed-sets from the same log. Every event already carries `hlc`
(`<millis>-<counter>-<replica_id>`), which is total and causally consistent.
So: compare `seq` while `mempalace_mesh_peers` reports no peers, `hlc` once it
reports any, and `created_at` never. (This is a legitimate use of `mesh_peers` —
choosing an ordering key — not the discredited gate on *whether* to read your
mailbox at all.)
**The failure directions are not symmetric, which is why this is safe to get
slightly wrong.** Local-`seq` skew can make an already-answered item *resurface*
as owed: noise, self-correcting, and visible. A timestamp comparison can
*suppress an unanswered ask forever*: silent and permanent. So if you ever see an
item you know you answered come back, do **not** "fix" it by reaching for
`created_at` — you would be trading the safe failure for the dangerous one.
This also supplies the "taken, not finished" state that looked missing: This also supplies the "taken, not finished" state that looked missing:
`claimed` and `ready` are deliberately **not** terminal, so work you have picked `claimed` and `ready` are deliberately **not** terminal, so work you have picked
up keeps resurfacing until you close it out. No extra convention, no new field. up keeps resurfacing until you close it out. No extra convention, no new field.
+125 -15
View File
@@ -2,8 +2,8 @@
# Runtime post-recreate verification for pi-devbox. # Runtime post-recreate verification for pi-devbox.
# #
# Verifies that after `docker compose up -d --force-recreate`: # Verifies that after `docker compose up -d --force-recreate`:
# - The new image is actually live (pi version matches, when an expected # - The new image is actually live (both the pi version and — when asked —
# version is supplied — see the version note below) # the pi-devbox image release tag; see the two version notes below)
# - Persisted named volumes survived (~/.pi config, shell history, zoxide, # - Persisted named volumes survived (~/.pi config, shell history, zoxide,
# nvim data, uv cache, ssh-local) # nvim data, uv cache, ssh-local)
# - pi runtime wiring is intact: keybindings symlink, AGENTS.md symlink, # - pi runtime wiring is intact: keybindings symlink, AGENTS.md symlink,
@@ -25,13 +25,33 @@
# the pi-devbox repo (which a maintainer already has for CI builds). A plain # the pi-devbox repo (which a maintainer already has for CI builds). A plain
# `docker pull` consumer is not the audience and will not have this file. # `docker pull` consumer is not the audience and will not have this file.
# #
# Version note: pi's version is resolved from `latest` at CI build time and is # TWO DIFFERENT VERSIONS, TWO DIFFERENT FLAGS. This distinction has already
# NOT pinned to a concrete value in Dockerfile.variant (ARG PI_VERSION=latest). # cost a release day, so it is spelled out here and in AGENTS.md step 4:
# So unlike opencode-devbox, this script cannot self-derive an expected version
# from the Dockerfile. Pass --expected-version to assert a match; without it the
# live pi version is reported as an informational WARN, not a failure.
# #
# Usage: ./scripts/recreate-sanity-check.sh [--expected-version X.Y.Z] [--variant studio|plain] # --expected-version the PI CODING AGENT version, e.g. 0.84.3
# (`pi --version`; pinned as ARG PI_VERSION in
# Dockerfile.variant, which CI reads as the source
# of truth)
# --expected-image-version the PI-DEVBOX IMAGE release tag, e.g. 1.8.9 or
# v1.8.9 (the `release_tag` baked into
# /etc/pi-devbox/build-manifest.json)
#
# Passing a release tag to --expected-version used to report
# "pi version mismatch: expected 1.8.8, got 0.84.3" — an accusation aimed at
# the wrong component, on the last gate of a release. Both flags now detect
# being handed the other one's value and say so instead.
#
# Neither flag is required. Both values are derivable from the image's own
# build manifest, so by default the script asserts the LIVE pi version against
# the version recorded at build time — which is not a tautology: a stale
# `pi` in the ~/.pi/npm-global volume can shadow the baked one, exactly the
# way a stale npm:pi-atelier can (see the packages[] check below). Pass the
# flags when you want an assertion against a value you name yourself, which
# is what a release checklist wants.
#
# Usage: ./scripts/recreate-sanity-check.sh [--expected-version X.Y.Z]
# [--expected-image-version X.Y.Z]
# [--variant studio|plain]
# #
# Exit codes: # Exit codes:
# 0 all checks passed # 0 all checks passed
@@ -41,22 +61,61 @@
set -euo pipefail set -euo pipefail
EXPECTED_VERSION="" EXPECTED_VERSION=""
EXPECTED_IMAGE_VERSION=""
VARIANT="" VARIANT=""
REPO_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" REPO_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
MANIFEST=/etc/pi-devbox/build-manifest.json
# Parse arguments usage() {
cat >&2 <<'EOF'
usage: recreate-sanity-check.sh [--expected-version X.Y.Z]
[--expected-image-version X.Y.Z]
[--variant studio|plain]
--expected-version pi coding agent version, e.g. 0.84.3 (`pi --version`)
--expected-image-version pi-devbox image release tag, e.g. 1.8.9 or v1.8.9
--variant studio|plain (auto-detected when omitted)
These are two different versions. Both are read from the image's own build
manifest when the corresponding flag is omitted.
EOF
}
# Parse arguments. Every flag takes a value, so reject a missing one rather
# than swallowing the next flag as if it were the value.
need_value() {
case "${2:-}" in
""|-*)
echo "$1 requires a value" >&2
usage
exit 2
;;
esac
}
while [[ $# -gt 0 ]]; do while [[ $# -gt 0 ]]; do
case "$1" in case "$1" in
--expected-version) --expected-version)
need_value "$@"
EXPECTED_VERSION="$2" EXPECTED_VERSION="$2"
shift 2 shift 2
;; ;;
--expected-image-version)
need_value "$@"
EXPECTED_IMAGE_VERSION="$2"
shift 2
;;
--variant) --variant)
need_value "$@"
VARIANT="$2" VARIANT="$2"
shift 2 shift 2
;; ;;
--help|-h)
usage
exit 0
;;
*) *)
echo "usage: $0 [--expected-version X.Y.Z] [--variant studio|plain]" >&2 echo "unknown option: $1" >&2
usage
exit 2 exit 2
;; ;;
esac esac
@@ -67,6 +126,19 @@ pass() { echo " ✓ $1"; }
fail() { echo " ✗ $1" >&2; FAILED=$((FAILED + 1)); } fail() { echo " ✗ $1" >&2; FAILED=$((FAILED + 1)); }
warn() { echo " ⚠ $1" >&2; } warn() { echo " ⚠ $1" >&2; }
# Read one top-level field from the build manifest, or print nothing. The
# manifest is the image's own ground truth (written at `docker build` time by
# Dockerfile.variant), so it needs no checkout and no network. Absent on an
# image built before it existed, hence every caller treats "" as unknown.
manifest_field() {
[ -f "$MANIFEST" ] || return 0
command -v jq >/dev/null 2>&1 || return 0
jq -r --arg k "$1" '.[$k] // empty' "$MANIFEST" 2>/dev/null || true
}
# Release tags are written with a leading v in the manifest and quoted without
# one in checklists; compare on the bare number so both spellings work.
strip_v() { printf '%s' "${1#v}"; }
# Auto-detect variant if not provided. The studio variant vendors pi-studio to # Auto-detect variant if not provided. The studio variant vendors pi-studio to
# /opt/pi-studio; the plain variant does not. # /opt/pi-studio; the plain variant does not.
if [ -z "$VARIANT" ]; then if [ -z "$VARIANT" ]; then
@@ -86,21 +158,59 @@ else
fi fi
echo echo
echo "-- pi version --" MANIFEST_PI_VERSION=$(manifest_field pi_version)
MANIFEST_RELEASE_TAG=$(manifest_field release_tag)
echo "-- pi (coding agent) version --"
if ACTUAL_VERSION=$(pi --version 2>&1 | head -1); then if ACTUAL_VERSION=$(pi --version 2>&1 | head -1); then
if [ -n "$EXPECTED_VERSION" ]; then if [ -n "$EXPECTED_VERSION" ]; then
if [ "$ACTUAL_VERSION" = "$EXPECTED_VERSION" ]; then if [ "$(strip_v "$EXPECTED_VERSION")" = "$(strip_v "$ACTUAL_VERSION")" ]; then
pass "pi version $ACTUAL_VERSION" pass "pi version $ACTUAL_VERSION (matches --expected-version)"
elif [ -n "$MANIFEST_RELEASE_TAG" ] &&
[ "$(strip_v "$EXPECTED_VERSION")" = "$(strip_v "$MANIFEST_RELEASE_TAG")" ]; then
# Exact, not heuristic: the value handed over IS this image's release
# tag, so it cannot be a pi version anyone meant.
fail "--expected-version $EXPECTED_VERSION is the pi-devbox IMAGE version, not the pi version — use --expected-image-version $EXPECTED_VERSION (live pi is $ACTUAL_VERSION)"
else else
fail "pi version mismatch: expected $EXPECTED_VERSION, got $ACTUAL_VERSION" fail "pi version mismatch: expected $EXPECTED_VERSION, got $ACTUAL_VERSION (this flag asserts the pi coding agent version; for the image release tag use --expected-image-version)"
fi
elif [ -n "$MANIFEST_PI_VERSION" ]; then
# Not a tautology: the manifest records what pi reported at BUILD time,
# while `pi --version` resolves through PATH, which a stale npm-global
# volume install can shadow.
if [ "$MANIFEST_PI_VERSION" = "$ACTUAL_VERSION" ]; then
pass "pi version $ACTUAL_VERSION (matches this image's build manifest)"
else
fail "live pi $ACTUAL_VERSION != $MANIFEST_PI_VERSION recorded in $MANIFEST — a stale pi in the ~/.pi/npm-global volume is shadowing the baked one"
fi fi
else else
warn "pi version $ACTUAL_VERSION (no --expected-version given; pi is built from 'latest', cannot self-derive — informational only)" warn "pi version $ACTUAL_VERSION (no --expected-version and no build manifest to compare against — informational only)"
fi fi
else else
fail "pi --version failed" fail "pi --version failed"
fi fi
echo
echo "-- pi-devbox image version --"
if [ -z "$MANIFEST_RELEASE_TAG" ]; then
if [ -n "$EXPECTED_IMAGE_VERSION" ]; then
fail "cannot verify --expected-image-version $EXPECTED_IMAGE_VERSION: no readable release_tag in $MANIFEST (image built before the manifest existed, or jq missing)"
else
warn "image release tag unknown (no readable $MANIFEST) — pi-devbox-version would say the same"
fi
elif [ -n "$EXPECTED_IMAGE_VERSION" ]; then
if [ "$(strip_v "$EXPECTED_IMAGE_VERSION")" = "$(strip_v "$MANIFEST_RELEASE_TAG")" ]; then
pass "image version $MANIFEST_RELEASE_TAG (matches --expected-image-version)"
elif [ -n "$MANIFEST_PI_VERSION" ] &&
[ "$(strip_v "$EXPECTED_IMAGE_VERSION")" = "$MANIFEST_PI_VERSION" ]; then
fail "--expected-image-version $EXPECTED_IMAGE_VERSION is the pi version, not the image release tag — use --expected-version $EXPECTED_IMAGE_VERSION (this image is $MANIFEST_RELEASE_TAG)"
else
fail "image version mismatch: expected $EXPECTED_IMAGE_VERSION, got $MANIFEST_RELEASE_TAG — the recreate did not pick up the intended image"
fi
else
warn "image version $MANIFEST_RELEASE_TAG (no --expected-image-version given — informational only)"
fi
echo echo
echo "-- Persisted named volumes (must survive --force-recreate) --" echo "-- Persisted named volumes (must survive --force-recreate) --"
+27 -3
View File
@@ -86,7 +86,13 @@ for arg in "$@"; do
case "$arg" in case "$arg" in
--check) MODE="check" ;; --check) MODE="check" ;;
--force) FORCE=1 ;; --force) FORCE=1 ;;
--*) die "unknown option: $arg" ;; # This is the one script whose argument ORDER was itself a landmine, so the
# path that documents the trap must not be the path that errors.
-h|--help)
awk 'NR>1 && /^#/ { sub(/^# ?/, ""); print; next } NR>1 { exit }' "$0"
exit 0
;;
--*) die "unknown option: $arg (try --help)" ;;
*) *)
[ -z "$ROOT" ] || die "unexpected extra argument: $arg (root already set to $ROOT)" [ -z "$ROOT" ] || die "unexpected extra argument: $arg (root already set to $ROOT)"
ROOT="$arg" ROOT="$arg"
@@ -110,6 +116,12 @@ fi
[ -d "$ROOT/.git" ] || die "not a git clone: $ROOT" [ -d "$ROOT/.git" ] || die "not a git clone: $ROOT"
[ -f "$ROOT/$REL_PATH" ] || die "no $REL_PATH in $ROOT" [ -f "$ROOT/$REL_PATH" ] || die "no $REL_PATH in $ROOT"
[ -f "$VENDORED" ] || die "vendored snapshot missing: $VENDORED" [ -f "$VENDORED" ] || die "vendored snapshot missing: $VENDORED"
# LOAD-BEARING, DO NOT DELETE AS "REDUNDANT WITH THE EXISTENCE PROBES": -f
# accepts an empty file, and sha256 of an empty file equals sha256 of a failed
# pipeline's empty stdin. Guarding it HERE, before mode dispatch, makes that
# collision unreachable by construction rather than by a probe further down --
# which also means no test below exercises the collision any more. Remove this
# line and the false "OK" for a nonexistent ref returns with nothing failing.
[ -s "$VENDORED" ] || die "vendored snapshot is empty: $VENDORED" [ -s "$VENDORED" ] || die "vendored snapshot is empty: $VENDORED"
head_sha=$(git -C "$ROOT" rev-parse HEAD 2>/dev/null) || die "cannot read HEAD of $ROOT" head_sha=$(git -C "$ROOT" rev-parse HEAD 2>/dev/null) || die "cannot read HEAD of $ROOT"
@@ -200,8 +212,20 @@ if [ "$MODE" = "check" ]; then
# clean and the ref simply moved on — a message that names the wrong cause # clean and the ref simply moved on — a message that names the wrong cause
# is the same defect class as a canary pinned to a deleted phrase. # is the same defect class as a canary pinned to a deleted phrase.
if [ "$recorded" != "$head_sha" ] && [ "$blob_sha" = "$upstream_sha" ]; then if [ "$recorded" != "$head_sha" ] && [ "$blob_sha" = "$upstream_sha" ]; then
printf 'NOTICE: %s has moved to %s; the snapshot describes the older %s (stale, not untruthful)\n' \ # Do not ASSERT which side is newer — test it. Asserting that HEAD is the
"$ROOT" "${head_sha:0:7}" "${recorded:0:7}" >&2 # newer side points the operator at a refresh (which costs a ~67-minute
# base rebuild) when the real remedy may be `git pull` in this clone. The
# refresh path below already uses this primitive; reuse it here.
if git -C "$ROOT" merge-base --is-ancestor "$recorded" "$head_sha" 2>/dev/null; then
printf 'NOTICE: %s has moved on to %s; the snapshot describes the older %s (stale, not untruthful — refresh to catch up)\n' \
"$ROOT" "${head_sha:0:7}" "${recorded:0:7}" >&2
elif git -C "$ROOT" merge-base --is-ancestor "$head_sha" "$recorded" 2>/dev/null; then
printf 'NOTICE: %s is BEHIND at %s; the snapshot describes the newer %s — pull this clone, do NOT refresh the snapshot\n' \
"$ROOT" "${head_sha:0:7}" "${recorded:0:7}" >&2
else
printf 'NOTICE: %s (HEAD %s) and the recorded %s have DIVERGED — neither is an ancestor of the other; reconcile the clone before refreshing\n' \
"$ROOT" "${head_sha:0:7}" "${recorded:0:7}" >&2
fi
else else
printf 'NOTICE: the working tree of %s differs from the snapshot (HEAD %s)\n' \ printf 'NOTICE: the working tree of %s differs from the snapshot (HEAD %s)\n' \
"$ROOT" "${head_sha:0:7}" >&2 "$ROOT" "${head_sha:0:7}" >&2