Compare commits
6 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 joakimp
|
6891dc32b8 | ||
 Joakim Persson
|
8a673ec143 | ||
|
joakimp
|
cdb6fc0950 | ||
|
Joakim Persson
|
14371e2da6 | ||
|
joakimp
|
aac4a1c323 | ||
|
joakimp
|
34cf1e3810 |
@@ -98,9 +98,23 @@ re-brand of opencode-devbox's `pi-only` variant.
|
||||
**post-recreate sanity check** inside the running container to confirm
|
||||
persisted volumes survived and the pi runtime wiring re-deployed (not just
|
||||
that the container booted):
|
||||
`docker compose exec devbox bash scripts/recreate-sanity-check.sh --expected-version X.Y.Z`
|
||||
(or just `pi-devbox-sanity --expected-version X.Y.Z` if `cli_utils/bin` is
|
||||
on PATH). This is the runtime peer of the build-time `smoke-test.sh` gate.
|
||||
`docker compose exec devbox bash scripts/recreate-sanity-check.sh --expected-image-version X.Y.Z`
|
||||
(or just `pi-devbox-sanity --expected-image-version X.Y.Z` if
|
||||
`cli_utils/bin` is on PATH). This is the runtime peer of the build-time
|
||||
`smoke-test.sh` gate.
|
||||
**`X.Y.Z` here is the pi-devbox release tag** you are shipping (e.g.
|
||||
`1.8.9`), which is what the rest of this checklist means by `vX.Y.Z`.
|
||||
`--expected-image-version` is the flag that asserts it. There is also an
|
||||
`--expected-version`, and it means something else — the **pi coding agent**
|
||||
version (e.g. `0.84.3`, the `ARG PI_VERSION` pin). Handing the release tag
|
||||
to that one used to report *"pi version mismatch: expected 1.8.8, got
|
||||
0.84.3"*, i.e. a red on the final gate of the release accusing the wrong
|
||||
component; it now tells you to use `--expected-image-version` instead, and
|
||||
the reverse mix-up is caught too. Both flags are optional — with neither,
|
||||
the live pi version is asserted against the version recorded in the image's
|
||||
own build manifest (which catches a stale `pi` in the `~/.pi/npm-global`
|
||||
volume shadowing the baked one) and the image tag is reported
|
||||
informationally.
|
||||
5. Push tag: `git tag vX.Y.Z && git push origin vX.Y.Z`.
|
||||
6. Watch CI: smoke job builds amd64 only and asserts size + extensions +
|
||||
pi version + new-base-tooling presence. Variant build is multi-arch
|
||||
|
||||
+535
-19
@@ -11,7 +11,486 @@ Pre-v1.0.0 tags followed the pi npm version (`v{pi_version}[letter]`).
|
||||
|
||||
---
|
||||
|
||||
## Unreleased
|
||||
## v1.8.10 — 2026-08-27
|
||||
|
||||
**This tag exists to deploy a fix and a safety net that are currently running on
|
||||
exactly one machine.** The feeder scrubber has been hand-copied to `/opt` on one
|
||||
device since this morning; every other device has kept staging unscrubbed
|
||||
transcripts into the shared palace. Nothing here is a new capability for its own
|
||||
sake.
|
||||
|
||||
`MEMPALACE_TOOLKIT_REF=main` floats: `docker-publish.yml` resolves it to a
|
||||
concrete SHA at build time, so whatever is on toolkit `main` when the tag is
|
||||
pushed ships in that image whether or not this repo has a commit. That is the
|
||||
rule v1.8.9 adopted after `553d865`/`5b8d78f` shipped undocumented twice — *name
|
||||
the behaviour change before tagging, not after* — and this entry is that rule
|
||||
being obeyed rather than re-learned.
|
||||
|
||||
**`mempalace-toolkit` main moves `5b8d78f` → `b2b50af`** (13 commits, ~2100
|
||||
insertions / ~520 deletions). No pi-devbox commit implements any of it.
|
||||
|
||||
### ⚠️ THE ONE CHECK THIS RELEASE MUST NOT SKIP
|
||||
|
||||
**On the first client that runs this image, verify the memory feed still stages.**
|
||||
The feeder is *fail-closed* by design: no redactor module, no staging (`exit 3`).
|
||||
That is correct behaviour and it is also the failure mode with no alarm — a
|
||||
packaging or path mistake stops the fleet's entire transcript feed and nothing
|
||||
complains loudly, because refusing to stage looks exactly like a quiet session.
|
||||
|
||||
This is not hypothetical. `f0bffd1` exists because the feeder is installed as a
|
||||
symlink (`/usr/local/bin/mempalace-pi-session` → `/opt/mempalace-toolkit/bin/…`)
|
||||
and `${BASH_SOURCE[0]}` reports the *symlink* path, so the module lookup landed in
|
||||
a directory where it does not exist. Had that shipped, every device would have
|
||||
refused to stage on first boot. It was caught by execution, not by review.
|
||||
|
||||
Acceptance, in order, on the first recreated client:
|
||||
|
||||
1. Run a session, then confirm the feeder logged a scrub summary — a
|
||||
`[scrub]` line with tier-tagged counts (`T1:env-value=…`, `T2:github-pat=…`),
|
||||
or an explicit "zero redactions". **Silence is the failure signal**, not success.
|
||||
2. Confirm the palace drawer count *moved* for that session (the feed reached the
|
||||
server, not just the stager).
|
||||
3. Confirm `exit 3` did **not** fire: `mempalace-pi-session` invoked through the
|
||||
`/usr/local/bin` symlink must find `mempalace_redact.py`.
|
||||
4. Only then trust the rest of this release.
|
||||
|
||||
If step 1 or 3 fails, the memory feed is down fleet-wide until it is fixed, and
|
||||
sessions that ran in the meantime are not recoverable from the palace — they were
|
||||
never staged. `MEMPALACE_FEED_ALLOW_UNSCRUBBED=1` is the loud escape hatch, and
|
||||
using it means accepting unscrubbed transcripts until the packaging is repaired.
|
||||
|
||||
### Dependency audit (2026-08-27)
|
||||
|
||||
Every component checked against upstream, not assumed:
|
||||
|
||||
| Component | Baked in v1.8.9 | Upstream now | Action |
|
||||
|---|---|---|---|
|
||||
| **mempalace-toolkit** | `5b8d78f` | **`b2b50af`** | ships the scrubber + symlink fix + `hlc` join |
|
||||
| **pi-studio** (studio variant) | `v0.9.48` | **`v0.9.52`** | 22 commits, additive only — see below |
|
||||
| pi | `0.84.3` (pinned) | `0.84.3` is npm latest | none |
|
||||
| mempalace | `3.8.0` (pinned) | `3.8.0` is PyPI latest | none |
|
||||
| pi-atelier | `v0.8.2` (pinned) | `v0.8.2` highest tag | none |
|
||||
| playwright | `1.62.1` (floats `latest`) | `1.62.1` | none — no drift this cycle |
|
||||
| pi-fork | `bf702b4` | `bf702b4` (2026-08-24) | none |
|
||||
| pi-observational-memory | `ce9fc98` (v3.0.4) | `ce9fc98` | none |
|
||||
| pi-toolkit | `0e1369e` | `0e1369e` (2026-08-07) | none |
|
||||
| pi-extensions | `2022887` | `2022887` (2026-08-17) | none |
|
||||
|
||||
**`pi-studio` `v0.9.48` → `v0.9.52`** — four releases, 22 commits, all additive:
|
||||
PDFs open directly in Studio with watched previews, the header can hide, and
|
||||
contextual *side questions* arrive (selected-tool use, frozen git context, export,
|
||||
keyboard shortcuts). No removals or renames in the diff; the changes are
|
||||
concentrated in `client/studio-client.js`, `index.ts` and three new `shared/`
|
||||
helpers.
|
||||
|
||||
**Its `pi` floor is `>=0.84.3` and we pin exactly `0.84.3` — satisfied with zero
|
||||
headroom.** Worth naming as a watch item rather than a problem: the next studio
|
||||
release that raises the floor breaks the studio variant until `PI_VERSION` moves,
|
||||
and that failure surfaces at build time in the studio job only, after the core
|
||||
variant has already published.
|
||||
|
||||
### Also pulled in by the floating toolkit ref (documentation only)
|
||||
|
||||
RFC 003 gains **§9.2**, a proposed direction for the one open decision this
|
||||
fleet keeps tripping over — that a report addressed to a device is never
|
||||
delivered, because mailbox candidacy requires exactly `status="open"`. It records
|
||||
a negative result worth keeping: widening the owed set to include terminal events
|
||||
cannot work, since the asserting shape and the clearing shape must be disjoint or
|
||||
every closure mints a fresh obligation. No code implements §9.2 in this release.
|
||||
|
||||
The skillset snapshot also moves, so this image bakes the mermaid-diagrams skill's
|
||||
Playwright driver and the honest note that a `claimed` ack notifies nobody.
|
||||
|
||||
### Transcripts get scrubbed before they are staged (`3d47937`, `836e35b`, `f0bffd1`)
|
||||
|
||||
`bin/mempalace_redact.py`, called from `mempalace-pi-session` at the moment the
|
||||
staged transcript is written — one hook covering both transports, because local
|
||||
mode mines that file and remote mode rsyncs the same bytes.
|
||||
|
||||
- **Why it exists, measured rather than argued.** One leaked bearer token had
|
||||
reached 3 drawers, 13 feeder inbox files across all three devices, and 10 local
|
||||
files spanning 10 days, from an agent printing an env var while debugging. A
|
||||
second sweep then found `GITEA_ACCESS_TOKEN` in 2 more drawers and
|
||||
`GITEA_EGL_ACCESS_TOKEN` in 3. This is routine agent behaviour, so the fix
|
||||
belongs in the pipeline, not in discipline.
|
||||
- **Detection is name-anchored, never entropy-anchored.** A palace's own primary
|
||||
keys — drawer ids, chunk ids, event ids, replica ids, HLCs, commit SHAs — *are*
|
||||
its high-entropy strings, so an entropy detector eats the memory it protects,
|
||||
silently and unrecoverably. Three tiers instead: T1 literal values from this
|
||||
process's env whose name says secret (zero false positives by construction);
|
||||
T2 vendor shapes (`ghp_`, `glpat-`, `xox*-`, `sk-`, `AKIA`, JWT, PEM, URL
|
||||
credentials, `Authorization:`); T3 key-name-says-secret.
|
||||
- **T3 is report-only, because the false-positive rate was measured.** On 52 MB
|
||||
of real fleet transcripts T3 fired 403 times, mostly `${VAR}` interpolation in
|
||||
compose files, TypeScript identifiers, a *type annotation*
|
||||
(`credentials: Credentials`), an IPA attribute holding a date
|
||||
(`krbPasswordExpiration`), AAAK diary shorthand, and terminal output following
|
||||
an ssh `Password:` prompt. With interpolation/code-context/key-suffix guards the
|
||||
enforced count fell **403 → 29** on the same corpus. `MEMPALACE_REDACT_STRICT=1`
|
||||
makes T3 enforce.
|
||||
- **Operational shape.** Fail closed — no redactor, no staging (`exit 3`),
|
||||
overridable with `MEMPALACE_FEED_ALLOW_UNSCRUBBED=1`. Every run prints a count
|
||||
*including* `0 redaction(s)`, because silence is indistinguishable from a
|
||||
scrubber that never ran. Findings carry rule, label, length and `sha256[:8]` —
|
||||
never the value.
|
||||
|
||||
**Near-miss this image would have shipped, caught before tagging (`f0bffd1`).**
|
||||
The image installs `/usr/local/bin/mempalace-pi-session` as a **symlink** into
|
||||
`/opt/mempalace-toolkit/bin`, and `${BASH_SOURCE[0]}` reports the invoked path,
|
||||
not the target — so the sibling-module lookup resolved to `/usr/local/bin`, the
|
||||
redactor was absent, and fail-closed did as instructed: `[FATAL] ... refusing to
|
||||
stage`. Measured side by side, the symlinked invocation FATALed while the direct
|
||||
one scrubbed 40 findings. **At the next bake that would have stopped every
|
||||
feeder tick on every device — a silent fleet-wide memory outage, worse than the
|
||||
leak the scrubber prevents.** Fixed by chasing the symlink chain in portable
|
||||
shell (`readlink -f` avoided: GNU/newer-BSD only, and this script also runs
|
||||
directly on macOS hosts) with colon-separated fallback candidates. Verified via
|
||||
the symlink, the direct path, and a second-hop symlink. General lesson: fail-closed
|
||||
converts "module not found" into an outage, which makes the module lookup
|
||||
load-bearing infrastructure that must be tested through the invocation path the
|
||||
fleet actually uses — not the convenient one from a checkout.
|
||||
|
||||
### The mailbox becomes explainable and mesh-safe (`bfe9c5c`, `a92c75d`, `e917662`, `ecc2a9c`)
|
||||
|
||||
- **Owed-set derivation joins on `hlc`, not `seq`** (`bfe9c5c`). `seq` is a
|
||||
replica-local arrival counter — the same event is `#7` in one database and `#12`
|
||||
in another — so a second replica would let already-answered asks resurrect.
|
||||
`hlc` is immutable and replicated, fixed-width, so string comparison *is* causal
|
||||
comparison. A safe no-op on today's single replica (verified: the positive-control
|
||||
pair orders identically under both keys), correct once a mesh exists.
|
||||
- **Delivered text now says it is queued** (`a92c75d`). Delivery uses `steer`
|
||||
with no `triggerTurn`, and the poll fires on `agent_settled`, so nothing wakes
|
||||
the model — a delivered ask sits until a human starts the next turn. Measured
|
||||
case: a directed report sat unread for 2.5 hours. The note explains the agent is
|
||||
not ignoring the ask, it is not running.
|
||||
- **`MEMPALACE_MAILBOX_NOTIFY` gains explicit `=kitty` / `=osc777` modes**
|
||||
(`e917662`). Terminal autodetection inside a container is not unreliable, it is
|
||||
*blind*: `docker exec` forwards neither `KITTY_WINDOW_ID` nor `TERM_PROGRAM`, and
|
||||
`TMUX` is unset because tmux runs on the host. Verified on a live process:
|
||||
`TERM=xterm-256color` and nothing else.
|
||||
- **The terminal path through tmux is documented as UNVERIFIED** (`ecc2a9c`).
|
||||
Test sequences written to the pty produced no notification on a remote client;
|
||||
tmux likely drops unknown OSC types without `allow-passthrough`, and multi-client
|
||||
routing (one ask pinging every attached client) is an open question.
|
||||
|
||||
### Documentation (`e1cc759`, `982b001`, `d4d8bb6`, `d2764bf`)
|
||||
|
||||
- **RFC 003, the coordination-log spec the code had been citing all along** — it
|
||||
did not exist anywhere. 11 sections, retrospective against mempalace 3.8.0
|
||||
`logstream.py`, incl. owed-set derivation, ten dogfooded landmines and seven open
|
||||
decisions. Non-obvious findings: `event_append` has **no** idempotency guard on
|
||||
the write path (verify-before-retry; the replication path *is* guarded),
|
||||
coordination tools are exempt from both palace locks by design,
|
||||
`GET /logstream/events` **never existed** in 3.8.0 (not proxy-blocked), and
|
||||
`mempalace sync` never touches the logstream — the log is permanent and unbounded.
|
||||
- **`docs/fleet-memory.md`**, operator-facing: five storage types, a decision tree,
|
||||
latency expectations (~2–5 min live session; next session while offline),
|
||||
broadcast exclusion by design, fan-out, and the search-before-answer /
|
||||
diary-at-session-end / verify-don't-retry habits.
|
||||
- **`docs/secret-hygiene.md`**, incl. the tier definitions, the measured FP data,
|
||||
stated false negatives, and the three server-side call sites (specified, not
|
||||
built — tier 2 only there, since the hub cannot see a client's env).
|
||||
- Phase 1 exposure record moved to the private fleet repo with a moved-note stub;
|
||||
retention direction for the unbounded log (logrotate-style: never rotate
|
||||
still-owed events, rotation invalidates held cursors, archive-verify-delete).
|
||||
|
||||
### The other memory system finally gets explained — `docs/observational-memory.md`
|
||||
|
||||
`pi-observational-memory` has been baked for several releases and described in
|
||||
one line of the feature list (*"the `recall` tool for session compaction"*),
|
||||
which is enough to name it and not nearly enough to use it. New 272-line
|
||||
explainer with five diagrams, aimed at someone who has seen `/om:status` or a
|
||||
"compacted memory" block and wondered whether to leave any of it switched on.
|
||||
|
||||
**Scoped to what this repo is authoritative for, because upstream already
|
||||
documents the mechanism well.** `/opt/pi-observational-memory/docs/` ships
|
||||
`concepts.md`, `how-it-works.md` and `configuration.md`, including a correct v3
|
||||
lifecycle diagram — so the new document links those for depth and spends its own
|
||||
words on the four facts pi-devbox owns and can change: the pinned commit it bakes
|
||||
(v3.0.4 `ce9fc98`, the value in `build-manifest.json`), the `packages[]` entry
|
||||
that decides which copy loads, the Haiku-workers-vs-Opus-session split seeded
|
||||
into `~/.pi/agent/settings.json`, and the `devbox-pi-config` volume that makes
|
||||
the ledger survive `--force-recreate`. Plus the confusion this image creates by
|
||||
shipping two things called memory: a section contrasting it with MemPalace, on
|
||||
the line *observational memory keeps a session coherent, the palace keeps the
|
||||
fleet coherent*.
|
||||
|
||||
Every stated number was read out of the live container or the baked tree rather
|
||||
than copied from release notes — including the correction that the dropper is
|
||||
gated on a **successful same-turn reflection** and not on a token threshold of
|
||||
its own, which is the one detail `pi-extensions/SKILL.md` still gets wrong.
|
||||
|
||||
Placement follows the audience split fleet-ops states for itself: reusable
|
||||
mechanism is not deployment data, so a "why is this in my container" document
|
||||
belongs in the repo that **pins and wires** the component, pointing upstream for
|
||||
depth. Linked twice from the README, because before this commit the README
|
||||
referenced `docs/` zero times and the one file already there
|
||||
(`mempalace-broker-design.md`) was reachable only by listing the directory.
|
||||
|
||||
### A README claim that v1.8.9 made false, and how it got there
|
||||
|
||||
**§ Cross-machine agent coordination ended with "Nothing in this image polls the
|
||||
log on the agent's behalf." The mailbox shipped in v1.8.9, so that sentence has
|
||||
been wrong since `aac4a1c`.** Replaced with the three knobs and their defaults
|
||||
(`MEMPALACE_MAILBOX`, `MEMPALACE_MAILBOX_POLL_MS` 300000,
|
||||
`MEMPALACE_MAILBOX_RESURFACE_MS` 3600000), the fact that owed-ness is *derived*
|
||||
rather than read off `status`, and the queued-into-the-next-turn delivery
|
||||
semantics measured on two devices.
|
||||
|
||||
The cause is the one v1.8.9 wrote a rule about: the behaviour arrived through the
|
||||
floating `MEMPALACE_TOOLKIT_REF`, so **no diff in this repo ever touched the
|
||||
paragraph that made the claim**. v1.8.9's rule ("name a floating-ref behaviour
|
||||
change in the CHANGELOG before tagging") fired for the CHANGELOG and nobody swept
|
||||
the README. The CHANGELOG records what *changed*; the README asserts what is
|
||||
*true*, and only the first is reviewed at release time. Extending the rule
|
||||
accordingly: grep the README for absolute claims — *nothing*, *never*, *does
|
||||
not*, *only* — about any component whose SHA moved.
|
||||
|
||||
**The replacement is dated on purpose.** It says it describes the bridge *as baked
|
||||
in v1.8.9* (`mempalace-toolkit` `5b8d78f`) and points at that repo's
|
||||
`docs/rfc-003-coordination-log.md` §7.11–§7.12 for the mechanism, because toolkit
|
||||
main is already ahead of the baked copy (`a92c75d` makes delivery say it is queued
|
||||
and ping the human who is not looking; `e917662` and `ecc2a9c` refine that notify
|
||||
path) and none of it reaches a container until a base rebuild. Documenting those
|
||||
here would have swapped a stale-behind claim for a stale-ahead one — the same
|
||||
defect with the sign flipped.
|
||||
|
||||
### Diagrams verified by rendering, not by parsing
|
||||
|
||||
Both comparison diagrams **parsed clean and rendered with their meaning
|
||||
reversed**: Mermaid laid the second declared `subgraph` out first, so "with
|
||||
observational memory" appeared before "without", and MemPalace before
|
||||
observational memory in the diagram whose entire job was that contrast. A third
|
||||
was legible only at 1280px. Rebuilt as declaration-ordered node chains, then
|
||||
re-rendered at mermaid@11 — the version `pi-studio` pins — in the baked headless
|
||||
browser and read back as an image. Recorded because it generalises:
|
||||
`mermaid.parse()` proves syntax and says nothing about layout, so a diagram is
|
||||
unverified until someone has looked at it.
|
||||
|
||||
### … and rendering it in *my* browser was still not enough
|
||||
|
||||
Reported from a real viewer: several boxes had their bottom line of text sliced
|
||||
off. Reproduced and root-caused rather than nudged — **Mermaid measures a node
|
||||
label with its own font metrics, computes the box, then renders the label as real
|
||||
HTML inside a `<foreignObject>`.** Any host stylesheet that touches the
|
||||
`line-height` or `font-size` of that HTML makes the text taller than the box
|
||||
already committed to, and the overflow is clipped at the box edge. Error
|
||||
accumulates per line, so the loss always lands on the last line of the tallest
|
||||
labels — which is exactly what was reported.
|
||||
|
||||
Two fixes were tried and only the second works:
|
||||
|
||||
- `%%{init: {'flowchart': {'htmlLabels': false}}}%%` — **rejected, and verified
|
||||
ineffective rather than assumed so.** The directive *is* honoured (label
|
||||
elements switch from 16 `foreignObject` to 7 `tspan`), and the clipping is
|
||||
identical, because the inflated font-size still inherits into SVG text.
|
||||
- **A hard limit of two short lines per node, with the detail moved into the prose
|
||||
under each diagram.** One- and two-line boxes have enough vertical slack to
|
||||
absorb the inflation; three- and four-line boxes do not. This is also better
|
||||
documentation — the old nodes were carrying paragraph-sized text.
|
||||
|
||||
The regression harness is now the interesting artefact: render every block with a
|
||||
deliberately inflated `line-height: 1.7 !important` on the label HTML, screenshot,
|
||||
and read it. Two survivors of the rewrite were caught only by that harness — a
|
||||
long unbreakable `/opt/pi-observational-memory` path silently wrapping to a third
|
||||
line, and a cylinder (`[( )]`) shape, whose curved bottom leaves less room than a
|
||||
rectangle for the same two lines.
|
||||
|
||||
### §4 answers the question the document left hanging: what compaction does to your context
|
||||
|
||||
Asked directly and worth writing down: *if the old conversation is folded away, is
|
||||
the session back to knowing nothing?* No — and the specifics are all checkable
|
||||
against pi 0.84.3's own `docs/compaction.md` and the extension's source:
|
||||
|
||||
- **A verbatim tail survives, sized by a token budget rather than a message
|
||||
count.** Pi walks back from the newest entry until `keepRecentTokens` [20000],
|
||||
and everything from that `firstKeptEntryId` onward is kept **unchanged**. Cut
|
||||
points land on turn boundaries, never mid-tool-call.
|
||||
- **The system prompt and `AGENTS.md` are not in the compacted region at all** —
|
||||
they are rebuilt from disk on every request, so compaction cannot lose them.
|
||||
- **Nothing is deleted from disk.** Compaction *appends* a `compaction` entry
|
||||
carrying the summary and the cut pointer; no session line is rewritten in place.
|
||||
- **`recall` therefore still resolves ids whose sources left the context**, because
|
||||
it reads the full branch via `sessionManager.getBranch()` and never consults the
|
||||
context window.
|
||||
- **Repeated compaction does not summarise the summary.** The text is always
|
||||
rendered from live observation/reflection records, so there is no
|
||||
generation-loss spiral; the projection is incremental against the last full-fold
|
||||
boundary and escalates to a true re-fold from the branch root at
|
||||
`observationsPoolMaxTokens` [20000].
|
||||
|
||||
And one correction to this repo's own earlier claim: **"compaction calls no model"
|
||||
is a steady-state property, not an absolute.** If the ledger is empty — compaction
|
||||
firing before the observer has ever run — the hook returns nothing and explicitly
|
||||
declines ownership (`// Decline ownership so Pi's native summarizer preserves the
|
||||
pre-cut context.`), and pi's own model-based summariser runs. The doc now says so,
|
||||
with the snippet.
|
||||
|
||||
### A shipped doc bug: the ledger entry type was stated exactly backwards
|
||||
|
||||
§9 told readers the entries are `custom_message` and specifically *not* `custom`.
|
||||
It is the other way round, so the one grep the section existed to get right was
|
||||
the one it got wrong. Corrected against the live session file — 11
|
||||
`om.observations.recorded` and 6 `om.reflections.recorded` entries, all
|
||||
`"type":"custom"`, alongside `"type":"custom_message"` entries whose `customType`
|
||||
is `mempalace-mailbox` and `mempalace-wakeup`, which is precisely where the
|
||||
confusion came from: **the mailbox uses the context-visible API, om's ledger uses
|
||||
the invisible one.**
|
||||
|
||||
That is not a typo but a load-bearing distinction, and the fix turns it into a
|
||||
feature the doc now advertises: `custom` entries *"do not participate in LLM
|
||||
context"* (pi `docs/session-format.md`), so **the ledger costs zero context until
|
||||
it is folded** — now a row in the cost table.
|
||||
### Not covered by any of this
|
||||
|
||||
The opencode bridge is a separate write path the feeder hook never sees, and the
|
||||
server-side layer is unbuilt — so a secret typed straight into `add_drawer`, or
|
||||
staged by a non-pi client, still lands unscrubbed.
|
||||
|
||||
---
|
||||
|
||||
## v1.8.9 — 2026-08-26
|
||||
|
||||
The coordination log gets a reader, and the release checklist's last gate stops
|
||||
accusing the wrong component.
|
||||
|
||||
### The mailbox arrives — named here *because nothing in this repo caused it*
|
||||
|
||||
**`mempalace-toolkit` main moves `e70bef2` → `5b8d78f` (exactly one commit, 281
|
||||
insertions / 9 deletions across `extensions/pi/mempalace.ts` and
|
||||
`extensions/pi/README.md`), and that is what actually ships the auto-delivered
|
||||
logstream mailbox.** No pi-devbox commit implements it. `docker-publish.yml`
|
||||
resolves `MEMPALACE_TOOLKIT_REF=main` to a concrete SHA at build time and folds
|
||||
that SHA into `base_tag`, so the mailbox would have landed in the next tagged
|
||||
image **whether or not this section existed** — which is precisely why it exists.
|
||||
That is the same shipped-undocumented shape as `553d865` in v1.8.7, and that one
|
||||
caused a cross-host misattribution: an agent on another machine reasoned about
|
||||
which image contained which behaviour from a CHANGELOG that never mentioned it.
|
||||
The rule this release adopts: **if a floating ref will pull a behaviour change
|
||||
into the image, name it in the CHANGELOG before tagging, not after.**
|
||||
|
||||
What the mailbox does, from the shipped code rather than from the design
|
||||
discussion:
|
||||
|
||||
- **The bridge was write-only.** It stamped provenance on the way *out* and never
|
||||
read the log back, so a directed ask reached an agent only if that agent
|
||||
happened to run `mempalace_event_list` itself. The channel carried real
|
||||
cross-machine traffic from 2026-08-18 onward with **zero readers** — every
|
||||
delivery in that window happened because a human said "check your mailbox".
|
||||
- **Doubly gated, exactly like the provenance stamper:** inert unless *both*
|
||||
`MEMPALACE_PI_DEVICE` and `MEMPALACE_REMOTE_URL` are set. An unstamped client
|
||||
has no address to be reached at, so there is nothing for it to read.
|
||||
- **On by default, opt out with `MEMPALACE_MAILBOX=0`.** Deliberate: an opt-in
|
||||
fix for a nobody-remembers-to-do-it problem only relocates the forgetting.
|
||||
Tunables: `MEMPALACE_MAILBOX_POLL_MS` (min gap between mid-session polls,
|
||||
default 300000) and `MEMPALACE_MAILBOX_RESURFACE_MS` (re-announce a still-owed
|
||||
ask after, default 3600000).
|
||||
- **Owed-ness is derived, never read off `status`.** `event_ack` appends and never
|
||||
mutates, and `status` is written once, so a directed `open` keeps matching the
|
||||
mailbox query forever — answered or not. A candidate counts as answered only
|
||||
when one of this device's own events has a **higher `seq`**, joins via
|
||||
`metadata.ack_of` or a shared `correlation_id`, and carries a terminal status
|
||||
(`applied`, `superseded`, `failed`, `blocked`). `claimed` and `ready` are
|
||||
deliberately **not** terminal — that is how "taken, but not finished" keeps
|
||||
resurfacing.
|
||||
- **`*` broadcasts are excluded from the owed set.** `to_agent: <me>` also matches
|
||||
broadcasts per the tool contract, so without this a broadcast written with
|
||||
`status="open"` would make every machine believe it personally owed the same
|
||||
answer — and the code would contradict the skill that documents it.
|
||||
- **The dedup map is in memory on purpose.** A restart forgets, so an already-seen
|
||||
ask can resurface: visible noise a human corrects in one turn. The opposite
|
||||
failure — suppressing an unanswered ask — is silent and permanent. Do not
|
||||
"fix" the noise by persisting it.
|
||||
- **Delivery queues, it never interrupts.** A sections push at
|
||||
`before_agent_start` plus a second `agent_settled` handler behind the 300 s
|
||||
floor, using `steer` and *not* `triggerTurn`: `agent_settled` means idle, so
|
||||
nothing wakes a model on inbound fleet traffic.
|
||||
|
||||
Measured on v1.8.8 (which bakes `e70bef2`, i.e. no mailbox) immediately before
|
||||
this release: the wake-up mailbox query had to be run by hand, returned **3**
|
||||
directed asks with `status="open"`, and the derivation above resolved **all
|
||||
three** as already answered — the third independent confirmation that the raw
|
||||
`status` filter never shrinks, and the first taken on a fresh container with no
|
||||
memory of having answered them.
|
||||
|
||||
### `--expected-image-version`: two versions, two flags
|
||||
|
||||
**`scripts/recreate-sanity-check.sh --expected-version 1.8.8` reported
|
||||
`✗ pi version mismatch: expected 1.8.8, got 0.84.3` and exit 1** — a red on the
|
||||
final runtime gate of a release, accusing the image of being the wrong version,
|
||||
when the flag had only ever asserted `pi --version`. `AGENTS.md` step 4 spelled
|
||||
it `--expected-version X.Y.Z` inside a checklist where every *other* `X.Y.Z` is
|
||||
the pi-devbox tag; `README.md` got it right, so the two documents disagreed.
|
||||
|
||||
Not hypothetical, and not one reader's slip: the v1.8.8 release-readiness handoff
|
||||
from `pi@emb-7kj4vr4g` (`evt_20260826T134919_a614ecfc2d4f`) propagated
|
||||
`--expected-version 1.8.8` twice, in its body and in
|
||||
`metadata.cannot_check_here`, while correctly calling step 4 "the runtime peer of
|
||||
the smoke gate, so it is not ceremonial". Two independent readers, one on another
|
||||
machine, converged on the wrong meaning. Left alone it puts a spurious red on
|
||||
every release, and the intuitive remedy — re-pull, re-recreate — is pure waste.
|
||||
|
||||
- **New `--expected-image-version X.Y.Z`** asserts the pi-devbox release tag,
|
||||
read from `release_tag` in `/etc/pi-devbox/build-manifest.json` (the image's
|
||||
own build-time ground truth — no checkout, no network, no Docker socket). A
|
||||
leading `v` is optional on either side, so `1.8.9` and `v1.8.9` both work.
|
||||
- **Both flags now detect being handed the other one's value**, and the test is
|
||||
exact rather than heuristic: the value is compared against the *other*
|
||||
quantity this image actually reports, so it can only fire when the mix-up is
|
||||
real. `--expected-version 1.8.9` now says *"is the pi-devbox IMAGE version,
|
||||
not the pi version — use `--expected-image-version`"*, and the reverse mix-up
|
||||
is caught the same way.
|
||||
- **Neither flag is required any more.** With none, the live `pi --version` is
|
||||
asserted against `pi_version` in the build manifest. That is not a tautology:
|
||||
`pi` resolves through `PATH`, and a stale install in the `~/.pi/npm-global`
|
||||
volume can shadow the baked one — the same shadowing this script already
|
||||
guards against for `npm:pi-atelier` in `packages[]`. Verified by mutating the
|
||||
manifest to a different version, which made the new check fail as intended.
|
||||
- **The header note it replaced was stale and load-bearing.** It claimed pi "is
|
||||
resolved from `latest` at CI build time and is NOT pinned … cannot self-derive
|
||||
an expected version". `Dockerfile.variant` pins `ARG PI_VERSION=0.84.3`, and
|
||||
`docker-publish.yml` *reads that ARG* as its source of truth (refusing to
|
||||
build on a floating value, checking it is published on npm, warning when npm
|
||||
is ahead). The same withdrawn claim also sat in `cli_utils`'s
|
||||
`pi-devbox-sanity --help`, the third place this confusion lived; fixed there
|
||||
too, in that repo.
|
||||
- Argument parsing hardened while in there: a flag whose value is missing — or
|
||||
is another flag — is now a usage error (exit 2) instead of silently consuming
|
||||
the next argument, and `--help` works.
|
||||
|
||||
All fourteen flag combinations were exercised by execution, including the two
|
||||
manifest-absent branches and the shadowing branch, which a healthy container
|
||||
cannot reach naturally — mutation-tested with a doctored manifest path so that
|
||||
each failure branch was observed *firing* rather than assumed present.
|
||||
|
||||
### Component audit: no bumps, and that is the finding
|
||||
|
||||
Checked before tagging, since a base rebuild was already forced:
|
||||
|
||||
| Component | In v1.8.8 | Upstream now | Action |
|
||||
|---|---|---|---|
|
||||
| pi (npm) | `0.84.3` (pinned) | `0.84.3` is `latest` | none |
|
||||
| mempalace (PyPI) | `3.8.0` (pinned) | `3.8.0` | none |
|
||||
| pi-atelier | `v0.8.2` (pinned) | `v0.8.2` highest tag | none |
|
||||
| pi-toolkit, pi-extensions, pi-fork, pi-observational-memory, pi-studio | floating | **identical to baked** | none |
|
||||
| skillset snapshot | `6eb20af` | `6eb20af` | none |
|
||||
| **mempalace-toolkit** | `e70bef2` | **`5b8d78f`** | ships the mailbox |
|
||||
|
||||
So the whole ~67-minute base rebuild this tag pays for is attributable to the
|
||||
toolkit SHA alone — `base_tag` folds it, and it moved. Every other floating ref
|
||||
resolved to the commit already baked (verified with `git ls-remote` per repo, not
|
||||
by reading a cached clone).
|
||||
|
||||
One claim in this audit came from a fork that had fabricated its findings — six
|
||||
plausible-looking toolkit commits with five nonexistent SHAs, a pi `0.84.4` that
|
||||
npm has never published, a pi-studio commit `ls-remote` says does not exist, and
|
||||
a compatibility floor of `0.8.2` where the code says `0.7.1`. Every row above was
|
||||
therefore re-measured directly. Recorded because the failure mode is specific:
|
||||
none of it looked wrong, and `git cat-file -e` is what caught it.
|
||||
|
||||
---
|
||||
|
||||
## v1.8.8 — 2026-08-26
|
||||
|
||||
The vendored `mempalace` skill snapshot stops being anonymous, and the
|
||||
container starts saying which copy of each skill it is actually reading.
|
||||
@@ -45,6 +524,23 @@ being fixed.**
|
||||
only, `2` cannot determine (ref absent from this clone). `AGENTS.md` step 2
|
||||
rewritten to state all three, since its promise that "the message
|
||||
distinguishes the two" was exactly what the branch was breaking.
|
||||
- **The staleness `NOTICE` then asserted a direction it had never tested** — the
|
||||
same defect one layer down, found by pi@emb-7kj4vr4g against the real state of
|
||||
its own host. The branch fired the notice on "recorded ≠ HEAD" and announced
|
||||
that HEAD was the newer side, so a clone that was merely *behind* was told
|
||||
*"has moved to 82a8d3c; the snapshot describes the older 5fd0d5c"* when
|
||||
`82a8d3c` is `5fd0d5c`'s **ancestor**. Harmless to the verdict (`rc` stayed 0,
|
||||
nothing was mis-verified) but it points the operator at a refresh — a
|
||||
~67-minute base rebuild — when the real remedy is `git pull`. It now tests
|
||||
ancestry with the `merge-base --is-ancestor` primitive the refresh path two
|
||||
sections above already used, and reports three distinct verdicts: **stale**
|
||||
(recorded is an ancestor — refresh), **your clone is behind** (HEAD is an
|
||||
ancestor — pull, do not refresh), **diverged** (neither). All three verified by
|
||||
execution; only the first was right before.
|
||||
- **`--help` died with `unknown option: --help`.** The strict argument loop that
|
||||
closed the silent-ignore hole never added a `--help` case, so the one script
|
||||
whose argument *order* was itself a landmine had an erroring discoverability
|
||||
path. It now prints its own header block.
|
||||
- **`VENDORED.md` contradicted itself, in the release whose stated invariant is
|
||||
non-contradiction.** Its hand-maintained "Snapshot provenance at last refresh"
|
||||
line named skillset `670f7f1` — seven commits behind the ARG, and *the very
|
||||
@@ -88,8 +584,22 @@ earlier. Fixed in skillset `5fd0d5c`, which derives owed-ness by joining on
|
||||
terminal reply suppresses every later ask on the same thread forever. Because
|
||||
the skillset is mounted live on every enrolled host, that correction was already
|
||||
deployed fleet-wide before this image was built; the vendored snapshot is
|
||||
resynced to it (`c04cd15` → `5fd0d5c`) so the no-clone fallback does not ship
|
||||
the withdrawn rule. Canary re-verified bidirectionally against the new bytes.
|
||||
resynced to it (`c04cd15` → `5fd0d5c` → `6eb20af`) so the no-clone fallback does
|
||||
not ship the withdrawn rule. Canary re-verified bidirectionally against the new
|
||||
bytes.
|
||||
|
||||
`6eb20af` adds the limit of that ordering test, found when pi@emb-7kj4vr4g
|
||||
verified it rather than adopting it: **`seq` is replica-local.** It equals
|
||||
`origin_seq` today only because one replica authors events for all four machines,
|
||||
so a second replica could order the same pair differently and derive a different
|
||||
owed-set from the same log — use `hlc` (already on every event, total and
|
||||
causally consistent) once `mesh_peers` reports any peer. Documented as reasoning,
|
||||
not measurement, since a second replica cannot be stood up to test it. The part
|
||||
worth keeping is the **asymmetry**: local-`seq` skew makes an answered item
|
||||
*resurface* (noise, self-correcting, visible), while a timestamp comparison
|
||||
*suppresses an unanswered ask forever* (silent, permanent) — so anyone tempted to
|
||||
"fix" a resurfacing item with `created_at` would be trading the safe failure for
|
||||
the dangerous one.
|
||||
|
||||
**Also carried, previously undocumented:** `dbb7879` resynced the vendored
|
||||
`mempalace` snapshot to skillset `c04cd15` ("the withdrawal only holds where the
|
||||
@@ -275,14 +785,17 @@ the skillset actually is — a maintainer's clone, or any running container.
|
||||
because when every host is a thin client of one palace the stamped agent name
|
||||
is the only thing distinguishing them. Set both or neither: a container
|
||||
without the device var can read the log but is addressable by nobody.
|
||||
- the skillset's `mempalace` skill (`82a8d3c`, live on every host that mounts
|
||||
- the skillset's `mempalace` skill (`6eb20af`, live on every host that mounts
|
||||
the skillset, no rebuild needed) — the **norms**: a mailbox query at wake-up,
|
||||
and the sender-declared ack contract, where a *directed* event with
|
||||
`status="open"` is owed a reply and a `*` broadcast owes nothing. Measured
|
||||
while designing it: an unfiltered mailbox returned 5 events, 4 of them
|
||||
finished broadcasts from eight days earlier, where `status="open"` returned
|
||||
exactly the 1 that needed an answer — an unfiltered mailbox trains you to
|
||||
ignore it, so the filter is the feature.
|
||||
`status="open"` is owed a reply and a `*` broadcast owes nothing. The
|
||||
`status` filter earns its place by dropping broadcast noise — measured, an
|
||||
unfiltered mailbox returned 5 events, 4 of them finished broadcasts from
|
||||
eight days earlier — but that is **all** it does; it does not compute
|
||||
owed-ness, and the version of this entry that claimed otherwise is withdrawn
|
||||
above. Measured today, both machines: the raw filter returns 2 asks here and
|
||||
1 there, **every one already answered**, while the derivation returns 0 for
|
||||
both. Dropping noise and deciding what is owed are two different jobs.
|
||||
- mempalace-toolkit `extensions/pi/README.md` (`e70bef2`) — the **mechanism**,
|
||||
including that the bridge is *write-only* today (it stamps events going out
|
||||
and never reads the log, so nothing in this image polls on the agent's
|
||||
@@ -290,16 +803,19 @@ the skillset actually is — a maintainer's clone, or any running container.
|
||||
implements `GET /logstream/stream`, but a reverse proxy exposing only `/mcp`
|
||||
makes it unreachable — verified by 404s against the real endpoint.
|
||||
|
||||
⚠️ **This makes the vendored snapshot stale on purpose.** The skill edit is in
|
||||
the skillset (`82a8d3c`), so `SKILLSET_SNAPSHOT_REF` still records `c04cd15`
|
||||
and `scripts/vendor-mempalace-skill.sh --check` now exits 1 with
|
||||
*"has moved to 82a8d3c; the snapshot describes the older c04cd15"*. Refreshing
|
||||
it is a deliberate release-day decision, not an oversight — hence the new
|
||||
step 2 in `AGENTS.md` § *Release-day checklist*, which states both that the
|
||||
refresh costs a base rebuild and that skipping it is legitimate because every
|
||||
enrolled host reads its live clone. What is not legitimate is skipping it
|
||||
*silently*, which is precisely what the new manifest fields and
|
||||
`pi-devbox-version` output make impossible.
|
||||
⚠️ **The snapshot was refreshed rather than left stale.** The skill edits landed
|
||||
in the skillset (`5fd0d5c`, then `6eb20af`), so `SKILLSET_SNAPSHOT_REF` was
|
||||
resynced to match and `scripts/vendor-mempalace-skill.sh --check` is a clean
|
||||
`OK` with no notice: the no-clone fallback carries the **corrected** protocol,
|
||||
not the withdrawn one. That mattered more than currency usually does, because
|
||||
the superseded copy contained an instruction — the `mesh_peers` gate — that
|
||||
actively told a reader to skip the feature. Refreshing remains a deliberate
|
||||
release-day decision rather than an automatic one: it costs a base rebuild, and
|
||||
skipping it is legitimate because every enrolled host reads its live clone.
|
||||
What is not legitimate is skipping it *silently*, which is what the new manifest
|
||||
fields and `pi-devbox-version` output make impossible — hence step 2 in
|
||||
`AGENTS.md` § *Release-day checklist*. In this release the refresh was free:
|
||||
`rootfs/` was already changing, so the base rebuild was forced anyway.
|
||||
|
||||
---
|
||||
|
||||
|
||||
+1
-1
@@ -305,7 +305,7 @@ ARG MEMPALACE_TOOLKIT_REF=main
|
||||
# no ~67-minute base rebuild. (scripts/check-base-hash.sh scans only
|
||||
# Dockerfile.base, so no folding into the base hash is required — nor would
|
||||
# it be correct, since this ARG changes nothing about the base's contents.)
|
||||
ARG SKILLSET_SNAPSHOT_REF=5fd0d5c406df506fd0e70977b6bd87f5fbc3b803
|
||||
ARG SKILLSET_SNAPSHOT_REF=6eb20af181f0147cb8c1377f6e36a6a47a68e8e5
|
||||
|
||||
# Dockerfile.base sets description="pi-devbox — base image (variant-independent)"
|
||||
# and every variant INHERITS it, so both published images used to advertise
|
||||
|
||||
@@ -20,7 +20,9 @@ on the host.
|
||||
- `pi-extensions` — TypeScript extensions for pi (preview, MCP bridges,
|
||||
mempalace integration, etc.)
|
||||
- `pi-fork` — the `fork` tool for spawning sub-agents
|
||||
- `pi-observational-memory` — the `recall` tool for session compaction
|
||||
- `pi-observational-memory` — durable session memory: the ledger that makes
|
||||
compaction cheap, plus the `recall` tool. See
|
||||
[`docs/observational-memory.md`](docs/observational-memory.md)
|
||||
- `pi-atelier` — TUI sidebar: ordered panels, split-pane, themes. Pinned to an
|
||||
audited tag; see [Version pins](#version-pins-pi-pi-atelier-mempalace)
|
||||
|
||||
@@ -579,7 +581,50 @@ convention that a directed event with `status="open"` is a request owed a reply
|
||||
while a `*` broadcast owes nothing. The mechanism side (what the bridge stamps,
|
||||
and why live SSE push depends on the palace deployment's reverse proxy rather
|
||||
than on this image) is documented in the toolkit's `extensions/pi/README.md`.
|
||||
Nothing in this image polls the log on the agent's behalf.
|
||||
|
||||
**Since v1.8.9 the bridge reads the log for you.** Earlier images were write-only
|
||||
— they stamped provenance on the way out and never read back, so a directed ask
|
||||
reached an agent only if that agent happened to run `mempalace_event_list`
|
||||
itself. The mailbox is gated on the same two variables as the stamper, is on by
|
||||
default, and derives what is *owed* rather than trusting `status` (an acked event
|
||||
keeps matching a `status="open"` query forever, because the log is append-only):
|
||||
|
||||
| Variable | Default | Effect |
|
||||
|---|---|---|
|
||||
| `MEMPALACE_MAILBOX` | unset (on) | `0` disables mailbox reads entirely |
|
||||
| `MEMPALACE_MAILBOX_POLL_MS` | `300000` | minimum gap between mid-session polls |
|
||||
| `MEMPALACE_MAILBOX_RESURFACE_MS` | `3600000` | re-announce a still-owed ask after this long |
|
||||
|
||||
Delivery **queues, it never interrupts**: the poll runs when pi goes idle and the
|
||||
message is steered into the *next* turn, so nothing wakes the model on inbound
|
||||
fleet traffic. The practical consequence, measured on two devices: the message
|
||||
appears in your session window and the agent acts on it when the next turn
|
||||
starts — you are the trigger. (That describes the bridge **as baked in v1.8.9**,
|
||||
`mempalace-toolkit` `5b8d78f`; the mailbox's own mechanism and landmines live in
|
||||
the toolkit's `docs/rfc-003-coordination-log.md` §7.11–§7.12, which moves ahead of
|
||||
whatever this image has baked.)
|
||||
|
||||
## Observational memory (in-session memory)
|
||||
|
||||
The image also bakes [pi-observational-memory](https://github.com/elpapi42/pi-observational-memory),
|
||||
which is memory of a *different kind* from the palace and is easy to confuse with
|
||||
it. It keeps a small branch-local ledger of observations and reflections while a
|
||||
session runs, so when pi compacts the conversation the summary is a
|
||||
**deterministic fold of that ledger rather than a model call**, and every item
|
||||
keeps a 12-character id that `recall(<id>)` resolves back to the exact source.
|
||||
|
||||
In one line: **observational memory keeps a session coherent; the palace keeps
|
||||
the fleet coherent.**
|
||||
|
||||
It is on by default, needs no habit from you, and sends its background work to a
|
||||
cheaper model than your session (Haiku while the session runs Opus, in the seeded
|
||||
`~/.pi/agent/settings.json`). Inspect it from inside pi with `/om:status` and
|
||||
`/om:view`; turn all proactive work off for one run with
|
||||
`PI_OBSERVATIONAL_MEMORY_PASSIVE=1 pi`.
|
||||
|
||||
What it is for, how the lifecycle works, what it costs, every setting and its
|
||||
default, and how it differs from MemPalace:
|
||||
[`docs/observational-memory.md`](docs/observational-memory.md).
|
||||
|
||||
## Agent skills
|
||||
|
||||
@@ -1017,10 +1062,21 @@ After `docker compose up -d --force-recreate`, run the **runtime** peer of
|
||||
persisted volumes survived, and pi runtime wiring is intact:
|
||||
|
||||
```bash
|
||||
./scripts/recreate-sanity-check.sh # auto-detects variant
|
||||
./scripts/recreate-sanity-check.sh --expected-version 0.79.4 # assert pi version
|
||||
./scripts/recreate-sanity-check.sh # auto-detects variant
|
||||
./scripts/recreate-sanity-check.sh --expected-image-version 1.8.9 # assert the pi-devbox release tag
|
||||
./scripts/recreate-sanity-check.sh --expected-version 0.84.3 # assert the pi coding agent version
|
||||
```
|
||||
|
||||
Those are **two different versions**, and the flags are not interchangeable:
|
||||
`--expected-image-version` takes the pi-devbox release tag (`v` optional),
|
||||
`--expected-version` takes `pi --version`. Hand one the other's value and it
|
||||
says so by name instead of reporting a mismatch against the wrong component.
|
||||
With neither flag, both values are read from the image's own build manifest
|
||||
(`/etc/pi-devbox/build-manifest.json`): the live pi version is asserted against
|
||||
the one recorded at build time — which catches a stale `pi` in the
|
||||
`~/.pi/npm-global` volume shadowing the baked one — and the release tag is
|
||||
reported informationally.
|
||||
|
||||
If `cli_utils` is on your PATH, the `pi-devbox-sanity` wrapper runs the same
|
||||
check by short name and locates the repo automatically (override with
|
||||
`PI_DEVBOX_REPO=/path/to/pi-devbox`). Like `smoke-test.sh`, this script is
|
||||
|
||||
@@ -0,0 +1,371 @@
|
||||
# Observational memory — why this image has it, and what it does for you
|
||||
|
||||
**Audience:** anyone using this container for long pi sessions who has wondered
|
||||
what `recall`, `/om:status` and "compacted memory" are, or whether they should
|
||||
leave any of it switched on.
|
||||
|
||||
**Companion documents:** the extension ships its own reference docs at
|
||||
`/opt/pi-observational-memory/docs/` —
|
||||
[`concepts.md`](https://github.com/elpapi42/pi-observational-memory/blob/main/docs/concepts.md)
|
||||
(the model),
|
||||
[`how-it-works.md`](https://github.com/elpapi42/pi-observational-memory/blob/main/docs/how-it-works.md)
|
||||
(hooks and internals) and
|
||||
[`configuration.md`](https://github.com/elpapi42/pi-observational-memory/blob/main/docs/configuration.md)
|
||||
(every setting). Pi's own compaction mechanics are in
|
||||
`/usr/lib/node_modules/@earendil-works/pi-coding-agent/docs/compaction.md`.
|
||||
Those are normative; this document is the **deployment** view — what is pinned
|
||||
here, how it is wired, what it costs, and how it differs from MemPalace. For the
|
||||
palace, see
|
||||
[`mempalace-toolkit/docs/fleet-memory.md`](https://gitea.jordbo.se/joakimp/mempalace-toolkit/src/branch/main/docs/fleet-memory.md).
|
||||
|
||||
> Verified on pi-devbox **v1.8.9** (`release_tag v1.8.9`, source `aac4a1c`),
|
||||
> which bakes pi-observational-memory **v3.0.4** at commit `ce9fc98` — the value
|
||||
> in `/etc/pi-devbox/build-manifest.json` → `components.pi-observational-memory`.
|
||||
> Every number below was read from that tree, from pi 0.84.3's own docs, or from
|
||||
> the live container.
|
||||
|
||||
---
|
||||
|
||||
## 1. The problem it solves
|
||||
|
||||
A long pi session outgrows the model's context window. Pi's answer is
|
||||
**compaction**: fold the older part of the conversation into a summary and keep
|
||||
recent messages verbatim. That is unavoidable, and it is where sessions go
|
||||
wrong — the summary is produced *at the moment of pressure*, by a model, about a
|
||||
transcript that is about to leave the context.
|
||||
|
||||
Observational memory changes *when* the remembering happens. Instead of
|
||||
summarising in a panic at the end, it keeps a small **ledger** up to date while
|
||||
the session runs, and compaction then just folds that ledger.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A0["plain compaction"] --> A1["context fills"]
|
||||
A1 --> A2["a model summarises<br/>under pressure"]
|
||||
A2 --> A3["prose summary,<br/>no way back"]
|
||||
B0["with observational<br/>memory"] --> B1["context fills"]
|
||||
B1 --> B2["ledger written<br/>as you work"]
|
||||
B2 --> B3["compaction folds<br/>the ledger"]
|
||||
B3 --> B4["ids you can<br/>recall"]
|
||||
```
|
||||
|
||||
Top row is pi on its own: one model call at the worst possible moment, detail
|
||||
chosen in a hurry, and the original wording gone from view. Bottom row is this
|
||||
image's default: the thinking happened earlier on a cheap model, the fold is
|
||||
deterministic, and every line in the result carries an id that resolves back to
|
||||
the exact source.
|
||||
|
||||
## 2. The mental model: three layers and a ledger
|
||||
|
||||
| Layer | What it is | Example |
|
||||
|---|---|---|
|
||||
| **Observation** | a timestamped, source-backed event from the conversation | "user rejected option B because it needs a base rebuild" |
|
||||
| **Reflection** | a durable conclusion *backed by* observations | "the user optimises for avoiding 67-minute rebuilds" |
|
||||
| **Drop** | a tombstone retiring an observation from active memory | the superseded detail of a bug that is now fixed |
|
||||
|
||||
These are appended to the session as silent ledger entries
|
||||
(`om.observations.recorded`, `om.reflections.recorded`,
|
||||
`om.observations.dropped`) and **folded** — replayed in order — to produce the
|
||||
memory state. The ledger is the source of truth; what you see in a compacted
|
||||
session is a rendering of it.
|
||||
|
||||
Two properties follow, and both matter later:
|
||||
|
||||
- **The ledger itself costs no context.** Those entries are pi `custom` entries,
|
||||
which *"do not participate in LLM context"* (pi `docs/session-format.md`). They
|
||||
sit in the session file and reach the model only via the fold at compaction.
|
||||
- **Memory is branch-local.** A pi session is a tree (resume, fork), and the fold
|
||||
follows the current branch only, so a forked branch does not inherit another
|
||||
branch's view.
|
||||
|
||||
## 3. The lifecycle
|
||||
|
||||
Three background workers and one compaction hook, driven by *raw token
|
||||
progress* rather than wall-clock time. Defaults in brackets.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
T(["turn_end"]) --> O{"10k raw tokens<br/>since observing?"}
|
||||
O -- yes --> OBS["<b>observer</b> runs"]
|
||||
O -- "no" --> R{"20k tokens<br/>since reflecting?"}
|
||||
R -- yes --> REF["<b>reflector</b> runs"]
|
||||
REF -- "if pool over 10k" --> DR["<b>dropper</b> prunes"]
|
||||
S(["agent_settled"]) --> C{"81k tokens<br/>since compacting?"}
|
||||
C -- yes --> CP["ctx.compact()"]
|
||||
CP --> H(["session_before_compact"])
|
||||
H --> F["fold the ledger<br/>no model call"]
|
||||
F --> VIS["compacted memory"]
|
||||
```
|
||||
|
||||
- **observer** — `observeAfterTokens` [10000]: writes observations for the
|
||||
conversation it has not covered yet.
|
||||
- **reflector** — `reflectAfterTokens` [20000]: promotes patterns across
|
||||
observations into durable reflections.
|
||||
- **dropper** — no clock of its own. It is post-reflection maintenance, gated on
|
||||
a *successful same-turn* reflection **and** an active pool above
|
||||
`observationsPoolTargetTokens` [10000]. Not a third worker on a third
|
||||
threshold.
|
||||
- **compaction** — `compactAfterTokens` [81000], checked when pi goes idle, so it
|
||||
never interrupts a turn. Pi will also compact on its own when the context is
|
||||
nearly full (`contextTokens > contextWindow - reserveTokens`, `reserveTokens`
|
||||
[16384]).
|
||||
|
||||
## 4. What compaction actually does to your context
|
||||
|
||||
This is the question the rest of the document used to leave hanging: if the old
|
||||
conversation is folded away, is the session back to knowing nothing?
|
||||
|
||||
**No.** Compaction replaces *part* of the context, not all of it, and it deletes
|
||||
nothing at all from disk.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
SYS["system prompt<br/>+ AGENTS.md"] --> CTX["what the model sees<br/>on the next turn"]
|
||||
SUM["folded memory:<br/>reflections + observations"] --> CTX
|
||||
TAIL["recent turns,<br/>verbatim"] --> CTX
|
||||
DISK[("session .jsonl: all of it")] -. "recall(id)" .-> CTX
|
||||
```
|
||||
|
||||
Where each piece comes from:
|
||||
|
||||
- **System prompt and `AGENTS.md` — never compacted, because they were never
|
||||
conversation.** Pi rebuilds them from disk on every request
|
||||
(`loadContextFileFromDir`), so they cannot be lost by compaction.
|
||||
- **The verbatim tail — sized by a token budget, not a message count.** Pi walks
|
||||
backwards from the newest entry accumulating token estimates until
|
||||
`keepRecentTokens` [20000] is reached; that entry becomes `firstKeptEntryId`,
|
||||
and *everything from there on is kept unchanged*. Cut points land on turn
|
||||
boundaries, never mid-tool-call. So the most recent ~20k tokens of real work —
|
||||
your last instructions, the diffs, the test output — survive word for word.
|
||||
- **The folded memory — replaces only what came before that cut.** Rendered from
|
||||
the ledger's records: reflections and observations, each with its 12-hex id.
|
||||
- **The session file — untouched.** Compaction *appends* a `compaction` entry
|
||||
(`{"type":"compaction", summary, firstKeptEntryId, tokensBefore, …}`) and
|
||||
rebuilds context from it on later turns. Nothing is rewritten in place; the
|
||||
only documented way to remove session content is deleting the whole `.jsonl`.
|
||||
|
||||
That last point is what makes the answer to "is the detail gone?" *no* rather
|
||||
than *mostly*: `recall` does not read the context window at all. It calls
|
||||
`sessionManager.getBranch()` — the full branch from the root — and resolves an
|
||||
observation id back to the original entries. Detail that left the model's view
|
||||
an hour ago is still one `recall` away.
|
||||
|
||||
**Repeated compaction does not summarise the summary.** The rendered text is
|
||||
always built from live observation/reflection *records*, never from the previous
|
||||
compaction's prose, so there is no generation-loss spiral. (Mechanically the
|
||||
projection is incremental — it re-derives back to the last full-fold boundary and
|
||||
carries the rest forward, escalating to a genuine re-fold from the branch root
|
||||
when the observation pool reaches `observationsPoolMaxTokens` [20000].)
|
||||
|
||||
So the honest summary of the state after compaction: **the model keeps its
|
||||
instructions, keeps recent work verbatim, trades older turns for a dense
|
||||
id-carrying digest of them, and can pull any of it back on demand.** Not a fresh
|
||||
start — a smaller, cheaper, still-navigable one.
|
||||
|
||||
### One caveat about "no model call"
|
||||
|
||||
If the ledger is empty — compaction fires before the observer has ever run — the
|
||||
hook returns nothing and *declines ownership*, and pi's own model-based
|
||||
summariser runs instead:
|
||||
|
||||
```ts
|
||||
const summary = renderSummary(projection.reflections, projection.observations);
|
||||
if (summary.length === 0) {
|
||||
// Decline ownership so Pi's native summarizer preserves the pre-cut context.
|
||||
return;
|
||||
}
|
||||
```
|
||||
|
||||
In steady state (any session old enough to have produced one observation) om's
|
||||
hook wins and compaction is model-free. "Never calls a model" is true in practice
|
||||
and false in principle; the fallback is deliberate, so an empty ledger degrades
|
||||
to normal pi rather than to no summary at all.
|
||||
|
||||
## 5. What you actually get
|
||||
|
||||
- **Compaction stops being a stall.** In steady state the latency path is
|
||||
deterministic work over ledger entries, not a summarisation call.
|
||||
- **Nothing important vanishes silently.** Compaction is lossy by design, but
|
||||
every item keeps a 12-character id, and `recall(<id>)` returns the exact
|
||||
evidence — original wording, reasoning, file path, error text.
|
||||
- **The bookkeeping runs on a cheaper model than your session.** In this image
|
||||
that is deliberate and visible (§7): background workers on Haiku, session on
|
||||
Opus.
|
||||
- **It is automatic.** No habit to maintain, unlike the palace protocol — which is
|
||||
exactly why the two complement each other (§11).
|
||||
- **Forks stay clean.** Branch-local memory means a `fork` sub-agent's noise does
|
||||
not leak into the parent's folded memory.
|
||||
|
||||
## 6. `recall` is not a search tool
|
||||
|
||||
`recall` takes **one specific 12-hex id** that already appears in compacted
|
||||
memory or in `/om:view`. It cannot be given a topic. It can return an observation
|
||||
(marked `active` or `dropped`), or a reflection together with the observations
|
||||
supporting it.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant M as compacted memory
|
||||
participant A as agent
|
||||
participant L as ledger
|
||||
M->>A: "[high] user rejected option B (a1b2c3d4e5f6)"
|
||||
A->>L: recall("a1b2c3d4e5f6")
|
||||
L-->>A: exact observation + source ids
|
||||
Note over A: acts on the original wording
|
||||
```
|
||||
|
||||
The rule of thumb the agent skill uses: recall **before a load-bearing action**
|
||||
that rests on a compressed memory — shipping a change, asserting a fact,
|
||||
answering "why do you believe that". One recall is cheap; redoing finished work
|
||||
is not.
|
||||
|
||||
## 7. How it is wired in this image
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
IMG["baked in the image:<br/>v3.0.4 @ ce9fc98"] --> REG["settings.json<br/>packages[]"]
|
||||
REG --> SESS["your pi session"]
|
||||
SESS -- "your turns" --> SM["session model:<br/>Opus"]
|
||||
SESS -- "observer, reflector,<br/>dropper" --> WM["memory model:<br/>Haiku"]
|
||||
SESS -- "ledger entries" --> JL["session .jsonl"]
|
||||
JL --> VOL[("devbox-pi-config<br/>volume")]
|
||||
```
|
||||
|
||||
Four consequences of that wiring:
|
||||
|
||||
1. **There is no separate database.** Memory *is* entries inside the ordinary pi
|
||||
session file (`~/.pi/agent/sessions/<project>/<timestamp>_<uuid>.jsonl`).
|
||||
Nothing extra to back up, nothing to migrate.
|
||||
2. **It survives container recreate**, because `~/.pi` is the `devbox-pi-config`
|
||||
named volume (`docker-compose.yml`) — the same one holding your pi config and
|
||||
session history.
|
||||
3. **`packages[]` is the only source of truth for which copy is loaded.** A clone
|
||||
at `/workspace/pi-observational-memory` may exist (and today matches `/opt`
|
||||
byte-for-byte at `ce9fc98`) — its presence proves nothing. To run a patched
|
||||
build you point `packages[]` at it explicitly and start a new session.
|
||||
4. **The worker model is a deliberate choice, and it is yours to change.** The
|
||||
seeded config sends background work to Haiku while your session runs Opus:
|
||||
|
||||
```json
|
||||
"observational-memory": {
|
||||
"model": { "provider": "amazon-bedrock", "id": "eu.anthropic.claude-haiku-4-5-20251001-v1:0" },
|
||||
"debugLog": false
|
||||
}
|
||||
```
|
||||
|
||||
## 8. What it costs
|
||||
|
||||
| Resource | Cost |
|
||||
|---|---|
|
||||
| Model calls | up to **three** background calls per consolidation pass (observer, reflector, dropper), each capped at `agentMaxTurns` [16], on the configured memory model — not your session model |
|
||||
| Latency in your turns | none by construction: workers run from `turn_end`, compaction runs when pi is idle, and the fold itself does no model work |
|
||||
| Disk | negligible — JSON lines inside a session file that would exist anyway (measured here: `~/.pi/agent/sessions` = 30 MB total, tens of `om.*` entries per session) |
|
||||
| Context window | **zero until compaction.** `custom` entries do not enter LLM context; only the folded summary does |
|
||||
| Attention | none once configured; there is no protocol for you or the agent to remember |
|
||||
|
||||
If that is still more than you want on a given run, §9's `passive` switch turns
|
||||
off all proactive work while keeping `recall` and `/om:*` usable.
|
||||
|
||||
## 9. Configuration
|
||||
|
||||
Global: `~/.pi/agent/settings.json` (persisted in the volume). Per project:
|
||||
`<project>/.pi/settings.json`, which overrides global. Precedence is
|
||||
project → global → environment, and the environment can only override `passive`.
|
||||
|
||||
| Key | Default | What it changes |
|
||||
|---|---|---|
|
||||
| `observeAfterTokens` | `10000` | observer cadence — lower means smaller chunks and more calls |
|
||||
| `reflectAfterTokens` | `20000` | reflector cadence (and thereby dropper opportunities) |
|
||||
| `observerChunkMaxTokens` | 20% of the memory model's context window, else `60000` | cap on one observer run's input |
|
||||
| `compactAfterTokens` | `81000` | when proactive auto-compaction fires |
|
||||
| `observationsPoolMaxTokens` | `20000` | pool size at which compaction does a full re-fold from the branch root |
|
||||
| `observationsPoolTargetTokens` | half of max (`10000`) | what the dropper aims back down to |
|
||||
| `agentMaxTurns` | `16` | shared turn cap for the three workers |
|
||||
| `model` | unset → session model | send background work to a cheaper/faster model |
|
||||
| `showWorkerNotifications` | `true` | routine "observer ran" notices |
|
||||
| `passive` | `false` | **kill switch** for all proactive background work; `recall` and `/om:*` still work |
|
||||
| `debugLog` | `false` | per-session NDJSON trace at `~/.pi/agent/observational-memory/debug/<session-id>.ndjson` |
|
||||
|
||||
Pi's own compaction knobs live under a separate `compaction` key —
|
||||
`keepRecentTokens` [20000] sets the verbatim tail from §4, `reserveTokens`
|
||||
[16384] the headroom that triggers pi's own compaction.
|
||||
|
||||
One-off passive run, no config edit:
|
||||
|
||||
```bash
|
||||
PI_OBSERVATIONAL_MEMORY_PASSIVE=1 pi
|
||||
```
|
||||
|
||||
Invalid values are ignored rather than fatal, so a typo degrades to the default
|
||||
instead of breaking your session — which also means a typo is silent. Check with
|
||||
`/om:status`.
|
||||
|
||||
## 10. Confirming it is actually working
|
||||
|
||||
Do not infer health from the absence of a warning; look:
|
||||
|
||||
```bash
|
||||
# 1. inside pi — the authoritative view
|
||||
/om:status # visible-vs-full drift, thresholds, worker state
|
||||
/om:view # what the agent currently sees
|
||||
/om:view full # full ledger truth at the branch tip
|
||||
|
||||
# 2. from a shell — are ledger entries being written, and has it compacted?
|
||||
grep -o '"customType":"om\.[a-z.]*"' \
|
||||
"$(ls -t ~/.pi/agent/sessions/*/*.jsonl | head -1)" | sort | uniq -c
|
||||
grep -c '"type":"compaction"' "$(ls -t ~/.pi/agent/sessions/*/*.jsonl | head -1)"
|
||||
|
||||
# 3. which copy is loaded, and at what commit
|
||||
python3 -c "import json;print(json.load(open('$HOME/.pi/agent/settings.json'))['packages'])"
|
||||
git -c safe.directory=/opt/pi-observational-memory -C /opt/pi-observational-memory rev-parse HEAD
|
||||
```
|
||||
|
||||
Ledger entries are `"type":"custom"` with `"customType":"om.…"`. Do not grep for
|
||||
`custom_message` — that is a *different* pi API for entries that **do** enter LLM
|
||||
context, used here by the MemPalace mailbox (`customType: "mempalace-mailbox"`),
|
||||
not by om.
|
||||
|
||||
## 11. It is not the same thing as MemPalace
|
||||
|
||||
Both are called "memory" and they solve different problems. Nothing is wrong with
|
||||
running both — this image does, and they cover each other's failure modes.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
O0["observational<br/>memory"] --> O1["horizon:<br/>this session"]
|
||||
O1 --> O2["scope: one branch,<br/>one machine"]
|
||||
O2 --> O3["automatic"]
|
||||
O3 --> O4["retrieval:<br/>recall(id)"]
|
||||
P0["MemPalace"] --> P1["horizon: months,<br/>machines"]
|
||||
P1 --> P2["scope:<br/>the fleet"]
|
||||
P2 --> P3["protocol-driven"]
|
||||
P3 --> P4["retrieval:<br/>search, KG, mailbox"]
|
||||
```
|
||||
|
||||
| Question | Answer |
|
||||
|---|---|
|
||||
| "What did we decide 200 turns ago in *this* session?" | observational memory (and `recall` for the exact wording) |
|
||||
| "What did we decide last month, or on another machine?" | MemPalace (`mempalace_search`, diaries) |
|
||||
| "What is true *right now* about version X?" | MemPalace knowledge graph |
|
||||
| "Does another machine need something from me?" | MemPalace coordination log — see [Cross-machine agent coordination](../README.md#cross-machine-agent-coordination) |
|
||||
| "Why is compaction not losing my session?" | observational memory |
|
||||
|
||||
The crisp version: **observational memory keeps a session coherent; the palace
|
||||
keeps the fleet coherent.** A container recreate wipes neither — but only because
|
||||
`~/.pi` and the palace both live outside the container filesystem.
|
||||
|
||||
## 12. Gotchas
|
||||
|
||||
- **Branch-local means branch-local.** Resuming or forking changes which ledger
|
||||
is folded. Memory that "disappeared" is usually on another branch.
|
||||
- **`recall` needs an id, not a topic.** If you only have a topic, that is a
|
||||
palace search, not a recall.
|
||||
- **A `/workspace` clone is not evidence of what is loaded** — see §7.3.
|
||||
- **`showWorkerNotifications: true` is not proof of work**; it reports runs, and
|
||||
an observer that deliberately emits nothing writes no ledger entry and simply
|
||||
retries after another `observeAfterTokens`.
|
||||
- **A turn bigger than `keepRecentTokens` splits.** The cut then lands mid-turn at
|
||||
an assistant message and pi merges two summaries — rare, but it is why a very
|
||||
large single turn can lose more verbatim detail than you would expect.
|
||||
- **`git log` in the baked tree needs `safe.directory`** (`/opt` is root-owned):
|
||||
`git -c safe.directory=/opt/pi-observational-memory -C /opt/pi-observational-memory log`.
|
||||
@@ -462,6 +462,24 @@ every later ask on the same `correlation_id` for good; verified on a live thread
|
||||
where a `ready` reply at `seq` 16 sits *before* the request at `seq` 17 that it
|
||||
obviously cannot have answered.
|
||||
|
||||
**On a real mesh, compare `hlc` instead.** `seq` is *replica-local*: it equals
|
||||
`origin_seq` today only because a single replica authors events for every
|
||||
machine. Enrol a second replica and a late-syncing peer event gets a late local
|
||||
`seq`, so two replicas can order the same pair differently and derive different
|
||||
owed-sets from the same log. Every event already carries `hlc`
|
||||
(`<millis>-<counter>-<replica_id>`), which is total and causally consistent.
|
||||
So: compare `seq` while `mempalace_mesh_peers` reports no peers, `hlc` once it
|
||||
reports any, and `created_at` never. (This is a legitimate use of `mesh_peers` —
|
||||
choosing an ordering key — not the discredited gate on *whether* to read your
|
||||
mailbox at all.)
|
||||
|
||||
**The failure directions are not symmetric, which is why this is safe to get
|
||||
slightly wrong.** Local-`seq` skew can make an already-answered item *resurface*
|
||||
as owed: noise, self-correcting, and visible. A timestamp comparison can
|
||||
*suppress an unanswered ask forever*: silent and permanent. So if you ever see an
|
||||
item you know you answered come back, do **not** "fix" it by reaching for
|
||||
`created_at` — you would be trading the safe failure for the dangerous one.
|
||||
|
||||
This also supplies the "taken, not finished" state that looked missing:
|
||||
`claimed` and `ready` are deliberately **not** terminal, so work you have picked
|
||||
up keeps resurfacing until you close it out. No extra convention, no new field.
|
||||
|
||||
@@ -2,8 +2,8 @@
|
||||
# Runtime post-recreate verification for pi-devbox.
|
||||
#
|
||||
# Verifies that after `docker compose up -d --force-recreate`:
|
||||
# - The new image is actually live (pi version matches, when an expected
|
||||
# version is supplied — see the version note below)
|
||||
# - The new image is actually live (both the pi version and — when asked —
|
||||
# the pi-devbox image release tag; see the two version notes below)
|
||||
# - Persisted named volumes survived (~/.pi config, shell history, zoxide,
|
||||
# nvim data, uv cache, ssh-local)
|
||||
# - pi runtime wiring is intact: keybindings symlink, AGENTS.md symlink,
|
||||
@@ -25,13 +25,33 @@
|
||||
# the pi-devbox repo (which a maintainer already has for CI builds). A plain
|
||||
# `docker pull` consumer is not the audience and will not have this file.
|
||||
#
|
||||
# Version note: pi's version is resolved from `latest` at CI build time and is
|
||||
# NOT pinned to a concrete value in Dockerfile.variant (ARG PI_VERSION=latest).
|
||||
# So unlike opencode-devbox, this script cannot self-derive an expected version
|
||||
# from the Dockerfile. Pass --expected-version to assert a match; without it the
|
||||
# live pi version is reported as an informational WARN, not a failure.
|
||||
# TWO DIFFERENT VERSIONS, TWO DIFFERENT FLAGS. This distinction has already
|
||||
# cost a release day, so it is spelled out here and in AGENTS.md step 4:
|
||||
#
|
||||
# Usage: ./scripts/recreate-sanity-check.sh [--expected-version X.Y.Z] [--variant studio|plain]
|
||||
# --expected-version the PI CODING AGENT version, e.g. 0.84.3
|
||||
# (`pi --version`; pinned as ARG PI_VERSION in
|
||||
# Dockerfile.variant, which CI reads as the source
|
||||
# of truth)
|
||||
# --expected-image-version the PI-DEVBOX IMAGE release tag, e.g. 1.8.9 or
|
||||
# v1.8.9 (the `release_tag` baked into
|
||||
# /etc/pi-devbox/build-manifest.json)
|
||||
#
|
||||
# Passing a release tag to --expected-version used to report
|
||||
# "pi version mismatch: expected 1.8.8, got 0.84.3" — an accusation aimed at
|
||||
# the wrong component, on the last gate of a release. Both flags now detect
|
||||
# being handed the other one's value and say so instead.
|
||||
#
|
||||
# Neither flag is required. Both values are derivable from the image's own
|
||||
# build manifest, so by default the script asserts the LIVE pi version against
|
||||
# the version recorded at build time — which is not a tautology: a stale
|
||||
# `pi` in the ~/.pi/npm-global volume can shadow the baked one, exactly the
|
||||
# way a stale npm:pi-atelier can (see the packages[] check below). Pass the
|
||||
# flags when you want an assertion against a value you name yourself, which
|
||||
# is what a release checklist wants.
|
||||
#
|
||||
# Usage: ./scripts/recreate-sanity-check.sh [--expected-version X.Y.Z]
|
||||
# [--expected-image-version X.Y.Z]
|
||||
# [--variant studio|plain]
|
||||
#
|
||||
# Exit codes:
|
||||
# 0 all checks passed
|
||||
@@ -41,22 +61,61 @@
|
||||
set -euo pipefail
|
||||
|
||||
EXPECTED_VERSION=""
|
||||
EXPECTED_IMAGE_VERSION=""
|
||||
VARIANT=""
|
||||
REPO_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
|
||||
MANIFEST=/etc/pi-devbox/build-manifest.json
|
||||
|
||||
# Parse arguments
|
||||
usage() {
|
||||
cat >&2 <<'EOF'
|
||||
usage: recreate-sanity-check.sh [--expected-version X.Y.Z]
|
||||
[--expected-image-version X.Y.Z]
|
||||
[--variant studio|plain]
|
||||
|
||||
--expected-version pi coding agent version, e.g. 0.84.3 (`pi --version`)
|
||||
--expected-image-version pi-devbox image release tag, e.g. 1.8.9 or v1.8.9
|
||||
--variant studio|plain (auto-detected when omitted)
|
||||
|
||||
These are two different versions. Both are read from the image's own build
|
||||
manifest when the corresponding flag is omitted.
|
||||
EOF
|
||||
}
|
||||
|
||||
# Parse arguments. Every flag takes a value, so reject a missing one rather
|
||||
# than swallowing the next flag as if it were the value.
|
||||
need_value() {
|
||||
case "${2:-}" in
|
||||
""|-*)
|
||||
echo "$1 requires a value" >&2
|
||||
usage
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
}
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--expected-version)
|
||||
need_value "$@"
|
||||
EXPECTED_VERSION="$2"
|
||||
shift 2
|
||||
;;
|
||||
--expected-image-version)
|
||||
need_value "$@"
|
||||
EXPECTED_IMAGE_VERSION="$2"
|
||||
shift 2
|
||||
;;
|
||||
--variant)
|
||||
need_value "$@"
|
||||
VARIANT="$2"
|
||||
shift 2
|
||||
;;
|
||||
--help|-h)
|
||||
usage
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
echo "usage: $0 [--expected-version X.Y.Z] [--variant studio|plain]" >&2
|
||||
echo "unknown option: $1" >&2
|
||||
usage
|
||||
exit 2
|
||||
;;
|
||||
esac
|
||||
@@ -67,6 +126,19 @@ pass() { echo " ✓ $1"; }
|
||||
fail() { echo " ✗ $1" >&2; FAILED=$((FAILED + 1)); }
|
||||
warn() { echo " ⚠ $1" >&2; }
|
||||
|
||||
# Read one top-level field from the build manifest, or print nothing. The
|
||||
# manifest is the image's own ground truth (written at `docker build` time by
|
||||
# Dockerfile.variant), so it needs no checkout and no network. Absent on an
|
||||
# image built before it existed, hence every caller treats "" as unknown.
|
||||
manifest_field() {
|
||||
[ -f "$MANIFEST" ] || return 0
|
||||
command -v jq >/dev/null 2>&1 || return 0
|
||||
jq -r --arg k "$1" '.[$k] // empty' "$MANIFEST" 2>/dev/null || true
|
||||
}
|
||||
# Release tags are written with a leading v in the manifest and quoted without
|
||||
# one in checklists; compare on the bare number so both spellings work.
|
||||
strip_v() { printf '%s' "${1#v}"; }
|
||||
|
||||
# Auto-detect variant if not provided. The studio variant vendors pi-studio to
|
||||
# /opt/pi-studio; the plain variant does not.
|
||||
if [ -z "$VARIANT" ]; then
|
||||
@@ -86,21 +158,59 @@ else
|
||||
fi
|
||||
echo
|
||||
|
||||
echo "-- pi version --"
|
||||
MANIFEST_PI_VERSION=$(manifest_field pi_version)
|
||||
MANIFEST_RELEASE_TAG=$(manifest_field release_tag)
|
||||
|
||||
echo "-- pi (coding agent) version --"
|
||||
if ACTUAL_VERSION=$(pi --version 2>&1 | head -1); then
|
||||
if [ -n "$EXPECTED_VERSION" ]; then
|
||||
if [ "$ACTUAL_VERSION" = "$EXPECTED_VERSION" ]; then
|
||||
pass "pi version $ACTUAL_VERSION"
|
||||
if [ "$(strip_v "$EXPECTED_VERSION")" = "$(strip_v "$ACTUAL_VERSION")" ]; then
|
||||
pass "pi version $ACTUAL_VERSION (matches --expected-version)"
|
||||
elif [ -n "$MANIFEST_RELEASE_TAG" ] &&
|
||||
[ "$(strip_v "$EXPECTED_VERSION")" = "$(strip_v "$MANIFEST_RELEASE_TAG")" ]; then
|
||||
# Exact, not heuristic: the value handed over IS this image's release
|
||||
# tag, so it cannot be a pi version anyone meant.
|
||||
fail "--expected-version $EXPECTED_VERSION is the pi-devbox IMAGE version, not the pi version — use --expected-image-version $EXPECTED_VERSION (live pi is $ACTUAL_VERSION)"
|
||||
else
|
||||
fail "pi version mismatch: expected $EXPECTED_VERSION, got $ACTUAL_VERSION"
|
||||
fail "pi version mismatch: expected $EXPECTED_VERSION, got $ACTUAL_VERSION (this flag asserts the pi coding agent version; for the image release tag use --expected-image-version)"
|
||||
fi
|
||||
elif [ -n "$MANIFEST_PI_VERSION" ]; then
|
||||
# Not a tautology: the manifest records what pi reported at BUILD time,
|
||||
# while `pi --version` resolves through PATH, which a stale npm-global
|
||||
# volume install can shadow.
|
||||
if [ "$MANIFEST_PI_VERSION" = "$ACTUAL_VERSION" ]; then
|
||||
pass "pi version $ACTUAL_VERSION (matches this image's build manifest)"
|
||||
else
|
||||
fail "live pi $ACTUAL_VERSION != $MANIFEST_PI_VERSION recorded in $MANIFEST — a stale pi in the ~/.pi/npm-global volume is shadowing the baked one"
|
||||
fi
|
||||
else
|
||||
warn "pi version $ACTUAL_VERSION (no --expected-version given; pi is built from 'latest', cannot self-derive — informational only)"
|
||||
warn "pi version $ACTUAL_VERSION (no --expected-version and no build manifest to compare against — informational only)"
|
||||
fi
|
||||
else
|
||||
fail "pi --version failed"
|
||||
fi
|
||||
|
||||
echo
|
||||
echo "-- pi-devbox image version --"
|
||||
if [ -z "$MANIFEST_RELEASE_TAG" ]; then
|
||||
if [ -n "$EXPECTED_IMAGE_VERSION" ]; then
|
||||
fail "cannot verify --expected-image-version $EXPECTED_IMAGE_VERSION: no readable release_tag in $MANIFEST (image built before the manifest existed, or jq missing)"
|
||||
else
|
||||
warn "image release tag unknown (no readable $MANIFEST) — pi-devbox-version would say the same"
|
||||
fi
|
||||
elif [ -n "$EXPECTED_IMAGE_VERSION" ]; then
|
||||
if [ "$(strip_v "$EXPECTED_IMAGE_VERSION")" = "$(strip_v "$MANIFEST_RELEASE_TAG")" ]; then
|
||||
pass "image version $MANIFEST_RELEASE_TAG (matches --expected-image-version)"
|
||||
elif [ -n "$MANIFEST_PI_VERSION" ] &&
|
||||
[ "$(strip_v "$EXPECTED_IMAGE_VERSION")" = "$MANIFEST_PI_VERSION" ]; then
|
||||
fail "--expected-image-version $EXPECTED_IMAGE_VERSION is the pi version, not the image release tag — use --expected-version $EXPECTED_IMAGE_VERSION (this image is $MANIFEST_RELEASE_TAG)"
|
||||
else
|
||||
fail "image version mismatch: expected $EXPECTED_IMAGE_VERSION, got $MANIFEST_RELEASE_TAG — the recreate did not pick up the intended image"
|
||||
fi
|
||||
else
|
||||
warn "image version $MANIFEST_RELEASE_TAG (no --expected-image-version given — informational only)"
|
||||
fi
|
||||
|
||||
echo
|
||||
echo "-- Persisted named volumes (must survive --force-recreate) --"
|
||||
|
||||
|
||||
@@ -86,7 +86,13 @@ for arg in "$@"; do
|
||||
case "$arg" in
|
||||
--check) MODE="check" ;;
|
||||
--force) FORCE=1 ;;
|
||||
--*) die "unknown option: $arg" ;;
|
||||
# This is the one script whose argument ORDER was itself a landmine, so the
|
||||
# path that documents the trap must not be the path that errors.
|
||||
-h|--help)
|
||||
awk 'NR>1 && /^#/ { sub(/^# ?/, ""); print; next } NR>1 { exit }' "$0"
|
||||
exit 0
|
||||
;;
|
||||
--*) die "unknown option: $arg (try --help)" ;;
|
||||
*)
|
||||
[ -z "$ROOT" ] || die "unexpected extra argument: $arg (root already set to $ROOT)"
|
||||
ROOT="$arg"
|
||||
@@ -110,6 +116,12 @@ fi
|
||||
[ -d "$ROOT/.git" ] || die "not a git clone: $ROOT"
|
||||
[ -f "$ROOT/$REL_PATH" ] || die "no $REL_PATH in $ROOT"
|
||||
[ -f "$VENDORED" ] || die "vendored snapshot missing: $VENDORED"
|
||||
# LOAD-BEARING, DO NOT DELETE AS "REDUNDANT WITH THE EXISTENCE PROBES": -f
|
||||
# accepts an empty file, and sha256 of an empty file equals sha256 of a failed
|
||||
# pipeline's empty stdin. Guarding it HERE, before mode dispatch, makes that
|
||||
# collision unreachable by construction rather than by a probe further down --
|
||||
# which also means no test below exercises the collision any more. Remove this
|
||||
# line and the false "OK" for a nonexistent ref returns with nothing failing.
|
||||
[ -s "$VENDORED" ] || die "vendored snapshot is empty: $VENDORED"
|
||||
|
||||
head_sha=$(git -C "$ROOT" rev-parse HEAD 2>/dev/null) || die "cannot read HEAD of $ROOT"
|
||||
@@ -200,8 +212,20 @@ if [ "$MODE" = "check" ]; then
|
||||
# clean and the ref simply moved on — a message that names the wrong cause
|
||||
# is the same defect class as a canary pinned to a deleted phrase.
|
||||
if [ "$recorded" != "$head_sha" ] && [ "$blob_sha" = "$upstream_sha" ]; then
|
||||
printf 'NOTICE: %s has moved to %s; the snapshot describes the older %s (stale, not untruthful)\n' \
|
||||
"$ROOT" "${head_sha:0:7}" "${recorded:0:7}" >&2
|
||||
# Do not ASSERT which side is newer — test it. Asserting that HEAD is the
|
||||
# newer side points the operator at a refresh (which costs a ~67-minute
|
||||
# base rebuild) when the real remedy may be `git pull` in this clone. The
|
||||
# refresh path below already uses this primitive; reuse it here.
|
||||
if git -C "$ROOT" merge-base --is-ancestor "$recorded" "$head_sha" 2>/dev/null; then
|
||||
printf 'NOTICE: %s has moved on to %s; the snapshot describes the older %s (stale, not untruthful — refresh to catch up)\n' \
|
||||
"$ROOT" "${head_sha:0:7}" "${recorded:0:7}" >&2
|
||||
elif git -C "$ROOT" merge-base --is-ancestor "$head_sha" "$recorded" 2>/dev/null; then
|
||||
printf 'NOTICE: %s is BEHIND at %s; the snapshot describes the newer %s — pull this clone, do NOT refresh the snapshot\n' \
|
||||
"$ROOT" "${head_sha:0:7}" "${recorded:0:7}" >&2
|
||||
else
|
||||
printf 'NOTICE: %s (HEAD %s) and the recorded %s have DIVERGED — neither is an ancestor of the other; reconcile the clone before refreshing\n' \
|
||||
"$ROOT" "${head_sha:0:7}" "${recorded:0:7}" >&2
|
||||
fi
|
||||
else
|
||||
printf 'NOTICE: the working tree of %s differs from the snapshot (HEAD %s)\n' \
|
||||
"$ROOT" "${head_sha:0:7}" >&2
|
||||
|
||||
Reference in New Issue
Block a user