Runbook gaps found while fixing the 2026-08-15 feed failure:
- §2.5 still titled "written but not installed" and still asserting "Not
installed, not enabled" — false since 2026-08-12. A reader landing there got
a flat contradiction of §4 item 3. Retitled, with the verified-2026-08-16
process line, and it now states the fact §2.6 depends on: the server is a
NATIVE process (no mempalace container on synlig), so it sees host paths.
- New §2.6 documents ~/mempalace-feed/<device>/: why transcripts cannot travel
over the HTTPS leg at all, the three client variables, and why
MEMPALACE_PI_REMOTE_PATH is the trap (its /data/feed default assumes a
containerized server; here it must equal the ssh-target path, and a mismatch
fails with rsync succeeding and only the mine failing). Plus the operational
notes that cost time: dedup keys on the absolute path so the inbox path is
load-bearing, grown sessions are purged+refiled by mtime, and a client-side
MCP timeout is NOT a failed mine.
- Header status: counts refreshed with an explicit "treat counts as timestamps".
Also, bin/mempalace-pi-session: default HOME from the passwd database when it is
unset. `docker run --entrypoint="" <image>` inherits no HOME when the image
config declares none, and every default is HOME-anchored under `set -u`, so the
script — including the palace-free --self-test — died with "HOME: unbound
variable" in exactly the environment pi-devbox's smoke suite uses. pi-devbox
v1.8.0 lost a release to the same assumption from the other side.
The remote-mine leg decided success with `'"error"' in body`. MCP answers a
hard tool failure with HTTP 200 and a JSON-RPC *result* whose content[].text
carries the tool's own JSON as an ESCAPED string, so those bytes are
\"error\" and the substring never matches. On 2026-08-15 (EMB-7KJ4VR4G, first
boot of the fresh pi-devbox image) a mine that failed with
{"success": false, "error": "source directory not found: '/data/feed/emb-7kj4vr4g'"}
printed "Done. Wing 'wing_conversations' updated." directly under that error
and exited 0. Transcripts had been rsynced for the whole session and filed
nowhere; the only artifact anyone would check said it worked.
- classify(): parse the envelope instead of grepping it. Catches JSON-RPC
errors, MCP isError, and inner success=false/error, and distinguishes
"verified ok" from "unverified: no JSON tool payload" rather than assuming.
- --self-test: six recorded MCP responses (fixture 1 is the real 2026-08-15
body) plus a regression guard asserting the old substring check is blind to
it. Needs no palace, no network, no sessions dir.
- Preflight warning when the rsync destination path and
MEMPALACE_PI_REMOTE_PATH disagree. The /data/feed default assumes a
CONTAINERIZED palace server; a native one (systemd unit / uv tool) sees host
paths, and then the two must match. Warned in preflight so --dry-run and
--prepare surface it too.
- Remote mode no longer previews NEW/SKIP from the LOCAL palace: dedup happens
on the palace host keyed on the remote inbox path, so this machine cannot
answer it. Tags become [?] and the summary says who decides. It had been
reporting "6 already filed" about a palace it was not feeding.
Staging default moves out of ~/.cache to <palace-root>/pi-stage (pi) and
<palace-root>/opencode-stage (opencode), resolved with mempalace's own
palace-path precedence ($MEMPALACE_PALACE_PATH -> $MEMPAL_PALACE_PATH ->
~/.mempalace/config.json -> ~/.mempalace/palace), then dirname.
Why: the convos miner keys dedup on the *staged* path, so a wiped stage plus a
sync scoped to include it prunes the drawers mined from those sources --
deleting memories, not a cache. Under ~/.cache that state was reachable by
anything treating a cache as disposable. Staging inside the palace makes the
coupling structural: the stage cannot be wiped without touching the palace
itself. Overrides ($MEMPALACE_PI_STAGE / $MEMPALACE_SESSION_STAGE, --stage) are
unchanged. Note the old default had never been created on any host, so this
closed a latent hazard, not a live one.
Measured, and the docs now claim only this much: sync prunes only within the
scope it is given -- wing-only, 1299 scanned / 1299 out of scope / 0 removed;
scoped at the palace root, 651 kept / 648 out of scope. The previous blanket
"sync prunes every drawer" wording overstated it, which is a liability: the next
reader disproves the overstatement and discards the real constraint with it.
Also in this change:
- cron log dir ~/.cache/mempalace-session -> ~/.cache/mempalace-logs. The stage
left that namespace, so the old name now read as "the stage".
- AGENTS.md: the convos miner *does* check mtime (verified against upstream
convo_miner.py); the previous "no mtime check" claim was wrong.
- smoke-test assertions use `mktemp -d` for --sessions-dir. One pointed at /tmp,
which still held earlier synthetic transcripts, so a --dry-run exported a fake
session into the real stage: --dry-run skips the mine, not the export.
docs/phase-1-exposure-runbook.md -- the newt/DNS/auth step that RFC 001 and the
synlig runbook leave open (runbook section 4, items 2 and 5). Port 8765 at /mcp,
newt targets 172.17.0.1, and the authentication is the single shared bearer
token (RFC 6.2, decided 2026-08-09) rather than per-device proxy users. The
latter cannot work today: mempalace validates exactly one token, and Pangolin's
SSO/PIN/password are browser-shaped while every client here is a headless
JSON-RPC POST -- enabling that protection breaks the clients it protects. The
per-device axis that *does* exist is the feeder's SSH key + per-device inbox.
New finding recorded there: a loopback bind does not merely 403 behind a tunnel
(already known, runbook 2.4) -- it also silently starts the server with no token
at all, because auto-minting is gated on the bind being non-loopback.
extensions/pi/README.md: the HTTP transport IS authenticated as of mempalace
3.6.0; the "sessionless and unauthenticated" note dated from the v1.3.0 era.
Closes the RFC section 8 Phase-0 hygiene item.
The three feeder wrappers (mempalace-docs, mempalace-pi-session,
mempalace-session) unconditionally ran 'mempalace repair --yes' after
mining, controllable only via --no-repair opt-out. The contrib launchd
and systemd templates did not pass --no-repair, so every scheduled tick
invoked the destructive in-place HNSW rebuild.
This has bitten us twice:
- 2026-05-04 09:08: a kickstart triggered repair while an MCP
subprocess held the DB open; the live collection was wiped (0
drawers) and had to be restored from the palace.backup snapshot.
- 2026-05-05 10:00: post-mine repair crashed mid-rebuild with
'NotFoundError: Collection [<uuid>] does not exist' - chromadb's
rebuild recreated the collection under a new UUID while the code
still held the old handle. Live DB survived only by luck (crash
hit before the swap).
Fix: flip the default.
- New flag: --repair (opt-in). Prints a warning and sleeps 3s before
invoking 'mempalace repair --yes'.
- --no-repair is retained as a deprecated no-op alias for backward
compatibility with any scripts/units still passing it.
- Default behavior: no repair. Routine ChromaDB add() keeps HNSW
consistent; repair is a recovery op, not a maintenance tick.
Docs updated to match: README, SKILL, ARCHITECTURE, AGENTS,
contrib/README. Scheduling guidance now explicitly warns against
enabling --repair on cron/launchd/systemd-timer runs.
Parallel to mempalace-session, this wrapper walks ~/.pi/agent/sessions/
JSONL files and mines qualifying sessions into wing_conversations via
'mempalace mine --mode convos'.
Design choices mirror mempalace-session:
- Export-stage-mine idiom with deterministic per-session staging paths
under ~/.cache/mempalace-pi-session/<wing>/, so 'mempalace mine' dedup
on source_file makes re-runs idempotent.
- --dry-run classifies each export as [NEW] or [SKIP] by matching staging
path against the palace's already-filed source_files.
- --min-messages filter skips throwaway single-prompt sessions.
Pi-specific parsing:
- Pi JSONL is a typed tree (id/parentId) per docs/session-format.md;
this walks in file order, which is correct for the overwhelmingly
linear case and harmlessly duplicative on branched sessions (palace
semantic dedup handles it).
- Roles mapped to Claude Code JSONL shape:
user -> {type:user, content:text}
assistant -> {type:assistant, content:[text, tool_use]}
toolResult-> {type:human, content:[tool_result]} (folded back by normalizer)
bashExecution/custom(display)/branchSummary/compactionSummary
-> rendered as text annotations
- thinking blocks and image blocks dropped (noise / palace is text-only).
Source labelling:
- Staging filenames prefixed 'pi_<uuid>.jsonl' so every drawer's
source_file metadata (visible in search results) unambiguously
identifies the harness. Opencode's convention ('<slug>_<id>.jsonl')
is preserved to keep the existing 19k+ drawers deduped.
- Inline synthetic header on first chunk:
[session: <title> | <cwd> | <date> | source: pi]
as a secondary signal.
Joakim Persson
joakimp