Staging default moves out of ~/.cache to <palace-root>/pi-stage (pi) and
<palace-root>/opencode-stage (opencode), resolved with mempalace's own
palace-path precedence ($MEMPALACE_PALACE_PATH -> $MEMPAL_PALACE_PATH ->
~/.mempalace/config.json -> ~/.mempalace/palace), then dirname.
Why: the convos miner keys dedup on the *staged* path, so a wiped stage plus a
sync scoped to include it prunes the drawers mined from those sources --
deleting memories, not a cache. Under ~/.cache that state was reachable by
anything treating a cache as disposable. Staging inside the palace makes the
coupling structural: the stage cannot be wiped without touching the palace
itself. Overrides ($MEMPALACE_PI_STAGE / $MEMPALACE_SESSION_STAGE, --stage) are
unchanged. Note the old default had never been created on any host, so this
closed a latent hazard, not a live one.
Measured, and the docs now claim only this much: sync prunes only within the
scope it is given -- wing-only, 1299 scanned / 1299 out of scope / 0 removed;
scoped at the palace root, 651 kept / 648 out of scope. The previous blanket
"sync prunes every drawer" wording overstated it, which is a liability: the next
reader disproves the overstatement and discards the real constraint with it.
Also in this change:
- cron log dir ~/.cache/mempalace-session -> ~/.cache/mempalace-logs. The stage
left that namespace, so the old name now read as "the stage".
- AGENTS.md: the convos miner *does* check mtime (verified against upstream
convo_miner.py); the previous "no mtime check" claim was wrong.
- smoke-test assertions use `mktemp -d` for --sessions-dir. One pointed at /tmp,
which still held earlier synthetic transcripts, so a --dry-run exported a fake
session into the real stage: --dry-run skips the mine, not the export.
docs/phase-1-exposure-runbook.md -- the newt/DNS/auth step that RFC 001 and the
synlig runbook leave open (runbook section 4, items 2 and 5). Port 8765 at /mcp,
newt targets 172.17.0.1, and the authentication is the single shared bearer
token (RFC 6.2, decided 2026-08-09) rather than per-device proxy users. The
latter cannot work today: mempalace validates exactly one token, and Pangolin's
SSO/PIN/password are browser-shaped while every client here is a headless
JSON-RPC POST -- enabling that protection breaks the clients it protects. The
per-device axis that *does* exist is the feeder's SSH key + per-device inbox.
New finding recorded there: a loopback bind does not merely 403 behind a tunnel
(already known, runbook 2.4) -- it also silently starts the server with no token
at all, because auto-minting is gated on the bind being non-loopback.
extensions/pi/README.md: the HTTP transport IS authenticated as of mempalace
3.6.0; the "sessionless and unauthenticated" note dated from the v1.3.0 era.
Closes the RFC section 8 Phase-0 hygiene item.
The three feeder wrappers (mempalace-docs, mempalace-pi-session,
mempalace-session) unconditionally ran 'mempalace repair --yes' after
mining, controllable only via --no-repair opt-out. The contrib launchd
and systemd templates did not pass --no-repair, so every scheduled tick
invoked the destructive in-place HNSW rebuild.
This has bitten us twice:
- 2026-05-04 09:08: a kickstart triggered repair while an MCP
subprocess held the DB open; the live collection was wiped (0
drawers) and had to be restored from the palace.backup snapshot.
- 2026-05-05 10:00: post-mine repair crashed mid-rebuild with
'NotFoundError: Collection [<uuid>] does not exist' - chromadb's
rebuild recreated the collection under a new UUID while the code
still held the old handle. Live DB survived only by luck (crash
hit before the swap).
Fix: flip the default.
- New flag: --repair (opt-in). Prints a warning and sleeps 3s before
invoking 'mempalace repair --yes'.
- --no-repair is retained as a deprecated no-op alias for backward
compatibility with any scripts/units still passing it.
- Default behavior: no repair. Routine ChromaDB add() keeps HNSW
consistent; repair is a recovery op, not a maintenance tick.
Docs updated to match: README, SKILL, ARCHITECTURE, AGENTS,
contrib/README. Scheduling guidance now explicitly warns against
enabling --repair on cron/launchd/systemd-timer runs.
Complement to the mempalace-pi-session feeder: now that a second source
mines into wing_conversations, every session's synthetic header carries
an explicit source tag so the LLM can discriminate at read time when
searches return first-chunk content:
[session: <title> | <directory> | <YYYY-MM-DD> | source: opencode]
The primary disambiguator in search results remains source_file basename
(opencode: '<slug>_<id>.jsonl', pi: 'pi_<uuid>.jsonl'), which is present
in every chunk's metadata regardless of where the search hit landed in
the session. This header tag is a cosmetic second signal on first-chunk
hits.
Caveat: existing drawers keep their old header — mempalace mine dedups
by source_file path, which didn't change, so old opencode sessions are
not re-mined. They are implicitly opencode (the only pre-pi source).
Parallel to mempalace-session, this wrapper walks ~/.pi/agent/sessions/
JSONL files and mines qualifying sessions into wing_conversations via
'mempalace mine --mode convos'.
Design choices mirror mempalace-session:
- Export-stage-mine idiom with deterministic per-session staging paths
under ~/.cache/mempalace-pi-session/<wing>/, so 'mempalace mine' dedup
on source_file makes re-runs idempotent.
- --dry-run classifies each export as [NEW] or [SKIP] by matching staging
path against the palace's already-filed source_files.
- --min-messages filter skips throwaway single-prompt sessions.
Pi-specific parsing:
- Pi JSONL is a typed tree (id/parentId) per docs/session-format.md;
this walks in file order, which is correct for the overwhelmingly
linear case and harmlessly duplicative on branched sessions (palace
semantic dedup handles it).
- Roles mapped to Claude Code JSONL shape:
user -> {type:user, content:text}
assistant -> {type:assistant, content:[text, tool_use]}
toolResult-> {type:human, content:[tool_result]} (folded back by normalizer)
bashExecution/custom(display)/branchSummary/compactionSummary
-> rendered as text annotations
- thinking blocks and image blocks dropped (noise / palace is text-only).
Source labelling:
- Staging filenames prefixed 'pi_<uuid>.jsonl' so every drawer's
source_file metadata (visible in search results) unambiguously
identifies the harness. Opencode's convention ('<slug>_<id>.jsonl')
is preserved to keep the existing 19k+ drawers deduped.
- Inline synthetic header on first chunk:
[session: <title> | <cwd> | <date> | source: pi]
as a secondary signal.
A --dry-run report showed all qualifying sessions without indicating
which would actually hit the palace on a real run. On a second run
against an already-mined corpus this was misleading — output said
'Exported 62 session(s)' but the real mine step would skip all 62.
The wrapper now queries the palace's chroma.sqlite3 (read-only, via
file:...?mode=ro URI) for source_file values under the staging dir,
then tags each exported session as [NEW] or [SKIP] during listing and
reports the split in the summary:
Exported 62 session(s) to ~/.cache/mempalace-session/wing_conversations
0 new → will be filed on mine
62 already filed → will be skipped (dedup by source_file)
--dry-run: no new sessions to mine. A real run would skip all 62.
Implementation notes:
- Classification is best-effort. If the palace is unreachable (fresh
install, moved, permission-denied, file missing) the wrapper falls
back to treating all exports as NEW — the real mine step still
delegates dedup to 'mempalace mine --mode convos' which is the
authoritative source of truth. Getting the classification wrong
in --dry-run is cosmetic; behaviour of a real run is unchanged.
- Palace path respects $MEMPALACE_PATH env var for non-default setups.
- Same classification also shown on a real (non-dry-run) mine so users
see upfront how much of the export set is actually new before the
miner runs.
Verified both directions:
- All-already-filed case (current box, 62 sessions in palace): reports
0 new, 62 skipped. --dry-run message correctly says 'would skip all'.
- Partial case (simulated by deleting one session's metadata from
palace): reports 1 new, 61 skipped. --dry-run message correctly
says 'would file 1 new'. Palace was restored from backup
immediately after the test.
README and SKILL.md both updated with the new dedup-aware output and
a direct answer to the FAQ 'will it mine the same sessions again?'
Producer-side MemPalace tooling: two bash wrappers that bridge opencode
session history and project documentation into the palace. Originally
developed in cli_utils (2026-04-28); split into its own repo on
2026-04-30 because the conceptual fit was weak — cli_utils is
interactive shell tooling, while this is agent memory infrastructure
with its own architecture, dependency surface, and growth trajectory.
Contents:
- bin/mempalace-docs — docs-only mining wrapper (originally a2ddcc9 in
cli_utils), bridges the gap until MemPalace PR #1213 (exclude_patterns)
merges upstream.
- bin/mempalace-session — opencode → palace session bridge (originally
dacca0e in cli_utils). Reads ~/.local/share/opencode/opencode.db,
exports each session to Claude Code JSONL, mines via
'mempalace mine --mode convos'. Bridges the gap until opencode
session-stopping hooks + an opencode harness in hooks_cli.py land
upstream.
- ARCHITECTURE.md — canonical spec: architecture diagram, component
details, setup recipe, operational notes, upstream-retirement
roadmap. Originally a4cf314 in cli_utils.
- SKILL.md — companion agent skill (producer side). Pairs with the
consumer-side mempalace skill. Symlinked into
~/.agents/skills/opencode-mempalace-bridge/ by install.sh.
- install.sh — idempotent installer, also handles --uninstall.
- AGENTS.md — repo conventions.
History of the individual files is not preserved in this split; see
cli_utils (gitea.jordbo.se/joakimp/cli_utils) commits a2ddcc9, dacca0e,
and a4cf314 for the original authorship context.
Joakim Persson
joakimp