Files
mempalace-toolkit/ARCHITECTURE.md
T
Joakim Persson ec436ed3ad docs: reconcile the RFC-001 docs with what is actually deployed
Audit of every doc touching the global-palace rollout against the running
fleet. Each correction below was verified against the filesystem or the host,
not against another doc:

- synlig-primary-runbook: the decommission `rm -rf ~/.mempalace` now carries a
  STOP block. That tree holds the fleet palace *and* the only copy of the
  bearer token every client authenticates with; the old "empty today" comment
  stopped being true when the palace was seeded on 2026-08-14. Adds an ordered
  safe decommission, and drops count-based join verification.

- phase-1-exposure-runbook: new S3.8, how to verify a flip actually took --
  the procedure that until now existed only in an untracked handover file.
  Three claims that fail independently (env var / curl / the palace-path
  discriminator) plus an explicit list of checks that produce FALSE POSITIVES:
  drawer counts (both sides were seeded from the same palace, and `status`
  counts chunks not drawers), write-then-read through the same transport, and
  the `mempalace` CLI -- which has no remote support at all, so post-flip it
  reads the dead local archive and reports success.

- rfc-001: status Draft -> Phases 0-1 implemented. Records that the join was a
  file-level copy, which SIDESTEPPED the S7.6 diary-dedup question rather than
  answering it -- so S7.6 remains a hard blocker for the second machine, which
  is the one that will actually exercise merge semantics.

- ARCHITECTURE, SKILL, contrib/README, extensions/pi/README all claimed pi
  feeds the palace automatically, unconditionally. That is gated on
  mempalace-toolkit >= 29e660e and every deployed image predates it, so the
  claim is currently false fleet-wide. Each site now states the gate plus a
  check that inspects the *deployed* file rather than repo HEAD.

- extensions/pi/README: plaintext http://mempalace.lan example -> https
  endpoint; the two transports are either/or (no dual-write, no local mirror);
  the bridge fails CLOSED, so "the agent has no mempalace_* tools" is the
  expected symptom of a server/token/DNS fault, not of a broken install.

- contrib/README: documents mempalace-serve.service, which this directory has
  shipped since day one without explaining it (linger, the load-bearing
  172.17.0.1 bind and why loopback is the unsafe-looking-safe option, the
  token path, and an uninstall warning).

- Fixes a pre-existing stray ```sh fence that was swallowing S3.2's heading and
  the token command into a code block.

Docs only; no behaviour change.
2026-08-14 22:57:22 +02:00

456 lines
31 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
# MemPalace Feeding Architecture
This repository wires [opencode](https://github.com/anomalyco/opencode) and arbitrary project directories into [MemPalace](https://github.com/MemPalace/mempalace) via two thin wrappers in `bin/`. This document explains why they exist and how they fit together.
**Audience:** someone setting up a new machine (or reviewing what's already set up) and asking "how does the palace actually get fed?". Pairs with the `mempalace` agent skill, which covers the *consumer* side (searching, diary, KG). This document covers the *producer* side.
---
## 1. The problem
MemPalace is a persistent memory layer for AI agents — vector search over drawers (chunks of verbatim content), a knowledge graph, and per-agent diaries, all behind an MCP server. To be useful it has to be *fed*: project docs, conversation transcripts, session summaries.
The stock mempalace CLI has two feeders:
| Feeder | What it ingests | Gap |
| ------------------------------------- | -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
| `mempalace mine` (default mode) | Any "readable" file in a directory (code + docs + misc) | Mines source code indiscriminately → embedding index floods with low-signal `__init__` fragments. |
| `mempalace mine --mode convos` | Claude Code JSONL, Claude.ai JSON, ChatGPT JSON, Slack, Codex JSONL | No opencode support. No SQLite support. Opencode persists its history in SQLite, not JSONL. |
And one auto-save path:
| Feeder | Harnesses supported | Gap |
| ------------------------------------- | -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
| `hooks_cli.py` (session-stop hooks) | `claude-code`, `codex` | No `opencode` harness → `/exit` mid-session leaves no diary entry behind. |
So on a machine using opencode + the "docs-first palace hygiene" policy, three gaps bite:
1. Mining a project floods the palace with source code we don't want.
2. Opencode session history is trapped in SQLite, invisible to `mine --mode convos`.
3. There's no auto-save on session stop — any persistence is best-effort heuristic.
The two wrappers in `bin/` close gaps **1** and **2**. Gap **3** is upstream work (see §6).
(Pi is a different story on gap 3: `bin/mempalace-pi-session` closes pi's
version of gap 2 the same way `mempalace-session` closes opencode's, and pi's
gap 3 is closed **without** any upstream dependency — its bridge
extension self-triggers the feed on `session_shutdown` and a debounced
`agent_settled`. **Version gate: extension-side feeding landed in
mempalace-toolkit `29e660e` (2026-08-12), so a copy baked into an older
container image cannot feed at all** — check the deployed file, not repo HEAD:
`grep -c MEMPALACE_FEED "$(readlink -f ~/.pi/agent/extensions/mempalace.ts)"`.
Zero means that machine still needs `contrib/`'s scheduled recipes. See §3's
`mempalace-pi-session` subsection and
[`extensions/pi/README.md`](extensions/pi/README.md).)
---
## 2. The architecture
```
Project dirs (/workspace/*) Opencode SQLite DB
├── *.md ~/.local/share/opencode/opencode.db
├── *.yaml ├── session (id, title, directory, time_created/updated)
├── Dockerfile ├── message (session_id, data JSON w/ role)
└── … └── part (message_id, data JSON w/ type: text|tool|…)
│ │
│ │
┌─────▼──────────┐ ┌────▼──────────────┐
│ mempalace-docs │ │ mempalace-session │
│ (bin/) │ │ (bin/) │
│ │ │ │
│ stage docs │ │ export each │
│ only via cp -p │ │ session as Claude │
│ to cache dir │ │ Code JSONL to │
│ │ │ cache dir │
└─────┬──────────┘ └────┬──────────────┘
│ │
│ ~/.cache/mempalace-docs/<wing>/ │ <palace-root>/opencode-stage/<wing>/
│ │
┌─────▼──────────┐ ┌────▼──────────────┐
│ mempalace mine │ │ mempalace mine │
│ │ │ --mode convos │
└─────┬──────────┘ └────┬──────────────┘
│ │
└───────────────────┬──────────────────────┘
┌──────▼─────────┐
│ ChromaDB │
│ ~/.mempalace/ │
│ palace/ │
└──────┬─────────┘
MCP server (mempalace_*)
AI agents (opencode, claude code, codex, …)
```
**Shared idiom:** *stage-to-cache-then-mine*.
Neither wrapper reimplements the mempalace miner. They each:
1. Curate input (filter / transform / rename).
2. Write it to a deterministic path under `~/.cache/…/<wing>/` with `mtime` preserved (via `cp -p` or explicit `os.utime`).
3. Delegate actual embedding + filing to `mempalace mine`, which already dedups on `source_file` path.
This keeps the wrappers thin. A third wrapper following the same idiom would justify factoring a shared helper library — two does not.
---
## 3. Component details
### `bin/mempalace-docs` (268 lines) — docs-first mining
**Input:** a project directory.
**Output:** palace drawers in `wing_<directory-name>` (or `--wing` override), only from documentation-class files.
What it files: `*.md`, `*.mdx`, `*.rst`, `*.txt`, `*.yml`, `*.yaml`, `*.toml`, selective `*.json`, shell scripts, Dockerfiles, Makefiles, license/notice files.
What it drops: source code (`.py`, `.ts`, `.go`, `.rs`, …), lockfiles, `.git`, `.venv`, `node_modules`, `__pycache__`, build output.
**Implementation notes:**
- Reads `mempalace.yaml` (if present) to discover the actual wing name — avoids drift if someone renamed the wing after init.
- Uses `cp -p` (not symlinks) because the miner skips symlinks (`miner.py` line 828).
- Auto-purges pre-existing drawers whose `source_file` is under the workspace path before re-mining, to prevent doubling on re-runs.
- Upstream [PR #1213](https://github.com/MemPalace/mempalace/pull/1213) will add `exclude_patterns` to `mempalace.yaml` — when merged, this wrapper should shrink to a thin shim.
### `bin/mempalace-session` (349 lines) — opencode → palace bridge
**Input:** the opencode SQLite DB (default `~/.local/share/opencode/opencode.db`).
**Output:** palace drawers in `wing_conversations` (or `--wing` override), one JSONL file per qualifying session.
**Transform pipeline, per session:**
1. Read `session` row (`id`, `title`, `directory`, `time_created`, `time_updated`).
2. Inject synthetic header as first user turn: `[session: <title> | <directory> | <YYYY-MM-DD>]` → makes title/dir/date semantically searchable.
3. For each `message` ordered by `id`:
- Read JSON `data` → get `role` (`user` / `assistant`).
- For each `part` under the message, read JSON `data` → dispatch on `type`:
- `text` → text block.
- `tool` → Claude Code `tool_use` block + deferred `tool_result` as synthetic human message (the mempalace normalizer folds it back into the assistant turn via its `is_tool_only` branch).
- `step-start` / `step-finish` → dropped as noise.
- `reasoning` → kept, prefixed with `[reasoning]`.
4. Serialize as Claude Code JSONL (`{"type": "user"|"assistant", "message": {"content": [...]}}`) — the one convos format the miner already understands.
5. Stage at `<palace-root>/opencode-stage/<wing>/<slug>_<id>.jsonl` with `mtime` = `session.time_updated` (deterministic, stable under dedup).
**Filters:**
- `--min-messages N` (default 3) — drops throwaway `/exit`'d sessions that would flood the palace.
- `--since YYYY-MM-DD` — incremental catch-up.
- `--session <id>` — one-shot mode.
**Then:** invokes `mempalace mine --mode convos` against the cache dir. A post-mine `mempalace repair` is **opt-in** via `--repair` — it is intentionally *not* the default because the in-place HNSW rebuild has corrupted live palaces on past runs. Never pass `--repair` from an unattended schedule.
### `bin/mempalace-pi-session` — pi coding-agent → palace bridge
**Input:** pi's own JSONL session transcripts under `~/.pi/agent/sessions/**/*.jsonl`
(no SQLite export step needed — pi already writes files, unlike opencode).
**Output:** palace drawers in `wing_conversations` (or `--wing` override), same
Claude Code JSONL staging shape as `mempalace-session` above.
The transform pipeline is the same shape as `mempalace-session`'s (synthetic
`[session: title | cwd | date | source: pi]` header, per-message dispatch,
`toolCall``tool_use`, `toolResult``tool_result`, `mtime` copied onto the
staged file for dedup stability) — see the script's own header comment for
the exhaustive per-role mapping. What's architecturally different from the
opencode wrapper is why this one has a **two-phase mode** and a **remote
transport**, neither of which `mempalace-session` needs:
**Two phases, because the palace is single-writer.** `mempalace` enforces
this with a per-palace `flock` (`palace.py`); the CLI's own error is explicit:
`palace ... is held by PID <n> (mempalace-mcp); wait for it to finish`. Unlike
opencode's session mine (which always runs *between* agent sessions, when
nothing else holds the palace), pi's own bridge extension
(`extensions/pi/mempalace.ts`) holds the palace open via its `mempalace-mcp`
subprocess for the *entire* live session — and that extension is also what
triggers the feed, on `session_shutdown` and a debounced `agent_settled`. An
unattended `mempalace mine` invoked from anywhere else during that window
would simply fail. So the wrapper splits:
- `--prepare` — export, threshold, stage (+ `rsync` in remote mode). Never
opens the palace. Prints `MINE_SOURCE=<path>`.
- (default, no `--prepare`) — the above, then mines it. If that mine hits
contention ("is held by"), it's treated as **success, not failure** (exit
0 with an informational message) — the holder's own extension will mine
what got staged. This inverts the CLI's own convention (`MineAlreadyRunning`
→ exit 1) deliberately, because at the wrapper layer the contention has a
benign interpretation the raw CLI can't know about.
The pi extension calls `--prepare` as a subprocess, then feeds `MINE_SOURCE`
to `mempalace_mine` through its own already-open MCP client — the only
process that can write during a live session, because it *is* the lock
holder. This is the mechanism behind [`extensions/pi/README.md` § Automatic
transcript feeding](extensions/pi/README.md#automatic-transcript-feeding).
**Remote transport, because there is no remote-palace CLI.** `mempalace`'s
`--backend` flag selects a *vector store* (chroma/qdrant/pgvector/milvus),
not a remote palace — the only remote surface is the HTTP MCP server
(`mempalace-mcp --transport http`, see [RFC-001](docs/rfc-001-global-palace.md)).
And `mempalace_mine`'s `source` path is expanded **in that server process**,
so a remote server has no way to see this machine's staged exports. `--mode
remote` (auto-selected when `$MEMPALACE_REMOTE_URL` is set) therefore
`rsync`s the stage to a per-device inbox on the palace host, then asks the
server to mine its own local copy of that inbox over the same HTTP
`tools/call` transport the extension uses. Requires
`$MEMPALACE_PI_SSH_TARGET`; see `--help` for the rest
(`MEMPALACE_PI_SSH_CONFIG`, `MEMPALACE_PI_REMOTE_PATH`, `MEMPALACE_PI_DEVICE`).
**Filters:** two gates, both required — stricter than `mempalace-session`'s
single filter because pi's transcripts have a failure mode opencode's don't:
- `--min-messages N` (default 4) — same idea as opencode's filter, raised
because pi's tool loops inflate turn counts fast.
- `--min-assistant-chars N` (default 1000) — counts assistant *text* only,
tool results excluded. Needed because pi expands skill/context text into
the user prompt: an abandoned session can have a huge "user" turn and
almost no assistant output (observed case: 13,380 injected-context user
chars answered with 38 assistant chars), so a message-count-only filter
would have filed 22 pure-noise drawers from that one session. Real
sessions on the same corpus measured 15,900100,000 assistant chars.
---
## 4. Setup recipe (new machine)
Assumes: opencode already installed, `~/.local/share/opencode/opencode.db` exists, `mempalace` CLI installed (v3.3.3+). If mempalace isn't installed yet, [`README.md`](README.md#installing-mempalace-itself-prerequisite) covers the `uv tool install mempalace` flow for both personal machines and the `/opt/uv-tools/` container pattern used by opencode-devbox.
```bash
# 1. Clone mempalace-toolkit (holds the two wrappers in bin/)
git clone ssh://git@gitea.jordbo.se:2222/joakimp/mempalace-toolkit.git ~/mempalace-toolkit
cd ~/mempalace-toolkit
# 2. Install (symlinks bin/* into ~/.local/bin, adds loader to rc file)
./install.sh
# 3. Ensure ~/.local/bin is on PATH (installer warns if not)
export PATH="$HOME/.local/bin:$PATH"
# 4. Mine opencode history into the palace
# (No global init step needed — the palace is created on first write.
# `mempalace init <dir>` is per-project, not global, and is optional.)
mempalace-session --dry-run # preview scope
mempalace-session # do it for real (~20 min for ~60 sessions)
# 5. Mine project docs (per project — run `mempalace init --yes <dir>`
# first if you want to customize the wing name or entity detection)
mempalace-docs /workspace/my_project --dry-run
mempalace-docs /workspace/my_project
# 7. Restart any MCP-connected agent, or call mempalace_reconnect from inside one
```
### Containerized setup (devbox)
The devbox uses two named Docker volumes so these persist across container recreate:
- `devbox-palace``~/.mempalace/palace` (the palace itself)
- `devbox-data``~/.local/share/opencode` (opencode's SQLite DB)
Code at `/workspace/mempalace-toolkit` is a bind mount from the host — survives container recreate and syncs via gitea. The docs staging dir (`~/.cache/mempalace-docs/`) is ephemeral and cheap to rebuild.
The **conversation** staging dirs are not: `<palace-root>/opencode-stage/` and
`<palace-root>/pi-stage/` hold the exact paths the palace's `source_file` dedup keys on, so they
live beside the palace deliberately and share its lifetime. Wiping one forces a full re-mine at
best, and lets a scoped `mempalace sync` prune every drawer mined from it at worst.
**After container recreate**, just re-run `./install.sh` (idempotent) to relink `bin/` into the fresh `~/.local/bin/`.
---
## 5. Operational notes
### Dedup behavior
Both wrappers dedup via `mempalace mine`'s built-in key:
- `mempalace-docs`: keys on `source_file` path + `mtime` → edit a doc, it re-mines; unchanged files are skipped.
- `mempalace-session`: keys on `source_file` path alone (convos miner doesn't check mtime) → a session's JSONL filename is `<slug>_<id>.jsonl`, stable per session, so re-runs skip already-filed sessions. To force re-mining, delete the staging dir.
**Verified:** a second full `mempalace-session` run immediately after the first produces 0 new drawers. The only cost is the post-mine `repair` step (index rebuild — ~5 min on 5k drawers).
### When to re-mine
- `mempalace-docs`: after significant doc changes in a project.
- `mempalace-session`: see the full **Operational Routine** below.
### Operational Routine (the `mempalace-session` workflow)
Until opencode grows session hooks and `hooks_cli.py` grows an opencode harness (see §6), **`mempalace-session` is the entire mechanism that gets opencode conversations into the palace.** Skip it and your session history exists only inside `~/.local/share/opencode/opencode.db` — a local SQLite file that's invisible to `mempalace_search`, vulnerable to volume wipes, and lost if the devbox is replaced.
That makes the routine worth codifying:
#### Triggers (when to run)
| Trigger | What to run | Why |
|---|---|---|
| **Substantive session you want preserved past `/exit`** | `mempalace-session --session <id>` | Targeted save before destructive action; see `~/.local/share/opencode/opencode.db` `session` table for the ID. |
| **Before a container recreate** | `mempalace-session` | The opencode DB lives in a named volume (`devbox-data`) so it normally survives, but a full mine right before is cheap insurance. |
| **Fresh machine, first provisioning** | `mempalace-session --dry-run` then `mempalace-session` | Backfills the whole corpus. Expect ~20 min / 60 sessions. |
| **Periodic sweep** | `mempalace-session` | Weekly catches anything you didn't explicitly save. Dedup is free, so running more often only costs the ~5 min repair. |
| **After upstream mempalace upgrade** | `mempalace-session` + `mempalace repair` | If the miner changed normalization or chunking, re-mine ensures the palace reflects current logic. Rare. |
#### Cadence (how often)
**Default: weekly.** Dedup is free on unchanged sessions, and `wing_conversations` growth is roughly linear in user activity. Weekly is frequent enough that searches almost always include recent context, and infrequent enough that the cost is negligible.
**Daily** is fine. Repair is now opt-in (`--repair`) and should never be set on an unattended schedule — run it manually from a quiet session if you suspect stale HNSW state.
**Monthly** is too infrequent. You'll search for "that thing we discussed last Tuesday" and miss it.
#### Relationship to the session lifecycle
`mempalace-session` is **offline, inter-session maintenance** — it runs between agent sessions, not during them. It does not replace the in-session habits from the consumer-side `mempalace` skill:
| Habit | When | Who |
|---|---|---|
| Wake-up search (load recent diary) | Agent session start | Agent, during session |
| Wind-down diary write | Agent session end | Agent, during session |
| `mempalace-session` mine | Between sessions (manual or scheduled) | Operator or automation |
The first two are live; the third is batched. They're complementary, not alternatives. The next subsection explains why both matter.
#### Diary vs session mine: why keep both?
A reasonable question: *"if every session is mined into `wing_conversations` anyway, what's the point of the agent also writing a diary entry?"* They're not redundant. They answer different questions and cover each other's failure modes.
| | Session mine (`wing_conversations`) | Diary (`wing_<agent>`) |
|---|---|---|
| Content | Every turn verbatim — prompts, responses, tool calls, dead ends, typos | Curated summary — what was decided, discovered, left pending |
| Granularity | One session ≈ 50200 drawers | One session ≈ 1 drawer |
| Compression | None (raw JSONL → normalized turns) | High (AAAK dialect — dots + pipes + entity codes, ~30× reduction) |
| Written by | Nothing — extracted from `opencode.db` | The agent that lived the session, at wind-down |
| Signal density | High noise (wrong turns, corrections, `/exit`'d threads) | High signal (agent's editorial judgment of what mattered) |
| Retrieval pattern | Semantic search (`mempalace_search("topic X")`) | Recency scan (`mempalace_diary_read(last_n=5)`) |
| Answers the question | *"What did we say exactly?"* | *"What did we accomplish / learn / decide?"* |
The distinguishing property of a diary entry is **editorial judgment by the author**. The diary captures things that were *never said aloud during the session* — meta-observations the agent made about the session as a whole:
- *"this pattern came up again, worth remembering"*
- *"user caught the bug before I shipped it — lesson: verify CLI examples against `--help` first"*
- *"10 commits across 3 repos today, all pushed"*
- *"healthy interruption: user stopped me before a long-running step"*
These are thoughts *about* the session, not utterances *during* it. Mining the raw turns will never surface them because the exact words were never spoken — they're the agent's reflection at wind-down.
**Three scenarios where the distinction matters in practice:**
1. **Wake-up token economics.** Reading `mempalace_diary_read(last_n=5)` returns five dense drawers, maybe 12k tokens total, 100% signal. Matching that orientation from the session mine would require semantic-searching for recent topics and reading chunks of raw turns — hundreds of drawers, tens of thousands of tokens, 90% noise.
2. **"What did we decide?" vs. "what did we say?"** If you ask *"when did we decide to split `mempalace-toolkit` from `cli_utils`?"* the diary gives you the crisp answer (date, trigger, rationale). The session mine gives you the actual seven-turn conversation that led up to the decision, including the turns where alternatives were considered. Both useful; different questions.
3. **Redundancy as safety.** If the agent `/exit`s without writing a diary (heuristic save missed it, no upstream hook), the session mine still catches the raw content. If `mempalace-session` hasn't run this week, the diary still captures the session's essence. The two systems cover each other's failure modes.
**Practical implications for how you work with mempalace:**
- **Don't skip diary writing** just because sessions are mined. A session without a diary entry is a session the next agent can read word-for-word but has no compressed summary of — expensive to orient against.
- **Don't skip session mining** just because agents write diaries. Diaries miss content (especially on `/exit`), and semantic search over raw turns is valuable when "what did we say exactly?" is the right question.
- **Do both, and let them specialize.** Treat the diary as your release notes (editorial, curated, recency-scanned) and the session mine as your git log (raw, searchable, complete). A repo keeps both; so should the palace.
If anything, automating session mining *increases* the value of diary entries. The agent can focus the diary on the parts mining cannot capture — meta-observations, self-critique, pattern noticing, pending work — rather than re-stating content the mine already has.
#### Automation
Pick one:
1. **systemd user timer** (recommended on Linux). Survives reboots, optional `Persistent=true` catch-up, logs to `journalctl`, background I/O priority. Templates in [`contrib/systemd/`](contrib/systemd/).
2. **launchd user agent** (recommended on macOS). The macOS-native equivalent — runs without a login session, logs to `~/Library/Logs/`, single-instance guarantees, `ProcessType=Background` throttling. Templates in [`contrib/launchd/`](contrib/launchd/).
3. **cron** (simplest, works on BSD and systemd-less Linux distros). Templates in [`contrib/cron/`](contrib/cron/).
4. **Manual** — run `mempalace-session` opportunistically. Fine on machines where you're in and out frequently; less fine on long-running devboxes.
Install recipes, verification commands, and uninstall steps for all four are in [`contrib/README.md`](contrib/README.md).
Quick-start (systemd user timer, Linux):
```bash
mkdir -p ~/.config/systemd/user
cp contrib/systemd/*.{service,timer} ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now mempalace-session.timer
# Optional on headless boxes: keep timer running when logged out
sudo loginctl enable-linger "$USER"
systemctl --user list-timers mempalace-session.timer
```
Quick-start (launchd user agent, macOS):
```bash
sed "s|USER|$USER|g" contrib/launchd/se.jordbo.mempalace-session.plist \
> ~/Library/LaunchAgents/se.jordbo.mempalace-session.plist
mkdir -p ~/Library/Logs
launchctl bootstrap "gui/$(id -u)" ~/Library/LaunchAgents/se.jordbo.mempalace-session.plist
launchctl enable "gui/$(id -u)/se.jordbo.mempalace-session"
launchctl list | grep mempalace-session
```
Quick-start (cron):
```bash
sed "s|USER|$USER|g" contrib/cron/mempalace-session.cron \
| (crontab -l 2>/dev/null; cat) | crontab -
mkdir -p ~/.cache/mempalace-logs
```
#### Verification
After any run (manual or scheduled), confirm the palace grew:
```bash
mempalace-session --dry-run # should list sessions
# Or from inside a live MCP client:
# mempalace_status — wing_conversations count
# mempalace_reconnect — refresh index after mine
```
A healthy run produces one of:
- **First run on fresh corpus**: several hundred to several thousand new drawers.
- **Incremental run**: zero to a few dozen new drawers (whatever grew since last run).
- **Rerun with no new activity**: zero new drawers, only the repair step runs.
A run that files far more drawers than expected may indicate a staging-dir wipe (forcing a full re-mine) — check `<palace-root>/opencode-stage/<wing>/` modification times.
### Cost profile (reference)
Measured on a ~10-day opencode corpus of 140 sessions / 1491 messages / 4656 parts:
- Dry run: seconds.
- Full mine: **21 minutes** (38 min user CPU). Produced 2378 drawers from 62 qualifying sessions.
- Dedup re-run: mine step instant; only the repair runs (~5 min).
Scaling is roughly linear in message count. Budget ~20 minutes per 60-session batch.
### Common failure modes
| Symptom | Cause | Fix |
| ---------------------------------------------- | ----------------------------------------------------- | --------------------------------------------------------- |
| `mempalace-session: command not found` after container recreate | `~/.local/bin` wiped with container | `cd ~/mempalace-toolkit && ./install.sh` |
| Search errors "Error finding id" post-mine | Stale HNSW index | `mempalace repair --yes` + `mempalace_reconnect` from MCP |
| Drawers doubled after re-mining a project | Someone renamed the wing or ran raw `mempalace mine` alongside the wrapper | Inspect `embedding_metadata` in `chroma.sqlite3`; purge duplicates by source prefix, then `mempalace repair` |
| Sessions missing from palace | Session has fewer than `--min-messages` messages | Lower the threshold or `--session <id>` explicitly |
---
## 6. Upstream roadmap
These gaps should ideally close upstream, making the wrappers thinner or obsolete:
1. **[MemPalace PR #1213](https://github.com/MemPalace/mempalace/pull/1213)** — `exclude_patterns` in `mempalace.yaml`. When merged, `mempalace-docs` shrinks to a thin shim (or disappears) since exclude-by-extension becomes a first-class config.
2. **Opencode session hooks** — [PR #16598](https://github.com/anomalyco/opencode/pull/16598) (session.stopping), [PR #16769](https://github.com/anomalyco/opencode/pull/16769) (shutdown), [PR #15224](https://github.com/anomalyco/opencode/pull/15224) (session.start), [issue #23503](https://github.com/anomalyco/opencode/issues/23503) (session.turn.completed). When at least one merges, opencode can fire hooks mempalace can receive.
3. **Opencode harness in `hooks_cli.py`** — mempalace's hooks CLI only knows `claude-code` + `codex` today. Adding `opencode` would let the auto-save diary path work on opencode too. Pairs with #2 above.
4. **SQLite mode for `mempalace mine --mode convos`** — if upstream ever adds direct SQLite ingest for opencode, `mempalace-session` loses its reason to exist (the export-to-JSONL dance goes away).
When #1 merges, retire `mempalace-docs` to a thin shim. When #2 + #3 land together, `mempalace-session` becomes a manual-only fallback (cron / backfill) while hooks handle live saves. (`mempalace-pi-session` has no equivalent entry here: pi's bridge extension already self-triggers the feed with no upstream dependency — see §3 — **provided the deployed copy is ≥ `29e660e` (2026-08-12); older baked images cannot feed, so `contrib/`'s schedulers remain load-bearing there**.)
Separately tracked in [`docs/rfc-001-global-palace.md`](docs/rfc-001-global-palace.md): moving from one palace *per machine* to a **single primary palace with per-machine local fallback** (`mempalace-edge`). That RFC also records upstream items of its own — server-side `origin_device` provenance stamped from a per-device credential, per-wing ACLs, a `mempalace_kg_supersede` tool-classification fix, and a guard against running `mempalace sync` on a shared palace.
---
## 7. See also
- [`docs/rfc-001-global-palace.md`](docs/rfc-001-global-palace.md) — design for a fleet-wide primary palace with offline-capable local fallback. **Read before attempting to "centralize" or "sync" palaces** — it documents two silently destructive footguns (`--palace` vs `MEMPALACE_PALACE_PATH`, and `mempalace sync`).
- [`README.md`](README.md) — human-facing quickstart + per-tool usage reference.
- [`AGENTS.md`](AGENTS.md) — repo conventions for AI agents modifying this codebase.
- [`SKILL.md`](SKILL.md) — agent skill (producer side), symlinked into `~/.agents/skills/opencode-mempalace-bridge/` by `install.sh`.
- [`extensions/pi/README.md`](extensions/pi/README.md) — pi coding-agent bring-up: the MemPalace MCP bridge extension, mosh-friendly keybindings, settings template for starting pi without `--model`, and the `~/.config/pi/.env` + zsh loader pattern for AWS env vars. Out of scope for this document (which is producer-side feeding), but linked from `install.sh` which handles both.
- `~/.agents/skills/mempalace/SKILL.md` — agent skill for the **consumer** side (searching, diary, KG). Pair with `SKILL.md` in this repo.
- [`cli_utils`](https://gitea.jordbo.se/joakimp/cli_utils) — sibling repo: shell quality-of-life tools. Origin of these wrappers before the 2026-04-30 split.