mempalace-toolkit/ARCHITECTURE.md

# MemPalace Feeding Architecture

This repository wires [opencode](https://github.com/anomalyco/opencode) and arbitrary project directories into [MemPalace](https://github.com/MemPalace/mempalace) via two thin wrappers in `bin/`. This document explains why they exist and how they fit together.

**Audience:** someone setting up a new machine (or reviewing what's already set up) and asking "how does the palace actually get fed?". Pairs with the `mempalace` agent skill, which covers the *consumer* side (searching, diary, KG). This document covers the *producer* side.

---

## 1. The problem

MemPalace is a persistent memory layer for AI agents — vector search over drawers (chunks of verbatim content), a knowledge graph, and per-agent diaries, all behind an MCP server. To be useful it has to be *fed*: project docs, conversation transcripts, session summaries.

The stock mempalace CLI has two feeders:

| Feeder                                | What it ingests                                          | Gap                                                                                                    |
| ------------------------------------- | -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
| `mempalace mine` (default mode)       | Any "readable" file in a directory (code + docs + misc)  | Mines source code indiscriminately → embedding index floods with low-signal `__init__` fragments.      |
| `mempalace mine --mode convos`        | Claude Code JSONL, Claude.ai JSON, ChatGPT JSON, Slack, Codex JSONL | No opencode support. No SQLite support. Opencode persists its history in SQLite, not JSONL.            |

And one auto-save path:

| Feeder                                | Harnesses supported                                      | Gap                                                                                                    |
| ------------------------------------- | -------------------------------------------------------- | ------------------------------------------------------------------------------------------------------ |
| `hooks_cli.py` (session-stop hooks)   | `claude-code`, `codex`                                   | No `opencode` harness → `/exit` mid-session leaves no diary entry behind.                              |

So on a machine using opencode + the "docs-first palace hygiene" policy, three gaps bite:

1. Mining a project floods the palace with source code we don't want.
2. Opencode session history is trapped in SQLite, invisible to `mine --mode convos`.
3. There's no auto-save on session stop — any persistence is best-effort heuristic.

The two wrappers in `bin/` close gaps **1** and **2**. Gap **3** is upstream work (see §6).

---

## 2. The architecture

```
  Project dirs (/workspace/*)                Opencode SQLite DB
  ├── *.md                                   ~/.local/share/opencode/opencode.db
  ├── *.yaml                                 ├── session  (id, title, directory, time_created/updated)
  ├── Dockerfile                             ├── message  (session_id, data JSON w/ role)
  └── …                                      └── part     (message_id, data JSON w/ type: text|tool|…)
        │                                          │
        │                                          │
  ┌─────▼──────────┐                          ┌────▼──────────────┐
  │ mempalace-docs │                          │ mempalace-session │
  │ (bin/)         │                          │ (bin/)            │
  │                │                          │                   │
  │ stage docs     │                          │ export each       │
  │ only via cp -p │                          │ session as Claude │
  │ to cache dir   │                          │ Code JSONL to     │
  │                │                          │ cache dir         │
  └─────┬──────────┘                          └────┬──────────────┘
        │                                          │
        │  ~/.cache/mempalace-docs/<wing>/          │  ~/.cache/mempalace-session/<wing>/
        │                                          │
  ┌─────▼──────────┐                          ┌────▼──────────────┐
  │ mempalace mine │                          │ mempalace mine    │
  │                │                          │ --mode convos     │
  └─────┬──────────┘                          └────┬──────────────┘
        │                                          │
        └───────────────────┬──────────────────────┘
                            │
                     ┌──────▼─────────┐
                     │ ChromaDB       │
                     │ ~/.mempalace/  │
                     │   palace/      │
                     └──────┬─────────┘
                            │
                     MCP server (mempalace_*)
                            │
                     AI agents (opencode, claude code, codex, …)
```

**Shared idiom:** *stage-to-cache-then-mine*.

Neither wrapper reimplements the mempalace miner. They each:

1. Curate input (filter / transform / rename).
2. Write it to a deterministic path under `~/.cache/…/<wing>/` with `mtime` preserved (via `cp -p` or explicit `os.utime`).
3. Delegate actual embedding + filing to `mempalace mine`, which already dedups on `source_file` path.

This keeps the wrappers thin. A third wrapper following the same idiom would justify factoring a shared helper library — two does not.

---

## 3. Component details

### `bin/mempalace-docs` (268 lines) — docs-first mining

**Input:** a project directory.
**Output:** palace drawers in `wing_<directory-name>` (or `--wing` override), only from documentation-class files.

What it files: `*.md`, `*.mdx`, `*.rst`, `*.txt`, `*.yml`, `*.yaml`, `*.toml`, selective `*.json`, shell scripts, Dockerfiles, Makefiles, license/notice files.

What it drops: source code (`.py`, `.ts`, `.go`, `.rs`, …), lockfiles, `.git`, `.venv`, `node_modules`, `__pycache__`, build output.

**Implementation notes:**

- Reads `mempalace.yaml` (if present) to discover the actual wing name — avoids drift if someone renamed the wing after init.
- Uses `cp -p` (not symlinks) because the miner skips symlinks (`miner.py` line 828).
- Auto-purges pre-existing drawers whose `source_file` is under the workspace path before re-mining, to prevent doubling on re-runs.
- Upstream [PR #1213](https://github.com/MemPalace/mempalace/pull/1213) will add `exclude_patterns` to `mempalace.yaml` — when merged, this wrapper should shrink to a thin shim.

### `bin/mempalace-session` (349 lines) — opencode → palace bridge

**Input:** the opencode SQLite DB (default `~/.local/share/opencode/opencode.db`).
**Output:** palace drawers in `wing_conversations` (or `--wing` override), one JSONL file per qualifying session.

**Transform pipeline, per session:**

1. Read `session` row (`id`, `title`, `directory`, `time_created`, `time_updated`).
2. Inject synthetic header as first user turn: `[session: <title> | <directory> | <YYYY-MM-DD>]` → makes title/dir/date semantically searchable.
3. For each `message` ordered by `id`:
   - Read JSON `data` → get `role` (`user` / `assistant`).
   - For each `part` under the message, read JSON `data` → dispatch on `type`:
     - `text` → text block.
     - `tool` → Claude Code `tool_use` block + deferred `tool_result` as synthetic human message (the mempalace normalizer folds it back into the assistant turn via its `is_tool_only` branch).
     - `step-start` / `step-finish` → dropped as noise.
     - `reasoning` → kept, prefixed with `[reasoning]`.
4. Serialize as Claude Code JSONL (`{"type": "user"|"assistant", "message": {"content": [...]}}`) — the one convos format the miner already understands.
5. Stage at `~/.cache/mempalace-session/<wing>/<slug>_<id>.jsonl` with `mtime` = `session.time_updated` (deterministic, stable under dedup).

**Filters:**

- `--min-messages N` (default 3) — drops throwaway `/exit`'d sessions that would flood the palace.
- `--since YYYY-MM-DD` — incremental catch-up.
- `--session <id>` — one-shot mode.

**Then:** invokes `mempalace mine --mode convos` against the cache dir. A post-mine `mempalace repair` is **opt-in** via `--repair` — it is intentionally *not* the default because the in-place HNSW rebuild has corrupted live palaces on past runs. Never pass `--repair` from an unattended schedule.

---

## 4. Setup recipe (new machine)

Assumes: opencode already installed, `~/.local/share/opencode/opencode.db` exists, `mempalace` CLI installed (v3.3.3+). If mempalace isn't installed yet, [`README.md`](README.md#installing-mempalace-itself-prerequisite) covers the `uv tool install mempalace` flow for both personal machines and the `/opt/uv-tools/` container pattern used by opencode-devbox.

```bash
# 1. Clone mempalace-toolkit (holds the two wrappers in bin/)
git clone ssh://git@gitea.jordbo.se:2222/joakimp/mempalace-toolkit.git ~/mempalace-toolkit
cd ~/mempalace-toolkit

# 2. Install (symlinks bin/* into ~/.local/bin, adds loader to rc file)
./install.sh

# 3. Ensure ~/.local/bin is on PATH (installer warns if not)
export PATH="$HOME/.local/bin:$PATH"

# 4. Mine opencode history into the palace
#    (No global init step needed — the palace is created on first write.
#     `mempalace init <dir>` is per-project, not global, and is optional.)
mempalace-session --dry-run              # preview scope
mempalace-session                        # do it for real (~20 min for ~60 sessions)

# 5. Mine project docs (per project — run `mempalace init --yes <dir>`
#    first if you want to customize the wing name or entity detection)
mempalace-docs /workspace/my_project --dry-run
mempalace-docs /workspace/my_project

# 7. Restart any MCP-connected agent, or call mempalace_reconnect from inside one
```

### Containerized setup (devbox)

The devbox uses two named Docker volumes so these persist across container recreate:

- `devbox-palace` → `~/.mempalace/palace` (the palace itself)
- `devbox-data` → `~/.local/share/opencode` (opencode's SQLite DB)

Code at `/workspace/mempalace-toolkit` is a bind mount from the host — survives container recreate and syncs via gitea. Staging directories (`~/.cache/mempalace-{docs,session}/`) are ephemeral but cheap to rebuild.

**After container recreate**, just re-run `./install.sh` (idempotent) to relink `bin/` into the fresh `~/.local/bin/`.

---

## 5. Operational notes

### Dedup behavior

Both wrappers dedup via `mempalace mine`'s built-in key:

- `mempalace-docs`: keys on `source_file` path + `mtime` → edit a doc, it re-mines; unchanged files are skipped.
- `mempalace-session`: keys on `source_file` path alone (convos miner doesn't check mtime) → a session's JSONL filename is `<slug>_<id>.jsonl`, stable per session, so re-runs skip already-filed sessions. To force re-mining, delete the staging dir.

**Verified:** a second full `mempalace-session` run immediately after the first produces 0 new drawers. The only cost is the post-mine `repair` step (index rebuild — ~5 min on 5k drawers).

### When to re-mine

- `mempalace-docs`: after significant doc changes in a project.
- `mempalace-session`: see the full **Operational Routine** below.

### Operational Routine (the `mempalace-session` workflow)

Until opencode grows session hooks and `hooks_cli.py` grows an opencode harness (see §6), **`mempalace-session` is the entire mechanism that gets opencode conversations into the palace.** Skip it and your session history exists only inside `~/.local/share/opencode/opencode.db` — a local SQLite file that's invisible to `mempalace_search`, vulnerable to volume wipes, and lost if the devbox is replaced.

That makes the routine worth codifying:

#### Triggers (when to run)

| Trigger | What to run | Why |
|---|---|---|
| **Substantive session you want preserved past `/exit`** | `mempalace-session --session <id>` | Targeted save before destructive action; see `~/.local/share/opencode/opencode.db` `session` table for the ID. |
| **Before a container recreate** | `mempalace-session` | The opencode DB lives in a named volume (`devbox-data`) so it normally survives, but a full mine right before is cheap insurance. |
| **Fresh machine, first provisioning** | `mempalace-session --dry-run` then `mempalace-session` | Backfills the whole corpus. Expect ~20 min / 60 sessions. |
| **Periodic sweep** | `mempalace-session` | Weekly catches anything you didn't explicitly save. Dedup is free, so running more often only costs the ~5 min repair. |
| **After upstream mempalace upgrade** | `mempalace-session` + `mempalace repair` | If the miner changed normalization or chunking, re-mine ensures the palace reflects current logic. Rare. |

#### Cadence (how often)

**Default: weekly.** Dedup is free on unchanged sessions, and `wing_conversations` growth is roughly linear in user activity. Weekly is frequent enough that searches almost always include recent context, and infrequent enough that the cost is negligible.

**Daily** is fine. Repair is now opt-in (`--repair`) and should never be set on an unattended schedule — run it manually from a quiet session if you suspect stale HNSW state.

**Monthly** is too infrequent. You'll search for "that thing we discussed last Tuesday" and miss it.

#### Relationship to the session lifecycle

`mempalace-session` is **offline, inter-session maintenance** — it runs between agent sessions, not during them. It does not replace the in-session habits from the consumer-side `mempalace` skill:

| Habit | When | Who |
|---|---|---|
| Wake-up search (load recent diary) | Agent session start | Agent, during session |
| Wind-down diary write | Agent session end | Agent, during session |
| `mempalace-session` mine | Between sessions (manual or scheduled) | Operator or automation |

The first two are live; the third is batched. They're complementary, not alternatives. The next subsection explains why both matter.

#### Diary vs session mine: why keep both?

A reasonable question: *"if every session is mined into `wing_conversations` anyway, what's the point of the agent also writing a diary entry?"* They're not redundant. They answer different questions and cover each other's failure modes.

|  | Session mine (`wing_conversations`) | Diary (`wing_<agent>`) |
|---|---|---|
| Content | Every turn verbatim — prompts, responses, tool calls, dead ends, typos | Curated summary — what was decided, discovered, left pending |
| Granularity | One session ≈ 50–200 drawers | One session ≈ 1 drawer |
| Compression | None (raw JSONL → normalized turns) | High (AAAK dialect — dots + pipes + entity codes, ~30× reduction) |
| Written by | Nothing — extracted from `opencode.db` | The agent that lived the session, at wind-down |
| Signal density | High noise (wrong turns, corrections, `/exit`'d threads) | High signal (agent's editorial judgment of what mattered) |
| Retrieval pattern | Semantic search (`mempalace_search("topic X")`) | Recency scan (`mempalace_diary_read(last_n=5)`) |
| Answers the question | *"What did we say exactly?"* | *"What did we accomplish / learn / decide?"* |

The distinguishing property of a diary entry is **editorial judgment by the author**. The diary captures things that were *never said aloud during the session* — meta-observations the agent made about the session as a whole:

- *"this pattern came up again, worth remembering"*
- *"user caught the bug before I shipped it — lesson: verify CLI examples against `--help` first"*
- *"10 commits across 3 repos today, all pushed"*
- *"healthy interruption: user stopped me before a long-running step"*

These are thoughts *about* the session, not utterances *during* it. Mining the raw turns will never surface them because the exact words were never spoken — they're the agent's reflection at wind-down.

**Three scenarios where the distinction matters in practice:**

1. **Wake-up token economics.** Reading `mempalace_diary_read(last_n=5)` returns five dense drawers, maybe 1–2k tokens total, 100% signal. Matching that orientation from the session mine would require semantic-searching for recent topics and reading chunks of raw turns — hundreds of drawers, tens of thousands of tokens, 90% noise.

2. **"What did we decide?" vs. "what did we say?"** If you ask *"when did we decide to split `mempalace-toolkit` from `cli_utils`?"* the diary gives you the crisp answer (date, trigger, rationale). The session mine gives you the actual seven-turn conversation that led up to the decision, including the turns where alternatives were considered. Both useful; different questions.

3. **Redundancy as safety.** If the agent `/exit`s without writing a diary (heuristic save missed it, no upstream hook), the session mine still catches the raw content. If `mempalace-session` hasn't run this week, the diary still captures the session's essence. The two systems cover each other's failure modes.

**Practical implications for how you work with mempalace:**

- **Don't skip diary writing** just because sessions are mined. A session without a diary entry is a session the next agent can read word-for-word but has no compressed summary of — expensive to orient against.
- **Don't skip session mining** just because agents write diaries. Diaries miss content (especially on `/exit`), and semantic search over raw turns is valuable when "what did we say exactly?" is the right question.
- **Do both, and let them specialize.** Treat the diary as your release notes (editorial, curated, recency-scanned) and the session mine as your git log (raw, searchable, complete). A repo keeps both; so should the palace.

If anything, automating session mining *increases* the value of diary entries. The agent can focus the diary on the parts mining cannot capture — meta-observations, self-critique, pattern noticing, pending work — rather than re-stating content the mine already has.

#### Automation

Pick one:

1. **systemd user timer** (recommended on Linux). Survives reboots, optional `Persistent=true` catch-up, logs to `journalctl`, background I/O priority. Templates in [`contrib/systemd/`](contrib/systemd/).
2. **launchd user agent** (recommended on macOS). The macOS-native equivalent — runs without a login session, logs to `~/Library/Logs/`, single-instance guarantees, `ProcessType=Background` throttling. Templates in [`contrib/launchd/`](contrib/launchd/).
3. **cron** (simplest, works on BSD and systemd-less Linux distros). Templates in [`contrib/cron/`](contrib/cron/).
4. **Manual** — run `mempalace-session` opportunistically. Fine on machines where you're in and out frequently; less fine on long-running devboxes.

Install recipes, verification commands, and uninstall steps for all four are in [`contrib/README.md`](contrib/README.md).

Quick-start (systemd user timer, Linux):

```bash
mkdir -p ~/.config/systemd/user
cp contrib/systemd/*.{service,timer} ~/.config/systemd/user/
systemctl --user daemon-reload
systemctl --user enable --now mempalace-session.timer
# Optional on headless boxes: keep timer running when logged out
sudo loginctl enable-linger "$USER"
systemctl --user list-timers mempalace-session.timer
```

Quick-start (launchd user agent, macOS):

```bash
sed "s|USER|$USER|g" contrib/launchd/se.jordbo.mempalace-session.plist \
  > ~/Library/LaunchAgents/se.jordbo.mempalace-session.plist
mkdir -p ~/Library/Logs
launchctl bootstrap "gui/$(id -u)" ~/Library/LaunchAgents/se.jordbo.mempalace-session.plist
launchctl enable "gui/$(id -u)/se.jordbo.mempalace-session"
launchctl list | grep mempalace-session
```

Quick-start (cron):

```bash
sed "s|USER|$USER|g" contrib/cron/mempalace-session.cron \
  | (crontab -l 2>/dev/null; cat) | crontab -
mkdir -p ~/.cache/mempalace-session
```

#### Verification

After any run (manual or scheduled), confirm the palace grew:

```bash
mempalace-session --dry-run                     # should list sessions
# Or from inside a live MCP client:
#   mempalace_status                             — wing_conversations count
#   mempalace_reconnect                          — refresh index after mine
```

A healthy run produces one of:
- **First run on fresh corpus**: several hundred to several thousand new drawers.
- **Incremental run**: zero to a few dozen new drawers (whatever grew since last run).
- **Rerun with no new activity**: zero new drawers, only the repair step runs.

A run that files far more drawers than expected may indicate a staging-dir wipe (forcing a full re-mine) — check `~/.cache/mempalace-session/<wing>/` modification times.

### Cost profile (reference)

Measured on a ~10-day opencode corpus of 140 sessions / 1491 messages / 4656 parts:

- Dry run: seconds.
- Full mine: **21 minutes** (38 min user CPU). Produced 2378 drawers from 62 qualifying sessions.
- Dedup re-run: mine step instant; only the repair runs (~5 min).

Scaling is roughly linear in message count. Budget ~20 minutes per 60-session batch.

### Common failure modes

| Symptom                                        | Cause                                                 | Fix                                                       |
| ---------------------------------------------- | ----------------------------------------------------- | --------------------------------------------------------- |
| `mempalace-session: command not found` after container recreate | `~/.local/bin` wiped with container                   | `cd ~/mempalace-toolkit && ./install.sh`                  |
| Search errors "Error finding id" post-mine     | Stale HNSW index                                      | `mempalace repair --yes` + `mempalace_reconnect` from MCP |
| Drawers doubled after re-mining a project      | Someone renamed the wing or ran raw `mempalace mine` alongside the wrapper | Inspect `embedding_metadata` in `chroma.sqlite3`; purge duplicates by source prefix, then `mempalace repair` |
| Sessions missing from palace                   | Session has fewer than `--min-messages` messages      | Lower the threshold or `--session <id>` explicitly        |

---

## 6. Upstream roadmap

These gaps should ideally close upstream, making the wrappers thinner or obsolete:

1. **[MemPalace PR #1213](https://github.com/MemPalace/mempalace/pull/1213)** — `exclude_patterns` in `mempalace.yaml`. When merged, `mempalace-docs` shrinks to a thin shim (or disappears) since exclude-by-extension becomes a first-class config.
2. **Opencode session hooks** — [PR #16598](https://github.com/anomalyco/opencode/pull/16598) (session.stopping), [PR #16769](https://github.com/anomalyco/opencode/pull/16769) (shutdown), [PR #15224](https://github.com/anomalyco/opencode/pull/15224) (session.start), [issue #23503](https://github.com/anomalyco/opencode/issues/23503) (session.turn.completed). When at least one merges, opencode can fire hooks mempalace can receive.
3. **Opencode harness in `hooks_cli.py`** — mempalace's hooks CLI only knows `claude-code` + `codex` today. Adding `opencode` would let the auto-save diary path work on opencode too. Pairs with #2 above.
4. **SQLite mode for `mempalace mine --mode convos`** — if upstream ever adds direct SQLite ingest for opencode, `mempalace-session` loses its reason to exist (the export-to-JSONL dance goes away).

When #1 merges, retire `mempalace-docs` to a thin shim. When #2 + #3 land together, `mempalace-session` becomes a manual-only fallback (cron / backfill) while hooks handle live saves.

---

## 7. See also

- [`README.md`](README.md) — human-facing quickstart + per-tool usage reference.
- [`AGENTS.md`](AGENTS.md) — repo conventions for AI agents modifying this codebase.
- [`SKILL.md`](SKILL.md) — agent skill (producer side), symlinked into `~/.agents/skills/opencode-mempalace-bridge/` by `install.sh`.
- `~/.agents/skills/mempalace/SKILL.md` — agent skill for the **consumer** side (searching, diary, KG). Pair with `SKILL.md` in this repo.
- [`cli_utils`](https://gitea.jordbo.se/joakimp/cli_utils) — sibling repo: shell quality-of-life tools. Origin of these wrappers before the 2026-04-30 split.