Joakim Persson
f60cf9c732
bin/mempalace-census classifies a palace on disk into the RFC 002 §2 classes
(mined / diary / agent-authored) and emits a human report or a --json manifest
that feeds Phases B/C. Read-only: every sqlite handle is opened mode=ro, no
-wal/-shm is created, safe against a live mempalace-serve. Reads LOCAL DISK only
and warns if MEMPALACE_REMOTE_URL is set, so a local census can't be mistaken
for a remote one.
The design point is that it SELF-VERIFIES instead of trusting ids.py's
docstrings: for every replayable drawer it reassembles content from chunks,
recomputes the upstream id and compares to the stored id. That single check
covers the id recipe, the chunk reassembly order and the classifier at once --
176/176 accounted for on the reference palace -- and it falsified three things
the RFC previously asserted from a docs-only reading (now RFC 002 §2.1):
(a) The hash input is LENGTH-PREFIXED, not "|"-joined. ids.py:31 defines
_DELIM = "|" and the make_* docstrings describe f"{wing}|{room}|{content}",
but _DELIM is dead code and _delimited_sha256 builds
"".join(f"{len(part)}:{part}"). Measured: length-prefixed reproduces real
ids 5/5, pipe-joined 0/5. Diary ids differ again -- a PLAIN sha256.
(b) id_recipe is NOT a mined-only marker. It looked like a clean
discriminator (same 14,586 count as source_file) but the server stamps
'v3' on content ids too, so classifying on `source_file OR id_recipe`
swallowed all 60 agent-authored drawers into MINED -- the dangerous
direction, since Phase C would try to re-mine drawers that have no source
file and silently drop them. Discriminator is a TRUTHY source_file (the
writer stores "" rather than omitting the key), cross-checked against the
miner-only keys source_mtime / normalize_version; disagreement is now a
first-class warning.
(c) Content ids DRIFT: 9 of 60 agent-authored drawers no longer reproduce
their own id, because update_drawer preserves the id while rewriting and
re-chunking. So "recompute the content id and skip if present" -- the
strategy this RFC specified for regime A -- misses every drifted drawer
and duplicates it. Phase C must key on the STORED id. Flagged as
edited_since_filing in the manifest so Phase C can be tested on them.
Also corrects §4.1's headline number: 15,949 mined / 98.9% was reconstructible
exactly as 16,338 (all embeddings rows) - 192 (diary rows) - 197 (agent rows),
i.e. it counted rows rather than parent drawers AND spanned both collections,
absorbing all 1,560 non-joinable mempalace_closets rows into the mined total.
Correct figures: 14,389 mined / 116 diary / 60 agent-authored = 14,565 parents,
replay surface 176 (1.2%). Two rules now enforced in the tool: always filter by
collection (one sqlite file holds both), and always say whether a count is rows
or parent drawers -- a chunked drawer contributes N rows and no parent row.
One implementation trap worth recording: Chroma splits metadata across
string_value and int_value, so reading only string_value nulls every numeric key
(source_mtime, chunk_index, line_start) -- which made the miner-marker
cross-check report 100% conflict until the loader coalesced the two columns.
601 lines
36 KiB
Markdown
601 lines
36 KiB
Markdown
# mempalace-toolkit
|
||
|
||
Producer-side tooling for [MemPalace](https://github.com/MemPalace/mempalace) — bridges that feed opencode session history and project documentation into the palace. Pairs with the consumer-side [`mempalace` agent skill](https://github.com/MemPalace/mempalace).
|
||
|
||
**What this repo contains:**
|
||
|
||
- `bin/mempalace-session` — exports [opencode](https://github.com/anomalyco/opencode) session history from its local SQLite DB to Claude Code JSONL, then mines it via `mempalace mine --mode convos`.
|
||
- `bin/mempalace-pi-session` — the same idea for the [pi coding-agent](https://github.com/earendil-works/pi): exports its native JSONL session transcripts and mines them the same way. Unlike `mempalace-session`, this one is normally invoked *for you* — the pi bridge extension (below) runs it automatically on `session_shutdown` and a debounced `agent_settled`, so most pi machines never need the `contrib/` scheduling templates at all. See [`mempalace-pi-session`](#mempalace-pi-session) below and [`extensions/pi/README.md` § Automatic transcript feeding](extensions/pi/README.md#automatic-transcript-feeding).
|
||
- `bin/mempalace-docs` — mines project directories into MemPalace while excluding source code, keeping the palace signal-dense.
|
||
- `bin/mempalace-census` — read-only census of a palace **on disk**: classifies every drawer as mined / diary / agent-authored, so you know what a cross-machine join would actually move (and what dedupes it) before writing anything. Implements [RFC 002](docs/rfc-002-joiner.md) Phase A. Never writes — safe against a live palace.
|
||
- [`ARCHITECTURE.md`](ARCHITECTURE.md) — **canonical spec**: architecture diagram, component details, setup recipe, operational notes, upstream-retirement roadmap.
|
||
- [`SKILL.md`](SKILL.md) — the companion agent skill, symlinked into `~/.agents/skills/opencode-mempalace-bridge/` on install.
|
||
- [`extensions/pi/`](extensions/pi/) — the pi↔mempalace MCP bridge (a TypeScript extension symlinked into `~/.pi/agent/extensions/`). Pi's own base config (keybindings, env loader, settings template) is in the sibling [`pi-toolkit`](https://gitea.jordbo.se/joakimp/pi-toolkit) repo — split out 2026-05-05 so `opencode-devbox` can build slim containers without mempalace.
|
||
|
||
**If you're just trying to get this working on a new machine → jump to [Setup](#setup).**
|
||
**If you want the full architecture story → read [`ARCHITECTURE.md`](ARCHITECTURE.md).**
|
||
|
||
---
|
||
|
||
## Ecosystem
|
||
|
||
This repo is the memory-layer hub in a family of composable toolkits.
|
||
Each `install.sh` is independent and idempotent; install only what you
|
||
need. MemPalace can be opted out entirely (the `opencode-devbox` slim
|
||
container path exercises this).
|
||
|
||
```
|
||
┌──────────────────┐
|
||
│ myconfigs │ dotfiles: ~/.config/pi/.env, ~/.config/opencode/.env,
|
||
└────────┬─────────┘ tmux CSI-u, zsh loaders, … (git-crypt encrypted)
|
||
│ provision
|
||
┌──────────────┼──────────────┐
|
||
▼ ▼ ▼
|
||
┌──────────┐ ┌──────────┐ ┌──────────────┐
|
||
│opencode- │ │pi-toolkit│ │ (pi/opencode │
|
||
│ toolkit │ │ │ │ installed │
|
||
│ │ │ │ │ upstream) │
|
||
└─────┬────┘ └─────┬────┘ └──────────────┘
|
||
│ │
|
||
│ (optional) │ (optional)
|
||
└──────┬───────┘
|
||
▼
|
||
┌─────────────────┐
|
||
│mempalace-toolkit│ ← this repo
|
||
└─────────────────┘ detects pi / opencode and installs bridges
|
||
for whichever are present
|
||
```
|
||
|
||
Who owns what:
|
||
|
||
| Repo | Scope |
|
||
|---|---|
|
||
| [`myconfigs`](https://gitea.jordbo.se/joakimp/myconfigs) | Dotfiles (git-crypt encrypted). Ships `.env` files each toolkit below sources. |
|
||
| [`opencode-toolkit`](https://gitea.jordbo.se/joakimp/opencode-toolkit) | Opencode's own shell glue (env loader for `~/.config/opencode/.env`). No mempalace dep. |
|
||
| [`pi-toolkit`](https://gitea.jordbo.se/joakimp/pi-toolkit) | Pi's own base config (keybindings, env loader, `settings.example.json`). No mempalace dep. |
|
||
| **`mempalace-toolkit` (this)** | Memory layer. Mining wrappers in `bin/`, `opencode-mempalace-bridge` skill, pi↔mempalace MCP extension. Probes for the other toolkits' artifacts; installs bridges where relevant. |
|
||
| [`opencode-devbox`](https://gitea.jordbo.se/joakimp/opencode-devbox) | Docker containers that compose any subset via independent `install.sh` invocations. |
|
||
|
||
---
|
||
|
||
## Why this exists
|
||
|
||
MemPalace is the agent memory layer. Its stock CLI has two gaps that bite on a machine running opencode with a docs-first palace policy:
|
||
|
||
1. **`mempalace mine` floods the palace with source code** — every `__init__` fragment, every generated file, hundreds of low-signal drawers per project. `mempalace-docs` fixes this by staging only documentation-class files (`*.md`, `*.yml`, `Dockerfile`, etc.) before mining.
|
||
2. **`mempalace mine --mode convos` can't read opencode's SQLite DB** — only file-based chat formats (Claude Code JSONL, Claude.ai JSON, ChatGPT, Slack, Codex). Opencode persists every turn in `~/.local/share/opencode/opencode.db` and has no upstream hook into mempalace's auto-save. `mempalace-session` fixes this by exporting each session to Claude Code JSONL before mining.
|
||
|
||
Both wrappers follow the same **stage-to-cache-then-mine** idiom. Neither reimplements the miner; they curate input and delegate.
|
||
|
||
Long-term, both should retire:
|
||
- `mempalace-docs` → retires when [MemPalace PR #1213](https://github.com/MemPalace/mempalace/pull/1213) (`exclude_patterns` in `mempalace.yaml`) merges.
|
||
- `mempalace-session` → retires when opencode session-stopping hooks ([PR #16598](https://github.com/anomalyco/opencode/pull/16598) et al.) merge **and** `hooks_cli.py` gains an `opencode` harness. Until both land, this repo fills the gap.
|
||
|
||
See [`ARCHITECTURE.md`](ARCHITECTURE.md) §6 for the full upstream roadmap.
|
||
|
||
---
|
||
|
||
## Setup
|
||
|
||
### Prerequisites
|
||
|
||
- [MemPalace](https://github.com/MemPalace/mempalace) CLI v3.3.3+ — **see [Installing mempalace itself](#installing-mempalace-itself-prerequisite) below if you haven't already**.
|
||
- Python 3 (stdlib `sqlite3` only — no extra deps)
|
||
- [opencode](https://github.com/anomalyco/opencode) with an active session DB at `~/.local/share/opencode/opencode.db` *(only needed for `mempalace-session`)*
|
||
- The mempalace **wake-up protocol** at `~/.config/opencode/instructions/mempalace.md` — without it, opencode loads the mempalace skill but never auto-runs it at session start, so most of mempalace's value is forfeited silently. Shipped by the [skillset repo](https://gitea.jordbo.se/joakimp/skillset); deploy via `./deploy-skills.sh --bootstrap` once per machine. `mempalace-toolkit/install.sh` probes for this file and warns if it's missing.
|
||
- The mempalace **MCP server registered in `~/.config/opencode/opencode.json`** — without this, opencode has no way to reach the mempalace tools and every `mempalace_*` call silently fails. See [Registering mempalace with opencode](#registering-mempalace-with-opencode-or-other-mcp-clients) below for the one-line JSON stanza. `install.sh` probes for this too.
|
||
|
||
### Installing mempalace itself (prerequisite)
|
||
|
||
mempalace-toolkit wraps the mempalace CLI but does not bundle it. The upstream [MemPalace repo](https://github.com/MemPalace/mempalace) documents `pip install mempalace` as the install method; `uv tool install` is cleaner and is the flow used in production containers like [opencode-devbox](https://gitea.jordbo.se/joakimp/opencode-devbox).
|
||
|
||
**Why uv over pip:**
|
||
- Isolated venv per tool — mempalace's dependencies (chromadb, embedding model runtime, …) don't leak into system Python or your project venvs.
|
||
- No PEP 668 fight — modern Debian / Ubuntu / Homebrew Python all refuse `pip install` into the system site-packages. `uv tool install` sidesteps this entirely.
|
||
- The shim (`~/.local/bin/mempalace` by default) is a thin wrapper that automatically activates the isolated venv on invocation, so `mempalace` is available from any bash or zsh terminal without manual `source venv/bin/activate`.
|
||
|
||
**Install uv** if it's not already on the machine:
|
||
|
||
```bash
|
||
# macOS / Linux, official installer — puts uv in ~/.local/bin
|
||
curl -LsSf https://astral.sh/uv/install.sh | sh
|
||
|
||
# Or: Homebrew on macOS
|
||
brew install uv
|
||
|
||
# Verify
|
||
uv --version
|
||
```
|
||
|
||
#### Personal machine (recommended default)
|
||
|
||
```bash
|
||
# Installs mempalace into an isolated venv under ~/.local/share/uv/tools/mempalace/,
|
||
# puts the `mempalace` shim into ~/.local/bin/.
|
||
uv tool install mempalace
|
||
|
||
# Make sure ~/.local/bin is on $PATH (uv prints this if it isn't)
|
||
export PATH="$HOME/.local/bin:$PATH" # add to ~/.bashrc or ~/.zshrc
|
||
|
||
# Verify
|
||
mempalace --version # should print the installed version
|
||
which mempalace # should point into ~/.local/bin/
|
||
```
|
||
|
||
After this, `mempalace` works the same from any bash or zsh terminal — interactive shell, script, cron, systemd user service, launchd agent, all fine.
|
||
|
||
To upgrade later: `uv tool upgrade mempalace` (or `--all`).
|
||
To uninstall: `uv tool uninstall mempalace`.
|
||
|
||
#### System-wide / container install (opencode-devbox pattern)
|
||
|
||
For a Docker image or a multi-user box where the shim should live on the system `PATH` rather than in each user's `~/.local/bin`, use `UV_TOOL_DIR` + `UV_TOOL_BIN_DIR` to relocate both the venv and the shim:
|
||
|
||
```bash
|
||
# In the Dockerfile — this is the pattern used by opencode-devbox
|
||
ENV UV_TOOL_DIR=/opt/uv-tools
|
||
ENV UV_TOOL_BIN_DIR=/usr/local/bin
|
||
|
||
RUN mkdir -p /opt/uv-tools && \
|
||
uv tool install --no-cache mempalace && \
|
||
/opt/uv-tools/mempalace/bin/python -c "import mempalace; print('mempalace installed')"
|
||
```
|
||
|
||
After this:
|
||
- `/opt/uv-tools/mempalace/` — the isolated venv.
|
||
- `/usr/local/bin/mempalace` — the CLI shim (globally on `PATH`, works for every user).
|
||
|
||
The last `python -c` line in the RUN step is a build-time sanity check: if the install silently failed, the build fails here rather than at runtime.
|
||
|
||
See [opencode-devbox/Dockerfile](https://gitea.jordbo.se/joakimp/opencode-devbox/src/branch/main/Dockerfile) §"MemPalace install" for the full production version (adds `INSTALL_MEMPALACE=true` build arg so the install can be skipped to shave ~300 MB off the image).
|
||
|
||
#### Registering mempalace with opencode (or other MCP clients)
|
||
|
||
Installing `mempalace` via `uv tool install` puts two shims on `PATH`: `mempalace` (the CLI) and `mempalace-mcp` (the MCP server). Neither the skill nor `mempalace-toolkit` will give you anything useful in opencode until the MCP server is registered in `~/.config/opencode/opencode.json`.
|
||
|
||
**Add to opencode.json:**
|
||
|
||
```json
|
||
{
|
||
"$schema": "https://opencode.ai/config.json",
|
||
"mcp": {
|
||
"mempalace": {
|
||
"type": "local",
|
||
"command": ["mempalace-mcp"]
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
If you already have other MCP servers configured, add the `mempalace` entry into the existing `mcp` object — don't replace the whole file. opencode's JSON is merged shallow-ly.
|
||
|
||
**Minimal full opencode.json for someone starting fresh** (adjust `model` to your preferred provider):
|
||
|
||
```json
|
||
{
|
||
"$schema": "https://opencode.ai/config.json",
|
||
"model": "anthropic/claude-opus-4-6",
|
||
"share": "disabled",
|
||
"autoupdate": false,
|
||
"instructions": [
|
||
"~/.config/opencode/instructions/mempalace.md"
|
||
],
|
||
"mcp": {
|
||
"mempalace": {
|
||
"type": "local",
|
||
"command": ["mempalace-mcp"]
|
||
}
|
||
}
|
||
}
|
||
```
|
||
|
||
Note the `instructions` array: this is what tells opencode to load the wake-up protocol at session start (see the previous section on installing that file via `skillset --bootstrap`).
|
||
|
||
**Custom palace path** (rare — default `~/.mempalace/palace/` works for everyone):
|
||
|
||
```json
|
||
"command": ["mempalace-mcp", "--palace", "/path/to/palace"]
|
||
```
|
||
|
||
**Claude Code** has a one-liner helper:
|
||
|
||
```bash
|
||
claude mcp add mempalace -- mempalace-mcp
|
||
```
|
||
|
||
…or `mempalace mcp` prints the current recommended snippets on demand:
|
||
|
||
```bash
|
||
mempalace mcp # shows `claude mcp add` + direct-run commands
|
||
```
|
||
|
||
**After editing the config**, restart opencode (or your MCP client). Verify the connection:
|
||
|
||
```bash
|
||
# From inside an opencode session:
|
||
# mempalace_status → should return palace stats, not "tool unavailable"
|
||
```
|
||
|
||
If the MCP tools don't show up in your agent's tool list, the most common causes are:
|
||
|
||
| Symptom | Likely cause | Fix |
|
||
|---|---|---|
|
||
| `mempalace_*` tools absent from tool list entirely | opencode.json not re-read | Restart opencode. |
|
||
| Server reported "unavailable" at startup | Shim not on `PATH`, or CLI itself broken | Run `mempalace-mcp --help` manually; fix PATH or re-run `uv tool install mempalace`. |
|
||
| `ModuleNotFoundError: No module named 'mempalace'` in logs | You configured `["python3", "-m", "mempalace.mcp_server"]` instead of `["mempalace-mcp"]` | Use the shim. See the legacy-fallback section below if you specifically need the python-invocation form. |
|
||
|
||
#### Legacy fallback: `mempalace-mcp-server` wrapper
|
||
|
||
Older MCP configs sometimes reference `["python3", "-m", "mempalace.mcp_server"]`. This worked when mempalace was installed via `pip install --break-system-packages` into the system site-packages, but breaks after switching to `uv tool install` — system `python3` cannot import mempalace from the isolated venv. On opencode-devbox a thin wrapper script on `PATH` bridged the two worlds during the transition:
|
||
|
||
```sh
|
||
#!/bin/sh
|
||
# /usr/local/bin/mempalace-mcp-server
|
||
exec /opt/uv-tools/mempalace/bin/python -m mempalace.mcp_server "$@"
|
||
```
|
||
|
||
With the wrapper in place, MCP configs referencing `["mempalace-mcp-server"]` work on a `uv tool install` setup.
|
||
|
||
**This is legacy; don't use it for new installs.** The modern `["mempalace-mcp"]` form — the uv-tool shim — does the same job without the extra script. The wrapper is documented here only so you know what to look for if you encounter an older config or a machine that was set up during the transition.
|
||
|
||
#### Verification checklist
|
||
|
||
After any install (personal or system-wide), confirm:
|
||
|
||
```bash
|
||
# CLI reachable from PATH
|
||
which mempalace # → a shim path
|
||
mempalace --version # → v3.3.3+ without import errors
|
||
|
||
# CLI can import its own modules (catches venv vs site-packages mismatch)
|
||
mempalace status 2>&1 | head -3 # → either palace stats or "No palace found" — not a Python traceback
|
||
|
||
# MCP shim reachable and runnable
|
||
which mempalace-mcp # → a shim path matching the `mempalace` CLI location
|
||
mempalace-mcp --help 2>&1 | head -5 # → MCP server help, not a Python traceback
|
||
```
|
||
|
||
If any of these produce `ModuleNotFoundError`, the isolated venv is broken — re-run `uv tool install --force mempalace` (or the system-install equivalent with `UV_TOOL_DIR` set) and check the verification again.
|
||
|
||
### Install mempalace-toolkit
|
||
|
||
```bash
|
||
git clone ssh://git@gitea.jordbo.se:2222/joakimp/mempalace-toolkit.git ~/mempalace-toolkit
|
||
cd ~/mempalace-toolkit
|
||
./install.sh
|
||
```
|
||
|
||
The installer symlinks `bin/*` into `~/.local/bin/` and installs the agent skill into `~/.agents/skills/opencode-mempalace-bridge/`. If [pi](https://github.com/earendil-works/pi) is installed (detected via `~/.pi/agent/extensions/`), it also symlinks [`extensions/pi/mempalace.ts`](extensions/pi/) into that directory so the pi↔mempalace bridge tracks version control. On machines without pi this step is silently skipped. Works on macOS and Linux.
|
||
|
||
Ensure `~/.local/bin` is on `$PATH`:
|
||
|
||
```bash
|
||
export PATH="$HOME/.local/bin:$PATH"
|
||
```
|
||
|
||
**If `install.sh` reports `Skipping <name> — already exists`:** there's a leftover symlink or file at `~/.local/bin/<name>` from a previous install (e.g. the pre-split `cli_utils` days). The installer prints the exact `rm && ./install.sh` command to fix it — remove the stale entry and re-run. It will never clobber an existing file without the user explicitly removing it first.
|
||
|
||
### Deploying pi on a new machine (full recipe)
|
||
|
||
If the target machine also runs [pi](https://github.com/earendil-works/pi), the recipe is: install [`pi-toolkit`](https://gitea.jordbo.se/joakimp/pi-toolkit) first (pi's own base config), then this toolkit (adds the pi↔mempalace MCP bridge). Full step-by-step: **[`extensions/pi/README.md` § Deploying pi with mempalace on a new machine](extensions/pi/README.md#deploying-pi-with-mempalace-on-a-new-machine)**.
|
||
|
||
Quick summary:
|
||
|
||
```bash
|
||
# 1. Dotfiles (tmux extended-keys, ~/.config/pi/.env, ...)
|
||
git clone <myconfigs> && cd myconfigs && ./provision.sh --profile <profile>
|
||
|
||
# 2. pi upstream 3. pi-toolkit (pi base config)
|
||
brew install pi-coding-agent git clone ssh://git@gitea.jordbo.se:2222/joakimp/pi-toolkit.git
|
||
cd pi-toolkit && ./install.sh
|
||
|
||
# 4. settings bootstrap
|
||
cp ~/pi-toolkit/settings.example.json ~/.pi/agent/settings.json && $EDITOR !$
|
||
|
||
# 5. mempalace CLI 6. This repo (adds the bridge)
|
||
uv tool install mempalace cd ~/mempalace-toolkit && ./install.sh
|
||
|
||
# 7. Open fresh shell, run `pi`. Wake-up auto-injection proves end-to-end.
|
||
```
|
||
|
||
### First mine
|
||
|
||
```bash
|
||
# Mine opencode session history into wing_conversations (no init needed)
|
||
mempalace-session --dry-run # preview qualifying sessions
|
||
mempalace-session # do it (~20 min per 60 sessions)
|
||
|
||
# Mine a project (docs only). If you want to pre-init the project with a
|
||
# custom wing name or entity config, run `mempalace init --yes <dir>` first;
|
||
# otherwise `mempalace-docs` derives the wing from the directory name.
|
||
mempalace-docs /workspace/my_project --dry-run
|
||
mempalace-docs /workspace/my_project
|
||
```
|
||
|
||
> **Note:** mempalace has no one-time global init. The palace itself is created lazily on first write (at `~/.mempalace/palace/`). `mempalace init <dir>` is a *per-project* command that sets up a `mempalace.yaml` + entity list for a specific source directory — optional, not a prerequisite for either wrapper.
|
||
|
||
### Diary vs session mine: why keep both?
|
||
|
||
Automated session mining captures every turn verbatim into `wing_conversations`. But agents are still expected to write a short AAAK-compressed diary entry at wind-down (the consumer-side `mempalace` skill calls this out as mandatory). They're not redundant — they answer different questions:
|
||
|
||
- **Session mine** = git log with diffs. *"What did we say exactly?"* Raw, searchable, complete. High noise.
|
||
- **Diary** = release notes. *"What did we decide / learn / accomplish?"* Curated, compressed, recency-scanned. The agent's editorial judgment of what mattered, including meta-observations that were never said aloud.
|
||
|
||
A machine running only one of these has half a memory. Full treatment with practical implications in [`ARCHITECTURE.md` §5 → "Diary vs session mine: why keep both?"](ARCHITECTURE.md#diary-vs-session-mine-why-keep-both). Short answer: automate the mine, keep writing diaries, and let them specialize.
|
||
|
||
### Keeping it fresh (automation)
|
||
|
||
**pi:** nothing to set up. The pi bridge extension
|
||
(`extensions/pi/mempalace.ts`) feeds the palace by itself — see
|
||
[`extensions/pi/README.md` § Automatic transcript feeding](extensions/pi/README.md#automatic-transcript-feeding).
|
||
The templates below aren't needed unless you're running
|
||
`mempalace-pi-session` somewhere without that extension (e.g. a bare pi
|
||
install, or a host-side catch-up job).
|
||
|
||
**opencode**, and pi installs without the bridge: manual invocation is fine
|
||
while you're actively driving the machine, but long-running devboxes benefit
|
||
from a weekly automated mine. [`contrib/`](contrib/) ships ready-to-install
|
||
templates:
|
||
|
||
- **systemd user timer** (recommended on Linux): survives reboots, catches missed runs, logs to `journalctl`.
|
||
- **launchd user agent** (recommended on macOS): native-equivalent — logs to `~/Library/Logs/`, single-instance guarantees, `ProcessType=Background` throttling.
|
||
- **cron**: simplest, works on BSD and systemd-less distros. No user-unit awareness needed.
|
||
- **Devbox variants** (`*-devbox.*`): if you run `mempalace-session` inside a long-lived container (e.g. [opencode-devbox](https://gitea.jordbo.se/joakimp/opencode-devbox)), the scheduler lives on the **host** and uses `docker exec` to reach the tool inside the container. Systemd and cron variants are included; both guard against "container currently stopped" so the timer is safe to leave enabled across dev cycles.
|
||
|
||
Quick-start (Linux / systemd, weekly Mon 03:00 local):
|
||
|
||
```bash
|
||
mkdir -p ~/.config/systemd/user
|
||
cp contrib/systemd/*.{service,timer} ~/.config/systemd/user/
|
||
systemctl --user daemon-reload
|
||
systemctl --user enable --now mempalace-session.timer
|
||
sudo loginctl enable-linger "$USER" # optional, for headless boxes
|
||
```
|
||
|
||
Quick-start (macOS / launchd, same schedule):
|
||
|
||
```bash
|
||
sed "s|USER|$USER|g" contrib/launchd/se.jordbo.mempalace-session.plist \
|
||
> ~/Library/LaunchAgents/se.jordbo.mempalace-session.plist
|
||
mkdir -p ~/Library/Logs
|
||
launchctl bootstrap "gui/$(id -u)" ~/Library/LaunchAgents/se.jordbo.mempalace-session.plist
|
||
launchctl enable "gui/$(id -u)/se.jordbo.mempalace-session"
|
||
```
|
||
|
||
Quick-start (host-side scheduling for a long-running opencode-devbox container):
|
||
|
||
```bash
|
||
# systemd on the host → docker exec into the container
|
||
mkdir -p ~/.config/systemd/user
|
||
cp contrib/systemd/mempalace-session-devbox.{service,timer} ~/.config/systemd/user/
|
||
systemctl --user daemon-reload
|
||
systemctl --user enable --now mempalace-session-devbox.timer
|
||
# If your container isn't named 'opencode-devbox' or its user isn't
|
||
# 'developer', run `systemctl --user edit mempalace-session-devbox.service`
|
||
# to set CONTAINER / CONTAINER_USER via an override.
|
||
```
|
||
|
||
See [`contrib/README.md`](contrib/README.md) for full install/verify/uninstall recipes, tuning, chooser table (host vs. devbox), and devbox/container caveats. The full operational routine (triggers, cadence, verification) is in [`ARCHITECTURE.md`](ARCHITECTURE.md) §5.
|
||
|
||
### Containerized (devbox) notes
|
||
|
||
On a Docker-based devbox, the palace and opencode DB should live on named volumes so they survive container recreate:
|
||
|
||
- `devbox-palace` → `~/.mempalace/palace`
|
||
- `devbox-data` → `~/.local/share/opencode`
|
||
|
||
This repo is typically bind-mounted from the host, so code survives recreate and syncs via git. After a container recreate, `~/.local/bin` is wiped — just re-run `./install.sh` (idempotent) to relink.
|
||
|
||
---
|
||
|
||
## `mempalace-docs`
|
||
|
||
Docs-only MemPalace miner. Stages documentation files into a cache dir and runs `mempalace mine` against the cache — never against the raw project dir.
|
||
|
||
```bash
|
||
mempalace-docs <directory> # mine with wing = dirname
|
||
mempalace-docs <directory> --wing my_project # override wing name
|
||
mempalace-docs <directory> --agent alice # record agent on drawers
|
||
mempalace-docs <directory> --dry-run # list files, don't file
|
||
mempalace-docs <directory> --repair # opt-in post-mine repair (risky, interactive only)
|
||
mempalace-docs --help
|
||
```
|
||
|
||
**What gets mined:** `*.md`, `*.mdx`, `*.rst`, `*.txt`, `*.yml`, `*.yaml`, `*.toml`, `*.json`, `*.sh`, `*.bash`, `*.zsh`, `*.fish`, `Dockerfile*`, `Makefile*`, `*.conf`, `*.cfg`, `*.ini`, `LICENSE*`, `COPYING*`, `NOTICE*`.
|
||
|
||
**What gets skipped:** `.py`, `.ts`, `.tsx`, `.js`, `.jsx`, `.go`, `.rs`, `.java`, `.cpp`, `.c`, `.rb`, `.kt`, `.swift`, build output directories (`.git`, `.venv`, `node_modules`, `__pycache__`, `.mypy_cache`, `.pytest_cache`, `.ruff_cache`, `dist`, `build`, `.next`, `target`, `coverage`), lockfiles.
|
||
|
||
**Rationale:** the palace is for *context and intent*. Agents already have `grep`/`glob`/`Read` for code — always authoritative, never stale. Embedding source code creates a parallel, lossier, drift-prone copy that pollutes semantic search for years.
|
||
|
||
---
|
||
|
||
## `mempalace-census`
|
||
|
||
Read-only join census of a palace **on local disk**. Answers *"what would joining this palace into the primary actually move, and what dedupes it?"* — [RFC 002](docs/rfc-002-joiner.md) Phase A.
|
||
|
||
```bash
|
||
mempalace-census # default palace, human report
|
||
mempalace-census --palace /mnt/tor-ms22/palace # a palace rsynced from another machine
|
||
mempalace-census --kg /path/knowledge_graph.sqlite3
|
||
mempalace-census --json > manifest.json # machine-readable, feeds Phase B/C
|
||
mempalace-census --no-verify-ids # skip id recompute (faster on huge palaces)
|
||
mempalace-census --help
|
||
```
|
||
|
||
**Classification**, by descending signal strength:
|
||
|
||
| Class | Identified by | Join action |
|
||
| --- | --- | --- |
|
||
| `DIARY` | metadata `type='diary_entry'` | replay + RFC 001 §7.6 suffix skip |
|
||
| `MINED` | truthy `source_file` | **re-mine on the target, never replay** — ids are path-derived, so replay duplicates |
|
||
| `AGENT-AUTHORED` | neither | replay, idempotent by content id |
|
||
|
||
**It self-verifies rather than trusting the docs.** For every replayable drawer it reassembles content from chunks and recomputes the upstream id, comparing against the stored one — checking the id recipe, the chunk reassembly order and the classifier in a single pass. On the reference palace: 176/176 accounted for. This check is what caught three wrong assumptions now written up in [RFC 002 §2.1](docs/rfc-002-joiner.md) — most usefully that `id_recipe` is *not* a mined-only marker, and that `update_drawer` preserves a drawer's id while rewriting content, so 15% of agent-authored drawers no longer reproduce their own content hash (`edited_since_filing` in the manifest).
|
||
|
||
**Safety:** opens every sqlite file with `mode=ro` and never writes — no `-wal`/`-shm` files are created, so it is safe to run while `mempalace-serve` is live. It reads **local disk only** and deliberately ignores `MEMPALACE_REMOTE_URL` (it warns if that is set, so you don't mistake a local census for a remote one).
|
||
|
||
---
|
||
|
||
## `mempalace-session`
|
||
|
||
Opencode → MemPalace session bridge. Reads `~/.local/share/opencode/opencode.db`, transforms each session into Claude Code JSONL, and files via `mempalace mine --mode convos`.
|
||
|
||
```bash
|
||
mempalace-session # mine all sessions (≥3 msgs)
|
||
mempalace-session --wing my_convos # custom wing (default: wing_conversations)
|
||
mempalace-session --session ses_abc123 # one session only
|
||
mempalace-session --since 2026-04-01 # only sessions updated on/after date
|
||
mempalace-session --min-messages 6 # stricter short-session filter
|
||
mempalace-session --db /custom/path/opencode.db # non-default DB location
|
||
mempalace-session --dry-run # export + list, skip mine
|
||
mempalace-session --repair # opt-in post-mine repair (risky, interactive only)
|
||
mempalace-session --help
|
||
```
|
||
|
||
**What gets exported per session:**
|
||
|
||
- Synthetic header injected as the first user turn (`[session: <title> | <dir> | <date>]`) so the palace can find sessions by topic, not just by ID.
|
||
- Each message → Claude Code JSONL line (`{"type": "user"|"assistant", "message": {"content": ...}}`).
|
||
- Tool calls → `tool_use` blocks. Known tools (`Bash`, `Read`, `Grep`, `Edit`, `Write`) get formatted summaries; unknown tools are JSON-serialized.
|
||
- Tool outputs → `tool_result` blocks in a follow-up human message, folded back into the assistant turn by the mempalace normalizer.
|
||
- `step-start` / `step-finish` parts are dropped as noise. `reasoning` parts are kept with a `[reasoning]` prefix.
|
||
|
||
**Dedup:** staging at `<palace-root>/opencode-stage/<wing>/` (override: `$MEMPALACE_SESSION_STAGE`) with deterministic per-session filenames (`<slug>_<id>.jsonl`). The convos miner keys on `source_file`, so re-runs skip unchanged sessions. To force re-mining a session, delete its JSONL from the staging dir.
|
||
|
||
**`--dry-run` is dedup-aware.** Each session is tagged `[NEW]` (would be filed) or `[SKIP]` (already in the palace), and the summary breaks down the count:
|
||
|
||
```
|
||
Exported 62 session(s) to /home/you/.mempalace/opencode-stage/wing_conversations
|
||
0 new → will be filed on mine
|
||
62 already filed → will be skipped (dedup by source_file)
|
||
|
||
--dry-run: no new sessions to mine. A real run would skip all 62.
|
||
```
|
||
|
||
If the palace is unreachable (fresh install, moved, permission-denied) the wrapper falls back to "everything is new" — the real mine step delegates dedup to `mempalace mine --mode convos`, which is always the source of truth. So running `mempalace-session` twice in a row is never destructive or wasteful: the second run's only cost is the post-mine HNSW `repair` step (~5 min on a ~5k-drawer palace).
|
||
|
||
**Filter:** sessions with fewer than `--min-messages` messages (default 3) are skipped — drops throwaway `/exit`'d sessions that would otherwise flood the palace. On a reference 140-session corpus, 78 were filtered this way.
|
||
|
||
**Cost profile:** ~20 minutes per 60-session batch. Scales roughly linearly with message count. Dedup re-run: mine step instant, only the post-mine `repair` runs (~5 min on 5k drawers).
|
||
|
||
---
|
||
|
||
## `mempalace-pi-session`
|
||
|
||
Pi coding-agent → MemPalace session bridge. Reads pi's own JSONL session
|
||
transcripts under `~/.pi/agent/sessions/`, converts each qualifying session
|
||
to Claude Code JSONL, and files via `mempalace mine --mode convos`. On most
|
||
machines you never run this by hand — the pi bridge extension
|
||
(`extensions/pi/mempalace.ts`) invokes it automatically; see
|
||
[`extensions/pi/README.md` § Automatic transcript feeding](extensions/pi/README.md#automatic-transcript-feeding).
|
||
Manual invocation is for a bare pi install without that extension, a
|
||
host-side catch-up job, or just poking at the export by hand.
|
||
|
||
```bash
|
||
mempalace-pi-session # export + mine everything qualifying
|
||
mempalace-pi-session --wing my_convos # custom wing (default: wing_conversations)
|
||
mempalace-pi-session --session <uuid-prefix> # one session only
|
||
mempalace-pi-session --since 2026-04-01 # only sessions updated on/after date
|
||
mempalace-pi-session --min-messages 6 # stricter turn-count filter (default: 4)
|
||
mempalace-pi-session --min-assistant-chars 2000 # stricter "did anything happen" filter (default: 1000)
|
||
mempalace-pi-session --dry-run # export + list, skip mine
|
||
mempalace-pi-session --prepare # export + stage only; print MINE_SOURCE=<path>, never opens the palace
|
||
mempalace-pi-session --mode remote # ship the stage to a remote palace host instead of mining locally
|
||
mempalace-pi-session --help
|
||
```
|
||
|
||
**Why `--prepare` exists:** the palace is single-writer. If a pi session for
|
||
this same palace is currently open, its own bridge extension already holds
|
||
the palace via `mempalace-mcp`, so an unattended `mempalace mine` from
|
||
anywhere else fails with `palace ... is held by PID <n>`. `--prepare` does
|
||
only the palace-free half (export, threshold, staging, and the `rsync` in
|
||
remote mode) and hands the mine off to whoever already holds the lock —
|
||
which is exactly what the pi extension does with its own MCP client. Run
|
||
without `--prepare`, the tool does the mine itself, and treats that
|
||
contention as success (exit 0, informational message) rather than failure —
|
||
the holder will mine what got staged.
|
||
|
||
**What gets exported per session:**
|
||
|
||
- Synthetic header injected as the first user turn
|
||
(`[session: <title> | <cwd> | <date> | source: pi]`) so the palace can find
|
||
sessions by topic, not just by ID, and so pi/opencode/other results stay
|
||
distinguishable in search.
|
||
- User/assistant messages extracted from pi's JSONL `message` entries.
|
||
- Assistant `toolCall` blocks → Claude Code `tool_use` blocks.
|
||
- `toolResult` role messages → `tool_result` blocks, folded back into the
|
||
assistant turn by the mempalace normalizer.
|
||
- `bashExecution`, `custom` (display-only), `branchSummary`,
|
||
`compactionSummary` → rendered as text annotations.
|
||
- `thinking` blocks and image content → dropped (noise / not text).
|
||
|
||
**Filter — two gates, both required** (this is stricter than
|
||
`mempalace-session`'s single message-count filter, and deliberately so —
|
||
see below):
|
||
|
||
1. `--min-messages` user+assistant turns (default **4**).
|
||
2. `--min-assistant-chars` characters of assistant *text*, tool results
|
||
excluded (default **1000**).
|
||
|
||
The second gate exists because message count alone isn't enough for pi: pi
|
||
expands skill/context text into the user prompt, so an abandoned session can
|
||
have a huge "user" turn and almost nothing on the assistant side — e.g. a
|
||
real observed case with a 13,380-char injected-context user turn answered
|
||
"Ready. What would you like to work on?" (38 chars). Total size said
|
||
substantial; assistant size correctly said nothing happened. Measured real
|
||
sessions on the same machine ran 15,900–100,000 assistant chars, so the
|
||
1000-char default sits with wide margin on both sides.
|
||
|
||
**Dedup:** staging under `$MEMPALACE_PI_STAGE/<wing>/` (default
|
||
`<palace-root>/pi-stage/<wing>/` — beside the palace, so the stage cannot be
|
||
wiped independently of the dedup keys pointing at it) with deterministic per-session-UUID
|
||
filenames, and the export preserves the source session's `mtime` on the
|
||
staged file. The convos miner is mtime-aware (see the Gotchas in
|
||
[`AGENTS.md`](AGENTS.md) — an older version of this doc claimed otherwise),
|
||
so re-runs on an unchanged session are a no-op, and re-feeding a **grown**
|
||
session (a live one being fed mid-conversation) purges and refiles that
|
||
session's drawers instead of duplicating them.
|
||
|
||
**Staging must persist.** Dedup keys on the *staged* path, not the original
|
||
transcript, so if the stage is wiped, a `mempalace sync` scoped to include it
|
||
sees those source files as gone and prunes the drawers mined from them —
|
||
deleting the memories, not just the cache. This is why the stage now defaults to
|
||
`<palace-root>/pi-stage`: it inherits whatever persistence the palace has, so
|
||
the files and the dedup keys that reference them cannot be separated by
|
||
wiping something that merely looks disposable. Override with
|
||
`$MEMPALACE_PI_STAGE` only if you have somewhere *more* durable than the palace.
|
||
|
||
**Remote palace:** if `$MEMPALACE_REMOTE_URL` is set, there is no
|
||
remote-palace CLI to mine into directly — `mempalace_mine` expands its
|
||
source path in the *server* process, which cannot see this machine's staged
|
||
exports. `--mode remote` (or `--mode auto`, which detects
|
||
`$MEMPALACE_REMOTE_URL`) instead `rsync`s the stage to a per-device inbox on
|
||
the palace host and asks the server to mine its own local copy. Requires
|
||
`$MEMPALACE_PI_SSH_TARGET` (`user@host:path`); see `MEMPALACE_PI_SSH_CONFIG`,
|
||
`MEMPALACE_PI_REMOTE_PATH`, and `MEMPALACE_PI_DEVICE` in `--help` for the
|
||
rest. Deploying that primary — newt, DNS, and why the auth is a shared bearer
|
||
token rather than per-device proxy users — is
|
||
[`docs/phase-1-exposure-runbook.md`](docs/phase-1-exposure-runbook.md).
|
||
|
||
---
|
||
|
||
## Companion agent skill
|
||
|
||
Installing this repo symlinks `SKILL.md` into `~/.agents/skills/opencode-mempalace-bridge/SKILL.md`, where it's auto-discovered by opencode (and by Claude Code / Kiro if you run `agents-sync` from [`cli_utils`](https://gitea.jordbo.se/joakimp/cli_utils)).
|
||
|
||
The skill is the *short-form checklist* for agents — when to use which wrapper, failure modes, setup recipes, anti-patterns. The canonical reference is always [`ARCHITECTURE.md`](ARCHITECTURE.md); the skill points there for deep context.
|
||
|
||
The skill pairs with the consumer-side [`mempalace` skill](https://github.com/MemPalace/mempalace) — that one covers using the palace (search, diary, KG); this one covers feeding it.
|
||
|
||
**Colocated skill pattern.** The skill lives here (not in [`skillset`](https://gitea.jordbo.se/joakimp/skillset)) because it moves in lockstep with the wrappers it documents. `install.sh` drops a `.skill-source` marker file in the deployed skill directory so sibling tooling (skillset's `deploy-skills.sh`, cli_utils's `agents-sync.zsh`) can tell the directory is externally owned. See [`AGENTS.md`](AGENTS.md) for the full convention and how to adopt it for future colocated skills.
|
||
|
||
---
|
||
|
||
## See also
|
||
|
||
- [`ARCHITECTURE.md`](ARCHITECTURE.md) — canonical spec: diagrams, setup recipe, failure modes, upstream roadmap.
|
||
- [`AGENTS.md`](AGENTS.md) — repo conventions for AI agents modifying this codebase.
|
||
- [MemPalace](https://github.com/MemPalace/mempalace) — the memory layer itself.
|
||
- [opencode](https://github.com/anomalyco/opencode) — the agent harness this bridges.
|
||
- [cli_utils](https://gitea.jordbo.se/joakimp/cli_utils) — sibling repo with shell quality-of-life tools (origin of these wrappers before the 2026-04-30 split).
|