joakimp/mempalace-toolkit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feeders: stage beside the palace, not in ~/.cache; document Phase 1 exposure

Browse Source 
Staging default moves out of ~/.cache to <palace-root>/pi-stage (pi) and
<palace-root>/opencode-stage (opencode), resolved with mempalace's own
palace-path precedence ($MEMPALACE_PALACE_PATH -> $MEMPAL_PALACE_PATH ->
~/.mempalace/config.json -> ~/.mempalace/palace), then dirname.

Why: the convos miner keys dedup on the *staged* path, so a wiped stage plus a
sync scoped to include it prunes the drawers mined from those sources --
deleting memories, not a cache. Under ~/.cache that state was reachable by
anything treating a cache as disposable. Staging inside the palace makes the
coupling structural: the stage cannot be wiped without touching the palace
itself. Overrides ($MEMPALACE_PI_STAGE / $MEMPALACE_SESSION_STAGE, --stage) are
unchanged. Note the old default had never been created on any host, so this
closed a latent hazard, not a live one.

Measured, and the docs now claim only this much: sync prunes only within the
scope it is given -- wing-only, 1299 scanned / 1299 out of scope / 0 removed;
scoped at the palace root, 651 kept / 648 out of scope. The previous blanket
"sync prunes every drawer" wording overstated it, which is a liability: the next
reader disproves the overstatement and discards the real constraint with it.

Also in this change:
- cron log dir ~/.cache/mempalace-session -> ~/.cache/mempalace-logs. The stage
  left that namespace, so the old name now read as "the stage".
- AGENTS.md: the convos miner *does* check mtime (verified against upstream
  convo_miner.py); the previous "no mtime check" claim was wrong.
- smoke-test assertions use `mktemp -d` for --sessions-dir. One pointed at /tmp,
  which still held earlier synthetic transcripts, so a --dry-run exported a fake
  session into the real stage: --dry-run skips the mine, not the export.

docs/phase-1-exposure-runbook.md -- the newt/DNS/auth step that RFC 001 and the
synlig runbook leave open (runbook section 4, items 2 and 5). Port 8765 at /mcp,
newt targets 172.17.0.1, and the authentication is the single shared bearer
token (RFC 6.2, decided 2026-08-09) rather than per-device proxy users. The
latter cannot work today: mempalace validates exactly one token, and Pangolin's
SSO/PIN/password are browser-shaped while every client here is a headless
JSON-RPC POST -- enabling that protection breaks the clients it protects. The
per-device axis that *does* exist is the feeder's SSH key + per-device inbox.

New finding recorded there: a loopback bind does not merely 403 behind a tunnel
(already known, runbook 2.4) -- it also silently starts the server with no token
at all, because auto-minting is gated on the bind being non-loopback.

extensions/pi/README.md: the HTTP transport IS authenticated as of mempalace
3.6.0; the "sessionless and unauthenticated" note dated from the v1.3.0 era.
Closes the RFC section 8 Phase-0 hygiene item.
This commit is contained in:
Joakim Persson
2026-08-12 17:04:01 +02:00
parent 3626946013
commit 29e660e18f
15 changed files with 1019 additions and 66 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+113 -3
View File
@@ -5,6 +5,7 @@ Producer-side tooling for [MemPalace](https://github.com/MemPalace/mempalace)
**What this repo contains:**
- `bin/mempalace-session` — exports [opencode](https://github.com/anomalyco/opencode) session history from its local SQLite DB to Claude Code JSONL, then mines it via `mempalace mine --mode convos`.
- `bin/mempalace-pi-session` — the same idea for the [pi coding-agent](https://github.com/earendil-works/pi): exports its native JSONL session transcripts and mines them the same way. Unlike `mempalace-session`, this one is normally invoked *for you* — the pi bridge extension (below) runs it automatically on `session_shutdown` and a debounced `agent_settled`, so most pi machines never need the `contrib/` scheduling templates at all. See [`mempalace-pi-session`](#mempalace-pi-session) below and [`extensions/pi/README.md` § Automatic transcript feeding](extensions/pi/README.md#automatic-transcript-feeding).
- `bin/mempalace-docs` — mines project directories into MemPalace while excluding source code, keeping the palace signal-dense.
- [`ARCHITECTURE.md`](ARCHITECTURE.md) — **canonical spec**: architecture diagram, component details, setup recipe, operational notes, upstream-retirement roadmap.
- [`SKILL.md`](SKILL.md) — the companion agent skill, symlinked into `~/.agents/skills/opencode-mempalace-bridge/` on install.
@@ -323,7 +324,17 @@ A machine running only one of these has half a memory. Full treatment with pract
### Keeping it fresh (automation)
Manual invocation is fine while you're actively driving the machine, but long-running devboxes benefit from a weekly automated mine. [`contrib/`](contrib/) ships ready-to-install templates:
**pi:** nothing to set up. The pi bridge extension
(`extensions/pi/mempalace.ts`) feeds the palace by itself — see
[`extensions/pi/README.md` § Automatic transcript feeding](extensions/pi/README.md#automatic-transcript-feeding).
The templates below aren't needed unless you're running
`mempalace-pi-session` somewhere without that extension (e.g. a bare pi
install, or a host-side catch-up job).
**opencode**, and pi installs without the bridge: manual invocation is fine
while you're actively driving the machine, but long-running devboxes benefit
from a weekly automated mine. [`contrib/`](contrib/) ships ready-to-install
templates:
- **systemd user timer** (recommended on Linux): survives reboots, catches missed runs, logs to `journalctl`.
- **launchd user agent** (recommended on macOS): native-equivalent — logs to `~/Library/Logs/`, single-instance guarantees, `ProcessType=Background` throttling.
@@ -421,12 +432,12 @@ mempalace-session --help
- Tool outputs → `tool_result` blocks in a follow-up human message, folded back into the assistant turn by the mempalace normalizer.
- `step-start` / `step-finish` parts are dropped as noise. `reasoning` parts are kept with a `[reasoning]` prefix.
**Dedup:** staging at `~/.cache/mempalace-session/<wing>/` with deterministic per-session filenames (`<slug>_<id>.jsonl`). The convos miner keys on `source_file`, so re-runs skip unchanged sessions. To force re-mining a session, delete its JSONL from the staging dir.
**Dedup:** staging at `<palace-root>/opencode-stage/<wing>/` (override: `$MEMPALACE_SESSION_STAGE`) with deterministic per-session filenames (`<slug>_<id>.jsonl`). The convos miner keys on `source_file`, so re-runs skip unchanged sessions. To force re-mining a session, delete its JSONL from the staging dir.
**`--dry-run` is dedup-aware.** Each session is tagged `[NEW]` (would be filed) or `[SKIP]` (already in the palace), and the summary breaks down the count:
```
Exported 62 session(s) to ~/.cache/mempalace-session/wing_conversations
Exported 62 session(s) to /home/you/.mempalace/opencode-stage/wing_conversations
0 new → will be filed on mine
62 already filed → will be skipped (dedup by source_file)
@@ -441,6 +452,105 @@ If the palace is unreachable (fresh install, moved, permission-denied) the wrapp
---
## `mempalace-pi-session`
Pi coding-agent → MemPalace session bridge. Reads pi's own JSONL session
transcripts under `~/.pi/agent/sessions/`, converts each qualifying session
to Claude Code JSONL, and files via `mempalace mine --mode convos`. On most
machines you never run this by hand — the pi bridge extension
(`extensions/pi/mempalace.ts`) invokes it automatically; see
[`extensions/pi/README.md` § Automatic transcript feeding](extensions/pi/README.md#automatic-transcript-feeding).
Manual invocation is for a bare pi install without that extension, a
host-side catch-up job, or just poking at the export by hand.
```bash
mempalace-pi-session # export + mine everything qualifying
mempalace-pi-session --wing my_convos # custom wing (default: wing_conversations)
mempalace-pi-session --session <uuid-prefix> # one session only
mempalace-pi-session --since 2026-04-01 # only sessions updated on/after date
mempalace-pi-session --min-messages 6 # stricter turn-count filter (default: 4)
mempalace-pi-session --min-assistant-chars 2000 # stricter "did anything happen" filter (default: 1000)
mempalace-pi-session --dry-run # export + list, skip mine
mempalace-pi-session --prepare # export + stage only; print MINE_SOURCE=<path>, never opens the palace
mempalace-pi-session --mode remote # ship the stage to a remote palace host instead of mining locally
mempalace-pi-session --help
```
**Why `--prepare` exists:** the palace is single-writer. If a pi session for
this same palace is currently open, its own bridge extension already holds
the palace via `mempalace-mcp`, so an unattended `mempalace mine` from
anywhere else fails with `palace ... is held by PID <n>`. `--prepare` does
only the palace-free half (export, threshold, staging, and the `rsync` in
remote mode) and hands the mine off to whoever already holds the lock —
which is exactly what the pi extension does with its own MCP client. Run
without `--prepare`, the tool does the mine itself, and treats that
contention as success (exit 0, informational message) rather than failure —
the holder will mine what got staged.
**What gets exported per session:**
- Synthetic header injected as the first user turn
(`[session: <title> | <cwd> | <date> | source: pi]`) so the palace can find
sessions by topic, not just by ID, and so pi/opencode/other results stay
distinguishable in search.
- User/assistant messages extracted from pi's JSONL `message` entries.
- Assistant `toolCall` blocks → Claude Code `tool_use` blocks.
- `toolResult` role messages → `tool_result` blocks, folded back into the
assistant turn by the mempalace normalizer.
- `bashExecution`, `custom` (display-only), `branchSummary`,
`compactionSummary` → rendered as text annotations.
- `thinking` blocks and image content → dropped (noise / not text).
**Filter — two gates, both required** (this is stricter than
`mempalace-session`'s single message-count filter, and deliberately so —
see below):
1. `--min-messages` user+assistant turns (default **4**).
2. `--min-assistant-chars` characters of assistant *text*, tool results
excluded (default **1000**).
The second gate exists because message count alone isn't enough for pi: pi
expands skill/context text into the user prompt, so an abandoned session can
have a huge "user" turn and almost nothing on the assistant side — e.g. a
real observed case with a 13,380-char injected-context user turn answered
"Ready. What would you like to work on?" (38 chars). Total size said
substantial; assistant size correctly said nothing happened. Measured real
sessions on the same machine ran 15,900–100,000 assistant chars, so the
1000-char default sits with wide margin on both sides.
**Dedup:** staging under `$MEMPALACE_PI_STAGE/<wing>/` (default
`<palace-root>/pi-stage/<wing>/` — beside the palace, so the stage cannot be
wiped independently of the dedup keys pointing at it) with deterministic per-session-UUID
filenames, and the export preserves the source session's `mtime` on the
staged file. The convos miner is mtime-aware (see the Gotchas in
[`AGENTS.md`](AGENTS.md) — an older version of this doc claimed otherwise),
so re-runs on an unchanged session are a no-op, and re-feeding a **grown**
session (a live one being fed mid-conversation) purges and refiles that
session's drawers instead of duplicating them.
**Staging must persist.** Dedup keys on the *staged* path, not the original
transcript, so if the stage is wiped, a `mempalace sync` scoped to include it
sees those source files as gone and prunes the drawers mined from them —
deleting the memories, not just the cache. This is why the stage now defaults to
`<palace-root>/pi-stage`: it inherits whatever persistence the palace has, so
the files and the dedup keys that reference them cannot be separated by
wiping something that merely looks disposable. Override with
`$MEMPALACE_PI_STAGE` only if you have somewhere *more* durable than the palace.
**Remote palace:** if `$MEMPALACE_REMOTE_URL` is set, there is no
remote-palace CLI to mine into directly — `mempalace_mine` expands its
source path in the *server* process, which cannot see this machine's staged
exports. `--mode remote` (or `--mode auto`, which detects
`$MEMPALACE_REMOTE_URL`) instead `rsync`s the stage to a per-device inbox on
the palace host and asks the server to mine its own local copy. Requires
`$MEMPALACE_PI_SSH_TARGET` (`user@host:path`); see `MEMPALACE_PI_SSH_CONFIG`,
`MEMPALACE_PI_REMOTE_PATH`, and `MEMPALACE_PI_DEVICE` in `--help` for the
rest. Deploying that primary — newt, DNS, and why the auth is a shared bearer
token rather than per-device proxy users — is
[`docs/phase-1-exposure-runbook.md`](docs/phase-1-exposure-runbook.md).
---
## Companion agent skill
Installing this repo symlinks `SKILL.md` into `~/.agents/skills/opencode-mempalace-bridge/SKILL.md`, where it's auto-discovered by opencode (and by Claude Code / Kiro if you run `agents-sync` from [`cli_utils`](https://gitea.jordbo.se/joakimp/cli_utils)).