joakimp/mempalace-toolkit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feeders: stage beside the palace, not in ~/.cache; document Phase 1 exposure

Browse Source 
Staging default moves out of ~/.cache to <palace-root>/pi-stage (pi) and
<palace-root>/opencode-stage (opencode), resolved with mempalace's own
palace-path precedence ($MEMPALACE_PALACE_PATH -> $MEMPAL_PALACE_PATH ->
~/.mempalace/config.json -> ~/.mempalace/palace), then dirname.

Why: the convos miner keys dedup on the *staged* path, so a wiped stage plus a
sync scoped to include it prunes the drawers mined from those sources --
deleting memories, not a cache. Under ~/.cache that state was reachable by
anything treating a cache as disposable. Staging inside the palace makes the
coupling structural: the stage cannot be wiped without touching the palace
itself. Overrides ($MEMPALACE_PI_STAGE / $MEMPALACE_SESSION_STAGE, --stage) are
unchanged. Note the old default had never been created on any host, so this
closed a latent hazard, not a live one.

Measured, and the docs now claim only this much: sync prunes only within the
scope it is given -- wing-only, 1299 scanned / 1299 out of scope / 0 removed;
scoped at the palace root, 651 kept / 648 out of scope. The previous blanket
"sync prunes every drawer" wording overstated it, which is a liability: the next
reader disproves the overstatement and discards the real constraint with it.

Also in this change:
- cron log dir ~/.cache/mempalace-session -> ~/.cache/mempalace-logs. The stage
  left that namespace, so the old name now read as "the stage".
- AGENTS.md: the convos miner *does* check mtime (verified against upstream
  convo_miner.py); the previous "no mtime check" claim was wrong.
- smoke-test assertions use `mktemp -d` for --sessions-dir. One pointed at /tmp,
  which still held earlier synthetic transcripts, so a --dry-run exported a fake
  session into the real stage: --dry-run skips the mine, not the export.

docs/phase-1-exposure-runbook.md -- the newt/DNS/auth step that RFC 001 and the
synlig runbook leave open (runbook section 4, items 2 and 5). Port 8765 at /mcp,
newt targets 172.17.0.1, and the authentication is the single shared bearer
token (RFC 6.2, decided 2026-08-09) rather than per-device proxy users. The
latter cannot work today: mempalace validates exactly one token, and Pangolin's
SSO/PIN/password are browser-shaped while every client here is a headless
JSON-RPC POST -- enabling that protection breaks the clients it protects. The
per-device axis that *does* exist is the feeder's SSH key + per-device inbox.

New finding recorded there: a loopback bind does not merely 403 behind a tunnel
(already known, runbook 2.4) -- it also silently starts the server with no token
at all, because auto-minting is gated on the bind being non-loopback.

extensions/pi/README.md: the HTTP transport IS authenticated as of mempalace
3.6.0; the "sessionless and unauthenticated" note dated from the v1.3.0 era.
Closes the RFC section 8 Phase-0 hygiene item.
This commit is contained in:
Joakim Persson
2026-08-12 17:04:01 +02:00
parent 3626946013
commit 29e660e18f
15 changed files with 1019 additions and 66 deletions
Download Patch File Download Diff File Expand all files Collapse all files
ARCHITECTURE.md
+84 -6
View File
@@ -31,6 +31,13 @@ So on a machine using opencode + the "docs-first palace hygiene" policy, three g
The two wrappers in `bin/` close gaps **1** and **2**. Gap **3** is upstream work (see §6).
(Pi is a different story on gap 3: `bin/mempalace-pi-session` closes pi's
version of gap 2 the same way `mempalace-session` closes opencode's, but pi's
gap 3 is already closed **without** any upstream dependency — its bridge
extension self-triggers the feed on `session_shutdown` and a debounced
`agent_settled`. See §3's `mempalace-pi-session` subsection and
[`extensions/pi/README.md`](extensions/pi/README.md).)
---
## 2. The architecture
@@ -53,7 +60,7 @@ The two wrappers in `bin/` close gaps **1** and **2**. Gap **3** is upstream wor
│ │ │ cache dir │
└─────┬──────────┘ └────┬──────────────┘
│ │
│ ~/.cache/mempalace-docs/<wing>/ │ ~/.cache/mempalace-session/<wing>/
│ ~/.cache/mempalace-docs/<wing>/ │ <palace-root>/opencode-stage/<wing>/
│ │
┌─────▼──────────┐ ┌────▼──────────────┐
│ mempalace mine │ │ mempalace mine │
@@ -120,7 +127,7 @@ What it drops: source code (`.py`, `.ts`, `.go`, `.rs`, …), lockfiles, `.git`,
- `step-start` / `step-finish` → dropped as noise.
- `reasoning` → kept, prefixed with `[reasoning]`.
4. Serialize as Claude Code JSONL (`{"type": "user"|"assistant", "message": {"content": [...]}}`) — the one convos format the miner already understands.
5. Stage at `~/.cache/mempalace-session/<wing>/<slug>_<id>.jsonl` with `mtime` = `session.time_updated` (deterministic, stable under dedup).
5. Stage at `<palace-root>/opencode-stage/<wing>/<slug>_<id>.jsonl` with `mtime` = `session.time_updated` (deterministic, stable under dedup).
**Filters:**
@@ -130,6 +137,73 @@ What it drops: source code (`.py`, `.ts`, `.go`, `.rs`, …), lockfiles, `.git`,
**Then:** invokes `mempalace mine --mode convos` against the cache dir. A post-mine `mempalace repair` is **opt-in** via `--repair` — it is intentionally *not* the default because the in-place HNSW rebuild has corrupted live palaces on past runs. Never pass `--repair` from an unattended schedule.
### `bin/mempalace-pi-session` — pi coding-agent → palace bridge
**Input:** pi's own JSONL session transcripts under `~/.pi/agent/sessions/**/*.jsonl`
(no SQLite export step needed — pi already writes files, unlike opencode).
**Output:** palace drawers in `wing_conversations` (or `--wing` override), same
Claude Code JSONL staging shape as `mempalace-session` above.
The transform pipeline is the same shape as `mempalace-session`'s (synthetic
`[session: title | cwd | date | source: pi]` header, per-message dispatch,
`toolCall`→`tool_use`, `toolResult`→`tool_result`, `mtime` copied onto the
staged file for dedup stability) — see the script's own header comment for
the exhaustive per-role mapping. What's architecturally different from the
opencode wrapper is why this one has a **two-phase mode** and a **remote
transport**, neither of which `mempalace-session` needs:
**Two phases, because the palace is single-writer.** `mempalace` enforces
this with a per-palace `flock` (`palace.py`); the CLI's own error is explicit:
`palace ... is held by PID <n> (mempalace-mcp); wait for it to finish`. Unlike
opencode's session mine (which always runs *between* agent sessions, when
nothing else holds the palace), pi's own bridge extension
(`extensions/pi/mempalace.ts`) holds the palace open via its `mempalace-mcp`
subprocess for the *entire* live session — and that extension is also what
triggers the feed, on `session_shutdown` and a debounced `agent_settled`. An
unattended `mempalace mine` invoked from anywhere else during that window
would simply fail. So the wrapper splits:
- `--prepare` — export, threshold, stage (+ `rsync` in remote mode). Never
opens the palace. Prints `MINE_SOURCE=<path>`.
- (default, no `--prepare`) — the above, then mines it. If that mine hits
contention ("is held by"), it's treated as **success, not failure** (exit
0 with an informational message) — the holder's own extension will mine
what got staged. This inverts the CLI's own convention (`MineAlreadyRunning`
→ exit 1) deliberately, because at the wrapper layer the contention has a
benign interpretation the raw CLI can't know about.
The pi extension calls `--prepare` as a subprocess, then feeds `MINE_SOURCE`
to `mempalace_mine` through its own already-open MCP client — the only
process that can write during a live session, because it *is* the lock
holder. This is the mechanism behind [`extensions/pi/README.md` § Automatic
transcript feeding](extensions/pi/README.md#automatic-transcript-feeding).
**Remote transport, because there is no remote-palace CLI.** `mempalace`'s
`--backend` flag selects a *vector store* (chroma/qdrant/pgvector/milvus),
not a remote palace — the only remote surface is the HTTP MCP server
(`mempalace-mcp --transport http`, see [RFC-001](docs/rfc-001-global-palace.md)).
And `mempalace_mine`'s `source` path is expanded **in that server process**,
so a remote server has no way to see this machine's staged exports. `--mode
remote` (auto-selected when `$MEMPALACE_REMOTE_URL` is set) therefore
`rsync`s the stage to a per-device inbox on the palace host, then asks the
server to mine its own local copy of that inbox over the same HTTP
`tools/call` transport the extension uses. Requires
`$MEMPALACE_PI_SSH_TARGET`; see `--help` for the rest
(`MEMPALACE_PI_SSH_CONFIG`, `MEMPALACE_PI_REMOTE_PATH`, `MEMPALACE_PI_DEVICE`).
**Filters:** two gates, both required — stricter than `mempalace-session`'s
single filter because pi's transcripts have a failure mode opencode's don't:
- `--min-messages N` (default 4) — same idea as opencode's filter, raised
because pi's tool loops inflate turn counts fast.
- `--min-assistant-chars N` (default 1000) — counts assistant *text* only,
tool results excluded. Needed because pi expands skill/context text into
the user prompt: an abandoned session can have a huge "user" turn and
almost no assistant output (observed case: 13,380 injected-context user
chars answered with 38 assistant chars), so a message-count-only filter
would have filed 22 pure-noise drawers from that one session. Real
sessions on the same corpus measured 15,900–100,000 assistant chars.
---
## 4. Setup recipe (new machine)
@@ -168,7 +242,11 @@ The devbox uses two named Docker volumes so these persist across container recre
- `devbox-palace` → `~/.mempalace/palace` (the palace itself)
- `devbox-data` → `~/.local/share/opencode` (opencode's SQLite DB)
Code at `/workspace/mempalace-toolkit` is a bind mount from the host — survives container recreate and syncs via gitea. Staging directories (`~/.cache/mempalace-{docs,session}/`) are ephemeral but cheap to rebuild.
Code at `/workspace/mempalace-toolkit` is a bind mount from the host — survives container recreate and syncs via gitea. The docs staging dir (`~/.cache/mempalace-docs/`) is ephemeral and cheap to rebuild.
The **conversation** staging dirs are not: `<palace-root>/opencode-stage/` and
`<palace-root>/pi-stage/` hold the exact paths the palace's `source_file` dedup keys on, so they
live beside the palace deliberately and share its lifetime. Wiping one forces a full re-mine at
best, and lets a scoped `mempalace sync` prune every drawer mined from it at worst.
**After container recreate**, just re-run `./install.sh` (idempotent) to relink `bin/` into the fresh `~/.local/bin/`.
@@ -304,7 +382,7 @@ Quick-start (cron):
```bash
sed "s|USER|$USER|g" contrib/cron/mempalace-session.cron \
| (crontab -l 2>/dev/null; cat) | crontab -
mkdir -p ~/.cache/mempalace-session
mkdir -p ~/.cache/mempalace-logs
```
#### Verification
@@ -323,7 +401,7 @@ A healthy run produces one of:
- **Incremental run**: zero to a few dozen new drawers (whatever grew since last run).
- **Rerun with no new activity**: zero new drawers, only the repair step runs.
A run that files far more drawers than expected may indicate a staging-dir wipe (forcing a full re-mine) — check `~/.cache/mempalace-session/<wing>/` modification times.
A run that files far more drawers than expected may indicate a staging-dir wipe (forcing a full re-mine) — check `<palace-root>/opencode-stage/<wing>/` modification times.
### Cost profile (reference)
@@ -355,7 +433,7 @@ These gaps should ideally close upstream, making the wrappers thinner or obsolet
3. **Opencode harness in `hooks_cli.py`** — mempalace's hooks CLI only knows `claude-code` + `codex` today. Adding `opencode` would let the auto-save diary path work on opencode too. Pairs with #2 above.
4. **SQLite mode for `mempalace mine --mode convos`** — if upstream ever adds direct SQLite ingest for opencode, `mempalace-session` loses its reason to exist (the export-to-JSONL dance goes away).
When #1 merges, retire `mempalace-docs` to a thin shim. When #2 + #3 land together, `mempalace-session` becomes a manual-only fallback (cron / backfill) while hooks handle live saves.
When #1 merges, retire `mempalace-docs` to a thin shim. When #2 + #3 land together, `mempalace-session` becomes a manual-only fallback (cron / backfill) while hooks handle live saves. (`mempalace-pi-session` has no equivalent entry here: pi's bridge extension already self-triggers the feed with no upstream dependency — see §3.)
Separately tracked in [`docs/rfc-001-global-palace.md`](docs/rfc-001-global-palace.md): moving from one palace *per machine* to a **single primary palace with per-machine local fallback** (`mempalace-edge`). That RFC also records upstream items of its own — server-side `origin_device` provenance stamped from a per-device credential, per-wing ACLs, a `mempalace_kg_supersede` tool-classification fix, and a guard against running `mempalace sync` on a shared palace.