diff --git a/AGENTS.md b/AGENTS.md index 2d66608..918d9c1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -152,7 +152,15 @@ For `mempalace-docs`, test on a small repo (e.g. this one) first: - The companion skill lives at `~/.agents/skills/opencode-mempalace-bridge/SKILL.md` and is a **symlink into this repo**. Editing that file edits `SKILL.md` here. To propagate to Claude Code / Kiro, run `agents-sync` from [`cli_utils`](https://gitea.jordbo.se/joakimp/cli_utils). - The opencode DB path defaults to `~/.local/share/opencode/opencode.db`. Override via `$OPENCODE_DB` or `--db`. - The mempalace miner **skips symlinks** (as of v3.3.3 — `miner.py` line ~828). That's why the wrappers use `cp -p` / explicit file writes for staging, not symlinks. -- The convos miner dedups on `source_file` path only (no mtime check). Staging filenames must be stable per session; deleting a staged JSONL forces a re-mine. +- The convos miner dedups on `source_file` path **and** `mtime` + (`file_already_mined(..., check_mtime=True)` in upstream `convo_miner.py`). + A changed/grown transcript is detected, purged, and refiled — it is not + silently skipped. What must still be stable across runs is the *staged + path itself*: if a wrapper's staging dir is wiped, dedup has nothing to + compare against and `mempalace sync` will treat the vanished sources as + deleted and prune the drawers mined from them. (An earlier version of this + file claimed "no mtime check" — that was wrong; verified against + `convo_miner.py` 2026-08.) - The docs miner dedups on `source_file` path + `mtime`. That's why staging uses `cp -p` (preserves mtime). ## Colocated skill pattern diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 1b0cab8..0c52fe7 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -31,6 +31,13 @@ So on a machine using opencode + the "docs-first palace hygiene" policy, three g The two wrappers in `bin/` close gaps **1** and **2**. Gap **3** is upstream work (see §6). +(Pi is a different story on gap 3: `bin/mempalace-pi-session` closes pi's +version of gap 2 the same way `mempalace-session` closes opencode's, but pi's +gap 3 is already closed **without** any upstream dependency — its bridge +extension self-triggers the feed on `session_shutdown` and a debounced +`agent_settled`. See §3's `mempalace-pi-session` subsection and +[`extensions/pi/README.md`](extensions/pi/README.md).) + --- ## 2. The architecture @@ -53,7 +60,7 @@ The two wrappers in `bin/` close gaps **1** and **2**. Gap **3** is upstream wor │ │ │ cache dir │ └─────┬──────────┘ └────┬──────────────┘ │ │ - │ ~/.cache/mempalace-docs// │ ~/.cache/mempalace-session// + │ ~/.cache/mempalace-docs// │ /opencode-stage// │ │ ┌─────▼──────────┐ ┌────▼──────────────┐ │ mempalace mine │ │ mempalace mine │ @@ -120,7 +127,7 @@ What it drops: source code (`.py`, `.ts`, `.go`, `.rs`, …), lockfiles, `.git`, - `step-start` / `step-finish` → dropped as noise. - `reasoning` → kept, prefixed with `[reasoning]`. 4. Serialize as Claude Code JSONL (`{"type": "user"|"assistant", "message": {"content": [...]}}`) — the one convos format the miner already understands. -5. Stage at `~/.cache/mempalace-session//_.jsonl` with `mtime` = `session.time_updated` (deterministic, stable under dedup). +5. Stage at `/opencode-stage//_.jsonl` with `mtime` = `session.time_updated` (deterministic, stable under dedup). **Filters:** @@ -130,6 +137,73 @@ What it drops: source code (`.py`, `.ts`, `.go`, `.rs`, …), lockfiles, `.git`, **Then:** invokes `mempalace mine --mode convos` against the cache dir. A post-mine `mempalace repair` is **opt-in** via `--repair` — it is intentionally *not* the default because the in-place HNSW rebuild has corrupted live palaces on past runs. Never pass `--repair` from an unattended schedule. +### `bin/mempalace-pi-session` — pi coding-agent → palace bridge + +**Input:** pi's own JSONL session transcripts under `~/.pi/agent/sessions/**/*.jsonl` +(no SQLite export step needed — pi already writes files, unlike opencode). +**Output:** palace drawers in `wing_conversations` (or `--wing` override), same +Claude Code JSONL staging shape as `mempalace-session` above. + +The transform pipeline is the same shape as `mempalace-session`'s (synthetic +`[session: title | cwd | date | source: pi]` header, per-message dispatch, +`toolCall`→`tool_use`, `toolResult`→`tool_result`, `mtime` copied onto the +staged file for dedup stability) — see the script's own header comment for +the exhaustive per-role mapping. What's architecturally different from the +opencode wrapper is why this one has a **two-phase mode** and a **remote +transport**, neither of which `mempalace-session` needs: + +**Two phases, because the palace is single-writer.** `mempalace` enforces +this with a per-palace `flock` (`palace.py`); the CLI's own error is explicit: +`palace ... is held by PID (mempalace-mcp); wait for it to finish`. Unlike +opencode's session mine (which always runs *between* agent sessions, when +nothing else holds the palace), pi's own bridge extension +(`extensions/pi/mempalace.ts`) holds the palace open via its `mempalace-mcp` +subprocess for the *entire* live session — and that extension is also what +triggers the feed, on `session_shutdown` and a debounced `agent_settled`. An +unattended `mempalace mine` invoked from anywhere else during that window +would simply fail. So the wrapper splits: + +- `--prepare` — export, threshold, stage (+ `rsync` in remote mode). Never + opens the palace. Prints `MINE_SOURCE=`. +- (default, no `--prepare`) — the above, then mines it. If that mine hits + contention ("is held by"), it's treated as **success, not failure** (exit + 0 with an informational message) — the holder's own extension will mine + what got staged. This inverts the CLI's own convention (`MineAlreadyRunning` + → exit 1) deliberately, because at the wrapper layer the contention has a + benign interpretation the raw CLI can't know about. + +The pi extension calls `--prepare` as a subprocess, then feeds `MINE_SOURCE` +to `mempalace_mine` through its own already-open MCP client — the only +process that can write during a live session, because it *is* the lock +holder. This is the mechanism behind [`extensions/pi/README.md` § Automatic +transcript feeding](extensions/pi/README.md#automatic-transcript-feeding). + +**Remote transport, because there is no remote-palace CLI.** `mempalace`'s +`--backend` flag selects a *vector store* (chroma/qdrant/pgvector/milvus), +not a remote palace — the only remote surface is the HTTP MCP server +(`mempalace-mcp --transport http`, see [RFC-001](docs/rfc-001-global-palace.md)). +And `mempalace_mine`'s `source` path is expanded **in that server process**, +so a remote server has no way to see this machine's staged exports. `--mode +remote` (auto-selected when `$MEMPALACE_REMOTE_URL` is set) therefore +`rsync`s the stage to a per-device inbox on the palace host, then asks the +server to mine its own local copy of that inbox over the same HTTP +`tools/call` transport the extension uses. Requires +`$MEMPALACE_PI_SSH_TARGET`; see `--help` for the rest +(`MEMPALACE_PI_SSH_CONFIG`, `MEMPALACE_PI_REMOTE_PATH`, `MEMPALACE_PI_DEVICE`). + +**Filters:** two gates, both required — stricter than `mempalace-session`'s +single filter because pi's transcripts have a failure mode opencode's don't: + +- `--min-messages N` (default 4) — same idea as opencode's filter, raised + because pi's tool loops inflate turn counts fast. +- `--min-assistant-chars N` (default 1000) — counts assistant *text* only, + tool results excluded. Needed because pi expands skill/context text into + the user prompt: an abandoned session can have a huge "user" turn and + almost no assistant output (observed case: 13,380 injected-context user + chars answered with 38 assistant chars), so a message-count-only filter + would have filed 22 pure-noise drawers from that one session. Real + sessions on the same corpus measured 15,900–100,000 assistant chars. + --- ## 4. Setup recipe (new machine) @@ -168,7 +242,11 @@ The devbox uses two named Docker volumes so these persist across container recre - `devbox-palace` → `~/.mempalace/palace` (the palace itself) - `devbox-data` → `~/.local/share/opencode` (opencode's SQLite DB) -Code at `/workspace/mempalace-toolkit` is a bind mount from the host — survives container recreate and syncs via gitea. Staging directories (`~/.cache/mempalace-{docs,session}/`) are ephemeral but cheap to rebuild. +Code at `/workspace/mempalace-toolkit` is a bind mount from the host — survives container recreate and syncs via gitea. The docs staging dir (`~/.cache/mempalace-docs/`) is ephemeral and cheap to rebuild. +The **conversation** staging dirs are not: `/opencode-stage/` and +`/pi-stage/` hold the exact paths the palace's `source_file` dedup keys on, so they +live beside the palace deliberately and share its lifetime. Wiping one forces a full re-mine at +best, and lets a scoped `mempalace sync` prune every drawer mined from it at worst. **After container recreate**, just re-run `./install.sh` (idempotent) to relink `bin/` into the fresh `~/.local/bin/`. @@ -304,7 +382,7 @@ Quick-start (cron): ```bash sed "s|USER|$USER|g" contrib/cron/mempalace-session.cron \ | (crontab -l 2>/dev/null; cat) | crontab - -mkdir -p ~/.cache/mempalace-session +mkdir -p ~/.cache/mempalace-logs ``` #### Verification @@ -323,7 +401,7 @@ A healthy run produces one of: - **Incremental run**: zero to a few dozen new drawers (whatever grew since last run). - **Rerun with no new activity**: zero new drawers, only the repair step runs. -A run that files far more drawers than expected may indicate a staging-dir wipe (forcing a full re-mine) — check `~/.cache/mempalace-session//` modification times. +A run that files far more drawers than expected may indicate a staging-dir wipe (forcing a full re-mine) — check `/opencode-stage//` modification times. ### Cost profile (reference) @@ -355,7 +433,7 @@ These gaps should ideally close upstream, making the wrappers thinner or obsolet 3. **Opencode harness in `hooks_cli.py`** — mempalace's hooks CLI only knows `claude-code` + `codex` today. Adding `opencode` would let the auto-save diary path work on opencode too. Pairs with #2 above. 4. **SQLite mode for `mempalace mine --mode convos`** — if upstream ever adds direct SQLite ingest for opencode, `mempalace-session` loses its reason to exist (the export-to-JSONL dance goes away). -When #1 merges, retire `mempalace-docs` to a thin shim. When #2 + #3 land together, `mempalace-session` becomes a manual-only fallback (cron / backfill) while hooks handle live saves. +When #1 merges, retire `mempalace-docs` to a thin shim. When #2 + #3 land together, `mempalace-session` becomes a manual-only fallback (cron / backfill) while hooks handle live saves. (`mempalace-pi-session` has no equivalent entry here: pi's bridge extension already self-triggers the feed with no upstream dependency — see §3.) Separately tracked in [`docs/rfc-001-global-palace.md`](docs/rfc-001-global-palace.md): moving from one palace *per machine* to a **single primary palace with per-machine local fallback** (`mempalace-edge`). That RFC also records upstream items of its own — server-side `origin_device` provenance stamped from a per-device credential, per-wing ACLs, a `mempalace_kg_supersede` tool-classification fix, and a guard against running `mempalace sync` on a shared palace. diff --git a/README.md b/README.md index 6d497af..176e8d2 100644 --- a/README.md +++ b/README.md @@ -5,6 +5,7 @@ Producer-side tooling for [MemPalace](https://github.com/MemPalace/mempalace) **What this repo contains:** - `bin/mempalace-session` — exports [opencode](https://github.com/anomalyco/opencode) session history from its local SQLite DB to Claude Code JSONL, then mines it via `mempalace mine --mode convos`. +- `bin/mempalace-pi-session` — the same idea for the [pi coding-agent](https://github.com/earendil-works/pi): exports its native JSONL session transcripts and mines them the same way. Unlike `mempalace-session`, this one is normally invoked *for you* — the pi bridge extension (below) runs it automatically on `session_shutdown` and a debounced `agent_settled`, so most pi machines never need the `contrib/` scheduling templates at all. See [`mempalace-pi-session`](#mempalace-pi-session) below and [`extensions/pi/README.md` § Automatic transcript feeding](extensions/pi/README.md#automatic-transcript-feeding). - `bin/mempalace-docs` — mines project directories into MemPalace while excluding source code, keeping the palace signal-dense. - [`ARCHITECTURE.md`](ARCHITECTURE.md) — **canonical spec**: architecture diagram, component details, setup recipe, operational notes, upstream-retirement roadmap. - [`SKILL.md`](SKILL.md) — the companion agent skill, symlinked into `~/.agents/skills/opencode-mempalace-bridge/` on install. @@ -323,7 +324,17 @@ A machine running only one of these has half a memory. Full treatment with pract ### Keeping it fresh (automation) -Manual invocation is fine while you're actively driving the machine, but long-running devboxes benefit from a weekly automated mine. [`contrib/`](contrib/) ships ready-to-install templates: +**pi:** nothing to set up. The pi bridge extension +(`extensions/pi/mempalace.ts`) feeds the palace by itself — see +[`extensions/pi/README.md` § Automatic transcript feeding](extensions/pi/README.md#automatic-transcript-feeding). +The templates below aren't needed unless you're running +`mempalace-pi-session` somewhere without that extension (e.g. a bare pi +install, or a host-side catch-up job). + +**opencode**, and pi installs without the bridge: manual invocation is fine +while you're actively driving the machine, but long-running devboxes benefit +from a weekly automated mine. [`contrib/`](contrib/) ships ready-to-install +templates: - **systemd user timer** (recommended on Linux): survives reboots, catches missed runs, logs to `journalctl`. - **launchd user agent** (recommended on macOS): native-equivalent — logs to `~/Library/Logs/`, single-instance guarantees, `ProcessType=Background` throttling. @@ -421,12 +432,12 @@ mempalace-session --help - Tool outputs → `tool_result` blocks in a follow-up human message, folded back into the assistant turn by the mempalace normalizer. - `step-start` / `step-finish` parts are dropped as noise. `reasoning` parts are kept with a `[reasoning]` prefix. -**Dedup:** staging at `~/.cache/mempalace-session//` with deterministic per-session filenames (`_.jsonl`). The convos miner keys on `source_file`, so re-runs skip unchanged sessions. To force re-mining a session, delete its JSONL from the staging dir. +**Dedup:** staging at `/opencode-stage//` (override: `$MEMPALACE_SESSION_STAGE`) with deterministic per-session filenames (`_.jsonl`). The convos miner keys on `source_file`, so re-runs skip unchanged sessions. To force re-mining a session, delete its JSONL from the staging dir. **`--dry-run` is dedup-aware.** Each session is tagged `[NEW]` (would be filed) or `[SKIP]` (already in the palace), and the summary breaks down the count: ``` -Exported 62 session(s) to ~/.cache/mempalace-session/wing_conversations +Exported 62 session(s) to /home/you/.mempalace/opencode-stage/wing_conversations 0 new → will be filed on mine 62 already filed → will be skipped (dedup by source_file) @@ -441,6 +452,105 @@ If the palace is unreachable (fresh install, moved, permission-denied) the wrapp --- +## `mempalace-pi-session` + +Pi coding-agent → MemPalace session bridge. Reads pi's own JSONL session +transcripts under `~/.pi/agent/sessions/`, converts each qualifying session +to Claude Code JSONL, and files via `mempalace mine --mode convos`. On most +machines you never run this by hand — the pi bridge extension +(`extensions/pi/mempalace.ts`) invokes it automatically; see +[`extensions/pi/README.md` § Automatic transcript feeding](extensions/pi/README.md#automatic-transcript-feeding). +Manual invocation is for a bare pi install without that extension, a +host-side catch-up job, or just poking at the export by hand. + +```bash +mempalace-pi-session # export + mine everything qualifying +mempalace-pi-session --wing my_convos # custom wing (default: wing_conversations) +mempalace-pi-session --session # one session only +mempalace-pi-session --since 2026-04-01 # only sessions updated on/after date +mempalace-pi-session --min-messages 6 # stricter turn-count filter (default: 4) +mempalace-pi-session --min-assistant-chars 2000 # stricter "did anything happen" filter (default: 1000) +mempalace-pi-session --dry-run # export + list, skip mine +mempalace-pi-session --prepare # export + stage only; print MINE_SOURCE=, never opens the palace +mempalace-pi-session --mode remote # ship the stage to a remote palace host instead of mining locally +mempalace-pi-session --help +``` + +**Why `--prepare` exists:** the palace is single-writer. If a pi session for +this same palace is currently open, its own bridge extension already holds +the palace via `mempalace-mcp`, so an unattended `mempalace mine` from +anywhere else fails with `palace ... is held by PID `. `--prepare` does +only the palace-free half (export, threshold, staging, and the `rsync` in +remote mode) and hands the mine off to whoever already holds the lock — +which is exactly what the pi extension does with its own MCP client. Run +without `--prepare`, the tool does the mine itself, and treats that +contention as success (exit 0, informational message) rather than failure — +the holder will mine what got staged. + +**What gets exported per session:** + +- Synthetic header injected as the first user turn + (`[session: | <cwd> | <date> | source: pi]`) so the palace can find + sessions by topic, not just by ID, and so pi/opencode/other results stay + distinguishable in search. +- User/assistant messages extracted from pi's JSONL `message` entries. +- Assistant `toolCall` blocks → Claude Code `tool_use` blocks. +- `toolResult` role messages → `tool_result` blocks, folded back into the + assistant turn by the mempalace normalizer. +- `bashExecution`, `custom` (display-only), `branchSummary`, + `compactionSummary` → rendered as text annotations. +- `thinking` blocks and image content → dropped (noise / not text). + +**Filter — two gates, both required** (this is stricter than +`mempalace-session`'s single message-count filter, and deliberately so — +see below): + +1. `--min-messages` user+assistant turns (default **4**). +2. `--min-assistant-chars` characters of assistant *text*, tool results + excluded (default **1000**). + +The second gate exists because message count alone isn't enough for pi: pi +expands skill/context text into the user prompt, so an abandoned session can +have a huge "user" turn and almost nothing on the assistant side — e.g. a +real observed case with a 13,380-char injected-context user turn answered +"Ready. What would you like to work on?" (38 chars). Total size said +substantial; assistant size correctly said nothing happened. Measured real +sessions on the same machine ran 15,900–100,000 assistant chars, so the +1000-char default sits with wide margin on both sides. + +**Dedup:** staging under `$MEMPALACE_PI_STAGE/<wing>/` (default +`<palace-root>/pi-stage/<wing>/` — beside the palace, so the stage cannot be +wiped independently of the dedup keys pointing at it) with deterministic per-session-UUID +filenames, and the export preserves the source session's `mtime` on the +staged file. The convos miner is mtime-aware (see the Gotchas in +[`AGENTS.md`](AGENTS.md) — an older version of this doc claimed otherwise), +so re-runs on an unchanged session are a no-op, and re-feeding a **grown** +session (a live one being fed mid-conversation) purges and refiles that +session's drawers instead of duplicating them. + +**Staging must persist.** Dedup keys on the *staged* path, not the original +transcript, so if the stage is wiped, a `mempalace sync` scoped to include it +sees those source files as gone and prunes the drawers mined from them — +deleting the memories, not just the cache. This is why the stage now defaults to +`<palace-root>/pi-stage`: it inherits whatever persistence the palace has, so +the files and the dedup keys that reference them cannot be separated by +wiping something that merely looks disposable. Override with +`$MEMPALACE_PI_STAGE` only if you have somewhere *more* durable than the palace. + +**Remote palace:** if `$MEMPALACE_REMOTE_URL` is set, there is no +remote-palace CLI to mine into directly — `mempalace_mine` expands its +source path in the *server* process, which cannot see this machine's staged +exports. `--mode remote` (or `--mode auto`, which detects +`$MEMPALACE_REMOTE_URL`) instead `rsync`s the stage to a per-device inbox on +the palace host and asks the server to mine its own local copy. Requires +`$MEMPALACE_PI_SSH_TARGET` (`user@host:path`); see `MEMPALACE_PI_SSH_CONFIG`, +`MEMPALACE_PI_REMOTE_PATH`, and `MEMPALACE_PI_DEVICE` in `--help` for the +rest. Deploying that primary — newt, DNS, and why the auth is a shared bearer +token rather than per-device proxy users — is +[`docs/phase-1-exposure-runbook.md`](docs/phase-1-exposure-runbook.md). + +--- + ## Companion agent skill Installing this repo symlinks `SKILL.md` into `~/.agents/skills/opencode-mempalace-bridge/SKILL.md`, where it's auto-discovered by opencode (and by Claude Code / Kiro if you run `agents-sync` from [`cli_utils`](https://gitea.jordbo.se/joakimp/cli_utils)). diff --git a/SKILL.md b/SKILL.md index 4bf93d8..7e2f1a3 100644 --- a/SKILL.md +++ b/SKILL.md @@ -20,6 +20,13 @@ The `mempalace` skill covers *using* the palace (search, diary, KG). This skill Both follow the same **stage-to-cache-then-mine** idiom — they curate input into `~/.cache/…/<wing>/`, then delegate to `mempalace mine`. +**Pi is out of scope for this skill.** The pi coding-agent has its own +wrapper, `mempalace-pi-session`, but — unlike opencode — pi's bridge +extension (`extensions/pi/mempalace.ts`) invokes it automatically on session +shutdown and a debounced mid-session tick, so a pi machine with that +extension installed needs none of this skill's manual/scheduled recipe. See +`extensions/pi/README.md` § Automatic transcript feeding in the same repo. + ## When to Load This Skill - User asks "how does the palace get fed?" or mentions setting up mempalace on a new machine. @@ -86,7 +93,14 @@ A docs-heavy repo should produce ~5–10 drawers per file. >15 drawers/file on a ### Dedup is free — re-running is safe - `mempalace-docs`: dedup keyed on `source_file` path + `mtime`. Unchanged files skipped. -- `mempalace-session`: dedup keyed on `source_file` path alone (no mtime check for convos). Staging filenames are deterministic per session (`<slug>_<id>.jsonl`), so re-runs skip already-filed sessions. +- `mempalace-session` / `mempalace-pi-session`: the convos miner also dedups + on `source_file` path + `mtime` (`file_already_mined(..., check_mtime=True)` + in upstream `convo_miner.py` — a prior version of this line said "no mtime + check", which was wrong). Staging filenames are deterministic per session, + so a re-run on unchanged content is a no-op, and a grown/changed session is + detected, purged, and refiled rather than duplicated. What must stay + stable is the staging *path itself* — wiping the staging dir makes dedup + (and `mempalace sync`) treat those sources as gone. Second run immediately after first → 0 new drawers, only the post-mine `repair` step runs (~5 min on 5k drawers). @@ -110,8 +124,10 @@ mempalace-session --session ses_abc123 # one specific session ### Force re-mine ```bash -rm -rf ~/.cache/mempalace-session/<wing>/ # nukes staging dir +rm -rf <palace-root>/opencode-stage/<wing>/ # nukes staging dir mempalace-session # stages + mines fresh +# Do NOT run `mempalace sync` between those two commands: with the stage gone, +# a scoped sync prunes the drawers mined from it instead of refiling them. ``` Staging is ephemeral by design; the palace is the source of truth. diff --git a/bin/mempalace-pi-session b/bin/mempalace-pi-session index 9d1fa8b..1b51286 100755 --- a/bin/mempalace-pi-session +++ b/bin/mempalace-pi-session @@ -10,22 +10,88 @@ # Strategy (mirrors mempalace-session): # 1. Walk ~/.pi/agent/sessions/**/*.jsonl and export each qualifying session # to a Claude Code JSONL file (format the mempalace normalizer speaks). -# 2. Stage exports under ~/.cache/mempalace-pi-session/<wing>/. +# 2. Stage exports under $MEMPALACE_PI_STAGE/<wing> (default +# <palace-root>/pi-stage/<wing> — alongside the palace it feeds). # 3. Run `mempalace mine --mode convos` against the staging dir. # +# TWO PHASES (--prepare), because the palace is single-writer +# mempalace refuses a CLI mine while another process holds the palace: +# "palace ... is held by PID <n> (mempalace-mcp); wait for it to finish" +# A live pi session ALWAYS has a holder — the mempalace extension's own +# mempalace-mcp. So an unattended CLI mine only works when no session is +# live (e.g. a container-start catch-up); during a session the mine must be +# performed by the process that already holds the palace. Hence: +# +# --prepare export + stage (+ rsync in remote mode) and print +# MINE_SOURCE=<path>, without ever opening the palace. +# (default) the above, then mine it ourselves. Contention is treated as +# success-with-nothing-to-do, not failure: the holder's own +# extension will mine what we staged. +# +# The pi mempalace extension drives exactly this: it runs --prepare on +# session_shutdown and on a debounced agent_settled, then calls +# mempalace_mine on MINE_SOURCE through its existing MCP client. +# +# TRANSPORTS (--mode, default auto) +# local Mine into the local palace with the mempalace CLI. +# remote $MEMPALACE_REMOTE_URL is set, so the palace lives on another host. +# There is no remote-palace CLI — only the HTTP MCP server — and +# mempalace_mine expands its source path in the SERVER process, so +# the server cannot see this machine's staged exports. We therefore +# rsync the stage into a per-device inbox on the palace host and ask +# the server to mine its own local path. Requires +# MEMPALACE_PI_SSH_TARGET (where to rsync) and +# MEMPALACE_PI_REMOTE_PATH (what that inbox is called server-side). +# # Labelling: every exported transcript begins with a synthetic header # [session: <title> | <cwd> | <YYYY-MM-DD> | source: pi] # so post-mine search results are self-identifying (pi vs opencode vs other). # # Dedup: mempalace convos mode keys on source_file (absolute staging path). -# Staging paths are deterministic per pi session UUID, so re-runs are -# idempotent until session content actually changes. +# Staging paths are deterministic per pi session UUID, and the export copies +# the source session's mtime onto the staged file, so re-runs are idempotent +# until session content actually changes. A GROWN session is purged and +# refiled for that source_file by the miner, so re-feeding a live session +# refreshes its drawers instead of duplicating them. # -# Session filter: sessions with fewer than --min-messages *user+assistant* -# messages (default 3) are skipped to avoid filing single-prompt throwaways. +# Staging location: source_file dedup keys on the staged path, so if the stage +# is wiped the palace is left with drawers whose source files look deleted. +# `mempalace sync` prunes exactly those — but only within the scope it is +# given. Measured on this layout: scoped at the palace root the staged sources +# are in scope (kept 651), while a wing-only sync reports them out_of_scope and +# leaves them alone. So the data loss is conditional on how sync is invoked, +# which is far too thin a margin to rely on. +# +# The stage therefore defaults NEXT TO THE PALACE (<palace-root>/pi-stage, +# resolved the way mempalace itself resolves the palace: $MEMPALACE_PALACE_PATH +# → $MEMPAL_PALACE_PATH → ~/.mempalace/config.json → ~/.mempalace/palace). +# +# That makes the invariant structural rather than documented: the stage and the +# dedup keys that reference it share one lifetime, so the dangerous state — +# palace survives, stage does not — can no longer be reached by wiping +# something that merely looks disposable. A cache dir (the obvious choice, and +# the old default) is exactly wrong here: it persists just long enough to look +# correct, then takes the memories with it. Override with MEMPALACE_PI_STAGE +# only if the target is at least as durable as the palace. +# +# In remote mode the local stage is only a shipping buffer — dedup lives on the +# server, keyed by the server-side inbox path — so its durability is moot there. +# +# Session filter: two gates, both required. +# 1. --min-messages <N> user+assistant turns (default 4). Tool loops inflate +# assistant turns fast in pi, so a real working session clears this +# easily; a single abandoned prompt does not. +# 2. --min-assistant-chars <N> characters of assistant *text* (default 1000), +# excluding tool results. Assistant volume, not total volume: pi expands +# skills/context into the user prompt, so an abandoned session can carry a +# 13k-char "user" message answered with "Ready. What would you like to +# work on?" — total size says substantial, assistant size correctly says +# nothing happened. # # Usage: # mempalace-pi-session +# mempalace-pi-session --prepare +# mempalace-pi-session --mode remote # mempalace-pi-session --wing <name> # mempalace-pi-session --session <uuid-prefix> # mempalace-pi-session --since 2026-04-01 @@ -34,13 +100,16 @@ # mempalace-pi-session --help # # Exit codes: -# 0 success +# 0 success (including "nothing qualified", "another run holds the lock", +# and "palace held by a live session") # 1 usage / argument error # 2 pi sessions dir missing -# 3 mempalace CLI not installed +# 3 mempalace CLI not installed / rsync missing in remote mode # 4 mine failed +# 5 remote transport failed (rsync or HTTP tools/call) # -# Dependencies: bash, python3 (stdlib only), mempalace (v3.3.3+) +# Dependencies: bash, python3 (stdlib only), mempalace (v3.3.3+); +# rsync + ssh in remote mode. set -euo pipefail @@ -49,11 +118,50 @@ AGENT="${USER:-mempalace}" WING="wing_conversations" SESSION_ID="" SINCE="" -MIN_MESSAGES=3 +MIN_MESSAGES=4 +MIN_ASSISTANT_CHARS=1000 DRY_RUN=0 DO_REPAIR=0 +PREPARE_ONLY=0 +MODE="auto" +REASON="" PI_SESSIONS_DIR="${PI_SESSIONS_DIR:-$HOME/.pi/agent/sessions}" +# Resolve the palace ROOT (the dir holding palace/, knowledge_graph.sqlite3, +# config.json) using mempalace's own precedence, so the stage lands next to +# whichever palace this host actually feeds. Mirrors config.py:palace_path() +# (env → config.json → default) and takes the parent. Only evaluated when +# MEMPALACE_PI_STAGE is unset, so the common path costs nothing. +palace_root() { + python3 - <<'PY' 2>/dev/null || echo "$HOME/.mempalace" +import json, os +p = os.environ.get("MEMPALACE_PALACE_PATH") or os.environ.get("MEMPAL_PALACE_PATH") +if p: + p = os.path.abspath(os.path.expanduser(p)) +else: + cfg = os.path.expanduser("~/.mempalace/config.json") + p = None + if os.path.exists(cfg): + try: + with open(cfg) as fh: + v = json.load(fh).get("palace_path") + p = os.path.expanduser(v) if v else None + except Exception: + p = None + p = p or os.path.expanduser("~/.mempalace/palace") +print(os.path.dirname(p.rstrip("/"))) +PY +} +STAGE_ROOT="${MEMPALACE_PI_STAGE:-$(palace_root)/pi-stage}" + +# Remote transport (see TRANSPORTS in the header) +REMOTE_URL="${MEMPALACE_REMOTE_URL:-}" +REMOTE_TOKEN="${MEMPALACE_REMOTE_TOKEN:-}" +SSH_TARGET="${MEMPALACE_PI_SSH_TARGET:-}" +SSH_CONFIG="${MEMPALACE_PI_SSH_CONFIG:-}" +REMOTE_PATH="${MEMPALACE_PI_REMOTE_PATH:-/data/feed}" +DEVICE="${MEMPALACE_PI_DEVICE:-$(hostname)}" + # ── Usage ──────────────────────────────────────────────────────────── usage() { cat <<'EOF' @@ -67,10 +175,28 @@ Options: --session <prefix> Export one session only (match on UUID prefix) --since <YYYY-MM-DD> Only sessions last modified on/after this date --min-messages <N> Skip sessions with fewer than N user+assistant - turns (default: 3) + turns (default: 4) + --min-assistant-chars <N> + Skip sessions with fewer than N characters of + assistant text, tool results excluded (default: 1000). + Catches abandoned sessions whose bulk is injected + skill/context text in the user prompt. --agent <name> Agent name recorded on drawers (default: $USER) --sessions-dir <path> Path to pi sessions dir (default: $PI_SESSIONS_DIR or ~/.pi/agent/sessions) + --stage <path> Staging root (default: $MEMPALACE_PI_STAGE, else + <palace-root>/pi-stage — next to the palace, so the + stage cannot be wiped independently of the dedup keys + that point at it). Exports go in <root>/<wing>. + See "Staging location" in the header before moving it. + --mode <m> auto|local|remote (default: auto — remote when + $MEMPALACE_REMOTE_URL is set) + --prepare Export + stage (+ rsync in remote mode), print + MINE_SOURCE=<path>, and stop without opening the + palace. For callers that will do the mine themselves + through a live MCP connection. + --reason <label> Label this run in its output (e.g. shutdown, tick, + container-start). Useful when triggers log to a file. --dry-run Export + list; do not mine into palace. Each session is tagged [NEW] or [SKIP] based on whether its source_file is already in the palace. @@ -102,14 +228,27 @@ Transcript shape per session: Dedup: - source_file = absolute staging path (deterministic per pi session UUID) - - Re-runs skip unchanged sessions. To force re-mining, delete the staging - dir: rm -rf ~/.cache/mempalace-pi-session/<wing>/ + - Re-runs skip unchanged sessions; a GROWN session (mtime changed) has its + old drawers purged and is refiled, so re-feeding a live session refreshes + rather than duplicates. + - To force re-mining, delete the staging dir: + rm -rf <palace-root>/pi-stage/<wing>/ + That forces a refile — but do NOT run `mempalace sync` while the stage is + missing, or the drawers mined from it get pruned instead. Rationale: - Pi's extension ecosystem could, in principle, stream sessions into the - palace as they happen, but that requires a resident MCP connection during - every pi session. This wrapper is the batch, harness-agnostic alternative: - it reads the durable on-disk JSONL and mines it on a schedule. + Two complementary paths feed the palace from pi, and they cover different + failure modes: + - The pi mempalace bridge extension (extensions/pi/mempalace.ts) drives + this script with --prepare on session_shutdown and on a debounced + agent_settled, then mines through its own live MCP connection. That is + the primary path: it needs no scheduling and it is the only way to write + while a session holds the palace. + - Running this script directly is the batch/recovery path: a + container-start or host-level catch-up that picks up transcripts nothing + mined at the time — notably after a SIGKILL, where no pi handler runs at + all. It reads the durable on-disk JSONL, so it does not care whether the + session that produced it exited cleanly. EOF } @@ -121,6 +260,11 @@ while [[ $# -gt 0 ]]; do --session) SESSION_ID="${2:-}"; shift 2 ;; --since) SINCE="${2:-}"; shift 2 ;; --min-messages) MIN_MESSAGES="${2:-}"; shift 2 ;; + --min-assistant-chars) MIN_ASSISTANT_CHARS="${2:-}"; shift 2 ;; + --stage) STAGE_ROOT="${2:-}"; shift 2 ;; + --mode) MODE="${2:-}"; shift 2 ;; + --prepare) PREPARE_ONLY=1; shift ;; + --reason) REASON="${2:-}"; shift 2 ;; --agent) AGENT="${2:-}"; shift 2 ;; --sessions-dir) PI_SESSIONS_DIR="${2:-}"; shift 2 ;; --dry-run) DRY_RUN=1; shift ;; @@ -138,32 +282,67 @@ if [[ ! -d "$PI_SESSIONS_DIR" ]]; then echo " override with --sessions-dir <path> or PI_SESSIONS_DIR env var" >&2 exit 2 fi -if ! command -v mempalace >/dev/null 2>&1; then +case "$MODE" in + auto) if [[ -n "$REMOTE_URL" ]]; then MODE="remote"; else MODE="local"; fi ;; + local|remote) ;; + *) echo "error: --mode must be auto|local|remote" >&2; exit 1 ;; +esac +# The mempalace CLI is only needed when WE do the mine. --prepare never opens +# the palace, and remote mode talks to the server over HTTP. +if [[ $PREPARE_ONLY -eq 0 && "$MODE" == "local" ]] && ! command -v mempalace >/dev/null 2>&1; then echo "error: mempalace CLI not found in PATH" >&2 exit 3 fi -if ! [[ "$MIN_MESSAGES" =~ ^[0-9]+$ ]]; then - echo "error: --min-messages must be an integer" >&2 - exit 1 +if [[ "$MODE" == "remote" ]]; then + command -v rsync >/dev/null 2>&1 || { echo "error: rsync not found (needed for --mode remote)" >&2; exit 3; } + if [[ -z "$SSH_TARGET" ]]; then + echo "error: MEMPALACE_PI_SSH_TARGET unset (needed for --mode remote)" >&2 + exit 1 + fi + # The devbox generates a dedicated LAN-jump key/config; prefer it if present. + if [[ -z "$SSH_CONFIG" && -f "$HOME/.ssh-local/config" ]]; then + SSH_CONFIG="$HOME/.ssh-local/config" + fi fi +for _n in MIN_MESSAGES MIN_ASSISTANT_CHARS; do + if ! [[ "${!_n}" =~ ^[0-9]+$ ]]; then + _flag="--$(printf '%s' "${_n,,}" | tr '_' '-')" + echo "error: $_flag must be an integer" >&2 + exit 1 + fi +done # ── Staging dir ────────────────────────────────────────────────────── -CACHE_ROOT="${XDG_CACHE_HOME:-$HOME/.cache}/mempalace-pi-session" -STAGE="$CACHE_ROOT/$WING" +# Deterministic per-wing path so source_file dedup works across re-runs. See +# "Staging location" in the header for why this should not be disposable. +STAGE="${STAGE_ROOT%/}/$WING" mkdir -p "$STAGE" +[[ -n "$REASON" ]] && echo "mempalace-pi-session [$REASON] mode=$MODE stage=$STAGE" + +# ── Single-writer guard ────────────────────────────────────────────── +# Non-blocking: overlapping triggers (a session_shutdown landing on top of a +# debounced mid-session run) must not queue or race. Losing a run is harmless +# — the next one re-exports from scratch. +exec 9>"${STAGE_ROOT%/}/.lock" +if command -v flock >/dev/null 2>&1 && ! flock -n 9; then + echo "another mempalace-pi-session run holds the lock; skipping" + exit 0 +fi + # ── Export sessions (Python heredoc) ──────────────────────────────── # Parses pi JSONL files and writes Claude Code JSONL per session into $STAGE. # Also classifies each export as NEW/ALREADY FILED (by source_file lookup) # so --dry-run reports the real mine-set size. Classification is advisory; # `mempalace mine --mode convos` is still the authoritative dedup. -export_count=$(python3 - "$PI_SESSIONS_DIR" "$STAGE" "$SESSION_ID" "$SINCE" "$MIN_MESSAGES" <<'PY' +export_count=$(python3 - "$PI_SESSIONS_DIR" "$STAGE" "$SESSION_ID" "$SINCE" "$MIN_MESSAGES" "$MIN_ASSISTANT_CHARS" <<'PY' import json, os, sqlite3, sys from datetime import datetime, timezone from pathlib import Path -sessions_dir, stage, session_filter, since, min_messages = sys.argv[1:6] +sessions_dir, stage, session_filter, since, min_messages, min_assistant_chars = sys.argv[1:7] min_messages = int(min_messages) +min_assistant_chars = int(min_assistant_chars) stage = Path(stage) sessions_dir = Path(sessions_dir) @@ -178,7 +357,15 @@ if since: # ── Load palace's already-filed source_files (best-effort, read-only) ── already_filed = set() -palace_path = os.environ.get("MEMPALACE_PATH", os.path.expanduser("~/.mempalace/palace")) +# Mirror mempalace's own resolution order (config.py): MEMPALACE_PALACE_PATH, +# then the legacy MEMPAL_PALACE_PATH, then the default. NOT "MEMPALACE_PATH" — +# that name is not a mempalace concept, and reading it silently degraded this +# NEW/SKIP preview to "everything is new" wherever some other tool had set it. +palace_path = ( + os.environ.get("MEMPALACE_PALACE_PATH") + or os.environ.get("MEMPAL_PALACE_PATH") + or os.path.expanduser("~/.mempalace/palace") +) chroma_db = Path(palace_path) / "chroma.sqlite3" if chroma_db.is_file(): try: @@ -282,6 +469,7 @@ if session_filter: exported = 0 skipped_short = 0 +skipped_quiet = 0 skipped_malformed = 0 skipped_already_filed = 0 @@ -318,6 +506,7 @@ for path in paths: continue title = derive_title(entries, fallback=session_uuid[:8]) + assistant_chars = 0 out_lines = [] out_lines.append({ "type": "user", @@ -337,6 +526,7 @@ for path in paths: out_lines.append({"type": "user", "message": {"content": text}}) elif role == "assistant": text, tool_uses, _ = extract_text(msg.get("content")) + assistant_chars += len(text.strip()) blocks = [] if text.strip(): blocks.append({"type": "text", "text": text}) @@ -406,6 +596,17 @@ for path in paths: skipped_short += 1 continue + # Assistant *text* volume, tool results excluded: the signal that the + # session actually did something, independent of how much injected + # skill/context text inflated the user side. + if assistant_chars < min_assistant_chars: + skipped_quiet += 1 + print( + f" [QUIET] {path.name} ({turn_count} turns, {assistant_chars} assistant chars)", + file=sys.stderr, + ) + continue + out_path = stage / f"pi_{session_uuid}.jsonl" with out_path.open("w", encoding="utf-8") as f: for obj in out_lines: @@ -428,6 +629,8 @@ print(f"EXPORTED {exported}") print(f"ALREADY_FILED {skipped_already_filed}") if skipped_short: print(f"SKIPPED_SHORT {skipped_short}", file=sys.stderr) +if skipped_quiet: + print(f"SKIPPED_QUIET {skipped_quiet}", file=sys.stderr) if skipped_malformed: print(f"SKIPPED_MALFORMED {skipped_malformed}", file=sys.stderr) PY @@ -461,12 +664,89 @@ if [[ $DRY_RUN -eq 1 ]]; then exit 0 fi +# ── Ship to the palace host (remote mode only) ─────────────────────── +# mempalace_mine expands its source path in the SERVER process, so in remote +# mode the exports have to physically exist over there. rsync --update is the +# idempotent half; the mine is the other half. +MINE_SOURCE="$STAGE" +if [[ "$MODE" == "remote" ]]; then + ssh_cmd="ssh" + [[ -n "$SSH_CONFIG" ]] && ssh_cmd="ssh -F $SSH_CONFIG" + echo "" + echo "Shipping stage to ${SSH_TARGET%/}/$DEVICE/ ..." + if ! rsync -a --update --no-owner --no-group \ + -e "$ssh_cmd" \ + --include='*.jsonl' --exclude='*' \ + "$STAGE/" "${SSH_TARGET%/}/$DEVICE/"; then + echo "error: rsync to ${SSH_TARGET%/}/$DEVICE/ failed" >&2 + exit 5 + fi + MINE_SOURCE="${REMOTE_PATH%/}/$DEVICE" +fi + +# ── Phase boundary ─────────────────────────────────────────────────── +# --prepare hands the source path to the caller (the pi mempalace extension), +# which mines it through the MCP client that already holds the palace. +if [[ $PREPARE_ONLY -eq 1 ]]; then + echo "" + printf 'MINE_SOURCE=%s\n' "$MINE_SOURCE" + exit 0 +fi + # ── Run the mine ───────────────────────────────────────────────────── echo "" echo "Mining into wing '$WING'..." -if ! mempalace mine "$STAGE" --mode convos --wing "$WING" --agent "$AGENT"; then - echo "error: mempalace mine failed" >&2 - exit 4 +if [[ "$MODE" == "remote" ]]; then + if ! python3 - "$REMOTE_URL" "$REMOTE_TOKEN" "$MINE_SOURCE" "$WING" "$AGENT" <<'PY' +import json, sys, urllib.error, urllib.request + +url, token, source, wing, agent = sys.argv[1:6] +payload = json.dumps({ + "jsonrpc": "2.0", + "id": 1, + "method": "tools/call", + "params": { + "name": "mempalace_mine", + "arguments": {"source": source, "mode": "convos", "wing": wing, "agent": agent}, + }, +}).encode() +headers = {"Content-Type": "application/json", "Accept": "application/json"} +if token: + headers["Authorization"] = f"Bearer {token}" +try: + with urllib.request.urlopen( + urllib.request.Request(url, data=payload, headers=headers), timeout=900 + ) as resp: + body = resp.read().decode("utf-8", "replace") +except urllib.error.URLError as exc: + print(f"error: remote mine transport failed: {exc}", file=sys.stderr) + sys.exit(1) +print(body[:4000]) +sys.exit(1 if '"error"' in body else 0) +PY + then + echo "error: remote mine failed" >&2 + exit 5 + fi +else + # Capture output so palace-level contention can be told apart from a real + # failure. A live pi session holds the palace through its own mempalace-mcp, + # and that session's extension mines what we just staged — so contention + # means "already handled", not "broken". + set +e + mine_out="$(mempalace mine "$MINE_SOURCE" --mode convos --wing "$WING" --agent "$AGENT" 2>&1)" + mine_rc=$? + set -e + printf '%s\n' "$mine_out" + if [[ $mine_rc -ne 0 ]]; then + if printf '%s' "$mine_out" | grep -q "is held by"; then + echo "" + echo "palace is held by a live session; it will mine the staged exports itself" + exit 0 + fi + echo "error: mempalace mine failed" >&2 + exit 4 + fi fi # ── Repair index ───────────────────────────────────────────────────── diff --git a/bin/mempalace-session b/bin/mempalace-session index 846ad92..7599228 100755 --- a/bin/mempalace-session +++ b/bin/mempalace-session @@ -10,13 +10,26 @@ # Strategy: # 1. Read opencode.db and export each qualifying session to a Claude Code # JSONL file (format the mempalace normalizer already understands). -# 2. Stage exports under ~/.cache/mempalace-session/<wing>/. +# 2. Stage exports under <palace-root>/opencode-stage/<wing>/ (override with +# MEMPALACE_SESSION_STAGE). # 3. Run `mempalace mine --mode convos` against the staging dir. # # Dedup: mempalace convos mode keys on source_file (absolute staging path). -# The staging path is deterministic (per-wing under XDG_CACHE_HOME) so re-runs +# The staging path is deterministic (per-wing under the palace root) so re-runs # are idempotent as long as session content hasn't changed. # +# Staging location: because dedup keys on the staged path, wiping the stage +# leaves the palace holding drawers whose source files look deleted, and +# `mempalace sync` prunes exactly those when they fall inside the scope it is +# given. The stage therefore lives next to the palace it feeds (resolved via +# $MEMPALACE_PALACE_PATH → $MEMPAL_PALACE_PATH → ~/.mempalace/config.json → +# ~/.mempalace/palace), so stage and dedup keys share one lifetime and the +# dangerous state — palace survives, stage does not — cannot be reached by +# wiping something that merely looks disposable. It used to default under +# ~/.cache, which is disposable on exactly the hosts where this runs +# unattended. Override with MEMPALACE_SESSION_STAGE only if the target is at +# least as durable as the palace. +# # Session filter: sessions with fewer than --min-messages messages (default 3) # are skipped to avoid filing throwaway /exit'd sessions. # @@ -87,7 +100,7 @@ Idempotency: What gets mined: - Each qualifying session → one Claude Code JSONL file - - Staged under ~/.cache/mempalace-session/<wing>/ + - Staged under <palace-root>/opencode-stage/<wing>/ - Filed via `mempalace mine --mode convos` Transcript shape per session: @@ -103,7 +116,9 @@ Transcript shape per session: Dedup: - source_file = absolute staging path (deterministic per session ID) - Re-runs skip unchanged sessions. To force re-mining, delete the staging - dir: rm -rf ~/.cache/mempalace-session/<wing>/ + dir: rm -rf <palace-root>/opencode-stage/<wing>/ + (that only forces a refile; do NOT run `mempalace sync` while the stage is + missing, or the drawers mined from it get pruned instead) Rationale: Opencode lacks a session-stopping hook (upstream PRs #16598, #16769 still @@ -147,8 +162,31 @@ if ! [[ "$MIN_MESSAGES" =~ ^[0-9]+$ ]]; then fi # ── Staging dir ────────────────────────────────────────────────────── -# Deterministic per-wing path so source_file dedup works across re-runs. -CACHE_ROOT="${XDG_CACHE_HOME:-$HOME/.cache}/mempalace-session" +# Deterministic per-wing path so source_file dedup works across re-runs, and +# anchored to the palace root so the stage cannot be wiped independently of the +# dedup keys that reference it (see "Staging location" in the header). +# Mirrors mempalace config.py:palace_path() precedence, then takes the parent. +palace_root() { + python3 - <<'PY' 2>/dev/null || echo "$HOME/.mempalace" +import json, os +p = os.environ.get("MEMPALACE_PALACE_PATH") or os.environ.get("MEMPAL_PALACE_PATH") +if p: + p = os.path.abspath(os.path.expanduser(p)) +else: + cfg = os.path.expanduser("~/.mempalace/config.json") + p = None + if os.path.exists(cfg): + try: + with open(cfg) as fh: + v = json.load(fh).get("palace_path") + p = os.path.expanduser(v) if v else None + except Exception: + p = None + p = p or os.path.expanduser("~/.mempalace/palace") +print(os.path.dirname(p.rstrip("/"))) +PY +} +CACHE_ROOT="${MEMPALACE_SESSION_STAGE:-$(palace_root)/opencode-stage}" STAGE="$CACHE_ROOT/$WING" mkdir -p "$STAGE" @@ -185,7 +223,15 @@ if since: # isn't reachable (first install, moved, permission-denied), we fall through # to "everything is new" — the mine step will do the real dedup anyway. already_filed = set() -palace_path = os.environ.get("MEMPALACE_PATH", os.path.expanduser("~/.mempalace/palace")) +# Mirror mempalace's own resolution order (config.py): MEMPALACE_PALACE_PATH, +# then the legacy MEMPAL_PALACE_PATH, then the default. NOT "MEMPALACE_PATH" — +# that name is not a mempalace concept, and reading it silently degraded this +# NEW/SKIP preview to "everything is new" wherever some other tool had set it. +palace_path = ( + os.environ.get("MEMPALACE_PALACE_PATH") + or os.environ.get("MEMPAL_PALACE_PATH") + or os.path.expanduser("~/.mempalace/palace") +) chroma_db = Path(palace_path) / "chroma.sqlite3" if chroma_db.is_file(): try: diff --git a/contrib/README.md b/contrib/README.md index 0ab3c50..fd3fecb 100644 --- a/contrib/README.md +++ b/contrib/README.md @@ -2,6 +2,16 @@ Manual invocation of the session-mining wrappers is fine on a machine you actively drive. For long-running devboxes, a weekly automated mine keeps the palace fresh without thinking about it. This directory ships ready-to-use templates for two common scheduling mechanisms, for each wrapper. +> **pi machines: check whether you need this at all.** If the pi bridge +> extension (`extensions/pi/mempalace.ts`) is installed, it already feeds the +> palace by itself on `session_shutdown` and a debounced `agent_settled` — +> see [`extensions/pi/README.md` § Automatic transcript feeding](../extensions/pi/README.md#automatic-transcript-feeding). +> The templates below were written when scheduling was the *only* path for +> both harnesses; that's still true for **opencode** (no such extension +> exists), but for pi they're now a fallback — useful for a bare pi install +> without the bridge, a host-level catch-up job, or belt-and-braces coverage +> of a hard container kill (the extension's triggers don't fire on `SIGKILL`). + > **Before using either**: confirm the toolkit is installed and the wrapper works — > `mempalace-session --dry-run` (and/or `mempalace-pi-session --dry-run`) should list qualifying sessions. If that errors, fix the install before scheduling. @@ -171,17 +181,22 @@ rm /tmp/mempalace-session.cron crontab -l | grep mempalace ``` -Ensure `~/.cache/mempalace-session/` exists so the log file can be written: +Ensure `~/.cache/mempalace-logs/` exists so the log file can be written: + +> This is the **log** directory only. The staging dir — the transcripts the +> palace keys its `source_file` dedup on — lives beside the palace +> (`<palace-root>/opencode-stage/`), not in `~/.cache`, precisely so it cannot +> be cleaned away while the palace survives. Logs here are disposable. ```bash -mkdir -p ~/.cache/mempalace-session +mkdir -p ~/.cache/mempalace-logs ``` **Verify a run is happening:** ```bash # Tail the log the cron entry writes to -tail -f ~/.cache/mempalace-session/cron.log +tail -f ~/.cache/mempalace-logs/cron.log # Or force a run manually to prove the command is well-formed mempalace-session @@ -282,11 +297,11 @@ cat contrib/cron/mempalace-session-devbox.cron (crontab -l 2>/dev/null; cat contrib/cron/mempalace-session-devbox.cron) | crontab - # Ensure the log directory exists -mkdir -p ~/.cache/mempalace-session +mkdir -p ~/.cache/mempalace-logs # Verify crontab -l | grep mempalace-session-devbox -tail -f ~/.cache/mempalace-session/cron-devbox.log +tail -f ~/.cache/mempalace-logs/cron-devbox.log ``` **Uninstall:** diff --git a/contrib/cron/mempalace-session-devbox.cron b/contrib/cron/mempalace-session-devbox.cron index ba6eff9..479429c 100644 --- a/contrib/cron/mempalace-session-devbox.cron +++ b/contrib/cron/mempalace-session-devbox.cron @@ -35,4 +35,4 @@ CONTAINER_USER=developer PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin # m h dom mon dow command -0 3 * * 1 /bin/sh -c 'docker ps --filter "name=^/${CONTAINER}$" --filter "status=running" -q | grep -q . && docker exec -u "${CONTAINER_USER}" "${CONTAINER}" mempalace-session >> "$HOME/.cache/mempalace-session/cron-devbox.log" 2>&1' +0 3 * * 1 /bin/sh -c 'docker ps --filter "name=^/${CONTAINER}$" --filter "status=running" -q | grep -q . && docker exec -u "${CONTAINER_USER}" "${CONTAINER}" mempalace-session >> "$HOME/.cache/mempalace-logs/cron-devbox.log" 2>&1' diff --git a/contrib/cron/mempalace-session.cron b/contrib/cron/mempalace-session.cron index b47809d..4e8f62a 100644 --- a/contrib/cron/mempalace-session.cron +++ b/contrib/cron/mempalace-session.cron @@ -16,4 +16,4 @@ PATH=/home/USER/.local/bin:/usr/local/bin:/usr/bin:/bin # m h dom mon dow command -0 3 * * 1 mempalace-session >> /home/USER/.cache/mempalace-session/cron.log 2>&1 +0 3 * * 1 mempalace-session >> /home/USER/.cache/mempalace-logs/cron.log 2>&1 diff --git a/contrib/systemd/mempalace-session.service b/contrib/systemd/mempalace-session.service index 43c3275..4576826 100644 --- a/contrib/systemd/mempalace-session.service +++ b/contrib/systemd/mempalace-session.service @@ -8,7 +8,7 @@ ConditionPathExists=!%t/mempalace-session.lock [Service] Type=oneshot -# The wrapper writes to ~/.cache/mempalace-session/ and the palace. +# The wrapper writes to ~/.cache/mempalace-logs/ and the palace. # Keep stdout/stderr in the journal — inspect with: # journalctl --user -u mempalace-session --since today ExecStart=%h/.local/bin/mempalace-session diff --git a/docs/phase-1-exposure-runbook.md b/docs/phase-1-exposure-runbook.md new file mode 100644 index 0000000..309825d --- /dev/null +++ b/docs/phase-1-exposure-runbook.md @@ -0,0 +1,185 @@ +# Phase 1 exposure — newt on synlig, DNS, and the client auth model + +Companion to [`rfc-001-global-palace.md`](./rfc-001-global-palace.md) (design + decisions) and +[`synlig-primary-runbook.md`](./synlig-primary-runbook.md) (what is already installed on the primary). +This doc covers only the step the other two leave open: **making the primary reachable** — runbook §4 +items 2 and 5. + +**Status 2026-08-12 — Pangolin updated on nyvaken (done, yours). newt not yet installed on synlig. +Nothing exposed. No client `.env` flipped.** + +Read this before touching Pangolin: three of the four questions this step raises were **already decided** +in RFC §6.2 on 2026-08-09, and re-deciding them differently is how the fleet ends up in two states. + +--- + +## 1. The four questions, answered + +| Question | Answer | Where it was decided | +| --- | --- | --- | +| Which port? | **8765**, path **`/mcp`** (liveness: `/healthz`) | `cli.py:2141` default; runbook §2.4 | +| What does newt target? | **`172.17.0.1:8765`** (docker0), **never** `127.0.0.1` | RFC §6.2 Transport; runbook §2.4 | +| Open, or authenticated? | **Authenticated. The primary is never an open public resource.** | RFC §6.2 Network posture | +| Per-device credentials? | **No — Phase 1 ships the single shared bearer token.** Per-device tokens are Phase 4. | RFC §6.2 Authentication | + +### 1.1 Why not per-device users at the proxy + +The instinct — "create a Pangolin user per container, put the credentials in each `.env`, keep the +usernames distinct" — is the right *goal* (revocation, attribution) reached through the wrong *layer*, +twice over: + +1. **mempalace validates exactly one token.** `hmac.compare_digest(provided, f"Bearer {srv.auth_token}")` + (`mcp_server.py:5292-5295`) — there is no user table and no second credential. Per-device HTTP identity + is not a configuration you can express today; it is Phase 4 work (a server-side + `token → {device_id, scopes}` registry). RFC §6.2 chose the shared token for Phase 1 deliberately: + *"iterate more feature rich but more complex solutions over time."* + +2. **Pangolin's HTTP auth is browser-shaped; the clients are not.** SSO login, resource PIN and resource + password all assume something that can follow a redirect, render a form and hold a session cookie. + Every MemPalace client here is a headless JSON-RPC `POST` with an `Authorization` header — pi's + extension, opencode's `type:remote` MCP entry, and `mempalace-pi-session --mode remote`'s + `urllib.request.urlopen`. Point those at a user-authenticated resource and they receive a login page + where JSON should be. Enabling that protection breaks precisely the clients it is meant to protect. + +So: **Pangolin terminates TLS and nothing more** (RFC §6.2 Transport, decided 2026-08-09). The bearer token +is the authentication. This is not "unprotected" — an unauthenticated request to `/mcp` gets a 401 from +mempalace itself, verified A4/A5 in runbook §2.4. + +**Consequence to accept consciously** (RFC §6.2, §7.3.2): until Phase 4 the primary **cannot tell devices +apart**. `origin_device` is client-asserted and advisory — nothing load-bearing may depend on it, and +revoking one laptop means rotating the token everywhere. + +### 1.2 The one place per-device identity *does* exist today + +Remote mode is not only HTTP. `mempalace_mine` expands its source path in the **server** process, so a +client's staged transcripts must physically exist on the primary. The feeder therefore ships them over +SSH into a **per-device inbox** before asking the server to mine its own copy: + +```sh +rsync -a --update -e "$ssh_cmd" "$STAGE/" "${SSH_TARGET%/}/$DEVICE/" # bin/mempalace-pi-session:677-680 +``` + +That SSH key **is** per-device identity, and it is individually revocable (one line out of +`authorized_keys`) years before Phase 4 lands. It costs nothing extra, because the mining path needs SSH +regardless. + +Two implications people miss: + +- **`mempalace.jordbo.se` alone does not enable mining.** HTTPS covers the read/write tool surface + (`search`, `add_drawer`, `diary_write`, `kg_*`) — genuinely useful on its own, and the reason to do this + at all. But `--mode remote` also needs `MEMPALACE_PI_SSH_TARGET` reachable. Budget for both paths. +- **`DEVICE` defaults to `$(hostname)`** (`bin/mempalace-pi-session:163`). In a container that is the + container hostname: either random per recreate (inboxes proliferate; each recreate re-mines into a fresh + empty inbox) or identical across sibling devboxes (two containers writing one inbox). **Set + `MEMPALACE_PI_DEVICE` explicitly per container.** It is a label, not a secret, so put it somewhere + reviewable — a committed compose file — where duplicates are visible. That, not username hygiene in + `.env`, is the discipline this design actually asks of you. + +--- + +## 2. The bind trap, in full + +RFC §6.2 and runbook §2.4 already say **do not bind loopback behind the tunnel**, because +`enforce_host_pin = _http_is_loopback(host)` (`mcp_server.py:5367`) makes a loopback bind reject the +proxy's forwarded `Host:` with a **403** that reads exactly like a Pangolin misconfiguration. + +**Additional finding, 2026-08-12 — the same reflex also silently removes authentication.** Token +resolution in `cmd_serve` (`cli.py:1447-1450`) is: + +```python +loopback = _server_is_loopback(host) +if not token and not loopback and not args.allow_insecure: + token, token_created = _load_or_create_server_token(palace_path) +``` + +Auto-minting is gated on the bind being **non-loopback**. A loopback bind therefore starts with **no token +at all** — no error, no warning, `--allow-insecure` not required — because the server has concluded it is +only reachable locally, while the tunnel is serving it to the internet. Bind loopback behind newt and you +get a 403 wall *and*, the moment anything relaxes the Host pin, an unauthenticated palace. + +Both failure modes have the same cure, already implemented in +`contrib/systemd/mempalace-serve.service`: **bind `172.17.0.1`**. Non-loopback, so the Host pin relaxes and +the token is mandatory; docker0-only, so newt reaches it and the LAN does not. + +> Belt and braces: set `MEMPALACE_MCP_HTTP_TOKEN` explicitly in the unit rather than relying on +> auto-minting. Then no future bind change can quietly drop authentication. + +--- + +## 3. Steps + +Ordered so nothing is reachable before it is authenticated. + +### 3.1 Start the primary (runbook §4.3 — one `sudo`, unit already staged) + +```sh +sudo loginctl enable-linger ecsjper +cd ~/.config/systemd/user && mv mempalace-serve.service.staged mempalace-serve.service +systemctl --user daemon-reload && systemctl --user enable --now mempalace-serve + +curl -s 172.17.0.1:8765/healthz # expect ok +curl -s 127.0.0.1:8765/healthz # expect 403 — correct, not a bug (§2) +ss -ltnp | grep 8765 # expect 172.17.0.1:8765 only +``` + +### 3.2 Collect the shared token + +```sh +cat ~/.mempalace/server/f5d849287f6d73f0141b29d7/token +``` + +Directory name is `sha256(realpath(palace))[:24]` — it changes if the palace path ever moves. Store via the +`.env.age` flow, 0600 (RFC §6.2). + +### 3.3 newt on synlig + +synlig runs Docker (Gitea Actions runner + digikam) but **no tunnel client** — runbook §4.2. Pangolin on +nyvaken cannot dial in; synlig must dial out. Add a `newt` container with the credentials Pangolin issues +for a new site. + +Because newt runs in Docker on this box, the docker0 bind is already correct for it: from inside the +container the primary is `172.17.0.1:8765`. Verify from *inside* newt's network namespace, not from the +host, before touching DNS. + +> synlig has 7.8 GiB shared with a CI runner (runbook §1). newt is small, but do not colocate anything +> else here casually. + +### 3.4 DNS at the web hotel + +One CNAME: `mempalace` → **the same target your existing Pangolin resources use** (nyvaken's public +hostname). RFC §6.2 costed this as *"one DNS record per service on the web hotel is the whole setup cost."* + +⚠️ Not verified from here: nyvaken's public FQDN, and whether your web hotel permits a CNAME at that label +(some require an A record, or forbid CNAME where other records exist). Confirm before assuming a 5-minute job. + +### 3.5 Pangolin resource + +- Target: newt site → `172.17.0.1:8765`, path `/mcp` (plus `/healthz` if you want the external probe). +- **Auth: none at the Pangolin layer** (§1.1). TLS termination only. +- Do **not** attach an `Origin`-injecting proxy or browser client: a *present* non-loopback `Origin` is a + hard 403 with no override (runbook §2.4 B3). + +### 3.6 Verify end-to-end before flipping any client + +```sh +curl -s https://mempalace.jordbo.se/healthz # ok +curl -s -o /dev/null -w '%{http_code}\n' -X POST \ + https://mempalace.jordbo.se/mcp # 401 — the token is doing its job +curl -s -X POST https://mempalace.jordbo.se/mcp \ + -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \ + -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | head -c 300 # 36 tools +``` + +The 401 check matters as much as the 200: it is the only evidence that the thing you just published to the +internet is not open. Then, and only then, Phase 1 client flip — **one machine first** (RFC §8), and +remember opencode containers need the §4.1 sidecar merge before their `.env` takes effect. + +--- + +## 4. Still open + +- **Per-device tokens** — Phase 4. Until then `origin_device` is advisory (§1.1). +- **§7.6 diary dedup** must be settled *before* the first §4.4 join; replay duplicates every entry. +- **§7.2**: never run `mempalace sync` against the shared palace. Doubly true now that the pi/opencode + feeders stage *inside* the palace root, which puts staged sources in scope for a sync of the palace dir. +- **nyvaken's public FQDN and the web hotel's CNAME rules** — unverified (§3.4). diff --git a/docs/rfc-001-global-palace.md b/docs/rfc-001-global-palace.md index a593ebb..159698c 100644 --- a/docs/rfc-001-global-palace.md +++ b/docs/rfc-001-global-palace.md @@ -100,6 +100,16 @@ Two of the three pieces already exist. This is not greenfield. > `pi-devbox/.env.example:21` still advertises `mempalace-mcp --transport http --host 0.0.0.0 --port > 8765` as the way to serve a shared palace — replace with `mempalace serve --token … --tls-cert …`, > and change the example URL from `http://mempalace.lan:8765/mcp` to `https://`. +> +> **✅ Done 2026-08-12** — `pi-devbox/.env.example` and +> `mempalace-toolkit/extensions/pi/README.md` both now recommend `mempalace serve`, bind docker0 rather +> than `0.0.0.0`/loopback, use an `https://` example URL, and state that the transport *is* +> authenticated. `docker-compose.mempalace.yml` audited too, and it was worse than stale — it was +> **broken on 3.6.0 in both directions**: `--host 0.0.0.0` with no token in the environment makes the +> server refuse to start (crash-looping under `restart: unless-stopped`), and once a token *is* supplied +> the healthcheck's unauthenticated `tools/list` POST 401s, marking a healthy server unhealthy forever. +> Fixed: token now required via `${MEMPALACE_REMOTE_TOKEN:?}` (fails fast at `up`), healthcheck switched +> to the token-free `/healthz`. ### Two things that sound like the feature and are not @@ -682,7 +692,7 @@ join replays (§4.4). | Phase | Effort | Deliverable | | --- | --- | --- | -| **0 — hygiene** | hours | §7 runbook: converge the KG/entities store paths **on synlig before first `serve`** (§7.1 — **done 2026-08-10**, runbook §2.3), ban `sync` on shared palaces (§7.2), fix stale "unauthenticated" docs (incl. `pi-devbox/.env.example:21`). Added 2026-08-09: settle the **diary dedup** approach and file its upstream ask (§7.6), and **dry-run the join from one palace only** (§4.4). **No provenance work here** — it is not backfill-critical (§7.3.3) and belongs to the stamper, not the agent | +| **0 — hygiene** | hours | §7 runbook: converge the KG/entities store paths **on synlig before first `serve`** (§7.1 — **done 2026-08-10**, runbook §2.3), ban `sync` on shared palaces (§7.2), ~~fix stale "unauthenticated" docs (incl. `pi-devbox/.env.example:21`)~~ — **done 2026-08-12**, and `docker-compose.mempalace.yml` turned out to be outright broken on 3.6.0, now fixed. Added 2026-08-09: settle the **diary dedup** approach and file its upstream ask (§7.6), and **dry-run the join from one palace only** (§4.4). **No provenance work here** — it is not backfill-critical (§7.3.3) and belongs to the stamper, not the agent | | **1 — primary up** | hours, **no code** | `mempalace serve --token --tls-cert` on a private-net host (reuse `docker-compose.mempalace.yml` — keep it a **separate standalone project**, R4 — and mind port 8765 vs pi-studio; tor-ms22 already moved to 8766). Repoint pi clients via `MEMPALACE_REMOTE_URL`/`MEMPALACE_REMOTE_TOKEN`. **opencode clients can be repointed in the same breath** — remote MCP is supported and `generate-config.py` already emits it (§9.1, §9.6), subject to the sidecar caveat in §4.1. **Shared memory today, no offline.** **Decided 2026-08-09: primary = synlig, TLS at Pangolin, single shared token (§8.1)** — mind the loopback Host-pin trap in §6.2. | | **1.5 — opencode env propagation** | hours | Make the `mcp.mempalace` subtree env-authoritative in `generate-config.py`, gated by a generated-value fingerprint (§4.1). Independent of the rest of this RFC. Without it, adopting *or reverting* the opt-in on an existing opencode container needs a manual sidecar merge or a `docker volume rm` — which also blocks R6 reversibility | | **2 — `mempalace-edge`** | ~1 week | The actual ask: local-first writes + outbox flush + merged reads + per-wing policy. **Not** "fixes opencode" — opencode's *transport* is already fine after Phase 1; what edge adds there is offline/local-first, since `generate-config.py`'s switch is remote **or** local with no failover. **Ships with the §1.2 opt-in wiring (compose + `.env.example` + a third branch in the existing `generate-config.py`) and must pass the R1 acceptance test.** | diff --git a/docs/synlig-primary-runbook.md b/docs/synlig-primary-runbook.md index 89c743c..629933e 100644 --- a/docs/synlig-primary-runbook.md +++ b/docs/synlig-primary-runbook.md @@ -128,7 +128,13 @@ standing up a network-reachable service while you were asleep was not mine to de ## 4. Tomorrow, in order -1. **Pangolin update on nyvaken** (yours). +> **2026-08-12: items 1–2 and 5 now have their own runbook —** +> [`phase-1-exposure-runbook.md`](./phase-1-exposure-runbook.md). Pangolin on nyvaken is updated (done). +> newt is still missing on synlig. That doc also records why per-device Pangolin users are the wrong layer, +> and an additional loopback finding: a loopback bind does not merely 403, it also silently starts the +> server with **no token at all** (auto-minting is gated on the bind being non-loopback). + +1. **Pangolin update on nyvaken** (yours). ✅ done 2026-08-12. 2. **⚠️ synlig has no tunnel client.** `docker ps` shows only the Gitea runner and digikam — no `newt`. Pangolin on nyvaken cannot reach synlig by itself; synlig needs a newt container (or an equivalent outbound tunnel) added. Easy to miss because Pangolin will look healthy on its own side. If newt runs diff --git a/extensions/pi/README.md b/extensions/pi/README.md index 26ed4ec..92a24d8 100644 --- a/extensions/pi/README.md +++ b/extensions/pi/README.md @@ -16,6 +16,7 @@ dependencies (~300 MB). **Jump to:** - [What it does](#what-it-does) - [Transport: local vs external](#transport-local-vs-external) +- [Automatic transcript feeding](#automatic-transcript-feeding) - [The `Type.Unsafe` gotcha](#the-typeunsafe-gotcha) - [Deploying pi with mempalace on a new machine](#deploying-pi-with-mempalace-on-a-new-machine) - [Fail-soft, identity, debugging](#fail-soft) @@ -37,11 +38,75 @@ dependencies (~300 MB). injects the result as a `mempalace-wakeup` system message so the agent orients itself the way `~/.agents/skills/mempalace/SKILL.md` describes. Skipped on resume/fork (context is already in the thread). -4. **Manual wind-down** via a `/mempalace-diary [topic]` slash command: +4. **Automatic transcript feeding** (`session_shutdown`, and a debounced + `agent_settled`): stages + mines this pi installation's own session + transcripts into the palace with no user action needed. Unlike the diary + below, this needs no LLM turn — it's a subprocess + a tool call — so it + *can* run on `session_shutdown` where the diary cannot. See + [Automatic transcript feeding](#automatic-transcript-feeding). +5. **Manual wind-down** via a `/mempalace-diary [topic]` slash command: sends a prompt asking the LLM to call `mempalace_diary_write` with - an AAAK-formatted entry summarizing the session. Not fully auto - because pi sessions are typically short/tactical and - `session_shutdown` fires too late to drive another LLM turn. + an AAAK-formatted entry summarizing the session. This one stays manual + because it needs the LLM to compose the entry, and `session_shutdown` + fires too late to drive another LLM turn — a constraint that applies to + the diary specifically, not to feeding (see above). + +## Automatic transcript feeding + +The bridge feeds this pi installation's own session transcripts into the +palace by itself — no scheduler, no cron, no manual invocation. It fires on +`session_shutdown` (covers quit, `/new`, `/resume`, `/fork`) and on a +debounced `agent_settled` (covers a long session that later crashes, since a +hard kill runs no shutdown handler at all). + +The work is split across two processes, and the reason is a hard constraint, +not a style choice: **the palace is single-writer.** A live pi session +always holds it through this extension's own `mempalace-mcp` subprocess, so +an unattended `mempalace mine` from anywhere else fails outright with +`palace ... is held by PID <n>`. The bridge therefore: + +1. Runs `mempalace-pi-session --prepare --reason <trigger> --wing <wing>` as a + subprocess. This does every palace-free step — parse pi's JSONL, apply + the quality threshold, stage the export, and (remote mode only) `rsync` + it to the palace host — and prints one line, `MINE_SOURCE=<path>`, + without ever touching the palace. +2. Calls the `mempalace_mine` MCP tool **through this extension's own + client** on that path. Going through the client that already holds the + lock is the only way to write during a live session, and it automatically + targets whichever palace the bridge is pointed at — local stdio or a + shared remote one. + +`mempalace-pi-session` (in this repo's `bin/`) is the actual exporter and +owns the quality gate, the remote transport, and every flag — see its +`--help` for the full reference; this section only covers the extension's +side of the wiring. + +**Env knobs (extension side):** + +| Var | Default | Effect | +|---|---|---| +| `MEMPALACE_FEED` | `1` | Set `0` to disable automatic feeding entirely. | +| `MEMPALACE_FEED_BIN` | `mempalace-pi-session` | Helper to run. | +| `MEMPALACE_FEED_WING` | `wing_conversations` | Target wing — passed to both the exporter and the `mempalace_mine` call. | +| `MEMPALACE_FEED_DEBOUNCE_MS` | `600000` (10 min) | Minimum gap between mid-session (`agent_settled`) feeds. Bounds crash loss to one window instead of a whole session. | +| `MEMPALACE_FEED_PREPARE_TIMEOUT_MS` | `120000` | Kills a wedged `--prepare` subprocess. | +| `MEMPALACE_FEED_MINE_TIMEOUT_MS` | `30000` | Caps the `mempalace_mine` call so a stalled palace can't hang session exit. | + +**Remote palace:** if `$MEMPALACE_REMOTE_URL` is set (see +[Transport](#transport-local-vs-external)), `mempalace_mine`'s source path is +expanded on the *server*, which cannot see this machine's transcripts — +that's exactly why step 1 above rsyncs first in that mode. Configure the +inbox with `MEMPALACE_PI_SSH_TARGET` (required for remote feeding — feeding +is silently skipped without it), `MEMPALACE_PI_SSH_CONFIG`, and +`MEMPALACE_PI_REMOTE_PATH`; see `mempalace-pi-session --help`. + +**Concurrency:** overlapping triggers coalesce — a `session_shutdown` landing +while a debounced tick is still running joins that in-flight feed instead of +racing it. `mempalace-pi-session` itself also takes a non-blocking `flock`, +so even two independent invocations (e.g. this extension and the +container-start catch-up some devbox images run) never race each other; +losing that race is harmless because the next trigger re-exports from +scratch. ## Transport: local vs external @@ -57,11 +122,25 @@ chosen at load time: harnesses/containers (pi + opencode + native). `MEMPALACE_REMOTE_TOKEN`, if set, is sent as `Authorization: Bearer <token>`. - Serve such an endpoint with `mempalace-mcp --transport http --host 0.0.0.0 - --port 8765` (the `pi-devbox` / `opencode-devbox` repos ship a - `docker-compose.mempalace.yml` for exactly this). Note: that HTTP transport is - currently sessionless and **unauthenticated** — keep it on a trusted network - or behind a reverse proxy that enforces the bearer token. + Serve such an endpoint with `mempalace serve --host 172.17.0.1 --port 8765` + (the `pi-devbox` / `opencode-devbox` repos ship a + `docker-compose.mempalace.yml` for exactly this). + + **The HTTP transport is authenticated as of mempalace 3.6.0** — earlier docs + here said otherwise, from the v1.3.0 era. `serve` mints a bearer token, keeps + it 0600, passes it via the environment (never argv), compares it with + `hmac.compare_digest`, and **refuses to bind a non-loopback host without one** + unless `--allow-insecure`. It also pins `Host` and allowlists `Origin` + (anti-DNS-rebinding), and can terminate TLS itself. + + Two binds to avoid. `0.0.0.0` publishes the palace to the whole LAN. And + `127.0.0.1` is the trap that looks safe: the Host pin is enforced *only* on + loopback binds, so behind a tunnel every proxied request 403s — and + token auto-minting is gated on the bind being non-loopback, so it starts with + **no authentication at all**, no warning. Bind the docker0 gateway + (`172.17.0.1`): reachable from the host and its containers, not from the LAN. + See + [`docs/phase-1-exposure-runbook.md`](../../docs/phase-1-exposure-runbook.md). Implementation note: the HTTP client (`RemoteMcpClient`) is **vendored** from [`pi-extensions`](https://gitea.jordbo.se/joakimp/pi-extensions)' diff --git a/extensions/pi/mempalace.ts b/extensions/pi/mempalace.ts index 76dfa18..0fc939e 100644 --- a/extensions/pi/mempalace.ts +++ b/extensions/pi/mempalace.ts @@ -21,6 +21,21 @@ * `mempalace_status` + `mempalace_diary_read` output as context so the * agent orients itself the way the mempalace skill describes. Skipped * on resume/fork (palace context is already in the thread). + * - Feeding (auto): stage + mine this container's pi transcripts into the + * palace on `session_shutdown` and on a debounced `agent_settled`. Needs + * no LLM turn (pi transcripts are JSONL on disk), which is why it CAN be + * automatic where the diary cannot. The file-side work is delegated to + * `mempalace-pi-session --prepare` (export + threshold + staging, plus the + * rsync to the palace host when the palace is remote); the mine itself + * must run through THIS client, because the palace is single-writer and + * this process is the holder — a CLI `mempalace mine` during a live + * session dies with "palace ... is held by PID <ours>". Going through the + * client also means it automatically targets whichever palace this bridge + * is pointed at (local stdio or a shared remote one). + * - MEMPALACE_FEED=0 disable feeding entirely + * - MEMPALACE_FEED_BIN helper to run (default mempalace-pi-session) + * - MEMPALACE_FEED_WING target wing (default wing_conversations) + * - MEMPALACE_FEED_DEBOUNCE_MS min gap between mid-session feeds (default 600000) * - Wind-down (manual): `/mempalace-diary` command prompts the LLM to * write an AAAK-formatted diary entry. Not fully auto because pi * sessions are typically short/tactical and session_shutdown is too @@ -724,7 +739,112 @@ export default async function mempalaceExtension(pi: ExtensionAPI) { }); } + // --- Automatic transcript feeding --- + // + // Split deliberately: `mempalace-pi-session --prepare` does the palace-free + // file work (export + quality threshold + staging, plus the rsync to the + // palace host in remote mode) and prints the path to mine; we then mine it + // through this client. See the header note on single-writer contention. + const feedEnabled = (process.env.MEMPALACE_FEED ?? "1") !== "0"; + const feedBin = process.env.MEMPALACE_FEED_BIN || "mempalace-pi-session"; + const feedWing = process.env.MEMPALACE_FEED_WING ?? "wing_conversations"; + const feedDebounceMs = num(process.env.MEMPALACE_FEED_DEBOUNCE_MS, 600_000); + const feedPrepareTimeoutMs = num(process.env.MEMPALACE_FEED_PREPARE_TIMEOUT_MS, 120_000); + const feedMineTimeoutMs = num(process.env.MEMPALACE_FEED_MINE_TIMEOUT_MS, 30_000); + let lastFeedAt = 0; // 0 => the first settled turn also acts as a catch-up + let feedInFlight: Promise<void> | null = null; + + /** Run `mempalace-feed --prepare`; resolve the path to mine, or null. */ + function prepareFeed(reason: string): Promise<string | null> { + return new Promise((resolve) => { + // A missing helper surfaces as an async 'error' event (ENOENT), not a + // throw, so the handler below is the fail-soft path. + const child = spawn(feedBin, ["--prepare", "--reason", reason, "--wing", feedWing], { + stdio: ["ignore", "pipe", "pipe"], + }); + let out = ""; + let settled = false; + const finish = (value: string | null) => { + if (settled) return; + settled = true; + clearTimeout(timer); + resolve(value); + }; + const timer = setTimeout(() => { + try { + child.kill("SIGKILL"); + } catch { + /* already gone */ + } + finish(null); + }, feedPrepareTimeoutMs); + child.stdout.on("data", (chunk) => { + out += String(chunk); + }); + child.stderr.on("data", () => { + /* the script keeps its own log */ + }); + child.on("error", () => finish(null)); + child.on("exit", (code) => { + if (code !== 0) return finish(null); + const match = out.match(/^MINE_SOURCE=(.+)$/m); + finish(match ? match[1].trim() : null); + }); + }); + } + + /** + * Stage + mine this container's transcripts. Never throws, and coalesces: + * an overlapping trigger joins the in-flight run instead of racing it. + */ + function feedPalace(reason: string): Promise<void> { + if (!feedEnabled || !available) return Promise.resolve(); + if (feedInFlight) return feedInFlight; + const run = (async () => { + try { + const source = await prepareFeed(reason); + if (!source) return; + await Promise.race([ + client.callTool("mempalace_mine", { + source, + mode: "convos", + wing: feedWing, + agent: agentName, + }), + new Promise((_resolve, reject) => + setTimeout( + () => reject(new Error(`mine timed out after ${feedMineTimeoutMs}ms`)), + feedMineTimeoutMs, + ), + ), + ]); + lastFeedAt = Date.now(); + } catch (err) { + process.stderr.write( + `[mempalace ext] feed (${reason}) failed: ${(err as Error).message}\n`, + ); + } + })(); + feedInFlight = run.finally(() => { + feedInFlight = null; + }); + return feedInFlight; + } + + // Mid-session feed. A hard container kill runs no handler at all, so this is + // what bounds crash loss to one debounce window instead of a whole session. + // Re-mining a grown transcript purges and refiles that source_file, so + // repeated ticks refresh a session's drawers rather than duplicating them. + pi.on("agent_settled", async () => { + if (Date.now() - lastFeedAt < feedDebounceMs) return; + void feedPalace("tick"); // deliberately not awaited: never stall a turn + }); + pi.on("session_shutdown", async () => { + // Feed before stopping the client: we are the palace holder, so nothing + // else can mine while we live. pi awaits this handler, so the mine really + // does complete; feedMineTimeoutMs keeps a wedged palace from hanging exit. + await feedPalace("shutdown"); client.stop(); });