feeders: stage beside the palace, not in ~/.cache; document Phase 1 exposure
Staging default moves out of ~/.cache to <palace-root>/pi-stage (pi) and <palace-root>/opencode-stage (opencode), resolved with mempalace's own palace-path precedence ($MEMPALACE_PALACE_PATH -> $MEMPAL_PALACE_PATH -> ~/.mempalace/config.json -> ~/.mempalace/palace), then dirname. Why: the convos miner keys dedup on the *staged* path, so a wiped stage plus a sync scoped to include it prunes the drawers mined from those sources -- deleting memories, not a cache. Under ~/.cache that state was reachable by anything treating a cache as disposable. Staging inside the palace makes the coupling structural: the stage cannot be wiped without touching the palace itself. Overrides ($MEMPALACE_PI_STAGE / $MEMPALACE_SESSION_STAGE, --stage) are unchanged. Note the old default had never been created on any host, so this closed a latent hazard, not a live one. Measured, and the docs now claim only this much: sync prunes only within the scope it is given -- wing-only, 1299 scanned / 1299 out of scope / 0 removed; scoped at the palace root, 651 kept / 648 out of scope. The previous blanket "sync prunes every drawer" wording overstated it, which is a liability: the next reader disproves the overstatement and discards the real constraint with it. Also in this change: - cron log dir ~/.cache/mempalace-session -> ~/.cache/mempalace-logs. The stage left that namespace, so the old name now read as "the stage". - AGENTS.md: the convos miner *does* check mtime (verified against upstream convo_miner.py); the previous "no mtime check" claim was wrong. - smoke-test assertions use `mktemp -d` for --sessions-dir. One pointed at /tmp, which still held earlier synthetic transcripts, so a --dry-run exported a fake session into the real stage: --dry-run skips the mine, not the export. docs/phase-1-exposure-runbook.md -- the newt/DNS/auth step that RFC 001 and the synlig runbook leave open (runbook section 4, items 2 and 5). Port 8765 at /mcp, newt targets 172.17.0.1, and the authentication is the single shared bearer token (RFC 6.2, decided 2026-08-09) rather than per-device proxy users. The latter cannot work today: mempalace validates exactly one token, and Pangolin's SSO/PIN/password are browser-shaped while every client here is a headless JSON-RPC POST -- enabling that protection breaks the clients it protects. The per-device axis that *does* exist is the feeder's SSH key + per-device inbox. New finding recorded there: a loopback bind does not merely 403 behind a tunnel (already known, runbook 2.4) -- it also silently starts the server with no token at all, because auto-minting is gated on the bind being non-loopback. extensions/pi/README.md: the HTTP transport IS authenticated as of mempalace 3.6.0; the "sessionless and unauthenticated" note dated from the v1.3.0 era. Closes the RFC section 8 Phase-0 hygiene item.
This commit is contained in:
Joakim Persson
@@ -152,7 +152,15 @@ For `mempalace-docs`, test on a small repo (e.g. this one) first:
|
||||
- The companion skill lives at `~/.agents/skills/opencode-mempalace-bridge/SKILL.md` and is a **symlink into this repo**. Editing that file edits `SKILL.md` here. To propagate to Claude Code / Kiro, run `agents-sync` from [`cli_utils`](https://gitea.jordbo.se/joakimp/cli_utils).
|
||||
- The opencode DB path defaults to `~/.local/share/opencode/opencode.db`. Override via `$OPENCODE_DB` or `--db`.
|
||||
- The mempalace miner **skips symlinks** (as of v3.3.3 — `miner.py` line ~828). That's why the wrappers use `cp -p` / explicit file writes for staging, not symlinks.
|
||||
- The convos miner dedups on `source_file` path only (no mtime check). Staging filenames must be stable per session; deleting a staged JSONL forces a re-mine.
|
||||
- The convos miner dedups on `source_file` path **and** `mtime`
|
||||
(`file_already_mined(..., check_mtime=True)` in upstream `convo_miner.py`).
|
||||
A changed/grown transcript is detected, purged, and refiled — it is not
|
||||
silently skipped. What must still be stable across runs is the *staged
|
||||
path itself*: if a wrapper's staging dir is wiped, dedup has nothing to
|
||||
compare against and `mempalace sync` will treat the vanished sources as
|
||||
deleted and prune the drawers mined from them. (An earlier version of this
|
||||
file claimed "no mtime check" — that was wrong; verified against
|
||||
`convo_miner.py` 2026-08.)
|
||||
- The docs miner dedups on `source_file` path + `mtime`. That's why staging uses `cp -p` (preserves mtime).
|
||||
|
||||
## Colocated skill pattern
|
||||
|
||||
+84
-6
@@ -31,6 +31,13 @@ So on a machine using opencode + the "docs-first palace hygiene" policy, three g
|
||||
|
||||
The two wrappers in `bin/` close gaps **1** and **2**. Gap **3** is upstream work (see §6).
|
||||
|
||||
(Pi is a different story on gap 3: `bin/mempalace-pi-session` closes pi's
|
||||
version of gap 2 the same way `mempalace-session` closes opencode's, but pi's
|
||||
gap 3 is already closed **without** any upstream dependency — its bridge
|
||||
extension self-triggers the feed on `session_shutdown` and a debounced
|
||||
`agent_settled`. See §3's `mempalace-pi-session` subsection and
|
||||
[`extensions/pi/README.md`](extensions/pi/README.md).)
|
||||
|
||||
---
|
||||
|
||||
## 2. The architecture
|
||||
@@ -53,7 +60,7 @@ The two wrappers in `bin/` close gaps **1** and **2**. Gap **3** is upstream wor
|
||||
│ │ │ cache dir │
|
||||
└─────┬──────────┘ └────┬──────────────┘
|
||||
│ │
|
||||
│ ~/.cache/mempalace-docs/<wing>/ │ ~/.cache/mempalace-session/<wing>/
|
||||
│ ~/.cache/mempalace-docs/<wing>/ │ <palace-root>/opencode-stage/<wing>/
|
||||
│ │
|
||||
┌─────▼──────────┐ ┌────▼──────────────┐
|
||||
│ mempalace mine │ │ mempalace mine │
|
||||
@@ -120,7 +127,7 @@ What it drops: source code (`.py`, `.ts`, `.go`, `.rs`, …), lockfiles, `.git`,
|
||||
- `step-start` / `step-finish` → dropped as noise.
|
||||
- `reasoning` → kept, prefixed with `[reasoning]`.
|
||||
4. Serialize as Claude Code JSONL (`{"type": "user"|"assistant", "message": {"content": [...]}}`) — the one convos format the miner already understands.
|
||||
5. Stage at `~/.cache/mempalace-session/<wing>/<slug>_<id>.jsonl` with `mtime` = `session.time_updated` (deterministic, stable under dedup).
|
||||
5. Stage at `<palace-root>/opencode-stage/<wing>/<slug>_<id>.jsonl` with `mtime` = `session.time_updated` (deterministic, stable under dedup).
|
||||
|
||||
**Filters:**
|
||||
|
||||
@@ -130,6 +137,73 @@ What it drops: source code (`.py`, `.ts`, `.go`, `.rs`, …), lockfiles, `.git`,
|
||||
|
||||
**Then:** invokes `mempalace mine --mode convos` against the cache dir. A post-mine `mempalace repair` is **opt-in** via `--repair` — it is intentionally *not* the default because the in-place HNSW rebuild has corrupted live palaces on past runs. Never pass `--repair` from an unattended schedule.
|
||||
|
||||
### `bin/mempalace-pi-session` — pi coding-agent → palace bridge
|
||||
|
||||
**Input:** pi's own JSONL session transcripts under `~/.pi/agent/sessions/**/*.jsonl`
|
||||
(no SQLite export step needed — pi already writes files, unlike opencode).
|
||||
**Output:** palace drawers in `wing_conversations` (or `--wing` override), same
|
||||
Claude Code JSONL staging shape as `mempalace-session` above.
|
||||
|
||||
The transform pipeline is the same shape as `mempalace-session`'s (synthetic
|
||||
`[session: title | cwd | date | source: pi]` header, per-message dispatch,
|
||||
`toolCall`→`tool_use`, `toolResult`→`tool_result`, `mtime` copied onto the
|
||||
staged file for dedup stability) — see the script's own header comment for
|
||||
the exhaustive per-role mapping. What's architecturally different from the
|
||||
opencode wrapper is why this one has a **two-phase mode** and a **remote
|
||||
transport**, neither of which `mempalace-session` needs:
|
||||
|
||||
**Two phases, because the palace is single-writer.** `mempalace` enforces
|
||||
this with a per-palace `flock` (`palace.py`); the CLI's own error is explicit:
|
||||
`palace ... is held by PID <n> (mempalace-mcp); wait for it to finish`. Unlike
|
||||
opencode's session mine (which always runs *between* agent sessions, when
|
||||
nothing else holds the palace), pi's own bridge extension
|
||||
(`extensions/pi/mempalace.ts`) holds the palace open via its `mempalace-mcp`
|
||||
subprocess for the *entire* live session — and that extension is also what
|
||||
triggers the feed, on `session_shutdown` and a debounced `agent_settled`. An
|
||||
unattended `mempalace mine` invoked from anywhere else during that window
|
||||
would simply fail. So the wrapper splits:
|
||||
|
||||
- `--prepare` — export, threshold, stage (+ `rsync` in remote mode). Never
|
||||
opens the palace. Prints `MINE_SOURCE=<path>`.
|
||||
- (default, no `--prepare`) — the above, then mines it. If that mine hits
|
||||
contention ("is held by"), it's treated as **success, not failure** (exit
|
||||
0 with an informational message) — the holder's own extension will mine
|
||||
what got staged. This inverts the CLI's own convention (`MineAlreadyRunning`
|
||||
→ exit 1) deliberately, because at the wrapper layer the contention has a
|
||||
benign interpretation the raw CLI can't know about.
|
||||
|
||||
The pi extension calls `--prepare` as a subprocess, then feeds `MINE_SOURCE`
|
||||
to `mempalace_mine` through its own already-open MCP client — the only
|
||||
process that can write during a live session, because it *is* the lock
|
||||
holder. This is the mechanism behind [`extensions/pi/README.md` § Automatic
|
||||
transcript feeding](extensions/pi/README.md#automatic-transcript-feeding).
|
||||
|
||||
**Remote transport, because there is no remote-palace CLI.** `mempalace`'s
|
||||
`--backend` flag selects a *vector store* (chroma/qdrant/pgvector/milvus),
|
||||
not a remote palace — the only remote surface is the HTTP MCP server
|
||||
(`mempalace-mcp --transport http`, see [RFC-001](docs/rfc-001-global-palace.md)).
|
||||
And `mempalace_mine`'s `source` path is expanded **in that server process**,
|
||||
so a remote server has no way to see this machine's staged exports. `--mode
|
||||
remote` (auto-selected when `$MEMPALACE_REMOTE_URL` is set) therefore
|
||||
`rsync`s the stage to a per-device inbox on the palace host, then asks the
|
||||
server to mine its own local copy of that inbox over the same HTTP
|
||||
`tools/call` transport the extension uses. Requires
|
||||
`$MEMPALACE_PI_SSH_TARGET`; see `--help` for the rest
|
||||
(`MEMPALACE_PI_SSH_CONFIG`, `MEMPALACE_PI_REMOTE_PATH`, `MEMPALACE_PI_DEVICE`).
|
||||
|
||||
**Filters:** two gates, both required — stricter than `mempalace-session`'s
|
||||
single filter because pi's transcripts have a failure mode opencode's don't:
|
||||
|
||||
- `--min-messages N` (default 4) — same idea as opencode's filter, raised
|
||||
because pi's tool loops inflate turn counts fast.
|
||||
- `--min-assistant-chars N` (default 1000) — counts assistant *text* only,
|
||||
tool results excluded. Needed because pi expands skill/context text into
|
||||
the user prompt: an abandoned session can have a huge "user" turn and
|
||||
almost no assistant output (observed case: 13,380 injected-context user
|
||||
chars answered with 38 assistant chars), so a message-count-only filter
|
||||
would have filed 22 pure-noise drawers from that one session. Real
|
||||
sessions on the same corpus measured 15,900–100,000 assistant chars.
|
||||
|
||||
---
|
||||
|
||||
## 4. Setup recipe (new machine)
|
||||
@@ -168,7 +242,11 @@ The devbox uses two named Docker volumes so these persist across container recre
|
||||
- `devbox-palace` → `~/.mempalace/palace` (the palace itself)
|
||||
- `devbox-data` → `~/.local/share/opencode` (opencode's SQLite DB)
|
||||
|
||||
Code at `/workspace/mempalace-toolkit` is a bind mount from the host — survives container recreate and syncs via gitea. Staging directories (`~/.cache/mempalace-{docs,session}/`) are ephemeral but cheap to rebuild.
|
||||
Code at `/workspace/mempalace-toolkit` is a bind mount from the host — survives container recreate and syncs via gitea. The docs staging dir (`~/.cache/mempalace-docs/`) is ephemeral and cheap to rebuild.
|
||||
The **conversation** staging dirs are not: `<palace-root>/opencode-stage/` and
|
||||
`<palace-root>/pi-stage/` hold the exact paths the palace's `source_file` dedup keys on, so they
|
||||
live beside the palace deliberately and share its lifetime. Wiping one forces a full re-mine at
|
||||
best, and lets a scoped `mempalace sync` prune every drawer mined from it at worst.
|
||||
|
||||
**After container recreate**, just re-run `./install.sh` (idempotent) to relink `bin/` into the fresh `~/.local/bin/`.
|
||||
|
||||
@@ -304,7 +382,7 @@ Quick-start (cron):
|
||||
```bash
|
||||
sed "s|USER|$USER|g" contrib/cron/mempalace-session.cron \
|
||||
| (crontab -l 2>/dev/null; cat) | crontab -
|
||||
mkdir -p ~/.cache/mempalace-session
|
||||
mkdir -p ~/.cache/mempalace-logs
|
||||
```
|
||||
|
||||
#### Verification
|
||||
@@ -323,7 +401,7 @@ A healthy run produces one of:
|
||||
- **Incremental run**: zero to a few dozen new drawers (whatever grew since last run).
|
||||
- **Rerun with no new activity**: zero new drawers, only the repair step runs.
|
||||
|
||||
A run that files far more drawers than expected may indicate a staging-dir wipe (forcing a full re-mine) — check `~/.cache/mempalace-session/<wing>/` modification times.
|
||||
A run that files far more drawers than expected may indicate a staging-dir wipe (forcing a full re-mine) — check `<palace-root>/opencode-stage/<wing>/` modification times.
|
||||
|
||||
### Cost profile (reference)
|
||||
|
||||
@@ -355,7 +433,7 @@ These gaps should ideally close upstream, making the wrappers thinner or obsolet
|
||||
3. **Opencode harness in `hooks_cli.py`** — mempalace's hooks CLI only knows `claude-code` + `codex` today. Adding `opencode` would let the auto-save diary path work on opencode too. Pairs with #2 above.
|
||||
4. **SQLite mode for `mempalace mine --mode convos`** — if upstream ever adds direct SQLite ingest for opencode, `mempalace-session` loses its reason to exist (the export-to-JSONL dance goes away).
|
||||
|
||||
When #1 merges, retire `mempalace-docs` to a thin shim. When #2 + #3 land together, `mempalace-session` becomes a manual-only fallback (cron / backfill) while hooks handle live saves.
|
||||
When #1 merges, retire `mempalace-docs` to a thin shim. When #2 + #3 land together, `mempalace-session` becomes a manual-only fallback (cron / backfill) while hooks handle live saves. (`mempalace-pi-session` has no equivalent entry here: pi's bridge extension already self-triggers the feed with no upstream dependency — see §3.)
|
||||
|
||||
Separately tracked in [`docs/rfc-001-global-palace.md`](docs/rfc-001-global-palace.md): moving from one palace *per machine* to a **single primary palace with per-machine local fallback** (`mempalace-edge`). That RFC also records upstream items of its own — server-side `origin_device` provenance stamped from a per-device credential, per-wing ACLs, a `mempalace_kg_supersede` tool-classification fix, and a guard against running `mempalace sync` on a shared palace.
|
||||
|
||||
|
||||
@@ -5,6 +5,7 @@ Producer-side tooling for [MemPalace](https://github.com/MemPalace/mempalace)
|
||||
**What this repo contains:**
|
||||
|
||||
- `bin/mempalace-session` — exports [opencode](https://github.com/anomalyco/opencode) session history from its local SQLite DB to Claude Code JSONL, then mines it via `mempalace mine --mode convos`.
|
||||
- `bin/mempalace-pi-session` — the same idea for the [pi coding-agent](https://github.com/earendil-works/pi): exports its native JSONL session transcripts and mines them the same way. Unlike `mempalace-session`, this one is normally invoked *for you* — the pi bridge extension (below) runs it automatically on `session_shutdown` and a debounced `agent_settled`, so most pi machines never need the `contrib/` scheduling templates at all. See [`mempalace-pi-session`](#mempalace-pi-session) below and [`extensions/pi/README.md` § Automatic transcript feeding](extensions/pi/README.md#automatic-transcript-feeding).
|
||||
- `bin/mempalace-docs` — mines project directories into MemPalace while excluding source code, keeping the palace signal-dense.
|
||||
- [`ARCHITECTURE.md`](ARCHITECTURE.md) — **canonical spec**: architecture diagram, component details, setup recipe, operational notes, upstream-retirement roadmap.
|
||||
- [`SKILL.md`](SKILL.md) — the companion agent skill, symlinked into `~/.agents/skills/opencode-mempalace-bridge/` on install.
|
||||
@@ -323,7 +324,17 @@ A machine running only one of these has half a memory. Full treatment with pract
|
||||
|
||||
### Keeping it fresh (automation)
|
||||
|
||||
Manual invocation is fine while you're actively driving the machine, but long-running devboxes benefit from a weekly automated mine. [`contrib/`](contrib/) ships ready-to-install templates:
|
||||
**pi:** nothing to set up. The pi bridge extension
|
||||
(`extensions/pi/mempalace.ts`) feeds the palace by itself — see
|
||||
[`extensions/pi/README.md` § Automatic transcript feeding](extensions/pi/README.md#automatic-transcript-feeding).
|
||||
The templates below aren't needed unless you're running
|
||||
`mempalace-pi-session` somewhere without that extension (e.g. a bare pi
|
||||
install, or a host-side catch-up job).
|
||||
|
||||
**opencode**, and pi installs without the bridge: manual invocation is fine
|
||||
while you're actively driving the machine, but long-running devboxes benefit
|
||||
from a weekly automated mine. [`contrib/`](contrib/) ships ready-to-install
|
||||
templates:
|
||||
|
||||
- **systemd user timer** (recommended on Linux): survives reboots, catches missed runs, logs to `journalctl`.
|
||||
- **launchd user agent** (recommended on macOS): native-equivalent — logs to `~/Library/Logs/`, single-instance guarantees, `ProcessType=Background` throttling.
|
||||
@@ -421,12 +432,12 @@ mempalace-session --help
|
||||
- Tool outputs → `tool_result` blocks in a follow-up human message, folded back into the assistant turn by the mempalace normalizer.
|
||||
- `step-start` / `step-finish` parts are dropped as noise. `reasoning` parts are kept with a `[reasoning]` prefix.
|
||||
|
||||
**Dedup:** staging at `~/.cache/mempalace-session/<wing>/` with deterministic per-session filenames (`<slug>_<id>.jsonl`). The convos miner keys on `source_file`, so re-runs skip unchanged sessions. To force re-mining a session, delete its JSONL from the staging dir.
|
||||
**Dedup:** staging at `<palace-root>/opencode-stage/<wing>/` (override: `$MEMPALACE_SESSION_STAGE`) with deterministic per-session filenames (`<slug>_<id>.jsonl`). The convos miner keys on `source_file`, so re-runs skip unchanged sessions. To force re-mining a session, delete its JSONL from the staging dir.
|
||||
|
||||
**`--dry-run` is dedup-aware.** Each session is tagged `[NEW]` (would be filed) or `[SKIP]` (already in the palace), and the summary breaks down the count:
|
||||
|
||||
```
|
||||
Exported 62 session(s) to ~/.cache/mempalace-session/wing_conversations
|
||||
Exported 62 session(s) to /home/you/.mempalace/opencode-stage/wing_conversations
|
||||
0 new → will be filed on mine
|
||||
62 already filed → will be skipped (dedup by source_file)
|
||||
|
||||
@@ -441,6 +452,105 @@ If the palace is unreachable (fresh install, moved, permission-denied) the wrapp
|
||||
|
||||
---
|
||||
|
||||
## `mempalace-pi-session`
|
||||
|
||||
Pi coding-agent → MemPalace session bridge. Reads pi's own JSONL session
|
||||
transcripts under `~/.pi/agent/sessions/`, converts each qualifying session
|
||||
to Claude Code JSONL, and files via `mempalace mine --mode convos`. On most
|
||||
machines you never run this by hand — the pi bridge extension
|
||||
(`extensions/pi/mempalace.ts`) invokes it automatically; see
|
||||
[`extensions/pi/README.md` § Automatic transcript feeding](extensions/pi/README.md#automatic-transcript-feeding).
|
||||
Manual invocation is for a bare pi install without that extension, a
|
||||
host-side catch-up job, or just poking at the export by hand.
|
||||
|
||||
```bash
|
||||
mempalace-pi-session # export + mine everything qualifying
|
||||
mempalace-pi-session --wing my_convos # custom wing (default: wing_conversations)
|
||||
mempalace-pi-session --session <uuid-prefix> # one session only
|
||||
mempalace-pi-session --since 2026-04-01 # only sessions updated on/after date
|
||||
mempalace-pi-session --min-messages 6 # stricter turn-count filter (default: 4)
|
||||
mempalace-pi-session --min-assistant-chars 2000 # stricter "did anything happen" filter (default: 1000)
|
||||
mempalace-pi-session --dry-run # export + list, skip mine
|
||||
mempalace-pi-session --prepare # export + stage only; print MINE_SOURCE=<path>, never opens the palace
|
||||
mempalace-pi-session --mode remote # ship the stage to a remote palace host instead of mining locally
|
||||
mempalace-pi-session --help
|
||||
```
|
||||
|
||||
**Why `--prepare` exists:** the palace is single-writer. If a pi session for
|
||||
this same palace is currently open, its own bridge extension already holds
|
||||
the palace via `mempalace-mcp`, so an unattended `mempalace mine` from
|
||||
anywhere else fails with `palace ... is held by PID <n>`. `--prepare` does
|
||||
only the palace-free half (export, threshold, staging, and the `rsync` in
|
||||
remote mode) and hands the mine off to whoever already holds the lock —
|
||||
which is exactly what the pi extension does with its own MCP client. Run
|
||||
without `--prepare`, the tool does the mine itself, and treats that
|
||||
contention as success (exit 0, informational message) rather than failure —
|
||||
the holder will mine what got staged.
|
||||
|
||||
**What gets exported per session:**
|
||||
|
||||
- Synthetic header injected as the first user turn
|
||||
(`[session: <title> | <cwd> | <date> | source: pi]`) so the palace can find
|
||||
sessions by topic, not just by ID, and so pi/opencode/other results stay
|
||||
distinguishable in search.
|
||||
- User/assistant messages extracted from pi's JSONL `message` entries.
|
||||
- Assistant `toolCall` blocks → Claude Code `tool_use` blocks.
|
||||
- `toolResult` role messages → `tool_result` blocks, folded back into the
|
||||
assistant turn by the mempalace normalizer.
|
||||
- `bashExecution`, `custom` (display-only), `branchSummary`,
|
||||
`compactionSummary` → rendered as text annotations.
|
||||
- `thinking` blocks and image content → dropped (noise / not text).
|
||||
|
||||
**Filter — two gates, both required** (this is stricter than
|
||||
`mempalace-session`'s single message-count filter, and deliberately so —
|
||||
see below):
|
||||
|
||||
1. `--min-messages` user+assistant turns (default **4**).
|
||||
2. `--min-assistant-chars` characters of assistant *text*, tool results
|
||||
excluded (default **1000**).
|
||||
|
||||
The second gate exists because message count alone isn't enough for pi: pi
|
||||
expands skill/context text into the user prompt, so an abandoned session can
|
||||
have a huge "user" turn and almost nothing on the assistant side — e.g. a
|
||||
real observed case with a 13,380-char injected-context user turn answered
|
||||
"Ready. What would you like to work on?" (38 chars). Total size said
|
||||
substantial; assistant size correctly said nothing happened. Measured real
|
||||
sessions on the same machine ran 15,900–100,000 assistant chars, so the
|
||||
1000-char default sits with wide margin on both sides.
|
||||
|
||||
**Dedup:** staging under `$MEMPALACE_PI_STAGE/<wing>/` (default
|
||||
`<palace-root>/pi-stage/<wing>/` — beside the palace, so the stage cannot be
|
||||
wiped independently of the dedup keys pointing at it) with deterministic per-session-UUID
|
||||
filenames, and the export preserves the source session's `mtime` on the
|
||||
staged file. The convos miner is mtime-aware (see the Gotchas in
|
||||
[`AGENTS.md`](AGENTS.md) — an older version of this doc claimed otherwise),
|
||||
so re-runs on an unchanged session are a no-op, and re-feeding a **grown**
|
||||
session (a live one being fed mid-conversation) purges and refiles that
|
||||
session's drawers instead of duplicating them.
|
||||
|
||||
**Staging must persist.** Dedup keys on the *staged* path, not the original
|
||||
transcript, so if the stage is wiped, a `mempalace sync` scoped to include it
|
||||
sees those source files as gone and prunes the drawers mined from them —
|
||||
deleting the memories, not just the cache. This is why the stage now defaults to
|
||||
`<palace-root>/pi-stage`: it inherits whatever persistence the palace has, so
|
||||
the files and the dedup keys that reference them cannot be separated by
|
||||
wiping something that merely looks disposable. Override with
|
||||
`$MEMPALACE_PI_STAGE` only if you have somewhere *more* durable than the palace.
|
||||
|
||||
**Remote palace:** if `$MEMPALACE_REMOTE_URL` is set, there is no
|
||||
remote-palace CLI to mine into directly — `mempalace_mine` expands its
|
||||
source path in the *server* process, which cannot see this machine's staged
|
||||
exports. `--mode remote` (or `--mode auto`, which detects
|
||||
`$MEMPALACE_REMOTE_URL`) instead `rsync`s the stage to a per-device inbox on
|
||||
the palace host and asks the server to mine its own local copy. Requires
|
||||
`$MEMPALACE_PI_SSH_TARGET` (`user@host:path`); see `MEMPALACE_PI_SSH_CONFIG`,
|
||||
`MEMPALACE_PI_REMOTE_PATH`, and `MEMPALACE_PI_DEVICE` in `--help` for the
|
||||
rest. Deploying that primary — newt, DNS, and why the auth is a shared bearer
|
||||
token rather than per-device proxy users — is
|
||||
[`docs/phase-1-exposure-runbook.md`](docs/phase-1-exposure-runbook.md).
|
||||
|
||||
---
|
||||
|
||||
## Companion agent skill
|
||||
|
||||
Installing this repo symlinks `SKILL.md` into `~/.agents/skills/opencode-mempalace-bridge/SKILL.md`, where it's auto-discovered by opencode (and by Claude Code / Kiro if you run `agents-sync` from [`cli_utils`](https://gitea.jordbo.se/joakimp/cli_utils)).
|
||||
|
||||
@@ -20,6 +20,13 @@ The `mempalace` skill covers *using* the palace (search, diary, KG). This skill
|
||||
|
||||
Both follow the same **stage-to-cache-then-mine** idiom — they curate input into `~/.cache/…/<wing>/`, then delegate to `mempalace mine`.
|
||||
|
||||
**Pi is out of scope for this skill.** The pi coding-agent has its own
|
||||
wrapper, `mempalace-pi-session`, but — unlike opencode — pi's bridge
|
||||
extension (`extensions/pi/mempalace.ts`) invokes it automatically on session
|
||||
shutdown and a debounced mid-session tick, so a pi machine with that
|
||||
extension installed needs none of this skill's manual/scheduled recipe. See
|
||||
`extensions/pi/README.md` § Automatic transcript feeding in the same repo.
|
||||
|
||||
## When to Load This Skill
|
||||
|
||||
- User asks "how does the palace get fed?" or mentions setting up mempalace on a new machine.
|
||||
@@ -86,7 +93,14 @@ A docs-heavy repo should produce ~5–10 drawers per file. >15 drawers/file on a
|
||||
### Dedup is free — re-running is safe
|
||||
|
||||
- `mempalace-docs`: dedup keyed on `source_file` path + `mtime`. Unchanged files skipped.
|
||||
- `mempalace-session`: dedup keyed on `source_file` path alone (no mtime check for convos). Staging filenames are deterministic per session (`<slug>_<id>.jsonl`), so re-runs skip already-filed sessions.
|
||||
- `mempalace-session` / `mempalace-pi-session`: the convos miner also dedups
|
||||
on `source_file` path + `mtime` (`file_already_mined(..., check_mtime=True)`
|
||||
in upstream `convo_miner.py` — a prior version of this line said "no mtime
|
||||
check", which was wrong). Staging filenames are deterministic per session,
|
||||
so a re-run on unchanged content is a no-op, and a grown/changed session is
|
||||
detected, purged, and refiled rather than duplicated. What must stay
|
||||
stable is the staging *path itself* — wiping the staging dir makes dedup
|
||||
(and `mempalace sync`) treat those sources as gone.
|
||||
|
||||
Second run immediately after first → 0 new drawers, only the post-mine `repair` step runs (~5 min on 5k drawers).
|
||||
|
||||
@@ -110,8 +124,10 @@ mempalace-session --session ses_abc123 # one specific session
|
||||
### Force re-mine
|
||||
|
||||
```bash
|
||||
rm -rf ~/.cache/mempalace-session/<wing>/ # nukes staging dir
|
||||
rm -rf <palace-root>/opencode-stage/<wing>/ # nukes staging dir
|
||||
mempalace-session # stages + mines fresh
|
||||
# Do NOT run `mempalace sync` between those two commands: with the stage gone,
|
||||
# a scoped sync prunes the drawers mined from it instead of refiling them.
|
||||
```
|
||||
|
||||
Staging is ephemeral by design; the palace is the source of truth.
|
||||
|
||||
+308
-28
@@ -10,22 +10,88 @@
|
||||
# Strategy (mirrors mempalace-session):
|
||||
# 1. Walk ~/.pi/agent/sessions/**/*.jsonl and export each qualifying session
|
||||
# to a Claude Code JSONL file (format the mempalace normalizer speaks).
|
||||
# 2. Stage exports under ~/.cache/mempalace-pi-session/<wing>/.
|
||||
# 2. Stage exports under $MEMPALACE_PI_STAGE/<wing> (default
|
||||
# <palace-root>/pi-stage/<wing> — alongside the palace it feeds).
|
||||
# 3. Run `mempalace mine --mode convos` against the staging dir.
|
||||
#
|
||||
# TWO PHASES (--prepare), because the palace is single-writer
|
||||
# mempalace refuses a CLI mine while another process holds the palace:
|
||||
# "palace ... is held by PID <n> (mempalace-mcp); wait for it to finish"
|
||||
# A live pi session ALWAYS has a holder — the mempalace extension's own
|
||||
# mempalace-mcp. So an unattended CLI mine only works when no session is
|
||||
# live (e.g. a container-start catch-up); during a session the mine must be
|
||||
# performed by the process that already holds the palace. Hence:
|
||||
#
|
||||
# --prepare export + stage (+ rsync in remote mode) and print
|
||||
# MINE_SOURCE=<path>, without ever opening the palace.
|
||||
# (default) the above, then mine it ourselves. Contention is treated as
|
||||
# success-with-nothing-to-do, not failure: the holder's own
|
||||
# extension will mine what we staged.
|
||||
#
|
||||
# The pi mempalace extension drives exactly this: it runs --prepare on
|
||||
# session_shutdown and on a debounced agent_settled, then calls
|
||||
# mempalace_mine on MINE_SOURCE through its existing MCP client.
|
||||
#
|
||||
# TRANSPORTS (--mode, default auto)
|
||||
# local Mine into the local palace with the mempalace CLI.
|
||||
# remote $MEMPALACE_REMOTE_URL is set, so the palace lives on another host.
|
||||
# There is no remote-palace CLI — only the HTTP MCP server — and
|
||||
# mempalace_mine expands its source path in the SERVER process, so
|
||||
# the server cannot see this machine's staged exports. We therefore
|
||||
# rsync the stage into a per-device inbox on the palace host and ask
|
||||
# the server to mine its own local path. Requires
|
||||
# MEMPALACE_PI_SSH_TARGET (where to rsync) and
|
||||
# MEMPALACE_PI_REMOTE_PATH (what that inbox is called server-side).
|
||||
#
|
||||
# Labelling: every exported transcript begins with a synthetic header
|
||||
# [session: <title> | <cwd> | <YYYY-MM-DD> | source: pi]
|
||||
# so post-mine search results are self-identifying (pi vs opencode vs other).
|
||||
#
|
||||
# Dedup: mempalace convos mode keys on source_file (absolute staging path).
|
||||
# Staging paths are deterministic per pi session UUID, so re-runs are
|
||||
# idempotent until session content actually changes.
|
||||
# Staging paths are deterministic per pi session UUID, and the export copies
|
||||
# the source session's mtime onto the staged file, so re-runs are idempotent
|
||||
# until session content actually changes. A GROWN session is purged and
|
||||
# refiled for that source_file by the miner, so re-feeding a live session
|
||||
# refreshes its drawers instead of duplicating them.
|
||||
#
|
||||
# Session filter: sessions with fewer than --min-messages *user+assistant*
|
||||
# messages (default 3) are skipped to avoid filing single-prompt throwaways.
|
||||
# Staging location: source_file dedup keys on the staged path, so if the stage
|
||||
# is wiped the palace is left with drawers whose source files look deleted.
|
||||
# `mempalace sync` prunes exactly those — but only within the scope it is
|
||||
# given. Measured on this layout: scoped at the palace root the staged sources
|
||||
# are in scope (kept 651), while a wing-only sync reports them out_of_scope and
|
||||
# leaves them alone. So the data loss is conditional on how sync is invoked,
|
||||
# which is far too thin a margin to rely on.
|
||||
#
|
||||
# The stage therefore defaults NEXT TO THE PALACE (<palace-root>/pi-stage,
|
||||
# resolved the way mempalace itself resolves the palace: $MEMPALACE_PALACE_PATH
|
||||
# → $MEMPAL_PALACE_PATH → ~/.mempalace/config.json → ~/.mempalace/palace).
|
||||
#
|
||||
# That makes the invariant structural rather than documented: the stage and the
|
||||
# dedup keys that reference it share one lifetime, so the dangerous state —
|
||||
# palace survives, stage does not — can no longer be reached by wiping
|
||||
# something that merely looks disposable. A cache dir (the obvious choice, and
|
||||
# the old default) is exactly wrong here: it persists just long enough to look
|
||||
# correct, then takes the memories with it. Override with MEMPALACE_PI_STAGE
|
||||
# only if the target is at least as durable as the palace.
|
||||
#
|
||||
# In remote mode the local stage is only a shipping buffer — dedup lives on the
|
||||
# server, keyed by the server-side inbox path — so its durability is moot there.
|
||||
#
|
||||
# Session filter: two gates, both required.
|
||||
# 1. --min-messages <N> user+assistant turns (default 4). Tool loops inflate
|
||||
# assistant turns fast in pi, so a real working session clears this
|
||||
# easily; a single abandoned prompt does not.
|
||||
# 2. --min-assistant-chars <N> characters of assistant *text* (default 1000),
|
||||
# excluding tool results. Assistant volume, not total volume: pi expands
|
||||
# skills/context into the user prompt, so an abandoned session can carry a
|
||||
# 13k-char "user" message answered with "Ready. What would you like to
|
||||
# work on?" — total size says substantial, assistant size correctly says
|
||||
# nothing happened.
|
||||
#
|
||||
# Usage:
|
||||
# mempalace-pi-session
|
||||
# mempalace-pi-session --prepare
|
||||
# mempalace-pi-session --mode remote
|
||||
# mempalace-pi-session --wing <name>
|
||||
# mempalace-pi-session --session <uuid-prefix>
|
||||
# mempalace-pi-session --since 2026-04-01
|
||||
@@ -34,13 +100,16 @@
|
||||
# mempalace-pi-session --help
|
||||
#
|
||||
# Exit codes:
|
||||
# 0 success
|
||||
# 0 success (including "nothing qualified", "another run holds the lock",
|
||||
# and "palace held by a live session")
|
||||
# 1 usage / argument error
|
||||
# 2 pi sessions dir missing
|
||||
# 3 mempalace CLI not installed
|
||||
# 3 mempalace CLI not installed / rsync missing in remote mode
|
||||
# 4 mine failed
|
||||
# 5 remote transport failed (rsync or HTTP tools/call)
|
||||
#
|
||||
# Dependencies: bash, python3 (stdlib only), mempalace (v3.3.3+)
|
||||
# Dependencies: bash, python3 (stdlib only), mempalace (v3.3.3+);
|
||||
# rsync + ssh in remote mode.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
@@ -49,11 +118,50 @@ AGENT="${USER:-mempalace}"
|
||||
WING="wing_conversations"
|
||||
SESSION_ID=""
|
||||
SINCE=""
|
||||
MIN_MESSAGES=3
|
||||
MIN_MESSAGES=4
|
||||
MIN_ASSISTANT_CHARS=1000
|
||||
DRY_RUN=0
|
||||
DO_REPAIR=0
|
||||
PREPARE_ONLY=0
|
||||
MODE="auto"
|
||||
REASON=""
|
||||
PI_SESSIONS_DIR="${PI_SESSIONS_DIR:-$HOME/.pi/agent/sessions}"
|
||||
|
||||
# Resolve the palace ROOT (the dir holding palace/, knowledge_graph.sqlite3,
|
||||
# config.json) using mempalace's own precedence, so the stage lands next to
|
||||
# whichever palace this host actually feeds. Mirrors config.py:palace_path()
|
||||
# (env → config.json → default) and takes the parent. Only evaluated when
|
||||
# MEMPALACE_PI_STAGE is unset, so the common path costs nothing.
|
||||
palace_root() {
|
||||
python3 - <<'PY' 2>/dev/null || echo "$HOME/.mempalace"
|
||||
import json, os
|
||||
p = os.environ.get("MEMPALACE_PALACE_PATH") or os.environ.get("MEMPAL_PALACE_PATH")
|
||||
if p:
|
||||
p = os.path.abspath(os.path.expanduser(p))
|
||||
else:
|
||||
cfg = os.path.expanduser("~/.mempalace/config.json")
|
||||
p = None
|
||||
if os.path.exists(cfg):
|
||||
try:
|
||||
with open(cfg) as fh:
|
||||
v = json.load(fh).get("palace_path")
|
||||
p = os.path.expanduser(v) if v else None
|
||||
except Exception:
|
||||
p = None
|
||||
p = p or os.path.expanduser("~/.mempalace/palace")
|
||||
print(os.path.dirname(p.rstrip("/")))
|
||||
PY
|
||||
}
|
||||
STAGE_ROOT="${MEMPALACE_PI_STAGE:-$(palace_root)/pi-stage}"
|
||||
|
||||
# Remote transport (see TRANSPORTS in the header)
|
||||
REMOTE_URL="${MEMPALACE_REMOTE_URL:-}"
|
||||
REMOTE_TOKEN="${MEMPALACE_REMOTE_TOKEN:-}"
|
||||
SSH_TARGET="${MEMPALACE_PI_SSH_TARGET:-}"
|
||||
SSH_CONFIG="${MEMPALACE_PI_SSH_CONFIG:-}"
|
||||
REMOTE_PATH="${MEMPALACE_PI_REMOTE_PATH:-/data/feed}"
|
||||
DEVICE="${MEMPALACE_PI_DEVICE:-$(hostname)}"
|
||||
|
||||
# ── Usage ────────────────────────────────────────────────────────────
|
||||
usage() {
|
||||
cat <<'EOF'
|
||||
@@ -67,10 +175,28 @@ Options:
|
||||
--session <prefix> Export one session only (match on UUID prefix)
|
||||
--since <YYYY-MM-DD> Only sessions last modified on/after this date
|
||||
--min-messages <N> Skip sessions with fewer than N user+assistant
|
||||
turns (default: 3)
|
||||
turns (default: 4)
|
||||
--min-assistant-chars <N>
|
||||
Skip sessions with fewer than N characters of
|
||||
assistant text, tool results excluded (default: 1000).
|
||||
Catches abandoned sessions whose bulk is injected
|
||||
skill/context text in the user prompt.
|
||||
--agent <name> Agent name recorded on drawers (default: $USER)
|
||||
--sessions-dir <path> Path to pi sessions dir (default: $PI_SESSIONS_DIR
|
||||
or ~/.pi/agent/sessions)
|
||||
--stage <path> Staging root (default: $MEMPALACE_PI_STAGE, else
|
||||
<palace-root>/pi-stage — next to the palace, so the
|
||||
stage cannot be wiped independently of the dedup keys
|
||||
that point at it). Exports go in <root>/<wing>.
|
||||
See "Staging location" in the header before moving it.
|
||||
--mode <m> auto|local|remote (default: auto — remote when
|
||||
$MEMPALACE_REMOTE_URL is set)
|
||||
--prepare Export + stage (+ rsync in remote mode), print
|
||||
MINE_SOURCE=<path>, and stop without opening the
|
||||
palace. For callers that will do the mine themselves
|
||||
through a live MCP connection.
|
||||
--reason <label> Label this run in its output (e.g. shutdown, tick,
|
||||
container-start). Useful when triggers log to a file.
|
||||
--dry-run Export + list; do not mine into palace. Each session
|
||||
is tagged [NEW] or [SKIP] based on whether its
|
||||
source_file is already in the palace.
|
||||
@@ -102,14 +228,27 @@ Transcript shape per session:
|
||||
|
||||
Dedup:
|
||||
- source_file = absolute staging path (deterministic per pi session UUID)
|
||||
- Re-runs skip unchanged sessions. To force re-mining, delete the staging
|
||||
dir: rm -rf ~/.cache/mempalace-pi-session/<wing>/
|
||||
- Re-runs skip unchanged sessions; a GROWN session (mtime changed) has its
|
||||
old drawers purged and is refiled, so re-feeding a live session refreshes
|
||||
rather than duplicates.
|
||||
- To force re-mining, delete the staging dir:
|
||||
rm -rf <palace-root>/pi-stage/<wing>/
|
||||
That forces a refile — but do NOT run `mempalace sync` while the stage is
|
||||
missing, or the drawers mined from it get pruned instead.
|
||||
|
||||
Rationale:
|
||||
Pi's extension ecosystem could, in principle, stream sessions into the
|
||||
palace as they happen, but that requires a resident MCP connection during
|
||||
every pi session. This wrapper is the batch, harness-agnostic alternative:
|
||||
it reads the durable on-disk JSONL and mines it on a schedule.
|
||||
Two complementary paths feed the palace from pi, and they cover different
|
||||
failure modes:
|
||||
- The pi mempalace bridge extension (extensions/pi/mempalace.ts) drives
|
||||
this script with --prepare on session_shutdown and on a debounced
|
||||
agent_settled, then mines through its own live MCP connection. That is
|
||||
the primary path: it needs no scheduling and it is the only way to write
|
||||
while a session holds the palace.
|
||||
- Running this script directly is the batch/recovery path: a
|
||||
container-start or host-level catch-up that picks up transcripts nothing
|
||||
mined at the time — notably after a SIGKILL, where no pi handler runs at
|
||||
all. It reads the durable on-disk JSONL, so it does not care whether the
|
||||
session that produced it exited cleanly.
|
||||
EOF
|
||||
}
|
||||
|
||||
@@ -121,6 +260,11 @@ while [[ $# -gt 0 ]]; do
|
||||
--session) SESSION_ID="${2:-}"; shift 2 ;;
|
||||
--since) SINCE="${2:-}"; shift 2 ;;
|
||||
--min-messages) MIN_MESSAGES="${2:-}"; shift 2 ;;
|
||||
--min-assistant-chars) MIN_ASSISTANT_CHARS="${2:-}"; shift 2 ;;
|
||||
--stage) STAGE_ROOT="${2:-}"; shift 2 ;;
|
||||
--mode) MODE="${2:-}"; shift 2 ;;
|
||||
--prepare) PREPARE_ONLY=1; shift ;;
|
||||
--reason) REASON="${2:-}"; shift 2 ;;
|
||||
--agent) AGENT="${2:-}"; shift 2 ;;
|
||||
--sessions-dir) PI_SESSIONS_DIR="${2:-}"; shift 2 ;;
|
||||
--dry-run) DRY_RUN=1; shift ;;
|
||||
@@ -138,32 +282,67 @@ if [[ ! -d "$PI_SESSIONS_DIR" ]]; then
|
||||
echo " override with --sessions-dir <path> or PI_SESSIONS_DIR env var" >&2
|
||||
exit 2
|
||||
fi
|
||||
if ! command -v mempalace >/dev/null 2>&1; then
|
||||
case "$MODE" in
|
||||
auto) if [[ -n "$REMOTE_URL" ]]; then MODE="remote"; else MODE="local"; fi ;;
|
||||
local|remote) ;;
|
||||
*) echo "error: --mode must be auto|local|remote" >&2; exit 1 ;;
|
||||
esac
|
||||
# The mempalace CLI is only needed when WE do the mine. --prepare never opens
|
||||
# the palace, and remote mode talks to the server over HTTP.
|
||||
if [[ $PREPARE_ONLY -eq 0 && "$MODE" == "local" ]] && ! command -v mempalace >/dev/null 2>&1; then
|
||||
echo "error: mempalace CLI not found in PATH" >&2
|
||||
exit 3
|
||||
fi
|
||||
if ! [[ "$MIN_MESSAGES" =~ ^[0-9]+$ ]]; then
|
||||
echo "error: --min-messages must be an integer" >&2
|
||||
exit 1
|
||||
if [[ "$MODE" == "remote" ]]; then
|
||||
command -v rsync >/dev/null 2>&1 || { echo "error: rsync not found (needed for --mode remote)" >&2; exit 3; }
|
||||
if [[ -z "$SSH_TARGET" ]]; then
|
||||
echo "error: MEMPALACE_PI_SSH_TARGET unset (needed for --mode remote)" >&2
|
||||
exit 1
|
||||
fi
|
||||
# The devbox generates a dedicated LAN-jump key/config; prefer it if present.
|
||||
if [[ -z "$SSH_CONFIG" && -f "$HOME/.ssh-local/config" ]]; then
|
||||
SSH_CONFIG="$HOME/.ssh-local/config"
|
||||
fi
|
||||
fi
|
||||
for _n in MIN_MESSAGES MIN_ASSISTANT_CHARS; do
|
||||
if ! [[ "${!_n}" =~ ^[0-9]+$ ]]; then
|
||||
_flag="--$(printf '%s' "${_n,,}" | tr '_' '-')"
|
||||
echo "error: $_flag must be an integer" >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
# ── Staging dir ──────────────────────────────────────────────────────
|
||||
CACHE_ROOT="${XDG_CACHE_HOME:-$HOME/.cache}/mempalace-pi-session"
|
||||
STAGE="$CACHE_ROOT/$WING"
|
||||
# Deterministic per-wing path so source_file dedup works across re-runs. See
|
||||
# "Staging location" in the header for why this should not be disposable.
|
||||
STAGE="${STAGE_ROOT%/}/$WING"
|
||||
mkdir -p "$STAGE"
|
||||
|
||||
[[ -n "$REASON" ]] && echo "mempalace-pi-session [$REASON] mode=$MODE stage=$STAGE"
|
||||
|
||||
# ── Single-writer guard ──────────────────────────────────────────────
|
||||
# Non-blocking: overlapping triggers (a session_shutdown landing on top of a
|
||||
# debounced mid-session run) must not queue or race. Losing a run is harmless
|
||||
# — the next one re-exports from scratch.
|
||||
exec 9>"${STAGE_ROOT%/}/.lock"
|
||||
if command -v flock >/dev/null 2>&1 && ! flock -n 9; then
|
||||
echo "another mempalace-pi-session run holds the lock; skipping"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ── Export sessions (Python heredoc) ────────────────────────────────
|
||||
# Parses pi JSONL files and writes Claude Code JSONL per session into $STAGE.
|
||||
# Also classifies each export as NEW/ALREADY FILED (by source_file lookup)
|
||||
# so --dry-run reports the real mine-set size. Classification is advisory;
|
||||
# `mempalace mine --mode convos` is still the authoritative dedup.
|
||||
export_count=$(python3 - "$PI_SESSIONS_DIR" "$STAGE" "$SESSION_ID" "$SINCE" "$MIN_MESSAGES" <<'PY'
|
||||
export_count=$(python3 - "$PI_SESSIONS_DIR" "$STAGE" "$SESSION_ID" "$SINCE" "$MIN_MESSAGES" "$MIN_ASSISTANT_CHARS" <<'PY'
|
||||
import json, os, sqlite3, sys
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
sessions_dir, stage, session_filter, since, min_messages = sys.argv[1:6]
|
||||
sessions_dir, stage, session_filter, since, min_messages, min_assistant_chars = sys.argv[1:7]
|
||||
min_messages = int(min_messages)
|
||||
min_assistant_chars = int(min_assistant_chars)
|
||||
stage = Path(stage)
|
||||
sessions_dir = Path(sessions_dir)
|
||||
|
||||
@@ -178,7 +357,15 @@ if since:
|
||||
|
||||
# ── Load palace's already-filed source_files (best-effort, read-only) ──
|
||||
already_filed = set()
|
||||
palace_path = os.environ.get("MEMPALACE_PATH", os.path.expanduser("~/.mempalace/palace"))
|
||||
# Mirror mempalace's own resolution order (config.py): MEMPALACE_PALACE_PATH,
|
||||
# then the legacy MEMPAL_PALACE_PATH, then the default. NOT "MEMPALACE_PATH" —
|
||||
# that name is not a mempalace concept, and reading it silently degraded this
|
||||
# NEW/SKIP preview to "everything is new" wherever some other tool had set it.
|
||||
palace_path = (
|
||||
os.environ.get("MEMPALACE_PALACE_PATH")
|
||||
or os.environ.get("MEMPAL_PALACE_PATH")
|
||||
or os.path.expanduser("~/.mempalace/palace")
|
||||
)
|
||||
chroma_db = Path(palace_path) / "chroma.sqlite3"
|
||||
if chroma_db.is_file():
|
||||
try:
|
||||
@@ -282,6 +469,7 @@ if session_filter:
|
||||
|
||||
exported = 0
|
||||
skipped_short = 0
|
||||
skipped_quiet = 0
|
||||
skipped_malformed = 0
|
||||
skipped_already_filed = 0
|
||||
|
||||
@@ -318,6 +506,7 @@ for path in paths:
|
||||
continue
|
||||
|
||||
title = derive_title(entries, fallback=session_uuid[:8])
|
||||
assistant_chars = 0
|
||||
out_lines = []
|
||||
out_lines.append({
|
||||
"type": "user",
|
||||
@@ -337,6 +526,7 @@ for path in paths:
|
||||
out_lines.append({"type": "user", "message": {"content": text}})
|
||||
elif role == "assistant":
|
||||
text, tool_uses, _ = extract_text(msg.get("content"))
|
||||
assistant_chars += len(text.strip())
|
||||
blocks = []
|
||||
if text.strip():
|
||||
blocks.append({"type": "text", "text": text})
|
||||
@@ -406,6 +596,17 @@ for path in paths:
|
||||
skipped_short += 1
|
||||
continue
|
||||
|
||||
# Assistant *text* volume, tool results excluded: the signal that the
|
||||
# session actually did something, independent of how much injected
|
||||
# skill/context text inflated the user side.
|
||||
if assistant_chars < min_assistant_chars:
|
||||
skipped_quiet += 1
|
||||
print(
|
||||
f" [QUIET] {path.name} ({turn_count} turns, {assistant_chars} assistant chars)",
|
||||
file=sys.stderr,
|
||||
)
|
||||
continue
|
||||
|
||||
out_path = stage / f"pi_{session_uuid}.jsonl"
|
||||
with out_path.open("w", encoding="utf-8") as f:
|
||||
for obj in out_lines:
|
||||
@@ -428,6 +629,8 @@ print(f"EXPORTED {exported}")
|
||||
print(f"ALREADY_FILED {skipped_already_filed}")
|
||||
if skipped_short:
|
||||
print(f"SKIPPED_SHORT {skipped_short}", file=sys.stderr)
|
||||
if skipped_quiet:
|
||||
print(f"SKIPPED_QUIET {skipped_quiet}", file=sys.stderr)
|
||||
if skipped_malformed:
|
||||
print(f"SKIPPED_MALFORMED {skipped_malformed}", file=sys.stderr)
|
||||
PY
|
||||
@@ -461,12 +664,89 @@ if [[ $DRY_RUN -eq 1 ]]; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ── Ship to the palace host (remote mode only) ───────────────────────
|
||||
# mempalace_mine expands its source path in the SERVER process, so in remote
|
||||
# mode the exports have to physically exist over there. rsync --update is the
|
||||
# idempotent half; the mine is the other half.
|
||||
MINE_SOURCE="$STAGE"
|
||||
if [[ "$MODE" == "remote" ]]; then
|
||||
ssh_cmd="ssh"
|
||||
[[ -n "$SSH_CONFIG" ]] && ssh_cmd="ssh -F $SSH_CONFIG"
|
||||
echo ""
|
||||
echo "Shipping stage to ${SSH_TARGET%/}/$DEVICE/ ..."
|
||||
if ! rsync -a --update --no-owner --no-group \
|
||||
-e "$ssh_cmd" \
|
||||
--include='*.jsonl' --exclude='*' \
|
||||
"$STAGE/" "${SSH_TARGET%/}/$DEVICE/"; then
|
||||
echo "error: rsync to ${SSH_TARGET%/}/$DEVICE/ failed" >&2
|
||||
exit 5
|
||||
fi
|
||||
MINE_SOURCE="${REMOTE_PATH%/}/$DEVICE"
|
||||
fi
|
||||
|
||||
# ── Phase boundary ───────────────────────────────────────────────────
|
||||
# --prepare hands the source path to the caller (the pi mempalace extension),
|
||||
# which mines it through the MCP client that already holds the palace.
|
||||
if [[ $PREPARE_ONLY -eq 1 ]]; then
|
||||
echo ""
|
||||
printf 'MINE_SOURCE=%s\n' "$MINE_SOURCE"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ── Run the mine ─────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "Mining into wing '$WING'..."
|
||||
if ! mempalace mine "$STAGE" --mode convos --wing "$WING" --agent "$AGENT"; then
|
||||
echo "error: mempalace mine failed" >&2
|
||||
exit 4
|
||||
if [[ "$MODE" == "remote" ]]; then
|
||||
if ! python3 - "$REMOTE_URL" "$REMOTE_TOKEN" "$MINE_SOURCE" "$WING" "$AGENT" <<'PY'
|
||||
import json, sys, urllib.error, urllib.request
|
||||
|
||||
url, token, source, wing, agent = sys.argv[1:6]
|
||||
payload = json.dumps({
|
||||
"jsonrpc": "2.0",
|
||||
"id": 1,
|
||||
"method": "tools/call",
|
||||
"params": {
|
||||
"name": "mempalace_mine",
|
||||
"arguments": {"source": source, "mode": "convos", "wing": wing, "agent": agent},
|
||||
},
|
||||
}).encode()
|
||||
headers = {"Content-Type": "application/json", "Accept": "application/json"}
|
||||
if token:
|
||||
headers["Authorization"] = f"Bearer {token}"
|
||||
try:
|
||||
with urllib.request.urlopen(
|
||||
urllib.request.Request(url, data=payload, headers=headers), timeout=900
|
||||
) as resp:
|
||||
body = resp.read().decode("utf-8", "replace")
|
||||
except urllib.error.URLError as exc:
|
||||
print(f"error: remote mine transport failed: {exc}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
print(body[:4000])
|
||||
sys.exit(1 if '"error"' in body else 0)
|
||||
PY
|
||||
then
|
||||
echo "error: remote mine failed" >&2
|
||||
exit 5
|
||||
fi
|
||||
else
|
||||
# Capture output so palace-level contention can be told apart from a real
|
||||
# failure. A live pi session holds the palace through its own mempalace-mcp,
|
||||
# and that session's extension mines what we just staged — so contention
|
||||
# means "already handled", not "broken".
|
||||
set +e
|
||||
mine_out="$(mempalace mine "$MINE_SOURCE" --mode convos --wing "$WING" --agent "$AGENT" 2>&1)"
|
||||
mine_rc=$?
|
||||
set -e
|
||||
printf '%s\n' "$mine_out"
|
||||
if [[ $mine_rc -ne 0 ]]; then
|
||||
if printf '%s' "$mine_out" | grep -q "is held by"; then
|
||||
echo ""
|
||||
echo "palace is held by a live session; it will mine the staged exports itself"
|
||||
exit 0
|
||||
fi
|
||||
echo "error: mempalace mine failed" >&2
|
||||
exit 4
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── Repair index ─────────────────────────────────────────────────────
|
||||
|
||||
+53
-7
@@ -10,13 +10,26 @@
|
||||
# Strategy:
|
||||
# 1. Read opencode.db and export each qualifying session to a Claude Code
|
||||
# JSONL file (format the mempalace normalizer already understands).
|
||||
# 2. Stage exports under ~/.cache/mempalace-session/<wing>/.
|
||||
# 2. Stage exports under <palace-root>/opencode-stage/<wing>/ (override with
|
||||
# MEMPALACE_SESSION_STAGE).
|
||||
# 3. Run `mempalace mine --mode convos` against the staging dir.
|
||||
#
|
||||
# Dedup: mempalace convos mode keys on source_file (absolute staging path).
|
||||
# The staging path is deterministic (per-wing under XDG_CACHE_HOME) so re-runs
|
||||
# The staging path is deterministic (per-wing under the palace root) so re-runs
|
||||
# are idempotent as long as session content hasn't changed.
|
||||
#
|
||||
# Staging location: because dedup keys on the staged path, wiping the stage
|
||||
# leaves the palace holding drawers whose source files look deleted, and
|
||||
# `mempalace sync` prunes exactly those when they fall inside the scope it is
|
||||
# given. The stage therefore lives next to the palace it feeds (resolved via
|
||||
# $MEMPALACE_PALACE_PATH → $MEMPAL_PALACE_PATH → ~/.mempalace/config.json →
|
||||
# ~/.mempalace/palace), so stage and dedup keys share one lifetime and the
|
||||
# dangerous state — palace survives, stage does not — cannot be reached by
|
||||
# wiping something that merely looks disposable. It used to default under
|
||||
# ~/.cache, which is disposable on exactly the hosts where this runs
|
||||
# unattended. Override with MEMPALACE_SESSION_STAGE only if the target is at
|
||||
# least as durable as the palace.
|
||||
#
|
||||
# Session filter: sessions with fewer than --min-messages messages (default 3)
|
||||
# are skipped to avoid filing throwaway /exit'd sessions.
|
||||
#
|
||||
@@ -87,7 +100,7 @@ Idempotency:
|
||||
|
||||
What gets mined:
|
||||
- Each qualifying session → one Claude Code JSONL file
|
||||
- Staged under ~/.cache/mempalace-session/<wing>/
|
||||
- Staged under <palace-root>/opencode-stage/<wing>/
|
||||
- Filed via `mempalace mine --mode convos`
|
||||
|
||||
Transcript shape per session:
|
||||
@@ -103,7 +116,9 @@ Transcript shape per session:
|
||||
Dedup:
|
||||
- source_file = absolute staging path (deterministic per session ID)
|
||||
- Re-runs skip unchanged sessions. To force re-mining, delete the staging
|
||||
dir: rm -rf ~/.cache/mempalace-session/<wing>/
|
||||
dir: rm -rf <palace-root>/opencode-stage/<wing>/
|
||||
(that only forces a refile; do NOT run `mempalace sync` while the stage is
|
||||
missing, or the drawers mined from it get pruned instead)
|
||||
|
||||
Rationale:
|
||||
Opencode lacks a session-stopping hook (upstream PRs #16598, #16769 still
|
||||
@@ -147,8 +162,31 @@ if ! [[ "$MIN_MESSAGES" =~ ^[0-9]+$ ]]; then
|
||||
fi
|
||||
|
||||
# ── Staging dir ──────────────────────────────────────────────────────
|
||||
# Deterministic per-wing path so source_file dedup works across re-runs.
|
||||
CACHE_ROOT="${XDG_CACHE_HOME:-$HOME/.cache}/mempalace-session"
|
||||
# Deterministic per-wing path so source_file dedup works across re-runs, and
|
||||
# anchored to the palace root so the stage cannot be wiped independently of the
|
||||
# dedup keys that reference it (see "Staging location" in the header).
|
||||
# Mirrors mempalace config.py:palace_path() precedence, then takes the parent.
|
||||
palace_root() {
|
||||
python3 - <<'PY' 2>/dev/null || echo "$HOME/.mempalace"
|
||||
import json, os
|
||||
p = os.environ.get("MEMPALACE_PALACE_PATH") or os.environ.get("MEMPAL_PALACE_PATH")
|
||||
if p:
|
||||
p = os.path.abspath(os.path.expanduser(p))
|
||||
else:
|
||||
cfg = os.path.expanduser("~/.mempalace/config.json")
|
||||
p = None
|
||||
if os.path.exists(cfg):
|
||||
try:
|
||||
with open(cfg) as fh:
|
||||
v = json.load(fh).get("palace_path")
|
||||
p = os.path.expanduser(v) if v else None
|
||||
except Exception:
|
||||
p = None
|
||||
p = p or os.path.expanduser("~/.mempalace/palace")
|
||||
print(os.path.dirname(p.rstrip("/")))
|
||||
PY
|
||||
}
|
||||
CACHE_ROOT="${MEMPALACE_SESSION_STAGE:-$(palace_root)/opencode-stage}"
|
||||
STAGE="$CACHE_ROOT/$WING"
|
||||
mkdir -p "$STAGE"
|
||||
|
||||
@@ -185,7 +223,15 @@ if since:
|
||||
# isn't reachable (first install, moved, permission-denied), we fall through
|
||||
# to "everything is new" — the mine step will do the real dedup anyway.
|
||||
already_filed = set()
|
||||
palace_path = os.environ.get("MEMPALACE_PATH", os.path.expanduser("~/.mempalace/palace"))
|
||||
# Mirror mempalace's own resolution order (config.py): MEMPALACE_PALACE_PATH,
|
||||
# then the legacy MEMPAL_PALACE_PATH, then the default. NOT "MEMPALACE_PATH" —
|
||||
# that name is not a mempalace concept, and reading it silently degraded this
|
||||
# NEW/SKIP preview to "everything is new" wherever some other tool had set it.
|
||||
palace_path = (
|
||||
os.environ.get("MEMPALACE_PALACE_PATH")
|
||||
or os.environ.get("MEMPAL_PALACE_PATH")
|
||||
or os.path.expanduser("~/.mempalace/palace")
|
||||
)
|
||||
chroma_db = Path(palace_path) / "chroma.sqlite3"
|
||||
if chroma_db.is_file():
|
||||
try:
|
||||
|
||||
+20
-5
@@ -2,6 +2,16 @@
|
||||
|
||||
Manual invocation of the session-mining wrappers is fine on a machine you actively drive. For long-running devboxes, a weekly automated mine keeps the palace fresh without thinking about it. This directory ships ready-to-use templates for two common scheduling mechanisms, for each wrapper.
|
||||
|
||||
> **pi machines: check whether you need this at all.** If the pi bridge
|
||||
> extension (`extensions/pi/mempalace.ts`) is installed, it already feeds the
|
||||
> palace by itself on `session_shutdown` and a debounced `agent_settled` —
|
||||
> see [`extensions/pi/README.md` § Automatic transcript feeding](../extensions/pi/README.md#automatic-transcript-feeding).
|
||||
> The templates below were written when scheduling was the *only* path for
|
||||
> both harnesses; that's still true for **opencode** (no such extension
|
||||
> exists), but for pi they're now a fallback — useful for a bare pi install
|
||||
> without the bridge, a host-level catch-up job, or belt-and-braces coverage
|
||||
> of a hard container kill (the extension's triggers don't fire on `SIGKILL`).
|
||||
|
||||
> **Before using either**: confirm the toolkit is installed and the wrapper works —
|
||||
> `mempalace-session --dry-run` (and/or `mempalace-pi-session --dry-run`) should list qualifying sessions. If that errors, fix the install before scheduling.
|
||||
|
||||
@@ -171,17 +181,22 @@ rm /tmp/mempalace-session.cron
|
||||
crontab -l | grep mempalace
|
||||
```
|
||||
|
||||
Ensure `~/.cache/mempalace-session/` exists so the log file can be written:
|
||||
Ensure `~/.cache/mempalace-logs/` exists so the log file can be written:
|
||||
|
||||
> This is the **log** directory only. The staging dir — the transcripts the
|
||||
> palace keys its `source_file` dedup on — lives beside the palace
|
||||
> (`<palace-root>/opencode-stage/`), not in `~/.cache`, precisely so it cannot
|
||||
> be cleaned away while the palace survives. Logs here are disposable.
|
||||
|
||||
```bash
|
||||
mkdir -p ~/.cache/mempalace-session
|
||||
mkdir -p ~/.cache/mempalace-logs
|
||||
```
|
||||
|
||||
**Verify a run is happening:**
|
||||
|
||||
```bash
|
||||
# Tail the log the cron entry writes to
|
||||
tail -f ~/.cache/mempalace-session/cron.log
|
||||
tail -f ~/.cache/mempalace-logs/cron.log
|
||||
|
||||
# Or force a run manually to prove the command is well-formed
|
||||
mempalace-session
|
||||
@@ -282,11 +297,11 @@ cat contrib/cron/mempalace-session-devbox.cron
|
||||
(crontab -l 2>/dev/null; cat contrib/cron/mempalace-session-devbox.cron) | crontab -
|
||||
|
||||
# Ensure the log directory exists
|
||||
mkdir -p ~/.cache/mempalace-session
|
||||
mkdir -p ~/.cache/mempalace-logs
|
||||
|
||||
# Verify
|
||||
crontab -l | grep mempalace-session-devbox
|
||||
tail -f ~/.cache/mempalace-session/cron-devbox.log
|
||||
tail -f ~/.cache/mempalace-logs/cron-devbox.log
|
||||
```
|
||||
|
||||
**Uninstall:**
|
||||
|
||||
@@ -35,4 +35,4 @@ CONTAINER_USER=developer
|
||||
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
|
||||
|
||||
# m h dom mon dow command
|
||||
0 3 * * 1 /bin/sh -c 'docker ps --filter "name=^/${CONTAINER}$" --filter "status=running" -q | grep -q . && docker exec -u "${CONTAINER_USER}" "${CONTAINER}" mempalace-session >> "$HOME/.cache/mempalace-session/cron-devbox.log" 2>&1'
|
||||
0 3 * * 1 /bin/sh -c 'docker ps --filter "name=^/${CONTAINER}$" --filter "status=running" -q | grep -q . && docker exec -u "${CONTAINER_USER}" "${CONTAINER}" mempalace-session >> "$HOME/.cache/mempalace-logs/cron-devbox.log" 2>&1'
|
||||
|
||||
@@ -16,4 +16,4 @@
|
||||
PATH=/home/USER/.local/bin:/usr/local/bin:/usr/bin:/bin
|
||||
|
||||
# m h dom mon dow command
|
||||
0 3 * * 1 mempalace-session >> /home/USER/.cache/mempalace-session/cron.log 2>&1
|
||||
0 3 * * 1 mempalace-session >> /home/USER/.cache/mempalace-logs/cron.log 2>&1
|
||||
|
||||
@@ -8,7 +8,7 @@ ConditionPathExists=!%t/mempalace-session.lock
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
# The wrapper writes to ~/.cache/mempalace-session/ and the palace.
|
||||
# The wrapper writes to ~/.cache/mempalace-logs/ and the palace.
|
||||
# Keep stdout/stderr in the journal — inspect with:
|
||||
# journalctl --user -u mempalace-session --since today
|
||||
ExecStart=%h/.local/bin/mempalace-session
|
||||
|
||||
@@ -0,0 +1,185 @@
|
||||
# Phase 1 exposure — newt on synlig, DNS, and the client auth model
|
||||
|
||||
Companion to [`rfc-001-global-palace.md`](./rfc-001-global-palace.md) (design + decisions) and
|
||||
[`synlig-primary-runbook.md`](./synlig-primary-runbook.md) (what is already installed on the primary).
|
||||
This doc covers only the step the other two leave open: **making the primary reachable** — runbook §4
|
||||
items 2 and 5.
|
||||
|
||||
**Status 2026-08-12 — Pangolin updated on nyvaken (done, yours). newt not yet installed on synlig.
|
||||
Nothing exposed. No client `.env` flipped.**
|
||||
|
||||
Read this before touching Pangolin: three of the four questions this step raises were **already decided**
|
||||
in RFC §6.2 on 2026-08-09, and re-deciding them differently is how the fleet ends up in two states.
|
||||
|
||||
---
|
||||
|
||||
## 1. The four questions, answered
|
||||
|
||||
| Question | Answer | Where it was decided |
|
||||
| --- | --- | --- |
|
||||
| Which port? | **8765**, path **`/mcp`** (liveness: `/healthz`) | `cli.py:2141` default; runbook §2.4 |
|
||||
| What does newt target? | **`172.17.0.1:8765`** (docker0), **never** `127.0.0.1` | RFC §6.2 Transport; runbook §2.4 |
|
||||
| Open, or authenticated? | **Authenticated. The primary is never an open public resource.** | RFC §6.2 Network posture |
|
||||
| Per-device credentials? | **No — Phase 1 ships the single shared bearer token.** Per-device tokens are Phase 4. | RFC §6.2 Authentication |
|
||||
|
||||
### 1.1 Why not per-device users at the proxy
|
||||
|
||||
The instinct — "create a Pangolin user per container, put the credentials in each `.env`, keep the
|
||||
usernames distinct" — is the right *goal* (revocation, attribution) reached through the wrong *layer*,
|
||||
twice over:
|
||||
|
||||
1. **mempalace validates exactly one token.** `hmac.compare_digest(provided, f"Bearer {srv.auth_token}")`
|
||||
(`mcp_server.py:5292-5295`) — there is no user table and no second credential. Per-device HTTP identity
|
||||
is not a configuration you can express today; it is Phase 4 work (a server-side
|
||||
`token → {device_id, scopes}` registry). RFC §6.2 chose the shared token for Phase 1 deliberately:
|
||||
*"iterate more feature rich but more complex solutions over time."*
|
||||
|
||||
2. **Pangolin's HTTP auth is browser-shaped; the clients are not.** SSO login, resource PIN and resource
|
||||
password all assume something that can follow a redirect, render a form and hold a session cookie.
|
||||
Every MemPalace client here is a headless JSON-RPC `POST` with an `Authorization` header — pi's
|
||||
extension, opencode's `type:remote` MCP entry, and `mempalace-pi-session --mode remote`'s
|
||||
`urllib.request.urlopen`. Point those at a user-authenticated resource and they receive a login page
|
||||
where JSON should be. Enabling that protection breaks precisely the clients it is meant to protect.
|
||||
|
||||
So: **Pangolin terminates TLS and nothing more** (RFC §6.2 Transport, decided 2026-08-09). The bearer token
|
||||
is the authentication. This is not "unprotected" — an unauthenticated request to `/mcp` gets a 401 from
|
||||
mempalace itself, verified A4/A5 in runbook §2.4.
|
||||
|
||||
**Consequence to accept consciously** (RFC §6.2, §7.3.2): until Phase 4 the primary **cannot tell devices
|
||||
apart**. `origin_device` is client-asserted and advisory — nothing load-bearing may depend on it, and
|
||||
revoking one laptop means rotating the token everywhere.
|
||||
|
||||
### 1.2 The one place per-device identity *does* exist today
|
||||
|
||||
Remote mode is not only HTTP. `mempalace_mine` expands its source path in the **server** process, so a
|
||||
client's staged transcripts must physically exist on the primary. The feeder therefore ships them over
|
||||
SSH into a **per-device inbox** before asking the server to mine its own copy:
|
||||
|
||||
```sh
|
||||
rsync -a --update -e "$ssh_cmd" "$STAGE/" "${SSH_TARGET%/}/$DEVICE/" # bin/mempalace-pi-session:677-680
|
||||
```
|
||||
|
||||
That SSH key **is** per-device identity, and it is individually revocable (one line out of
|
||||
`authorized_keys`) years before Phase 4 lands. It costs nothing extra, because the mining path needs SSH
|
||||
regardless.
|
||||
|
||||
Two implications people miss:
|
||||
|
||||
- **`mempalace.jordbo.se` alone does not enable mining.** HTTPS covers the read/write tool surface
|
||||
(`search`, `add_drawer`, `diary_write`, `kg_*`) — genuinely useful on its own, and the reason to do this
|
||||
at all. But `--mode remote` also needs `MEMPALACE_PI_SSH_TARGET` reachable. Budget for both paths.
|
||||
- **`DEVICE` defaults to `$(hostname)`** (`bin/mempalace-pi-session:163`). In a container that is the
|
||||
container hostname: either random per recreate (inboxes proliferate; each recreate re-mines into a fresh
|
||||
empty inbox) or identical across sibling devboxes (two containers writing one inbox). **Set
|
||||
`MEMPALACE_PI_DEVICE` explicitly per container.** It is a label, not a secret, so put it somewhere
|
||||
reviewable — a committed compose file — where duplicates are visible. That, not username hygiene in
|
||||
`.env`, is the discipline this design actually asks of you.
|
||||
|
||||
---
|
||||
|
||||
## 2. The bind trap, in full
|
||||
|
||||
RFC §6.2 and runbook §2.4 already say **do not bind loopback behind the tunnel**, because
|
||||
`enforce_host_pin = _http_is_loopback(host)` (`mcp_server.py:5367`) makes a loopback bind reject the
|
||||
proxy's forwarded `Host:` with a **403** that reads exactly like a Pangolin misconfiguration.
|
||||
|
||||
**Additional finding, 2026-08-12 — the same reflex also silently removes authentication.** Token
|
||||
resolution in `cmd_serve` (`cli.py:1447-1450`) is:
|
||||
|
||||
```python
|
||||
loopback = _server_is_loopback(host)
|
||||
if not token and not loopback and not args.allow_insecure:
|
||||
token, token_created = _load_or_create_server_token(palace_path)
|
||||
```
|
||||
|
||||
Auto-minting is gated on the bind being **non-loopback**. A loopback bind therefore starts with **no token
|
||||
at all** — no error, no warning, `--allow-insecure` not required — because the server has concluded it is
|
||||
only reachable locally, while the tunnel is serving it to the internet. Bind loopback behind newt and you
|
||||
get a 403 wall *and*, the moment anything relaxes the Host pin, an unauthenticated palace.
|
||||
|
||||
Both failure modes have the same cure, already implemented in
|
||||
`contrib/systemd/mempalace-serve.service`: **bind `172.17.0.1`**. Non-loopback, so the Host pin relaxes and
|
||||
the token is mandatory; docker0-only, so newt reaches it and the LAN does not.
|
||||
|
||||
> Belt and braces: set `MEMPALACE_MCP_HTTP_TOKEN` explicitly in the unit rather than relying on
|
||||
> auto-minting. Then no future bind change can quietly drop authentication.
|
||||
|
||||
---
|
||||
|
||||
## 3. Steps
|
||||
|
||||
Ordered so nothing is reachable before it is authenticated.
|
||||
|
||||
### 3.1 Start the primary (runbook §4.3 — one `sudo`, unit already staged)
|
||||
|
||||
```sh
|
||||
sudo loginctl enable-linger ecsjper
|
||||
cd ~/.config/systemd/user && mv mempalace-serve.service.staged mempalace-serve.service
|
||||
systemctl --user daemon-reload && systemctl --user enable --now mempalace-serve
|
||||
|
||||
curl -s 172.17.0.1:8765/healthz # expect ok
|
||||
curl -s 127.0.0.1:8765/healthz # expect 403 — correct, not a bug (§2)
|
||||
ss -ltnp | grep 8765 # expect 172.17.0.1:8765 only
|
||||
```
|
||||
|
||||
### 3.2 Collect the shared token
|
||||
|
||||
```sh
|
||||
cat ~/.mempalace/server/f5d849287f6d73f0141b29d7/token
|
||||
```
|
||||
|
||||
Directory name is `sha256(realpath(palace))[:24]` — it changes if the palace path ever moves. Store via the
|
||||
`.env.age` flow, 0600 (RFC §6.2).
|
||||
|
||||
### 3.3 newt on synlig
|
||||
|
||||
synlig runs Docker (Gitea Actions runner + digikam) but **no tunnel client** — runbook §4.2. Pangolin on
|
||||
nyvaken cannot dial in; synlig must dial out. Add a `newt` container with the credentials Pangolin issues
|
||||
for a new site.
|
||||
|
||||
Because newt runs in Docker on this box, the docker0 bind is already correct for it: from inside the
|
||||
container the primary is `172.17.0.1:8765`. Verify from *inside* newt's network namespace, not from the
|
||||
host, before touching DNS.
|
||||
|
||||
> synlig has 7.8 GiB shared with a CI runner (runbook §1). newt is small, but do not colocate anything
|
||||
> else here casually.
|
||||
|
||||
### 3.4 DNS at the web hotel
|
||||
|
||||
One CNAME: `mempalace` → **the same target your existing Pangolin resources use** (nyvaken's public
|
||||
hostname). RFC §6.2 costed this as *"one DNS record per service on the web hotel is the whole setup cost."*
|
||||
|
||||
⚠️ Not verified from here: nyvaken's public FQDN, and whether your web hotel permits a CNAME at that label
|
||||
(some require an A record, or forbid CNAME where other records exist). Confirm before assuming a 5-minute job.
|
||||
|
||||
### 3.5 Pangolin resource
|
||||
|
||||
- Target: newt site → `172.17.0.1:8765`, path `/mcp` (plus `/healthz` if you want the external probe).
|
||||
- **Auth: none at the Pangolin layer** (§1.1). TLS termination only.
|
||||
- Do **not** attach an `Origin`-injecting proxy or browser client: a *present* non-loopback `Origin` is a
|
||||
hard 403 with no override (runbook §2.4 B3).
|
||||
|
||||
### 3.6 Verify end-to-end before flipping any client
|
||||
|
||||
```sh
|
||||
curl -s https://mempalace.jordbo.se/healthz # ok
|
||||
curl -s -o /dev/null -w '%{http_code}\n' -X POST \
|
||||
https://mempalace.jordbo.se/mcp # 401 — the token is doing its job
|
||||
curl -s -X POST https://mempalace.jordbo.se/mcp \
|
||||
-H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
|
||||
-d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | head -c 300 # 36 tools
|
||||
```
|
||||
|
||||
The 401 check matters as much as the 200: it is the only evidence that the thing you just published to the
|
||||
internet is not open. Then, and only then, Phase 1 client flip — **one machine first** (RFC §8), and
|
||||
remember opencode containers need the §4.1 sidecar merge before their `.env` takes effect.
|
||||
|
||||
---
|
||||
|
||||
## 4. Still open
|
||||
|
||||
- **Per-device tokens** — Phase 4. Until then `origin_device` is advisory (§1.1).
|
||||
- **§7.6 diary dedup** must be settled *before* the first §4.4 join; replay duplicates every entry.
|
||||
- **§7.2**: never run `mempalace sync` against the shared palace. Doubly true now that the pi/opencode
|
||||
feeders stage *inside* the palace root, which puts staged sources in scope for a sync of the palace dir.
|
||||
- **nyvaken's public FQDN and the web hotel's CNAME rules** — unverified (§3.4).
|
||||
@@ -100,6 +100,16 @@ Two of the three pieces already exist. This is not greenfield.
|
||||
> `pi-devbox/.env.example:21` still advertises `mempalace-mcp --transport http --host 0.0.0.0 --port
|
||||
> 8765` as the way to serve a shared palace — replace with `mempalace serve --token … --tls-cert …`,
|
||||
> and change the example URL from `http://mempalace.lan:8765/mcp` to `https://`.
|
||||
>
|
||||
> **✅ Done 2026-08-12** — `pi-devbox/.env.example` and
|
||||
> `mempalace-toolkit/extensions/pi/README.md` both now recommend `mempalace serve`, bind docker0 rather
|
||||
> than `0.0.0.0`/loopback, use an `https://` example URL, and state that the transport *is*
|
||||
> authenticated. `docker-compose.mempalace.yml` audited too, and it was worse than stale — it was
|
||||
> **broken on 3.6.0 in both directions**: `--host 0.0.0.0` with no token in the environment makes the
|
||||
> server refuse to start (crash-looping under `restart: unless-stopped`), and once a token *is* supplied
|
||||
> the healthcheck's unauthenticated `tools/list` POST 401s, marking a healthy server unhealthy forever.
|
||||
> Fixed: token now required via `${MEMPALACE_REMOTE_TOKEN:?}` (fails fast at `up`), healthcheck switched
|
||||
> to the token-free `/healthz`.
|
||||
|
||||
### Two things that sound like the feature and are not
|
||||
|
||||
@@ -682,7 +692,7 @@ join replays (§4.4).
|
||||
|
||||
| Phase | Effort | Deliverable |
|
||||
| --- | --- | --- |
|
||||
| **0 — hygiene** | hours | §7 runbook: converge the KG/entities store paths **on synlig before first `serve`** (§7.1 — **done 2026-08-10**, runbook §2.3), ban `sync` on shared palaces (§7.2), fix stale "unauthenticated" docs (incl. `pi-devbox/.env.example:21`). Added 2026-08-09: settle the **diary dedup** approach and file its upstream ask (§7.6), and **dry-run the join from one palace only** (§4.4). **No provenance work here** — it is not backfill-critical (§7.3.3) and belongs to the stamper, not the agent |
|
||||
| **0 — hygiene** | hours | §7 runbook: converge the KG/entities store paths **on synlig before first `serve`** (§7.1 — **done 2026-08-10**, runbook §2.3), ban `sync` on shared palaces (§7.2), ~~fix stale "unauthenticated" docs (incl. `pi-devbox/.env.example:21`)~~ — **done 2026-08-12**, and `docker-compose.mempalace.yml` turned out to be outright broken on 3.6.0, now fixed. Added 2026-08-09: settle the **diary dedup** approach and file its upstream ask (§7.6), and **dry-run the join from one palace only** (§4.4). **No provenance work here** — it is not backfill-critical (§7.3.3) and belongs to the stamper, not the agent |
|
||||
| **1 — primary up** | hours, **no code** | `mempalace serve --token --tls-cert` on a private-net host (reuse `docker-compose.mempalace.yml` — keep it a **separate standalone project**, R4 — and mind port 8765 vs pi-studio; tor-ms22 already moved to 8766). Repoint pi clients via `MEMPALACE_REMOTE_URL`/`MEMPALACE_REMOTE_TOKEN`. **opencode clients can be repointed in the same breath** — remote MCP is supported and `generate-config.py` already emits it (§9.1, §9.6), subject to the sidecar caveat in §4.1. **Shared memory today, no offline.** **Decided 2026-08-09: primary = synlig, TLS at Pangolin, single shared token (§8.1)** — mind the loopback Host-pin trap in §6.2. |
|
||||
| **1.5 — opencode env propagation** | hours | Make the `mcp.mempalace` subtree env-authoritative in `generate-config.py`, gated by a generated-value fingerprint (§4.1). Independent of the rest of this RFC. Without it, adopting *or reverting* the opt-in on an existing opencode container needs a manual sidecar merge or a `docker volume rm` — which also blocks R6 reversibility |
|
||||
| **2 — `mempalace-edge`** | ~1 week | The actual ask: local-first writes + outbox flush + merged reads + per-wing policy. **Not** "fixes opencode" — opencode's *transport* is already fine after Phase 1; what edge adds there is offline/local-first, since `generate-config.py`'s switch is remote **or** local with no failover. **Ships with the §1.2 opt-in wiring (compose + `.env.example` + a third branch in the existing `generate-config.py`) and must pass the R1 acceptance test.** |
|
||||
|
||||
@@ -128,7 +128,13 @@ standing up a network-reachable service while you were asleep was not mine to de
|
||||
|
||||
## 4. Tomorrow, in order
|
||||
|
||||
1. **Pangolin update on nyvaken** (yours).
|
||||
> **2026-08-12: items 1–2 and 5 now have their own runbook —**
|
||||
> [`phase-1-exposure-runbook.md`](./phase-1-exposure-runbook.md). Pangolin on nyvaken is updated (done).
|
||||
> newt is still missing on synlig. That doc also records why per-device Pangolin users are the wrong layer,
|
||||
> and an additional loopback finding: a loopback bind does not merely 403, it also silently starts the
|
||||
> server with **no token at all** (auto-minting is gated on the bind being non-loopback).
|
||||
|
||||
1. **Pangolin update on nyvaken** (yours). ✅ done 2026-08-12.
|
||||
2. **⚠️ synlig has no tunnel client.** `docker ps` shows only the Gitea runner and digikam — no `newt`.
|
||||
Pangolin on nyvaken cannot reach synlig by itself; synlig needs a newt container (or an equivalent
|
||||
outbound tunnel) added. Easy to miss because Pangolin will look healthy on its own side. If newt runs
|
||||
|
||||
+88
-9
@@ -16,6 +16,7 @@ dependencies (~300 MB).
|
||||
**Jump to:**
|
||||
- [What it does](#what-it-does)
|
||||
- [Transport: local vs external](#transport-local-vs-external)
|
||||
- [Automatic transcript feeding](#automatic-transcript-feeding)
|
||||
- [The `Type.Unsafe` gotcha](#the-typeunsafe-gotcha)
|
||||
- [Deploying pi with mempalace on a new machine](#deploying-pi-with-mempalace-on-a-new-machine)
|
||||
- [Fail-soft, identity, debugging](#fail-soft)
|
||||
@@ -37,11 +38,75 @@ dependencies (~300 MB).
|
||||
injects the result as a `mempalace-wakeup` system message so the
|
||||
agent orients itself the way `~/.agents/skills/mempalace/SKILL.md`
|
||||
describes. Skipped on resume/fork (context is already in the thread).
|
||||
4. **Manual wind-down** via a `/mempalace-diary [topic]` slash command:
|
||||
4. **Automatic transcript feeding** (`session_shutdown`, and a debounced
|
||||
`agent_settled`): stages + mines this pi installation's own session
|
||||
transcripts into the palace with no user action needed. Unlike the diary
|
||||
below, this needs no LLM turn — it's a subprocess + a tool call — so it
|
||||
*can* run on `session_shutdown` where the diary cannot. See
|
||||
[Automatic transcript feeding](#automatic-transcript-feeding).
|
||||
5. **Manual wind-down** via a `/mempalace-diary [topic]` slash command:
|
||||
sends a prompt asking the LLM to call `mempalace_diary_write` with
|
||||
an AAAK-formatted entry summarizing the session. Not fully auto
|
||||
because pi sessions are typically short/tactical and
|
||||
`session_shutdown` fires too late to drive another LLM turn.
|
||||
an AAAK-formatted entry summarizing the session. This one stays manual
|
||||
because it needs the LLM to compose the entry, and `session_shutdown`
|
||||
fires too late to drive another LLM turn — a constraint that applies to
|
||||
the diary specifically, not to feeding (see above).
|
||||
|
||||
## Automatic transcript feeding
|
||||
|
||||
The bridge feeds this pi installation's own session transcripts into the
|
||||
palace by itself — no scheduler, no cron, no manual invocation. It fires on
|
||||
`session_shutdown` (covers quit, `/new`, `/resume`, `/fork`) and on a
|
||||
debounced `agent_settled` (covers a long session that later crashes, since a
|
||||
hard kill runs no shutdown handler at all).
|
||||
|
||||
The work is split across two processes, and the reason is a hard constraint,
|
||||
not a style choice: **the palace is single-writer.** A live pi session
|
||||
always holds it through this extension's own `mempalace-mcp` subprocess, so
|
||||
an unattended `mempalace mine` from anywhere else fails outright with
|
||||
`palace ... is held by PID <n>`. The bridge therefore:
|
||||
|
||||
1. Runs `mempalace-pi-session --prepare --reason <trigger> --wing <wing>` as a
|
||||
subprocess. This does every palace-free step — parse pi's JSONL, apply
|
||||
the quality threshold, stage the export, and (remote mode only) `rsync`
|
||||
it to the palace host — and prints one line, `MINE_SOURCE=<path>`,
|
||||
without ever touching the palace.
|
||||
2. Calls the `mempalace_mine` MCP tool **through this extension's own
|
||||
client** on that path. Going through the client that already holds the
|
||||
lock is the only way to write during a live session, and it automatically
|
||||
targets whichever palace the bridge is pointed at — local stdio or a
|
||||
shared remote one.
|
||||
|
||||
`mempalace-pi-session` (in this repo's `bin/`) is the actual exporter and
|
||||
owns the quality gate, the remote transport, and every flag — see its
|
||||
`--help` for the full reference; this section only covers the extension's
|
||||
side of the wiring.
|
||||
|
||||
**Env knobs (extension side):**
|
||||
|
||||
| Var | Default | Effect |
|
||||
|---|---|---|
|
||||
| `MEMPALACE_FEED` | `1` | Set `0` to disable automatic feeding entirely. |
|
||||
| `MEMPALACE_FEED_BIN` | `mempalace-pi-session` | Helper to run. |
|
||||
| `MEMPALACE_FEED_WING` | `wing_conversations` | Target wing — passed to both the exporter and the `mempalace_mine` call. |
|
||||
| `MEMPALACE_FEED_DEBOUNCE_MS` | `600000` (10 min) | Minimum gap between mid-session (`agent_settled`) feeds. Bounds crash loss to one window instead of a whole session. |
|
||||
| `MEMPALACE_FEED_PREPARE_TIMEOUT_MS` | `120000` | Kills a wedged `--prepare` subprocess. |
|
||||
| `MEMPALACE_FEED_MINE_TIMEOUT_MS` | `30000` | Caps the `mempalace_mine` call so a stalled palace can't hang session exit. |
|
||||
|
||||
**Remote palace:** if `$MEMPALACE_REMOTE_URL` is set (see
|
||||
[Transport](#transport-local-vs-external)), `mempalace_mine`'s source path is
|
||||
expanded on the *server*, which cannot see this machine's transcripts —
|
||||
that's exactly why step 1 above rsyncs first in that mode. Configure the
|
||||
inbox with `MEMPALACE_PI_SSH_TARGET` (required for remote feeding — feeding
|
||||
is silently skipped without it), `MEMPALACE_PI_SSH_CONFIG`, and
|
||||
`MEMPALACE_PI_REMOTE_PATH`; see `mempalace-pi-session --help`.
|
||||
|
||||
**Concurrency:** overlapping triggers coalesce — a `session_shutdown` landing
|
||||
while a debounced tick is still running joins that in-flight feed instead of
|
||||
racing it. `mempalace-pi-session` itself also takes a non-blocking `flock`,
|
||||
so even two independent invocations (e.g. this extension and the
|
||||
container-start catch-up some devbox images run) never race each other;
|
||||
losing that race is harmless because the next trigger re-exports from
|
||||
scratch.
|
||||
|
||||
## Transport: local vs external
|
||||
|
||||
@@ -57,11 +122,25 @@ chosen at load time:
|
||||
harnesses/containers (pi + opencode + native). `MEMPALACE_REMOTE_TOKEN`, if
|
||||
set, is sent as `Authorization: Bearer <token>`.
|
||||
|
||||
Serve such an endpoint with `mempalace-mcp --transport http --host 0.0.0.0
|
||||
--port 8765` (the `pi-devbox` / `opencode-devbox` repos ship a
|
||||
`docker-compose.mempalace.yml` for exactly this). Note: that HTTP transport is
|
||||
currently sessionless and **unauthenticated** — keep it on a trusted network
|
||||
or behind a reverse proxy that enforces the bearer token.
|
||||
Serve such an endpoint with `mempalace serve --host 172.17.0.1 --port 8765`
|
||||
(the `pi-devbox` / `opencode-devbox` repos ship a
|
||||
`docker-compose.mempalace.yml` for exactly this).
|
||||
|
||||
**The HTTP transport is authenticated as of mempalace 3.6.0** — earlier docs
|
||||
here said otherwise, from the v1.3.0 era. `serve` mints a bearer token, keeps
|
||||
it 0600, passes it via the environment (never argv), compares it with
|
||||
`hmac.compare_digest`, and **refuses to bind a non-loopback host without one**
|
||||
unless `--allow-insecure`. It also pins `Host` and allowlists `Origin`
|
||||
(anti-DNS-rebinding), and can terminate TLS itself.
|
||||
|
||||
Two binds to avoid. `0.0.0.0` publishes the palace to the whole LAN. And
|
||||
`127.0.0.1` is the trap that looks safe: the Host pin is enforced *only* on
|
||||
loopback binds, so behind a tunnel every proxied request 403s — and
|
||||
token auto-minting is gated on the bind being non-loopback, so it starts with
|
||||
**no authentication at all**, no warning. Bind the docker0 gateway
|
||||
(`172.17.0.1`): reachable from the host and its containers, not from the LAN.
|
||||
See
|
||||
[`docs/phase-1-exposure-runbook.md`](../../docs/phase-1-exposure-runbook.md).
|
||||
|
||||
Implementation note: the HTTP client (`RemoteMcpClient`) is **vendored** from
|
||||
[`pi-extensions`](https://gitea.jordbo.se/joakimp/pi-extensions)'
|
||||
|
||||
@@ -21,6 +21,21 @@
|
||||
* `mempalace_status` + `mempalace_diary_read` output as context so the
|
||||
* agent orients itself the way the mempalace skill describes. Skipped
|
||||
* on resume/fork (palace context is already in the thread).
|
||||
* - Feeding (auto): stage + mine this container's pi transcripts into the
|
||||
* palace on `session_shutdown` and on a debounced `agent_settled`. Needs
|
||||
* no LLM turn (pi transcripts are JSONL on disk), which is why it CAN be
|
||||
* automatic where the diary cannot. The file-side work is delegated to
|
||||
* `mempalace-pi-session --prepare` (export + threshold + staging, plus the
|
||||
* rsync to the palace host when the palace is remote); the mine itself
|
||||
* must run through THIS client, because the palace is single-writer and
|
||||
* this process is the holder — a CLI `mempalace mine` during a live
|
||||
* session dies with "palace ... is held by PID <ours>". Going through the
|
||||
* client also means it automatically targets whichever palace this bridge
|
||||
* is pointed at (local stdio or a shared remote one).
|
||||
* - MEMPALACE_FEED=0 disable feeding entirely
|
||||
* - MEMPALACE_FEED_BIN helper to run (default mempalace-pi-session)
|
||||
* - MEMPALACE_FEED_WING target wing (default wing_conversations)
|
||||
* - MEMPALACE_FEED_DEBOUNCE_MS min gap between mid-session feeds (default 600000)
|
||||
* - Wind-down (manual): `/mempalace-diary` command prompts the LLM to
|
||||
* write an AAAK-formatted diary entry. Not fully auto because pi
|
||||
* sessions are typically short/tactical and session_shutdown is too
|
||||
@@ -724,7 +739,112 @@ export default async function mempalaceExtension(pi: ExtensionAPI) {
|
||||
});
|
||||
}
|
||||
|
||||
// --- Automatic transcript feeding ---
|
||||
//
|
||||
// Split deliberately: `mempalace-pi-session --prepare` does the palace-free
|
||||
// file work (export + quality threshold + staging, plus the rsync to the
|
||||
// palace host in remote mode) and prints the path to mine; we then mine it
|
||||
// through this client. See the header note on single-writer contention.
|
||||
const feedEnabled = (process.env.MEMPALACE_FEED ?? "1") !== "0";
|
||||
const feedBin = process.env.MEMPALACE_FEED_BIN || "mempalace-pi-session";
|
||||
const feedWing = process.env.MEMPALACE_FEED_WING ?? "wing_conversations";
|
||||
const feedDebounceMs = num(process.env.MEMPALACE_FEED_DEBOUNCE_MS, 600_000);
|
||||
const feedPrepareTimeoutMs = num(process.env.MEMPALACE_FEED_PREPARE_TIMEOUT_MS, 120_000);
|
||||
const feedMineTimeoutMs = num(process.env.MEMPALACE_FEED_MINE_TIMEOUT_MS, 30_000);
|
||||
let lastFeedAt = 0; // 0 => the first settled turn also acts as a catch-up
|
||||
let feedInFlight: Promise<void> | null = null;
|
||||
|
||||
/** Run `mempalace-feed --prepare`; resolve the path to mine, or null. */
|
||||
function prepareFeed(reason: string): Promise<string | null> {
|
||||
return new Promise((resolve) => {
|
||||
// A missing helper surfaces as an async 'error' event (ENOENT), not a
|
||||
// throw, so the handler below is the fail-soft path.
|
||||
const child = spawn(feedBin, ["--prepare", "--reason", reason, "--wing", feedWing], {
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
});
|
||||
let out = "";
|
||||
let settled = false;
|
||||
const finish = (value: string | null) => {
|
||||
if (settled) return;
|
||||
settled = true;
|
||||
clearTimeout(timer);
|
||||
resolve(value);
|
||||
};
|
||||
const timer = setTimeout(() => {
|
||||
try {
|
||||
child.kill("SIGKILL");
|
||||
} catch {
|
||||
/* already gone */
|
||||
}
|
||||
finish(null);
|
||||
}, feedPrepareTimeoutMs);
|
||||
child.stdout.on("data", (chunk) => {
|
||||
out += String(chunk);
|
||||
});
|
||||
child.stderr.on("data", () => {
|
||||
/* the script keeps its own log */
|
||||
});
|
||||
child.on("error", () => finish(null));
|
||||
child.on("exit", (code) => {
|
||||
if (code !== 0) return finish(null);
|
||||
const match = out.match(/^MINE_SOURCE=(.+)$/m);
|
||||
finish(match ? match[1].trim() : null);
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Stage + mine this container's transcripts. Never throws, and coalesces:
|
||||
* an overlapping trigger joins the in-flight run instead of racing it.
|
||||
*/
|
||||
function feedPalace(reason: string): Promise<void> {
|
||||
if (!feedEnabled || !available) return Promise.resolve();
|
||||
if (feedInFlight) return feedInFlight;
|
||||
const run = (async () => {
|
||||
try {
|
||||
const source = await prepareFeed(reason);
|
||||
if (!source) return;
|
||||
await Promise.race([
|
||||
client.callTool("mempalace_mine", {
|
||||
source,
|
||||
mode: "convos",
|
||||
wing: feedWing,
|
||||
agent: agentName,
|
||||
}),
|
||||
new Promise((_resolve, reject) =>
|
||||
setTimeout(
|
||||
() => reject(new Error(`mine timed out after ${feedMineTimeoutMs}ms`)),
|
||||
feedMineTimeoutMs,
|
||||
),
|
||||
),
|
||||
]);
|
||||
lastFeedAt = Date.now();
|
||||
} catch (err) {
|
||||
process.stderr.write(
|
||||
`[mempalace ext] feed (${reason}) failed: ${(err as Error).message}\n`,
|
||||
);
|
||||
}
|
||||
})();
|
||||
feedInFlight = run.finally(() => {
|
||||
feedInFlight = null;
|
||||
});
|
||||
return feedInFlight;
|
||||
}
|
||||
|
||||
// Mid-session feed. A hard container kill runs no handler at all, so this is
|
||||
// what bounds crash loss to one debounce window instead of a whole session.
|
||||
// Re-mining a grown transcript purges and refiles that source_file, so
|
||||
// repeated ticks refresh a session's drawers rather than duplicating them.
|
||||
pi.on("agent_settled", async () => {
|
||||
if (Date.now() - lastFeedAt < feedDebounceMs) return;
|
||||
void feedPalace("tick"); // deliberately not awaited: never stall a turn
|
||||
});
|
||||
|
||||
pi.on("session_shutdown", async () => {
|
||||
// Feed before stopping the client: we are the palace holder, so nothing
|
||||
// else can mine while we live. pi awaits this handler, so the mine really
|
||||
// does complete; feedMineTimeoutMs keeps a wedged palace from hanging exit.
|
||||
await feedPalace("shutdown");
|
||||
client.stop();
|
||||
});
|
||||
|
||||
|
||||
Reference in New Issue
Block a user