joakimp/mempalace-toolkit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feeders: stage beside the palace, not in ~/.cache; document Phase 1 exposure

Browse Source 
Staging default moves out of ~/.cache to <palace-root>/pi-stage (pi) and
<palace-root>/opencode-stage (opencode), resolved with mempalace's own
palace-path precedence ($MEMPALACE_PALACE_PATH -> $MEMPAL_PALACE_PATH ->
~/.mempalace/config.json -> ~/.mempalace/palace), then dirname.

Why: the convos miner keys dedup on the *staged* path, so a wiped stage plus a
sync scoped to include it prunes the drawers mined from those sources --
deleting memories, not a cache. Under ~/.cache that state was reachable by
anything treating a cache as disposable. Staging inside the palace makes the
coupling structural: the stage cannot be wiped without touching the palace
itself. Overrides ($MEMPALACE_PI_STAGE / $MEMPALACE_SESSION_STAGE, --stage) are
unchanged. Note the old default had never been created on any host, so this
closed a latent hazard, not a live one.

Measured, and the docs now claim only this much: sync prunes only within the
scope it is given -- wing-only, 1299 scanned / 1299 out of scope / 0 removed;
scoped at the palace root, 651 kept / 648 out of scope. The previous blanket
"sync prunes every drawer" wording overstated it, which is a liability: the next
reader disproves the overstatement and discards the real constraint with it.

Also in this change:
- cron log dir ~/.cache/mempalace-session -> ~/.cache/mempalace-logs. The stage
  left that namespace, so the old name now read as "the stage".
- AGENTS.md: the convos miner *does* check mtime (verified against upstream
  convo_miner.py); the previous "no mtime check" claim was wrong.
- smoke-test assertions use `mktemp -d` for --sessions-dir. One pointed at /tmp,
  which still held earlier synthetic transcripts, so a --dry-run exported a fake
  session into the real stage: --dry-run skips the mine, not the export.

docs/phase-1-exposure-runbook.md -- the newt/DNS/auth step that RFC 001 and the
synlig runbook leave open (runbook section 4, items 2 and 5). Port 8765 at /mcp,
newt targets 172.17.0.1, and the authentication is the single shared bearer
token (RFC 6.2, decided 2026-08-09) rather than per-device proxy users. The
latter cannot work today: mempalace validates exactly one token, and Pangolin's
SSO/PIN/password are browser-shaped while every client here is a headless
JSON-RPC POST -- enabling that protection breaks the clients it protects. The
per-device axis that *does* exist is the feeder's SSH key + per-device inbox.

New finding recorded there: a loopback bind does not merely 403 behind a tunnel
(already known, runbook 2.4) -- it also silently starts the server with no token
at all, because auto-minting is gated on the bind being non-loopback.

extensions/pi/README.md: the HTTP transport IS authenticated as of mempalace
3.6.0; the "sessionless and unauthenticated" note dated from the v1.3.0 era.
Closes the RFC section 8 Phase-0 hygiene item.
This commit is contained in:
Joakim Persson
2026-08-12 17:04:01 +02:00
parent 3626946013
commit 29e660e18f
15 changed files with 1019 additions and 66 deletions
Download Patch File Download Diff File Expand all files Collapse all files
extensions/pi/README.md
+88 -9
View File
@@ -16,6 +16,7 @@ dependencies (~300 MB).
**Jump to:**
- [What it does](#what-it-does)
- [Transport: local vs external](#transport-local-vs-external)
- [Automatic transcript feeding](#automatic-transcript-feeding)
- [The `Type.Unsafe` gotcha](#the-typeunsafe-gotcha)
- [Deploying pi with mempalace on a new machine](#deploying-pi-with-mempalace-on-a-new-machine)
- [Fail-soft, identity, debugging](#fail-soft)
@@ -37,11 +38,75 @@ dependencies (~300 MB).
injects the result as a `mempalace-wakeup` system message so the
agent orients itself the way `~/.agents/skills/mempalace/SKILL.md`
describes. Skipped on resume/fork (context is already in the thread).
4. **Manual wind-down** via a `/mempalace-diary [topic]` slash command:
4. **Automatic transcript feeding** (`session_shutdown`, and a debounced
`agent_settled`): stages + mines this pi installation's own session
transcripts into the palace with no user action needed. Unlike the diary
below, this needs no LLM turn — it's a subprocess + a tool call — so it
*can* run on `session_shutdown` where the diary cannot. See
[Automatic transcript feeding](#automatic-transcript-feeding).
5. **Manual wind-down** via a `/mempalace-diary [topic]` slash command:
sends a prompt asking the LLM to call `mempalace_diary_write` with
an AAAK-formatted entry summarizing the session. Not fully auto
because pi sessions are typically short/tactical and
`session_shutdown` fires too late to drive another LLM turn.
an AAAK-formatted entry summarizing the session. This one stays manual
because it needs the LLM to compose the entry, and `session_shutdown`
fires too late to drive another LLM turn — a constraint that applies to
the diary specifically, not to feeding (see above).
## Automatic transcript feeding
The bridge feeds this pi installation's own session transcripts into the
palace by itself — no scheduler, no cron, no manual invocation. It fires on
`session_shutdown` (covers quit, `/new`, `/resume`, `/fork`) and on a
debounced `agent_settled` (covers a long session that later crashes, since a
hard kill runs no shutdown handler at all).
The work is split across two processes, and the reason is a hard constraint,
not a style choice: **the palace is single-writer.** A live pi session
always holds it through this extension's own `mempalace-mcp` subprocess, so
an unattended `mempalace mine` from anywhere else fails outright with
`palace ... is held by PID <n>`. The bridge therefore:
1. Runs `mempalace-pi-session --prepare --reason <trigger> --wing <wing>` as a
subprocess. This does every palace-free step — parse pi's JSONL, apply
the quality threshold, stage the export, and (remote mode only) `rsync`
it to the palace host — and prints one line, `MINE_SOURCE=<path>`,
without ever touching the palace.
2. Calls the `mempalace_mine` MCP tool **through this extension's own
client** on that path. Going through the client that already holds the
lock is the only way to write during a live session, and it automatically
targets whichever palace the bridge is pointed at — local stdio or a
shared remote one.
`mempalace-pi-session` (in this repo's `bin/`) is the actual exporter and
owns the quality gate, the remote transport, and every flag — see its
`--help` for the full reference; this section only covers the extension's
side of the wiring.
**Env knobs (extension side):**
| Var | Default | Effect |
|---|---|---|
| `MEMPALACE_FEED` | `1` | Set `0` to disable automatic feeding entirely. |
| `MEMPALACE_FEED_BIN` | `mempalace-pi-session` | Helper to run. |
| `MEMPALACE_FEED_WING` | `wing_conversations` | Target wing — passed to both the exporter and the `mempalace_mine` call. |
| `MEMPALACE_FEED_DEBOUNCE_MS` | `600000` (10 min) | Minimum gap between mid-session (`agent_settled`) feeds. Bounds crash loss to one window instead of a whole session. |
| `MEMPALACE_FEED_PREPARE_TIMEOUT_MS` | `120000` | Kills a wedged `--prepare` subprocess. |
| `MEMPALACE_FEED_MINE_TIMEOUT_MS` | `30000` | Caps the `mempalace_mine` call so a stalled palace can't hang session exit. |
**Remote palace:** if `$MEMPALACE_REMOTE_URL` is set (see
[Transport](#transport-local-vs-external)), `mempalace_mine`'s source path is
expanded on the *server*, which cannot see this machine's transcripts —
that's exactly why step 1 above rsyncs first in that mode. Configure the
inbox with `MEMPALACE_PI_SSH_TARGET` (required for remote feeding — feeding
is silently skipped without it), `MEMPALACE_PI_SSH_CONFIG`, and
`MEMPALACE_PI_REMOTE_PATH`; see `mempalace-pi-session --help`.
**Concurrency:** overlapping triggers coalesce — a `session_shutdown` landing
while a debounced tick is still running joins that in-flight feed instead of
racing it. `mempalace-pi-session` itself also takes a non-blocking `flock`,
so even two independent invocations (e.g. this extension and the
container-start catch-up some devbox images run) never race each other;
losing that race is harmless because the next trigger re-exports from
scratch.
## Transport: local vs external
@@ -57,11 +122,25 @@ chosen at load time:
harnesses/containers (pi + opencode + native). `MEMPALACE_REMOTE_TOKEN`, if
set, is sent as `Authorization: Bearer <token>`.
Serve such an endpoint with `mempalace-mcp --transport http --host 0.0.0.0
--port 8765` (the `pi-devbox` / `opencode-devbox` repos ship a
`docker-compose.mempalace.yml` for exactly this). Note: that HTTP transport is
currently sessionless and **unauthenticated** — keep it on a trusted network
or behind a reverse proxy that enforces the bearer token.
Serve such an endpoint with `mempalace serve --host 172.17.0.1 --port 8765`
(the `pi-devbox` / `opencode-devbox` repos ship a
`docker-compose.mempalace.yml` for exactly this).
**The HTTP transport is authenticated as of mempalace 3.6.0** — earlier docs
here said otherwise, from the v1.3.0 era. `serve` mints a bearer token, keeps
it 0600, passes it via the environment (never argv), compares it with
`hmac.compare_digest`, and **refuses to bind a non-loopback host without one**
unless `--allow-insecure`. It also pins `Host` and allowlists `Origin`
(anti-DNS-rebinding), and can terminate TLS itself.
Two binds to avoid. `0.0.0.0` publishes the palace to the whole LAN. And
`127.0.0.1` is the trap that looks safe: the Host pin is enforced *only* on
loopback binds, so behind a tunnel every proxied request 403s — and
token auto-minting is gated on the bind being non-loopback, so it starts with
**no authentication at all**, no warning. Bind the docker0 gateway
(`172.17.0.1`): reachable from the host and its containers, not from the LAN.
See
[`docs/phase-1-exposure-runbook.md`](../../docs/phase-1-exposure-runbook.md).
Implementation note: the HTTP client (`RemoteMcpClient`) is **vendored** from
[`pi-extensions`](https://gitea.jordbo.se/joakimp/pi-extensions)'
extensions/pi/mempalace.ts
+120
View File
@@ -21,6 +21,21 @@
* `mempalace_status` + `mempalace_diary_read` output as context so the
* agent orients itself the way the mempalace skill describes. Skipped
* on resume/fork (palace context is already in the thread).
* - Feeding (auto): stage + mine this container's pi transcripts into the
* palace on `session_shutdown` and on a debounced `agent_settled`. Needs
* no LLM turn (pi transcripts are JSONL on disk), which is why it CAN be
* automatic where the diary cannot. The file-side work is delegated to
* `mempalace-pi-session --prepare` (export + threshold + staging, plus the
* rsync to the palace host when the palace is remote); the mine itself
* must run through THIS client, because the palace is single-writer and
* this process is the holder — a CLI `mempalace mine` during a live
* session dies with "palace ... is held by PID <ours>". Going through the
* client also means it automatically targets whichever palace this bridge
* is pointed at (local stdio or a shared remote one).
* - MEMPALACE_FEED=0 disable feeding entirely
* - MEMPALACE_FEED_BIN helper to run (default mempalace-pi-session)
* - MEMPALACE_FEED_WING target wing (default wing_conversations)
* - MEMPALACE_FEED_DEBOUNCE_MS min gap between mid-session feeds (default 600000)
* - Wind-down (manual): `/mempalace-diary` command prompts the LLM to
* write an AAAK-formatted diary entry. Not fully auto because pi
* sessions are typically short/tactical and session_shutdown is too
@@ -724,7 +739,112 @@ export default async function mempalaceExtension(pi: ExtensionAPI) {
});
}
// --- Automatic transcript feeding ---
//
// Split deliberately: `mempalace-pi-session --prepare` does the palace-free
// file work (export + quality threshold + staging, plus the rsync to the
// palace host in remote mode) and prints the path to mine; we then mine it
// through this client. See the header note on single-writer contention.
const feedEnabled = (process.env.MEMPALACE_FEED ?? "1") !== "0";
const feedBin = process.env.MEMPALACE_FEED_BIN || "mempalace-pi-session";
const feedWing = process.env.MEMPALACE_FEED_WING ?? "wing_conversations";
const feedDebounceMs = num(process.env.MEMPALACE_FEED_DEBOUNCE_MS, 600_000);
const feedPrepareTimeoutMs = num(process.env.MEMPALACE_FEED_PREPARE_TIMEOUT_MS, 120_000);
const feedMineTimeoutMs = num(process.env.MEMPALACE_FEED_MINE_TIMEOUT_MS, 30_000);
let lastFeedAt = 0; // 0 => the first settled turn also acts as a catch-up
let feedInFlight: Promise<void> | null = null;
/** Run `mempalace-feed --prepare`; resolve the path to mine, or null. */
function prepareFeed(reason: string): Promise<string | null> {
return new Promise((resolve) => {
// A missing helper surfaces as an async 'error' event (ENOENT), not a
// throw, so the handler below is the fail-soft path.
const child = spawn(feedBin, ["--prepare", "--reason", reason, "--wing", feedWing], {
stdio: ["ignore", "pipe", "pipe"],
});
let out = "";
let settled = false;
const finish = (value: string | null) => {
if (settled) return;
settled = true;
clearTimeout(timer);
resolve(value);
};
const timer = setTimeout(() => {
try {
child.kill("SIGKILL");
} catch {
/* already gone */
}
finish(null);
}, feedPrepareTimeoutMs);
child.stdout.on("data", (chunk) => {
out += String(chunk);
});
child.stderr.on("data", () => {
/* the script keeps its own log */
});
child.on("error", () => finish(null));
child.on("exit", (code) => {
if (code !== 0) return finish(null);
const match = out.match(/^MINE_SOURCE=(.+)$/m);
finish(match ? match[1].trim() : null);
});
});
}
/**
* Stage + mine this container's transcripts. Never throws, and coalesces:
* an overlapping trigger joins the in-flight run instead of racing it.
*/
function feedPalace(reason: string): Promise<void> {
if (!feedEnabled || !available) return Promise.resolve();
if (feedInFlight) return feedInFlight;
const run = (async () => {
try {
const source = await prepareFeed(reason);
if (!source) return;
await Promise.race([
client.callTool("mempalace_mine", {
source,
mode: "convos",
wing: feedWing,
agent: agentName,
}),
new Promise((_resolve, reject) =>
setTimeout(
() => reject(new Error(`mine timed out after ${feedMineTimeoutMs}ms`)),
feedMineTimeoutMs,
),
),
]);
lastFeedAt = Date.now();
} catch (err) {
process.stderr.write(
`[mempalace ext] feed (${reason}) failed: ${(err as Error).message}\n`,
);
}
})();
feedInFlight = run.finally(() => {
feedInFlight = null;
});
return feedInFlight;
}
// Mid-session feed. A hard container kill runs no handler at all, so this is
// what bounds crash loss to one debounce window instead of a whole session.
// Re-mining a grown transcript purges and refiles that source_file, so
// repeated ticks refresh a session's drawers rather than duplicating them.
pi.on("agent_settled", async () => {
if (Date.now() - lastFeedAt < feedDebounceMs) return;
void feedPalace("tick"); // deliberately not awaited: never stall a turn
});
pi.on("session_shutdown", async () => {
// Feed before stopping the client: we are the palace holder, so nothing
// else can mine while we live. pi awaits this handler, so the mine really
// does complete; feedMineTimeoutMs keeps a wedged palace from hanging exit.
await feedPalace("shutdown");
client.stop();
});