feeders: stage beside the palace, not in ~/.cache; document Phase 1 exposure

Staging default moves out of ~/.cache to <palace-root>/pi-stage (pi) and
<palace-root>/opencode-stage (opencode), resolved with mempalace's own
palace-path precedence ($MEMPALACE_PALACE_PATH -> $MEMPAL_PALACE_PATH ->
~/.mempalace/config.json -> ~/.mempalace/palace), then dirname.

Why: the convos miner keys dedup on the *staged* path, so a wiped stage plus a
sync scoped to include it prunes the drawers mined from those sources --
deleting memories, not a cache. Under ~/.cache that state was reachable by
anything treating a cache as disposable. Staging inside the palace makes the
coupling structural: the stage cannot be wiped without touching the palace
itself. Overrides ($MEMPALACE_PI_STAGE / $MEMPALACE_SESSION_STAGE, --stage) are
unchanged. Note the old default had never been created on any host, so this
closed a latent hazard, not a live one.

Measured, and the docs now claim only this much: sync prunes only within the
scope it is given -- wing-only, 1299 scanned / 1299 out of scope / 0 removed;
scoped at the palace root, 651 kept / 648 out of scope. The previous blanket
"sync prunes every drawer" wording overstated it, which is a liability: the next
reader disproves the overstatement and discards the real constraint with it.

Also in this change:
- cron log dir ~/.cache/mempalace-session -> ~/.cache/mempalace-logs. The stage
  left that namespace, so the old name now read as "the stage".
- AGENTS.md: the convos miner *does* check mtime (verified against upstream
  convo_miner.py); the previous "no mtime check" claim was wrong.
- smoke-test assertions use `mktemp -d` for --sessions-dir. One pointed at /tmp,
  which still held earlier synthetic transcripts, so a --dry-run exported a fake
  session into the real stage: --dry-run skips the mine, not the export.

docs/phase-1-exposure-runbook.md -- the newt/DNS/auth step that RFC 001 and the
synlig runbook leave open (runbook section 4, items 2 and 5). Port 8765 at /mcp,
newt targets 172.17.0.1, and the authentication is the single shared bearer
token (RFC 6.2, decided 2026-08-09) rather than per-device proxy users. The
latter cannot work today: mempalace validates exactly one token, and Pangolin's
SSO/PIN/password are browser-shaped while every client here is a headless
JSON-RPC POST -- enabling that protection breaks the clients it protects. The
per-device axis that *does* exist is the feeder's SSH key + per-device inbox.

New finding recorded there: a loopback bind does not merely 403 behind a tunnel
(already known, runbook 2.4) -- it also silently starts the server with no token
at all, because auto-minting is gated on the bind being non-loopback.

extensions/pi/README.md: the HTTP transport IS authenticated as of mempalace
3.6.0; the "sessionless and unauthenticated" note dated from the v1.3.0 era.
Closes the RFC section 8 Phase-0 hygiene item.
This commit is contained in:
Joakim Persson
2026-08-12 17:04:01 +02:00
parent 3626946013
commit 29e660e18f
15 changed files with 1019 additions and 66 deletions
+53 -7
View File
@@ -10,13 +10,26 @@
# Strategy:
# 1. Read opencode.db and export each qualifying session to a Claude Code
# JSONL file (format the mempalace normalizer already understands).
# 2. Stage exports under ~/.cache/mempalace-session/<wing>/.
# 2. Stage exports under <palace-root>/opencode-stage/<wing>/ (override with
# MEMPALACE_SESSION_STAGE).
# 3. Run `mempalace mine --mode convos` against the staging dir.
#
# Dedup: mempalace convos mode keys on source_file (absolute staging path).
# The staging path is deterministic (per-wing under XDG_CACHE_HOME) so re-runs
# The staging path is deterministic (per-wing under the palace root) so re-runs
# are idempotent as long as session content hasn't changed.
#
# Staging location: because dedup keys on the staged path, wiping the stage
# leaves the palace holding drawers whose source files look deleted, and
# `mempalace sync` prunes exactly those when they fall inside the scope it is
# given. The stage therefore lives next to the palace it feeds (resolved via
# $MEMPALACE_PALACE_PATH → $MEMPAL_PALACE_PATH → ~/.mempalace/config.json →
# ~/.mempalace/palace), so stage and dedup keys share one lifetime and the
# dangerous state — palace survives, stage does not — cannot be reached by
# wiping something that merely looks disposable. It used to default under
# ~/.cache, which is disposable on exactly the hosts where this runs
# unattended. Override with MEMPALACE_SESSION_STAGE only if the target is at
# least as durable as the palace.
#
# Session filter: sessions with fewer than --min-messages messages (default 3)
# are skipped to avoid filing throwaway /exit'd sessions.
#
@@ -87,7 +100,7 @@ Idempotency:
What gets mined:
- Each qualifying session → one Claude Code JSONL file
- Staged under ~/.cache/mempalace-session/<wing>/
- Staged under <palace-root>/opencode-stage/<wing>/
- Filed via `mempalace mine --mode convos`
Transcript shape per session:
@@ -103,7 +116,9 @@ Transcript shape per session:
Dedup:
- source_file = absolute staging path (deterministic per session ID)
- Re-runs skip unchanged sessions. To force re-mining, delete the staging
dir: rm -rf ~/.cache/mempalace-session/<wing>/
dir: rm -rf <palace-root>/opencode-stage/<wing>/
(that only forces a refile; do NOT run `mempalace sync` while the stage is
missing, or the drawers mined from it get pruned instead)
Rationale:
Opencode lacks a session-stopping hook (upstream PRs #16598, #16769 still
@@ -147,8 +162,31 @@ if ! [[ "$MIN_MESSAGES" =~ ^[0-9]+$ ]]; then
fi
# ── Staging dir ──────────────────────────────────────────────────────
# Deterministic per-wing path so source_file dedup works across re-runs.
CACHE_ROOT="${XDG_CACHE_HOME:-$HOME/.cache}/mempalace-session"
# Deterministic per-wing path so source_file dedup works across re-runs, and
# anchored to the palace root so the stage cannot be wiped independently of the
# dedup keys that reference it (see "Staging location" in the header).
# Mirrors mempalace config.py:palace_path() precedence, then takes the parent.
palace_root() {
python3 - <<'PY' 2>/dev/null || echo "$HOME/.mempalace"
import json, os
p = os.environ.get("MEMPALACE_PALACE_PATH") or os.environ.get("MEMPAL_PALACE_PATH")
if p:
p = os.path.abspath(os.path.expanduser(p))
else:
cfg = os.path.expanduser("~/.mempalace/config.json")
p = None
if os.path.exists(cfg):
try:
with open(cfg) as fh:
v = json.load(fh).get("palace_path")
p = os.path.expanduser(v) if v else None
except Exception:
p = None
p = p or os.path.expanduser("~/.mempalace/palace")
print(os.path.dirname(p.rstrip("/")))
PY
}
CACHE_ROOT="${MEMPALACE_SESSION_STAGE:-$(palace_root)/opencode-stage}"
STAGE="$CACHE_ROOT/$WING"
mkdir -p "$STAGE"
@@ -185,7 +223,15 @@ if since:
# isn't reachable (first install, moved, permission-denied), we fall through
# to "everything is new" — the mine step will do the real dedup anyway.
already_filed = set()
palace_path = os.environ.get("MEMPALACE_PATH", os.path.expanduser("~/.mempalace/palace"))
# Mirror mempalace's own resolution order (config.py): MEMPALACE_PALACE_PATH,
# then the legacy MEMPAL_PALACE_PATH, then the default. NOT "MEMPALACE_PATH" —
# that name is not a mempalace concept, and reading it silently degraded this
# NEW/SKIP preview to "everything is new" wherever some other tool had set it.
palace_path = (
os.environ.get("MEMPALACE_PALACE_PATH")
or os.environ.get("MEMPAL_PALACE_PATH")
or os.path.expanduser("~/.mempalace/palace")
)
chroma_db = Path(palace_path) / "chroma.sqlite3"
if chroma_db.is_file():
try: