feeders: stage beside the palace, not in ~/.cache; document Phase 1 exposure
Staging default moves out of ~/.cache to <palace-root>/pi-stage (pi) and <palace-root>/opencode-stage (opencode), resolved with mempalace's own palace-path precedence ($MEMPALACE_PALACE_PATH -> $MEMPAL_PALACE_PATH -> ~/.mempalace/config.json -> ~/.mempalace/palace), then dirname. Why: the convos miner keys dedup on the *staged* path, so a wiped stage plus a sync scoped to include it prunes the drawers mined from those sources -- deleting memories, not a cache. Under ~/.cache that state was reachable by anything treating a cache as disposable. Staging inside the palace makes the coupling structural: the stage cannot be wiped without touching the palace itself. Overrides ($MEMPALACE_PI_STAGE / $MEMPALACE_SESSION_STAGE, --stage) are unchanged. Note the old default had never been created on any host, so this closed a latent hazard, not a live one. Measured, and the docs now claim only this much: sync prunes only within the scope it is given -- wing-only, 1299 scanned / 1299 out of scope / 0 removed; scoped at the palace root, 651 kept / 648 out of scope. The previous blanket "sync prunes every drawer" wording overstated it, which is a liability: the next reader disproves the overstatement and discards the real constraint with it. Also in this change: - cron log dir ~/.cache/mempalace-session -> ~/.cache/mempalace-logs. The stage left that namespace, so the old name now read as "the stage". - AGENTS.md: the convos miner *does* check mtime (verified against upstream convo_miner.py); the previous "no mtime check" claim was wrong. - smoke-test assertions use `mktemp -d` for --sessions-dir. One pointed at /tmp, which still held earlier synthetic transcripts, so a --dry-run exported a fake session into the real stage: --dry-run skips the mine, not the export. docs/phase-1-exposure-runbook.md -- the newt/DNS/auth step that RFC 001 and the synlig runbook leave open (runbook section 4, items 2 and 5). Port 8765 at /mcp, newt targets 172.17.0.1, and the authentication is the single shared bearer token (RFC 6.2, decided 2026-08-09) rather than per-device proxy users. The latter cannot work today: mempalace validates exactly one token, and Pangolin's SSO/PIN/password are browser-shaped while every client here is a headless JSON-RPC POST -- enabling that protection breaks the clients it protects. The per-device axis that *does* exist is the feeder's SSH key + per-device inbox. New finding recorded there: a loopback bind does not merely 403 behind a tunnel (already known, runbook 2.4) -- it also silently starts the server with no token at all, because auto-minting is gated on the bind being non-loopback. extensions/pi/README.md: the HTTP transport IS authenticated as of mempalace 3.6.0; the "sessionless and unauthenticated" note dated from the v1.3.0 era. Closes the RFC section 8 Phase-0 hygiene item.
This commit is contained in:
+53
-7
@@ -10,13 +10,26 @@
|
||||
# Strategy:
|
||||
# 1. Read opencode.db and export each qualifying session to a Claude Code
|
||||
# JSONL file (format the mempalace normalizer already understands).
|
||||
# 2. Stage exports under ~/.cache/mempalace-session/<wing>/.
|
||||
# 2. Stage exports under <palace-root>/opencode-stage/<wing>/ (override with
|
||||
# MEMPALACE_SESSION_STAGE).
|
||||
# 3. Run `mempalace mine --mode convos` against the staging dir.
|
||||
#
|
||||
# Dedup: mempalace convos mode keys on source_file (absolute staging path).
|
||||
# The staging path is deterministic (per-wing under XDG_CACHE_HOME) so re-runs
|
||||
# The staging path is deterministic (per-wing under the palace root) so re-runs
|
||||
# are idempotent as long as session content hasn't changed.
|
||||
#
|
||||
# Staging location: because dedup keys on the staged path, wiping the stage
|
||||
# leaves the palace holding drawers whose source files look deleted, and
|
||||
# `mempalace sync` prunes exactly those when they fall inside the scope it is
|
||||
# given. The stage therefore lives next to the palace it feeds (resolved via
|
||||
# $MEMPALACE_PALACE_PATH → $MEMPAL_PALACE_PATH → ~/.mempalace/config.json →
|
||||
# ~/.mempalace/palace), so stage and dedup keys share one lifetime and the
|
||||
# dangerous state — palace survives, stage does not — cannot be reached by
|
||||
# wiping something that merely looks disposable. It used to default under
|
||||
# ~/.cache, which is disposable on exactly the hosts where this runs
|
||||
# unattended. Override with MEMPALACE_SESSION_STAGE only if the target is at
|
||||
# least as durable as the palace.
|
||||
#
|
||||
# Session filter: sessions with fewer than --min-messages messages (default 3)
|
||||
# are skipped to avoid filing throwaway /exit'd sessions.
|
||||
#
|
||||
@@ -87,7 +100,7 @@ Idempotency:
|
||||
|
||||
What gets mined:
|
||||
- Each qualifying session → one Claude Code JSONL file
|
||||
- Staged under ~/.cache/mempalace-session/<wing>/
|
||||
- Staged under <palace-root>/opencode-stage/<wing>/
|
||||
- Filed via `mempalace mine --mode convos`
|
||||
|
||||
Transcript shape per session:
|
||||
@@ -103,7 +116,9 @@ Transcript shape per session:
|
||||
Dedup:
|
||||
- source_file = absolute staging path (deterministic per session ID)
|
||||
- Re-runs skip unchanged sessions. To force re-mining, delete the staging
|
||||
dir: rm -rf ~/.cache/mempalace-session/<wing>/
|
||||
dir: rm -rf <palace-root>/opencode-stage/<wing>/
|
||||
(that only forces a refile; do NOT run `mempalace sync` while the stage is
|
||||
missing, or the drawers mined from it get pruned instead)
|
||||
|
||||
Rationale:
|
||||
Opencode lacks a session-stopping hook (upstream PRs #16598, #16769 still
|
||||
@@ -147,8 +162,31 @@ if ! [[ "$MIN_MESSAGES" =~ ^[0-9]+$ ]]; then
|
||||
fi
|
||||
|
||||
# ── Staging dir ──────────────────────────────────────────────────────
|
||||
# Deterministic per-wing path so source_file dedup works across re-runs.
|
||||
CACHE_ROOT="${XDG_CACHE_HOME:-$HOME/.cache}/mempalace-session"
|
||||
# Deterministic per-wing path so source_file dedup works across re-runs, and
|
||||
# anchored to the palace root so the stage cannot be wiped independently of the
|
||||
# dedup keys that reference it (see "Staging location" in the header).
|
||||
# Mirrors mempalace config.py:palace_path() precedence, then takes the parent.
|
||||
palace_root() {
|
||||
python3 - <<'PY' 2>/dev/null || echo "$HOME/.mempalace"
|
||||
import json, os
|
||||
p = os.environ.get("MEMPALACE_PALACE_PATH") or os.environ.get("MEMPAL_PALACE_PATH")
|
||||
if p:
|
||||
p = os.path.abspath(os.path.expanduser(p))
|
||||
else:
|
||||
cfg = os.path.expanduser("~/.mempalace/config.json")
|
||||
p = None
|
||||
if os.path.exists(cfg):
|
||||
try:
|
||||
with open(cfg) as fh:
|
||||
v = json.load(fh).get("palace_path")
|
||||
p = os.path.expanduser(v) if v else None
|
||||
except Exception:
|
||||
p = None
|
||||
p = p or os.path.expanduser("~/.mempalace/palace")
|
||||
print(os.path.dirname(p.rstrip("/")))
|
||||
PY
|
||||
}
|
||||
CACHE_ROOT="${MEMPALACE_SESSION_STAGE:-$(palace_root)/opencode-stage}"
|
||||
STAGE="$CACHE_ROOT/$WING"
|
||||
mkdir -p "$STAGE"
|
||||
|
||||
@@ -185,7 +223,15 @@ if since:
|
||||
# isn't reachable (first install, moved, permission-denied), we fall through
|
||||
# to "everything is new" — the mine step will do the real dedup anyway.
|
||||
already_filed = set()
|
||||
palace_path = os.environ.get("MEMPALACE_PATH", os.path.expanduser("~/.mempalace/palace"))
|
||||
# Mirror mempalace's own resolution order (config.py): MEMPALACE_PALACE_PATH,
|
||||
# then the legacy MEMPAL_PALACE_PATH, then the default. NOT "MEMPALACE_PATH" —
|
||||
# that name is not a mempalace concept, and reading it silently degraded this
|
||||
# NEW/SKIP preview to "everything is new" wherever some other tool had set it.
|
||||
palace_path = (
|
||||
os.environ.get("MEMPALACE_PALACE_PATH")
|
||||
or os.environ.get("MEMPAL_PALACE_PATH")
|
||||
or os.path.expanduser("~/.mempalace/palace")
|
||||
)
|
||||
chroma_db = Path(palace_path) / "chroma.sqlite3"
|
||||
if chroma_db.is_file():
|
||||
try:
|
||||
|
||||
Reference in New Issue
Block a user