feeders: stage beside the palace, not in ~/.cache; document Phase 1 exposure
Staging default moves out of ~/.cache to <palace-root>/pi-stage (pi) and <palace-root>/opencode-stage (opencode), resolved with mempalace's own palace-path precedence ($MEMPALACE_PALACE_PATH -> $MEMPAL_PALACE_PATH -> ~/.mempalace/config.json -> ~/.mempalace/palace), then dirname. Why: the convos miner keys dedup on the *staged* path, so a wiped stage plus a sync scoped to include it prunes the drawers mined from those sources -- deleting memories, not a cache. Under ~/.cache that state was reachable by anything treating a cache as disposable. Staging inside the palace makes the coupling structural: the stage cannot be wiped without touching the palace itself. Overrides ($MEMPALACE_PI_STAGE / $MEMPALACE_SESSION_STAGE, --stage) are unchanged. Note the old default had never been created on any host, so this closed a latent hazard, not a live one. Measured, and the docs now claim only this much: sync prunes only within the scope it is given -- wing-only, 1299 scanned / 1299 out of scope / 0 removed; scoped at the palace root, 651 kept / 648 out of scope. The previous blanket "sync prunes every drawer" wording overstated it, which is a liability: the next reader disproves the overstatement and discards the real constraint with it. Also in this change: - cron log dir ~/.cache/mempalace-session -> ~/.cache/mempalace-logs. The stage left that namespace, so the old name now read as "the stage". - AGENTS.md: the convos miner *does* check mtime (verified against upstream convo_miner.py); the previous "no mtime check" claim was wrong. - smoke-test assertions use `mktemp -d` for --sessions-dir. One pointed at /tmp, which still held earlier synthetic transcripts, so a --dry-run exported a fake session into the real stage: --dry-run skips the mine, not the export. docs/phase-1-exposure-runbook.md -- the newt/DNS/auth step that RFC 001 and the synlig runbook leave open (runbook section 4, items 2 and 5). Port 8765 at /mcp, newt targets 172.17.0.1, and the authentication is the single shared bearer token (RFC 6.2, decided 2026-08-09) rather than per-device proxy users. The latter cannot work today: mempalace validates exactly one token, and Pangolin's SSO/PIN/password are browser-shaped while every client here is a headless JSON-RPC POST -- enabling that protection breaks the clients it protects. The per-device axis that *does* exist is the feeder's SSH key + per-device inbox. New finding recorded there: a loopback bind does not merely 403 behind a tunnel (already known, runbook 2.4) -- it also silently starts the server with no token at all, because auto-minting is gated on the bind being non-loopback. extensions/pi/README.md: the HTTP transport IS authenticated as of mempalace 3.6.0; the "sessionless and unauthenticated" note dated from the v1.3.0 era. Closes the RFC section 8 Phase-0 hygiene item.
This commit is contained in:
Joakim Persson
+308
-28
@@ -10,22 +10,88 @@
|
||||
# Strategy (mirrors mempalace-session):
|
||||
# 1. Walk ~/.pi/agent/sessions/**/*.jsonl and export each qualifying session
|
||||
# to a Claude Code JSONL file (format the mempalace normalizer speaks).
|
||||
# 2. Stage exports under ~/.cache/mempalace-pi-session/<wing>/.
|
||||
# 2. Stage exports under $MEMPALACE_PI_STAGE/<wing> (default
|
||||
# <palace-root>/pi-stage/<wing> — alongside the palace it feeds).
|
||||
# 3. Run `mempalace mine --mode convos` against the staging dir.
|
||||
#
|
||||
# TWO PHASES (--prepare), because the palace is single-writer
|
||||
# mempalace refuses a CLI mine while another process holds the palace:
|
||||
# "palace ... is held by PID <n> (mempalace-mcp); wait for it to finish"
|
||||
# A live pi session ALWAYS has a holder — the mempalace extension's own
|
||||
# mempalace-mcp. So an unattended CLI mine only works when no session is
|
||||
# live (e.g. a container-start catch-up); during a session the mine must be
|
||||
# performed by the process that already holds the palace. Hence:
|
||||
#
|
||||
# --prepare export + stage (+ rsync in remote mode) and print
|
||||
# MINE_SOURCE=<path>, without ever opening the palace.
|
||||
# (default) the above, then mine it ourselves. Contention is treated as
|
||||
# success-with-nothing-to-do, not failure: the holder's own
|
||||
# extension will mine what we staged.
|
||||
#
|
||||
# The pi mempalace extension drives exactly this: it runs --prepare on
|
||||
# session_shutdown and on a debounced agent_settled, then calls
|
||||
# mempalace_mine on MINE_SOURCE through its existing MCP client.
|
||||
#
|
||||
# TRANSPORTS (--mode, default auto)
|
||||
# local Mine into the local palace with the mempalace CLI.
|
||||
# remote $MEMPALACE_REMOTE_URL is set, so the palace lives on another host.
|
||||
# There is no remote-palace CLI — only the HTTP MCP server — and
|
||||
# mempalace_mine expands its source path in the SERVER process, so
|
||||
# the server cannot see this machine's staged exports. We therefore
|
||||
# rsync the stage into a per-device inbox on the palace host and ask
|
||||
# the server to mine its own local path. Requires
|
||||
# MEMPALACE_PI_SSH_TARGET (where to rsync) and
|
||||
# MEMPALACE_PI_REMOTE_PATH (what that inbox is called server-side).
|
||||
#
|
||||
# Labelling: every exported transcript begins with a synthetic header
|
||||
# [session: <title> | <cwd> | <YYYY-MM-DD> | source: pi]
|
||||
# so post-mine search results are self-identifying (pi vs opencode vs other).
|
||||
#
|
||||
# Dedup: mempalace convos mode keys on source_file (absolute staging path).
|
||||
# Staging paths are deterministic per pi session UUID, so re-runs are
|
||||
# idempotent until session content actually changes.
|
||||
# Staging paths are deterministic per pi session UUID, and the export copies
|
||||
# the source session's mtime onto the staged file, so re-runs are idempotent
|
||||
# until session content actually changes. A GROWN session is purged and
|
||||
# refiled for that source_file by the miner, so re-feeding a live session
|
||||
# refreshes its drawers instead of duplicating them.
|
||||
#
|
||||
# Session filter: sessions with fewer than --min-messages *user+assistant*
|
||||
# messages (default 3) are skipped to avoid filing single-prompt throwaways.
|
||||
# Staging location: source_file dedup keys on the staged path, so if the stage
|
||||
# is wiped the palace is left with drawers whose source files look deleted.
|
||||
# `mempalace sync` prunes exactly those — but only within the scope it is
|
||||
# given. Measured on this layout: scoped at the palace root the staged sources
|
||||
# are in scope (kept 651), while a wing-only sync reports them out_of_scope and
|
||||
# leaves them alone. So the data loss is conditional on how sync is invoked,
|
||||
# which is far too thin a margin to rely on.
|
||||
#
|
||||
# The stage therefore defaults NEXT TO THE PALACE (<palace-root>/pi-stage,
|
||||
# resolved the way mempalace itself resolves the palace: $MEMPALACE_PALACE_PATH
|
||||
# → $MEMPAL_PALACE_PATH → ~/.mempalace/config.json → ~/.mempalace/palace).
|
||||
#
|
||||
# That makes the invariant structural rather than documented: the stage and the
|
||||
# dedup keys that reference it share one lifetime, so the dangerous state —
|
||||
# palace survives, stage does not — can no longer be reached by wiping
|
||||
# something that merely looks disposable. A cache dir (the obvious choice, and
|
||||
# the old default) is exactly wrong here: it persists just long enough to look
|
||||
# correct, then takes the memories with it. Override with MEMPALACE_PI_STAGE
|
||||
# only if the target is at least as durable as the palace.
|
||||
#
|
||||
# In remote mode the local stage is only a shipping buffer — dedup lives on the
|
||||
# server, keyed by the server-side inbox path — so its durability is moot there.
|
||||
#
|
||||
# Session filter: two gates, both required.
|
||||
# 1. --min-messages <N> user+assistant turns (default 4). Tool loops inflate
|
||||
# assistant turns fast in pi, so a real working session clears this
|
||||
# easily; a single abandoned prompt does not.
|
||||
# 2. --min-assistant-chars <N> characters of assistant *text* (default 1000),
|
||||
# excluding tool results. Assistant volume, not total volume: pi expands
|
||||
# skills/context into the user prompt, so an abandoned session can carry a
|
||||
# 13k-char "user" message answered with "Ready. What would you like to
|
||||
# work on?" — total size says substantial, assistant size correctly says
|
||||
# nothing happened.
|
||||
#
|
||||
# Usage:
|
||||
# mempalace-pi-session
|
||||
# mempalace-pi-session --prepare
|
||||
# mempalace-pi-session --mode remote
|
||||
# mempalace-pi-session --wing <name>
|
||||
# mempalace-pi-session --session <uuid-prefix>
|
||||
# mempalace-pi-session --since 2026-04-01
|
||||
@@ -34,13 +100,16 @@
|
||||
# mempalace-pi-session --help
|
||||
#
|
||||
# Exit codes:
|
||||
# 0 success
|
||||
# 0 success (including "nothing qualified", "another run holds the lock",
|
||||
# and "palace held by a live session")
|
||||
# 1 usage / argument error
|
||||
# 2 pi sessions dir missing
|
||||
# 3 mempalace CLI not installed
|
||||
# 3 mempalace CLI not installed / rsync missing in remote mode
|
||||
# 4 mine failed
|
||||
# 5 remote transport failed (rsync or HTTP tools/call)
|
||||
#
|
||||
# Dependencies: bash, python3 (stdlib only), mempalace (v3.3.3+)
|
||||
# Dependencies: bash, python3 (stdlib only), mempalace (v3.3.3+);
|
||||
# rsync + ssh in remote mode.
|
||||
|
||||
set -euo pipefail
|
||||
|
||||
@@ -49,11 +118,50 @@ AGENT="${USER:-mempalace}"
|
||||
WING="wing_conversations"
|
||||
SESSION_ID=""
|
||||
SINCE=""
|
||||
MIN_MESSAGES=3
|
||||
MIN_MESSAGES=4
|
||||
MIN_ASSISTANT_CHARS=1000
|
||||
DRY_RUN=0
|
||||
DO_REPAIR=0
|
||||
PREPARE_ONLY=0
|
||||
MODE="auto"
|
||||
REASON=""
|
||||
PI_SESSIONS_DIR="${PI_SESSIONS_DIR:-$HOME/.pi/agent/sessions}"
|
||||
|
||||
# Resolve the palace ROOT (the dir holding palace/, knowledge_graph.sqlite3,
|
||||
# config.json) using mempalace's own precedence, so the stage lands next to
|
||||
# whichever palace this host actually feeds. Mirrors config.py:palace_path()
|
||||
# (env → config.json → default) and takes the parent. Only evaluated when
|
||||
# MEMPALACE_PI_STAGE is unset, so the common path costs nothing.
|
||||
palace_root() {
|
||||
python3 - <<'PY' 2>/dev/null || echo "$HOME/.mempalace"
|
||||
import json, os
|
||||
p = os.environ.get("MEMPALACE_PALACE_PATH") or os.environ.get("MEMPAL_PALACE_PATH")
|
||||
if p:
|
||||
p = os.path.abspath(os.path.expanduser(p))
|
||||
else:
|
||||
cfg = os.path.expanduser("~/.mempalace/config.json")
|
||||
p = None
|
||||
if os.path.exists(cfg):
|
||||
try:
|
||||
with open(cfg) as fh:
|
||||
v = json.load(fh).get("palace_path")
|
||||
p = os.path.expanduser(v) if v else None
|
||||
except Exception:
|
||||
p = None
|
||||
p = p or os.path.expanduser("~/.mempalace/palace")
|
||||
print(os.path.dirname(p.rstrip("/")))
|
||||
PY
|
||||
}
|
||||
STAGE_ROOT="${MEMPALACE_PI_STAGE:-$(palace_root)/pi-stage}"
|
||||
|
||||
# Remote transport (see TRANSPORTS in the header)
|
||||
REMOTE_URL="${MEMPALACE_REMOTE_URL:-}"
|
||||
REMOTE_TOKEN="${MEMPALACE_REMOTE_TOKEN:-}"
|
||||
SSH_TARGET="${MEMPALACE_PI_SSH_TARGET:-}"
|
||||
SSH_CONFIG="${MEMPALACE_PI_SSH_CONFIG:-}"
|
||||
REMOTE_PATH="${MEMPALACE_PI_REMOTE_PATH:-/data/feed}"
|
||||
DEVICE="${MEMPALACE_PI_DEVICE:-$(hostname)}"
|
||||
|
||||
# ── Usage ────────────────────────────────────────────────────────────
|
||||
usage() {
|
||||
cat <<'EOF'
|
||||
@@ -67,10 +175,28 @@ Options:
|
||||
--session <prefix> Export one session only (match on UUID prefix)
|
||||
--since <YYYY-MM-DD> Only sessions last modified on/after this date
|
||||
--min-messages <N> Skip sessions with fewer than N user+assistant
|
||||
turns (default: 3)
|
||||
turns (default: 4)
|
||||
--min-assistant-chars <N>
|
||||
Skip sessions with fewer than N characters of
|
||||
assistant text, tool results excluded (default: 1000).
|
||||
Catches abandoned sessions whose bulk is injected
|
||||
skill/context text in the user prompt.
|
||||
--agent <name> Agent name recorded on drawers (default: $USER)
|
||||
--sessions-dir <path> Path to pi sessions dir (default: $PI_SESSIONS_DIR
|
||||
or ~/.pi/agent/sessions)
|
||||
--stage <path> Staging root (default: $MEMPALACE_PI_STAGE, else
|
||||
<palace-root>/pi-stage — next to the palace, so the
|
||||
stage cannot be wiped independently of the dedup keys
|
||||
that point at it). Exports go in <root>/<wing>.
|
||||
See "Staging location" in the header before moving it.
|
||||
--mode <m> auto|local|remote (default: auto — remote when
|
||||
$MEMPALACE_REMOTE_URL is set)
|
||||
--prepare Export + stage (+ rsync in remote mode), print
|
||||
MINE_SOURCE=<path>, and stop without opening the
|
||||
palace. For callers that will do the mine themselves
|
||||
through a live MCP connection.
|
||||
--reason <label> Label this run in its output (e.g. shutdown, tick,
|
||||
container-start). Useful when triggers log to a file.
|
||||
--dry-run Export + list; do not mine into palace. Each session
|
||||
is tagged [NEW] or [SKIP] based on whether its
|
||||
source_file is already in the palace.
|
||||
@@ -102,14 +228,27 @@ Transcript shape per session:
|
||||
|
||||
Dedup:
|
||||
- source_file = absolute staging path (deterministic per pi session UUID)
|
||||
- Re-runs skip unchanged sessions. To force re-mining, delete the staging
|
||||
dir: rm -rf ~/.cache/mempalace-pi-session/<wing>/
|
||||
- Re-runs skip unchanged sessions; a GROWN session (mtime changed) has its
|
||||
old drawers purged and is refiled, so re-feeding a live session refreshes
|
||||
rather than duplicates.
|
||||
- To force re-mining, delete the staging dir:
|
||||
rm -rf <palace-root>/pi-stage/<wing>/
|
||||
That forces a refile — but do NOT run `mempalace sync` while the stage is
|
||||
missing, or the drawers mined from it get pruned instead.
|
||||
|
||||
Rationale:
|
||||
Pi's extension ecosystem could, in principle, stream sessions into the
|
||||
palace as they happen, but that requires a resident MCP connection during
|
||||
every pi session. This wrapper is the batch, harness-agnostic alternative:
|
||||
it reads the durable on-disk JSONL and mines it on a schedule.
|
||||
Two complementary paths feed the palace from pi, and they cover different
|
||||
failure modes:
|
||||
- The pi mempalace bridge extension (extensions/pi/mempalace.ts) drives
|
||||
this script with --prepare on session_shutdown and on a debounced
|
||||
agent_settled, then mines through its own live MCP connection. That is
|
||||
the primary path: it needs no scheduling and it is the only way to write
|
||||
while a session holds the palace.
|
||||
- Running this script directly is the batch/recovery path: a
|
||||
container-start or host-level catch-up that picks up transcripts nothing
|
||||
mined at the time — notably after a SIGKILL, where no pi handler runs at
|
||||
all. It reads the durable on-disk JSONL, so it does not care whether the
|
||||
session that produced it exited cleanly.
|
||||
EOF
|
||||
}
|
||||
|
||||
@@ -121,6 +260,11 @@ while [[ $# -gt 0 ]]; do
|
||||
--session) SESSION_ID="${2:-}"; shift 2 ;;
|
||||
--since) SINCE="${2:-}"; shift 2 ;;
|
||||
--min-messages) MIN_MESSAGES="${2:-}"; shift 2 ;;
|
||||
--min-assistant-chars) MIN_ASSISTANT_CHARS="${2:-}"; shift 2 ;;
|
||||
--stage) STAGE_ROOT="${2:-}"; shift 2 ;;
|
||||
--mode) MODE="${2:-}"; shift 2 ;;
|
||||
--prepare) PREPARE_ONLY=1; shift ;;
|
||||
--reason) REASON="${2:-}"; shift 2 ;;
|
||||
--agent) AGENT="${2:-}"; shift 2 ;;
|
||||
--sessions-dir) PI_SESSIONS_DIR="${2:-}"; shift 2 ;;
|
||||
--dry-run) DRY_RUN=1; shift ;;
|
||||
@@ -138,32 +282,67 @@ if [[ ! -d "$PI_SESSIONS_DIR" ]]; then
|
||||
echo " override with --sessions-dir <path> or PI_SESSIONS_DIR env var" >&2
|
||||
exit 2
|
||||
fi
|
||||
if ! command -v mempalace >/dev/null 2>&1; then
|
||||
case "$MODE" in
|
||||
auto) if [[ -n "$REMOTE_URL" ]]; then MODE="remote"; else MODE="local"; fi ;;
|
||||
local|remote) ;;
|
||||
*) echo "error: --mode must be auto|local|remote" >&2; exit 1 ;;
|
||||
esac
|
||||
# The mempalace CLI is only needed when WE do the mine. --prepare never opens
|
||||
# the palace, and remote mode talks to the server over HTTP.
|
||||
if [[ $PREPARE_ONLY -eq 0 && "$MODE" == "local" ]] && ! command -v mempalace >/dev/null 2>&1; then
|
||||
echo "error: mempalace CLI not found in PATH" >&2
|
||||
exit 3
|
||||
fi
|
||||
if ! [[ "$MIN_MESSAGES" =~ ^[0-9]+$ ]]; then
|
||||
echo "error: --min-messages must be an integer" >&2
|
||||
exit 1
|
||||
if [[ "$MODE" == "remote" ]]; then
|
||||
command -v rsync >/dev/null 2>&1 || { echo "error: rsync not found (needed for --mode remote)" >&2; exit 3; }
|
||||
if [[ -z "$SSH_TARGET" ]]; then
|
||||
echo "error: MEMPALACE_PI_SSH_TARGET unset (needed for --mode remote)" >&2
|
||||
exit 1
|
||||
fi
|
||||
# The devbox generates a dedicated LAN-jump key/config; prefer it if present.
|
||||
if [[ -z "$SSH_CONFIG" && -f "$HOME/.ssh-local/config" ]]; then
|
||||
SSH_CONFIG="$HOME/.ssh-local/config"
|
||||
fi
|
||||
fi
|
||||
for _n in MIN_MESSAGES MIN_ASSISTANT_CHARS; do
|
||||
if ! [[ "${!_n}" =~ ^[0-9]+$ ]]; then
|
||||
_flag="--$(printf '%s' "${_n,,}" | tr '_' '-')"
|
||||
echo "error: $_flag must be an integer" >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
# ── Staging dir ──────────────────────────────────────────────────────
|
||||
CACHE_ROOT="${XDG_CACHE_HOME:-$HOME/.cache}/mempalace-pi-session"
|
||||
STAGE="$CACHE_ROOT/$WING"
|
||||
# Deterministic per-wing path so source_file dedup works across re-runs. See
|
||||
# "Staging location" in the header for why this should not be disposable.
|
||||
STAGE="${STAGE_ROOT%/}/$WING"
|
||||
mkdir -p "$STAGE"
|
||||
|
||||
[[ -n "$REASON" ]] && echo "mempalace-pi-session [$REASON] mode=$MODE stage=$STAGE"
|
||||
|
||||
# ── Single-writer guard ──────────────────────────────────────────────
|
||||
# Non-blocking: overlapping triggers (a session_shutdown landing on top of a
|
||||
# debounced mid-session run) must not queue or race. Losing a run is harmless
|
||||
# — the next one re-exports from scratch.
|
||||
exec 9>"${STAGE_ROOT%/}/.lock"
|
||||
if command -v flock >/dev/null 2>&1 && ! flock -n 9; then
|
||||
echo "another mempalace-pi-session run holds the lock; skipping"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ── Export sessions (Python heredoc) ────────────────────────────────
|
||||
# Parses pi JSONL files and writes Claude Code JSONL per session into $STAGE.
|
||||
# Also classifies each export as NEW/ALREADY FILED (by source_file lookup)
|
||||
# so --dry-run reports the real mine-set size. Classification is advisory;
|
||||
# `mempalace mine --mode convos` is still the authoritative dedup.
|
||||
export_count=$(python3 - "$PI_SESSIONS_DIR" "$STAGE" "$SESSION_ID" "$SINCE" "$MIN_MESSAGES" <<'PY'
|
||||
export_count=$(python3 - "$PI_SESSIONS_DIR" "$STAGE" "$SESSION_ID" "$SINCE" "$MIN_MESSAGES" "$MIN_ASSISTANT_CHARS" <<'PY'
|
||||
import json, os, sqlite3, sys
|
||||
from datetime import datetime, timezone
|
||||
from pathlib import Path
|
||||
|
||||
sessions_dir, stage, session_filter, since, min_messages = sys.argv[1:6]
|
||||
sessions_dir, stage, session_filter, since, min_messages, min_assistant_chars = sys.argv[1:7]
|
||||
min_messages = int(min_messages)
|
||||
min_assistant_chars = int(min_assistant_chars)
|
||||
stage = Path(stage)
|
||||
sessions_dir = Path(sessions_dir)
|
||||
|
||||
@@ -178,7 +357,15 @@ if since:
|
||||
|
||||
# ── Load palace's already-filed source_files (best-effort, read-only) ──
|
||||
already_filed = set()
|
||||
palace_path = os.environ.get("MEMPALACE_PATH", os.path.expanduser("~/.mempalace/palace"))
|
||||
# Mirror mempalace's own resolution order (config.py): MEMPALACE_PALACE_PATH,
|
||||
# then the legacy MEMPAL_PALACE_PATH, then the default. NOT "MEMPALACE_PATH" —
|
||||
# that name is not a mempalace concept, and reading it silently degraded this
|
||||
# NEW/SKIP preview to "everything is new" wherever some other tool had set it.
|
||||
palace_path = (
|
||||
os.environ.get("MEMPALACE_PALACE_PATH")
|
||||
or os.environ.get("MEMPAL_PALACE_PATH")
|
||||
or os.path.expanduser("~/.mempalace/palace")
|
||||
)
|
||||
chroma_db = Path(palace_path) / "chroma.sqlite3"
|
||||
if chroma_db.is_file():
|
||||
try:
|
||||
@@ -282,6 +469,7 @@ if session_filter:
|
||||
|
||||
exported = 0
|
||||
skipped_short = 0
|
||||
skipped_quiet = 0
|
||||
skipped_malformed = 0
|
||||
skipped_already_filed = 0
|
||||
|
||||
@@ -318,6 +506,7 @@ for path in paths:
|
||||
continue
|
||||
|
||||
title = derive_title(entries, fallback=session_uuid[:8])
|
||||
assistant_chars = 0
|
||||
out_lines = []
|
||||
out_lines.append({
|
||||
"type": "user",
|
||||
@@ -337,6 +526,7 @@ for path in paths:
|
||||
out_lines.append({"type": "user", "message": {"content": text}})
|
||||
elif role == "assistant":
|
||||
text, tool_uses, _ = extract_text(msg.get("content"))
|
||||
assistant_chars += len(text.strip())
|
||||
blocks = []
|
||||
if text.strip():
|
||||
blocks.append({"type": "text", "text": text})
|
||||
@@ -406,6 +596,17 @@ for path in paths:
|
||||
skipped_short += 1
|
||||
continue
|
||||
|
||||
# Assistant *text* volume, tool results excluded: the signal that the
|
||||
# session actually did something, independent of how much injected
|
||||
# skill/context text inflated the user side.
|
||||
if assistant_chars < min_assistant_chars:
|
||||
skipped_quiet += 1
|
||||
print(
|
||||
f" [QUIET] {path.name} ({turn_count} turns, {assistant_chars} assistant chars)",
|
||||
file=sys.stderr,
|
||||
)
|
||||
continue
|
||||
|
||||
out_path = stage / f"pi_{session_uuid}.jsonl"
|
||||
with out_path.open("w", encoding="utf-8") as f:
|
||||
for obj in out_lines:
|
||||
@@ -428,6 +629,8 @@ print(f"EXPORTED {exported}")
|
||||
print(f"ALREADY_FILED {skipped_already_filed}")
|
||||
if skipped_short:
|
||||
print(f"SKIPPED_SHORT {skipped_short}", file=sys.stderr)
|
||||
if skipped_quiet:
|
||||
print(f"SKIPPED_QUIET {skipped_quiet}", file=sys.stderr)
|
||||
if skipped_malformed:
|
||||
print(f"SKIPPED_MALFORMED {skipped_malformed}", file=sys.stderr)
|
||||
PY
|
||||
@@ -461,12 +664,89 @@ if [[ $DRY_RUN -eq 1 ]]; then
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ── Ship to the palace host (remote mode only) ───────────────────────
|
||||
# mempalace_mine expands its source path in the SERVER process, so in remote
|
||||
# mode the exports have to physically exist over there. rsync --update is the
|
||||
# idempotent half; the mine is the other half.
|
||||
MINE_SOURCE="$STAGE"
|
||||
if [[ "$MODE" == "remote" ]]; then
|
||||
ssh_cmd="ssh"
|
||||
[[ -n "$SSH_CONFIG" ]] && ssh_cmd="ssh -F $SSH_CONFIG"
|
||||
echo ""
|
||||
echo "Shipping stage to ${SSH_TARGET%/}/$DEVICE/ ..."
|
||||
if ! rsync -a --update --no-owner --no-group \
|
||||
-e "$ssh_cmd" \
|
||||
--include='*.jsonl' --exclude='*' \
|
||||
"$STAGE/" "${SSH_TARGET%/}/$DEVICE/"; then
|
||||
echo "error: rsync to ${SSH_TARGET%/}/$DEVICE/ failed" >&2
|
||||
exit 5
|
||||
fi
|
||||
MINE_SOURCE="${REMOTE_PATH%/}/$DEVICE"
|
||||
fi
|
||||
|
||||
# ── Phase boundary ───────────────────────────────────────────────────
|
||||
# --prepare hands the source path to the caller (the pi mempalace extension),
|
||||
# which mines it through the MCP client that already holds the palace.
|
||||
if [[ $PREPARE_ONLY -eq 1 ]]; then
|
||||
echo ""
|
||||
printf 'MINE_SOURCE=%s\n' "$MINE_SOURCE"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
# ── Run the mine ─────────────────────────────────────────────────────
|
||||
echo ""
|
||||
echo "Mining into wing '$WING'..."
|
||||
if ! mempalace mine "$STAGE" --mode convos --wing "$WING" --agent "$AGENT"; then
|
||||
echo "error: mempalace mine failed" >&2
|
||||
exit 4
|
||||
if [[ "$MODE" == "remote" ]]; then
|
||||
if ! python3 - "$REMOTE_URL" "$REMOTE_TOKEN" "$MINE_SOURCE" "$WING" "$AGENT" <<'PY'
|
||||
import json, sys, urllib.error, urllib.request
|
||||
|
||||
url, token, source, wing, agent = sys.argv[1:6]
|
||||
payload = json.dumps({
|
||||
"jsonrpc": "2.0",
|
||||
"id": 1,
|
||||
"method": "tools/call",
|
||||
"params": {
|
||||
"name": "mempalace_mine",
|
||||
"arguments": {"source": source, "mode": "convos", "wing": wing, "agent": agent},
|
||||
},
|
||||
}).encode()
|
||||
headers = {"Content-Type": "application/json", "Accept": "application/json"}
|
||||
if token:
|
||||
headers["Authorization"] = f"Bearer {token}"
|
||||
try:
|
||||
with urllib.request.urlopen(
|
||||
urllib.request.Request(url, data=payload, headers=headers), timeout=900
|
||||
) as resp:
|
||||
body = resp.read().decode("utf-8", "replace")
|
||||
except urllib.error.URLError as exc:
|
||||
print(f"error: remote mine transport failed: {exc}", file=sys.stderr)
|
||||
sys.exit(1)
|
||||
print(body[:4000])
|
||||
sys.exit(1 if '"error"' in body else 0)
|
||||
PY
|
||||
then
|
||||
echo "error: remote mine failed" >&2
|
||||
exit 5
|
||||
fi
|
||||
else
|
||||
# Capture output so palace-level contention can be told apart from a real
|
||||
# failure. A live pi session holds the palace through its own mempalace-mcp,
|
||||
# and that session's extension mines what we just staged — so contention
|
||||
# means "already handled", not "broken".
|
||||
set +e
|
||||
mine_out="$(mempalace mine "$MINE_SOURCE" --mode convos --wing "$WING" --agent "$AGENT" 2>&1)"
|
||||
mine_rc=$?
|
||||
set -e
|
||||
printf '%s\n' "$mine_out"
|
||||
if [[ $mine_rc -ne 0 ]]; then
|
||||
if printf '%s' "$mine_out" | grep -q "is held by"; then
|
||||
echo ""
|
||||
echo "palace is held by a live session; it will mine the staged exports itself"
|
||||
exit 0
|
||||
fi
|
||||
echo "error: mempalace mine failed" >&2
|
||||
exit 4
|
||||
fi
|
||||
fi
|
||||
|
||||
# ── Repair index ─────────────────────────────────────────────────────
|
||||
|
||||
+53
-7
@@ -10,13 +10,26 @@
|
||||
# Strategy:
|
||||
# 1. Read opencode.db and export each qualifying session to a Claude Code
|
||||
# JSONL file (format the mempalace normalizer already understands).
|
||||
# 2. Stage exports under ~/.cache/mempalace-session/<wing>/.
|
||||
# 2. Stage exports under <palace-root>/opencode-stage/<wing>/ (override with
|
||||
# MEMPALACE_SESSION_STAGE).
|
||||
# 3. Run `mempalace mine --mode convos` against the staging dir.
|
||||
#
|
||||
# Dedup: mempalace convos mode keys on source_file (absolute staging path).
|
||||
# The staging path is deterministic (per-wing under XDG_CACHE_HOME) so re-runs
|
||||
# The staging path is deterministic (per-wing under the palace root) so re-runs
|
||||
# are idempotent as long as session content hasn't changed.
|
||||
#
|
||||
# Staging location: because dedup keys on the staged path, wiping the stage
|
||||
# leaves the palace holding drawers whose source files look deleted, and
|
||||
# `mempalace sync` prunes exactly those when they fall inside the scope it is
|
||||
# given. The stage therefore lives next to the palace it feeds (resolved via
|
||||
# $MEMPALACE_PALACE_PATH → $MEMPAL_PALACE_PATH → ~/.mempalace/config.json →
|
||||
# ~/.mempalace/palace), so stage and dedup keys share one lifetime and the
|
||||
# dangerous state — palace survives, stage does not — cannot be reached by
|
||||
# wiping something that merely looks disposable. It used to default under
|
||||
# ~/.cache, which is disposable on exactly the hosts where this runs
|
||||
# unattended. Override with MEMPALACE_SESSION_STAGE only if the target is at
|
||||
# least as durable as the palace.
|
||||
#
|
||||
# Session filter: sessions with fewer than --min-messages messages (default 3)
|
||||
# are skipped to avoid filing throwaway /exit'd sessions.
|
||||
#
|
||||
@@ -87,7 +100,7 @@ Idempotency:
|
||||
|
||||
What gets mined:
|
||||
- Each qualifying session → one Claude Code JSONL file
|
||||
- Staged under ~/.cache/mempalace-session/<wing>/
|
||||
- Staged under <palace-root>/opencode-stage/<wing>/
|
||||
- Filed via `mempalace mine --mode convos`
|
||||
|
||||
Transcript shape per session:
|
||||
@@ -103,7 +116,9 @@ Transcript shape per session:
|
||||
Dedup:
|
||||
- source_file = absolute staging path (deterministic per session ID)
|
||||
- Re-runs skip unchanged sessions. To force re-mining, delete the staging
|
||||
dir: rm -rf ~/.cache/mempalace-session/<wing>/
|
||||
dir: rm -rf <palace-root>/opencode-stage/<wing>/
|
||||
(that only forces a refile; do NOT run `mempalace sync` while the stage is
|
||||
missing, or the drawers mined from it get pruned instead)
|
||||
|
||||
Rationale:
|
||||
Opencode lacks a session-stopping hook (upstream PRs #16598, #16769 still
|
||||
@@ -147,8 +162,31 @@ if ! [[ "$MIN_MESSAGES" =~ ^[0-9]+$ ]]; then
|
||||
fi
|
||||
|
||||
# ── Staging dir ──────────────────────────────────────────────────────
|
||||
# Deterministic per-wing path so source_file dedup works across re-runs.
|
||||
CACHE_ROOT="${XDG_CACHE_HOME:-$HOME/.cache}/mempalace-session"
|
||||
# Deterministic per-wing path so source_file dedup works across re-runs, and
|
||||
# anchored to the palace root so the stage cannot be wiped independently of the
|
||||
# dedup keys that reference it (see "Staging location" in the header).
|
||||
# Mirrors mempalace config.py:palace_path() precedence, then takes the parent.
|
||||
palace_root() {
|
||||
python3 - <<'PY' 2>/dev/null || echo "$HOME/.mempalace"
|
||||
import json, os
|
||||
p = os.environ.get("MEMPALACE_PALACE_PATH") or os.environ.get("MEMPAL_PALACE_PATH")
|
||||
if p:
|
||||
p = os.path.abspath(os.path.expanduser(p))
|
||||
else:
|
||||
cfg = os.path.expanduser("~/.mempalace/config.json")
|
||||
p = None
|
||||
if os.path.exists(cfg):
|
||||
try:
|
||||
with open(cfg) as fh:
|
||||
v = json.load(fh).get("palace_path")
|
||||
p = os.path.expanduser(v) if v else None
|
||||
except Exception:
|
||||
p = None
|
||||
p = p or os.path.expanduser("~/.mempalace/palace")
|
||||
print(os.path.dirname(p.rstrip("/")))
|
||||
PY
|
||||
}
|
||||
CACHE_ROOT="${MEMPALACE_SESSION_STAGE:-$(palace_root)/opencode-stage}"
|
||||
STAGE="$CACHE_ROOT/$WING"
|
||||
mkdir -p "$STAGE"
|
||||
|
||||
@@ -185,7 +223,15 @@ if since:
|
||||
# isn't reachable (first install, moved, permission-denied), we fall through
|
||||
# to "everything is new" — the mine step will do the real dedup anyway.
|
||||
already_filed = set()
|
||||
palace_path = os.environ.get("MEMPALACE_PATH", os.path.expanduser("~/.mempalace/palace"))
|
||||
# Mirror mempalace's own resolution order (config.py): MEMPALACE_PALACE_PATH,
|
||||
# then the legacy MEMPAL_PALACE_PATH, then the default. NOT "MEMPALACE_PATH" —
|
||||
# that name is not a mempalace concept, and reading it silently degraded this
|
||||
# NEW/SKIP preview to "everything is new" wherever some other tool had set it.
|
||||
palace_path = (
|
||||
os.environ.get("MEMPALACE_PALACE_PATH")
|
||||
or os.environ.get("MEMPAL_PALACE_PATH")
|
||||
or os.path.expanduser("~/.mempalace/palace")
|
||||
)
|
||||
chroma_db = Path(palace_path) / "chroma.sqlite3"
|
||||
if chroma_db.is_file():
|
||||
try:
|
||||
|
||||
Reference in New Issue
Block a user