docs: backup and recovery, plus units; move host runbook to a private repo
Adds bin/mempalace-backup and docs/backup-and-recovery.md — the mechanism a palace actually needs, none of it site-specific. Why a palace cannot be backed up with cp: it is chroma.sqlite3 (authoritative), knowledge_graph.sqlite3 (usually WAL, so -wal/-shm make a plain copy a same-instant gamble), derived HNSW segment dirs, hallways.json, the embedder descriptor, and a HIDDEN .mempalace/origin.json. Both SQLite files are therefore copied through the online-backup API. Two bugs are documented because both produce a backup that looks fine: "$PALACE"/*/ silently skips the hidden dir, and per-directory rsync collides the identically named data_level0.bin in every HNSW segment. Treating the palace as one tree fixes both and makes a backup a faithful palace IMAGE, so restore is a copy rather than a procedure. Two modes: hot (default, zero downtime, ~4 s, index may lag but SQLite is authoritative and repair --mode from-sqlite rebuilds) and cold (--cold, ~5 s downtime, byte-consistent, restart trapped so a failed run still brings the server back). Verification runs on the COPY — quick_check plus row counts — and the backup is committed by mv only after it passes, with retention pruned only after a verified commit, so a broken new backup cannot delete the last good one. Documented because they are easy to get wrong: the sqlite3 CLI is often absent where the Python module is present; mempalace_embedder.json must be restored with the drawers or search silently degrades; a tested restore means running status AND search against the restored copy, since search is what actually exercises the index; mempalace-serve is a USER unit, so root systemctl reports "not found"; Persistent=true is what makes a missed window run after boot; and installing against the system Python couples the palace's availability to distribution upgrades, with the uv-managed-interpreter fix plus the two PATH traps that bite scripted upgrades. Also moves docs/synlig-primary-runbook.md out to a private fleet repository, leaving a stub that explains the split, since a host inventory is operator data for one deployment rather than part of a public toolkit. The path stays valid so existing links do not break. Remaining host references in the README, RFCs and ARCHITECTURE are left alone deliberately: they are load-bearing prose, contain no secrets, and are best generalised as they are next edited rather than in one churn-heavy pass.
This commit is contained in:
Joakim Persson
@@ -0,0 +1,282 @@
|
||||
#!/usr/bin/env bash
|
||||
# mempalace-backup — consistent, verified backups of a file-backed MemPalace palace.
|
||||
#
|
||||
# WHY THIS EXISTS
|
||||
# The fleet primary (RFC-001) had no backups at all. A palace is not one file:
|
||||
# it is two SQLite databases (one of them in WAL mode), a set of chromadb HNSW
|
||||
# segment directories, and a few JSON sidecars. `cp -r` on a live palace can
|
||||
# capture a torn WAL and an index that disagrees with the sqlite.
|
||||
#
|
||||
# TWO MODES
|
||||
# hot (default) Zero downtime. Both SQLite files are copied with the sqlite
|
||||
# ONLINE BACKUP API (a transactionally consistent snapshot,
|
||||
# WAL included). The HNSW segment dirs are rsynced and may be
|
||||
# slightly skewed relative to the sqlite — which is RECOVERABLE,
|
||||
# because the sqlite is the source of truth and the index can be
|
||||
# rebuilt: `mempalace repair --mode from-sqlite`.
|
||||
# cold (--cold) Stops the server, copies everything byte-for-byte, restarts.
|
||||
# Guaranteed self-consistent including the index. The restart is
|
||||
# trapped so an error still brings the server back.
|
||||
#
|
||||
# BONUS: restoring the hot backup via `repair --mode from-sqlite --archive-existing`
|
||||
# re-CREATES the collections, which is also the only path that applies the modern
|
||||
# chromadb HNSW defaults (batch_size=100 / sync_threshold=1000). Palaces created
|
||||
# under mempalace <=3.6 are stuck on 2/2 and sync the index every 2 records.
|
||||
#
|
||||
# NOTE: synlig has no sqlite3 CLI, so every SQLite operation here goes through
|
||||
# python3's stdlib sqlite3 module. Do not "simplify" this to `VACUUM INTO`.
|
||||
set -euo pipefail
|
||||
|
||||
PALACE="${MEMPALACE_PALACE_PATH:-$HOME/.mempalace/palace}"
|
||||
DEST="${MEMPALACE_BACKUP_DIR:-$HOME/backups/mempalace}"
|
||||
UNIT="mempalace-serve"
|
||||
KEEP=7
|
||||
COLD=0
|
||||
DRY=0
|
||||
VERIFY=""
|
||||
|
||||
usage() {
|
||||
cat <<'USAGE'
|
||||
usage: mempalace-backup [--cold] [--dest DIR] [--keep N] [--palace DIR] [--dry-run]
|
||||
mempalace-backup --verify BACKUP_DIR
|
||||
|
||||
--cold quiesce the server (stop/copy/start) for a byte-consistent copy
|
||||
--dest DIR backup root (default: $HOME/backups/mempalace)
|
||||
--keep N how many backups to retain (default: 7; 0 = keep all)
|
||||
--palace DIR palace to back up (default: $MEMPALACE_PALACE_PATH or ~/.mempalace/palace)
|
||||
--dry-run report what would happen, touch nothing
|
||||
--verify DIR integrity-check an existing backup and print its counts
|
||||
USAGE
|
||||
}
|
||||
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--cold) COLD=1 ;;
|
||||
--dest) DEST="$2"; shift ;;
|
||||
--keep) KEEP="$2"; shift ;;
|
||||
--palace) PALACE="$2"; shift ;;
|
||||
--dry-run) DRY=1 ;;
|
||||
--verify) VERIFY="$2"; shift ;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
*) echo "mempalace-backup: unknown option $1" >&2; usage >&2; exit 2 ;;
|
||||
esac
|
||||
shift
|
||||
done
|
||||
|
||||
log() { printf '%s %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*"; }
|
||||
die() { log "ERROR: $*"; exit 1; }
|
||||
|
||||
# ── shared python helpers ────────────────────────────────────────────────────
|
||||
# check_db: PRAGMA quick_check + row counts, on a copy. Prints JSON.
|
||||
py_check() {
|
||||
python3 - "$1" <<'PY'
|
||||
import json, os, sqlite3, sys
|
||||
path = sys.argv[1]
|
||||
out = {"file": os.path.basename(path), "bytes": os.path.getsize(path)}
|
||||
try:
|
||||
c = sqlite3.connect(f"file:{path}?mode=ro", uri=True)
|
||||
out["quick_check"] = c.execute("PRAGMA quick_check").fetchone()[0]
|
||||
tables = [r[0] for r in c.execute(
|
||||
"SELECT name FROM sqlite_master WHERE type='table'")]
|
||||
out["tables"] = len(tables)
|
||||
counts = {}
|
||||
if "embedding_metadata" in tables:
|
||||
counts["drawers"] = c.execute(
|
||||
"SELECT count(*) FROM embedding_metadata "
|
||||
"WHERE key='chroma:document'").fetchone()[0]
|
||||
for t in ("facts", "triples", "entities"):
|
||||
if t in tables:
|
||||
counts[t] = c.execute(f"SELECT count(*) FROM {t}").fetchone()[0]
|
||||
out["counts"] = counts
|
||||
c.close()
|
||||
except Exception as exc: # noqa: BLE001 — report, never mask
|
||||
out["error"] = repr(exc)
|
||||
print(json.dumps(out))
|
||||
PY
|
||||
}
|
||||
|
||||
# ── --verify: inspect an existing backup, then exit ──────────────────────────
|
||||
if [ -n "$VERIFY" ]; then
|
||||
[ -d "$VERIFY" ] || die "no such backup dir: $VERIFY"
|
||||
log "verifying $VERIFY"
|
||||
rc=0
|
||||
for db in "$VERIFY"/chroma.sqlite3 "$VERIFY"/knowledge_graph.sqlite3; do
|
||||
[ -f "$db" ] || { log " MISSING $(basename "$db")"; rc=1; continue; }
|
||||
res=$(py_check "$db")
|
||||
log " $res"
|
||||
printf '%s' "$res" | grep -q '"quick_check": "ok"' || rc=1
|
||||
done
|
||||
if [ -f "$VERIFY/MANIFEST.json" ]; then
|
||||
python3 - "$VERIFY/MANIFEST.json" <<'PY'
|
||||
import json, sys
|
||||
m = json.load(open(sys.argv[1]))
|
||||
print(" manifest: created={created_utc} mode={mode} bytes={bytes_total} "
|
||||
"mempalace={mempalace_version}".format(**m))
|
||||
print(" index :", m.get("index_consistency", "?"))
|
||||
PY
|
||||
fi
|
||||
[ "$rc" -eq 0 ] && log "VERIFY OK" || log "VERIFY FAILED"
|
||||
exit "$rc"
|
||||
fi
|
||||
|
||||
# ── preflight ───────────────────────────────────────────────────────────────
|
||||
[ -d "$PALACE" ] || die "palace not found: $PALACE"
|
||||
[ -f "$PALACE/chroma.sqlite3" ] || die "no chroma.sqlite3 in $PALACE — wrong dir?"
|
||||
command -v rsync >/dev/null || die "rsync not found"
|
||||
|
||||
STAMP="$(date -u +%Y%m%dT%H%M%SZ)"
|
||||
MODE=$([ "$COLD" -eq 1 ] && echo cold || echo hot)
|
||||
OUT="$DEST/mp-$STAMP-$MODE"
|
||||
TMP="$OUT.incomplete"
|
||||
|
||||
log "palace : $PALACE ($(du -sh "$PALACE" | cut -f1))"
|
||||
log "dest : $OUT"
|
||||
log "mode : $MODE keep: $KEEP"
|
||||
|
||||
if [ "$DRY" -eq 1 ]; then
|
||||
log "dry-run: would create $OUT"
|
||||
log "dry-run: would copy chroma.sqlite3 + knowledge_graph.sqlite3 via sqlite backup API"
|
||||
log "dry-run: would rsync the palace tree ($(find "$PALACE" -mindepth 1 -maxdepth 1 | wc -l) top-level entries, hidden included) minus SQLite"
|
||||
[ "$COLD" -eq 1 ] && log "dry-run: would stop/start $UNIT"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
mkdir -p "$TMP"
|
||||
|
||||
# ── cold mode: quiesce, with a trap so the server always comes back ─────────
|
||||
STOPPED=0
|
||||
restart_if_stopped() {
|
||||
if [ "$STOPPED" -eq 1 ]; then
|
||||
log "restarting $UNIT"
|
||||
systemctl --user start "$UNIT" || log "WARNING: failed to start $UNIT — CHECK MANUALLY"
|
||||
STOPPED=0
|
||||
fi
|
||||
}
|
||||
trap restart_if_stopped EXIT INT TERM
|
||||
|
||||
if [ "$COLD" -eq 1 ]; then
|
||||
log "stopping $UNIT (quiesce)"
|
||||
systemctl --user stop "$UNIT"
|
||||
STOPPED=1
|
||||
# Give an in-flight mine a moment to release the writer lease.
|
||||
sleep 2
|
||||
fi
|
||||
|
||||
# ── the copy ────────────────────────────────────────────────────────────────
|
||||
# Order matters: rsync the tree FIRST, then write the two SQLite files as
|
||||
# consistent snapshots, so the databases are the freshest thing in the backup.
|
||||
#
|
||||
# rsync takes everything EXCEPT the SQLite files (and their -wal/-shm): the HNSW
|
||||
# segment dirs, the JSON sidecars, and hidden dirs such as .mempalace/origin.json
|
||||
# — a `"$PALACE"/*/` glob silently skips those, which is exactly the bug this
|
||||
# structure removes. The payoff: a backup dir IS a palace image, so restore is a
|
||||
# copy rather than a procedure.
|
||||
log "rsyncing palace tree (excluding SQLite)"
|
||||
rsync -a \
|
||||
--exclude 'chroma.sqlite3' --exclude 'chroma.sqlite3-*' \
|
||||
--exclude 'knowledge_graph.sqlite3' --exclude 'knowledge_graph.sqlite3-*' \
|
||||
"$PALACE"/ "$TMP"/
|
||||
|
||||
# SQLite: online backup API. Consistent even mid-write, and WAL-aware — which
|
||||
# matters because knowledge_graph.sqlite3 runs in WAL mode (-wal/-shm present).
|
||||
for db in chroma.sqlite3 knowledge_graph.sqlite3; do
|
||||
[ -f "$PALACE/$db" ] || { log "skip $db (absent)"; continue; }
|
||||
log "sqlite-backup $db"
|
||||
python3 - "$PALACE/$db" "$TMP/$db" <<'PY'
|
||||
import sqlite3, sys
|
||||
src_path, dst_path = sys.argv[1], sys.argv[2]
|
||||
try:
|
||||
src = sqlite3.connect(f"file:{src_path}?mode=ro", uri=True)
|
||||
except sqlite3.OperationalError:
|
||||
src = sqlite3.connect(src_path) # fall back if ro open is refused
|
||||
dst = sqlite3.connect(dst_path)
|
||||
with dst:
|
||||
src.backup(dst) # atomic, consistent snapshot
|
||||
dst.close(); src.close()
|
||||
PY
|
||||
done
|
||||
|
||||
restart_if_stopped
|
||||
trap - EXIT INT TERM
|
||||
|
||||
# ── verify the COPY, not the original ───────────────────────────────────────
|
||||
log "verifying copies"
|
||||
CHROMA_JSON=$(py_check "$TMP/chroma.sqlite3")
|
||||
log " $CHROMA_JSON"
|
||||
printf '%s' "$CHROMA_JSON" | grep -q '"quick_check": "ok"' \
|
||||
|| die "chroma.sqlite3 copy failed quick_check — backup NOT committed"
|
||||
KG_JSON='{}'
|
||||
if [ -f "$TMP/knowledge_graph.sqlite3" ]; then
|
||||
KG_JSON=$(py_check "$TMP/knowledge_graph.sqlite3")
|
||||
log " $KG_JSON"
|
||||
printf '%s' "$KG_JSON" | grep -q '"quick_check": "ok"' \
|
||||
|| die "knowledge_graph.sqlite3 copy failed quick_check — backup NOT committed"
|
||||
fi
|
||||
|
||||
# ── manifest ────────────────────────────────────────────────────────────────
|
||||
MEMPALACE_VER=$("$HOME/.local/bin/mempalace" --version 2>/dev/null | head -1 || echo unknown)
|
||||
python3 - "$TMP/MANIFEST.json" "$STAMP" "$MODE" "$PALACE" "$MEMPALACE_VER" \
|
||||
"$CHROMA_JSON" "$KG_JSON" <<'PY'
|
||||
import json, os, socket, subprocess, sys
|
||||
out, stamp, mode, palace, ver, chroma, kg = sys.argv[1:8]
|
||||
def size(p):
|
||||
t = 0
|
||||
for root, _dirs, files in os.walk(p):
|
||||
for f in files:
|
||||
try: t += os.path.getsize(os.path.join(root, f))
|
||||
except OSError: pass
|
||||
return t
|
||||
man = {
|
||||
"created_utc": stamp,
|
||||
"mode": mode,
|
||||
"host": socket.gethostname(),
|
||||
"palace_path": palace,
|
||||
"mempalace_version": ver.strip(),
|
||||
"bytes_total": size(os.path.dirname(out)),
|
||||
"chroma": json.loads(chroma),
|
||||
"knowledge_graph": json.loads(kg) if kg.strip() != "{}" else None,
|
||||
"index_consistency": (
|
||||
"byte-consistent (server quiesced)" if mode == "cold" else
|
||||
"sqlite is authoritative; HNSW segments may lag a few writes — "
|
||||
"restore with `mempalace repair --mode from-sqlite` if search misbehaves"
|
||||
),
|
||||
"restore": [
|
||||
"# a backup dir IS a palace image — restore is a copy, not a procedure",
|
||||
"systemctl --user stop mempalace-serve",
|
||||
"mv ~/.mempalace/palace ~/.mempalace/palace.broken-$(date -u +%Y%m%dT%H%M%SZ)",
|
||||
"mkdir -p ~/.mempalace/palace",
|
||||
"rsync -a --exclude MANIFEST.json <backup>/ ~/.mempalace/palace/",
|
||||
"systemctl --user start mempalace-serve",
|
||||
"# hot backups only: if search misbehaves or the index count looks stale,",
|
||||
"# rebuild the HNSW index from the authoritative sqlite -- this also",
|
||||
"# re-creates collections with modern HNSW defaults (100/1000):",
|
||||
"mempalace repair --mode from-sqlite --archive-existing --yes",
|
||||
],
|
||||
}
|
||||
with open(out, "w") as fh:
|
||||
json.dump(man, fh, indent=2)
|
||||
print(json.dumps({k: man[k] for k in ("created_utc", "mode", "bytes_total")}))
|
||||
PY
|
||||
|
||||
mv "$TMP" "$OUT"
|
||||
log "committed $OUT ($(du -sh "$OUT" | cut -f1))"
|
||||
|
||||
# ── retention: only ever prune AFTER a committed, verified backup ───────────
|
||||
if [ "$KEEP" -gt 0 ]; then
|
||||
mapfile -t all < <(find "$DEST" -maxdepth 1 -type d -name 'mp-*' | sort)
|
||||
n=${#all[@]}
|
||||
if [ "$n" -gt "$KEEP" ]; then
|
||||
for old in "${all[@]:0:$((n - KEEP))}"; do
|
||||
log "pruning $old"
|
||||
rm -rf "$old"
|
||||
done
|
||||
fi
|
||||
log "retained $(find "$DEST" -maxdepth 1 -type d -name 'mp-*' | wc -l)/$KEEP"
|
||||
fi
|
||||
|
||||
# Leftover .incomplete dirs mean a previous run died — surface, don't hide.
|
||||
for stale in "$DEST"/mp-*.incomplete; do
|
||||
[ -e "$stale" ] && log "NOTE: stale incomplete backup present: $stale"
|
||||
done
|
||||
|
||||
log "done"
|
||||
@@ -37,6 +37,8 @@ Pick **one scheduler** (systemd *or* launchd *or* cron). The opencode and pi job
|
||||
| `cron/mempalace-pi-session.cron` | pi → palace | Tue 03:00 |
|
||||
| `cron/mempalace-session-devbox.cron` | opencode (devbox) → palace | Mon 03:00 |
|
||||
| `systemd/mempalace-serve.service` | **not a mining job** — runs the shared palace *server* | always-on |
|
||||
| `systemd/mempalace-backup.{service,timer}` | **not a mining job** — hot palace backup (zero downtime) | daily 04:00 |
|
||||
| `systemd/mempalace-backup-cold.{service,timer}` | **not a mining job** — cold palace backup (~5 s downtime, byte-consistent) | Sun 04:30 |
|
||||
|
||||
The pi variants are drop-in copies of the opencode variants with script name and schedule updated; the install recipes below apply equally — just swap `mempalace-session` for `mempalace-pi-session` and the schedule day.
|
||||
|
||||
|
||||
@@ -0,0 +1,20 @@
|
||||
[Unit]
|
||||
Description=MemPalace backup (cold — quiesces the server for a byte-consistent copy)
|
||||
Documentation=file:%h/mempalace-toolkit/docs/backup-and-recovery.md
|
||||
ConditionPathExists=%h/.mempalace/palace
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
# Cold mode stops mempalace-serve, copies the palace at rest (so the HNSW index
|
||||
# is byte-consistent with the SQLite, not merely repairable from it), and starts
|
||||
# it again. The restart is trapped on EXIT/INT/TERM inside the script, so an
|
||||
# interrupted or failed run still brings the server back up.
|
||||
#
|
||||
# Measured downtime on a 223 MB palace: ~5 seconds.
|
||||
#
|
||||
# NOTE: mempalace-serve.service ships a drop-in with Restart=always. That is
|
||||
# correct for crash recovery but means an explicit `systemctl --user stop` is the
|
||||
# only way to keep it down for the duration of the copy — do not expect a plain
|
||||
# SIGTERM to hold it.
|
||||
ExecStart=%h/.local/bin/mempalace-backup --cold --keep 7
|
||||
Nice=10
|
||||
@@ -0,0 +1,12 @@
|
||||
[Unit]
|
||||
Description=Weekly cold MemPalace backup
|
||||
Documentation=file:%h/mempalace-toolkit/docs/backup-and-recovery.md
|
||||
|
||||
[Timer]
|
||||
# Half an hour after the daily hot backup, so the two never overlap on the
|
||||
# palace lock even if the hot run is unusually slow.
|
||||
OnCalendar=Sun 04:30:00
|
||||
Persistent=true
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -0,0 +1,19 @@
|
||||
[Unit]
|
||||
Description=MemPalace backup (hot — zero downtime, SQLite online-backup API)
|
||||
Documentation=file:%h/mempalace-toolkit/docs/backup-and-recovery.md
|
||||
# Better a skipped run than a run that creates an empty palace somewhere and
|
||||
# then dutifully backs it up.
|
||||
ConditionPathExists=%h/.mempalace/palace
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
# Hot mode never stops mempalace-serve. Both SQLite files are copied through the
|
||||
# online-backup API (transactionally consistent, WAL-aware); the HNSW index
|
||||
# segments are copied live and may lag a few records. That is acceptable because
|
||||
# SQLite is authoritative — see docs/backup-and-recovery.md §3.
|
||||
ExecStart=%h/.local/bin/mempalace-backup --keep 7
|
||||
# Backups are not urgent; the palace server and any running mine are. Yielding
|
||||
# CPU and I/O keeps a 4-second backup from lengthening someone's interactive
|
||||
# search.
|
||||
Nice=10
|
||||
IOSchedulingClass=idle
|
||||
@@ -0,0 +1,15 @@
|
||||
[Unit]
|
||||
Description=Daily hot MemPalace backup
|
||||
Documentation=file:%h/mempalace-toolkit/docs/backup-and-recovery.md
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*-*-* 04:00:00
|
||||
# If the host was powered off at 04:00, run shortly after the next boot instead
|
||||
# of silently skipping that day. Without this, a laptop or an occasionally-off
|
||||
# server quietly accumulates days with no backup.
|
||||
Persistent=true
|
||||
# Avoid colliding with other maintenance that also picked the top of the hour.
|
||||
RandomizedDelaySec=300
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -0,0 +1,275 @@
|
||||
# Backing up and recovering a palace
|
||||
|
||||
A palace is not one file, so `cp -r` is not a backup. This document explains what
|
||||
a palace is made of, why the obvious approaches silently corrupt it, and how
|
||||
`bin/mempalace-backup` avoids that — plus how to actually restore one and prove
|
||||
the restore worked.
|
||||
|
||||
Host-specific details (which machine is primary, addresses, users, offsite
|
||||
targets) deliberately live outside this repo. Substitute `<PRIMARY_HOST>` and
|
||||
`<USER>` throughout.
|
||||
|
||||
## 1. What is actually in a palace
|
||||
|
||||
Run `ls -a` on a palace directory and you get something like this — sizes are
|
||||
from a real ~25k-drawer palace, for a sense of proportion:
|
||||
|
||||
| Path | Size | What it is |
|
||||
|---|---|---|
|
||||
| `chroma.sqlite3` | 167 MB | **The authoritative store.** Drawers, chunks, metadata, embeddings. |
|
||||
| `<uuid>/` (one per segment) | 50 MB, 2.8 MB | chromadb HNSW index segments (`data_level0.bin`, `index_metadata.pickle`). Derived data. |
|
||||
| `knowledge_graph.sqlite3` | 64 KB | KG triples + entities. **Usually in WAL mode** — see §2. |
|
||||
| `knowledge_graph.sqlite3-wal` / `-shm` | varies | Live write-ahead log. Their existence is the whole problem. |
|
||||
| `hallways.json` | 10.8 MB | Within-wing co-occurrence links, rebuilt by `mempalace hallways`. |
|
||||
| `known_entities.json`, `mempalace_embedder.json` | < 1 KB | Entity list; **which embedder model this palace was built with**. |
|
||||
| `.mempalace/origin.json` | small | Provenance. **Hidden** — see the glob trap in §3. |
|
||||
| `.collection_type_fixed`, `.blob_seq_ids_migrated` | 0 B | Migration marker files. Presence is the signal. |
|
||||
|
||||
Two consequences worth internalising:
|
||||
|
||||
- **The SQLite files are the truth; the index segments are derived.** If the
|
||||
index is stale or damaged you can rebuild it from SQLite. The reverse is not
|
||||
true. This asymmetry is what makes a zero-downtime backup possible at all.
|
||||
- **`mempalace_embedder.json` is not optional.** Restoring drawers next to a
|
||||
*different* embedder model gives you a palace whose stored vectors and future
|
||||
queries live in different spaces — search silently degrades rather than
|
||||
failing. Back it up, and check it after a restore.
|
||||
|
||||
## 2. Why `cp` is unsafe, specifically
|
||||
|
||||
SQLite in WAL mode keeps recent commits in `-wal` and shared state in `-shm`.
|
||||
Copying those three files with `cp` gives you no guarantee they are from the same
|
||||
instant: the copier can read `chroma.sqlite3` before a checkpoint and `-wal`
|
||||
after it. The result usually *opens* — which is the dangerous part — and then
|
||||
misbehaves later, or fails `PRAGMA integrity_check` under load.
|
||||
|
||||
`rsync` has exactly the same problem: it is not a snapshot, it is a sequence of
|
||||
reads.
|
||||
|
||||
The supported approach is SQLite's **online backup API**, which takes a
|
||||
transactionally consistent copy of a live database while writers continue:
|
||||
|
||||
```python
|
||||
import sqlite3
|
||||
src = sqlite3.connect(f"file:{src_path}?mode=ro", uri=True)
|
||||
dst = sqlite3.connect(dst_path)
|
||||
with dst:
|
||||
src.backup(dst) # WAL-aware, consistent, no writer downtime
|
||||
```
|
||||
|
||||
Two practical notes:
|
||||
|
||||
- **`VACUUM INTO` is a fine alternative in principle**, but it needs either the
|
||||
`sqlite3` CLI or a Python build new enough for it. Do not assume the CLI is
|
||||
present: a stock Ubuntu server often has the Python `sqlite3` *module* and no
|
||||
`sqlite3` *binary*. Check with `command -v sqlite3` before writing tooling
|
||||
that depends on it.
|
||||
- Open the source with `mode=ro` so a backup can never be the thing that
|
||||
creates a `-wal` file in a directory you meant only to read.
|
||||
|
||||
## 3. Two backup modes, and the honest tradeoff
|
||||
|
||||
`bin/mempalace-backup` implements both. Neither is strictly better.
|
||||
|
||||
### hot (default) — zero downtime
|
||||
|
||||
1. `rsync` the whole palace tree **excluding** the SQLite files.
|
||||
2. Write both SQLite files via the online-backup API.
|
||||
|
||||
The server never stops. The index segments are copied while they may be being
|
||||
written, so in principle they can lag the SQLite by a few records. That is
|
||||
acceptable *because the SQLite is authoritative*: if search misbehaves after a
|
||||
restore, rebuild the index (§5).
|
||||
|
||||
Measured on a 223 MB palace: **~4 s**.
|
||||
|
||||
### cold (`--cold`) — byte-consistent
|
||||
|
||||
1. Stop the server (`systemctl --user stop mempalace-serve`).
|
||||
2. Copy everything, including the index, at rest.
|
||||
3. Restart the server.
|
||||
|
||||
The restart is in a `trap … EXIT INT TERM`, so an interruption mid-copy still
|
||||
brings the server back. Measured downtime: **~5 s**.
|
||||
|
||||
Recommended schedule: **hot daily, cold weekly**. The cold copy is your
|
||||
byte-exact reference; the hot copies bound your data loss between them.
|
||||
|
||||
### The glob trap that cost two rewrites
|
||||
|
||||
Two bugs are worth naming because both produce a backup that looks fine:
|
||||
|
||||
- **`"$PALACE"/*/` skips hidden directories.** `.mempalace/origin.json` is
|
||||
silently omitted, so the restored palace loses its provenance. Shell globs do
|
||||
not match dotfiles without `dotglob`/`GLOBIGNORE` fiddling.
|
||||
- **Per-directory rsync collides identically named files.** Every HNSW segment
|
||||
contains a `data_level0.bin`. Copying segment dirs into one destination
|
||||
merges them and the last write wins — a corrupt index that still loads.
|
||||
|
||||
The fix for both is to treat the palace as **one tree**: a single
|
||||
`rsync -a --exclude` of the SQLite files, so a backup directory is a faithful
|
||||
*image* of the palace. That makes restore a copy rather than a procedure.
|
||||
|
||||
## 4. Verify the copy, not the original
|
||||
|
||||
A backup you have not read back is a hypothesis. After copying, and **before**
|
||||
committing the backup or pruning old ones, the tool runs against the *copy*:
|
||||
|
||||
- `PRAGMA quick_check` on both databases
|
||||
- row counts (`drawers`, `triples`, `entities`)
|
||||
- a `MANIFEST.json` recording counts, sizes, mode, tool version, and the
|
||||
index-consistency caveat for hot backups
|
||||
|
||||
Sequencing matters as much as the checks:
|
||||
|
||||
1. write into `<dest>.incomplete/`
|
||||
2. verify
|
||||
3. `mv` into place (atomic commit)
|
||||
4. **only now** prune old backups to the retention limit
|
||||
|
||||
Retention pruning that runs before verification can delete your last good
|
||||
backup on the night the new one is broken. A leftover `.incomplete` directory is
|
||||
a visible symptom, not silent data loss, so the tool reports stale ones.
|
||||
|
||||
## 5. Restoring
|
||||
|
||||
Because a backup is a palace image, restore is a copy:
|
||||
|
||||
```bash
|
||||
systemctl --user stop mempalace-serve
|
||||
mv ~/.mempalace/palace ~/.mempalace/palace.broken-$(date -u +%Y%m%dT%H%M%SZ)
|
||||
mkdir -p ~/.mempalace/palace
|
||||
rsync -a --exclude MANIFEST.json <backup-dir>/ ~/.mempalace/palace/
|
||||
systemctl --user start mempalace-serve
|
||||
```
|
||||
|
||||
Keep the old directory until the restore is proven. It costs disk and buys you a
|
||||
second attempt.
|
||||
|
||||
**If the restore came from a hot backup and search behaves oddly**, rebuild the
|
||||
derived index from the authoritative SQLite:
|
||||
|
||||
```bash
|
||||
mempalace repair --mode from-sqlite
|
||||
```
|
||||
|
||||
Note that a full `repair --mode from-sqlite --archive-existing` re-creates the
|
||||
chromadb collections, which means new collections pick up **current** HNSW
|
||||
tuning defaults. On palaces created by older versions this is also the only way
|
||||
to escape legacy `hnsw:batch_size=2 / hnsw:sync_threshold=2` settings, which sync
|
||||
the index every two records. Check yours with:
|
||||
|
||||
```sql
|
||||
SELECT c.name, k.key, k.str_value FROM collections c
|
||||
JOIN collection_metadata k ON k.collection_id = c.id
|
||||
WHERE k.key LIKE 'hnsw%';
|
||||
```
|
||||
|
||||
## 6. Prove the restore, don't assume it
|
||||
|
||||
"The files copied" is not a tested restore. Restore to a throwaway directory and
|
||||
make the palace *answer questions*:
|
||||
|
||||
```bash
|
||||
rsync -a --exclude MANIFEST.json <backup-dir>/ /tmp/restore-test/
|
||||
export MEMPALACE_PALACE_PATH=/tmp/restore-test
|
||||
mempalace status # drawer count — compare to the source
|
||||
mempalace search "some phrase you know is in there"
|
||||
```
|
||||
|
||||
`status` proves the schema and metadata survived. **`search` is the stronger
|
||||
test**: it exercises the HNSW segments, so it is what tells you whether a
|
||||
*hot* backup's index came across usable. On the reference run above it returned
|
||||
correctly ranked results (`cosine_sim=0.398 bm25=2.992`) from a zero-downtime
|
||||
backup — the theoretical index skew did not materialise.
|
||||
|
||||
Do this at least once when you set backups up, and again after any version
|
||||
upgrade that touches storage.
|
||||
|
||||
## 7. Scheduling that survives a reboot
|
||||
|
||||
Use **systemd user timers** with lingering enabled, so backups run without
|
||||
anyone logging in:
|
||||
|
||||
```bash
|
||||
loginctl enable-linger <USER> # required, or user units die at logout
|
||||
systemctl --user enable --now mempalace-backup.timer mempalace-backup-cold.timer
|
||||
systemctl --user list-timers 'mempalace*'
|
||||
```
|
||||
|
||||
Units are in [`contrib/systemd/`](../contrib/systemd/). Two settings earn their
|
||||
keep:
|
||||
|
||||
- **`Persistent=true`** — a timer whose window was missed (host powered off)
|
||||
runs shortly after boot instead of skipping the day.
|
||||
- **`RandomizedDelaySec`** — avoids a thundering herd if other maintenance
|
||||
shares the hour.
|
||||
|
||||
`mempalace-serve` is itself a **user** unit in the reference deployment. That is
|
||||
worth stating explicitly because `systemctl status mempalace-serve` as root
|
||||
reports "not found" and invites the conclusion that the server is not installed.
|
||||
Always `systemctl --user`.
|
||||
|
||||
### The interpreter is part of your durability story
|
||||
|
||||
If the tool is installed against the **system** Python (`/usr/bin/python3`), then
|
||||
a distribution release upgrade — which can remove or replace that interpreter —
|
||||
breaks the server and its backups together, at the moment you least want it.
|
||||
Install against a **managed** interpreter instead:
|
||||
|
||||
```bash
|
||||
uv python install 3.12
|
||||
uv tool install --force --python "$HOME/.local/bin/python3.12" 'mempalace==<version>'
|
||||
systemctl --user restart mempalace-serve
|
||||
```
|
||||
|
||||
Now an OS upgrade cannot take the interpreter away. Two gotchas:
|
||||
|
||||
- `uv` is typically in `~/.local/bin`, which is **not** on a non-interactive
|
||||
ssh PATH. Scripted upgrades must `export PATH="$HOME/.local/bin:$PATH"`.
|
||||
- `uv python list` prints `$HOME`-relative paths, so a regex anchored on a
|
||||
leading `/` matches inside `.local/share` and yields a bogus absolute path.
|
||||
Resolve the shim and assert on it:
|
||||
`readlink -f ~/.local/bin/python3.12 | grep share/uv/python`.
|
||||
|
||||
**Restart the server after any upgrade.** Recent versions refuse mutating tools
|
||||
when the served library drifts from what is installed on disk, and
|
||||
`mempalace_reconnect` cannot clear that — it reopens the database but cannot
|
||||
reload Python modules.
|
||||
|
||||
## 8. Sizing and offsite
|
||||
|
||||
At ~225 MB per copy, 7 hot + 7 cold is ~3 GB — trivial against any modern disk,
|
||||
so prefer generous retention over clever incremental schemes. If space is
|
||||
genuinely tight, `zstd -T0` on the SQLite files compresses well, at the cost of
|
||||
making a backup no longer a directly-mountable palace image.
|
||||
|
||||
**Local backups protect against corruption and mistakes, not against losing the
|
||||
host.** Until a copy leaves the machine, a single disk or a single `rm` ends the
|
||||
palace. Whatever offsite target you choose (object store, NAS, another host),
|
||||
two rules matter more than the choice:
|
||||
|
||||
- **Put a copy of the restore procedure with the backups.** Documentation that
|
||||
lives only inside the thing you are restoring is not available at the moment
|
||||
you need it.
|
||||
- **Verify the offsite copy by restoring from it**, not by checking that the
|
||||
upload exited 0.
|
||||
|
||||
## 9. Usage
|
||||
|
||||
```
|
||||
mempalace-backup [--cold] [--dest DIR] [--keep N] [--palace DIR] [--dry-run]
|
||||
mempalace-backup --verify <backup-dir>
|
||||
```
|
||||
|
||||
| Flag | Meaning |
|
||||
|---|---|
|
||||
| `--cold` | Quiesce the server for a byte-consistent copy (default: hot) |
|
||||
| `--dest DIR` | Backup root (default `~/backups/mempalace`) |
|
||||
| `--keep N` | Retention count, pruned only after a verified commit (default 7) |
|
||||
| `--palace DIR` | Palace to back up (default `~/.mempalace/palace`) |
|
||||
| `--verify DIR` | Re-verify an existing backup and print its manifest |
|
||||
| `--dry-run` | Report what would happen, write nothing |
|
||||
|
||||
Backups are named `mp-<UTC timestamp>-<hot|cold>`, so they sort
|
||||
chronologically and their mode is visible without opening the manifest.
|
||||
+15
-309
@@ -1,315 +1,21 @@
|
||||
# synlig primary — Phase 0 runbook and handoff
|
||||
# (moved) primary-host runbook
|
||||
|
||||
Companion to [`rfc-001-global-palace.md`](./rfc-001-global-palace.md). Records what was actually done
|
||||
on the primary, with verified evidence, so the next session (or the next machine) does not re-derive it.
|
||||
This file used to contain the deployment runbook for one specific primary host —
|
||||
its hostname, addresses, user, service wiring and rollback steps.
|
||||
|
||||
> **Status 2026-08-14 17:00 — SUPERSEDED IN PART. The primary is live, exposed, and seeded.**
|
||||
> Serving since 2026-08-12 at `https://mempalace.jordbo.se/mcp`. Seeded 2026-08-14 15:07 from
|
||||
> EMB-7KJ4VR4G's palace (itself a carry-over from the previous work computer EMB-X1JY06WJ — rfc-001 §4.4)
|
||||
> — 14,777 drawers / 9 wings / 16,337 embeddings / KG 46 entities, 34 triples,
|
||||
> now 14,803 drawers. One client (EMB-7KJ4VR4G's pi-devbox container) is flipped and verified
|
||||
> end-to-end. Both Phase 0 blockers below are cleared.
|
||||
>
|
||||
> **Update 2026-08-16 00:20 — the transcript feed is live too, and it is the primary's third moving
|
||||
> part** (alongside the HTTPS tunnel and the palace itself). See §2.6: transcripts arrive over SSH into
|
||||
> `~/mempalace-feed/<device>/` and are mined *by this host's own server process*. 15,478 drawers as of
|
||||
> that check — but treat every count in this document as a timestamp, not a fact: `status` counts chunk
|
||||
> rows, and §4 item 7 explains why counts adjudicate nothing.
|
||||
>
|
||||
> **Read §3 "Deliberately NOT done" as a record of the 2026-08-10 state, not of today's** — every
|
||||
> item in it has since been done. And before running anything in §5 Rollback, read the warning at the
|
||||
> top of it: `~/.mempalace` on synlig is no longer disposable.
|
||||
**That content now lives in a private repository**, because a host inventory is
|
||||
operator data for one deployment, not part of the toolkit. This repository is
|
||||
public and keeps only host-agnostic *mechanism*.
|
||||
|
||||
Original status, kept for the record:
|
||||
What lives where:
|
||||
|
||||
**Status 2026-08-10 00:30 — Phase 0 prep complete. Not serving. Nothing exposed.**
|
||||
Blocked on two things, both deliberately left to Joakim: the Pangolin update on nyvaken, and one `sudo`.
|
||||
|
||||
---
|
||||
|
||||
## 1. What synlig is (discovered, not assumed)
|
||||
|
||||
| Fact | Value |
|
||||
| Content | Home |
|
||||
|---|---|
|
||||
| SSH | `synlig` → `synlig.erdc.ericsson.net`, user `ecsjper` (from `~/.ssh/config`) |
|
||||
| OS | Ubuntu 24.04.4 LTS, 7.8 GiB RAM, 78 G disk (**29 G free**), uptime 12 d |
|
||||
| Python / uv | system `python3` 3.12.3; `uv` at `~/.local/bin/uv` (**not** on the non-login `PATH`) |
|
||||
| Interfaces | `lo` 127.0.0.1, `ens3` 10.0.0.4/16, `docker0` 172.17.0.1/16, `br-…` 172.19.0.1/16 |
|
||||
| Already listening | 22, 80, 443, 3000 (node), 3389 + 3350 + 4822 (xrdp/guacamole), 631 |
|
||||
| Docker | present; running `act_runner-runner-1` (**Gitea Actions runner**) and `digikam` |
|
||||
| Pre-existing MemPalace | **none** — no `mempalace` binary, no `~/.mempalace`. Greenfield. |
|
||||
| How to expose a palace over HTTP, and the Host/Origin pin | `docs/phase-1-exposure-runbook.md` (here) |
|
||||
| Why a palace needs a special backup, and how to restore one | `docs/backup-and-recovery.md` (here) |
|
||||
| Unit/timer/plist templates | `contrib/` (here) |
|
||||
| Which machine is primary, its addresses, users, tunnels, offsite target | private fleet repository |
|
||||
| Per-host feeder device names and schedules | private fleet repository |
|
||||
|
||||
The Gitea Actions runner living here is worth remembering: synlig is not a dedicated appliance, and CI
|
||||
load competes with the palace for the same 7.8 GiB.
|
||||
|
||||
## 2. Done tonight
|
||||
|
||||
### 2.1 MemPalace installed, pinned to the fleet version
|
||||
|
||||
```sh
|
||||
~/.local/bin/uv tool install "mempalace==3.6.0" # → mempalace, mempalace-mcp
|
||||
```
|
||||
|
||||
Pinned deliberately: the clients run 3.6.0, and the id recipes / idempotency probes this RFC leans on are
|
||||
version-specific. Reversible with `uv tool uninstall mempalace`.
|
||||
|
||||
### 2.2 Embedder model pre-warmed — the corporate-network risk that wasn't
|
||||
|
||||
The first embed pulls `all-MiniLM-L6-v2` ONNX (79.3 MB) from the chroma CDN into
|
||||
`~/.cache/chroma/onnx_models/` (167 M on disk once unpacked). **This was the main unknown** — an
|
||||
egress-filtered work VM would have failed here, at the worst possible moment (first client write).
|
||||
It downloaded at ~20 MB/s with no proxy interference. Done in a throwaway palace, since deleted, so the
|
||||
real palace never saw it. Same model as the clients use, so the semantic space matches.
|
||||
|
||||
### 2.3 Palace created with the §7.1 landmine structurally removed
|
||||
|
||||
`~/.mempalace/palace` — the **stock default**, so no `MEMPALACE_PALACE_PATH` and no `config.json` is
|
||||
needed anywhere on synlig. One less thing to drift.
|
||||
|
||||
RFC §7.1 says to `mv` three HOME-anchored stores into the palace dir before first `serve`. **On a
|
||||
greenfield primary there is nothing to move — but the hazard is not actually a migration hazard, and the
|
||||
RFC understated it:** with *stock defaults* `palace_path` is `~/.mempalace/palace` while `DEFAULT_KG_PATH`
|
||||
is `~/.mempalace/knowledge_graph.sqlite3`. Those differ, so the split is the **out-of-the-box** behaviour,
|
||||
not a consequence of a custom path. It is permanent, not one-time: `serve` always passes `--palace` (KG
|
||||
inside the palace), while any CLI command run *without* `--palace` uses the HOME path. Two KGs on one box,
|
||||
forever, silently.
|
||||
|
||||
Fixed by making both resolution rules land on one inode:
|
||||
|
||||
```sh
|
||||
ln -sfn palace/knowledge_graph.sqlite3 ~/.mempalace/knowledge_graph.sqlite3
|
||||
ln -sfn palace/known_entities.json ~/.mempalace/known_entities.json
|
||||
```
|
||||
|
||||
Relative targets, so a home-directory move survives. `hallways.json` was originally left **un**symlinked:
|
||||
it is already palace-derived, and its HOME path is a warning-only legacy probe (`hallways.py:73-95`) that
|
||||
never auto-migrates.
|
||||
|
||||
> **Update 2026-08-14 — `hallways.json` is now symlinked too**, during the seeding session:
|
||||
> ```sh
|
||||
> ln -sfn palace/hallways.json ~/.mempalace/hallways.json
|
||||
> ```
|
||||
> Rationale changed: the point is no longer "only symlink what the code demands" but *all real state
|
||||
> lives under `palace/` as a single backup unit*, so one `palace/` copy is a complete copy. All three
|
||||
> parent-level paths now resolve, which matters because mempalace 3.6.0 resolves these three paths
|
||||
> inconsistently (MCP server: palace-relative; KG CLI default: `~/.mempalace`; `hallways.json`:
|
||||
> `dirname(palace_path)`; `known_entities.json`: hardcoded `~`). Revert by deleting the symlink if it
|
||||
> ever causes trouble.
|
||||
|
||||
Verified the symlink assumption rather than trusting it (`python3 sqlite3` on synlig, temp dir):
|
||||
|
||||
| Check | Result |
|
||||
| --- | --- |
|
||||
| Dangling symlink + `sqlite3.connect` | creates the target |
|
||||
| `-wal` / `-shm` placement | next to the **target**, inside the palace dir — *not* beside the symlink |
|
||||
| Write via symlink → read via palace path | same data, **same inode** |
|
||||
|
||||
The WAL placement is the part that mattered: it keeps the palace directory a single self-contained
|
||||
backup/bind-mount unit.
|
||||
|
||||
### 2.4 §6.2's Host/Origin policy verified by experiment, not by reading
|
||||
|
||||
Ran on synlig, loopback and docker0 binds, then stopped. **11/11 as predicted:**
|
||||
|
||||
| # | Bind | Request | Expected | Got |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| A1 | 127.0.0.1 | `/healthz`, correct Host | 200 | ✅ 200 |
|
||||
| A2 | 127.0.0.1 | `/healthz`, `Host: palace.example.com` | **403** | ✅ 403 |
|
||||
| A3 | 127.0.0.1 | `/healthz`, `Origin: https://evil.example` | 403 | ✅ 403 |
|
||||
| A4 | 127.0.0.1 | `POST /mcp`, no token | 401 | ✅ 401 |
|
||||
| A5 | 127.0.0.1 | `POST /mcp`, wrong token | 401 | ✅ 401 |
|
||||
| A6 | 127.0.0.1 | `POST /mcp`, correct token | 200 | ✅ 200 (`tools/list` → **36 tools**) |
|
||||
| B1 | 172.17.0.1 | `/healthz`, bound-host Host | 200 | ✅ 200 |
|
||||
| B2 | 172.17.0.1 | `/healthz`, `Host: palace.example.com` | **200** | ✅ 200 |
|
||||
| B3 | 172.17.0.1 | `/healthz`, `Origin: https://evil.example` | 403 | ✅ 403 |
|
||||
| B4 | 172.17.0.1 | `/healthz`, loopback Origin | 200 | ✅ 200 |
|
||||
| B5 | 172.17.0.1 | `POST /mcp`, foreign Host + token | 200 | ✅ 200 |
|
||||
|
||||
**Operational conclusions:**
|
||||
|
||||
1. **Do not bind loopback behind the tunnel.** A2 vs B2 is the whole story: the reflex "bind 127.0.0.1,
|
||||
it's safer" produces a 403 that looks like a Pangolin misconfiguration and is not one.
|
||||
2. **Bind `172.17.0.1` (docker0).** Non-loopback, so the Host pin relaxes — but reachable only from
|
||||
synlig and its containers, so a newt container on this box can reach it while the LAN cannot. This is
|
||||
strictly better than `0.0.0.0` here. It is what `contrib/systemd/mempalace-serve.service` uses.
|
||||
3. **`Origin` is never relaxed** (B3). No browser-based MCP client, and no proxy that injects `Origin`.
|
||||
4. `/healthz` is Host/Origin-gated but token-free — a usable liveness probe for the tunnel.
|
||||
|
||||
Test script kept at `/tmp/synlig-phase0-test.sh` on this container (ephemeral — re-create from the table
|
||||
above if needed; it starts, probes and stops the server, and asserts nothing is left listening).
|
||||
|
||||
### 2.5 A start unit — written, staged, and (since 2026-08-12) installed and running
|
||||
|
||||
> **This subsection describes 2026-08-10. The unit is now live.** Verified 2026-08-16 00:15:
|
||||
> `systemctl --user list-units` shows `mempalace-serve.service … loaded active running`, and the
|
||||
> process is
|
||||
> `~/.local/share/uv/tools/mempalace/bin/python -m mempalace.mcp_server --transport http
|
||||
> --host 172.17.0.1 --port 8765 --palace /home/ecsjper/.mempalace/palace`.
|
||||
> Note what that means and §2.6 depends on: **the server is a NATIVE process, not a container** — it
|
||||
> sees synlig's real filesystem paths, and `docker ps` on synlig lists no mempalace container.
|
||||
|
||||
`contrib/systemd/mempalace-serve.service` — user unit, follows the existing `contrib/systemd/` style,
|
||||
carries the bind rationale inline so nobody "fixes" it back to loopback. A copy is already staged on synlig
|
||||
at `~/.config/systemd/user/mempalace-serve.service.staged` — **the `.staged` suffix is deliberate**:
|
||||
systemd only reads `*.service`, so the file cannot be activated by accident, not even by a stray
|
||||
`daemon-reload`. **Not** installed, **not** enabled: it needs one `sudo loginctl enable-linger`, and
|
||||
standing up a network-reachable service while you were asleep was not mine to decide.
|
||||
(Both were done on 2026-08-12 — §4 item 3 has the exact commands that were run.)
|
||||
|
||||
### 2.6 The transcript inbox — `~/mempalace-feed/<device>/` (added 2026-08-16)
|
||||
|
||||
Flipped clients write drawers over HTTPS, but their **session transcripts** cannot travel that way:
|
||||
`mempalace_mine` resolves its `source` path *in the server process*, so the server cannot see a
|
||||
client's staged exports. `mempalace-pi-session --mode remote` therefore rsyncs each client's stage into
|
||||
a per-device inbox here and then asks the server to mine its own local path:
|
||||
|
||||
```sh
|
||||
ls ~/mempalace-feed/ # one dir per device, e.g. emb-7kj4vr4g/
|
||||
ls ~/mempalace-feed/emb-7kj4vr4g/ # pi_<session-uuid>.jsonl, mtimes preserved
|
||||
```
|
||||
|
||||
Client side, that needs three variables — and **the third one is the trap**:
|
||||
|
||||
| Variable | Value for this fleet | Why |
|
||||
| --- | --- | --- |
|
||||
| `MEMPALACE_PI_SSH_TARGET` | `ecsjper@synlig:/home/ecsjper/mempalace-feed` | where rsync puts the files |
|
||||
| `MEMPALACE_PI_DEVICE` | e.g. `emb-7kj4vr4g` | inbox subdirectory per machine |
|
||||
| `MEMPALACE_PI_REMOTE_PATH` | `/home/ecsjper/mempalace-feed` | the inbox **as the server process sees it** |
|
||||
|
||||
The feeder's default for the third is `/data/feed`, which assumes a *containerized* palace server with
|
||||
the inbox bind-mounted there. **This primary is native (§2.5), so it only ever sees host paths and the
|
||||
value must equal the path half of the SSH target.** Get it wrong and the failure is quiet in the worst
|
||||
way: rsync succeeds, the files are all present here, and only the mine fails with
|
||||
`source directory not found: '/data/feed/<device>'`.
|
||||
|
||||
That is exactly what happened on 2026-08-15, and it went unnoticed for a session because the feeder
|
||||
decided success with `'"error"' in body` — MCP returns HTTP 200 with the tool's own JSON **escaped**
|
||||
inside `result.content[].text`, so those bytes are `\"error\"`, the substring never matched, and
|
||||
`~/.pi/agent/mempalace-catchup.log` printed `Done. Wing 'wing_conversations' updated.` directly under the
|
||||
error. Fixed in `6e1f4f3`: the envelope is parsed, `--self-test` pins that exact response body, and a
|
||||
preflight warning fires whenever the ship path and `MEMPALACE_PI_REMOTE_PATH` disagree.
|
||||
|
||||
**Operational notes for this inbox:**
|
||||
|
||||
- Dedup keys on the **absolute source path**, so the inbox path is load-bearing: it must stay stable, or
|
||||
every transcript re-files under its new name. Migrating it on 2026-08-15 (from the clients' old
|
||||
container-local stage paths, which arrived with the seed) cost a full re-mine plus
|
||||
`mempalace_delete_by_source` on 6 old paths — 651 drawers purged, 1243 re-filed. `wing_conversations`
|
||||
is now keyed entirely on `/home/ecsjper/mempalace-feed/<device>/…`.
|
||||
- A **grown** session is purged and re-filed for the same path (mtime-based), so re-feeding a live
|
||||
session refreshes it instead of duplicating it. That is why the inbox keeps whole transcripts rather
|
||||
than deltas — do not "tidy" it by deleting files the palace still references.
|
||||
- `mempalace_mine` over MCP can **exceed a client's request timeout while the server keeps working and
|
||||
finishes normally**. A client-side timeout is not a failed mine: check
|
||||
`SELECT COUNT(*) FROM embedding_metadata WHERE key='source_file' AND string_value LIKE '<inbox>%'`
|
||||
(read-only, `file:…?mode=ro`) before retrying anything.
|
||||
- Health check after any client recreate, from the client: the tail of
|
||||
`~/.pi/agent/mempalace-catchup.log` should end in `Done. Wing … updated.` with no `error:` line above
|
||||
it. With the fixed feeder a broken run exits 5 and names the reason.
|
||||
|
||||
## 3. Deliberately NOT done
|
||||
|
||||
> **⚠️ Historical — this section describes 2026-08-10 and is no longer true.** All five items were
|
||||
> done between 2026-08-12 and 2026-08-14. Kept because the *reasoning* for deferring them is still
|
||||
> the record of why the order was chosen. Current state per item is inlined below.
|
||||
|
||||
- **Nothing is serving.** No listener on 8765; no mempalace process. Re-verified at the end of the run.
|
||||
→ **Now serving** since 2026-08-12 (`mempalace-serve.service`, `systemctl --user`), reachable at
|
||||
`https://mempalace.jordbo.se/mcp` via newt/Pangolin.
|
||||
- **No client `.env` was touched.** Your working setup is exactly as you left it (R6: reversible).
|
||||
→ **One client flipped 2026-08-14**: four variables on EMB-7KJ4VR4G, `docker-compose.yaml` unchanged.
|
||||
Still reversible in ~30s (§3.8 of [`phase-1-exposure-runbook.md`](./phase-1-exposure-runbook.md)).
|
||||
- **No data joined.** The palace is empty. The §4.4 join needs the diary-dedup decision (§7.6) first —
|
||||
replaying diaries today duplicates them, and the primary is the one place that must stay clean.
|
||||
→ **Seeded 2026-08-14** from *one* palace by file-level copy. This sidestepped §7.6 rather than
|
||||
solving it: a file-level copy replays no diaries, so it cannot duplicate them. **§7.6 is still a
|
||||
hard blocker for the second machine to join.**
|
||||
- **nyvaken untouched.** Read nothing, changed nothing.
|
||||
- **No sudo.** `sudo -n` on synlig requires a password.
|
||||
|
||||
## 4. Tomorrow, in order
|
||||
|
||||
> **2026-08-12: items 1–2 and 5 now have their own runbook —**
|
||||
> [`phase-1-exposure-runbook.md`](./phase-1-exposure-runbook.md). Pangolin on nyvaken is updated (done),
|
||||
> and **newt is now installed on synlig and connected to Pangolin (done 2026-08-12)** — so the blocker is
|
||||
> now item 3, the one `sudo`. That doc also records why per-device Pangolin users are the wrong layer, why
|
||||
> the HTTPS tunnel and the feeder's SSH path are **not** redundant (§1.3), the client-flip variable trap
|
||||
> (§3.7), and an additional loopback finding: a loopback bind does not merely 403, it also silently starts
|
||||
> the server with **no token at all** (auto-minting is gated on the bind being non-loopback).
|
||||
|
||||
1. **Pangolin update on nyvaken** (yours). ✅ done 2026-08-12.
|
||||
2. ~~**⚠️ synlig has no tunnel client.**~~ ✅ **done 2026-08-12** — newt installed and connected to Pangolin.
|
||||
(Kept for the reasoning: `docker ps` showed only the Gitea runner and digikam. Pangolin on nyvaken
|
||||
cannot reach synlig by itself; synlig had to dial out. Easy to miss because Pangolin looks healthy on
|
||||
its own side — which is also why "connected" is not yet proof it can reach the palace: verify
|
||||
`172.17.0.1:8765/healthz` from *inside* newt's namespace, exposure runbook §3.3.) Since newt runs in
|
||||
Docker here, the docker0 bind above is already correct for it.
|
||||
3. **One sudo, then start** (the unit is already staged; just drop the suffix). ✅ **done 2026-08-12** —
|
||||
linger enabled, unit enabled, `172.17.0.1:8765/healthz` → `ok`.
|
||||
```sh
|
||||
sudo loginctl enable-linger ecsjper
|
||||
cd ~/.config/systemd/user && mv mempalace-serve.service.staged mempalace-serve.service
|
||||
systemctl --user daemon-reload && systemctl --user enable --now mempalace-serve
|
||||
curl -s 172.17.0.1:8765/healthz # ok
|
||||
curl -s 127.0.0.1:8765/healthz # NOTHING — refused, exit 7 (not 403; see below)
|
||||
ss -ltnp | grep 8765 # 172.17.0.1:8765 only
|
||||
```
|
||||
⚠ **Corrected 2026-08-12:** this line predicted `403`. The real run returned empty, which is *more*
|
||||
reassuring. With the docker0-only bind nothing listens on loopback, so the connection is refused before
|
||||
any header is sent (`%{http_code}` → `000`, `$?` → `7`). The 403 in §2.4 is the **loopback-bind** case:
|
||||
a server on `127.0.0.1` answering a proxy-forwarded foreign `Host:`. Two different failures that were
|
||||
collapsed into one expectation here.
|
||||
4. **Collect the shared token** (auto-minted on first non-loopback start, stable across restarts):
|
||||
```sh
|
||||
cat ~/.mempalace/server/f5d849287f6d73f0141b29d7/token
|
||||
```
|
||||
That directory name is `sha256(realpath(palace))[:24]` — it changes if the palace path ever changes.
|
||||
5. **Route it through Pangolin**, then verify `/healthz` end-to-end through the public hostname *before*
|
||||
pointing any client at it.
|
||||
6. **Then, and only then**, Phase 1 client flip — one machine first, and remember opencode containers
|
||||
need the §4.1 sidecar merge (or Phase 1.5) before the `.env` takes effect.
|
||||
7. **Before the first join:** settle §7.6 diary dedup, then dry-run §4.4 from **one** palace.
|
||||
> **Correction 2026-08-14 — do NOT verify a join "by checking counts", which is what this item
|
||||
> originally said.** Counts are not evidence, in either direction. `mempalace status` counts
|
||||
> **chunk rows**, not logical drawers (3 drawers plus one 2-chunk diary presented as +9), and chunk
|
||||
> counts legitimately differ between two palaces whenever a drawer was updated on either side,
|
||||
> because an update re-chunks to the new length and deletes the surplus rows. Diffing chunk-id sets
|
||||
> is a useful first pass but **over**-reports: a chunk id present on one side only is the ordinary
|
||||
> signature of an edit, not of loss. This cost real time on 2026-08-14 — a missing
|
||||
> `chunk_000007` was read as a truncated seed, when in fact the two palaces held two revisions of
|
||||
> one drawer and nothing was lost. **Adjudicate by fetching the parent drawer on both sides and
|
||||
> comparing the reassembled `content`.**
|
||||
|
||||
## 5. Rollback
|
||||
|
||||
> **⚠️ STOP — 2026-08-14. Do not run this block as it was originally written.** `~/.mempalace` on
|
||||
> synlig is now the fleet primary. That tree holds the only central palace (14,803 drawers, seeded
|
||||
> from EMB-7KJ4VR4G) **and the server's bearer token** at `~/.mempalace/server/<hash>/token` — the
|
||||
> single credential every flipped client authenticates with, of which there is no second copy.
|
||||
> `rm -rf ~/.mempalace` destroys both. The original comment ("empty today — check before running once
|
||||
> it isn't") is far too soft for a destructive command someone runs under pressure, which is exactly
|
||||
> why it is being replaced rather than amended.
|
||||
|
||||
Stopping the service is safe and reversible on its own, and is the whole of what "rollback" should
|
||||
normally mean now:
|
||||
|
||||
```sh
|
||||
systemctl --user disable --now mempalace-serve # clients fail CLOSED — they lose the palace
|
||||
# tools; they do NOT fall back to a local palace
|
||||
```
|
||||
|
||||
To genuinely decommission the primary, in this order:
|
||||
|
||||
1. Flip every client back first (§3.8 of [`phase-1-exposure-runbook.md`](./phase-1-exposure-runbook.md),
|
||||
in reverse) so nothing is pointed at a host that is about to lose its palace.
|
||||
2. Copy `~/.mempalace/palace/` **and** the token file off the host, and verify the copy by comparing
|
||||
reassembled drawer `content`, not counts (see §4 item 7).
|
||||
3. Only then remove anything. Never `rsync --delete` into `~/.mempalace` — the token lives inside it.
|
||||
|
||||
The two destructive steps below were written on 2026-08-10, when `~/.mempalace` was genuinely empty.
|
||||
Kept for the record; **must not be run while the primary is live**:
|
||||
|
||||
```sh
|
||||
~/.local/bin/uv tool uninstall mempalace
|
||||
rm -rf ~/.mempalace # ⚠️ DESTROYS THE FLEET PALACE AND THE ONLY TOKEN
|
||||
```
|
||||
If you are looking for the mechanism, the two runbooks above are the same
|
||||
procedures with `<PRIMARY_HOST>` and `<USER>` in place of one site's specifics.
|
||||
|
||||
Reference in New Issue
Block a user