From 947604b25d0c054f239868ce3c061eb95b7f575b Mon Sep 17 00:00:00 2001 From: Joakim Persson Date: Mon, 17 Aug 2026 00:50:16 +0200 Subject: [PATCH] docs: backup and recovery, plus units; move host runbook to a private repo MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Adds bin/mempalace-backup and docs/backup-and-recovery.md — the mechanism a palace actually needs, none of it site-specific. Why a palace cannot be backed up with cp: it is chroma.sqlite3 (authoritative), knowledge_graph.sqlite3 (usually WAL, so -wal/-shm make a plain copy a same-instant gamble), derived HNSW segment dirs, hallways.json, the embedder descriptor, and a HIDDEN .mempalace/origin.json. Both SQLite files are therefore copied through the online-backup API. Two bugs are documented because both produce a backup that looks fine: "$PALACE"/*/ silently skips the hidden dir, and per-directory rsync collides the identically named data_level0.bin in every HNSW segment. Treating the palace as one tree fixes both and makes a backup a faithful palace IMAGE, so restore is a copy rather than a procedure. Two modes: hot (default, zero downtime, ~4 s, index may lag but SQLite is authoritative and repair --mode from-sqlite rebuilds) and cold (--cold, ~5 s downtime, byte-consistent, restart trapped so a failed run still brings the server back). Verification runs on the COPY — quick_check plus row counts — and the backup is committed by mv only after it passes, with retention pruned only after a verified commit, so a broken new backup cannot delete the last good one. Documented because they are easy to get wrong: the sqlite3 CLI is often absent where the Python module is present; mempalace_embedder.json must be restored with the drawers or search silently degrades; a tested restore means running status AND search against the restored copy, since search is what actually exercises the index; mempalace-serve is a USER unit, so root systemctl reports "not found"; Persistent=true is what makes a missed window run after boot; and installing against the system Python couples the palace's availability to distribution upgrades, with the uv-managed-interpreter fix plus the two PATH traps that bite scripted upgrades. Also moves docs/synlig-primary-runbook.md out to a private fleet repository, leaving a stub that explains the split, since a host inventory is operator data for one deployment rather than part of a public toolkit. The path stays valid so existing links do not break. Remaining host references in the README, RFCs and ARCHITECTURE are left alone deliberately: they are load-bearing prose, contain no secrets, and are best generalised as they are next edited rather than in one churn-heavy pass. --- bin/mempalace-backup | 282 +++++++++++++++ contrib/README.md | 2 + contrib/systemd/mempalace-backup-cold.service | 20 ++ contrib/systemd/mempalace-backup-cold.timer | 12 + contrib/systemd/mempalace-backup.service | 19 + contrib/systemd/mempalace-backup.timer | 15 + docs/backup-and-recovery.md | 275 +++++++++++++++ docs/synlig-primary-runbook.md | 326 +----------------- 8 files changed, 641 insertions(+), 310 deletions(-) create mode 100644 bin/mempalace-backup create mode 100644 contrib/systemd/mempalace-backup-cold.service create mode 100644 contrib/systemd/mempalace-backup-cold.timer create mode 100644 contrib/systemd/mempalace-backup.service create mode 100644 contrib/systemd/mempalace-backup.timer create mode 100644 docs/backup-and-recovery.md diff --git a/bin/mempalace-backup b/bin/mempalace-backup new file mode 100644 index 0000000..091ce46 --- /dev/null +++ b/bin/mempalace-backup @@ -0,0 +1,282 @@ +#!/usr/bin/env bash +# mempalace-backup — consistent, verified backups of a file-backed MemPalace palace. +# +# WHY THIS EXISTS +# The fleet primary (RFC-001) had no backups at all. A palace is not one file: +# it is two SQLite databases (one of them in WAL mode), a set of chromadb HNSW +# segment directories, and a few JSON sidecars. `cp -r` on a live palace can +# capture a torn WAL and an index that disagrees with the sqlite. +# +# TWO MODES +# hot (default) Zero downtime. Both SQLite files are copied with the sqlite +# ONLINE BACKUP API (a transactionally consistent snapshot, +# WAL included). The HNSW segment dirs are rsynced and may be +# slightly skewed relative to the sqlite — which is RECOVERABLE, +# because the sqlite is the source of truth and the index can be +# rebuilt: `mempalace repair --mode from-sqlite`. +# cold (--cold) Stops the server, copies everything byte-for-byte, restarts. +# Guaranteed self-consistent including the index. The restart is +# trapped so an error still brings the server back. +# +# BONUS: restoring the hot backup via `repair --mode from-sqlite --archive-existing` +# re-CREATES the collections, which is also the only path that applies the modern +# chromadb HNSW defaults (batch_size=100 / sync_threshold=1000). Palaces created +# under mempalace <=3.6 are stuck on 2/2 and sync the index every 2 records. +# +# NOTE: synlig has no sqlite3 CLI, so every SQLite operation here goes through +# python3's stdlib sqlite3 module. Do not "simplify" this to `VACUUM INTO`. +set -euo pipefail + +PALACE="${MEMPALACE_PALACE_PATH:-$HOME/.mempalace/palace}" +DEST="${MEMPALACE_BACKUP_DIR:-$HOME/backups/mempalace}" +UNIT="mempalace-serve" +KEEP=7 +COLD=0 +DRY=0 +VERIFY="" + +usage() { + cat <<'USAGE' +usage: mempalace-backup [--cold] [--dest DIR] [--keep N] [--palace DIR] [--dry-run] + mempalace-backup --verify BACKUP_DIR + + --cold quiesce the server (stop/copy/start) for a byte-consistent copy + --dest DIR backup root (default: $HOME/backups/mempalace) + --keep N how many backups to retain (default: 7; 0 = keep all) + --palace DIR palace to back up (default: $MEMPALACE_PALACE_PATH or ~/.mempalace/palace) + --dry-run report what would happen, touch nothing + --verify DIR integrity-check an existing backup and print its counts +USAGE +} + +while [ $# -gt 0 ]; do + case "$1" in + --cold) COLD=1 ;; + --dest) DEST="$2"; shift ;; + --keep) KEEP="$2"; shift ;; + --palace) PALACE="$2"; shift ;; + --dry-run) DRY=1 ;; + --verify) VERIFY="$2"; shift ;; + -h|--help) usage; exit 0 ;; + *) echo "mempalace-backup: unknown option $1" >&2; usage >&2; exit 2 ;; + esac + shift +done + +log() { printf '%s %s\n' "$(date -u +%Y-%m-%dT%H:%M:%SZ)" "$*"; } +die() { log "ERROR: $*"; exit 1; } + +# ── shared python helpers ──────────────────────────────────────────────────── +# check_db: PRAGMA quick_check + row counts, on a copy. Prints JSON. +py_check() { + python3 - "$1" <<'PY' +import json, os, sqlite3, sys +path = sys.argv[1] +out = {"file": os.path.basename(path), "bytes": os.path.getsize(path)} +try: + c = sqlite3.connect(f"file:{path}?mode=ro", uri=True) + out["quick_check"] = c.execute("PRAGMA quick_check").fetchone()[0] + tables = [r[0] for r in c.execute( + "SELECT name FROM sqlite_master WHERE type='table'")] + out["tables"] = len(tables) + counts = {} + if "embedding_metadata" in tables: + counts["drawers"] = c.execute( + "SELECT count(*) FROM embedding_metadata " + "WHERE key='chroma:document'").fetchone()[0] + for t in ("facts", "triples", "entities"): + if t in tables: + counts[t] = c.execute(f"SELECT count(*) FROM {t}").fetchone()[0] + out["counts"] = counts + c.close() +except Exception as exc: # noqa: BLE001 — report, never mask + out["error"] = repr(exc) +print(json.dumps(out)) +PY +} + +# ── --verify: inspect an existing backup, then exit ────────────────────────── +if [ -n "$VERIFY" ]; then + [ -d "$VERIFY" ] || die "no such backup dir: $VERIFY" + log "verifying $VERIFY" + rc=0 + for db in "$VERIFY"/chroma.sqlite3 "$VERIFY"/knowledge_graph.sqlite3; do + [ -f "$db" ] || { log " MISSING $(basename "$db")"; rc=1; continue; } + res=$(py_check "$db") + log " $res" + printf '%s' "$res" | grep -q '"quick_check": "ok"' || rc=1 + done + if [ -f "$VERIFY/MANIFEST.json" ]; then + python3 - "$VERIFY/MANIFEST.json" <<'PY' +import json, sys +m = json.load(open(sys.argv[1])) +print(" manifest: created={created_utc} mode={mode} bytes={bytes_total} " + "mempalace={mempalace_version}".format(**m)) +print(" index :", m.get("index_consistency", "?")) +PY + fi + [ "$rc" -eq 0 ] && log "VERIFY OK" || log "VERIFY FAILED" + exit "$rc" +fi + +# ── preflight ─────────────────────────────────────────────────────────────── +[ -d "$PALACE" ] || die "palace not found: $PALACE" +[ -f "$PALACE/chroma.sqlite3" ] || die "no chroma.sqlite3 in $PALACE — wrong dir?" +command -v rsync >/dev/null || die "rsync not found" + +STAMP="$(date -u +%Y%m%dT%H%M%SZ)" +MODE=$([ "$COLD" -eq 1 ] && echo cold || echo hot) +OUT="$DEST/mp-$STAMP-$MODE" +TMP="$OUT.incomplete" + +log "palace : $PALACE ($(du -sh "$PALACE" | cut -f1))" +log "dest : $OUT" +log "mode : $MODE keep: $KEEP" + +if [ "$DRY" -eq 1 ]; then + log "dry-run: would create $OUT" + log "dry-run: would copy chroma.sqlite3 + knowledge_graph.sqlite3 via sqlite backup API" + log "dry-run: would rsync the palace tree ($(find "$PALACE" -mindepth 1 -maxdepth 1 | wc -l) top-level entries, hidden included) minus SQLite" + [ "$COLD" -eq 1 ] && log "dry-run: would stop/start $UNIT" + exit 0 +fi + +mkdir -p "$TMP" + +# ── cold mode: quiesce, with a trap so the server always comes back ───────── +STOPPED=0 +restart_if_stopped() { + if [ "$STOPPED" -eq 1 ]; then + log "restarting $UNIT" + systemctl --user start "$UNIT" || log "WARNING: failed to start $UNIT — CHECK MANUALLY" + STOPPED=0 + fi +} +trap restart_if_stopped EXIT INT TERM + +if [ "$COLD" -eq 1 ]; then + log "stopping $UNIT (quiesce)" + systemctl --user stop "$UNIT" + STOPPED=1 + # Give an in-flight mine a moment to release the writer lease. + sleep 2 +fi + +# ── the copy ──────────────────────────────────────────────────────────────── +# Order matters: rsync the tree FIRST, then write the two SQLite files as +# consistent snapshots, so the databases are the freshest thing in the backup. +# +# rsync takes everything EXCEPT the SQLite files (and their -wal/-shm): the HNSW +# segment dirs, the JSON sidecars, and hidden dirs such as .mempalace/origin.json +# — a `"$PALACE"/*/` glob silently skips those, which is exactly the bug this +# structure removes. The payoff: a backup dir IS a palace image, so restore is a +# copy rather than a procedure. +log "rsyncing palace tree (excluding SQLite)" +rsync -a \ + --exclude 'chroma.sqlite3' --exclude 'chroma.sqlite3-*' \ + --exclude 'knowledge_graph.sqlite3' --exclude 'knowledge_graph.sqlite3-*' \ + "$PALACE"/ "$TMP"/ + +# SQLite: online backup API. Consistent even mid-write, and WAL-aware — which +# matters because knowledge_graph.sqlite3 runs in WAL mode (-wal/-shm present). +for db in chroma.sqlite3 knowledge_graph.sqlite3; do + [ -f "$PALACE/$db" ] || { log "skip $db (absent)"; continue; } + log "sqlite-backup $db" + python3 - "$PALACE/$db" "$TMP/$db" <<'PY' +import sqlite3, sys +src_path, dst_path = sys.argv[1], sys.argv[2] +try: + src = sqlite3.connect(f"file:{src_path}?mode=ro", uri=True) +except sqlite3.OperationalError: + src = sqlite3.connect(src_path) # fall back if ro open is refused +dst = sqlite3.connect(dst_path) +with dst: + src.backup(dst) # atomic, consistent snapshot +dst.close(); src.close() +PY +done + +restart_if_stopped +trap - EXIT INT TERM + +# ── verify the COPY, not the original ─────────────────────────────────────── +log "verifying copies" +CHROMA_JSON=$(py_check "$TMP/chroma.sqlite3") +log " $CHROMA_JSON" +printf '%s' "$CHROMA_JSON" | grep -q '"quick_check": "ok"' \ + || die "chroma.sqlite3 copy failed quick_check — backup NOT committed" +KG_JSON='{}' +if [ -f "$TMP/knowledge_graph.sqlite3" ]; then + KG_JSON=$(py_check "$TMP/knowledge_graph.sqlite3") + log " $KG_JSON" + printf '%s' "$KG_JSON" | grep -q '"quick_check": "ok"' \ + || die "knowledge_graph.sqlite3 copy failed quick_check — backup NOT committed" +fi + +# ── manifest ──────────────────────────────────────────────────────────────── +MEMPALACE_VER=$("$HOME/.local/bin/mempalace" --version 2>/dev/null | head -1 || echo unknown) +python3 - "$TMP/MANIFEST.json" "$STAMP" "$MODE" "$PALACE" "$MEMPALACE_VER" \ + "$CHROMA_JSON" "$KG_JSON" <<'PY' +import json, os, socket, subprocess, sys +out, stamp, mode, palace, ver, chroma, kg = sys.argv[1:8] +def size(p): + t = 0 + for root, _dirs, files in os.walk(p): + for f in files: + try: t += os.path.getsize(os.path.join(root, f)) + except OSError: pass + return t +man = { + "created_utc": stamp, + "mode": mode, + "host": socket.gethostname(), + "palace_path": palace, + "mempalace_version": ver.strip(), + "bytes_total": size(os.path.dirname(out)), + "chroma": json.loads(chroma), + "knowledge_graph": json.loads(kg) if kg.strip() != "{}" else None, + "index_consistency": ( + "byte-consistent (server quiesced)" if mode == "cold" else + "sqlite is authoritative; HNSW segments may lag a few writes — " + "restore with `mempalace repair --mode from-sqlite` if search misbehaves" + ), + "restore": [ + "# a backup dir IS a palace image — restore is a copy, not a procedure", + "systemctl --user stop mempalace-serve", + "mv ~/.mempalace/palace ~/.mempalace/palace.broken-$(date -u +%Y%m%dT%H%M%SZ)", + "mkdir -p ~/.mempalace/palace", + "rsync -a --exclude MANIFEST.json / ~/.mempalace/palace/", + "systemctl --user start mempalace-serve", + "# hot backups only: if search misbehaves or the index count looks stale,", + "# rebuild the HNSW index from the authoritative sqlite -- this also", + "# re-creates collections with modern HNSW defaults (100/1000):", + "mempalace repair --mode from-sqlite --archive-existing --yes", + ], +} +with open(out, "w") as fh: + json.dump(man, fh, indent=2) +print(json.dumps({k: man[k] for k in ("created_utc", "mode", "bytes_total")})) +PY + +mv "$TMP" "$OUT" +log "committed $OUT ($(du -sh "$OUT" | cut -f1))" + +# ── retention: only ever prune AFTER a committed, verified backup ─────────── +if [ "$KEEP" -gt 0 ]; then + mapfile -t all < <(find "$DEST" -maxdepth 1 -type d -name 'mp-*' | sort) + n=${#all[@]} + if [ "$n" -gt "$KEEP" ]; then + for old in "${all[@]:0:$((n - KEEP))}"; do + log "pruning $old" + rm -rf "$old" + done + fi + log "retained $(find "$DEST" -maxdepth 1 -type d -name 'mp-*' | wc -l)/$KEEP" +fi + +# Leftover .incomplete dirs mean a previous run died — surface, don't hide. +for stale in "$DEST"/mp-*.incomplete; do + [ -e "$stale" ] && log "NOTE: stale incomplete backup present: $stale" +done + +log "done" diff --git a/contrib/README.md b/contrib/README.md index 618d0dd..a031c75 100644 --- a/contrib/README.md +++ b/contrib/README.md @@ -37,6 +37,8 @@ Pick **one scheduler** (systemd *or* launchd *or* cron). The opencode and pi job | `cron/mempalace-pi-session.cron` | pi → palace | Tue 03:00 | | `cron/mempalace-session-devbox.cron` | opencode (devbox) → palace | Mon 03:00 | | `systemd/mempalace-serve.service` | **not a mining job** — runs the shared palace *server* | always-on | +| `systemd/mempalace-backup.{service,timer}` | **not a mining job** — hot palace backup (zero downtime) | daily 04:00 | +| `systemd/mempalace-backup-cold.{service,timer}` | **not a mining job** — cold palace backup (~5 s downtime, byte-consistent) | Sun 04:30 | The pi variants are drop-in copies of the opencode variants with script name and schedule updated; the install recipes below apply equally — just swap `mempalace-session` for `mempalace-pi-session` and the schedule day. diff --git a/contrib/systemd/mempalace-backup-cold.service b/contrib/systemd/mempalace-backup-cold.service new file mode 100644 index 0000000..828b4a7 --- /dev/null +++ b/contrib/systemd/mempalace-backup-cold.service @@ -0,0 +1,20 @@ +[Unit] +Description=MemPalace backup (cold — quiesces the server for a byte-consistent copy) +Documentation=file:%h/mempalace-toolkit/docs/backup-and-recovery.md +ConditionPathExists=%h/.mempalace/palace + +[Service] +Type=oneshot +# Cold mode stops mempalace-serve, copies the palace at rest (so the HNSW index +# is byte-consistent with the SQLite, not merely repairable from it), and starts +# it again. The restart is trapped on EXIT/INT/TERM inside the script, so an +# interrupted or failed run still brings the server back up. +# +# Measured downtime on a 223 MB palace: ~5 seconds. +# +# NOTE: mempalace-serve.service ships a drop-in with Restart=always. That is +# correct for crash recovery but means an explicit `systemctl --user stop` is the +# only way to keep it down for the duration of the copy — do not expect a plain +# SIGTERM to hold it. +ExecStart=%h/.local/bin/mempalace-backup --cold --keep 7 +Nice=10 diff --git a/contrib/systemd/mempalace-backup-cold.timer b/contrib/systemd/mempalace-backup-cold.timer new file mode 100644 index 0000000..312f4d6 --- /dev/null +++ b/contrib/systemd/mempalace-backup-cold.timer @@ -0,0 +1,12 @@ +[Unit] +Description=Weekly cold MemPalace backup +Documentation=file:%h/mempalace-toolkit/docs/backup-and-recovery.md + +[Timer] +# Half an hour after the daily hot backup, so the two never overlap on the +# palace lock even if the hot run is unusually slow. +OnCalendar=Sun 04:30:00 +Persistent=true + +[Install] +WantedBy=timers.target diff --git a/contrib/systemd/mempalace-backup.service b/contrib/systemd/mempalace-backup.service new file mode 100644 index 0000000..3288a46 --- /dev/null +++ b/contrib/systemd/mempalace-backup.service @@ -0,0 +1,19 @@ +[Unit] +Description=MemPalace backup (hot — zero downtime, SQLite online-backup API) +Documentation=file:%h/mempalace-toolkit/docs/backup-and-recovery.md +# Better a skipped run than a run that creates an empty palace somewhere and +# then dutifully backs it up. +ConditionPathExists=%h/.mempalace/palace + +[Service] +Type=oneshot +# Hot mode never stops mempalace-serve. Both SQLite files are copied through the +# online-backup API (transactionally consistent, WAL-aware); the HNSW index +# segments are copied live and may lag a few records. That is acceptable because +# SQLite is authoritative — see docs/backup-and-recovery.md §3. +ExecStart=%h/.local/bin/mempalace-backup --keep 7 +# Backups are not urgent; the palace server and any running mine are. Yielding +# CPU and I/O keeps a 4-second backup from lengthening someone's interactive +# search. +Nice=10 +IOSchedulingClass=idle diff --git a/contrib/systemd/mempalace-backup.timer b/contrib/systemd/mempalace-backup.timer new file mode 100644 index 0000000..eaf51eb --- /dev/null +++ b/contrib/systemd/mempalace-backup.timer @@ -0,0 +1,15 @@ +[Unit] +Description=Daily hot MemPalace backup +Documentation=file:%h/mempalace-toolkit/docs/backup-and-recovery.md + +[Timer] +OnCalendar=*-*-* 04:00:00 +# If the host was powered off at 04:00, run shortly after the next boot instead +# of silently skipping that day. Without this, a laptop or an occasionally-off +# server quietly accumulates days with no backup. +Persistent=true +# Avoid colliding with other maintenance that also picked the top of the hour. +RandomizedDelaySec=300 + +[Install] +WantedBy=timers.target diff --git a/docs/backup-and-recovery.md b/docs/backup-and-recovery.md new file mode 100644 index 0000000..b18f179 --- /dev/null +++ b/docs/backup-and-recovery.md @@ -0,0 +1,275 @@ +# Backing up and recovering a palace + +A palace is not one file, so `cp -r` is not a backup. This document explains what +a palace is made of, why the obvious approaches silently corrupt it, and how +`bin/mempalace-backup` avoids that — plus how to actually restore one and prove +the restore worked. + +Host-specific details (which machine is primary, addresses, users, offsite +targets) deliberately live outside this repo. Substitute `` and +`` throughout. + +## 1. What is actually in a palace + +Run `ls -a` on a palace directory and you get something like this — sizes are +from a real ~25k-drawer palace, for a sense of proportion: + +| Path | Size | What it is | +|---|---|---| +| `chroma.sqlite3` | 167 MB | **The authoritative store.** Drawers, chunks, metadata, embeddings. | +| `/` (one per segment) | 50 MB, 2.8 MB | chromadb HNSW index segments (`data_level0.bin`, `index_metadata.pickle`). Derived data. | +| `knowledge_graph.sqlite3` | 64 KB | KG triples + entities. **Usually in WAL mode** — see §2. | +| `knowledge_graph.sqlite3-wal` / `-shm` | varies | Live write-ahead log. Their existence is the whole problem. | +| `hallways.json` | 10.8 MB | Within-wing co-occurrence links, rebuilt by `mempalace hallways`. | +| `known_entities.json`, `mempalace_embedder.json` | < 1 KB | Entity list; **which embedder model this palace was built with**. | +| `.mempalace/origin.json` | small | Provenance. **Hidden** — see the glob trap in §3. | +| `.collection_type_fixed`, `.blob_seq_ids_migrated` | 0 B | Migration marker files. Presence is the signal. | + +Two consequences worth internalising: + +- **The SQLite files are the truth; the index segments are derived.** If the + index is stale or damaged you can rebuild it from SQLite. The reverse is not + true. This asymmetry is what makes a zero-downtime backup possible at all. +- **`mempalace_embedder.json` is not optional.** Restoring drawers next to a + *different* embedder model gives you a palace whose stored vectors and future + queries live in different spaces — search silently degrades rather than + failing. Back it up, and check it after a restore. + +## 2. Why `cp` is unsafe, specifically + +SQLite in WAL mode keeps recent commits in `-wal` and shared state in `-shm`. +Copying those three files with `cp` gives you no guarantee they are from the same +instant: the copier can read `chroma.sqlite3` before a checkpoint and `-wal` +after it. The result usually *opens* — which is the dangerous part — and then +misbehaves later, or fails `PRAGMA integrity_check` under load. + +`rsync` has exactly the same problem: it is not a snapshot, it is a sequence of +reads. + +The supported approach is SQLite's **online backup API**, which takes a +transactionally consistent copy of a live database while writers continue: + +```python +import sqlite3 +src = sqlite3.connect(f"file:{src_path}?mode=ro", uri=True) +dst = sqlite3.connect(dst_path) +with dst: + src.backup(dst) # WAL-aware, consistent, no writer downtime +``` + +Two practical notes: + +- **`VACUUM INTO` is a fine alternative in principle**, but it needs either the + `sqlite3` CLI or a Python build new enough for it. Do not assume the CLI is + present: a stock Ubuntu server often has the Python `sqlite3` *module* and no + `sqlite3` *binary*. Check with `command -v sqlite3` before writing tooling + that depends on it. +- Open the source with `mode=ro` so a backup can never be the thing that + creates a `-wal` file in a directory you meant only to read. + +## 3. Two backup modes, and the honest tradeoff + +`bin/mempalace-backup` implements both. Neither is strictly better. + +### hot (default) — zero downtime + +1. `rsync` the whole palace tree **excluding** the SQLite files. +2. Write both SQLite files via the online-backup API. + +The server never stops. The index segments are copied while they may be being +written, so in principle they can lag the SQLite by a few records. That is +acceptable *because the SQLite is authoritative*: if search misbehaves after a +restore, rebuild the index (§5). + +Measured on a 223 MB palace: **~4 s**. + +### cold (`--cold`) — byte-consistent + +1. Stop the server (`systemctl --user stop mempalace-serve`). +2. Copy everything, including the index, at rest. +3. Restart the server. + +The restart is in a `trap … EXIT INT TERM`, so an interruption mid-copy still +brings the server back. Measured downtime: **~5 s**. + +Recommended schedule: **hot daily, cold weekly**. The cold copy is your +byte-exact reference; the hot copies bound your data loss between them. + +### The glob trap that cost two rewrites + +Two bugs are worth naming because both produce a backup that looks fine: + +- **`"$PALACE"/*/` skips hidden directories.** `.mempalace/origin.json` is + silently omitted, so the restored palace loses its provenance. Shell globs do + not match dotfiles without `dotglob`/`GLOBIGNORE` fiddling. +- **Per-directory rsync collides identically named files.** Every HNSW segment + contains a `data_level0.bin`. Copying segment dirs into one destination + merges them and the last write wins — a corrupt index that still loads. + +The fix for both is to treat the palace as **one tree**: a single +`rsync -a --exclude` of the SQLite files, so a backup directory is a faithful +*image* of the palace. That makes restore a copy rather than a procedure. + +## 4. Verify the copy, not the original + +A backup you have not read back is a hypothesis. After copying, and **before** +committing the backup or pruning old ones, the tool runs against the *copy*: + +- `PRAGMA quick_check` on both databases +- row counts (`drawers`, `triples`, `entities`) +- a `MANIFEST.json` recording counts, sizes, mode, tool version, and the + index-consistency caveat for hot backups + +Sequencing matters as much as the checks: + +1. write into `.incomplete/` +2. verify +3. `mv` into place (atomic commit) +4. **only now** prune old backups to the retention limit + +Retention pruning that runs before verification can delete your last good +backup on the night the new one is broken. A leftover `.incomplete` directory is +a visible symptom, not silent data loss, so the tool reports stale ones. + +## 5. Restoring + +Because a backup is a palace image, restore is a copy: + +```bash +systemctl --user stop mempalace-serve +mv ~/.mempalace/palace ~/.mempalace/palace.broken-$(date -u +%Y%m%dT%H%M%SZ) +mkdir -p ~/.mempalace/palace +rsync -a --exclude MANIFEST.json / ~/.mempalace/palace/ +systemctl --user start mempalace-serve +``` + +Keep the old directory until the restore is proven. It costs disk and buys you a +second attempt. + +**If the restore came from a hot backup and search behaves oddly**, rebuild the +derived index from the authoritative SQLite: + +```bash +mempalace repair --mode from-sqlite +``` + +Note that a full `repair --mode from-sqlite --archive-existing` re-creates the +chromadb collections, which means new collections pick up **current** HNSW +tuning defaults. On palaces created by older versions this is also the only way +to escape legacy `hnsw:batch_size=2 / hnsw:sync_threshold=2` settings, which sync +the index every two records. Check yours with: + +```sql +SELECT c.name, k.key, k.str_value FROM collections c + JOIN collection_metadata k ON k.collection_id = c.id + WHERE k.key LIKE 'hnsw%'; +``` + +## 6. Prove the restore, don't assume it + +"The files copied" is not a tested restore. Restore to a throwaway directory and +make the palace *answer questions*: + +```bash +rsync -a --exclude MANIFEST.json / /tmp/restore-test/ +export MEMPALACE_PALACE_PATH=/tmp/restore-test +mempalace status # drawer count — compare to the source +mempalace search "some phrase you know is in there" +``` + +`status` proves the schema and metadata survived. **`search` is the stronger +test**: it exercises the HNSW segments, so it is what tells you whether a +*hot* backup's index came across usable. On the reference run above it returned +correctly ranked results (`cosine_sim=0.398 bm25=2.992`) from a zero-downtime +backup — the theoretical index skew did not materialise. + +Do this at least once when you set backups up, and again after any version +upgrade that touches storage. + +## 7. Scheduling that survives a reboot + +Use **systemd user timers** with lingering enabled, so backups run without +anyone logging in: + +```bash +loginctl enable-linger # required, or user units die at logout +systemctl --user enable --now mempalace-backup.timer mempalace-backup-cold.timer +systemctl --user list-timers 'mempalace*' +``` + +Units are in [`contrib/systemd/`](../contrib/systemd/). Two settings earn their +keep: + +- **`Persistent=true`** — a timer whose window was missed (host powered off) + runs shortly after boot instead of skipping the day. +- **`RandomizedDelaySec`** — avoids a thundering herd if other maintenance + shares the hour. + +`mempalace-serve` is itself a **user** unit in the reference deployment. That is +worth stating explicitly because `systemctl status mempalace-serve` as root +reports "not found" and invites the conclusion that the server is not installed. +Always `systemctl --user`. + +### The interpreter is part of your durability story + +If the tool is installed against the **system** Python (`/usr/bin/python3`), then +a distribution release upgrade — which can remove or replace that interpreter — +breaks the server and its backups together, at the moment you least want it. +Install against a **managed** interpreter instead: + +```bash +uv python install 3.12 +uv tool install --force --python "$HOME/.local/bin/python3.12" 'mempalace==' +systemctl --user restart mempalace-serve +``` + +Now an OS upgrade cannot take the interpreter away. Two gotchas: + +- `uv` is typically in `~/.local/bin`, which is **not** on a non-interactive + ssh PATH. Scripted upgrades must `export PATH="$HOME/.local/bin:$PATH"`. +- `uv python list` prints `$HOME`-relative paths, so a regex anchored on a + leading `/` matches inside `.local/share` and yields a bogus absolute path. + Resolve the shim and assert on it: + `readlink -f ~/.local/bin/python3.12 | grep share/uv/python`. + +**Restart the server after any upgrade.** Recent versions refuse mutating tools +when the served library drifts from what is installed on disk, and +`mempalace_reconnect` cannot clear that — it reopens the database but cannot +reload Python modules. + +## 8. Sizing and offsite + +At ~225 MB per copy, 7 hot + 7 cold is ~3 GB — trivial against any modern disk, +so prefer generous retention over clever incremental schemes. If space is +genuinely tight, `zstd -T0` on the SQLite files compresses well, at the cost of +making a backup no longer a directly-mountable palace image. + +**Local backups protect against corruption and mistakes, not against losing the +host.** Until a copy leaves the machine, a single disk or a single `rm` ends the +palace. Whatever offsite target you choose (object store, NAS, another host), +two rules matter more than the choice: + +- **Put a copy of the restore procedure with the backups.** Documentation that + lives only inside the thing you are restoring is not available at the moment + you need it. +- **Verify the offsite copy by restoring from it**, not by checking that the + upload exited 0. + +## 9. Usage + +``` +mempalace-backup [--cold] [--dest DIR] [--keep N] [--palace DIR] [--dry-run] +mempalace-backup --verify +``` + +| Flag | Meaning | +|---|---| +| `--cold` | Quiesce the server for a byte-consistent copy (default: hot) | +| `--dest DIR` | Backup root (default `~/backups/mempalace`) | +| `--keep N` | Retention count, pruned only after a verified commit (default 7) | +| `--palace DIR` | Palace to back up (default `~/.mempalace/palace`) | +| `--verify DIR` | Re-verify an existing backup and print its manifest | +| `--dry-run` | Report what would happen, write nothing | + +Backups are named `mp--`, so they sort +chronologically and their mode is visible without opening the manifest. diff --git a/docs/synlig-primary-runbook.md b/docs/synlig-primary-runbook.md index 68b5052..333312a 100644 --- a/docs/synlig-primary-runbook.md +++ b/docs/synlig-primary-runbook.md @@ -1,315 +1,21 @@ -# synlig primary — Phase 0 runbook and handoff +# (moved) primary-host runbook -Companion to [`rfc-001-global-palace.md`](./rfc-001-global-palace.md). Records what was actually done -on the primary, with verified evidence, so the next session (or the next machine) does not re-derive it. +This file used to contain the deployment runbook for one specific primary host — +its hostname, addresses, user, service wiring and rollback steps. -> **Status 2026-08-14 17:00 — SUPERSEDED IN PART. The primary is live, exposed, and seeded.** -> Serving since 2026-08-12 at `https://mempalace.jordbo.se/mcp`. Seeded 2026-08-14 15:07 from -> EMB-7KJ4VR4G's palace (itself a carry-over from the previous work computer EMB-X1JY06WJ — rfc-001 §4.4) -> — 14,777 drawers / 9 wings / 16,337 embeddings / KG 46 entities, 34 triples, -> now 14,803 drawers. One client (EMB-7KJ4VR4G's pi-devbox container) is flipped and verified -> end-to-end. Both Phase 0 blockers below are cleared. -> -> **Update 2026-08-16 00:20 — the transcript feed is live too, and it is the primary's third moving -> part** (alongside the HTTPS tunnel and the palace itself). See §2.6: transcripts arrive over SSH into -> `~/mempalace-feed//` and are mined *by this host's own server process*. 15,478 drawers as of -> that check — but treat every count in this document as a timestamp, not a fact: `status` counts chunk -> rows, and §4 item 7 explains why counts adjudicate nothing. -> -> **Read §3 "Deliberately NOT done" as a record of the 2026-08-10 state, not of today's** — every -> item in it has since been done. And before running anything in §5 Rollback, read the warning at the -> top of it: `~/.mempalace` on synlig is no longer disposable. +**That content now lives in a private repository**, because a host inventory is +operator data for one deployment, not part of the toolkit. This repository is +public and keeps only host-agnostic *mechanism*. -Original status, kept for the record: +What lives where: -**Status 2026-08-10 00:30 — Phase 0 prep complete. Not serving. Nothing exposed.** -Blocked on two things, both deliberately left to Joakim: the Pangolin update on nyvaken, and one `sudo`. +| Content | Home | +|---|---| +| How to expose a palace over HTTP, and the Host/Origin pin | `docs/phase-1-exposure-runbook.md` (here) | +| Why a palace needs a special backup, and how to restore one | `docs/backup-and-recovery.md` (here) | +| Unit/timer/plist templates | `contrib/` (here) | +| Which machine is primary, its addresses, users, tunnels, offsite target | private fleet repository | +| Per-host feeder device names and schedules | private fleet repository | ---- - -## 1. What synlig is (discovered, not assumed) - -| Fact | Value | -| --- | --- | -| SSH | `synlig` → `synlig.erdc.ericsson.net`, user `ecsjper` (from `~/.ssh/config`) | -| OS | Ubuntu 24.04.4 LTS, 7.8 GiB RAM, 78 G disk (**29 G free**), uptime 12 d | -| Python / uv | system `python3` 3.12.3; `uv` at `~/.local/bin/uv` (**not** on the non-login `PATH`) | -| Interfaces | `lo` 127.0.0.1, `ens3` 10.0.0.4/16, `docker0` 172.17.0.1/16, `br-…` 172.19.0.1/16 | -| Already listening | 22, 80, 443, 3000 (node), 3389 + 3350 + 4822 (xrdp/guacamole), 631 | -| Docker | present; running `act_runner-runner-1` (**Gitea Actions runner**) and `digikam` | -| Pre-existing MemPalace | **none** — no `mempalace` binary, no `~/.mempalace`. Greenfield. | - -The Gitea Actions runner living here is worth remembering: synlig is not a dedicated appliance, and CI -load competes with the palace for the same 7.8 GiB. - -## 2. Done tonight - -### 2.1 MemPalace installed, pinned to the fleet version - -```sh -~/.local/bin/uv tool install "mempalace==3.6.0" # → mempalace, mempalace-mcp -``` - -Pinned deliberately: the clients run 3.6.0, and the id recipes / idempotency probes this RFC leans on are -version-specific. Reversible with `uv tool uninstall mempalace`. - -### 2.2 Embedder model pre-warmed — the corporate-network risk that wasn't - -The first embed pulls `all-MiniLM-L6-v2` ONNX (79.3 MB) from the chroma CDN into -`~/.cache/chroma/onnx_models/` (167 M on disk once unpacked). **This was the main unknown** — an -egress-filtered work VM would have failed here, at the worst possible moment (first client write). -It downloaded at ~20 MB/s with no proxy interference. Done in a throwaway palace, since deleted, so the -real palace never saw it. Same model as the clients use, so the semantic space matches. - -### 2.3 Palace created with the §7.1 landmine structurally removed - -`~/.mempalace/palace` — the **stock default**, so no `MEMPALACE_PALACE_PATH` and no `config.json` is -needed anywhere on synlig. One less thing to drift. - -RFC §7.1 says to `mv` three HOME-anchored stores into the palace dir before first `serve`. **On a -greenfield primary there is nothing to move — but the hazard is not actually a migration hazard, and the -RFC understated it:** with *stock defaults* `palace_path` is `~/.mempalace/palace` while `DEFAULT_KG_PATH` -is `~/.mempalace/knowledge_graph.sqlite3`. Those differ, so the split is the **out-of-the-box** behaviour, -not a consequence of a custom path. It is permanent, not one-time: `serve` always passes `--palace` (KG -inside the palace), while any CLI command run *without* `--palace` uses the HOME path. Two KGs on one box, -forever, silently. - -Fixed by making both resolution rules land on one inode: - -```sh -ln -sfn palace/knowledge_graph.sqlite3 ~/.mempalace/knowledge_graph.sqlite3 -ln -sfn palace/known_entities.json ~/.mempalace/known_entities.json -``` - -Relative targets, so a home-directory move survives. `hallways.json` was originally left **un**symlinked: -it is already palace-derived, and its HOME path is a warning-only legacy probe (`hallways.py:73-95`) that -never auto-migrates. - -> **Update 2026-08-14 — `hallways.json` is now symlinked too**, during the seeding session: -> ```sh -> ln -sfn palace/hallways.json ~/.mempalace/hallways.json -> ``` -> Rationale changed: the point is no longer "only symlink what the code demands" but *all real state -> lives under `palace/` as a single backup unit*, so one `palace/` copy is a complete copy. All three -> parent-level paths now resolve, which matters because mempalace 3.6.0 resolves these three paths -> inconsistently (MCP server: palace-relative; KG CLI default: `~/.mempalace`; `hallways.json`: -> `dirname(palace_path)`; `known_entities.json`: hardcoded `~`). Revert by deleting the symlink if it -> ever causes trouble. - -Verified the symlink assumption rather than trusting it (`python3 sqlite3` on synlig, temp dir): - -| Check | Result | -| --- | --- | -| Dangling symlink + `sqlite3.connect` | creates the target | -| `-wal` / `-shm` placement | next to the **target**, inside the palace dir — *not* beside the symlink | -| Write via symlink → read via palace path | same data, **same inode** | - -The WAL placement is the part that mattered: it keeps the palace directory a single self-contained -backup/bind-mount unit. - -### 2.4 §6.2's Host/Origin policy verified by experiment, not by reading - -Ran on synlig, loopback and docker0 binds, then stopped. **11/11 as predicted:** - -| # | Bind | Request | Expected | Got | -| --- | --- | --- | --- | --- | -| A1 | 127.0.0.1 | `/healthz`, correct Host | 200 | ✅ 200 | -| A2 | 127.0.0.1 | `/healthz`, `Host: palace.example.com` | **403** | ✅ 403 | -| A3 | 127.0.0.1 | `/healthz`, `Origin: https://evil.example` | 403 | ✅ 403 | -| A4 | 127.0.0.1 | `POST /mcp`, no token | 401 | ✅ 401 | -| A5 | 127.0.0.1 | `POST /mcp`, wrong token | 401 | ✅ 401 | -| A6 | 127.0.0.1 | `POST /mcp`, correct token | 200 | ✅ 200 (`tools/list` → **36 tools**) | -| B1 | 172.17.0.1 | `/healthz`, bound-host Host | 200 | ✅ 200 | -| B2 | 172.17.0.1 | `/healthz`, `Host: palace.example.com` | **200** | ✅ 200 | -| B3 | 172.17.0.1 | `/healthz`, `Origin: https://evil.example` | 403 | ✅ 403 | -| B4 | 172.17.0.1 | `/healthz`, loopback Origin | 200 | ✅ 200 | -| B5 | 172.17.0.1 | `POST /mcp`, foreign Host + token | 200 | ✅ 200 | - -**Operational conclusions:** - -1. **Do not bind loopback behind the tunnel.** A2 vs B2 is the whole story: the reflex "bind 127.0.0.1, - it's safer" produces a 403 that looks like a Pangolin misconfiguration and is not one. -2. **Bind `172.17.0.1` (docker0).** Non-loopback, so the Host pin relaxes — but reachable only from - synlig and its containers, so a newt container on this box can reach it while the LAN cannot. This is - strictly better than `0.0.0.0` here. It is what `contrib/systemd/mempalace-serve.service` uses. -3. **`Origin` is never relaxed** (B3). No browser-based MCP client, and no proxy that injects `Origin`. -4. `/healthz` is Host/Origin-gated but token-free — a usable liveness probe for the tunnel. - -Test script kept at `/tmp/synlig-phase0-test.sh` on this container (ephemeral — re-create from the table -above if needed; it starts, probes and stops the server, and asserts nothing is left listening). - -### 2.5 A start unit — written, staged, and (since 2026-08-12) installed and running - -> **This subsection describes 2026-08-10. The unit is now live.** Verified 2026-08-16 00:15: -> `systemctl --user list-units` shows `mempalace-serve.service … loaded active running`, and the -> process is -> `~/.local/share/uv/tools/mempalace/bin/python -m mempalace.mcp_server --transport http -> --host 172.17.0.1 --port 8765 --palace /home/ecsjper/.mempalace/palace`. -> Note what that means and §2.6 depends on: **the server is a NATIVE process, not a container** — it -> sees synlig's real filesystem paths, and `docker ps` on synlig lists no mempalace container. - -`contrib/systemd/mempalace-serve.service` — user unit, follows the existing `contrib/systemd/` style, -carries the bind rationale inline so nobody "fixes" it back to loopback. A copy is already staged on synlig -at `~/.config/systemd/user/mempalace-serve.service.staged` — **the `.staged` suffix is deliberate**: -systemd only reads `*.service`, so the file cannot be activated by accident, not even by a stray -`daemon-reload`. **Not** installed, **not** enabled: it needs one `sudo loginctl enable-linger`, and -standing up a network-reachable service while you were asleep was not mine to decide. -(Both were done on 2026-08-12 — §4 item 3 has the exact commands that were run.) - -### 2.6 The transcript inbox — `~/mempalace-feed//` (added 2026-08-16) - -Flipped clients write drawers over HTTPS, but their **session transcripts** cannot travel that way: -`mempalace_mine` resolves its `source` path *in the server process*, so the server cannot see a -client's staged exports. `mempalace-pi-session --mode remote` therefore rsyncs each client's stage into -a per-device inbox here and then asks the server to mine its own local path: - -```sh -ls ~/mempalace-feed/ # one dir per device, e.g. emb-7kj4vr4g/ -ls ~/mempalace-feed/emb-7kj4vr4g/ # pi_.jsonl, mtimes preserved -``` - -Client side, that needs three variables — and **the third one is the trap**: - -| Variable | Value for this fleet | Why | -| --- | --- | --- | -| `MEMPALACE_PI_SSH_TARGET` | `ecsjper@synlig:/home/ecsjper/mempalace-feed` | where rsync puts the files | -| `MEMPALACE_PI_DEVICE` | e.g. `emb-7kj4vr4g` | inbox subdirectory per machine | -| `MEMPALACE_PI_REMOTE_PATH` | `/home/ecsjper/mempalace-feed` | the inbox **as the server process sees it** | - -The feeder's default for the third is `/data/feed`, which assumes a *containerized* palace server with -the inbox bind-mounted there. **This primary is native (§2.5), so it only ever sees host paths and the -value must equal the path half of the SSH target.** Get it wrong and the failure is quiet in the worst -way: rsync succeeds, the files are all present here, and only the mine fails with -`source directory not found: '/data/feed/'`. - -That is exactly what happened on 2026-08-15, and it went unnoticed for a session because the feeder -decided success with `'"error"' in body` — MCP returns HTTP 200 with the tool's own JSON **escaped** -inside `result.content[].text`, so those bytes are `\"error\"`, the substring never matched, and -`~/.pi/agent/mempalace-catchup.log` printed `Done. Wing 'wing_conversations' updated.` directly under the -error. Fixed in `6e1f4f3`: the envelope is parsed, `--self-test` pins that exact response body, and a -preflight warning fires whenever the ship path and `MEMPALACE_PI_REMOTE_PATH` disagree. - -**Operational notes for this inbox:** - -- Dedup keys on the **absolute source path**, so the inbox path is load-bearing: it must stay stable, or - every transcript re-files under its new name. Migrating it on 2026-08-15 (from the clients' old - container-local stage paths, which arrived with the seed) cost a full re-mine plus - `mempalace_delete_by_source` on 6 old paths — 651 drawers purged, 1243 re-filed. `wing_conversations` - is now keyed entirely on `/home/ecsjper/mempalace-feed//…`. -- A **grown** session is purged and re-filed for the same path (mtime-based), so re-feeding a live - session refreshes it instead of duplicating it. That is why the inbox keeps whole transcripts rather - than deltas — do not "tidy" it by deleting files the palace still references. -- `mempalace_mine` over MCP can **exceed a client's request timeout while the server keeps working and - finishes normally**. A client-side timeout is not a failed mine: check - `SELECT COUNT(*) FROM embedding_metadata WHERE key='source_file' AND string_value LIKE '%'` - (read-only, `file:…?mode=ro`) before retrying anything. -- Health check after any client recreate, from the client: the tail of - `~/.pi/agent/mempalace-catchup.log` should end in `Done. Wing … updated.` with no `error:` line above - it. With the fixed feeder a broken run exits 5 and names the reason. - -## 3. Deliberately NOT done - -> **⚠️ Historical — this section describes 2026-08-10 and is no longer true.** All five items were -> done between 2026-08-12 and 2026-08-14. Kept because the *reasoning* for deferring them is still -> the record of why the order was chosen. Current state per item is inlined below. - -- **Nothing is serving.** No listener on 8765; no mempalace process. Re-verified at the end of the run. - → **Now serving** since 2026-08-12 (`mempalace-serve.service`, `systemctl --user`), reachable at - `https://mempalace.jordbo.se/mcp` via newt/Pangolin. -- **No client `.env` was touched.** Your working setup is exactly as you left it (R6: reversible). - → **One client flipped 2026-08-14**: four variables on EMB-7KJ4VR4G, `docker-compose.yaml` unchanged. - Still reversible in ~30s (§3.8 of [`phase-1-exposure-runbook.md`](./phase-1-exposure-runbook.md)). -- **No data joined.** The palace is empty. The §4.4 join needs the diary-dedup decision (§7.6) first — - replaying diaries today duplicates them, and the primary is the one place that must stay clean. - → **Seeded 2026-08-14** from *one* palace by file-level copy. This sidestepped §7.6 rather than - solving it: a file-level copy replays no diaries, so it cannot duplicate them. **§7.6 is still a - hard blocker for the second machine to join.** -- **nyvaken untouched.** Read nothing, changed nothing. -- **No sudo.** `sudo -n` on synlig requires a password. - -## 4. Tomorrow, in order - -> **2026-08-12: items 1–2 and 5 now have their own runbook —** -> [`phase-1-exposure-runbook.md`](./phase-1-exposure-runbook.md). Pangolin on nyvaken is updated (done), -> and **newt is now installed on synlig and connected to Pangolin (done 2026-08-12)** — so the blocker is -> now item 3, the one `sudo`. That doc also records why per-device Pangolin users are the wrong layer, why -> the HTTPS tunnel and the feeder's SSH path are **not** redundant (§1.3), the client-flip variable trap -> (§3.7), and an additional loopback finding: a loopback bind does not merely 403, it also silently starts -> the server with **no token at all** (auto-minting is gated on the bind being non-loopback). - -1. **Pangolin update on nyvaken** (yours). ✅ done 2026-08-12. -2. ~~**⚠️ synlig has no tunnel client.**~~ ✅ **done 2026-08-12** — newt installed and connected to Pangolin. - (Kept for the reasoning: `docker ps` showed only the Gitea runner and digikam. Pangolin on nyvaken - cannot reach synlig by itself; synlig had to dial out. Easy to miss because Pangolin looks healthy on - its own side — which is also why "connected" is not yet proof it can reach the palace: verify - `172.17.0.1:8765/healthz` from *inside* newt's namespace, exposure runbook §3.3.) Since newt runs in - Docker here, the docker0 bind above is already correct for it. -3. **One sudo, then start** (the unit is already staged; just drop the suffix). ✅ **done 2026-08-12** — - linger enabled, unit enabled, `172.17.0.1:8765/healthz` → `ok`. - ```sh - sudo loginctl enable-linger ecsjper - cd ~/.config/systemd/user && mv mempalace-serve.service.staged mempalace-serve.service - systemctl --user daemon-reload && systemctl --user enable --now mempalace-serve - curl -s 172.17.0.1:8765/healthz # ok - curl -s 127.0.0.1:8765/healthz # NOTHING — refused, exit 7 (not 403; see below) - ss -ltnp | grep 8765 # 172.17.0.1:8765 only - ``` - ⚠ **Corrected 2026-08-12:** this line predicted `403`. The real run returned empty, which is *more* - reassuring. With the docker0-only bind nothing listens on loopback, so the connection is refused before - any header is sent (`%{http_code}` → `000`, `$?` → `7`). The 403 in §2.4 is the **loopback-bind** case: - a server on `127.0.0.1` answering a proxy-forwarded foreign `Host:`. Two different failures that were - collapsed into one expectation here. -4. **Collect the shared token** (auto-minted on first non-loopback start, stable across restarts): - ```sh - cat ~/.mempalace/server/f5d849287f6d73f0141b29d7/token - ``` - That directory name is `sha256(realpath(palace))[:24]` — it changes if the palace path ever changes. -5. **Route it through Pangolin**, then verify `/healthz` end-to-end through the public hostname *before* - pointing any client at it. -6. **Then, and only then**, Phase 1 client flip — one machine first, and remember opencode containers - need the §4.1 sidecar merge (or Phase 1.5) before the `.env` takes effect. -7. **Before the first join:** settle §7.6 diary dedup, then dry-run §4.4 from **one** palace. - > **Correction 2026-08-14 — do NOT verify a join "by checking counts", which is what this item - > originally said.** Counts are not evidence, in either direction. `mempalace status` counts - > **chunk rows**, not logical drawers (3 drawers plus one 2-chunk diary presented as +9), and chunk - > counts legitimately differ between two palaces whenever a drawer was updated on either side, - > because an update re-chunks to the new length and deletes the surplus rows. Diffing chunk-id sets - > is a useful first pass but **over**-reports: a chunk id present on one side only is the ordinary - > signature of an edit, not of loss. This cost real time on 2026-08-14 — a missing - > `chunk_000007` was read as a truncated seed, when in fact the two palaces held two revisions of - > one drawer and nothing was lost. **Adjudicate by fetching the parent drawer on both sides and - > comparing the reassembled `content`.** - -## 5. Rollback - -> **⚠️ STOP — 2026-08-14. Do not run this block as it was originally written.** `~/.mempalace` on -> synlig is now the fleet primary. That tree holds the only central palace (14,803 drawers, seeded -> from EMB-7KJ4VR4G) **and the server's bearer token** at `~/.mempalace/server//token` — the -> single credential every flipped client authenticates with, of which there is no second copy. -> `rm -rf ~/.mempalace` destroys both. The original comment ("empty today — check before running once -> it isn't") is far too soft for a destructive command someone runs under pressure, which is exactly -> why it is being replaced rather than amended. - -Stopping the service is safe and reversible on its own, and is the whole of what "rollback" should -normally mean now: - -```sh -systemctl --user disable --now mempalace-serve # clients fail CLOSED — they lose the palace - # tools; they do NOT fall back to a local palace -``` - -To genuinely decommission the primary, in this order: - -1. Flip every client back first (§3.8 of [`phase-1-exposure-runbook.md`](./phase-1-exposure-runbook.md), - in reverse) so nothing is pointed at a host that is about to lose its palace. -2. Copy `~/.mempalace/palace/` **and** the token file off the host, and verify the copy by comparing - reassembled drawer `content`, not counts (see §4 item 7). -3. Only then remove anything. Never `rsync --delete` into `~/.mempalace` — the token lives inside it. - -The two destructive steps below were written on 2026-08-10, when `~/.mempalace` was genuinely empty. -Kept for the record; **must not be run while the primary is live**: - -```sh -~/.local/bin/uv tool uninstall mempalace -rm -rf ~/.mempalace # ⚠️ DESTROYS THE FLEET PALACE AND THE ONLY TOKEN -``` +If you are looking for the mechanism, the two runbooks above are the same +procedures with `` and `` in place of one site's specifics.