docs: backup and recovery, plus units; move host runbook to a private repo

Adds bin/mempalace-backup and docs/backup-and-recovery.md — the mechanism a
palace actually needs, none of it site-specific.

Why a palace cannot be backed up with cp: it is chroma.sqlite3 (authoritative),
knowledge_graph.sqlite3 (usually WAL, so -wal/-shm make a plain copy a
same-instant gamble), derived HNSW segment dirs, hallways.json, the embedder
descriptor, and a HIDDEN .mempalace/origin.json. Both SQLite files are therefore
copied through the online-backup API. Two bugs are documented because both
produce a backup that looks fine: "$PALACE"/*/ silently skips the hidden dir, and
per-directory rsync collides the identically named data_level0.bin in every HNSW
segment. Treating the palace as one tree fixes both and makes a backup a faithful
palace IMAGE, so restore is a copy rather than a procedure.

Two modes: hot (default, zero downtime, ~4 s, index may lag but SQLite is
authoritative and repair --mode from-sqlite rebuilds) and cold (--cold, ~5 s
downtime, byte-consistent, restart trapped so a failed run still brings the
server back). Verification runs on the COPY — quick_check plus row counts — and
the backup is committed by mv only after it passes, with retention pruned only
after a verified commit, so a broken new backup cannot delete the last good one.

Documented because they are easy to get wrong: the sqlite3 CLI is often absent
where the Python module is present; mempalace_embedder.json must be restored with
the drawers or search silently degrades; a tested restore means running status
AND search against the restored copy, since search is what actually exercises the
index; mempalace-serve is a USER unit, so root systemctl reports "not found";
Persistent=true is what makes a missed window run after boot; and installing
against the system Python couples the palace's availability to distribution
upgrades, with the uv-managed-interpreter fix plus the two PATH traps that bite
scripted upgrades.

Also moves docs/synlig-primary-runbook.md out to a private fleet repository,
leaving a stub that explains the split, since a host inventory is operator data
for one deployment rather than part of a public toolkit. The path stays valid so
existing links do not break. Remaining host references in the README, RFCs and
ARCHITECTURE are left alone deliberately: they are load-bearing prose, contain no
secrets, and are best generalised as they are next edited rather than in one
churn-heavy pass.

contrib/README.md

@@ -37,6 +37,8 @@ Pick **one scheduler** (systemd *or* launchd *or* cron). The opencode and pi job
 | `cron/mempalace-pi-session.cron` | pi → palace | Tue 03:00 |
 | `cron/mempalace-session-devbox.cron` | opencode (devbox) → palace | Mon 03:00 |
 | `systemd/mempalace-serve.service` | **not a mining job** — runs the shared palace *server* | always-on |
+| `systemd/mempalace-backup.{service,timer}` | **not a mining job** — hot palace backup (zero downtime) | daily 04:00 |
+| `systemd/mempalace-backup-cold.{service,timer}` | **not a mining job** — cold palace backup (~5 s downtime, byte-consistent) | Sun 04:30 |
 
 The pi variants are drop-in copies of the opencode variants with script name and schedule updated; the install recipes below apply equally — just swap `mempalace-session` for `mempalace-pi-session` and the schedule day.
 

contrib/systemd/mempalace-backup-cold.service

@@ -0,0 +1,20 @@
+[Unit]
+Description=MemPalace backup (cold — quiesces the server for a byte-consistent copy)
+Documentation=file:%h/mempalace-toolkit/docs/backup-and-recovery.md
+ConditionPathExists=%h/.mempalace/palace
+
+[Service]
+Type=oneshot
+# Cold mode stops mempalace-serve, copies the palace at rest (so the HNSW index
+# is byte-consistent with the SQLite, not merely repairable from it), and starts
+# it again. The restart is trapped on EXIT/INT/TERM inside the script, so an
+# interrupted or failed run still brings the server back up.
+#
+# Measured downtime on a 223 MB palace: ~5 seconds.
+#
+# NOTE: mempalace-serve.service ships a drop-in with Restart=always. That is
+# correct for crash recovery but means an explicit `systemctl --user stop` is the
+# only way to keep it down for the duration of the copy — do not expect a plain
+# SIGTERM to hold it.
+ExecStart=%h/.local/bin/mempalace-backup --cold --keep 7
+Nice=10

contrib/systemd/mempalace-backup-cold.timer

@@ -0,0 +1,12 @@
+[Unit]
+Description=Weekly cold MemPalace backup
+Documentation=file:%h/mempalace-toolkit/docs/backup-and-recovery.md
+
+[Timer]
+# Half an hour after the daily hot backup, so the two never overlap on the
+# palace lock even if the hot run is unusually slow.
+OnCalendar=Sun 04:30:00
+Persistent=true
+
+[Install]
+WantedBy=timers.target

contrib/systemd/mempalace-backup.service

@@ -0,0 +1,19 @@
+[Unit]
+Description=MemPalace backup (hot — zero downtime, SQLite online-backup API)
+Documentation=file:%h/mempalace-toolkit/docs/backup-and-recovery.md
+# Better a skipped run than a run that creates an empty palace somewhere and
+# then dutifully backs it up.
+ConditionPathExists=%h/.mempalace/palace
+
+[Service]
+Type=oneshot
+# Hot mode never stops mempalace-serve. Both SQLite files are copied through the
+# online-backup API (transactionally consistent, WAL-aware); the HNSW index
+# segments are copied live and may lag a few records. That is acceptable because
+# SQLite is authoritative — see docs/backup-and-recovery.md §3.
+ExecStart=%h/.local/bin/mempalace-backup --keep 7
+# Backups are not urgent; the palace server and any running mine are. Yielding
+# CPU and I/O keeps a 4-second backup from lengthening someone's interactive
+# search.
+Nice=10
+IOSchedulingClass=idle

contrib/systemd/mempalace-backup.timer

@@ -0,0 +1,15 @@
+[Unit]
+Description=Daily hot MemPalace backup
+Documentation=file:%h/mempalace-toolkit/docs/backup-and-recovery.md
+
+[Timer]
+OnCalendar=*-*-* 04:00:00
+# If the host was powered off at 04:00, run shortly after the next boot instead
+# of silently skipping that day. Without this, a laptop or an occasionally-off
+# server quietly accumulates days with no backup.
+Persistent=true
+# Avoid colliding with other maintenance that also picked the top of the hour.
+RandomizedDelaySec=300
+
+[Install]
+WantedBy=timers.target