docs: backup and recovery, plus units; move host runbook to a private repo
Adds bin/mempalace-backup and docs/backup-and-recovery.md — the mechanism a palace actually needs, none of it site-specific. Why a palace cannot be backed up with cp: it is chroma.sqlite3 (authoritative), knowledge_graph.sqlite3 (usually WAL, so -wal/-shm make a plain copy a same-instant gamble), derived HNSW segment dirs, hallways.json, the embedder descriptor, and a HIDDEN .mempalace/origin.json. Both SQLite files are therefore copied through the online-backup API. Two bugs are documented because both produce a backup that looks fine: "$PALACE"/*/ silently skips the hidden dir, and per-directory rsync collides the identically named data_level0.bin in every HNSW segment. Treating the palace as one tree fixes both and makes a backup a faithful palace IMAGE, so restore is a copy rather than a procedure. Two modes: hot (default, zero downtime, ~4 s, index may lag but SQLite is authoritative and repair --mode from-sqlite rebuilds) and cold (--cold, ~5 s downtime, byte-consistent, restart trapped so a failed run still brings the server back). Verification runs on the COPY — quick_check plus row counts — and the backup is committed by mv only after it passes, with retention pruned only after a verified commit, so a broken new backup cannot delete the last good one. Documented because they are easy to get wrong: the sqlite3 CLI is often absent where the Python module is present; mempalace_embedder.json must be restored with the drawers or search silently degrades; a tested restore means running status AND search against the restored copy, since search is what actually exercises the index; mempalace-serve is a USER unit, so root systemctl reports "not found"; Persistent=true is what makes a missed window run after boot; and installing against the system Python couples the palace's availability to distribution upgrades, with the uv-managed-interpreter fix plus the two PATH traps that bite scripted upgrades. Also moves docs/synlig-primary-runbook.md out to a private fleet repository, leaving a stub that explains the split, since a host inventory is operator data for one deployment rather than part of a public toolkit. The path stays valid so existing links do not break. Remaining host references in the README, RFCs and ARCHITECTURE are left alone deliberately: they are load-bearing prose, contain no secrets, and are best generalised as they are next edited rather than in one churn-heavy pass.
This commit is contained in:
@@ -0,0 +1,20 @@
|
||||
[Unit]
|
||||
Description=MemPalace backup (cold — quiesces the server for a byte-consistent copy)
|
||||
Documentation=file:%h/mempalace-toolkit/docs/backup-and-recovery.md
|
||||
ConditionPathExists=%h/.mempalace/palace
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
# Cold mode stops mempalace-serve, copies the palace at rest (so the HNSW index
|
||||
# is byte-consistent with the SQLite, not merely repairable from it), and starts
|
||||
# it again. The restart is trapped on EXIT/INT/TERM inside the script, so an
|
||||
# interrupted or failed run still brings the server back up.
|
||||
#
|
||||
# Measured downtime on a 223 MB palace: ~5 seconds.
|
||||
#
|
||||
# NOTE: mempalace-serve.service ships a drop-in with Restart=always. That is
|
||||
# correct for crash recovery but means an explicit `systemctl --user stop` is the
|
||||
# only way to keep it down for the duration of the copy — do not expect a plain
|
||||
# SIGTERM to hold it.
|
||||
ExecStart=%h/.local/bin/mempalace-backup --cold --keep 7
|
||||
Nice=10
|
||||
Reference in New Issue
Block a user