mempalace-toolkit/docs/backup-and-recovery.md

Backing up and recovering a palace

A palace is not one file, so cp -r is not a backup. This document explains what a palace is made of, why the obvious approaches silently corrupt it, and how bin/mempalace-backup avoids that — plus how to actually restore one and prove the restore worked.

Host-specific details (which machine is primary, addresses, users, offsite targets) deliberately live outside this repo. Substitute <PRIMARY_HOST> and <USER> throughout.

1. What is actually in a palace

Run ls -a on a palace directory and you get something like this — sizes are from a real ~25k-drawer palace, for a sense of proportion:

Path	Size	What it is
chroma.sqlite3	167 MB	The authoritative store. Drawers, chunks, metadata, embeddings.
<uuid>/ (one per segment)	50 MB, 2.8 MB	chromadb HNSW index segments (data_level0.bin, index_metadata.pickle). Derived data.
knowledge_graph.sqlite3	64 KB	KG triples + entities. Usually in WAL mode — see §2.
knowledge_graph.sqlite3-wal / -shm	varies	Live write-ahead log. Their existence is the whole problem.
hallways.json	10.8 MB	Within-wing co-occurrence links, rebuilt by mempalace hallways.
known_entities.json, mempalace_embedder.json	< 1 KB	Entity list; which embedder model this palace was built with.
.mempalace/origin.json	small	Provenance. Hidden — see the glob trap in §3.
.collection_type_fixed, .blob_seq_ids_migrated	0 B	Migration marker files. Presence is the signal.

Two consequences worth internalising:

• The SQLite files are the truth; the index segments are derived. If the index is stale or damaged you can rebuild it from SQLite. The reverse is not true. This asymmetry is what makes a zero-downtime backup possible at all.
• mempalace_embedder.json is not optional. Restoring drawers next to a different embedder model gives you a palace whose stored vectors and future queries live in different spaces — search silently degrades rather than failing. Back it up, and check it after a restore.

2. Why cp is unsafe, specifically

SQLite in WAL mode keeps recent commits in -wal and shared state in -shm. Copying those three files with cp gives you no guarantee they are from the same instant: the copier can read chroma.sqlite3 before a checkpoint and -wal after it. The result usually opens — which is the dangerous part — and then misbehaves later, or fails PRAGMA integrity_check under load.

rsync has exactly the same problem: it is not a snapshot, it is a sequence of reads.

The supported approach is SQLite's online backup API, which takes a transactionally consistent copy of a live database while writers continue:

import sqlite3
src = sqlite3.connect(f"file:{src_path}?mode=ro", uri=True)
dst = sqlite3.connect(dst_path)
with dst:
    src.backup(dst)          # WAL-aware, consistent, no writer downtime

Two practical notes:

• VACUUM INTO is a fine alternative in principle, but it needs either the sqlite3 CLI or a Python build new enough for it. Do not assume the CLI is present: a stock Ubuntu server often has the Python sqlite3 module and no sqlite3 binary. Check with command -v sqlite3 before writing tooling that depends on it.
• Open the source with mode=ro so a backup can never be the thing that creates a -wal file in a directory you meant only to read.

3. Two backup modes, and the honest tradeoff

bin/mempalace-backup implements both. Neither is strictly better.

hot (default) — zero downtime

1. rsync the whole palace tree excluding the SQLite files.
2. Write both SQLite files via the online-backup API.

The server never stops. The index segments are copied while they may be being written, so in principle they can lag the SQLite by a few records. That is acceptable because the SQLite is authoritative: if search misbehaves after a restore, rebuild the index (§5).

Measured on a 223 MB palace: ~4 s.

cold (--cold) — byte-consistent

1. Stop the server (systemctl --user stop mempalace-serve).
2. Copy everything, including the index, at rest.
3. Restart the server.

The restart is in a trap … EXIT INT TERM, so an interruption mid-copy still brings the server back. Measured downtime: ~5 s.

Recommended schedule: hot daily, cold weekly. The cold copy is your byte-exact reference; the hot copies bound your data loss between them.

The glob trap that cost two rewrites

Two bugs are worth naming because both produce a backup that looks fine:

• "$PALACE"/*/ skips hidden directories. .mempalace/origin.json is silently omitted, so the restored palace loses its provenance. Shell globs do not match dotfiles without dotglob/GLOBIGNORE fiddling.
• Per-directory rsync collides identically named files. Every HNSW segment contains a data_level0.bin. Copying segment dirs into one destination merges them and the last write wins — a corrupt index that still loads.

The fix for both is to treat the palace as one tree: a single rsync -a --exclude of the SQLite files, so a backup directory is a faithful image of the palace. That makes restore a copy rather than a procedure.

4. Verify the copy, not the original

A backup you have not read back is a hypothesis. After copying, and before committing the backup or pruning old ones, the tool runs against the copy:

• PRAGMA quick_check on both databases
• row counts (drawers, triples, entities)
• a MANIFEST.json recording counts, sizes, mode, tool version, and the index-consistency caveat for hot backups

Sequencing matters as much as the checks:

1. write into <dest>.incomplete/
2. verify
3. mv into place (atomic commit)
4. only now prune old backups to the retention limit

Retention pruning that runs before verification can delete your last good backup on the night the new one is broken. A leftover .incomplete directory is a visible symptom, not silent data loss, so the tool reports stale ones.

5. Restoring

Because a backup is a palace image, restore is a copy:

systemctl --user stop mempalace-serve
mv ~/.mempalace/palace ~/.mempalace/palace.broken-$(date -u +%Y%m%dT%H%M%SZ)
mkdir -p ~/.mempalace/palace
rsync -a --exclude MANIFEST.json <backup-dir>/ ~/.mempalace/palace/
systemctl --user start mempalace-serve

Keep the old directory until the restore is proven. It costs disk and buys you a second attempt.

If the restore came from a hot backup and search behaves oddly, rebuild the derived index from the authoritative SQLite:

mempalace repair --mode from-sqlite

Note that a full repair --mode from-sqlite --archive-existing re-creates the chromadb collections, which means new collections pick up current HNSW tuning defaults. On palaces created by older versions this is also the only way to escape legacy hnsw:batch_size=2 / hnsw:sync_threshold=2 settings, which sync the index every two records. Check yours with:

SELECT c.name, k.key, k.str_value FROM collections c
  JOIN collection_metadata k ON k.collection_id = c.id
 WHERE k.key LIKE 'hnsw%';

6. Prove the restore, don't assume it

"The files copied" is not a tested restore. Restore to a throwaway directory and make the palace answer questions:

rsync -a --exclude MANIFEST.json <backup-dir>/ /tmp/restore-test/
export MEMPALACE_PALACE_PATH=/tmp/restore-test
mempalace status                      # drawer count — compare to the source
mempalace search "some phrase you know is in there"

status proves the schema and metadata survived. search is the stronger test: it exercises the HNSW segments, so it is what tells you whether a hot backup's index came across usable. On the reference run above it returned correctly ranked results (cosine_sim=0.398 bm25=2.992) from a zero-downtime backup — the theoretical index skew did not materialise.

Do this at least once when you set backups up, and again after any version upgrade that touches storage.

7. Scheduling that survives a reboot

Use systemd user timers with lingering enabled, so backups run without anyone logging in:

loginctl enable-linger <USER>          # required, or user units die at logout
systemctl --user enable --now mempalace-backup.timer mempalace-backup-cold.timer
systemctl --user list-timers 'mempalace*'

Units are in contrib/systemd/. Two settings earn their keep:

• Persistent=true — a timer whose window was missed (host powered off) runs shortly after boot instead of skipping the day.
• RandomizedDelaySec — avoids a thundering herd if other maintenance shares the hour.

mempalace-serve is itself a user unit in the reference deployment. That is worth stating explicitly because systemctl status mempalace-serve as root reports "not found" and invites the conclusion that the server is not installed. Always systemctl --user.

The interpreter is part of your durability story

If the tool is installed against the system Python (/usr/bin/python3), then a distribution release upgrade — which can remove or replace that interpreter — breaks the server and its backups together, at the moment you least want it. Install against a managed interpreter instead:

uv python install 3.12
uv tool install --force --python "$HOME/.local/bin/python3.12" 'mempalace==<version>'
systemctl --user restart mempalace-serve

Now an OS upgrade cannot take the interpreter away. Two gotchas:

• uv is typically in ~/.local/bin, which is not on a non-interactive ssh PATH. Scripted upgrades must export PATH="$HOME/.local/bin:$PATH".
• uv python list prints $HOME-relative paths, so a regex anchored on a leading / matches inside .local/share and yields a bogus absolute path. Resolve the shim and assert on it: readlink -f ~/.local/bin/python3.12 | grep share/uv/python.

Restart the server after any upgrade. Recent versions refuse mutating tools when the served library drifts from what is installed on disk, and mempalace_reconnect cannot clear that — it reopens the database but cannot reload Python modules.

8. Sizing and offsite

At ~225 MB per copy, 7 hot + 7 cold is ~3 GB — trivial against any modern disk, so prefer generous retention over clever incremental schemes. If space is genuinely tight, zstd -T0 on the SQLite files compresses well, at the cost of making a backup no longer a directly-mountable palace image.

Local backups protect against corruption and mistakes, not against losing the host. Until a copy leaves the machine, a single disk or a single rm ends the palace. Whatever offsite target you choose (object store, NAS, another host), two rules matter more than the choice:

• Put a copy of the restore procedure with the backups. Documentation that lives only inside the thing you are restoring is not available at the moment you need it.
• Verify the offsite copy by restoring from it, not by checking that the upload exited 0.

9. Usage

mempalace-backup [--cold] [--dest DIR] [--keep N] [--palace DIR] [--dry-run]
mempalace-backup --verify <backup-dir>

Flag	Meaning
--cold	Quiesce the server for a byte-consistent copy (default: hot)
--dest DIR	Backup root (default ~/backups/mempalace)
--keep N	Retention count, pruned only after a verified commit (default 7)
--palace DIR	Palace to back up (default ~/.mempalace/palace)
--verify DIR	Re-verify an existing backup and print its manifest
--dry-run	Report what would happen, write nothing

Backups are named mp-<UTC timestamp>-<hot|cold>, so they sort chronologically and their mode is visible without opening the manifest.