mempalace-toolkit/docs/backup-and-recovery.md

# Backing up and recovering a palace

A palace is not one file, so `cp -r` is not a backup. This document explains what
a palace is made of, why the obvious approaches silently corrupt it, and how
`bin/mempalace-backup` avoids that — plus how to actually restore one and prove
the restore worked.

Host-specific details (which machine is primary, addresses, users, offsite
targets) deliberately live outside this repo. Substitute `<PRIMARY_HOST>` and
`<USER>` throughout.

## 1. What is actually in a palace

Run `ls -a` on a palace directory and you get something like this — sizes are
from a real ~25k-drawer palace, for a sense of proportion:

| Path | Size | What it is |
|---|---|---|
| `chroma.sqlite3` | 167 MB | **The authoritative store.** Drawers, chunks, metadata, embeddings. |
| `<uuid>/` (one per segment) | 50 MB, 2.8 MB | chromadb HNSW index segments (`data_level0.bin`, `index_metadata.pickle`). Derived data. |
| `knowledge_graph.sqlite3` | 64 KB | KG triples + entities. **Usually in WAL mode** — see §2. |
| `knowledge_graph.sqlite3-wal` / `-shm` | varies | Live write-ahead log. Their existence is the whole problem. |
| `hallways.json` | 10.8 MB | Within-wing co-occurrence links, rebuilt by `mempalace hallways`. |
| `known_entities.json`, `mempalace_embedder.json` | < 1 KB | Entity list; **which embedder model this palace was built with**. |
| `.mempalace/origin.json` | small | Provenance. **Hidden** — see the glob trap in §3. |
| `.collection_type_fixed`, `.blob_seq_ids_migrated` | 0 B | Migration marker files. Presence is the signal. |

Two consequences worth internalising:

- **The SQLite files are the truth; the index segments are derived.** If the
  index is stale or damaged you can rebuild it from SQLite. The reverse is not
  true. This asymmetry is what makes a zero-downtime backup possible at all.
- **`mempalace_embedder.json` is not optional.** Restoring drawers next to a
  *different* embedder model gives you a palace whose stored vectors and future
  queries live in different spaces — search silently degrades rather than
  failing. Back it up, and check it after a restore.

## 2. Why `cp` is unsafe, specifically

SQLite in WAL mode keeps recent commits in `-wal` and shared state in `-shm`.
Copying those three files with `cp` gives you no guarantee they are from the same
instant: the copier can read `chroma.sqlite3` before a checkpoint and `-wal`
after it. The result usually *opens* — which is the dangerous part — and then
misbehaves later, or fails `PRAGMA integrity_check` under load.

`rsync` has exactly the same problem: it is not a snapshot, it is a sequence of
reads.

The supported approach is SQLite's **online backup API**, which takes a
transactionally consistent copy of a live database while writers continue:

```python
import sqlite3
src = sqlite3.connect(f"file:{src_path}?mode=ro", uri=True)
dst = sqlite3.connect(dst_path)
with dst:
    src.backup(dst)          # WAL-aware, consistent, no writer downtime
```

Two practical notes:

- **`VACUUM INTO` is a fine alternative in principle**, but it needs either the
  `sqlite3` CLI or a Python build new enough for it. Do not assume the CLI is
  present: a stock Ubuntu server often has the Python `sqlite3` *module* and no
  `sqlite3` *binary*. Check with `command -v sqlite3` before writing tooling
  that depends on it.
- Open the source with `mode=ro` so a backup can never be the thing that
  creates a `-wal` file in a directory you meant only to read.

## 3. Two backup modes, and the honest tradeoff

`bin/mempalace-backup` implements both. Neither is strictly better.

### hot (default) — zero downtime

1. `rsync` the whole palace tree **excluding** the SQLite files.
2. Write both SQLite files via the online-backup API.

The server never stops. The index segments are copied while they may be being
written, so in principle they can lag the SQLite by a few records. That is
acceptable *because the SQLite is authoritative*: if search misbehaves after a
restore, rebuild the index (§5).

Measured on a 223 MB palace: **~4 s**.

### cold (`--cold`) — byte-consistent

1. Stop the server (`systemctl --user stop mempalace-serve`).
2. Copy everything, including the index, at rest.
3. Restart the server.

The restart is in a `trap … EXIT INT TERM`, so an interruption mid-copy still
brings the server back. Measured downtime: **~5 s**.

Recommended schedule: **hot daily, cold weekly**. The cold copy is your
byte-exact reference; the hot copies bound your data loss between them.

### The glob trap that cost two rewrites

Two bugs are worth naming because both produce a backup that looks fine:

- **`"$PALACE"/*/` skips hidden directories.** `.mempalace/origin.json` is
  silently omitted, so the restored palace loses its provenance. Shell globs do
  not match dotfiles without `dotglob`/`GLOBIGNORE` fiddling.
- **Per-directory rsync collides identically named files.** Every HNSW segment
  contains a `data_level0.bin`. Copying segment dirs into one destination
  merges them and the last write wins — a corrupt index that still loads.

The fix for both is to treat the palace as **one tree**: a single
`rsync -a --exclude` of the SQLite files, so a backup directory is a faithful
*image* of the palace. That makes restore a copy rather than a procedure.

## 4. Verify the copy, not the original

A backup you have not read back is a hypothesis. After copying, and **before**
committing the backup or pruning old ones, the tool runs against the *copy*:

- `PRAGMA quick_check` on both databases
- row counts (`drawers`, `triples`, `entities`)
- a `MANIFEST.json` recording counts, sizes, mode, tool version, and the
  index-consistency caveat for hot backups

Sequencing matters as much as the checks:

1. write into `<dest>.incomplete/`
2. verify
3. `mv` into place (atomic commit)
4. **only now** prune old backups to the retention limit

Retention pruning that runs before verification can delete your last good
backup on the night the new one is broken. A leftover `.incomplete` directory is
a visible symptom, not silent data loss, so the tool reports stale ones.

## 5. Restoring

Because a backup is a palace image, restore is a copy:

```bash
systemctl --user stop mempalace-serve
mv ~/.mempalace/palace ~/.mempalace/palace.broken-$(date -u +%Y%m%dT%H%M%SZ)
mkdir -p ~/.mempalace/palace
rsync -a --exclude MANIFEST.json <backup-dir>/ ~/.mempalace/palace/
systemctl --user start mempalace-serve
```

Keep the old directory until the restore is proven. It costs disk and buys you a
second attempt.

**If the restore came from a hot backup and search behaves oddly**, rebuild the
derived index from the authoritative SQLite:

```bash
mempalace repair --mode from-sqlite
```

Note that a full `repair --mode from-sqlite --archive-existing` re-creates the
chromadb collections, which means new collections pick up **current** HNSW
tuning defaults. On palaces created by older versions this is also the only way
to escape legacy `hnsw:batch_size=2 / hnsw:sync_threshold=2` settings, which sync
the index every two records. Check yours with:

```sql
SELECT c.name, k.key, k.str_value FROM collections c
  JOIN collection_metadata k ON k.collection_id = c.id
 WHERE k.key LIKE 'hnsw%';
```

## 6. Prove the restore, don't assume it

"The files copied" is not a tested restore. Restore to a throwaway directory and
make the palace *answer questions*:

```bash
rsync -a --exclude MANIFEST.json <backup-dir>/ /tmp/restore-test/
export MEMPALACE_PALACE_PATH=/tmp/restore-test
mempalace status                      # drawer count — compare to the source
mempalace search "some phrase you know is in there"
```

`status` proves the schema and metadata survived. **`search` is the stronger
test**: it exercises the HNSW segments, so it is what tells you whether a
*hot* backup's index came across usable. On the reference run above it returned
correctly ranked results (`cosine_sim=0.398 bm25=2.992`) from a zero-downtime
backup — the theoretical index skew did not materialise.

Do this at least once when you set backups up, and again after any version
upgrade that touches storage.

## 7. Scheduling that survives a reboot

Use **systemd user timers** with lingering enabled, so backups run without
anyone logging in:

```bash
loginctl enable-linger <USER>          # required, or user units die at logout
systemctl --user enable --now mempalace-backup.timer mempalace-backup-cold.timer
systemctl --user list-timers 'mempalace*'
```

Units are in [`contrib/systemd/`](../contrib/systemd/). Two settings earn their
keep:

- **`Persistent=true`** — a timer whose window was missed (host powered off)
  runs shortly after boot instead of skipping the day.
- **`RandomizedDelaySec`** — avoids a thundering herd if other maintenance
  shares the hour.

`mempalace-serve` is itself a **user** unit in the reference deployment. That is
worth stating explicitly because `systemctl status mempalace-serve` as root
reports "not found" and invites the conclusion that the server is not installed.
Always `systemctl --user`.

### The interpreter is part of your durability story

If the tool is installed against the **system** Python (`/usr/bin/python3`), then
a distribution release upgrade — which can remove or replace that interpreter —
breaks the server and its backups together, at the moment you least want it.
Install against a **managed** interpreter instead:

```bash
uv python install 3.12
uv tool install --force --python "$HOME/.local/bin/python3.12" 'mempalace==<version>'
systemctl --user restart mempalace-serve
```

Now an OS upgrade cannot take the interpreter away. Two gotchas:

- `uv` is typically in `~/.local/bin`, which is **not** on a non-interactive
  ssh PATH. Scripted upgrades must `export PATH="$HOME/.local/bin:$PATH"`.
- `uv python list` prints `$HOME`-relative paths, so a regex anchored on a
  leading `/` matches inside `.local/share` and yields a bogus absolute path.
  Resolve the shim and assert on it:
  `readlink -f ~/.local/bin/python3.12 | grep share/uv/python`.

**Restart the server after any upgrade.** Recent versions refuse mutating tools
when the served library drifts from what is installed on disk, and
`mempalace_reconnect` cannot clear that — it reopens the database but cannot
reload Python modules.

## 8. Sizing and offsite

At ~225 MB per copy, 7 hot + 7 cold is ~3 GB — trivial against any modern disk,
so prefer generous retention over clever incremental schemes. If space is
genuinely tight, `zstd -T0` on the SQLite files compresses well, at the cost of
making a backup no longer a directly-mountable palace image.

**Local backups protect against corruption and mistakes, not against losing the
host.** Until a copy leaves the machine, a single disk or a single `rm` ends the
palace. Whatever offsite target you choose (object store, NAS, another host),
two rules matter more than the choice:

- **Put a copy of the restore procedure with the backups.** Documentation that
  lives only inside the thing you are restoring is not available at the moment
  you need it.
- **Verify the offsite copy by restoring from it**, not by checking that the
  upload exited 0.

## 9. Usage

```
mempalace-backup [--cold] [--dest DIR] [--keep N] [--palace DIR] [--dry-run]
mempalace-backup --verify <backup-dir>
```

| Flag | Meaning |
|---|---|
| `--cold` | Quiesce the server for a byte-consistent copy (default: hot) |
| `--dest DIR` | Backup root (default `~/backups/mempalace`) |
| `--keep N` | Retention count, pruned only after a verified commit (default 7) |
| `--palace DIR` | Palace to back up (default `~/.mempalace/palace`) |
| `--verify DIR` | Re-verify an existing backup and print its manifest |
| `--dry-run` | Report what would happen, write nothing |

Backups are named `mp-<UTC timestamp>-<hot|cold>`, so they sort
chronologically and their mode is visible without opening the manifest.