docs: backup and recovery, plus units; move host runbook to a private repo
Adds bin/mempalace-backup and docs/backup-and-recovery.md — the mechanism a palace actually needs, none of it site-specific. Why a palace cannot be backed up with cp: it is chroma.sqlite3 (authoritative), knowledge_graph.sqlite3 (usually WAL, so -wal/-shm make a plain copy a same-instant gamble), derived HNSW segment dirs, hallways.json, the embedder descriptor, and a HIDDEN .mempalace/origin.json. Both SQLite files are therefore copied through the online-backup API. Two bugs are documented because both produce a backup that looks fine: "$PALACE"/*/ silently skips the hidden dir, and per-directory rsync collides the identically named data_level0.bin in every HNSW segment. Treating the palace as one tree fixes both and makes a backup a faithful palace IMAGE, so restore is a copy rather than a procedure. Two modes: hot (default, zero downtime, ~4 s, index may lag but SQLite is authoritative and repair --mode from-sqlite rebuilds) and cold (--cold, ~5 s downtime, byte-consistent, restart trapped so a failed run still brings the server back). Verification runs on the COPY — quick_check plus row counts — and the backup is committed by mv only after it passes, with retention pruned only after a verified commit, so a broken new backup cannot delete the last good one. Documented because they are easy to get wrong: the sqlite3 CLI is often absent where the Python module is present; mempalace_embedder.json must be restored with the drawers or search silently degrades; a tested restore means running status AND search against the restored copy, since search is what actually exercises the index; mempalace-serve is a USER unit, so root systemctl reports "not found"; Persistent=true is what makes a missed window run after boot; and installing against the system Python couples the palace's availability to distribution upgrades, with the uv-managed-interpreter fix plus the two PATH traps that bite scripted upgrades. Also moves docs/synlig-primary-runbook.md out to a private fleet repository, leaving a stub that explains the split, since a host inventory is operator data for one deployment rather than part of a public toolkit. The path stays valid so existing links do not break. Remaining host references in the README, RFCs and ARCHITECTURE are left alone deliberately: they are load-bearing prose, contain no secrets, and are best generalised as they are next edited rather than in one churn-heavy pass.
This commit is contained in:
Joakim Persson
@@ -0,0 +1,275 @@
|
||||
# Backing up and recovering a palace
|
||||
|
||||
A palace is not one file, so `cp -r` is not a backup. This document explains what
|
||||
a palace is made of, why the obvious approaches silently corrupt it, and how
|
||||
`bin/mempalace-backup` avoids that — plus how to actually restore one and prove
|
||||
the restore worked.
|
||||
|
||||
Host-specific details (which machine is primary, addresses, users, offsite
|
||||
targets) deliberately live outside this repo. Substitute `<PRIMARY_HOST>` and
|
||||
`<USER>` throughout.
|
||||
|
||||
## 1. What is actually in a palace
|
||||
|
||||
Run `ls -a` on a palace directory and you get something like this — sizes are
|
||||
from a real ~25k-drawer palace, for a sense of proportion:
|
||||
|
||||
| Path | Size | What it is |
|
||||
|---|---|---|
|
||||
| `chroma.sqlite3` | 167 MB | **The authoritative store.** Drawers, chunks, metadata, embeddings. |
|
||||
| `<uuid>/` (one per segment) | 50 MB, 2.8 MB | chromadb HNSW index segments (`data_level0.bin`, `index_metadata.pickle`). Derived data. |
|
||||
| `knowledge_graph.sqlite3` | 64 KB | KG triples + entities. **Usually in WAL mode** — see §2. |
|
||||
| `knowledge_graph.sqlite3-wal` / `-shm` | varies | Live write-ahead log. Their existence is the whole problem. |
|
||||
| `hallways.json` | 10.8 MB | Within-wing co-occurrence links, rebuilt by `mempalace hallways`. |
|
||||
| `known_entities.json`, `mempalace_embedder.json` | < 1 KB | Entity list; **which embedder model this palace was built with**. |
|
||||
| `.mempalace/origin.json` | small | Provenance. **Hidden** — see the glob trap in §3. |
|
||||
| `.collection_type_fixed`, `.blob_seq_ids_migrated` | 0 B | Migration marker files. Presence is the signal. |
|
||||
|
||||
Two consequences worth internalising:
|
||||
|
||||
- **The SQLite files are the truth; the index segments are derived.** If the
|
||||
index is stale or damaged you can rebuild it from SQLite. The reverse is not
|
||||
true. This asymmetry is what makes a zero-downtime backup possible at all.
|
||||
- **`mempalace_embedder.json` is not optional.** Restoring drawers next to a
|
||||
*different* embedder model gives you a palace whose stored vectors and future
|
||||
queries live in different spaces — search silently degrades rather than
|
||||
failing. Back it up, and check it after a restore.
|
||||
|
||||
## 2. Why `cp` is unsafe, specifically
|
||||
|
||||
SQLite in WAL mode keeps recent commits in `-wal` and shared state in `-shm`.
|
||||
Copying those three files with `cp` gives you no guarantee they are from the same
|
||||
instant: the copier can read `chroma.sqlite3` before a checkpoint and `-wal`
|
||||
after it. The result usually *opens* — which is the dangerous part — and then
|
||||
misbehaves later, or fails `PRAGMA integrity_check` under load.
|
||||
|
||||
`rsync` has exactly the same problem: it is not a snapshot, it is a sequence of
|
||||
reads.
|
||||
|
||||
The supported approach is SQLite's **online backup API**, which takes a
|
||||
transactionally consistent copy of a live database while writers continue:
|
||||
|
||||
```python
|
||||
import sqlite3
|
||||
src = sqlite3.connect(f"file:{src_path}?mode=ro", uri=True)
|
||||
dst = sqlite3.connect(dst_path)
|
||||
with dst:
|
||||
src.backup(dst) # WAL-aware, consistent, no writer downtime
|
||||
```
|
||||
|
||||
Two practical notes:
|
||||
|
||||
- **`VACUUM INTO` is a fine alternative in principle**, but it needs either the
|
||||
`sqlite3` CLI or a Python build new enough for it. Do not assume the CLI is
|
||||
present: a stock Ubuntu server often has the Python `sqlite3` *module* and no
|
||||
`sqlite3` *binary*. Check with `command -v sqlite3` before writing tooling
|
||||
that depends on it.
|
||||
- Open the source with `mode=ro` so a backup can never be the thing that
|
||||
creates a `-wal` file in a directory you meant only to read.
|
||||
|
||||
## 3. Two backup modes, and the honest tradeoff
|
||||
|
||||
`bin/mempalace-backup` implements both. Neither is strictly better.
|
||||
|
||||
### hot (default) — zero downtime
|
||||
|
||||
1. `rsync` the whole palace tree **excluding** the SQLite files.
|
||||
2. Write both SQLite files via the online-backup API.
|
||||
|
||||
The server never stops. The index segments are copied while they may be being
|
||||
written, so in principle they can lag the SQLite by a few records. That is
|
||||
acceptable *because the SQLite is authoritative*: if search misbehaves after a
|
||||
restore, rebuild the index (§5).
|
||||
|
||||
Measured on a 223 MB palace: **~4 s**.
|
||||
|
||||
### cold (`--cold`) — byte-consistent
|
||||
|
||||
1. Stop the server (`systemctl --user stop mempalace-serve`).
|
||||
2. Copy everything, including the index, at rest.
|
||||
3. Restart the server.
|
||||
|
||||
The restart is in a `trap … EXIT INT TERM`, so an interruption mid-copy still
|
||||
brings the server back. Measured downtime: **~5 s**.
|
||||
|
||||
Recommended schedule: **hot daily, cold weekly**. The cold copy is your
|
||||
byte-exact reference; the hot copies bound your data loss between them.
|
||||
|
||||
### The glob trap that cost two rewrites
|
||||
|
||||
Two bugs are worth naming because both produce a backup that looks fine:
|
||||
|
||||
- **`"$PALACE"/*/` skips hidden directories.** `.mempalace/origin.json` is
|
||||
silently omitted, so the restored palace loses its provenance. Shell globs do
|
||||
not match dotfiles without `dotglob`/`GLOBIGNORE` fiddling.
|
||||
- **Per-directory rsync collides identically named files.** Every HNSW segment
|
||||
contains a `data_level0.bin`. Copying segment dirs into one destination
|
||||
merges them and the last write wins — a corrupt index that still loads.
|
||||
|
||||
The fix for both is to treat the palace as **one tree**: a single
|
||||
`rsync -a --exclude` of the SQLite files, so a backup directory is a faithful
|
||||
*image* of the palace. That makes restore a copy rather than a procedure.
|
||||
|
||||
## 4. Verify the copy, not the original
|
||||
|
||||
A backup you have not read back is a hypothesis. After copying, and **before**
|
||||
committing the backup or pruning old ones, the tool runs against the *copy*:
|
||||
|
||||
- `PRAGMA quick_check` on both databases
|
||||
- row counts (`drawers`, `triples`, `entities`)
|
||||
- a `MANIFEST.json` recording counts, sizes, mode, tool version, and the
|
||||
index-consistency caveat for hot backups
|
||||
|
||||
Sequencing matters as much as the checks:
|
||||
|
||||
1. write into `<dest>.incomplete/`
|
||||
2. verify
|
||||
3. `mv` into place (atomic commit)
|
||||
4. **only now** prune old backups to the retention limit
|
||||
|
||||
Retention pruning that runs before verification can delete your last good
|
||||
backup on the night the new one is broken. A leftover `.incomplete` directory is
|
||||
a visible symptom, not silent data loss, so the tool reports stale ones.
|
||||
|
||||
## 5. Restoring
|
||||
|
||||
Because a backup is a palace image, restore is a copy:
|
||||
|
||||
```bash
|
||||
systemctl --user stop mempalace-serve
|
||||
mv ~/.mempalace/palace ~/.mempalace/palace.broken-$(date -u +%Y%m%dT%H%M%SZ)
|
||||
mkdir -p ~/.mempalace/palace
|
||||
rsync -a --exclude MANIFEST.json <backup-dir>/ ~/.mempalace/palace/
|
||||
systemctl --user start mempalace-serve
|
||||
```
|
||||
|
||||
Keep the old directory until the restore is proven. It costs disk and buys you a
|
||||
second attempt.
|
||||
|
||||
**If the restore came from a hot backup and search behaves oddly**, rebuild the
|
||||
derived index from the authoritative SQLite:
|
||||
|
||||
```bash
|
||||
mempalace repair --mode from-sqlite
|
||||
```
|
||||
|
||||
Note that a full `repair --mode from-sqlite --archive-existing` re-creates the
|
||||
chromadb collections, which means new collections pick up **current** HNSW
|
||||
tuning defaults. On palaces created by older versions this is also the only way
|
||||
to escape legacy `hnsw:batch_size=2 / hnsw:sync_threshold=2` settings, which sync
|
||||
the index every two records. Check yours with:
|
||||
|
||||
```sql
|
||||
SELECT c.name, k.key, k.str_value FROM collections c
|
||||
JOIN collection_metadata k ON k.collection_id = c.id
|
||||
WHERE k.key LIKE 'hnsw%';
|
||||
```
|
||||
|
||||
## 6. Prove the restore, don't assume it
|
||||
|
||||
"The files copied" is not a tested restore. Restore to a throwaway directory and
|
||||
make the palace *answer questions*:
|
||||
|
||||
```bash
|
||||
rsync -a --exclude MANIFEST.json <backup-dir>/ /tmp/restore-test/
|
||||
export MEMPALACE_PALACE_PATH=/tmp/restore-test
|
||||
mempalace status # drawer count — compare to the source
|
||||
mempalace search "some phrase you know is in there"
|
||||
```
|
||||
|
||||
`status` proves the schema and metadata survived. **`search` is the stronger
|
||||
test**: it exercises the HNSW segments, so it is what tells you whether a
|
||||
*hot* backup's index came across usable. On the reference run above it returned
|
||||
correctly ranked results (`cosine_sim=0.398 bm25=2.992`) from a zero-downtime
|
||||
backup — the theoretical index skew did not materialise.
|
||||
|
||||
Do this at least once when you set backups up, and again after any version
|
||||
upgrade that touches storage.
|
||||
|
||||
## 7. Scheduling that survives a reboot
|
||||
|
||||
Use **systemd user timers** with lingering enabled, so backups run without
|
||||
anyone logging in:
|
||||
|
||||
```bash
|
||||
loginctl enable-linger <USER> # required, or user units die at logout
|
||||
systemctl --user enable --now mempalace-backup.timer mempalace-backup-cold.timer
|
||||
systemctl --user list-timers 'mempalace*'
|
||||
```
|
||||
|
||||
Units are in [`contrib/systemd/`](../contrib/systemd/). Two settings earn their
|
||||
keep:
|
||||
|
||||
- **`Persistent=true`** — a timer whose window was missed (host powered off)
|
||||
runs shortly after boot instead of skipping the day.
|
||||
- **`RandomizedDelaySec`** — avoids a thundering herd if other maintenance
|
||||
shares the hour.
|
||||
|
||||
`mempalace-serve` is itself a **user** unit in the reference deployment. That is
|
||||
worth stating explicitly because `systemctl status mempalace-serve` as root
|
||||
reports "not found" and invites the conclusion that the server is not installed.
|
||||
Always `systemctl --user`.
|
||||
|
||||
### The interpreter is part of your durability story
|
||||
|
||||
If the tool is installed against the **system** Python (`/usr/bin/python3`), then
|
||||
a distribution release upgrade — which can remove or replace that interpreter —
|
||||
breaks the server and its backups together, at the moment you least want it.
|
||||
Install against a **managed** interpreter instead:
|
||||
|
||||
```bash
|
||||
uv python install 3.12
|
||||
uv tool install --force --python "$HOME/.local/bin/python3.12" 'mempalace==<version>'
|
||||
systemctl --user restart mempalace-serve
|
||||
```
|
||||
|
||||
Now an OS upgrade cannot take the interpreter away. Two gotchas:
|
||||
|
||||
- `uv` is typically in `~/.local/bin`, which is **not** on a non-interactive
|
||||
ssh PATH. Scripted upgrades must `export PATH="$HOME/.local/bin:$PATH"`.
|
||||
- `uv python list` prints `$HOME`-relative paths, so a regex anchored on a
|
||||
leading `/` matches inside `.local/share` and yields a bogus absolute path.
|
||||
Resolve the shim and assert on it:
|
||||
`readlink -f ~/.local/bin/python3.12 | grep share/uv/python`.
|
||||
|
||||
**Restart the server after any upgrade.** Recent versions refuse mutating tools
|
||||
when the served library drifts from what is installed on disk, and
|
||||
`mempalace_reconnect` cannot clear that — it reopens the database but cannot
|
||||
reload Python modules.
|
||||
|
||||
## 8. Sizing and offsite
|
||||
|
||||
At ~225 MB per copy, 7 hot + 7 cold is ~3 GB — trivial against any modern disk,
|
||||
so prefer generous retention over clever incremental schemes. If space is
|
||||
genuinely tight, `zstd -T0` on the SQLite files compresses well, at the cost of
|
||||
making a backup no longer a directly-mountable palace image.
|
||||
|
||||
**Local backups protect against corruption and mistakes, not against losing the
|
||||
host.** Until a copy leaves the machine, a single disk or a single `rm` ends the
|
||||
palace. Whatever offsite target you choose (object store, NAS, another host),
|
||||
two rules matter more than the choice:
|
||||
|
||||
- **Put a copy of the restore procedure with the backups.** Documentation that
|
||||
lives only inside the thing you are restoring is not available at the moment
|
||||
you need it.
|
||||
- **Verify the offsite copy by restoring from it**, not by checking that the
|
||||
upload exited 0.
|
||||
|
||||
## 9. Usage
|
||||
|
||||
```
|
||||
mempalace-backup [--cold] [--dest DIR] [--keep N] [--palace DIR] [--dry-run]
|
||||
mempalace-backup --verify <backup-dir>
|
||||
```
|
||||
|
||||
| Flag | Meaning |
|
||||
|---|---|
|
||||
| `--cold` | Quiesce the server for a byte-consistent copy (default: hot) |
|
||||
| `--dest DIR` | Backup root (default `~/backups/mempalace`) |
|
||||
| `--keep N` | Retention count, pruned only after a verified commit (default 7) |
|
||||
| `--palace DIR` | Palace to back up (default `~/.mempalace/palace`) |
|
||||
| `--verify DIR` | Re-verify an existing backup and print its manifest |
|
||||
| `--dry-run` | Report what would happen, write nothing |
|
||||
|
||||
Backups are named `mp-<UTC timestamp>-<hot|cold>`, so they sort
|
||||
chronologically and their mode is visible without opening the manifest.
|
||||
Reference in New Issue
Block a user