joakimp/mempalace-toolkit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: backup and recovery, plus units; move host runbook to a private repo

Browse Source 
Adds bin/mempalace-backup and docs/backup-and-recovery.md — the mechanism a
palace actually needs, none of it site-specific.

Why a palace cannot be backed up with cp: it is chroma.sqlite3 (authoritative),
knowledge_graph.sqlite3 (usually WAL, so -wal/-shm make a plain copy a
same-instant gamble), derived HNSW segment dirs, hallways.json, the embedder
descriptor, and a HIDDEN .mempalace/origin.json. Both SQLite files are therefore
copied through the online-backup API. Two bugs are documented because both
produce a backup that looks fine: "$PALACE"/*/ silently skips the hidden dir, and
per-directory rsync collides the identically named data_level0.bin in every HNSW
segment. Treating the palace as one tree fixes both and makes a backup a faithful
palace IMAGE, so restore is a copy rather than a procedure.

Two modes: hot (default, zero downtime, ~4 s, index may lag but SQLite is
authoritative and repair --mode from-sqlite rebuilds) and cold (--cold, ~5 s
downtime, byte-consistent, restart trapped so a failed run still brings the
server back). Verification runs on the COPY — quick_check plus row counts — and
the backup is committed by mv only after it passes, with retention pruned only
after a verified commit, so a broken new backup cannot delete the last good one.

Documented because they are easy to get wrong: the sqlite3 CLI is often absent
where the Python module is present; mempalace_embedder.json must be restored with
the drawers or search silently degrades; a tested restore means running status
AND search against the restored copy, since search is what actually exercises the
index; mempalace-serve is a USER unit, so root systemctl reports "not found";
Persistent=true is what makes a missed window run after boot; and installing
against the system Python couples the palace's availability to distribution
upgrades, with the uv-managed-interpreter fix plus the two PATH traps that bite
scripted upgrades.

Also moves docs/synlig-primary-runbook.md out to a private fleet repository,
leaving a stub that explains the split, since a host inventory is operator data
for one deployment rather than part of a public toolkit. The path stays valid so
existing links do not break. Remaining host references in the README, RFCs and
ARCHITECTURE are left alone deliberately: they are load-bearing prose, contain no
secrets, and are best generalised as they are next edited rather than in one
churn-heavy pass.
This commit is contained in:
Joakim Persson
2026-08-17 00:50:16 +02:00
parent b609cf5a69
commit 947604b25d
8 changed files with 641 additions and 310 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/backup-and-recovery.md
+275
View File
@@ -0,0 +1,275 @@
# Backing up and recovering a palace
A palace is not one file, so `cp -r` is not a backup. This document explains what
a palace is made of, why the obvious approaches silently corrupt it, and how
`bin/mempalace-backup` avoids that — plus how to actually restore one and prove
the restore worked.
Host-specific details (which machine is primary, addresses, users, offsite
targets) deliberately live outside this repo. Substitute `<PRIMARY_HOST>` and
`<USER>` throughout.
## 1. What is actually in a palace
Run `ls -a` on a palace directory and you get something like this — sizes are
from a real ~25k-drawer palace, for a sense of proportion:
| Path | Size | What it is |
|---|---|---|
| `chroma.sqlite3` | 167 MB | **The authoritative store.** Drawers, chunks, metadata, embeddings. |
| `<uuid>/` (one per segment) | 50 MB, 2.8 MB | chromadb HNSW index segments (`data_level0.bin`, `index_metadata.pickle`). Derived data. |
| `knowledge_graph.sqlite3` | 64 KB | KG triples + entities. **Usually in WAL mode** — see §2. |
| `knowledge_graph.sqlite3-wal` / `-shm` | varies | Live write-ahead log. Their existence is the whole problem. |
| `hallways.json` | 10.8 MB | Within-wing co-occurrence links, rebuilt by `mempalace hallways`. |
| `known_entities.json`, `mempalace_embedder.json` | < 1 KB | Entity list; **which embedder model this palace was built with**. |
| `.mempalace/origin.json` | small | Provenance. **Hidden** — see the glob trap in §3. |
| `.collection_type_fixed`, `.blob_seq_ids_migrated` | 0 B | Migration marker files. Presence is the signal. |
Two consequences worth internalising:
- **The SQLite files are the truth; the index segments are derived.** If the
index is stale or damaged you can rebuild it from SQLite. The reverse is not
true. This asymmetry is what makes a zero-downtime backup possible at all.
- **`mempalace_embedder.json` is not optional.** Restoring drawers next to a
*different* embedder model gives you a palace whose stored vectors and future
queries live in different spaces — search silently degrades rather than
failing. Back it up, and check it after a restore.
## 2. Why `cp` is unsafe, specifically
SQLite in WAL mode keeps recent commits in `-wal` and shared state in `-shm`.
Copying those three files with `cp` gives you no guarantee they are from the same
instant: the copier can read `chroma.sqlite3` before a checkpoint and `-wal`
after it. The result usually *opens* — which is the dangerous part — and then
misbehaves later, or fails `PRAGMA integrity_check` under load.
`rsync` has exactly the same problem: it is not a snapshot, it is a sequence of
reads.
The supported approach is SQLite's **online backup API**, which takes a
transactionally consistent copy of a live database while writers continue:
```python
import sqlite3
src = sqlite3.connect(f"file:{src_path}?mode=ro", uri=True)
dst = sqlite3.connect(dst_path)
with dst:
src.backup(dst) # WAL-aware, consistent, no writer downtime
```
Two practical notes:
- **`VACUUM INTO` is a fine alternative in principle**, but it needs either the
`sqlite3` CLI or a Python build new enough for it. Do not assume the CLI is
present: a stock Ubuntu server often has the Python `sqlite3` *module* and no
`sqlite3` *binary*. Check with `command -v sqlite3` before writing tooling
that depends on it.
- Open the source with `mode=ro` so a backup can never be the thing that
creates a `-wal` file in a directory you meant only to read.
## 3. Two backup modes, and the honest tradeoff
`bin/mempalace-backup` implements both. Neither is strictly better.
### hot (default) — zero downtime
1. `rsync` the whole palace tree **excluding** the SQLite files.
2. Write both SQLite files via the online-backup API.
The server never stops. The index segments are copied while they may be being
written, so in principle they can lag the SQLite by a few records. That is
acceptable *because the SQLite is authoritative*: if search misbehaves after a
restore, rebuild the index (§5).
Measured on a 223 MB palace: **~4 s**.
### cold (`--cold`) — byte-consistent
1. Stop the server (`systemctl --user stop mempalace-serve`).
2. Copy everything, including the index, at rest.
3. Restart the server.
The restart is in a `trap … EXIT INT TERM`, so an interruption mid-copy still
brings the server back. Measured downtime: **~5 s**.
Recommended schedule: **hot daily, cold weekly**. The cold copy is your
byte-exact reference; the hot copies bound your data loss between them.
### The glob trap that cost two rewrites
Two bugs are worth naming because both produce a backup that looks fine:
- **`"$PALACE"/*/` skips hidden directories.** `.mempalace/origin.json` is
silently omitted, so the restored palace loses its provenance. Shell globs do
not match dotfiles without `dotglob`/`GLOBIGNORE` fiddling.
- **Per-directory rsync collides identically named files.** Every HNSW segment
contains a `data_level0.bin`. Copying segment dirs into one destination
merges them and the last write wins — a corrupt index that still loads.
The fix for both is to treat the palace as **one tree**: a single
`rsync -a --exclude` of the SQLite files, so a backup directory is a faithful
*image* of the palace. That makes restore a copy rather than a procedure.
## 4. Verify the copy, not the original
A backup you have not read back is a hypothesis. After copying, and **before**
committing the backup or pruning old ones, the tool runs against the *copy*:
- `PRAGMA quick_check` on both databases
- row counts (`drawers`, `triples`, `entities`)
- a `MANIFEST.json` recording counts, sizes, mode, tool version, and the
index-consistency caveat for hot backups
Sequencing matters as much as the checks:
1. write into `<dest>.incomplete/`
2. verify
3. `mv` into place (atomic commit)
4. **only now** prune old backups to the retention limit
Retention pruning that runs before verification can delete your last good
backup on the night the new one is broken. A leftover `.incomplete` directory is
a visible symptom, not silent data loss, so the tool reports stale ones.
## 5. Restoring
Because a backup is a palace image, restore is a copy:
```bash
systemctl --user stop mempalace-serve
mv ~/.mempalace/palace ~/.mempalace/palace.broken-$(date -u +%Y%m%dT%H%M%SZ)
mkdir -p ~/.mempalace/palace
rsync -a --exclude MANIFEST.json <backup-dir>/ ~/.mempalace/palace/
systemctl --user start mempalace-serve
```
Keep the old directory until the restore is proven. It costs disk and buys you a
second attempt.
**If the restore came from a hot backup and search behaves oddly**, rebuild the
derived index from the authoritative SQLite:
```bash
mempalace repair --mode from-sqlite
```
Note that a full `repair --mode from-sqlite --archive-existing` re-creates the
chromadb collections, which means new collections pick up **current** HNSW
tuning defaults. On palaces created by older versions this is also the only way
to escape legacy `hnsw:batch_size=2 / hnsw:sync_threshold=2` settings, which sync
the index every two records. Check yours with:
```sql
SELECT c.name, k.key, k.str_value FROM collections c
JOIN collection_metadata k ON k.collection_id = c.id
WHERE k.key LIKE 'hnsw%';
```
## 6. Prove the restore, don't assume it
"The files copied" is not a tested restore. Restore to a throwaway directory and
make the palace *answer questions*:
```bash
rsync -a --exclude MANIFEST.json <backup-dir>/ /tmp/restore-test/
export MEMPALACE_PALACE_PATH=/tmp/restore-test
mempalace status # drawer count — compare to the source
mempalace search "some phrase you know is in there"
```
`status` proves the schema and metadata survived. **`search` is the stronger
test**: it exercises the HNSW segments, so it is what tells you whether a
*hot* backup's index came across usable. On the reference run above it returned
correctly ranked results (`cosine_sim=0.398 bm25=2.992`) from a zero-downtime
backup — the theoretical index skew did not materialise.
Do this at least once when you set backups up, and again after any version
upgrade that touches storage.
## 7. Scheduling that survives a reboot
Use **systemd user timers** with lingering enabled, so backups run without
anyone logging in:
```bash
loginctl enable-linger <USER> # required, or user units die at logout
systemctl --user enable --now mempalace-backup.timer mempalace-backup-cold.timer
systemctl --user list-timers 'mempalace*'
```
Units are in [`contrib/systemd/`](../contrib/systemd/). Two settings earn their
keep:
- **`Persistent=true`** — a timer whose window was missed (host powered off)
runs shortly after boot instead of skipping the day.
- **`RandomizedDelaySec`** — avoids a thundering herd if other maintenance
shares the hour.
`mempalace-serve` is itself a **user** unit in the reference deployment. That is
worth stating explicitly because `systemctl status mempalace-serve` as root
reports "not found" and invites the conclusion that the server is not installed.
Always `systemctl --user`.
### The interpreter is part of your durability story
If the tool is installed against the **system** Python (`/usr/bin/python3`), then
a distribution release upgrade — which can remove or replace that interpreter —
breaks the server and its backups together, at the moment you least want it.
Install against a **managed** interpreter instead:
```bash
uv python install 3.12
uv tool install --force --python "$HOME/.local/bin/python3.12" 'mempalace==<version>'
systemctl --user restart mempalace-serve
```
Now an OS upgrade cannot take the interpreter away. Two gotchas:
- `uv` is typically in `~/.local/bin`, which is **not** on a non-interactive
ssh PATH. Scripted upgrades must `export PATH="$HOME/.local/bin:$PATH"`.
- `uv python list` prints `$HOME`-relative paths, so a regex anchored on a
leading `/` matches inside `.local/share` and yields a bogus absolute path.
Resolve the shim and assert on it:
`readlink -f ~/.local/bin/python3.12 | grep share/uv/python`.
**Restart the server after any upgrade.** Recent versions refuse mutating tools
when the served library drifts from what is installed on disk, and
`mempalace_reconnect` cannot clear that — it reopens the database but cannot
reload Python modules.
## 8. Sizing and offsite
At ~225 MB per copy, 7 hot + 7 cold is ~3 GB — trivial against any modern disk,
so prefer generous retention over clever incremental schemes. If space is
genuinely tight, `zstd -T0` on the SQLite files compresses well, at the cost of
making a backup no longer a directly-mountable palace image.
**Local backups protect against corruption and mistakes, not against losing the
host.** Until a copy leaves the machine, a single disk or a single `rm` ends the
palace. Whatever offsite target you choose (object store, NAS, another host),
two rules matter more than the choice:
- **Put a copy of the restore procedure with the backups.** Documentation that
lives only inside the thing you are restoring is not available at the moment
you need it.
- **Verify the offsite copy by restoring from it**, not by checking that the
upload exited 0.
## 9. Usage
```
mempalace-backup [--cold] [--dest DIR] [--keep N] [--palace DIR] [--dry-run]
mempalace-backup --verify <backup-dir>
```
| Flag | Meaning |
|---|---|
| `--cold` | Quiesce the server for a byte-consistent copy (default: hot) |
| `--dest DIR` | Backup root (default `~/backups/mempalace`) |
| `--keep N` | Retention count, pruned only after a verified commit (default 7) |
| `--palace DIR` | Palace to back up (default `~/.mempalace/palace`) |
| `--verify DIR` | Re-verify an existing backup and print its manifest |
| `--dry-run` | Report what would happen, write nothing |
Backups are named `mp-<UTC timestamp>-<hot|cold>`, so they sort
chronologically and their mode is visible without opening the manifest.
docs/synlig-primary-runbook.md
+16 -310
View File
@@ -1,315 +1,21 @@
# synlig primary — Phase 0 runbook and handoff
# (moved) primary-host runbook
Companion to [`rfc-001-global-palace.md`](./rfc-001-global-palace.md). Records what was actually done
on the primary, with verified evidence, so the next session (or the next machine) does not re-derive it.
This file used to contain the deployment runbook for one specific primary host —
its hostname, addresses, user, service wiring and rollback steps.
> **Status 2026-08-14 17:00 — SUPERSEDED IN PART. The primary is live, exposed, and seeded.**
> Serving since 2026-08-12 at `https://mempalace.jordbo.se/mcp`. Seeded 2026-08-14 15:07 from
> EMB-7KJ4VR4G's palace (itself a carry-over from the previous work computer EMB-X1JY06WJ — rfc-001 §4.4)
> — 14,777 drawers / 9 wings / 16,337 embeddings / KG 46 entities, 34 triples,
> now 14,803 drawers. One client (EMB-7KJ4VR4G's pi-devbox container) is flipped and verified
> end-to-end. Both Phase 0 blockers below are cleared.
>
> **Update 2026-08-16 00:20 — the transcript feed is live too, and it is the primary's third moving
> part** (alongside the HTTPS tunnel and the palace itself). See §2.6: transcripts arrive over SSH into
> `~/mempalace-feed/<device>/` and are mined *by this host's own server process*. 15,478 drawers as of
> that check — but treat every count in this document as a timestamp, not a fact: `status` counts chunk
> rows, and §4 item 7 explains why counts adjudicate nothing.
>
> **Read §3 "Deliberately NOT done" as a record of the 2026-08-10 state, not of today's** — every
> item in it has since been done. And before running anything in §5 Rollback, read the warning at the
> top of it: `~/.mempalace` on synlig is no longer disposable.
**That content now lives in a private repository**, because a host inventory is
operator data for one deployment, not part of the toolkit. This repository is
public and keeps only host-agnostic *mechanism*.
Original status, kept for the record:
What lives where:
**Status 2026-08-10 00:30 — Phase 0 prep complete. Not serving. Nothing exposed.**
Blocked on two things, both deliberately left to Joakim: the Pangolin update on nyvaken, and one `sudo`.
| Content | Home |
|---|---|
| How to expose a palace over HTTP, and the Host/Origin pin | `docs/phase-1-exposure-runbook.md` (here) |
| Why a palace needs a special backup, and how to restore one | `docs/backup-and-recovery.md` (here) |
| Unit/timer/plist templates | `contrib/` (here) |
| Which machine is primary, its addresses, users, tunnels, offsite target | private fleet repository |
| Per-host feeder device names and schedules | private fleet repository |
---
## 1. What synlig is (discovered, not assumed)
| Fact | Value |
| --- | --- |
| SSH | `synlig` → `synlig.erdc.ericsson.net`, user `ecsjper` (from `~/.ssh/config`) |
| OS | Ubuntu 24.04.4 LTS, 7.8 GiB RAM, 78 G disk (**29 G free**), uptime 12 d |
| Python / uv | system `python3` 3.12.3; `uv` at `~/.local/bin/uv` (**not** on the non-login `PATH`) |
| Interfaces | `lo` 127.0.0.1, `ens3` 10.0.0.4/16, `docker0` 172.17.0.1/16, `br-…` 172.19.0.1/16 |
| Already listening | 22, 80, 443, 3000 (node), 3389 + 3350 + 4822 (xrdp/guacamole), 631 |
| Docker | present; running `act_runner-runner-1` (**Gitea Actions runner**) and `digikam` |
| Pre-existing MemPalace | **none** — no `mempalace` binary, no `~/.mempalace`. Greenfield. |
The Gitea Actions runner living here is worth remembering: synlig is not a dedicated appliance, and CI
load competes with the palace for the same 7.8 GiB.
## 2. Done tonight
### 2.1 MemPalace installed, pinned to the fleet version
```sh
~/.local/bin/uv tool install "mempalace==3.6.0" # → mempalace, mempalace-mcp
```
Pinned deliberately: the clients run 3.6.0, and the id recipes / idempotency probes this RFC leans on are
version-specific. Reversible with `uv tool uninstall mempalace`.
### 2.2 Embedder model pre-warmed — the corporate-network risk that wasn't
The first embed pulls `all-MiniLM-L6-v2` ONNX (79.3 MB) from the chroma CDN into
`~/.cache/chroma/onnx_models/` (167 M on disk once unpacked). **This was the main unknown** — an
egress-filtered work VM would have failed here, at the worst possible moment (first client write).
It downloaded at ~20 MB/s with no proxy interference. Done in a throwaway palace, since deleted, so the
real palace never saw it. Same model as the clients use, so the semantic space matches.
### 2.3 Palace created with the §7.1 landmine structurally removed
`~/.mempalace/palace` — the **stock default**, so no `MEMPALACE_PALACE_PATH` and no `config.json` is
needed anywhere on synlig. One less thing to drift.
RFC §7.1 says to `mv` three HOME-anchored stores into the palace dir before first `serve`. **On a
greenfield primary there is nothing to move — but the hazard is not actually a migration hazard, and the
RFC understated it:** with *stock defaults* `palace_path` is `~/.mempalace/palace` while `DEFAULT_KG_PATH`
is `~/.mempalace/knowledge_graph.sqlite3`. Those differ, so the split is the **out-of-the-box** behaviour,
not a consequence of a custom path. It is permanent, not one-time: `serve` always passes `--palace` (KG
inside the palace), while any CLI command run *without* `--palace` uses the HOME path. Two KGs on one box,
forever, silently.
Fixed by making both resolution rules land on one inode:
```sh
ln -sfn palace/knowledge_graph.sqlite3 ~/.mempalace/knowledge_graph.sqlite3
ln -sfn palace/known_entities.json ~/.mempalace/known_entities.json
```
Relative targets, so a home-directory move survives. `hallways.json` was originally left **un**symlinked:
it is already palace-derived, and its HOME path is a warning-only legacy probe (`hallways.py:73-95`) that
never auto-migrates.
> **Update 2026-08-14 — `hallways.json` is now symlinked too**, during the seeding session:
> ```sh
> ln -sfn palace/hallways.json ~/.mempalace/hallways.json
> ```
> Rationale changed: the point is no longer "only symlink what the code demands" but *all real state
> lives under `palace/` as a single backup unit*, so one `palace/` copy is a complete copy. All three
> parent-level paths now resolve, which matters because mempalace 3.6.0 resolves these three paths
> inconsistently (MCP server: palace-relative; KG CLI default: `~/.mempalace`; `hallways.json`:
> `dirname(palace_path)`; `known_entities.json`: hardcoded `~`). Revert by deleting the symlink if it
> ever causes trouble.
Verified the symlink assumption rather than trusting it (`python3 sqlite3` on synlig, temp dir):
| Check | Result |
| --- | --- |
| Dangling symlink + `sqlite3.connect` | creates the target |
| `-wal` / `-shm` placement | next to the **target**, inside the palace dir — *not* beside the symlink |
| Write via symlink → read via palace path | same data, **same inode** |
The WAL placement is the part that mattered: it keeps the palace directory a single self-contained
backup/bind-mount unit.
### 2.4 §6.2's Host/Origin policy verified by experiment, not by reading
Ran on synlig, loopback and docker0 binds, then stopped. **11/11 as predicted:**
| # | Bind | Request | Expected | Got |
| --- | --- | --- | --- | --- |
| A1 | 127.0.0.1 | `/healthz`, correct Host | 200 | ✅ 200 |
| A2 | 127.0.0.1 | `/healthz`, `Host: palace.example.com` | **403** | ✅ 403 |
| A3 | 127.0.0.1 | `/healthz`, `Origin: https://evil.example` | 403 | ✅ 403 |
| A4 | 127.0.0.1 | `POST /mcp`, no token | 401 | ✅ 401 |
| A5 | 127.0.0.1 | `POST /mcp`, wrong token | 401 | ✅ 401 |
| A6 | 127.0.0.1 | `POST /mcp`, correct token | 200 | ✅ 200 (`tools/list` → **36 tools**) |
| B1 | 172.17.0.1 | `/healthz`, bound-host Host | 200 | ✅ 200 |
| B2 | 172.17.0.1 | `/healthz`, `Host: palace.example.com` | **200** | ✅ 200 |
| B3 | 172.17.0.1 | `/healthz`, `Origin: https://evil.example` | 403 | ✅ 403 |
| B4 | 172.17.0.1 | `/healthz`, loopback Origin | 200 | ✅ 200 |
| B5 | 172.17.0.1 | `POST /mcp`, foreign Host + token | 200 | ✅ 200 |
**Operational conclusions:**
1. **Do not bind loopback behind the tunnel.** A2 vs B2 is the whole story: the reflex "bind 127.0.0.1,
it's safer" produces a 403 that looks like a Pangolin misconfiguration and is not one.
2. **Bind `172.17.0.1` (docker0).** Non-loopback, so the Host pin relaxes — but reachable only from
synlig and its containers, so a newt container on this box can reach it while the LAN cannot. This is
strictly better than `0.0.0.0` here. It is what `contrib/systemd/mempalace-serve.service` uses.
3. **`Origin` is never relaxed** (B3). No browser-based MCP client, and no proxy that injects `Origin`.
4. `/healthz` is Host/Origin-gated but token-free — a usable liveness probe for the tunnel.
Test script kept at `/tmp/synlig-phase0-test.sh` on this container (ephemeral — re-create from the table
above if needed; it starts, probes and stops the server, and asserts nothing is left listening).
### 2.5 A start unit — written, staged, and (since 2026-08-12) installed and running
> **This subsection describes 2026-08-10. The unit is now live.** Verified 2026-08-16 00:15:
> `systemctl --user list-units` shows `mempalace-serve.service … loaded active running`, and the
> process is
> `~/.local/share/uv/tools/mempalace/bin/python -m mempalace.mcp_server --transport http
> --host 172.17.0.1 --port 8765 --palace /home/ecsjper/.mempalace/palace`.
> Note what that means and §2.6 depends on: **the server is a NATIVE process, not a container** — it
> sees synlig's real filesystem paths, and `docker ps` on synlig lists no mempalace container.
`contrib/systemd/mempalace-serve.service` — user unit, follows the existing `contrib/systemd/` style,
carries the bind rationale inline so nobody "fixes" it back to loopback. A copy is already staged on synlig
at `~/.config/systemd/user/mempalace-serve.service.staged` — **the `.staged` suffix is deliberate**:
systemd only reads `*.service`, so the file cannot be activated by accident, not even by a stray
`daemon-reload`. **Not** installed, **not** enabled: it needs one `sudo loginctl enable-linger`, and
standing up a network-reachable service while you were asleep was not mine to decide.
(Both were done on 2026-08-12 — §4 item 3 has the exact commands that were run.)
### 2.6 The transcript inbox — `~/mempalace-feed/<device>/` (added 2026-08-16)
Flipped clients write drawers over HTTPS, but their **session transcripts** cannot travel that way:
`mempalace_mine` resolves its `source` path *in the server process*, so the server cannot see a
client's staged exports. `mempalace-pi-session --mode remote` therefore rsyncs each client's stage into
a per-device inbox here and then asks the server to mine its own local path:
```sh
ls ~/mempalace-feed/ # one dir per device, e.g. emb-7kj4vr4g/
ls ~/mempalace-feed/emb-7kj4vr4g/ # pi_<session-uuid>.jsonl, mtimes preserved
```
Client side, that needs three variables — and **the third one is the trap**:
| Variable | Value for this fleet | Why |
| --- | --- | --- |
| `MEMPALACE_PI_SSH_TARGET` | `ecsjper@synlig:/home/ecsjper/mempalace-feed` | where rsync puts the files |
| `MEMPALACE_PI_DEVICE` | e.g. `emb-7kj4vr4g` | inbox subdirectory per machine |
| `MEMPALACE_PI_REMOTE_PATH` | `/home/ecsjper/mempalace-feed` | the inbox **as the server process sees it** |
The feeder's default for the third is `/data/feed`, which assumes a *containerized* palace server with
the inbox bind-mounted there. **This primary is native (§2.5), so it only ever sees host paths and the
value must equal the path half of the SSH target.** Get it wrong and the failure is quiet in the worst
way: rsync succeeds, the files are all present here, and only the mine fails with
`source directory not found: '/data/feed/<device>'`.
That is exactly what happened on 2026-08-15, and it went unnoticed for a session because the feeder
decided success with `'"error"' in body` — MCP returns HTTP 200 with the tool's own JSON **escaped**
inside `result.content[].text`, so those bytes are `\"error\"`, the substring never matched, and
`~/.pi/agent/mempalace-catchup.log` printed `Done. Wing 'wing_conversations' updated.` directly under the
error. Fixed in `6e1f4f3`: the envelope is parsed, `--self-test` pins that exact response body, and a
preflight warning fires whenever the ship path and `MEMPALACE_PI_REMOTE_PATH` disagree.
**Operational notes for this inbox:**
- Dedup keys on the **absolute source path**, so the inbox path is load-bearing: it must stay stable, or
every transcript re-files under its new name. Migrating it on 2026-08-15 (from the clients' old
container-local stage paths, which arrived with the seed) cost a full re-mine plus
`mempalace_delete_by_source` on 6 old paths — 651 drawers purged, 1243 re-filed. `wing_conversations`
is now keyed entirely on `/home/ecsjper/mempalace-feed/<device>/…`.
- A **grown** session is purged and re-filed for the same path (mtime-based), so re-feeding a live
session refreshes it instead of duplicating it. That is why the inbox keeps whole transcripts rather
than deltas — do not "tidy" it by deleting files the palace still references.
- `mempalace_mine` over MCP can **exceed a client's request timeout while the server keeps working and
finishes normally**. A client-side timeout is not a failed mine: check
`SELECT COUNT(*) FROM embedding_metadata WHERE key='source_file' AND string_value LIKE '<inbox>%'`
(read-only, `file:…?mode=ro`) before retrying anything.
- Health check after any client recreate, from the client: the tail of
`~/.pi/agent/mempalace-catchup.log` should end in `Done. Wing … updated.` with no `error:` line above
it. With the fixed feeder a broken run exits 5 and names the reason.
## 3. Deliberately NOT done
> **⚠️ Historical — this section describes 2026-08-10 and is no longer true.** All five items were
> done between 2026-08-12 and 2026-08-14. Kept because the *reasoning* for deferring them is still
> the record of why the order was chosen. Current state per item is inlined below.
- **Nothing is serving.** No listener on 8765; no mempalace process. Re-verified at the end of the run.
→ **Now serving** since 2026-08-12 (`mempalace-serve.service`, `systemctl --user`), reachable at
`https://mempalace.jordbo.se/mcp` via newt/Pangolin.
- **No client `.env` was touched.** Your working setup is exactly as you left it (R6: reversible).
→ **One client flipped 2026-08-14**: four variables on EMB-7KJ4VR4G, `docker-compose.yaml` unchanged.
Still reversible in ~30s (§3.8 of [`phase-1-exposure-runbook.md`](./phase-1-exposure-runbook.md)).
- **No data joined.** The palace is empty. The §4.4 join needs the diary-dedup decision (§7.6) first —
replaying diaries today duplicates them, and the primary is the one place that must stay clean.
→ **Seeded 2026-08-14** from *one* palace by file-level copy. This sidestepped §7.6 rather than
solving it: a file-level copy replays no diaries, so it cannot duplicate them. **§7.6 is still a
hard blocker for the second machine to join.**
- **nyvaken untouched.** Read nothing, changed nothing.
- **No sudo.** `sudo -n` on synlig requires a password.
## 4. Tomorrow, in order
> **2026-08-12: items 1–2 and 5 now have their own runbook —**
> [`phase-1-exposure-runbook.md`](./phase-1-exposure-runbook.md). Pangolin on nyvaken is updated (done),
> and **newt is now installed on synlig and connected to Pangolin (done 2026-08-12)** — so the blocker is
> now item 3, the one `sudo`. That doc also records why per-device Pangolin users are the wrong layer, why
> the HTTPS tunnel and the feeder's SSH path are **not** redundant (§1.3), the client-flip variable trap
> (§3.7), and an additional loopback finding: a loopback bind does not merely 403, it also silently starts
> the server with **no token at all** (auto-minting is gated on the bind being non-loopback).
1. **Pangolin update on nyvaken** (yours). ✅ done 2026-08-12.
2. ~~**⚠️ synlig has no tunnel client.**~~ ✅ **done 2026-08-12** — newt installed and connected to Pangolin.
(Kept for the reasoning: `docker ps` showed only the Gitea runner and digikam. Pangolin on nyvaken
cannot reach synlig by itself; synlig had to dial out. Easy to miss because Pangolin looks healthy on
its own side — which is also why "connected" is not yet proof it can reach the palace: verify
`172.17.0.1:8765/healthz` from *inside* newt's namespace, exposure runbook §3.3.) Since newt runs in
Docker here, the docker0 bind above is already correct for it.
3. **One sudo, then start** (the unit is already staged; just drop the suffix). ✅ **done 2026-08-12** —
linger enabled, unit enabled, `172.17.0.1:8765/healthz` → `ok`.
```sh
sudo loginctl enable-linger ecsjper
cd ~/.config/systemd/user && mv mempalace-serve.service.staged mempalace-serve.service
systemctl --user daemon-reload && systemctl --user enable --now mempalace-serve
curl -s 172.17.0.1:8765/healthz # ok
curl -s 127.0.0.1:8765/healthz # NOTHING — refused, exit 7 (not 403; see below)
ss -ltnp | grep 8765 # 172.17.0.1:8765 only
```
⚠ **Corrected 2026-08-12:** this line predicted `403`. The real run returned empty, which is *more*
reassuring. With the docker0-only bind nothing listens on loopback, so the connection is refused before
any header is sent (`%{http_code}` → `000`, `$?` → `7`). The 403 in §2.4 is the **loopback-bind** case:
a server on `127.0.0.1` answering a proxy-forwarded foreign `Host:`. Two different failures that were
collapsed into one expectation here.
4. **Collect the shared token** (auto-minted on first non-loopback start, stable across restarts):
```sh
cat ~/.mempalace/server/f5d849287f6d73f0141b29d7/token
```
That directory name is `sha256(realpath(palace))[:24]` — it changes if the palace path ever changes.
5. **Route it through Pangolin**, then verify `/healthz` end-to-end through the public hostname *before*
pointing any client at it.
6. **Then, and only then**, Phase 1 client flip — one machine first, and remember opencode containers
need the §4.1 sidecar merge (or Phase 1.5) before the `.env` takes effect.
7. **Before the first join:** settle §7.6 diary dedup, then dry-run §4.4 from **one** palace.
> **Correction 2026-08-14 — do NOT verify a join "by checking counts", which is what this item
> originally said.** Counts are not evidence, in either direction. `mempalace status` counts
> **chunk rows**, not logical drawers (3 drawers plus one 2-chunk diary presented as +9), and chunk
> counts legitimately differ between two palaces whenever a drawer was updated on either side,
> because an update re-chunks to the new length and deletes the surplus rows. Diffing chunk-id sets
> is a useful first pass but **over**-reports: a chunk id present on one side only is the ordinary
> signature of an edit, not of loss. This cost real time on 2026-08-14 — a missing
> `chunk_000007` was read as a truncated seed, when in fact the two palaces held two revisions of
> one drawer and nothing was lost. **Adjudicate by fetching the parent drawer on both sides and
> comparing the reassembled `content`.**
## 5. Rollback
> **⚠️ STOP — 2026-08-14. Do not run this block as it was originally written.** `~/.mempalace` on
> synlig is now the fleet primary. That tree holds the only central palace (14,803 drawers, seeded
> from EMB-7KJ4VR4G) **and the server's bearer token** at `~/.mempalace/server/<hash>/token` — the
> single credential every flipped client authenticates with, of which there is no second copy.
> `rm -rf ~/.mempalace` destroys both. The original comment ("empty today — check before running once
> it isn't") is far too soft for a destructive command someone runs under pressure, which is exactly
> why it is being replaced rather than amended.
Stopping the service is safe and reversible on its own, and is the whole of what "rollback" should
normally mean now:
```sh
systemctl --user disable --now mempalace-serve # clients fail CLOSED — they lose the palace
# tools; they do NOT fall back to a local palace
```
To genuinely decommission the primary, in this order:
1. Flip every client back first (§3.8 of [`phase-1-exposure-runbook.md`](./phase-1-exposure-runbook.md),
in reverse) so nothing is pointed at a host that is about to lose its palace.
2. Copy `~/.mempalace/palace/` **and** the token file off the host, and verify the copy by comparing
reassembled drawer `content`, not counts (see §4 item 7).
3. Only then remove anything. Never `rsync --delete` into `~/.mempalace` — the token lives inside it.
The two destructive steps below were written on 2026-08-10, when `~/.mempalace` was genuinely empty.
Kept for the record; **must not be run while the primary is live**:
```sh
~/.local/bin/uv tool uninstall mempalace
rm -rf ~/.mempalace # ⚠️ DESTROYS THE FLEET PALACE AND THE ONLY TOKEN
```
If you are looking for the mechanism, the two runbooks above are the same
procedures with `<PRIMARY_HOST>` and `<USER>` in place of one site's specifics.