docs: backup and recovery, plus units; move host runbook to a private repo
Adds bin/mempalace-backup and docs/backup-and-recovery.md — the mechanism a palace actually needs, none of it site-specific. Why a palace cannot be backed up with cp: it is chroma.sqlite3 (authoritative), knowledge_graph.sqlite3 (usually WAL, so -wal/-shm make a plain copy a same-instant gamble), derived HNSW segment dirs, hallways.json, the embedder descriptor, and a HIDDEN .mempalace/origin.json. Both SQLite files are therefore copied through the online-backup API. Two bugs are documented because both produce a backup that looks fine: "$PALACE"/*/ silently skips the hidden dir, and per-directory rsync collides the identically named data_level0.bin in every HNSW segment. Treating the palace as one tree fixes both and makes a backup a faithful palace IMAGE, so restore is a copy rather than a procedure. Two modes: hot (default, zero downtime, ~4 s, index may lag but SQLite is authoritative and repair --mode from-sqlite rebuilds) and cold (--cold, ~5 s downtime, byte-consistent, restart trapped so a failed run still brings the server back). Verification runs on the COPY — quick_check plus row counts — and the backup is committed by mv only after it passes, with retention pruned only after a verified commit, so a broken new backup cannot delete the last good one. Documented because they are easy to get wrong: the sqlite3 CLI is often absent where the Python module is present; mempalace_embedder.json must be restored with the drawers or search silently degrades; a tested restore means running status AND search against the restored copy, since search is what actually exercises the index; mempalace-serve is a USER unit, so root systemctl reports "not found"; Persistent=true is what makes a missed window run after boot; and installing against the system Python couples the palace's availability to distribution upgrades, with the uv-managed-interpreter fix plus the two PATH traps that bite scripted upgrades. Also moves docs/synlig-primary-runbook.md out to a private fleet repository, leaving a stub that explains the split, since a host inventory is operator data for one deployment rather than part of a public toolkit. The path stays valid so existing links do not break. Remaining host references in the README, RFCs and ARCHITECTURE are left alone deliberately: they are load-bearing prose, contain no secrets, and are best generalised as they are next edited rather than in one churn-heavy pass.
This commit is contained in:
Joakim Persson
+16
-310
@@ -1,315 +1,21 @@
|
||||
# synlig primary — Phase 0 runbook and handoff
|
||||
# (moved) primary-host runbook
|
||||
|
||||
Companion to [`rfc-001-global-palace.md`](./rfc-001-global-palace.md). Records what was actually done
|
||||
on the primary, with verified evidence, so the next session (or the next machine) does not re-derive it.
|
||||
This file used to contain the deployment runbook for one specific primary host —
|
||||
its hostname, addresses, user, service wiring and rollback steps.
|
||||
|
||||
> **Status 2026-08-14 17:00 — SUPERSEDED IN PART. The primary is live, exposed, and seeded.**
|
||||
> Serving since 2026-08-12 at `https://mempalace.jordbo.se/mcp`. Seeded 2026-08-14 15:07 from
|
||||
> EMB-7KJ4VR4G's palace (itself a carry-over from the previous work computer EMB-X1JY06WJ — rfc-001 §4.4)
|
||||
> — 14,777 drawers / 9 wings / 16,337 embeddings / KG 46 entities, 34 triples,
|
||||
> now 14,803 drawers. One client (EMB-7KJ4VR4G's pi-devbox container) is flipped and verified
|
||||
> end-to-end. Both Phase 0 blockers below are cleared.
|
||||
>
|
||||
> **Update 2026-08-16 00:20 — the transcript feed is live too, and it is the primary's third moving
|
||||
> part** (alongside the HTTPS tunnel and the palace itself). See §2.6: transcripts arrive over SSH into
|
||||
> `~/mempalace-feed/<device>/` and are mined *by this host's own server process*. 15,478 drawers as of
|
||||
> that check — but treat every count in this document as a timestamp, not a fact: `status` counts chunk
|
||||
> rows, and §4 item 7 explains why counts adjudicate nothing.
|
||||
>
|
||||
> **Read §3 "Deliberately NOT done" as a record of the 2026-08-10 state, not of today's** — every
|
||||
> item in it has since been done. And before running anything in §5 Rollback, read the warning at the
|
||||
> top of it: `~/.mempalace` on synlig is no longer disposable.
|
||||
**That content now lives in a private repository**, because a host inventory is
|
||||
operator data for one deployment, not part of the toolkit. This repository is
|
||||
public and keeps only host-agnostic *mechanism*.
|
||||
|
||||
Original status, kept for the record:
|
||||
What lives where:
|
||||
|
||||
**Status 2026-08-10 00:30 — Phase 0 prep complete. Not serving. Nothing exposed.**
|
||||
Blocked on two things, both deliberately left to Joakim: the Pangolin update on nyvaken, and one `sudo`.
|
||||
| Content | Home |
|
||||
|---|---|
|
||||
| How to expose a palace over HTTP, and the Host/Origin pin | `docs/phase-1-exposure-runbook.md` (here) |
|
||||
| Why a palace needs a special backup, and how to restore one | `docs/backup-and-recovery.md` (here) |
|
||||
| Unit/timer/plist templates | `contrib/` (here) |
|
||||
| Which machine is primary, its addresses, users, tunnels, offsite target | private fleet repository |
|
||||
| Per-host feeder device names and schedules | private fleet repository |
|
||||
|
||||
---
|
||||
|
||||
## 1. What synlig is (discovered, not assumed)
|
||||
|
||||
| Fact | Value |
|
||||
| --- | --- |
|
||||
| SSH | `synlig` → `synlig.erdc.ericsson.net`, user `ecsjper` (from `~/.ssh/config`) |
|
||||
| OS | Ubuntu 24.04.4 LTS, 7.8 GiB RAM, 78 G disk (**29 G free**), uptime 12 d |
|
||||
| Python / uv | system `python3` 3.12.3; `uv` at `~/.local/bin/uv` (**not** on the non-login `PATH`) |
|
||||
| Interfaces | `lo` 127.0.0.1, `ens3` 10.0.0.4/16, `docker0` 172.17.0.1/16, `br-…` 172.19.0.1/16 |
|
||||
| Already listening | 22, 80, 443, 3000 (node), 3389 + 3350 + 4822 (xrdp/guacamole), 631 |
|
||||
| Docker | present; running `act_runner-runner-1` (**Gitea Actions runner**) and `digikam` |
|
||||
| Pre-existing MemPalace | **none** — no `mempalace` binary, no `~/.mempalace`. Greenfield. |
|
||||
|
||||
The Gitea Actions runner living here is worth remembering: synlig is not a dedicated appliance, and CI
|
||||
load competes with the palace for the same 7.8 GiB.
|
||||
|
||||
## 2. Done tonight
|
||||
|
||||
### 2.1 MemPalace installed, pinned to the fleet version
|
||||
|
||||
```sh
|
||||
~/.local/bin/uv tool install "mempalace==3.6.0" # → mempalace, mempalace-mcp
|
||||
```
|
||||
|
||||
Pinned deliberately: the clients run 3.6.0, and the id recipes / idempotency probes this RFC leans on are
|
||||
version-specific. Reversible with `uv tool uninstall mempalace`.
|
||||
|
||||
### 2.2 Embedder model pre-warmed — the corporate-network risk that wasn't
|
||||
|
||||
The first embed pulls `all-MiniLM-L6-v2` ONNX (79.3 MB) from the chroma CDN into
|
||||
`~/.cache/chroma/onnx_models/` (167 M on disk once unpacked). **This was the main unknown** — an
|
||||
egress-filtered work VM would have failed here, at the worst possible moment (first client write).
|
||||
It downloaded at ~20 MB/s with no proxy interference. Done in a throwaway palace, since deleted, so the
|
||||
real palace never saw it. Same model as the clients use, so the semantic space matches.
|
||||
|
||||
### 2.3 Palace created with the §7.1 landmine structurally removed
|
||||
|
||||
`~/.mempalace/palace` — the **stock default**, so no `MEMPALACE_PALACE_PATH` and no `config.json` is
|
||||
needed anywhere on synlig. One less thing to drift.
|
||||
|
||||
RFC §7.1 says to `mv` three HOME-anchored stores into the palace dir before first `serve`. **On a
|
||||
greenfield primary there is nothing to move — but the hazard is not actually a migration hazard, and the
|
||||
RFC understated it:** with *stock defaults* `palace_path` is `~/.mempalace/palace` while `DEFAULT_KG_PATH`
|
||||
is `~/.mempalace/knowledge_graph.sqlite3`. Those differ, so the split is the **out-of-the-box** behaviour,
|
||||
not a consequence of a custom path. It is permanent, not one-time: `serve` always passes `--palace` (KG
|
||||
inside the palace), while any CLI command run *without* `--palace` uses the HOME path. Two KGs on one box,
|
||||
forever, silently.
|
||||
|
||||
Fixed by making both resolution rules land on one inode:
|
||||
|
||||
```sh
|
||||
ln -sfn palace/knowledge_graph.sqlite3 ~/.mempalace/knowledge_graph.sqlite3
|
||||
ln -sfn palace/known_entities.json ~/.mempalace/known_entities.json
|
||||
```
|
||||
|
||||
Relative targets, so a home-directory move survives. `hallways.json` was originally left **un**symlinked:
|
||||
it is already palace-derived, and its HOME path is a warning-only legacy probe (`hallways.py:73-95`) that
|
||||
never auto-migrates.
|
||||
|
||||
> **Update 2026-08-14 — `hallways.json` is now symlinked too**, during the seeding session:
|
||||
> ```sh
|
||||
> ln -sfn palace/hallways.json ~/.mempalace/hallways.json
|
||||
> ```
|
||||
> Rationale changed: the point is no longer "only symlink what the code demands" but *all real state
|
||||
> lives under `palace/` as a single backup unit*, so one `palace/` copy is a complete copy. All three
|
||||
> parent-level paths now resolve, which matters because mempalace 3.6.0 resolves these three paths
|
||||
> inconsistently (MCP server: palace-relative; KG CLI default: `~/.mempalace`; `hallways.json`:
|
||||
> `dirname(palace_path)`; `known_entities.json`: hardcoded `~`). Revert by deleting the symlink if it
|
||||
> ever causes trouble.
|
||||
|
||||
Verified the symlink assumption rather than trusting it (`python3 sqlite3` on synlig, temp dir):
|
||||
|
||||
| Check | Result |
|
||||
| --- | --- |
|
||||
| Dangling symlink + `sqlite3.connect` | creates the target |
|
||||
| `-wal` / `-shm` placement | next to the **target**, inside the palace dir — *not* beside the symlink |
|
||||
| Write via symlink → read via palace path | same data, **same inode** |
|
||||
|
||||
The WAL placement is the part that mattered: it keeps the palace directory a single self-contained
|
||||
backup/bind-mount unit.
|
||||
|
||||
### 2.4 §6.2's Host/Origin policy verified by experiment, not by reading
|
||||
|
||||
Ran on synlig, loopback and docker0 binds, then stopped. **11/11 as predicted:**
|
||||
|
||||
| # | Bind | Request | Expected | Got |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| A1 | 127.0.0.1 | `/healthz`, correct Host | 200 | ✅ 200 |
|
||||
| A2 | 127.0.0.1 | `/healthz`, `Host: palace.example.com` | **403** | ✅ 403 |
|
||||
| A3 | 127.0.0.1 | `/healthz`, `Origin: https://evil.example` | 403 | ✅ 403 |
|
||||
| A4 | 127.0.0.1 | `POST /mcp`, no token | 401 | ✅ 401 |
|
||||
| A5 | 127.0.0.1 | `POST /mcp`, wrong token | 401 | ✅ 401 |
|
||||
| A6 | 127.0.0.1 | `POST /mcp`, correct token | 200 | ✅ 200 (`tools/list` → **36 tools**) |
|
||||
| B1 | 172.17.0.1 | `/healthz`, bound-host Host | 200 | ✅ 200 |
|
||||
| B2 | 172.17.0.1 | `/healthz`, `Host: palace.example.com` | **200** | ✅ 200 |
|
||||
| B3 | 172.17.0.1 | `/healthz`, `Origin: https://evil.example` | 403 | ✅ 403 |
|
||||
| B4 | 172.17.0.1 | `/healthz`, loopback Origin | 200 | ✅ 200 |
|
||||
| B5 | 172.17.0.1 | `POST /mcp`, foreign Host + token | 200 | ✅ 200 |
|
||||
|
||||
**Operational conclusions:**
|
||||
|
||||
1. **Do not bind loopback behind the tunnel.** A2 vs B2 is the whole story: the reflex "bind 127.0.0.1,
|
||||
it's safer" produces a 403 that looks like a Pangolin misconfiguration and is not one.
|
||||
2. **Bind `172.17.0.1` (docker0).** Non-loopback, so the Host pin relaxes — but reachable only from
|
||||
synlig and its containers, so a newt container on this box can reach it while the LAN cannot. This is
|
||||
strictly better than `0.0.0.0` here. It is what `contrib/systemd/mempalace-serve.service` uses.
|
||||
3. **`Origin` is never relaxed** (B3). No browser-based MCP client, and no proxy that injects `Origin`.
|
||||
4. `/healthz` is Host/Origin-gated but token-free — a usable liveness probe for the tunnel.
|
||||
|
||||
Test script kept at `/tmp/synlig-phase0-test.sh` on this container (ephemeral — re-create from the table
|
||||
above if needed; it starts, probes and stops the server, and asserts nothing is left listening).
|
||||
|
||||
### 2.5 A start unit — written, staged, and (since 2026-08-12) installed and running
|
||||
|
||||
> **This subsection describes 2026-08-10. The unit is now live.** Verified 2026-08-16 00:15:
|
||||
> `systemctl --user list-units` shows `mempalace-serve.service … loaded active running`, and the
|
||||
> process is
|
||||
> `~/.local/share/uv/tools/mempalace/bin/python -m mempalace.mcp_server --transport http
|
||||
> --host 172.17.0.1 --port 8765 --palace /home/ecsjper/.mempalace/palace`.
|
||||
> Note what that means and §2.6 depends on: **the server is a NATIVE process, not a container** — it
|
||||
> sees synlig's real filesystem paths, and `docker ps` on synlig lists no mempalace container.
|
||||
|
||||
`contrib/systemd/mempalace-serve.service` — user unit, follows the existing `contrib/systemd/` style,
|
||||
carries the bind rationale inline so nobody "fixes" it back to loopback. A copy is already staged on synlig
|
||||
at `~/.config/systemd/user/mempalace-serve.service.staged` — **the `.staged` suffix is deliberate**:
|
||||
systemd only reads `*.service`, so the file cannot be activated by accident, not even by a stray
|
||||
`daemon-reload`. **Not** installed, **not** enabled: it needs one `sudo loginctl enable-linger`, and
|
||||
standing up a network-reachable service while you were asleep was not mine to decide.
|
||||
(Both were done on 2026-08-12 — §4 item 3 has the exact commands that were run.)
|
||||
|
||||
### 2.6 The transcript inbox — `~/mempalace-feed/<device>/` (added 2026-08-16)
|
||||
|
||||
Flipped clients write drawers over HTTPS, but their **session transcripts** cannot travel that way:
|
||||
`mempalace_mine` resolves its `source` path *in the server process*, so the server cannot see a
|
||||
client's staged exports. `mempalace-pi-session --mode remote` therefore rsyncs each client's stage into
|
||||
a per-device inbox here and then asks the server to mine its own local path:
|
||||
|
||||
```sh
|
||||
ls ~/mempalace-feed/ # one dir per device, e.g. emb-7kj4vr4g/
|
||||
ls ~/mempalace-feed/emb-7kj4vr4g/ # pi_<session-uuid>.jsonl, mtimes preserved
|
||||
```
|
||||
|
||||
Client side, that needs three variables — and **the third one is the trap**:
|
||||
|
||||
| Variable | Value for this fleet | Why |
|
||||
| --- | --- | --- |
|
||||
| `MEMPALACE_PI_SSH_TARGET` | `ecsjper@synlig:/home/ecsjper/mempalace-feed` | where rsync puts the files |
|
||||
| `MEMPALACE_PI_DEVICE` | e.g. `emb-7kj4vr4g` | inbox subdirectory per machine |
|
||||
| `MEMPALACE_PI_REMOTE_PATH` | `/home/ecsjper/mempalace-feed` | the inbox **as the server process sees it** |
|
||||
|
||||
The feeder's default for the third is `/data/feed`, which assumes a *containerized* palace server with
|
||||
the inbox bind-mounted there. **This primary is native (§2.5), so it only ever sees host paths and the
|
||||
value must equal the path half of the SSH target.** Get it wrong and the failure is quiet in the worst
|
||||
way: rsync succeeds, the files are all present here, and only the mine fails with
|
||||
`source directory not found: '/data/feed/<device>'`.
|
||||
|
||||
That is exactly what happened on 2026-08-15, and it went unnoticed for a session because the feeder
|
||||
decided success with `'"error"' in body` — MCP returns HTTP 200 with the tool's own JSON **escaped**
|
||||
inside `result.content[].text`, so those bytes are `\"error\"`, the substring never matched, and
|
||||
`~/.pi/agent/mempalace-catchup.log` printed `Done. Wing 'wing_conversations' updated.` directly under the
|
||||
error. Fixed in `6e1f4f3`: the envelope is parsed, `--self-test` pins that exact response body, and a
|
||||
preflight warning fires whenever the ship path and `MEMPALACE_PI_REMOTE_PATH` disagree.
|
||||
|
||||
**Operational notes for this inbox:**
|
||||
|
||||
- Dedup keys on the **absolute source path**, so the inbox path is load-bearing: it must stay stable, or
|
||||
every transcript re-files under its new name. Migrating it on 2026-08-15 (from the clients' old
|
||||
container-local stage paths, which arrived with the seed) cost a full re-mine plus
|
||||
`mempalace_delete_by_source` on 6 old paths — 651 drawers purged, 1243 re-filed. `wing_conversations`
|
||||
is now keyed entirely on `/home/ecsjper/mempalace-feed/<device>/…`.
|
||||
- A **grown** session is purged and re-filed for the same path (mtime-based), so re-feeding a live
|
||||
session refreshes it instead of duplicating it. That is why the inbox keeps whole transcripts rather
|
||||
than deltas — do not "tidy" it by deleting files the palace still references.
|
||||
- `mempalace_mine` over MCP can **exceed a client's request timeout while the server keeps working and
|
||||
finishes normally**. A client-side timeout is not a failed mine: check
|
||||
`SELECT COUNT(*) FROM embedding_metadata WHERE key='source_file' AND string_value LIKE '<inbox>%'`
|
||||
(read-only, `file:…?mode=ro`) before retrying anything.
|
||||
- Health check after any client recreate, from the client: the tail of
|
||||
`~/.pi/agent/mempalace-catchup.log` should end in `Done. Wing … updated.` with no `error:` line above
|
||||
it. With the fixed feeder a broken run exits 5 and names the reason.
|
||||
|
||||
## 3. Deliberately NOT done
|
||||
|
||||
> **⚠️ Historical — this section describes 2026-08-10 and is no longer true.** All five items were
|
||||
> done between 2026-08-12 and 2026-08-14. Kept because the *reasoning* for deferring them is still
|
||||
> the record of why the order was chosen. Current state per item is inlined below.
|
||||
|
||||
- **Nothing is serving.** No listener on 8765; no mempalace process. Re-verified at the end of the run.
|
||||
→ **Now serving** since 2026-08-12 (`mempalace-serve.service`, `systemctl --user`), reachable at
|
||||
`https://mempalace.jordbo.se/mcp` via newt/Pangolin.
|
||||
- **No client `.env` was touched.** Your working setup is exactly as you left it (R6: reversible).
|
||||
→ **One client flipped 2026-08-14**: four variables on EMB-7KJ4VR4G, `docker-compose.yaml` unchanged.
|
||||
Still reversible in ~30s (§3.8 of [`phase-1-exposure-runbook.md`](./phase-1-exposure-runbook.md)).
|
||||
- **No data joined.** The palace is empty. The §4.4 join needs the diary-dedup decision (§7.6) first —
|
||||
replaying diaries today duplicates them, and the primary is the one place that must stay clean.
|
||||
→ **Seeded 2026-08-14** from *one* palace by file-level copy. This sidestepped §7.6 rather than
|
||||
solving it: a file-level copy replays no diaries, so it cannot duplicate them. **§7.6 is still a
|
||||
hard blocker for the second machine to join.**
|
||||
- **nyvaken untouched.** Read nothing, changed nothing.
|
||||
- **No sudo.** `sudo -n` on synlig requires a password.
|
||||
|
||||
## 4. Tomorrow, in order
|
||||
|
||||
> **2026-08-12: items 1–2 and 5 now have their own runbook —**
|
||||
> [`phase-1-exposure-runbook.md`](./phase-1-exposure-runbook.md). Pangolin on nyvaken is updated (done),
|
||||
> and **newt is now installed on synlig and connected to Pangolin (done 2026-08-12)** — so the blocker is
|
||||
> now item 3, the one `sudo`. That doc also records why per-device Pangolin users are the wrong layer, why
|
||||
> the HTTPS tunnel and the feeder's SSH path are **not** redundant (§1.3), the client-flip variable trap
|
||||
> (§3.7), and an additional loopback finding: a loopback bind does not merely 403, it also silently starts
|
||||
> the server with **no token at all** (auto-minting is gated on the bind being non-loopback).
|
||||
|
||||
1. **Pangolin update on nyvaken** (yours). ✅ done 2026-08-12.
|
||||
2. ~~**⚠️ synlig has no tunnel client.**~~ ✅ **done 2026-08-12** — newt installed and connected to Pangolin.
|
||||
(Kept for the reasoning: `docker ps` showed only the Gitea runner and digikam. Pangolin on nyvaken
|
||||
cannot reach synlig by itself; synlig had to dial out. Easy to miss because Pangolin looks healthy on
|
||||
its own side — which is also why "connected" is not yet proof it can reach the palace: verify
|
||||
`172.17.0.1:8765/healthz` from *inside* newt's namespace, exposure runbook §3.3.) Since newt runs in
|
||||
Docker here, the docker0 bind above is already correct for it.
|
||||
3. **One sudo, then start** (the unit is already staged; just drop the suffix). ✅ **done 2026-08-12** —
|
||||
linger enabled, unit enabled, `172.17.0.1:8765/healthz` → `ok`.
|
||||
```sh
|
||||
sudo loginctl enable-linger ecsjper
|
||||
cd ~/.config/systemd/user && mv mempalace-serve.service.staged mempalace-serve.service
|
||||
systemctl --user daemon-reload && systemctl --user enable --now mempalace-serve
|
||||
curl -s 172.17.0.1:8765/healthz # ok
|
||||
curl -s 127.0.0.1:8765/healthz # NOTHING — refused, exit 7 (not 403; see below)
|
||||
ss -ltnp | grep 8765 # 172.17.0.1:8765 only
|
||||
```
|
||||
⚠ **Corrected 2026-08-12:** this line predicted `403`. The real run returned empty, which is *more*
|
||||
reassuring. With the docker0-only bind nothing listens on loopback, so the connection is refused before
|
||||
any header is sent (`%{http_code}` → `000`, `$?` → `7`). The 403 in §2.4 is the **loopback-bind** case:
|
||||
a server on `127.0.0.1` answering a proxy-forwarded foreign `Host:`. Two different failures that were
|
||||
collapsed into one expectation here.
|
||||
4. **Collect the shared token** (auto-minted on first non-loopback start, stable across restarts):
|
||||
```sh
|
||||
cat ~/.mempalace/server/f5d849287f6d73f0141b29d7/token
|
||||
```
|
||||
That directory name is `sha256(realpath(palace))[:24]` — it changes if the palace path ever changes.
|
||||
5. **Route it through Pangolin**, then verify `/healthz` end-to-end through the public hostname *before*
|
||||
pointing any client at it.
|
||||
6. **Then, and only then**, Phase 1 client flip — one machine first, and remember opencode containers
|
||||
need the §4.1 sidecar merge (or Phase 1.5) before the `.env` takes effect.
|
||||
7. **Before the first join:** settle §7.6 diary dedup, then dry-run §4.4 from **one** palace.
|
||||
> **Correction 2026-08-14 — do NOT verify a join "by checking counts", which is what this item
|
||||
> originally said.** Counts are not evidence, in either direction. `mempalace status` counts
|
||||
> **chunk rows**, not logical drawers (3 drawers plus one 2-chunk diary presented as +9), and chunk
|
||||
> counts legitimately differ between two palaces whenever a drawer was updated on either side,
|
||||
> because an update re-chunks to the new length and deletes the surplus rows. Diffing chunk-id sets
|
||||
> is a useful first pass but **over**-reports: a chunk id present on one side only is the ordinary
|
||||
> signature of an edit, not of loss. This cost real time on 2026-08-14 — a missing
|
||||
> `chunk_000007` was read as a truncated seed, when in fact the two palaces held two revisions of
|
||||
> one drawer and nothing was lost. **Adjudicate by fetching the parent drawer on both sides and
|
||||
> comparing the reassembled `content`.**
|
||||
|
||||
## 5. Rollback
|
||||
|
||||
> **⚠️ STOP — 2026-08-14. Do not run this block as it was originally written.** `~/.mempalace` on
|
||||
> synlig is now the fleet primary. That tree holds the only central palace (14,803 drawers, seeded
|
||||
> from EMB-7KJ4VR4G) **and the server's bearer token** at `~/.mempalace/server/<hash>/token` — the
|
||||
> single credential every flipped client authenticates with, of which there is no second copy.
|
||||
> `rm -rf ~/.mempalace` destroys both. The original comment ("empty today — check before running once
|
||||
> it isn't") is far too soft for a destructive command someone runs under pressure, which is exactly
|
||||
> why it is being replaced rather than amended.
|
||||
|
||||
Stopping the service is safe and reversible on its own, and is the whole of what "rollback" should
|
||||
normally mean now:
|
||||
|
||||
```sh
|
||||
systemctl --user disable --now mempalace-serve # clients fail CLOSED — they lose the palace
|
||||
# tools; they do NOT fall back to a local palace
|
||||
```
|
||||
|
||||
To genuinely decommission the primary, in this order:
|
||||
|
||||
1. Flip every client back first (§3.8 of [`phase-1-exposure-runbook.md`](./phase-1-exposure-runbook.md),
|
||||
in reverse) so nothing is pointed at a host that is about to lose its palace.
|
||||
2. Copy `~/.mempalace/palace/` **and** the token file off the host, and verify the copy by comparing
|
||||
reassembled drawer `content`, not counts (see §4 item 7).
|
||||
3. Only then remove anything. Never `rsync --delete` into `~/.mempalace` — the token lives inside it.
|
||||
|
||||
The two destructive steps below were written on 2026-08-10, when `~/.mempalace` was genuinely empty.
|
||||
Kept for the record; **must not be run while the primary is live**:
|
||||
|
||||
```sh
|
||||
~/.local/bin/uv tool uninstall mempalace
|
||||
rm -rf ~/.mempalace # ⚠️ DESTROYS THE FLEET PALACE AND THE ONLY TOKEN
|
||||
```
|
||||
If you are looking for the mechanism, the two runbooks above are the same
|
||||
procedures with `<PRIMARY_HOST>` and `<USER>` in place of one site's specifics.
|
||||
|
||||
Reference in New Issue
Block a user