mempalace-toolkit/docs/synlig-primary-runbook.md

synlig primary — Phase 0 runbook and handoff

Companion to rfc-001-global-palace.md. Records what was actually done on the primary, with verified evidence, so the next session (or the next machine) does not re-derive it.

Status 2026-08-14 17:00 — SUPERSEDED IN PART. The primary is live, exposed, and seeded. Serving since 2026-08-12 at https://mempalace.jordbo.se/mcp. Seeded 2026-08-14 15:07 from EMB-7KJ4VR4G's palace (itself a carry-over from the previous work computer EMB-X1JY06WJ — rfc-001 §4.4) — 14,777 drawers / 9 wings / 16,337 embeddings / KG 46 entities, 34 triples, now 14,803 drawers. One client (EMB-7KJ4VR4G's pi-devbox container) is flipped and verified end-to-end. Both Phase 0 blockers below are cleared.

Update 2026-08-16 00:20 — the transcript feed is live too, and it is the primary's third moving part (alongside the HTTPS tunnel and the palace itself). See §2.6: transcripts arrive over SSH into ~/mempalace-feed/<device>/ and are mined by this host's own server process. 15,478 drawers as of that check — but treat every count in this document as a timestamp, not a fact: status counts chunk rows, and §4 item 7 explains why counts adjudicate nothing.

Read §3 "Deliberately NOT done" as a record of the 2026-08-10 state, not of today's — every item in it has since been done. And before running anything in §5 Rollback, read the warning at the top of it: ~/.mempalace on synlig is no longer disposable.

Original status, kept for the record:

Status 2026-08-10 00:30 — Phase 0 prep complete. Not serving. Nothing exposed. Blocked on two things, both deliberately left to Joakim: the Pangolin update on nyvaken, and one sudo.

---

1. What synlig is (discovered, not assumed)

Fact	Value
SSH	synlig → synlig.erdc.ericsson.net, user ecsjper (from ~/.ssh/config)
OS	Ubuntu 24.04.4 LTS, 7.8 GiB RAM, 78 G disk (29 G free), uptime 12 d
Python / uv	system python3 3.12.3; uv at ~/.local/bin/uv (not on the non-login PATH)
Interfaces	lo 127.0.0.1, ens3 10.0.0.4/16, docker0 172.17.0.1/16, br-… 172.19.0.1/16
Already listening	22, 80, 443, 3000 (node), 3389 + 3350 + 4822 (xrdp/guacamole), 631
Docker	present; running act_runner-runner-1 (Gitea Actions runner) and digikam
Pre-existing MemPalace	none — no mempalace binary, no ~/.mempalace. Greenfield.

The Gitea Actions runner living here is worth remembering: synlig is not a dedicated appliance, and CI load competes with the palace for the same 7.8 GiB.

2. Done tonight

2.1 MemPalace installed, pinned to the fleet version

~/.local/bin/uv tool install "mempalace==3.6.0"     # → mempalace, mempalace-mcp

Pinned deliberately: the clients run 3.6.0, and the id recipes / idempotency probes this RFC leans on are version-specific. Reversible with uv tool uninstall mempalace.

2.2 Embedder model pre-warmed — the corporate-network risk that wasn't

The first embed pulls all-MiniLM-L6-v2 ONNX (79.3 MB) from the chroma CDN into ~/.cache/chroma/onnx_models/ (167 M on disk once unpacked). This was the main unknown — an egress-filtered work VM would have failed here, at the worst possible moment (first client write). It downloaded at ~20 MB/s with no proxy interference. Done in a throwaway palace, since deleted, so the real palace never saw it. Same model as the clients use, so the semantic space matches.

2.3 Palace created with the §7.1 landmine structurally removed

~/.mempalace/palace — the stock default, so no MEMPALACE_PALACE_PATH and no config.json is needed anywhere on synlig. One less thing to drift.

RFC §7.1 says to mv three HOME-anchored stores into the palace dir before first serve. On a greenfield primary there is nothing to move — but the hazard is not actually a migration hazard, and the RFC understated it: with stock defaults palace_path is ~/.mempalace/palace while DEFAULT_KG_PATH is ~/.mempalace/knowledge_graph.sqlite3. Those differ, so the split is the out-of-the-box behaviour, not a consequence of a custom path. It is permanent, not one-time: serve always passes --palace (KG inside the palace), while any CLI command run without --palace uses the HOME path. Two KGs on one box, forever, silently.

Fixed by making both resolution rules land on one inode:

ln -sfn palace/knowledge_graph.sqlite3 ~/.mempalace/knowledge_graph.sqlite3
ln -sfn palace/known_entities.json     ~/.mempalace/known_entities.json

Relative targets, so a home-directory move survives. hallways.json was originally left unsymlinked: it is already palace-derived, and its HOME path is a warning-only legacy probe (hallways.py:73-95) that never auto-migrates.

Update 2026-08-14 — hallways.json is now symlinked too, during the seeding session:

ln -sfn palace/hallways.json ~/.mempalace/hallways.json

Rationale changed: the point is no longer "only symlink what the code demands" but all real state lives under palace/ as a single backup unit, so one palace/ copy is a complete copy. All three parent-level paths now resolve, which matters because mempalace 3.6.0 resolves these three paths inconsistently (MCP server: palace-relative; KG CLI default: ~/.mempalace; hallways.json: dirname(palace_path); known_entities.json: hardcoded ~). Revert by deleting the symlink if it ever causes trouble.

Verified the symlink assumption rather than trusting it (python3 sqlite3 on synlig, temp dir):

Check	Result
Dangling symlink + sqlite3.connect	creates the target
-wal / -shm placement	next to the target, inside the palace dir — not beside the symlink
Write via symlink → read via palace path	same data, same inode

The WAL placement is the part that mattered: it keeps the palace directory a single self-contained backup/bind-mount unit.

2.4 §6.2's Host/Origin policy verified by experiment, not by reading

Ran on synlig, loopback and docker0 binds, then stopped. 11/11 as predicted:

#	Bind	Request	Expected	Got
A1	127.0.0.1	/healthz, correct Host	200	✅ 200
A2	127.0.0.1	/healthz, Host: palace.example.com	403	✅ 403
A3	127.0.0.1	/healthz, Origin: https://evil.example	403	✅ 403
A4	127.0.0.1	POST /mcp, no token	401	✅ 401
A5	127.0.0.1	POST /mcp, wrong token	401	✅ 401
A6	127.0.0.1	POST /mcp, correct token	200	✅ 200 (tools/list → 36 tools)
B1	172.17.0.1	/healthz, bound-host Host	200	✅ 200
B2	172.17.0.1	/healthz, Host: palace.example.com	200	✅ 200
B3	172.17.0.1	/healthz, Origin: https://evil.example	403	✅ 403
B4	172.17.0.1	/healthz, loopback Origin	200	✅ 200
B5	172.17.0.1	POST /mcp, foreign Host + token	200	✅ 200

Operational conclusions:

1. Do not bind loopback behind the tunnel. A2 vs B2 is the whole story: the reflex "bind 127.0.0.1, it's safer" produces a 403 that looks like a Pangolin misconfiguration and is not one.
2. Bind 172.17.0.1 (docker0). Non-loopback, so the Host pin relaxes — but reachable only from synlig and its containers, so a newt container on this box can reach it while the LAN cannot. This is strictly better than 0.0.0.0 here. It is what contrib/systemd/mempalace-serve.service uses.
3. Origin is never relaxed (B3). No browser-based MCP client, and no proxy that injects Origin.
4. /healthz is Host/Origin-gated but token-free — a usable liveness probe for the tunnel.

Test script kept at /tmp/synlig-phase0-test.sh on this container (ephemeral — re-create from the table above if needed; it starts, probes and stops the server, and asserts nothing is left listening).

2.5 A start unit — written, staged, and (since 2026-08-12) installed and running

This subsection describes 2026-08-10. The unit is now live. Verified 2026-08-16 00:15: systemctl --user list-units shows mempalace-serve.service … loaded active running, and the process is ~/.local/share/uv/tools/mempalace/bin/python -m mempalace.mcp_server --transport http --host 172.17.0.1 --port 8765 --palace /home/ecsjper/.mempalace/palace. Note what that means and §2.6 depends on: the server is a NATIVE process, not a container — it sees synlig's real filesystem paths, and docker ps on synlig lists no mempalace container.

contrib/systemd/mempalace-serve.service — user unit, follows the existing contrib/systemd/ style, carries the bind rationale inline so nobody "fixes" it back to loopback. A copy is already staged on synlig at ~/.config/systemd/user/mempalace-serve.service.staged — the .staged suffix is deliberate: systemd only reads *.service, so the file cannot be activated by accident, not even by a stray daemon-reload. Not installed, not enabled: it needs one sudo loginctl enable-linger, and standing up a network-reachable service while you were asleep was not mine to decide. (Both were done on 2026-08-12 — §4 item 3 has the exact commands that were run.)

2.6 The transcript inbox — ~/mempalace-feed/<device>/ (added 2026-08-16)

Flipped clients write drawers over HTTPS, but their session transcripts cannot travel that way: mempalace_mine resolves its source path in the server process, so the server cannot see a client's staged exports. mempalace-pi-session --mode remote therefore rsyncs each client's stage into a per-device inbox here and then asks the server to mine its own local path:

ls ~/mempalace-feed/                    # one dir per device, e.g. emb-7kj4vr4g/
ls ~/mempalace-feed/emb-7kj4vr4g/       # pi_<session-uuid>.jsonl, mtimes preserved

Client side, that needs three variables — and the third one is the trap:

Variable	Value for this fleet	Why
MEMPALACE_PI_SSH_TARGET	ecsjper@synlig:/home/ecsjper/mempalace-feed	where rsync puts the files
MEMPALACE_PI_DEVICE	e.g. emb-7kj4vr4g	inbox subdirectory per machine
MEMPALACE_PI_REMOTE_PATH	/home/ecsjper/mempalace-feed	the inbox as the server process sees it

The feeder's default for the third is /data/feed, which assumes a containerized palace server with the inbox bind-mounted there. This primary is native (§2.5), so it only ever sees host paths and the value must equal the path half of the SSH target. Get it wrong and the failure is quiet in the worst way: rsync succeeds, the files are all present here, and only the mine fails with source directory not found: '/data/feed/<device>'.

That is exactly what happened on 2026-08-15, and it went unnoticed for a session because the feeder decided success with '"error"' in body — MCP returns HTTP 200 with the tool's own JSON escaped inside result.content[].text, so those bytes are \"error\", the substring never matched, and ~/.pi/agent/mempalace-catchup.log printed Done. Wing 'wing_conversations' updated. directly under the error. Fixed in 6e1f4f3: the envelope is parsed, --self-test pins that exact response body, and a preflight warning fires whenever the ship path and MEMPALACE_PI_REMOTE_PATH disagree.

Operational notes for this inbox:

• Dedup keys on the absolute source path, so the inbox path is load-bearing: it must stay stable, or every transcript re-files under its new name. Migrating it on 2026-08-15 (from the clients' old container-local stage paths, which arrived with the seed) cost a full re-mine plus mempalace_delete_by_source on 6 old paths — 651 drawers purged, 1243 re-filed. wing_conversations is now keyed entirely on /home/ecsjper/mempalace-feed/<device>/….
• A grown session is purged and re-filed for the same path (mtime-based), so re-feeding a live session refreshes it instead of duplicating it. That is why the inbox keeps whole transcripts rather than deltas — do not "tidy" it by deleting files the palace still references.
• mempalace_mine over MCP can exceed a client's request timeout while the server keeps working and finishes normally. A client-side timeout is not a failed mine: check SELECT COUNT(*) FROM embedding_metadata WHERE key='source_file' AND string_value LIKE '<inbox>%' (read-only, file:…?mode=ro) before retrying anything.
• Health check after any client recreate, from the client: the tail of ~/.pi/agent/mempalace-catchup.log should end in Done. Wing … updated. with no error: line above it. With the fixed feeder a broken run exits 5 and names the reason.

3. Deliberately NOT done

⚠️ Historical — this section describes 2026-08-10 and is no longer true. All five items were done between 2026-08-12 and 2026-08-14. Kept because the reasoning for deferring them is still the record of why the order was chosen. Current state per item is inlined below.

• Nothing is serving. No listener on 8765; no mempalace process. Re-verified at the end of the run. → Now serving since 2026-08-12 (mempalace-serve.service, systemctl --user), reachable at https://mempalace.jordbo.se/mcp via newt/Pangolin.
• No client .env was touched. Your working setup is exactly as you left it (R6: reversible). → One client flipped 2026-08-14: four variables on EMB-7KJ4VR4G, docker-compose.yaml unchanged. Still reversible in ~30s (§3.8 of phase-1-exposure-runbook.md).
• No data joined. The palace is empty. The §4.4 join needs the diary-dedup decision (§7.6) first — replaying diaries today duplicates them, and the primary is the one place that must stay clean. → Seeded 2026-08-14 from one palace by file-level copy. This sidestepped §7.6 rather than solving it: a file-level copy replays no diaries, so it cannot duplicate them. §7.6 is still a hard blocker for the second machine to join.
• nyvaken untouched. Read nothing, changed nothing.
• No sudo. sudo -n on synlig requires a password.

4. Tomorrow, in order

2026-08-12: items 1–2 and 5 now have their own runbook — phase-1-exposure-runbook.md. Pangolin on nyvaken is updated (done), and newt is now installed on synlig and connected to Pangolin (done 2026-08-12) — so the blocker is now item 3, the one sudo. That doc also records why per-device Pangolin users are the wrong layer, why the HTTPS tunnel and the feeder's SSH path are not redundant (§1.3), the client-flip variable trap (§3.7), and an additional loopback finding: a loopback bind does not merely 403, it also silently starts the server with no token at all (auto-minting is gated on the bind being non-loopback).

1. Pangolin update on nyvaken (yours). ✅ done 2026-08-12.
2. ⚠️ synlig has no tunnel client. ✅ done 2026-08-12 — newt installed and connected to Pangolin. (Kept for the reasoning: docker ps showed only the Gitea runner and digikam. Pangolin on nyvaken cannot reach synlig by itself; synlig had to dial out. Easy to miss because Pangolin looks healthy on its own side — which is also why "connected" is not yet proof it can reach the palace: verify 172.17.0.1:8765/healthz from inside newt's namespace, exposure runbook §3.3.) Since newt runs in Docker here, the docker0 bind above is already correct for it.
3. One sudo, then start (the unit is already staged; just drop the suffix). ✅ done 2026-08-12 — linger enabled, unit enabled, 172.17.0.1:8765/healthz → ok.

   sudo loginctl enable-linger ecsjper
   cd ~/.config/systemd/user && mv mempalace-serve.service.staged mempalace-serve.service
   systemctl --user daemon-reload && systemctl --user enable --now mempalace-serve
   curl -s 172.17.0.1:8765/healthz          # ok
   curl -s 127.0.0.1:8765/healthz           # NOTHING — refused, exit 7 (not 403; see below)
   ss -ltnp | grep 8765                     # 172.17.0.1:8765 only
   

   ⚠ Corrected 2026-08-12: this line predicted 403. The real run returned empty, which is more reassuring. With the docker0-only bind nothing listens on loopback, so the connection is refused before any header is sent (%{http_code} → 000, $? → 7). The 403 in §2.4 is the loopback-bind case: a server on 127.0.0.1 answering a proxy-forwarded foreign Host:. Two different failures that were collapsed into one expectation here.
4. Collect the shared token (auto-minted on first non-loopback start, stable across restarts):

   cat ~/.mempalace/server/f5d849287f6d73f0141b29d7/token
   

   That directory name is sha256(realpath(palace))[:24] — it changes if the palace path ever changes.
5. Route it through Pangolin, then verify /healthz end-to-end through the public hostname before pointing any client at it.
6. Then, and only then, Phase 1 client flip — one machine first, and remember opencode containers need the §4.1 sidecar merge (or Phase 1.5) before the .env takes effect.
7. Before the first join: settle §7.6 diary dedup, then dry-run §4.4 from one palace.

   Correction 2026-08-14 — do NOT verify a join "by checking counts", which is what this item originally said. Counts are not evidence, in either direction. mempalace status counts chunk rows, not logical drawers (3 drawers plus one 2-chunk diary presented as +9), and chunk counts legitimately differ between two palaces whenever a drawer was updated on either side, because an update re-chunks to the new length and deletes the surplus rows. Diffing chunk-id sets is a useful first pass but over-reports: a chunk id present on one side only is the ordinary signature of an edit, not of loss. This cost real time on 2026-08-14 — a missing chunk_000007 was read as a truncated seed, when in fact the two palaces held two revisions of one drawer and nothing was lost. Adjudicate by fetching the parent drawer on both sides and comparing the reassembled content.

5. Rollback

⚠️ STOP — 2026-08-14. Do not run this block as it was originally written. ~/.mempalace on synlig is now the fleet primary. That tree holds the only central palace (14,803 drawers, seeded from EMB-7KJ4VR4G) and the server's bearer token at ~/.mempalace/server/<hash>/token — the single credential every flipped client authenticates with, of which there is no second copy. rm -rf ~/.mempalace destroys both. The original comment ("empty today — check before running once it isn't") is far too soft for a destructive command someone runs under pressure, which is exactly why it is being replaced rather than amended.

Stopping the service is safe and reversible on its own, and is the whole of what "rollback" should normally mean now:

systemctl --user disable --now mempalace-serve      # clients fail CLOSED — they lose the palace
                                                    # tools; they do NOT fall back to a local palace

To genuinely decommission the primary, in this order:

1. Flip every client back first (§3.8 of phase-1-exposure-runbook.md, in reverse) so nothing is pointed at a host that is about to lose its palace.
2. Copy ~/.mempalace/palace/ and the token file off the host, and verify the copy by comparing reassembled drawer content, not counts (see §4 item 7).
3. Only then remove anything. Never rsync --delete into ~/.mempalace — the token lives inside it.

The two destructive steps below were written on 2026-08-10, when ~/.mempalace was genuinely empty. Kept for the record; must not be run while the primary is live:

~/.local/bin/uv tool uninstall mempalace
rm -rf ~/.mempalace                                 # ⚠️ DESTROYS THE FLEET PALACE AND THE ONLY TOKEN