mempalace-toolkit/docs/synlig-primary-runbook.md

synlig primary — Phase 0 runbook and handoff

Companion to rfc-001-global-palace.md. Records what was actually done on the primary, with verified evidence, so the next session (or the next machine) does not re-derive it.

Status 2026-08-10 00:30 — Phase 0 prep complete. Not serving. Nothing exposed. Blocked on two things, both deliberately left to Joakim: the Pangolin update on nyvaken, and one sudo.

---

1. What synlig is (discovered, not assumed)

Fact	Value
SSH	synlig → synlig.erdc.ericsson.net, user ecsjper (from ~/.ssh/config)
OS	Ubuntu 24.04.4 LTS, 7.8 GiB RAM, 78 G disk (29 G free), uptime 12 d
Python / uv	system python3 3.12.3; uv at ~/.local/bin/uv (not on the non-login PATH)
Interfaces	lo 127.0.0.1, ens3 10.0.0.4/16, docker0 172.17.0.1/16, br-… 172.19.0.1/16
Already listening	22, 80, 443, 3000 (node), 3389 + 3350 + 4822 (xrdp/guacamole), 631
Docker	present; running act_runner-runner-1 (Gitea Actions runner) and digikam
Pre-existing MemPalace	none — no mempalace binary, no ~/.mempalace. Greenfield.

The Gitea Actions runner living here is worth remembering: synlig is not a dedicated appliance, and CI load competes with the palace for the same 7.8 GiB.

2. Done tonight

2.1 MemPalace installed, pinned to the fleet version

~/.local/bin/uv tool install "mempalace==3.6.0"     # → mempalace, mempalace-mcp

Pinned deliberately: the clients run 3.6.0, and the id recipes / idempotency probes this RFC leans on are version-specific. Reversible with uv tool uninstall mempalace.

2.2 Embedder model pre-warmed — the corporate-network risk that wasn't

The first embed pulls all-MiniLM-L6-v2 ONNX (79.3 MB) from the chroma CDN into ~/.cache/chroma/onnx_models/ (167 M on disk once unpacked). This was the main unknown — an egress-filtered work VM would have failed here, at the worst possible moment (first client write). It downloaded at ~20 MB/s with no proxy interference. Done in a throwaway palace, since deleted, so the real palace never saw it. Same model as the clients use, so the semantic space matches.

2.3 Palace created with the §7.1 landmine structurally removed

~/.mempalace/palace — the stock default, so no MEMPALACE_PALACE_PATH and no config.json is needed anywhere on synlig. One less thing to drift.

RFC §7.1 says to mv three HOME-anchored stores into the palace dir before first serve. On a greenfield primary there is nothing to move — but the hazard is not actually a migration hazard, and the RFC understated it: with stock defaults palace_path is ~/.mempalace/palace while DEFAULT_KG_PATH is ~/.mempalace/knowledge_graph.sqlite3. Those differ, so the split is the out-of-the-box behaviour, not a consequence of a custom path. It is permanent, not one-time: serve always passes --palace (KG inside the palace), while any CLI command run without --palace uses the HOME path. Two KGs on one box, forever, silently.

Fixed by making both resolution rules land on one inode:

ln -sfn palace/knowledge_graph.sqlite3 ~/.mempalace/knowledge_graph.sqlite3
ln -sfn palace/known_entities.json     ~/.mempalace/known_entities.json

Relative targets, so a home-directory move survives. hallways.json is deliberately not symlinked: it is already palace-derived, and its HOME path is a warning-only legacy probe (hallways.py:73-95) that never auto-migrates.

Verified the symlink assumption rather than trusting it (python3 sqlite3 on synlig, temp dir):

Check	Result
Dangling symlink + sqlite3.connect	creates the target
-wal / -shm placement	next to the target, inside the palace dir — not beside the symlink
Write via symlink → read via palace path	same data, same inode

The WAL placement is the part that mattered: it keeps the palace directory a single self-contained backup/bind-mount unit.

2.4 §6.2's Host/Origin policy verified by experiment, not by reading

Ran on synlig, loopback and docker0 binds, then stopped. 11/11 as predicted:

#	Bind	Request	Expected	Got
A1	127.0.0.1	/healthz, correct Host	200	✅ 200
A2	127.0.0.1	/healthz, Host: palace.example.com	403	✅ 403
A3	127.0.0.1	/healthz, Origin: https://evil.example	403	✅ 403
A4	127.0.0.1	POST /mcp, no token	401	✅ 401
A5	127.0.0.1	POST /mcp, wrong token	401	✅ 401
A6	127.0.0.1	POST /mcp, correct token	200	✅ 200 (tools/list → 36 tools)
B1	172.17.0.1	/healthz, bound-host Host	200	✅ 200
B2	172.17.0.1	/healthz, Host: palace.example.com	200	✅ 200
B3	172.17.0.1	/healthz, Origin: https://evil.example	403	✅ 403
B4	172.17.0.1	/healthz, loopback Origin	200	✅ 200
B5	172.17.0.1	POST /mcp, foreign Host + token	200	✅ 200

Operational conclusions:

1. Do not bind loopback behind the tunnel. A2 vs B2 is the whole story: the reflex "bind 127.0.0.1, it's safer" produces a 403 that looks like a Pangolin misconfiguration and is not one.
2. Bind 172.17.0.1 (docker0). Non-loopback, so the Host pin relaxes — but reachable only from synlig and its containers, so a newt container on this box can reach it while the LAN cannot. This is strictly better than 0.0.0.0 here. It is what contrib/systemd/mempalace-serve.service uses.
3. Origin is never relaxed (B3). No browser-based MCP client, and no proxy that injects Origin.
4. /healthz is Host/Origin-gated but token-free — a usable liveness probe for the tunnel.

Test script kept at /tmp/synlig-phase0-test.sh on this container (ephemeral — re-create from the table above if needed; it starts, probes and stops the server, and asserts nothing is left listening).

2.5 A start unit, written but not installed

contrib/systemd/mempalace-serve.service — user unit, follows the existing contrib/systemd/ style, carries the bind rationale inline so nobody "fixes" it back to loopback. A copy is already staged on synlig at ~/.config/systemd/user/mempalace-serve.service.staged — the .staged suffix is deliberate: systemd only reads *.service, so the file cannot be activated by accident, not even by a stray daemon-reload. Not installed, not enabled: it needs one sudo loginctl enable-linger, and standing up a network-reachable service while you were asleep was not mine to decide.

3. Deliberately NOT done

• Nothing is serving. No listener on 8765; no mempalace process. Re-verified at the end of the run.
• No client .env was touched. Your working setup is exactly as you left it (R6: reversible).
• No data joined. The palace is empty. The §4.4 join needs the diary-dedup decision (§7.6) first — replaying diaries today duplicates them, and the primary is the one place that must stay clean.
• nyvaken untouched. Read nothing, changed nothing.
• No sudo. sudo -n on synlig requires a password.

4. Tomorrow, in order

2026-08-12: items 1–2 and 5 now have their own runbook — phase-1-exposure-runbook.md. Pangolin on nyvaken is updated (done), and newt is now installed on synlig and connected to Pangolin (done 2026-08-12) — so the blocker is now item 3, the one sudo. That doc also records why per-device Pangolin users are the wrong layer, why the HTTPS tunnel and the feeder's SSH path are not redundant (§1.3), the client-flip variable trap (§3.7), and an additional loopback finding: a loopback bind does not merely 403, it also silently starts the server with no token at all (auto-minting is gated on the bind being non-loopback).

1. Pangolin update on nyvaken (yours). ✅ done 2026-08-12.
2. ⚠️ synlig has no tunnel client. ✅ done 2026-08-12 — newt installed and connected to Pangolin. (Kept for the reasoning: docker ps showed only the Gitea runner and digikam. Pangolin on nyvaken cannot reach synlig by itself; synlig had to dial out. Easy to miss because Pangolin looks healthy on its own side — which is also why "connected" is not yet proof it can reach the palace: verify 172.17.0.1:8765/healthz from inside newt's namespace, exposure runbook §3.3.) Since newt runs in Docker here, the docker0 bind above is already correct for it.
3. One sudo, then start (the unit is already staged; just drop the suffix). ✅ done 2026-08-12 — linger enabled, unit enabled, 172.17.0.1:8765/healthz → ok.

   sudo loginctl enable-linger ecsjper
   cd ~/.config/systemd/user && mv mempalace-serve.service.staged mempalace-serve.service
   systemctl --user daemon-reload && systemctl --user enable --now mempalace-serve
   curl -s 172.17.0.1:8765/healthz          # ok
   curl -s 127.0.0.1:8765/healthz           # NOTHING — refused, exit 7 (not 403; see below)
   ss -ltnp | grep 8765                     # 172.17.0.1:8765 only
   

   ⚠ Corrected 2026-08-12: this line predicted 403. The real run returned empty, which is more reassuring. With the docker0-only bind nothing listens on loopback, so the connection is refused before any header is sent (%{http_code} → 000, $? → 7). The 403 in §2.4 is the loopback-bind case: a server on 127.0.0.1 answering a proxy-forwarded foreign Host:. Two different failures that were collapsed into one expectation here.
4. Collect the shared token (auto-minted on first non-loopback start, stable across restarts):

   cat ~/.mempalace/server/f5d849287f6d73f0141b29d7/token
   

   That directory name is sha256(realpath(palace))[:24] — it changes if the palace path ever changes.
5. Route it through Pangolin, then verify /healthz end-to-end through the public hostname before pointing any client at it.
6. Then, and only then, Phase 1 client flip — one machine first, and remember opencode containers need the §4.1 sidecar merge (or Phase 1.5) before the .env takes effect.
7. Before the first join: settle §7.6 diary dedup, then dry-run §4.4 from one palace and check counts.

5. Rollback

systemctl --user disable --now mempalace-serve     # if enabled
~/.local/bin/uv tool uninstall mempalace
rm -rf ~/.mempalace                                # empty today — check before running once it isn't