Phase 0 on synlig: install, palace layout, verified Host/Origin policy
synlig is greenfield — no mempalace, no ~/.mempalace at all — so Phase 0 became "provision correctly from birth" rather than "migrate carefully". Nothing is serving; no client config was touched. Done: - mempalace 3.6.0 installed via uv, pinned to the fleet version (the id recipes and idempotency probes this RFC leans on are version-specific). - Embedder pre-warmed. This was the real unknown: the first embed pulls a 79.3 MB ONNX model from the chroma CDN, and an egress-filtered work VM would have failed at the worst moment — the first client write. Pulled at ~20 MB/s, no proxy interference. Done in a throwaway palace so the real one never saw it. - Palace at the stock default ~/.mempalace/palace, so no config file and no MEMPALACE_PALACE_PATH is needed on synlig at all. Two corrections to the RFC, both from provisioning rather than reading: - §7.1 was understated. DEFAULT_PALACE_PATH (~/.mempalace/palace) and DEFAULT_KG_PATH (~/.mempalace/knowledge_graph.sqlite3) already differ with stock defaults, so the KG split is out-of-the-box behaviour, not a consequence of a custom --palace, and it is permanent rather than one-time: serve always passes --palace, any CLI call without it uses the HOME path. A one-time mv does not fix that, it only picks which of the two files gets populated. Fixed instead by converging both rules on one inode via relative symlinks, and verified the load-bearing assumption: a dangling symlink is created on connect, -wal/-shm land next to the target (so the palace dir stays a self-contained backup unit, which is the part that mattered), cross-path read works, same inode. hallways.json deliberately left alone — already palace-derived, HOME path is a warning-only probe. - §6.2 upgraded from "test early" to verified: 11/11 as predicted. The headline is that the safe-sounding reflex is the failure mode — loopback bind + proxy forwarding a public Host is 403, non-loopback is 200. Also confirmed Origin is never relaxed on either bind, and /healthz is Host/Origin-gated but token-free, so it works as the tunnel probe. Recommends binding the docker0 gateway over 0.0.0.0: non-loopback so the pin relaxes, but reachable only from the host and its containers. New docs/synlig-primary-runbook.md carries the discovered facts about the box, the evidence tables, an explicit "deliberately not done" list, and tomorrow's ordered steps. New contrib/systemd/mempalace-serve.service carries the bind rationale inline so nobody "fixes" it back to loopback; staged on synlig with a .staged suffix so systemd cannot pick it up by accident. Flagged for tomorrow: synlig has no newt/tunnel client (docker ps shows only the Gitea runner and digikam), so Pangolin on nyvaken cannot reach it until one is added — easy to miss, because Pangolin will look healthy from its own side.
This commit is contained in:
Joakim Persson
@@ -0,0 +1,162 @@
|
||||
# synlig primary — Phase 0 runbook and handoff
|
||||
|
||||
Companion to [`rfc-001-global-palace.md`](./rfc-001-global-palace.md). Records what was actually done
|
||||
on the primary, with verified evidence, so the next session (or the next machine) does not re-derive it.
|
||||
|
||||
**Status 2026-08-10 00:30 — Phase 0 prep complete. Not serving. Nothing exposed.**
|
||||
Blocked on two things, both deliberately left to Joakim: the Pangolin update on nyvaken, and one `sudo`.
|
||||
|
||||
---
|
||||
|
||||
## 1. What synlig is (discovered, not assumed)
|
||||
|
||||
| Fact | Value |
|
||||
| --- | --- |
|
||||
| SSH | `synlig` → `synlig.erdc.ericsson.net`, user `ecsjper` (from `~/.ssh/config`) |
|
||||
| OS | Ubuntu 24.04.4 LTS, 7.8 GiB RAM, 78 G disk (**29 G free**), uptime 12 d |
|
||||
| Python / uv | system `python3` 3.12.3; `uv` at `~/.local/bin/uv` (**not** on the non-login `PATH`) |
|
||||
| Interfaces | `lo` 127.0.0.1, `ens3` 10.0.0.4/16, `docker0` 172.17.0.1/16, `br-…` 172.19.0.1/16 |
|
||||
| Already listening | 22, 80, 443, 3000 (node), 3389 + 3350 + 4822 (xrdp/guacamole), 631 |
|
||||
| Docker | present; running `act_runner-runner-1` (**Gitea Actions runner**) and `digikam` |
|
||||
| Pre-existing MemPalace | **none** — no `mempalace` binary, no `~/.mempalace`. Greenfield. |
|
||||
|
||||
The Gitea Actions runner living here is worth remembering: synlig is not a dedicated appliance, and CI
|
||||
load competes with the palace for the same 7.8 GiB.
|
||||
|
||||
## 2. Done tonight
|
||||
|
||||
### 2.1 MemPalace installed, pinned to the fleet version
|
||||
|
||||
```sh
|
||||
~/.local/bin/uv tool install "mempalace==3.6.0" # → mempalace, mempalace-mcp
|
||||
```
|
||||
|
||||
Pinned deliberately: the clients run 3.6.0, and the id recipes / idempotency probes this RFC leans on are
|
||||
version-specific. Reversible with `uv tool uninstall mempalace`.
|
||||
|
||||
### 2.2 Embedder model pre-warmed — the corporate-network risk that wasn't
|
||||
|
||||
The first embed pulls `all-MiniLM-L6-v2` ONNX (79.3 MB) from the chroma CDN into
|
||||
`~/.cache/chroma/onnx_models/` (167 M on disk once unpacked). **This was the main unknown** — an
|
||||
egress-filtered work VM would have failed here, at the worst possible moment (first client write).
|
||||
It downloaded at ~20 MB/s with no proxy interference. Done in a throwaway palace, since deleted, so the
|
||||
real palace never saw it. Same model as the clients use, so the semantic space matches.
|
||||
|
||||
### 2.3 Palace created with the §7.1 landmine structurally removed
|
||||
|
||||
`~/.mempalace/palace` — the **stock default**, so no `MEMPALACE_PALACE_PATH` and no `config.json` is
|
||||
needed anywhere on synlig. One less thing to drift.
|
||||
|
||||
RFC §7.1 says to `mv` three HOME-anchored stores into the palace dir before first `serve`. **On a
|
||||
greenfield primary there is nothing to move — but the hazard is not actually a migration hazard, and the
|
||||
RFC understated it:** with *stock defaults* `palace_path` is `~/.mempalace/palace` while `DEFAULT_KG_PATH`
|
||||
is `~/.mempalace/knowledge_graph.sqlite3`. Those differ, so the split is the **out-of-the-box** behaviour,
|
||||
not a consequence of a custom path. It is permanent, not one-time: `serve` always passes `--palace` (KG
|
||||
inside the palace), while any CLI command run *without* `--palace` uses the HOME path. Two KGs on one box,
|
||||
forever, silently.
|
||||
|
||||
Fixed by making both resolution rules land on one inode:
|
||||
|
||||
```sh
|
||||
ln -sfn palace/knowledge_graph.sqlite3 ~/.mempalace/knowledge_graph.sqlite3
|
||||
ln -sfn palace/known_entities.json ~/.mempalace/known_entities.json
|
||||
```
|
||||
|
||||
Relative targets, so a home-directory move survives. `hallways.json` is deliberately **not** symlinked:
|
||||
it is already palace-derived, and its HOME path is a warning-only legacy probe (`hallways.py:73-95`) that
|
||||
never auto-migrates.
|
||||
|
||||
Verified the symlink assumption rather than trusting it (`python3 sqlite3` on synlig, temp dir):
|
||||
|
||||
| Check | Result |
|
||||
| --- | --- |
|
||||
| Dangling symlink + `sqlite3.connect` | creates the target |
|
||||
| `-wal` / `-shm` placement | next to the **target**, inside the palace dir — *not* beside the symlink |
|
||||
| Write via symlink → read via palace path | same data, **same inode** |
|
||||
|
||||
The WAL placement is the part that mattered: it keeps the palace directory a single self-contained
|
||||
backup/bind-mount unit.
|
||||
|
||||
### 2.4 §6.2's Host/Origin policy verified by experiment, not by reading
|
||||
|
||||
Ran on synlig, loopback and docker0 binds, then stopped. **11/11 as predicted:**
|
||||
|
||||
| # | Bind | Request | Expected | Got |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| A1 | 127.0.0.1 | `/healthz`, correct Host | 200 | ✅ 200 |
|
||||
| A2 | 127.0.0.1 | `/healthz`, `Host: palace.example.com` | **403** | ✅ 403 |
|
||||
| A3 | 127.0.0.1 | `/healthz`, `Origin: https://evil.example` | 403 | ✅ 403 |
|
||||
| A4 | 127.0.0.1 | `POST /mcp`, no token | 401 | ✅ 401 |
|
||||
| A5 | 127.0.0.1 | `POST /mcp`, wrong token | 401 | ✅ 401 |
|
||||
| A6 | 127.0.0.1 | `POST /mcp`, correct token | 200 | ✅ 200 (`tools/list` → **36 tools**) |
|
||||
| B1 | 172.17.0.1 | `/healthz`, bound-host Host | 200 | ✅ 200 |
|
||||
| B2 | 172.17.0.1 | `/healthz`, `Host: palace.example.com` | **200** | ✅ 200 |
|
||||
| B3 | 172.17.0.1 | `/healthz`, `Origin: https://evil.example` | 403 | ✅ 403 |
|
||||
| B4 | 172.17.0.1 | `/healthz`, loopback Origin | 200 | ✅ 200 |
|
||||
| B5 | 172.17.0.1 | `POST /mcp`, foreign Host + token | 200 | ✅ 200 |
|
||||
|
||||
**Operational conclusions:**
|
||||
|
||||
1. **Do not bind loopback behind the tunnel.** A2 vs B2 is the whole story: the reflex "bind 127.0.0.1,
|
||||
it's safer" produces a 403 that looks like a Pangolin misconfiguration and is not one.
|
||||
2. **Bind `172.17.0.1` (docker0).** Non-loopback, so the Host pin relaxes — but reachable only from
|
||||
synlig and its containers, so a newt container on this box can reach it while the LAN cannot. This is
|
||||
strictly better than `0.0.0.0` here. It is what `contrib/systemd/mempalace-serve.service` uses.
|
||||
3. **`Origin` is never relaxed** (B3). No browser-based MCP client, and no proxy that injects `Origin`.
|
||||
4. `/healthz` is Host/Origin-gated but token-free — a usable liveness probe for the tunnel.
|
||||
|
||||
Test script kept at `/tmp/synlig-phase0-test.sh` on this container (ephemeral — re-create from the table
|
||||
above if needed; it starts, probes and stops the server, and asserts nothing is left listening).
|
||||
|
||||
### 2.5 A start unit, written but not installed
|
||||
|
||||
`contrib/systemd/mempalace-serve.service` — user unit, follows the existing `contrib/systemd/` style,
|
||||
carries the bind rationale inline so nobody "fixes" it back to loopback. A copy is already staged on synlig
|
||||
at `~/.config/systemd/user/mempalace-serve.service.staged` — **the `.staged` suffix is deliberate**:
|
||||
systemd only reads `*.service`, so the file cannot be activated by accident, not even by a stray
|
||||
`daemon-reload`. **Not** installed, **not** enabled: it needs one `sudo loginctl enable-linger`, and
|
||||
standing up a network-reachable service while you were asleep was not mine to decide.
|
||||
|
||||
## 3. Deliberately NOT done
|
||||
|
||||
- **Nothing is serving.** No listener on 8765; no mempalace process. Re-verified at the end of the run.
|
||||
- **No client `.env` was touched.** Your working setup is exactly as you left it (R6: reversible).
|
||||
- **No data joined.** The palace is empty. The §4.4 join needs the diary-dedup decision (§7.6) first —
|
||||
replaying diaries today duplicates them, and the primary is the one place that must stay clean.
|
||||
- **nyvaken untouched.** Read nothing, changed nothing.
|
||||
- **No sudo.** `sudo -n` on synlig requires a password.
|
||||
|
||||
## 4. Tomorrow, in order
|
||||
|
||||
1. **Pangolin update on nyvaken** (yours).
|
||||
2. **⚠️ synlig has no tunnel client.** `docker ps` shows only the Gitea runner and digikam — no `newt`.
|
||||
Pangolin on nyvaken cannot reach synlig by itself; synlig needs a newt container (or an equivalent
|
||||
outbound tunnel) added. Easy to miss because Pangolin will look healthy on its own side. If newt runs
|
||||
in Docker here, the docker0 bind above is already correct for it.
|
||||
3. **One sudo, then start** (the unit is already staged; just drop the suffix):
|
||||
```sh
|
||||
sudo loginctl enable-linger ecsjper
|
||||
cd ~/.config/systemd/user && mv mempalace-serve.service.staged mempalace-serve.service
|
||||
systemctl --user daemon-reload && systemctl --user enable --now mempalace-serve
|
||||
curl -s 172.17.0.1:8765/healthz # ok
|
||||
curl -s 127.0.0.1:8765/healthz # 403 — expected, not a bug (§2.4)
|
||||
```
|
||||
4. **Collect the shared token** (auto-minted on first non-loopback start, stable across restarts):
|
||||
```sh
|
||||
cat ~/.mempalace/server/f5d849287f6d73f0141b29d7/token
|
||||
```
|
||||
That directory name is `sha256(realpath(palace))[:24]` — it changes if the palace path ever changes.
|
||||
5. **Route it through Pangolin**, then verify `/healthz` end-to-end through the public hostname *before*
|
||||
pointing any client at it.
|
||||
6. **Then, and only then**, Phase 1 client flip — one machine first, and remember opencode containers
|
||||
need the §4.1 sidecar merge (or Phase 1.5) before the `.env` takes effect.
|
||||
7. **Before the first join:** settle §7.6 diary dedup, then dry-run §4.4 from **one** palace and check
|
||||
counts.
|
||||
|
||||
## 5. Rollback
|
||||
|
||||
```sh
|
||||
systemctl --user disable --now mempalace-serve # if enabled
|
||||
~/.local/bin/uv tool uninstall mempalace
|
||||
rm -rf ~/.mempalace # empty today — check before running once it isn't
|
||||
```
|
||||
Reference in New Issue
Block a user