Joakim Persson
08e344b047
Both corrections come from the first real Phase 1 start on synlig (2026-08-12).
1. The Pangolin resource target was documented as bare `172.17.0.1:8765` with no
scheme, and the obvious guess from that is `https://` -- which cannot work.
`contrib/systemd/mempalace-serve.service` runs `serve --host 172.17.0.1
--port 8765` with no --tls-cert, so the primary speaks plaintext HTTP; TLS
terminates at Pangolin, which is the whole point of the RFC 6.2 decision.
Point a proxy at https:// and it attempts a TLS handshake against a plaintext
listener: 502 from outside, while `curl 172.17.0.1:8765/healthz` on the box
still says ok -- a confusing pair of symptoms. Now spelled `http://` with the
failure mode named, in the runbook and in the unit's comments.
2. `curl -s 127.0.0.1:8765/healthz` was documented as "expect 403". Wrong: the
real run returns empty. With a docker0-only bind nothing is listening on
loopback, so the connection is refused at TCP level before any header is sent
(%{http_code} -> 000, exit 7). The 403 is the *loopback-bind* case verified
2026-08-10 -- server on 127.0.0.1 answering a proxy-forwarded foreign Host.
Two distinct behaviours had been collapsed into one expectation in three
places (both runbooks and the unit).
Worth stating why the correction matters rather than just fixing it: refusal
is the *stronger* signal. A 403 proves only that a request was rejected; a
refused connection proves the loopback and LAN surface is not listening at
all. Someone who expected 403, saw silence, and "fixed" it by rebinding to
0.0.0.0 would have converted a correct configuration into an exposed one.
The docs now also say what to do if it hangs, or if ss shows 0.0.0.0:8765.
180 lines
10 KiB
Markdown
180 lines
10 KiB
Markdown
# synlig primary — Phase 0 runbook and handoff
|
||
|
||
Companion to [`rfc-001-global-palace.md`](./rfc-001-global-palace.md). Records what was actually done
|
||
on the primary, with verified evidence, so the next session (or the next machine) does not re-derive it.
|
||
|
||
**Status 2026-08-10 00:30 — Phase 0 prep complete. Not serving. Nothing exposed.**
|
||
Blocked on two things, both deliberately left to Joakim: the Pangolin update on nyvaken, and one `sudo`.
|
||
|
||
---
|
||
|
||
## 1. What synlig is (discovered, not assumed)
|
||
|
||
| Fact | Value |
|
||
| --- | --- |
|
||
| SSH | `synlig` → `synlig.erdc.ericsson.net`, user `ecsjper` (from `~/.ssh/config`) |
|
||
| OS | Ubuntu 24.04.4 LTS, 7.8 GiB RAM, 78 G disk (**29 G free**), uptime 12 d |
|
||
| Python / uv | system `python3` 3.12.3; `uv` at `~/.local/bin/uv` (**not** on the non-login `PATH`) |
|
||
| Interfaces | `lo` 127.0.0.1, `ens3` 10.0.0.4/16, `docker0` 172.17.0.1/16, `br-…` 172.19.0.1/16 |
|
||
| Already listening | 22, 80, 443, 3000 (node), 3389 + 3350 + 4822 (xrdp/guacamole), 631 |
|
||
| Docker | present; running `act_runner-runner-1` (**Gitea Actions runner**) and `digikam` |
|
||
| Pre-existing MemPalace | **none** — no `mempalace` binary, no `~/.mempalace`. Greenfield. |
|
||
|
||
The Gitea Actions runner living here is worth remembering: synlig is not a dedicated appliance, and CI
|
||
load competes with the palace for the same 7.8 GiB.
|
||
|
||
## 2. Done tonight
|
||
|
||
### 2.1 MemPalace installed, pinned to the fleet version
|
||
|
||
```sh
|
||
~/.local/bin/uv tool install "mempalace==3.6.0" # → mempalace, mempalace-mcp
|
||
```
|
||
|
||
Pinned deliberately: the clients run 3.6.0, and the id recipes / idempotency probes this RFC leans on are
|
||
version-specific. Reversible with `uv tool uninstall mempalace`.
|
||
|
||
### 2.2 Embedder model pre-warmed — the corporate-network risk that wasn't
|
||
|
||
The first embed pulls `all-MiniLM-L6-v2` ONNX (79.3 MB) from the chroma CDN into
|
||
`~/.cache/chroma/onnx_models/` (167 M on disk once unpacked). **This was the main unknown** — an
|
||
egress-filtered work VM would have failed here, at the worst possible moment (first client write).
|
||
It downloaded at ~20 MB/s with no proxy interference. Done in a throwaway palace, since deleted, so the
|
||
real palace never saw it. Same model as the clients use, so the semantic space matches.
|
||
|
||
### 2.3 Palace created with the §7.1 landmine structurally removed
|
||
|
||
`~/.mempalace/palace` — the **stock default**, so no `MEMPALACE_PALACE_PATH` and no `config.json` is
|
||
needed anywhere on synlig. One less thing to drift.
|
||
|
||
RFC §7.1 says to `mv` three HOME-anchored stores into the palace dir before first `serve`. **On a
|
||
greenfield primary there is nothing to move — but the hazard is not actually a migration hazard, and the
|
||
RFC understated it:** with *stock defaults* `palace_path` is `~/.mempalace/palace` while `DEFAULT_KG_PATH`
|
||
is `~/.mempalace/knowledge_graph.sqlite3`. Those differ, so the split is the **out-of-the-box** behaviour,
|
||
not a consequence of a custom path. It is permanent, not one-time: `serve` always passes `--palace` (KG
|
||
inside the palace), while any CLI command run *without* `--palace` uses the HOME path. Two KGs on one box,
|
||
forever, silently.
|
||
|
||
Fixed by making both resolution rules land on one inode:
|
||
|
||
```sh
|
||
ln -sfn palace/knowledge_graph.sqlite3 ~/.mempalace/knowledge_graph.sqlite3
|
||
ln -sfn palace/known_entities.json ~/.mempalace/known_entities.json
|
||
```
|
||
|
||
Relative targets, so a home-directory move survives. `hallways.json` is deliberately **not** symlinked:
|
||
it is already palace-derived, and its HOME path is a warning-only legacy probe (`hallways.py:73-95`) that
|
||
never auto-migrates.
|
||
|
||
Verified the symlink assumption rather than trusting it (`python3 sqlite3` on synlig, temp dir):
|
||
|
||
| Check | Result |
|
||
| --- | --- |
|
||
| Dangling symlink + `sqlite3.connect` | creates the target |
|
||
| `-wal` / `-shm` placement | next to the **target**, inside the palace dir — *not* beside the symlink |
|
||
| Write via symlink → read via palace path | same data, **same inode** |
|
||
|
||
The WAL placement is the part that mattered: it keeps the palace directory a single self-contained
|
||
backup/bind-mount unit.
|
||
|
||
### 2.4 §6.2's Host/Origin policy verified by experiment, not by reading
|
||
|
||
Ran on synlig, loopback and docker0 binds, then stopped. **11/11 as predicted:**
|
||
|
||
| # | Bind | Request | Expected | Got |
|
||
| --- | --- | --- | --- | --- |
|
||
| A1 | 127.0.0.1 | `/healthz`, correct Host | 200 | ✅ 200 |
|
||
| A2 | 127.0.0.1 | `/healthz`, `Host: palace.example.com` | **403** | ✅ 403 |
|
||
| A3 | 127.0.0.1 | `/healthz`, `Origin: https://evil.example` | 403 | ✅ 403 |
|
||
| A4 | 127.0.0.1 | `POST /mcp`, no token | 401 | ✅ 401 |
|
||
| A5 | 127.0.0.1 | `POST /mcp`, wrong token | 401 | ✅ 401 |
|
||
| A6 | 127.0.0.1 | `POST /mcp`, correct token | 200 | ✅ 200 (`tools/list` → **36 tools**) |
|
||
| B1 | 172.17.0.1 | `/healthz`, bound-host Host | 200 | ✅ 200 |
|
||
| B2 | 172.17.0.1 | `/healthz`, `Host: palace.example.com` | **200** | ✅ 200 |
|
||
| B3 | 172.17.0.1 | `/healthz`, `Origin: https://evil.example` | 403 | ✅ 403 |
|
||
| B4 | 172.17.0.1 | `/healthz`, loopback Origin | 200 | ✅ 200 |
|
||
| B5 | 172.17.0.1 | `POST /mcp`, foreign Host + token | 200 | ✅ 200 |
|
||
|
||
**Operational conclusions:**
|
||
|
||
1. **Do not bind loopback behind the tunnel.** A2 vs B2 is the whole story: the reflex "bind 127.0.0.1,
|
||
it's safer" produces a 403 that looks like a Pangolin misconfiguration and is not one.
|
||
2. **Bind `172.17.0.1` (docker0).** Non-loopback, so the Host pin relaxes — but reachable only from
|
||
synlig and its containers, so a newt container on this box can reach it while the LAN cannot. This is
|
||
strictly better than `0.0.0.0` here. It is what `contrib/systemd/mempalace-serve.service` uses.
|
||
3. **`Origin` is never relaxed** (B3). No browser-based MCP client, and no proxy that injects `Origin`.
|
||
4. `/healthz` is Host/Origin-gated but token-free — a usable liveness probe for the tunnel.
|
||
|
||
Test script kept at `/tmp/synlig-phase0-test.sh` on this container (ephemeral — re-create from the table
|
||
above if needed; it starts, probes and stops the server, and asserts nothing is left listening).
|
||
|
||
### 2.5 A start unit, written but not installed
|
||
|
||
`contrib/systemd/mempalace-serve.service` — user unit, follows the existing `contrib/systemd/` style,
|
||
carries the bind rationale inline so nobody "fixes" it back to loopback. A copy is already staged on synlig
|
||
at `~/.config/systemd/user/mempalace-serve.service.staged` — **the `.staged` suffix is deliberate**:
|
||
systemd only reads `*.service`, so the file cannot be activated by accident, not even by a stray
|
||
`daemon-reload`. **Not** installed, **not** enabled: it needs one `sudo loginctl enable-linger`, and
|
||
standing up a network-reachable service while you were asleep was not mine to decide.
|
||
|
||
## 3. Deliberately NOT done
|
||
|
||
- **Nothing is serving.** No listener on 8765; no mempalace process. Re-verified at the end of the run.
|
||
- **No client `.env` was touched.** Your working setup is exactly as you left it (R6: reversible).
|
||
- **No data joined.** The palace is empty. The §4.4 join needs the diary-dedup decision (§7.6) first —
|
||
replaying diaries today duplicates them, and the primary is the one place that must stay clean.
|
||
- **nyvaken untouched.** Read nothing, changed nothing.
|
||
- **No sudo.** `sudo -n` on synlig requires a password.
|
||
|
||
## 4. Tomorrow, in order
|
||
|
||
> **2026-08-12: items 1–2 and 5 now have their own runbook —**
|
||
> [`phase-1-exposure-runbook.md`](./phase-1-exposure-runbook.md). Pangolin on nyvaken is updated (done),
|
||
> and **newt is now installed on synlig and connected to Pangolin (done 2026-08-12)** — so the blocker is
|
||
> now item 3, the one `sudo`. That doc also records why per-device Pangolin users are the wrong layer, why
|
||
> the HTTPS tunnel and the feeder's SSH path are **not** redundant (§1.3), the client-flip variable trap
|
||
> (§3.7), and an additional loopback finding: a loopback bind does not merely 403, it also silently starts
|
||
> the server with **no token at all** (auto-minting is gated on the bind being non-loopback).
|
||
|
||
1. **Pangolin update on nyvaken** (yours). ✅ done 2026-08-12.
|
||
2. ~~**⚠️ synlig has no tunnel client.**~~ ✅ **done 2026-08-12** — newt installed and connected to Pangolin.
|
||
(Kept for the reasoning: `docker ps` showed only the Gitea runner and digikam. Pangolin on nyvaken
|
||
cannot reach synlig by itself; synlig had to dial out. Easy to miss because Pangolin looks healthy on
|
||
its own side — which is also why "connected" is not yet proof it can reach the palace: verify
|
||
`172.17.0.1:8765/healthz` from *inside* newt's namespace, exposure runbook §3.3.) Since newt runs in
|
||
Docker here, the docker0 bind above is already correct for it.
|
||
3. **One sudo, then start** (the unit is already staged; just drop the suffix). ✅ **done 2026-08-12** —
|
||
linger enabled, unit enabled, `172.17.0.1:8765/healthz` → `ok`.
|
||
```sh
|
||
sudo loginctl enable-linger ecsjper
|
||
cd ~/.config/systemd/user && mv mempalace-serve.service.staged mempalace-serve.service
|
||
systemctl --user daemon-reload && systemctl --user enable --now mempalace-serve
|
||
curl -s 172.17.0.1:8765/healthz # ok
|
||
curl -s 127.0.0.1:8765/healthz # NOTHING — refused, exit 7 (not 403; see below)
|
||
ss -ltnp | grep 8765 # 172.17.0.1:8765 only
|
||
```
|
||
⚠ **Corrected 2026-08-12:** this line predicted `403`. The real run returned empty, which is *more*
|
||
reassuring. With the docker0-only bind nothing listens on loopback, so the connection is refused before
|
||
any header is sent (`%{http_code}` → `000`, `$?` → `7`). The 403 in §2.4 is the **loopback-bind** case:
|
||
a server on `127.0.0.1` answering a proxy-forwarded foreign `Host:`. Two different failures that were
|
||
collapsed into one expectation here.
|
||
4. **Collect the shared token** (auto-minted on first non-loopback start, stable across restarts):
|
||
```sh
|
||
cat ~/.mempalace/server/f5d849287f6d73f0141b29d7/token
|
||
```
|
||
That directory name is `sha256(realpath(palace))[:24]` — it changes if the palace path ever changes.
|
||
5. **Route it through Pangolin**, then verify `/healthz` end-to-end through the public hostname *before*
|
||
pointing any client at it.
|
||
6. **Then, and only then**, Phase 1 client flip — one machine first, and remember opencode containers
|
||
need the §4.1 sidecar merge (or Phase 1.5) before the `.env` takes effect.
|
||
7. **Before the first join:** settle §7.6 diary dedup, then dry-run §4.4 from **one** palace and check
|
||
counts.
|
||
|
||
## 5. Rollback
|
||
|
||
```sh
|
||
systemctl --user disable --now mempalace-serve # if enabled
|
||
~/.local/bin/uv tool uninstall mempalace
|
||
rm -rf ~/.mempalace # empty today — check before running once it isn't
|
||
```
|