Joakim Persson
00a95d1a2f
Asked "why do we need Pangolin if you proposed rsync/ssh?", and the runbook did
not actually answer it -- it stated both were needed without saying why neither
substitutes. New section 1.3:
- Pangolin/HTTPS carries the MCP tool surface (search, add_drawer,
diary_write, kg_*) -- every live tool call, from any MCP client.
- SSH/rsync carries transcript *files* only, because mempalace_mine expands
its source path server-side, so the server can only mine its own disk.
HTTPS alone is a palace you can query but cannot feed; SSH alone is files with
no query API. The rsync is not a transport preference, it is a workaround for
where `mine` resolves paths.
Records honestly that `ssh -L 8765:172.17.0.1:8765 synlig` *would* replace the
tunnel for MCP, and why we don't: synlig dials out (reaching for a dial-out
tunnel is itself the evidence inbound was unavailable), MCP clients want a
durable URL rather than a per-session forward, and the forward must be up on
every device before every session.
And the design's weak point, stated instead of glossed: the rsync runs
client -> synlig, so mining needs synlig's SSH reachable *from the client*. Were
that true everywhere, no tunnel would be needed for MCP either. Honest
expectation after Phase 1 is therefore: query/write from anywhere, mine only
from devices that can reach synlig's SSH. Section 4 now carries the upstream ask
that would close the gap -- have the feeder send content over MCP (add_drawer /
diary_write, which it already calls) instead of asking the server to mine a path
it must first rsync there.
New section 3.7, a live trap for the imminent client flip:
MEMPALACE_REMOTE_URL on its own does not degrade to local feeding, it *stops*
feeding. auto mode switches to remote as soon as the URL is set (:286) and
remote mode then exits 1 without MEMPALACE_PI_SSH_TARGET (:298-300), before
anything is staged or filed -- so a cron feeder just starts failing, and the
loudest symptom is silence. Two safe orders given: set all three variables in
one edit, or set URL+token and pin --mode local until the SSH target exists.
newt is installed on synlig and connected to Pangolin (done 2026-08-12), marked
here and in the synlig runbook's item 2; the blocker is now item 3, the one
sudo. Added the follow-up that "connected to Pangolin" only proves newt reached
nyvaken -- reaching the *palace* is a separate claim that fails independently,
so probe 172.17.0.1:8765/healthz from inside newt's namespace.
All seven code citations verified against the source at commit time.
173 lines
9.8 KiB
Markdown
173 lines
9.8 KiB
Markdown
# synlig primary — Phase 0 runbook and handoff
|
||
|
||
Companion to [`rfc-001-global-palace.md`](./rfc-001-global-palace.md). Records what was actually done
|
||
on the primary, with verified evidence, so the next session (or the next machine) does not re-derive it.
|
||
|
||
**Status 2026-08-10 00:30 — Phase 0 prep complete. Not serving. Nothing exposed.**
|
||
Blocked on two things, both deliberately left to Joakim: the Pangolin update on nyvaken, and one `sudo`.
|
||
|
||
---
|
||
|
||
## 1. What synlig is (discovered, not assumed)
|
||
|
||
| Fact | Value |
|
||
| --- | --- |
|
||
| SSH | `synlig` → `synlig.erdc.ericsson.net`, user `ecsjper` (from `~/.ssh/config`) |
|
||
| OS | Ubuntu 24.04.4 LTS, 7.8 GiB RAM, 78 G disk (**29 G free**), uptime 12 d |
|
||
| Python / uv | system `python3` 3.12.3; `uv` at `~/.local/bin/uv` (**not** on the non-login `PATH`) |
|
||
| Interfaces | `lo` 127.0.0.1, `ens3` 10.0.0.4/16, `docker0` 172.17.0.1/16, `br-…` 172.19.0.1/16 |
|
||
| Already listening | 22, 80, 443, 3000 (node), 3389 + 3350 + 4822 (xrdp/guacamole), 631 |
|
||
| Docker | present; running `act_runner-runner-1` (**Gitea Actions runner**) and `digikam` |
|
||
| Pre-existing MemPalace | **none** — no `mempalace` binary, no `~/.mempalace`. Greenfield. |
|
||
|
||
The Gitea Actions runner living here is worth remembering: synlig is not a dedicated appliance, and CI
|
||
load competes with the palace for the same 7.8 GiB.
|
||
|
||
## 2. Done tonight
|
||
|
||
### 2.1 MemPalace installed, pinned to the fleet version
|
||
|
||
```sh
|
||
~/.local/bin/uv tool install "mempalace==3.6.0" # → mempalace, mempalace-mcp
|
||
```
|
||
|
||
Pinned deliberately: the clients run 3.6.0, and the id recipes / idempotency probes this RFC leans on are
|
||
version-specific. Reversible with `uv tool uninstall mempalace`.
|
||
|
||
### 2.2 Embedder model pre-warmed — the corporate-network risk that wasn't
|
||
|
||
The first embed pulls `all-MiniLM-L6-v2` ONNX (79.3 MB) from the chroma CDN into
|
||
`~/.cache/chroma/onnx_models/` (167 M on disk once unpacked). **This was the main unknown** — an
|
||
egress-filtered work VM would have failed here, at the worst possible moment (first client write).
|
||
It downloaded at ~20 MB/s with no proxy interference. Done in a throwaway palace, since deleted, so the
|
||
real palace never saw it. Same model as the clients use, so the semantic space matches.
|
||
|
||
### 2.3 Palace created with the §7.1 landmine structurally removed
|
||
|
||
`~/.mempalace/palace` — the **stock default**, so no `MEMPALACE_PALACE_PATH` and no `config.json` is
|
||
needed anywhere on synlig. One less thing to drift.
|
||
|
||
RFC §7.1 says to `mv` three HOME-anchored stores into the palace dir before first `serve`. **On a
|
||
greenfield primary there is nothing to move — but the hazard is not actually a migration hazard, and the
|
||
RFC understated it:** with *stock defaults* `palace_path` is `~/.mempalace/palace` while `DEFAULT_KG_PATH`
|
||
is `~/.mempalace/knowledge_graph.sqlite3`. Those differ, so the split is the **out-of-the-box** behaviour,
|
||
not a consequence of a custom path. It is permanent, not one-time: `serve` always passes `--palace` (KG
|
||
inside the palace), while any CLI command run *without* `--palace` uses the HOME path. Two KGs on one box,
|
||
forever, silently.
|
||
|
||
Fixed by making both resolution rules land on one inode:
|
||
|
||
```sh
|
||
ln -sfn palace/knowledge_graph.sqlite3 ~/.mempalace/knowledge_graph.sqlite3
|
||
ln -sfn palace/known_entities.json ~/.mempalace/known_entities.json
|
||
```
|
||
|
||
Relative targets, so a home-directory move survives. `hallways.json` is deliberately **not** symlinked:
|
||
it is already palace-derived, and its HOME path is a warning-only legacy probe (`hallways.py:73-95`) that
|
||
never auto-migrates.
|
||
|
||
Verified the symlink assumption rather than trusting it (`python3 sqlite3` on synlig, temp dir):
|
||
|
||
| Check | Result |
|
||
| --- | --- |
|
||
| Dangling symlink + `sqlite3.connect` | creates the target |
|
||
| `-wal` / `-shm` placement | next to the **target**, inside the palace dir — *not* beside the symlink |
|
||
| Write via symlink → read via palace path | same data, **same inode** |
|
||
|
||
The WAL placement is the part that mattered: it keeps the palace directory a single self-contained
|
||
backup/bind-mount unit.
|
||
|
||
### 2.4 §6.2's Host/Origin policy verified by experiment, not by reading
|
||
|
||
Ran on synlig, loopback and docker0 binds, then stopped. **11/11 as predicted:**
|
||
|
||
| # | Bind | Request | Expected | Got |
|
||
| --- | --- | --- | --- | --- |
|
||
| A1 | 127.0.0.1 | `/healthz`, correct Host | 200 | ✅ 200 |
|
||
| A2 | 127.0.0.1 | `/healthz`, `Host: palace.example.com` | **403** | ✅ 403 |
|
||
| A3 | 127.0.0.1 | `/healthz`, `Origin: https://evil.example` | 403 | ✅ 403 |
|
||
| A4 | 127.0.0.1 | `POST /mcp`, no token | 401 | ✅ 401 |
|
||
| A5 | 127.0.0.1 | `POST /mcp`, wrong token | 401 | ✅ 401 |
|
||
| A6 | 127.0.0.1 | `POST /mcp`, correct token | 200 | ✅ 200 (`tools/list` → **36 tools**) |
|
||
| B1 | 172.17.0.1 | `/healthz`, bound-host Host | 200 | ✅ 200 |
|
||
| B2 | 172.17.0.1 | `/healthz`, `Host: palace.example.com` | **200** | ✅ 200 |
|
||
| B3 | 172.17.0.1 | `/healthz`, `Origin: https://evil.example` | 403 | ✅ 403 |
|
||
| B4 | 172.17.0.1 | `/healthz`, loopback Origin | 200 | ✅ 200 |
|
||
| B5 | 172.17.0.1 | `POST /mcp`, foreign Host + token | 200 | ✅ 200 |
|
||
|
||
**Operational conclusions:**
|
||
|
||
1. **Do not bind loopback behind the tunnel.** A2 vs B2 is the whole story: the reflex "bind 127.0.0.1,
|
||
it's safer" produces a 403 that looks like a Pangolin misconfiguration and is not one.
|
||
2. **Bind `172.17.0.1` (docker0).** Non-loopback, so the Host pin relaxes — but reachable only from
|
||
synlig and its containers, so a newt container on this box can reach it while the LAN cannot. This is
|
||
strictly better than `0.0.0.0` here. It is what `contrib/systemd/mempalace-serve.service` uses.
|
||
3. **`Origin` is never relaxed** (B3). No browser-based MCP client, and no proxy that injects `Origin`.
|
||
4. `/healthz` is Host/Origin-gated but token-free — a usable liveness probe for the tunnel.
|
||
|
||
Test script kept at `/tmp/synlig-phase0-test.sh` on this container (ephemeral — re-create from the table
|
||
above if needed; it starts, probes and stops the server, and asserts nothing is left listening).
|
||
|
||
### 2.5 A start unit, written but not installed
|
||
|
||
`contrib/systemd/mempalace-serve.service` — user unit, follows the existing `contrib/systemd/` style,
|
||
carries the bind rationale inline so nobody "fixes" it back to loopback. A copy is already staged on synlig
|
||
at `~/.config/systemd/user/mempalace-serve.service.staged` — **the `.staged` suffix is deliberate**:
|
||
systemd only reads `*.service`, so the file cannot be activated by accident, not even by a stray
|
||
`daemon-reload`. **Not** installed, **not** enabled: it needs one `sudo loginctl enable-linger`, and
|
||
standing up a network-reachable service while you were asleep was not mine to decide.
|
||
|
||
## 3. Deliberately NOT done
|
||
|
||
- **Nothing is serving.** No listener on 8765; no mempalace process. Re-verified at the end of the run.
|
||
- **No client `.env` was touched.** Your working setup is exactly as you left it (R6: reversible).
|
||
- **No data joined.** The palace is empty. The §4.4 join needs the diary-dedup decision (§7.6) first —
|
||
replaying diaries today duplicates them, and the primary is the one place that must stay clean.
|
||
- **nyvaken untouched.** Read nothing, changed nothing.
|
||
- **No sudo.** `sudo -n` on synlig requires a password.
|
||
|
||
## 4. Tomorrow, in order
|
||
|
||
> **2026-08-12: items 1–2 and 5 now have their own runbook —**
|
||
> [`phase-1-exposure-runbook.md`](./phase-1-exposure-runbook.md). Pangolin on nyvaken is updated (done),
|
||
> and **newt is now installed on synlig and connected to Pangolin (done 2026-08-12)** — so the blocker is
|
||
> now item 3, the one `sudo`. That doc also records why per-device Pangolin users are the wrong layer, why
|
||
> the HTTPS tunnel and the feeder's SSH path are **not** redundant (§1.3), the client-flip variable trap
|
||
> (§3.7), and an additional loopback finding: a loopback bind does not merely 403, it also silently starts
|
||
> the server with **no token at all** (auto-minting is gated on the bind being non-loopback).
|
||
|
||
1. **Pangolin update on nyvaken** (yours). ✅ done 2026-08-12.
|
||
2. ~~**⚠️ synlig has no tunnel client.**~~ ✅ **done 2026-08-12** — newt installed and connected to Pangolin.
|
||
(Kept for the reasoning: `docker ps` showed only the Gitea runner and digikam. Pangolin on nyvaken
|
||
cannot reach synlig by itself; synlig had to dial out. Easy to miss because Pangolin looks healthy on
|
||
its own side — which is also why "connected" is not yet proof it can reach the palace: verify
|
||
`172.17.0.1:8765/healthz` from *inside* newt's namespace, exposure runbook §3.3.) Since newt runs in
|
||
Docker here, the docker0 bind above is already correct for it.
|
||
3. **One sudo, then start** (the unit is already staged; just drop the suffix):
|
||
```sh
|
||
sudo loginctl enable-linger ecsjper
|
||
cd ~/.config/systemd/user && mv mempalace-serve.service.staged mempalace-serve.service
|
||
systemctl --user daemon-reload && systemctl --user enable --now mempalace-serve
|
||
curl -s 172.17.0.1:8765/healthz # ok
|
||
curl -s 127.0.0.1:8765/healthz # 403 — expected, not a bug (§2.4)
|
||
```
|
||
4. **Collect the shared token** (auto-minted on first non-loopback start, stable across restarts):
|
||
```sh
|
||
cat ~/.mempalace/server/f5d849287f6d73f0141b29d7/token
|
||
```
|
||
That directory name is `sha256(realpath(palace))[:24]` — it changes if the palace path ever changes.
|
||
5. **Route it through Pangolin**, then verify `/healthz` end-to-end through the public hostname *before*
|
||
pointing any client at it.
|
||
6. **Then, and only then**, Phase 1 client flip — one machine first, and remember opencode containers
|
||
need the §4.1 sidecar merge (or Phase 1.5) before the `.env` takes effect.
|
||
7. **Before the first join:** settle §7.6 diary dedup, then dry-run §4.4 from **one** palace and check
|
||
counts.
|
||
|
||
## 5. Rollback
|
||
|
||
```sh
|
||
systemctl --user disable --now mempalace-serve # if enabled
|
||
~/.local/bin/uv tool uninstall mempalace
|
||
rm -rf ~/.mempalace # empty today — check before running once it isn't
|
||
```
|