diff --git a/contrib/systemd/mempalace-serve.service b/contrib/systemd/mempalace-serve.service new file mode 100644 index 0000000..beefad1 --- /dev/null +++ b/contrib/systemd/mempalace-serve.service @@ -0,0 +1,56 @@ +[Unit] +Description=MemPalace remote MCP server (the fleet primary — RFC-001) +Documentation=https://gitea.jordbo.se/joakimp/mempalace-toolkit +Documentation=file:%h/mempalace-toolkit/docs/rfc-001-global-palace.md +After=network-online.target +Wants=network-online.target +# Refuse to start if the palace is missing — better a clear failure than a +# server quietly creating an empty palace somewhere unexpected. +ConditionPathExists=%h/.mempalace/palace + +[Service] +Type=simple +# ── Bind address: NOT 127.0.0.1. This is deliberate and load-bearing. ──────── +# mempalace pins the HTTP Host header to loopback literals *only on a loopback +# bind* (mcp_server.py: enforce_host_pin = _http_is_loopback(host)). Behind a +# reverse proxy / tunnel that forwards the public hostname, a loopback bind +# answers 403 Forbidden. Verified empirically on synlig 2026-08-10: +# bind 127.0.0.1 + Host: palace.example.com -> 403 +# bind 172.17.0.1 + Host: palace.example.com -> 200 +# 172.17.0.1 is the docker0 gateway: non-loopback (so the pin relaxes), but +# reachable only from this host and its containers — so a newt/Pangolin tunnel +# container on this box can reach it while the LAN cannot. Use 0.0.0.0 only if +# the tunnel does not run in Docker here, and only with the firewall closed. +# +# The Origin check is NEVER relaxed: a request carrying a non-loopback Origin +# is 403 with no override. Fine for MCP clients (they send none); fatal for +# browser-based clients. +# +# No --token here on purpose: for a non-loopback bind, serve reuses or mints a +# 0600 token at ~/.mempalace/server//token and keeps it +# stable across restarts. Read it from there to configure clients; never paste +# it into this unit (units are world-readable). +ExecStart=%h/.local/bin/mempalace serve --host 172.17.0.1 --port 8765 +Restart=on-failure +RestartSec=5s +# The server serializes every request behind one lock, so a wedged process is +# a fleet-wide outage. Fail fast and let Restart= recover. +TimeoutStopSec=30 +# Journal: journalctl --user -u mempalace-serve -f +StandardOutput=journal +StandardError=journal +# Modest hardening (user units can't do much, but these are free) +NoNewPrivileges=true +PrivateTmp=true +ProtectKernelTunables=true +ProtectControlGroups=true + +[Install] +WantedBy=default.target + +# ── Enabling (needs one sudo, hence not done by the Phase 0 prep) ─────────── +# sudo loginctl enable-linger $USER # else the unit dies with your login +# systemctl --user daemon-reload +# systemctl --user enable --now mempalace-serve +# curl -s localhost:8765/healthz # -> 403 (expected! see bind note) +# curl -s 172.17.0.1:8765/healthz # -> ok diff --git a/docs/rfc-001-global-palace.md b/docs/rfc-001-global-palace.md index ea6fff9..a593ebb 100644 --- a/docs/rfc-001-global-palace.md +++ b/docs/rfc-001-global-palace.md @@ -398,7 +398,7 @@ The transport is largely solved; **the gap is authorization, not cryptography.** | Control | Decision | | --- | --- | | **Network posture** | Primary **never** internet-exposed. Publish only onto the private overlay / existing tunnel (Pangolin/newt). ⚠️ **The reflexive "bind loopback in the container" is the failure mode here — see Transport.** | -| **Transport** | **Decided 2026-08-09: terminate TLS in the existing Pangolin/newt tunnel**, not in `serve` (cheaper than patching Python's TLS surface; one DNS record per service on the web hotel is the whole setup cost). ⚠️ **Host-pinning is coupled to the bind address**, verified: `enforce_host_pin = _http_is_loopback(host)` (`mcp_server.py:5367`). On a **loopback** bind, `Host` is pinned to loopback literals + the bound host, so a proxy forwarding `Host: palace.example.com` gets **403 Forbidden** — either make the proxy rewrite `Host` to `127.0.0.1:`, or bind the private interface instead. On a **non-loopback** bind the pin is deliberately **relaxed** ("*may sit behind a proxy that rewrites Host … lean on the Origin check + optional token instead*", `:5362-5365`), which is also why `cli.py:1450` makes a tokenless non-loopback bind require `--allow-insecure`. **The `Origin` check is never relaxed:** an absent `Origin` is allowed (every non-browser MCP client, incl. pi and opencode), but a *present* non-loopback `Origin` is 403 with no override — so keep browser-based clients and `Origin`-injecting proxies out of the path. | +| **Transport** | **Decided 2026-08-09: terminate TLS in the existing Pangolin/newt tunnel**, not in `serve` (cheaper than patching Python's TLS surface; one DNS record per service on the web hotel is the whole setup cost). ✅ **Host/Origin policy verified by experiment on synlig 2026-08-10, 11/11 as predicted** (runbook §2.4) — the summary: **do not bind loopback behind the tunnel.** `enforce_host_pin = _http_is_loopback(host)` (`mcp_server.py:5367`), so a loopback bind + a proxy forwarding `Host: palace.example.com` → **403**, while a non-loopback bind → **200** (pin deliberately relaxed, "*may sit behind a proxy that rewrites Host*", `:5362-5365`); tokenless non-loopback binds require `--allow-insecure` (`cli.py:1450`). **Prefer binding the docker0 gateway (e.g. `172.17.0.1`) over `0.0.0.0`:** non-loopback, so the pin relaxes, yet reachable only from the host and its containers — so a newt container on the box reaches it and the LAN cannot. **The `Origin` check is never relaxed:** absent `Origin` is fine (every non-browser MCP client, incl. pi and opencode), a *present* non-loopback `Origin` is 403 with no override — so keep browser-based clients and `Origin`-injecting proxies out of the path. `/healthz` is Host/Origin-gated but token-free, so it works as the tunnel's liveness probe. | | **Authentication** | Target: **per-device bearer tokens** with a server-side `token → {device_id, scopes}` registry — revoke one laptop, rotate without a fleet outage. **Decided 2026-08-09: Phase 1 ships the single shared token** ("*iterate more feature rich but more complex solutions over time*"), so per-device lands with Phase 4. Store in the existing `.env.age` flow, 0600 on disk. **Consequence: until then the primary cannot tell devices apart, so `origin_device` stays client-asserted and advisory — nothing load-bearing may depend on it (§7.3.2).** | | **Authorization** | Per-device read/write **wing globs**. Server-side refusal of `mine`/`sync`/`delete_*` except for an admin device. `--read-only` gives a free observer tier. | | **Provenance/audit** | `origin_device` (+ optional `origin_label`) + `op_id` on every record, stamped **server-side from the authenticated credential** — a client-asserted origin is a hint, not a fact (§7.3.2); server-side append-only op log. | @@ -440,6 +440,30 @@ Four stores, **three** location rules: nothing else** — it would leave the KG as fragmented as it is today. That is why the shared-backend option is *not* the answer. +⚠️ **Understated above, corrected 2026-08-10 while provisioning synlig: this is not a migration hazard, +it is the stock-default behaviour, and it is permanent.** `DEFAULT_PALACE_PATH` is +`~/.mempalace/palace` (`config.py:221`) while `DEFAULT_KG_PATH` is +`~/.mempalace/knowledge_graph.sqlite3` (`knowledge_graph.py:49`) — **those differ out of the box**, with no +custom path involved. So on *every* host, forever: `serve` (always passes `--palace`) uses the KG inside +the palace, while any CLI command run without `--palace` uses the HOME one. A one-time `mv` does not fix +that; it just relocates which of the two files is populated. + +**Better action — converge the two resolution rules onto one inode** (and the only option on a greenfield +primary, where there is nothing to move): + +```sh +ln -sfn palace/knowledge_graph.sqlite3 ~/.mempalace/knowledge_graph.sqlite3 +ln -sfn palace/known_entities.json ~/.mempalace/known_entities.json +# hallways.json deliberately NOT symlinked: already palace-derived, and its HOME +# path is a warning-only legacy probe that never auto-migrates (hallways.py:73-95) +``` + +Verified on synlig 2026-08-10, because the WAL behaviour was the load-bearing assumption: a **dangling** +symlink is created on first `sqlite3.connect`; `-wal`/`-shm` land next to the **target** (inside the palace +dir, so the palace stays a self-contained backup/bind-mount unit) and *not* beside the symlink; a write +through one path reads back through the other; **same inode**. Full record in +[`synlig-primary-runbook.md`](./synlig-primary-runbook.md) §2.3. + ### 7.2 Never run `mempalace sync` against a shared palace It classifies drawers whose source files are absent **on the running host** as orphans and deletes @@ -658,7 +682,7 @@ join replays (§4.4). | Phase | Effort | Deliverable | | --- | --- | --- | -| **0 — hygiene** | hours | §7 runbook: move KG/hallways/entities **on synlig, before first `serve`** (§7.1), ban `sync` on shared palaces (§7.2), fix stale "unauthenticated" docs (incl. `pi-devbox/.env.example:21`). Added 2026-08-09: settle the **diary dedup** approach and file its upstream ask (§7.6), and **dry-run the join from one palace only** (§4.4). **No provenance work here** — it is not backfill-critical (§7.3.3) and belongs to the stamper, not the agent | +| **0 — hygiene** | hours | §7 runbook: converge the KG/entities store paths **on synlig before first `serve`** (§7.1 — **done 2026-08-10**, runbook §2.3), ban `sync` on shared palaces (§7.2), fix stale "unauthenticated" docs (incl. `pi-devbox/.env.example:21`). Added 2026-08-09: settle the **diary dedup** approach and file its upstream ask (§7.6), and **dry-run the join from one palace only** (§4.4). **No provenance work here** — it is not backfill-critical (§7.3.3) and belongs to the stamper, not the agent | | **1 — primary up** | hours, **no code** | `mempalace serve --token --tls-cert` on a private-net host (reuse `docker-compose.mempalace.yml` — keep it a **separate standalone project**, R4 — and mind port 8765 vs pi-studio; tor-ms22 already moved to 8766). Repoint pi clients via `MEMPALACE_REMOTE_URL`/`MEMPALACE_REMOTE_TOKEN`. **opencode clients can be repointed in the same breath** — remote MCP is supported and `generate-config.py` already emits it (§9.1, §9.6), subject to the sidecar caveat in §4.1. **Shared memory today, no offline.** **Decided 2026-08-09: primary = synlig, TLS at Pangolin, single shared token (§8.1)** — mind the loopback Host-pin trap in §6.2. | | **1.5 — opencode env propagation** | hours | Make the `mcp.mempalace` subtree env-authoritative in `generate-config.py`, gated by a generated-value fingerprint (§4.1). Independent of the rest of this RFC. Without it, adopting *or reverting* the opt-in on an existing opencode container needs a manual sidecar merge or a `docker volume rm` — which also blocks R6 reversibility | | **2 — `mempalace-edge`** | ~1 week | The actual ask: local-first writes + outbox flush + merged reads + per-wing policy. **Not** "fixes opencode" — opencode's *transport* is already fine after Phase 1; what edge adds there is offline/local-first, since `generate-config.py`'s switch is remote **or** local with no failover. **Ships with the §1.2 opt-in wiring (compose + `.env.example` + a third branch in the existing `generate-config.py`) and must pass the R1 acceptance test.** | @@ -762,7 +786,9 @@ Re-verify without re-reading the package. Paths relative to | Chunk probe is deliberate (all-or-nothing upsert) | `mcp_server.py:2586-2592` | | Metadata is exhaustive — no session/PID/conversation field | `mcp_server.py:2578-2585` (drawers), `3527-3536` (diaries) | | pi extension sets identity for **diaries only** | `extensions/pi/mempalace.ts:758` (`agent_name`); no `added_by` set anywhere in the file | -| Host-pin coupled to bind address; Origin never relaxed | `mcp_server.py:5160-5194`, `5276-5290`, `5362-5367`; `cli.py:1450` | +| Anti-rebinding Host/Origin checks | `mcp_server.py:5160-5194`, `5276-5290`, `5362-5367`; `cli.py:1450`. **Verified empirically** — runbook §2.4 | +| Stock defaults already split the KG | `config.py:221` (`~/.mempalace/palace`) vs `knowledge_graph.py:49` (`~/.mempalace/knowledge_graph.sqlite3`) | +| Server token path (stable across restarts) | `cli.py:_server_token_path` — `~/.mempalace/server/sha256(realpath(palace))[:24]/token` | | pi-devbox non-destructive settings merge (pattern to port) | `pi-devbox/entrypoint-user.sh:131-162` (`jq -s '.[0] * .[1]'`, `.bak`, `PI_SETTINGS_MERGE=0`) | | No `OPENCODE_CONFIG*` env layer upstream | absent from `https://opencode.ai/config.json`; zero hits in `opencode-devbox`, `myconfigs` | | Tool classification (+ `kg_supersede` gap) | `service.py:29`, `54`, `72-84` | diff --git a/docs/synlig-primary-runbook.md b/docs/synlig-primary-runbook.md new file mode 100644 index 0000000..89c743c --- /dev/null +++ b/docs/synlig-primary-runbook.md @@ -0,0 +1,162 @@ +# synlig primary — Phase 0 runbook and handoff + +Companion to [`rfc-001-global-palace.md`](./rfc-001-global-palace.md). Records what was actually done +on the primary, with verified evidence, so the next session (or the next machine) does not re-derive it. + +**Status 2026-08-10 00:30 — Phase 0 prep complete. Not serving. Nothing exposed.** +Blocked on two things, both deliberately left to Joakim: the Pangolin update on nyvaken, and one `sudo`. + +--- + +## 1. What synlig is (discovered, not assumed) + +| Fact | Value | +| --- | --- | +| SSH | `synlig` → `synlig.erdc.ericsson.net`, user `ecsjper` (from `~/.ssh/config`) | +| OS | Ubuntu 24.04.4 LTS, 7.8 GiB RAM, 78 G disk (**29 G free**), uptime 12 d | +| Python / uv | system `python3` 3.12.3; `uv` at `~/.local/bin/uv` (**not** on the non-login `PATH`) | +| Interfaces | `lo` 127.0.0.1, `ens3` 10.0.0.4/16, `docker0` 172.17.0.1/16, `br-…` 172.19.0.1/16 | +| Already listening | 22, 80, 443, 3000 (node), 3389 + 3350 + 4822 (xrdp/guacamole), 631 | +| Docker | present; running `act_runner-runner-1` (**Gitea Actions runner**) and `digikam` | +| Pre-existing MemPalace | **none** — no `mempalace` binary, no `~/.mempalace`. Greenfield. | + +The Gitea Actions runner living here is worth remembering: synlig is not a dedicated appliance, and CI +load competes with the palace for the same 7.8 GiB. + +## 2. Done tonight + +### 2.1 MemPalace installed, pinned to the fleet version + +```sh +~/.local/bin/uv tool install "mempalace==3.6.0" # → mempalace, mempalace-mcp +``` + +Pinned deliberately: the clients run 3.6.0, and the id recipes / idempotency probes this RFC leans on are +version-specific. Reversible with `uv tool uninstall mempalace`. + +### 2.2 Embedder model pre-warmed — the corporate-network risk that wasn't + +The first embed pulls `all-MiniLM-L6-v2` ONNX (79.3 MB) from the chroma CDN into +`~/.cache/chroma/onnx_models/` (167 M on disk once unpacked). **This was the main unknown** — an +egress-filtered work VM would have failed here, at the worst possible moment (first client write). +It downloaded at ~20 MB/s with no proxy interference. Done in a throwaway palace, since deleted, so the +real palace never saw it. Same model as the clients use, so the semantic space matches. + +### 2.3 Palace created with the §7.1 landmine structurally removed + +`~/.mempalace/palace` — the **stock default**, so no `MEMPALACE_PALACE_PATH` and no `config.json` is +needed anywhere on synlig. One less thing to drift. + +RFC §7.1 says to `mv` three HOME-anchored stores into the palace dir before first `serve`. **On a +greenfield primary there is nothing to move — but the hazard is not actually a migration hazard, and the +RFC understated it:** with *stock defaults* `palace_path` is `~/.mempalace/palace` while `DEFAULT_KG_PATH` +is `~/.mempalace/knowledge_graph.sqlite3`. Those differ, so the split is the **out-of-the-box** behaviour, +not a consequence of a custom path. It is permanent, not one-time: `serve` always passes `--palace` (KG +inside the palace), while any CLI command run *without* `--palace` uses the HOME path. Two KGs on one box, +forever, silently. + +Fixed by making both resolution rules land on one inode: + +```sh +ln -sfn palace/knowledge_graph.sqlite3 ~/.mempalace/knowledge_graph.sqlite3 +ln -sfn palace/known_entities.json ~/.mempalace/known_entities.json +``` + +Relative targets, so a home-directory move survives. `hallways.json` is deliberately **not** symlinked: +it is already palace-derived, and its HOME path is a warning-only legacy probe (`hallways.py:73-95`) that +never auto-migrates. + +Verified the symlink assumption rather than trusting it (`python3 sqlite3` on synlig, temp dir): + +| Check | Result | +| --- | --- | +| Dangling symlink + `sqlite3.connect` | creates the target | +| `-wal` / `-shm` placement | next to the **target**, inside the palace dir — *not* beside the symlink | +| Write via symlink → read via palace path | same data, **same inode** | + +The WAL placement is the part that mattered: it keeps the palace directory a single self-contained +backup/bind-mount unit. + +### 2.4 §6.2's Host/Origin policy verified by experiment, not by reading + +Ran on synlig, loopback and docker0 binds, then stopped. **11/11 as predicted:** + +| # | Bind | Request | Expected | Got | +| --- | --- | --- | --- | --- | +| A1 | 127.0.0.1 | `/healthz`, correct Host | 200 | ✅ 200 | +| A2 | 127.0.0.1 | `/healthz`, `Host: palace.example.com` | **403** | ✅ 403 | +| A3 | 127.0.0.1 | `/healthz`, `Origin: https://evil.example` | 403 | ✅ 403 | +| A4 | 127.0.0.1 | `POST /mcp`, no token | 401 | ✅ 401 | +| A5 | 127.0.0.1 | `POST /mcp`, wrong token | 401 | ✅ 401 | +| A6 | 127.0.0.1 | `POST /mcp`, correct token | 200 | ✅ 200 (`tools/list` → **36 tools**) | +| B1 | 172.17.0.1 | `/healthz`, bound-host Host | 200 | ✅ 200 | +| B2 | 172.17.0.1 | `/healthz`, `Host: palace.example.com` | **200** | ✅ 200 | +| B3 | 172.17.0.1 | `/healthz`, `Origin: https://evil.example` | 403 | ✅ 403 | +| B4 | 172.17.0.1 | `/healthz`, loopback Origin | 200 | ✅ 200 | +| B5 | 172.17.0.1 | `POST /mcp`, foreign Host + token | 200 | ✅ 200 | + +**Operational conclusions:** + +1. **Do not bind loopback behind the tunnel.** A2 vs B2 is the whole story: the reflex "bind 127.0.0.1, + it's safer" produces a 403 that looks like a Pangolin misconfiguration and is not one. +2. **Bind `172.17.0.1` (docker0).** Non-loopback, so the Host pin relaxes — but reachable only from + synlig and its containers, so a newt container on this box can reach it while the LAN cannot. This is + strictly better than `0.0.0.0` here. It is what `contrib/systemd/mempalace-serve.service` uses. +3. **`Origin` is never relaxed** (B3). No browser-based MCP client, and no proxy that injects `Origin`. +4. `/healthz` is Host/Origin-gated but token-free — a usable liveness probe for the tunnel. + +Test script kept at `/tmp/synlig-phase0-test.sh` on this container (ephemeral — re-create from the table +above if needed; it starts, probes and stops the server, and asserts nothing is left listening). + +### 2.5 A start unit, written but not installed + +`contrib/systemd/mempalace-serve.service` — user unit, follows the existing `contrib/systemd/` style, +carries the bind rationale inline so nobody "fixes" it back to loopback. A copy is already staged on synlig +at `~/.config/systemd/user/mempalace-serve.service.staged` — **the `.staged` suffix is deliberate**: +systemd only reads `*.service`, so the file cannot be activated by accident, not even by a stray +`daemon-reload`. **Not** installed, **not** enabled: it needs one `sudo loginctl enable-linger`, and +standing up a network-reachable service while you were asleep was not mine to decide. + +## 3. Deliberately NOT done + +- **Nothing is serving.** No listener on 8765; no mempalace process. Re-verified at the end of the run. +- **No client `.env` was touched.** Your working setup is exactly as you left it (R6: reversible). +- **No data joined.** The palace is empty. The §4.4 join needs the diary-dedup decision (§7.6) first — + replaying diaries today duplicates them, and the primary is the one place that must stay clean. +- **nyvaken untouched.** Read nothing, changed nothing. +- **No sudo.** `sudo -n` on synlig requires a password. + +## 4. Tomorrow, in order + +1. **Pangolin update on nyvaken** (yours). +2. **⚠️ synlig has no tunnel client.** `docker ps` shows only the Gitea runner and digikam — no `newt`. + Pangolin on nyvaken cannot reach synlig by itself; synlig needs a newt container (or an equivalent + outbound tunnel) added. Easy to miss because Pangolin will look healthy on its own side. If newt runs + in Docker here, the docker0 bind above is already correct for it. +3. **One sudo, then start** (the unit is already staged; just drop the suffix): + ```sh + sudo loginctl enable-linger ecsjper + cd ~/.config/systemd/user && mv mempalace-serve.service.staged mempalace-serve.service + systemctl --user daemon-reload && systemctl --user enable --now mempalace-serve + curl -s 172.17.0.1:8765/healthz # ok + curl -s 127.0.0.1:8765/healthz # 403 — expected, not a bug (§2.4) + ``` +4. **Collect the shared token** (auto-minted on first non-loopback start, stable across restarts): + ```sh + cat ~/.mempalace/server/f5d849287f6d73f0141b29d7/token + ``` + That directory name is `sha256(realpath(palace))[:24]` — it changes if the palace path ever changes. +5. **Route it through Pangolin**, then verify `/healthz` end-to-end through the public hostname *before* + pointing any client at it. +6. **Then, and only then**, Phase 1 client flip — one machine first, and remember opencode containers + need the §4.1 sidecar merge (or Phase 1.5) before the `.env` takes effect. +7. **Before the first join:** settle §7.6 diary dedup, then dry-run §4.4 from **one** palace and check + counts. + +## 5. Rollback + +```sh +systemctl --user disable --now mempalace-serve # if enabled +~/.local/bin/uv tool uninstall mempalace +rm -rf ~/.mempalace # empty today — check before running once it isn't +```