mempalace-toolkit/docs/phase-1-exposure-runbook.md

Phase 1 exposure — newt on synlig, DNS, and the client auth model

Companion to rfc-001-global-palace.md (design + decisions) and synlig-primary-runbook.md (what is already installed on the primary). This doc covers only the step the other two leave open: making the primary reachable — runbook §4 items 2 and 5.

Status 2026-08-12 — Pangolin updated on nyvaken (done, yours). newt not yet installed on synlig. Nothing exposed. No client .env flipped.

Read this before touching Pangolin: three of the four questions this step raises were already decided in RFC §6.2 on 2026-08-09, and re-deciding them differently is how the fleet ends up in two states.

---

1. The four questions, answered

Question	Answer	Where it was decided
Which port?	8765, path /mcp (liveness: /healthz)	cli.py:2141 default; runbook §2.4
What does newt target?	172.17.0.1:8765 (docker0), never 127.0.0.1	RFC §6.2 Transport; runbook §2.4
Open, or authenticated?	Authenticated. The primary is never an open public resource.	RFC §6.2 Network posture
Per-device credentials?	No — Phase 1 ships the single shared bearer token. Per-device tokens are Phase 4.	RFC §6.2 Authentication

1.1 Why not per-device users at the proxy

The instinct — "create a Pangolin user per container, put the credentials in each .env, keep the usernames distinct" — is the right goal (revocation, attribution) reached through the wrong layer, twice over:

1. mempalace validates exactly one token. hmac.compare_digest(provided, f"Bearer {srv.auth_token}") (mcp_server.py:5292-5295) — there is no user table and no second credential. Per-device HTTP identity is not a configuration you can express today; it is Phase 4 work (a server-side token → {device_id, scopes} registry). RFC §6.2 chose the shared token for Phase 1 deliberately: "iterate more feature rich but more complex solutions over time."

2. Pangolin's HTTP auth is browser-shaped; the clients are not. SSO login, resource PIN and resource password all assume something that can follow a redirect, render a form and hold a session cookie. Every MemPalace client here is a headless JSON-RPC POST with an Authorization header — pi's extension, opencode's type:remote MCP entry, and mempalace-pi-session --mode remote's urllib.request.urlopen. Point those at a user-authenticated resource and they receive a login page where JSON should be. Enabling that protection breaks precisely the clients it is meant to protect.

So: Pangolin terminates TLS and nothing more (RFC §6.2 Transport, decided 2026-08-09). The bearer token is the authentication. This is not "unprotected" — an unauthenticated request to /mcp gets a 401 from mempalace itself, verified A4/A5 in runbook §2.4.

Consequence to accept consciously (RFC §6.2, §7.3.2): until Phase 4 the primary cannot tell devices apart. origin_device is client-asserted and advisory — nothing load-bearing may depend on it, and revoking one laptop means rotating the token everywhere.

1.2 The one place per-device identity does exist today

Remote mode is not only HTTP. mempalace_mine expands its source path in the server process, so a client's staged transcripts must physically exist on the primary. The feeder therefore ships them over SSH into a per-device inbox before asking the server to mine its own copy:

rsync -a --update -e "$ssh_cmd" "$STAGE/" "${SSH_TARGET%/}/$DEVICE/"   # bin/mempalace-pi-session:677-680

That SSH key is per-device identity, and it is individually revocable (one line out of authorized_keys) years before Phase 4 lands. It costs nothing extra, because the mining path needs SSH regardless.

Two implications people miss:

• mempalace.jordbo.se alone does not enable mining. HTTPS covers the read/write tool surface (search, add_drawer, diary_write, kg_*) — genuinely useful on its own, and the reason to do this at all. But --mode remote also needs MEMPALACE_PI_SSH_TARGET reachable. Budget for both paths.
• DEVICE defaults to $(hostname) (bin/mempalace-pi-session:163). In a container that is the container hostname: either random per recreate (inboxes proliferate; each recreate re-mines into a fresh empty inbox) or identical across sibling devboxes (two containers writing one inbox). Set MEMPALACE_PI_DEVICE explicitly per container. It is a label, not a secret, so put it somewhere reviewable — a committed compose file — where duplicates are visible. That, not username hygiene in .env, is the discipline this design actually asks of you.

1.3 "Then why Pangolin at all, if the feeder uses SSH?"

Because they are not alternatives — they carry different traffic, and neither substitutes for the other.

	Pangolin/newt (HTTPS)	SSH + rsync
Carries	the MCP tool surface: search, add_drawer, diary_write, kg_* — every live tool call	transcript files only, once per session or cron run
Used by	the pi extension, opencode type:remote, any MCP client	the feeder, internally (bin/mempalace-pi-session:677-680)
Needed because	clients need one stable URL, reachable from wherever they are	mempalace_mine expands its source path server-side, so the server can only mine files on its own disk

HTTPS alone is a palace you can query but cannot feed. SSH alone is files shipped with no live query API. The rsync is not a transport preference; it is a workaround for where mine resolves paths.

Could SSH replace Pangolin? Partly, and it is worth being honest about it: ssh -L 8765:172.17.0.1:8765 synlig yields a working local MCP endpoint with no public HTTPS at all. Three reasons this runbook does not do that:

1. Direction. synlig dials out through newt. That we reached for a dial-out tunnel rather than a port-forward is itself the evidence that inbound was not available — a corporate host does not accept connections from a phone on a foreign network.
2. MCP clients want a durable URL, not a per-session forwarded port. opencode type:remote takes a URL; a forward that drops takes the tools down mid-session.
3. The forward must be up on every device before every session. Pangolin is up once.

The weak point, stated plainly. The rsync runs client → synlig, so it needs synlig's SSH reachable from the client. Were that already true everywhere, no tunnel would be needed for MCP either. So the honest expectation after Phase 1 is: query and write from anywhere, mine only from devices that can reach synlig's SSH (corporate network / VPN / LAN). See §4 for the change that would remove that limit.

---

2. The bind trap, in full

RFC §6.2 and runbook §2.4 already say do not bind loopback behind the tunnel, because enforce_host_pin = _http_is_loopback(host) (mcp_server.py:5367) makes a loopback bind reject the proxy's forwarded Host: with a 403 that reads exactly like a Pangolin misconfiguration.

Additional finding, 2026-08-12 — the same reflex also silently removes authentication. Token resolution in cmd_serve (cli.py:1447-1450) is:

loopback = _server_is_loopback(host)
if not token and not loopback and not args.allow_insecure:
    token, token_created = _load_or_create_server_token(palace_path)

Auto-minting is gated on the bind being non-loopback. A loopback bind therefore starts with no token at all — no error, no warning, --allow-insecure not required — because the server has concluded it is only reachable locally, while the tunnel is serving it to the internet. Bind loopback behind newt and you get a 403 wall and, the moment anything relaxes the Host pin, an unauthenticated palace.

Both failure modes have the same cure, already implemented in contrib/systemd/mempalace-serve.service: bind 172.17.0.1. Non-loopback, so the Host pin relaxes and the token is mandatory; docker0-only, so newt reaches it and the LAN does not.

Belt and braces: set MEMPALACE_MCP_HTTP_TOKEN explicitly in the unit rather than relying on auto-minting. Then no future bind change can quietly drop authentication.

---

3. Steps

Ordered so nothing is reachable before it is authenticated.

3.1 Start the primary (runbook §4.3 — one sudo, unit already staged)

sudo loginctl enable-linger ecsjper
cd ~/.config/systemd/user && mv mempalace-serve.service.staged mempalace-serve.service
systemctl --user daemon-reload && systemctl --user enable --now mempalace-serve

curl -s 172.17.0.1:8765/healthz     # expect ok
curl -s 127.0.0.1:8765/healthz      # expect NOTHING — connection refused, exit 7 (see below)
ss -ltnp | grep 8765                # expect 172.17.0.1:8765 only

⚠ The loopback probe returns empty, not 403 — corrected 2026-08-12 against the real run. Nothing is listening on 127.0.0.1, so the connection is refused at TCP level and curl -s prints nothing; check it with -w '%{http_code}' → 000 and $? → 7. The 403 belongs to a different configuration: server bound to loopback, receiving a proxy-forwarded foreign Host: (§2, verified 2026-08-10). With a docker0-only bind you cannot get 403 from loopback, because you never get far enough to send a header. Refusal is the stronger signal of the two: it proves the loopback and LAN surface is not listening at all. If it hangs instead, or ss shows 0.0.0.0:8765, stop — that is not this configuration.

### 3.2 Collect the shared token

```sh
cat ~/.mempalace/server/f5d849287f6d73f0141b29d7/token

Directory name is sha256(realpath(palace))[:24] — it changes if the palace path ever moves. Store via the .env.age flow, 0600 (RFC §6.2).

3.3 newt on synlig — ✅ done 2026-08-12 (installed, connected to Pangolin)

synlig runs Docker (Gitea Actions runner + digikam) but no tunnel client — runbook §4.2. Pangolin on nyvaken cannot dial in; synlig must dial out. Add a newt container with the credentials Pangolin issues for a new site.

Because newt runs in Docker on this box, the docker0 bind is already correct for it: from inside the container the primary is 172.17.0.1:8765. Verify from inside newt's network namespace, not from the host, before touching DNS.

synlig has 7.8 GiB shared with a CI runner (runbook §1). newt is small, but do not colocate anything else here casually.

Confirm next, now that newt is up. "Connected to Pangolin" proves newt reached nyvaken — a different claim from newt reaching the palace, and the two fail independently:

# from inside newt's namespace, not from the host
docker exec <newt-container> wget -qO- http://172.17.0.1:8765/healthz    # expect ok

If that hangs or refuses while the Pangolin dashboard shows the site online, the tunnel is fine and the target is wrong — look at the resource's upstream address (§3.5), not at newt. Note this check needs §3.1 done first: if mempalace-serve is not running yet, it fails for that reason alone.

3.4 DNS at the web hotel — ✅ done 2026-08-12

One CNAME: mempalace → the same target your existing Pangolin resources use (nyvaken's public hostname). RFC §6.2 costed this as "one DNS record per service on the web hotel is the whole setup cost." Done: mempalace.jordbo.se resolves and terminates TLS at Pangolin on nyvaken — the §6.2 estimate held.

⚠️ Not verified from here: nyvaken's public FQDN, and whether your web hotel permits a CNAME at that label (some require an A record, or forbid CNAME where other records exist). Confirm before assuming a 5-minute job.

3.5 Pangolin resource

• Target: newt site → http://172.17.0.1:8765 — path /mcp (plus /healthz for the external probe).
• ⚠️ The scheme is http, not https. The primary runs serve --host 172.17.0.1 --port 8765 with no cert (contrib/systemd/mempalace-serve.service): TLS terminates at Pangolin, which is the entire point of the §6.2 decision. Point the resource at https://172.17.0.1:8765 and Pangolin attempts a TLS handshake against a plaintext listener — you get a 502/Bad Gateway from outside while the server itself looks perfectly healthy on curl 172.17.0.1:8765/healthz. (Got this wrong on the first attempt 2026-08-12, because this line used to omit the scheme.)
• Auth: none at the Pangolin layer (§1.1). TLS termination only.
• ⚠️ If resource auth is left on, the signature is a 302, not a 401 or 403 — hit for real 2026-08-12:

  HTTP/2 302
  location: https://pangolin.jordbo.se/auth/resource/<uuid>?redirect=https%3A%2F%2Fmempalace.jordbo.se%2Fhealthz
  content-length: 0
  

  This is §1.1's "browser-shaped auth" arriving as a concrete symptom: Pangolin sends the login redirect before proxying, so the palace never sees the request and its journal stays silent. curl -s shows an empty body and an MCP client sees non-JSON. Diagnose with -D- or -w '%{http_code} %{redirect_url}' — a location: pointing at /auth/resource/… means the fix is in the Pangolin UI (switch the site's authentication off), not in the palace, the unit, or newt. A 302 is unambiguously good news: DNS, TLS and routing all worked — only the auth layer intervened.
• Do not attach an Origin-injecting proxy or browser client: a present non-loopback Origin is a hard 403 with no override (runbook §2.4 B3).

The server's entire HTTP surface is two exact paths, so path-scoped rules cover it completely (mcp_server.py:5299-5318, read 2026-08-12):

Method + path	Auth	Notes
GET /healthz	none (Host/Origin gated only)	the liveness probe; works with no creds by design
POST /mcp	Authorization: Bearer <token>, hmac.compare_digest on the exact string	the whole tool surface
anything else	—	send_error(404) from the palace itself

Three consequences worth having in writing:

• Path-scoped Pangolin rules are not a compromise here, they are tighter than a host-wide proxy and lose nothing — there is no third endpoint to forget.
• /mcp is matched exactly (if path != "/mcp"), so a client URL with a trailing slash gets a 404 from the palace. Configure clients as https://mempalace.jordbo.se/mcp — no trailing slash.
• There is no GET /mcp, no SSE, no session id, no DELETE. This is plain JSON-RPC over POST, not MCP streamable-HTTP. A strict client that opens with a GET handshake will see 404; pi's extension, opencode type:remote and the feeder all POST directly and are fine.

3.6 Verify end-to-end before flipping any client — ✅ passed 2026-08-12

curl -s https://mempalace.jordbo.se/healthz                      # ok        ✓ from devbox AND synlig
curl -s -o /dev/null -w '%{http_code}\n' -X POST \
     https://mempalace.jordbo.se/mcp                             # 401       ✓ the token is doing its job
TOKEN=$(cat ~/.mempalace/server/f5d849287f6d73f0141b29d7/token)  # on synlig
curl -s -X POST https://mempalace.jordbo.se/mcp \
     -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
     -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | head -c 300   # 36 tools

No initialize and no Accept: text/event-stream needed — see the surface table above; a bare tools/list POST is a complete request. Run the 200 and the 401 from two different networks (a client box and the primary itself): passing from only one leaves split-horizon DNS untested.

The 401 check matters as much as the 200: it is the only evidence that the thing you just published to the internet is not open. Then, and only then, Phase 1 client flip — one machine first (RFC §8), and remember opencode containers need the §4.1 sidecar merge before their .env takes effect.

3.7 Flipping a client: set three variables, or none

⚠ MEMPALACE_REMOTE_URL on its own does not degrade to local feeding — it stops feeding. auto mode switches to remote the moment the URL is set, and remote mode then refuses to run without an SSH target:

auto) if [[ -n "$REMOTE_URL" ]]; then MODE="remote"; else MODE="local"; fi ;;   # :286
...
command -v rsync >/dev/null 2>&1 || { echo "error: rsync not found ..."; exit 3; }   # :297
if [[ -z "$SSH_TARGET" ]]; then
  echo "error: MEMPALACE_PI_SSH_TARGET unset (needed for --mode remote)" >&2; exit 1  # :298-300
fi

That exit happens before anything is staged or filed, and a cron-driven feeder will simply start failing — the loudest symptom is silence, which is the hardest kind to notice. Two safe orders:

• Both paths at once: set MEMPALACE_REMOTE_URL, MEMPALACE_REMOTE_TOKEN and MEMPALACE_PI_SSH_TARGET (plus MEMPALACE_PI_DEVICE, §1.2) in the same edit.
• HTTPS first, mining later: set the URL and token, and pin the feeder to --mode local until the SSH target exists. Tools then read/write the shared palace while transcripts keep landing in the local one.

MEMPALACE_REMOTE_URL is the full endpoint including /mcp, with no trailing slash — the feeder POSTs to it verbatim (urllib.request.Request(url, data=payload, …), :718), and §3.5 shows the server matches /mcp exactly. Matches the existing examples (docker-compose.mempalace.yml:7, MEMPALACE_REMOTE_URL=http://<reachable-host>:8765/mcp). For this fleet: MEMPALACE_REMOTE_URL=https://mempalace.jordbo.se/mcp

Either way, run the feeder once by hand and read its exit code before trusting the timer. This is precisely the failure "one machine first" is meant to contain.

The same misconfiguration has two different symptoms depending on who invokes the feeder — checked in the code 2026-08-12, and the quieter one is the trap:

Caller	Behaviour with REMOTE_URL set and SSH_TARGET unset
direct run, session-end hook, cron	exit 1 with error: MEMPALACE_PI_SSH_TARGET unset (:298-300)
pi-devbox container start	silently skips — no error, no log file (entrypoint-user.sh:134: : # remote palace but no inbox configured — nothing we can ship to; skip quietly)

The entrypoint's skip happens before the subshell that writes ~/.pi/agent/mempalace-catchup.log, so there is not even an empty log to notice. Someone asking "why is nothing from this container in the palace?" finds no artifact at all. The skip is deliberate — there is genuinely nothing to ship to — but it is indistinguishable from a healthy run that had nothing to do.

→ On a pi-devbox container, confirm the feeder is actually alive after a flip rather than assuming:

mempalace-pi-session --reason manual-check; echo "exit=$?"   # exit=0 and a filed count, not silence
cat ~/.pi/agent/mempalace-catchup.log                        # missing file = the entrypoint skipped

---

4. Still open

• Feeding without SSH — the upstream ask that would close §1.3's gap. Have the feeder send content over MCP (add_drawer / diary_write, which it already calls) instead of asking the server to mine a path it must first rsync there. HTTPS would then be genuinely sufficient and mining would work from any network. Until then, mining is limited to devices that can reach synlig's SSH.
• Per-device tokens — Phase 4. Until then origin_device is advisory (§1.1).
• §7.6 diary dedup must be settled before the first §4.4 join; replay duplicates every entry.
• §7.2: never run mempalace sync against the shared palace. Doubly true now that the pi/opencode feeders stage inside the palace root, which puts staged sources in scope for a sync of the palace dir.
• nyvaken's public FQDN and the web hotel's CNAME rules — unverified (§3.4).