mempalace-toolkit/docs/phase-1-exposure-runbook.md

# Phase 1 exposure — newt on synlig, DNS, and the client auth model

Companion to [`rfc-001-global-palace.md`](./rfc-001-global-palace.md) (design + decisions) and
[`synlig-primary-runbook.md`](./synlig-primary-runbook.md) (what is already installed on the primary).
This doc covers only the step the other two leave open: **making the primary reachable** — runbook §4
items 2 and 5.

**Status 2026-08-12 — Pangolin updated on nyvaken (done, yours). newt not yet installed on synlig.
Nothing exposed. No client `.env` flipped.**

Read this before touching Pangolin: three of the four questions this step raises were **already decided**
in RFC §6.2 on 2026-08-09, and re-deciding them differently is how the fleet ends up in two states.

---

## 1. The four questions, answered

| Question | Answer | Where it was decided |
| --- | --- | --- |
| Which port? | **8765**, path **`/mcp`** (liveness: `/healthz`) | `cli.py:2141` default; runbook §2.4 |
| What does newt target? | **`172.17.0.1:8765`** (docker0), **never** `127.0.0.1` | RFC §6.2 Transport; runbook §2.4 |
| Open, or authenticated? | **Authenticated. The primary is never an open public resource.** | RFC §6.2 Network posture |
| Per-device credentials? | **No — Phase 1 ships the single shared bearer token.** Per-device tokens are Phase 4. | RFC §6.2 Authentication |

### 1.1 Why not per-device users at the proxy

The instinct — "create a Pangolin user per container, put the credentials in each `.env`, keep the
usernames distinct" — is the right *goal* (revocation, attribution) reached through the wrong *layer*,
twice over:

1. **mempalace validates exactly one token.** `hmac.compare_digest(provided, f"Bearer {srv.auth_token}")`
   (`mcp_server.py:5292-5295`) — there is no user table and no second credential. Per-device HTTP identity
   is not a configuration you can express today; it is Phase 4 work (a server-side
   `token → {device_id, scopes}` registry). RFC §6.2 chose the shared token for Phase 1 deliberately:
   *"iterate more feature rich but more complex solutions over time."*

2. **Pangolin's HTTP auth is browser-shaped; the clients are not.** SSO login, resource PIN and resource
   password all assume something that can follow a redirect, render a form and hold a session cookie.
   Every MemPalace client here is a headless JSON-RPC `POST` with an `Authorization` header — pi's
   extension, opencode's `type:remote` MCP entry, and `mempalace-pi-session --mode remote`'s
   `urllib.request.urlopen`. Point those at a user-authenticated resource and they receive a login page
   where JSON should be. Enabling that protection breaks precisely the clients it is meant to protect.

So: **Pangolin terminates TLS and nothing more** (RFC §6.2 Transport, decided 2026-08-09). The bearer token
is the authentication. This is not "unprotected" — an unauthenticated request to `/mcp` gets a 401 from
mempalace itself, verified A4/A5 in runbook §2.4.

**Consequence to accept consciously** (RFC §6.2, §7.3.2): until Phase 4 the primary **cannot tell devices
apart**. `origin_device` is client-asserted and advisory — nothing load-bearing may depend on it, and
revoking one laptop means rotating the token everywhere.

### 1.2 The one place per-device identity *does* exist today

Remote mode is not only HTTP. `mempalace_mine` expands its source path in the **server** process, so a
client's staged transcripts must physically exist on the primary. The feeder therefore ships them over
SSH into a **per-device inbox** before asking the server to mine its own copy:

```sh
rsync -a --update -e "$ssh_cmd" "$STAGE/" "${SSH_TARGET%/}/$DEVICE/"   # bin/mempalace-pi-session:677-680
```

That SSH key **is** per-device identity, and it is individually revocable (one line out of
`authorized_keys`) years before Phase 4 lands. It costs nothing extra, because the mining path needs SSH
regardless.

Two implications people miss:

- **`mempalace.jordbo.se` alone does not enable mining.** HTTPS covers the read/write tool surface
  (`search`, `add_drawer`, `diary_write`, `kg_*`) — genuinely useful on its own, and the reason to do this
  at all. But `--mode remote` also needs `MEMPALACE_PI_SSH_TARGET` reachable. Budget for both paths.
- **`DEVICE` defaults to `$(hostname)`** (`bin/mempalace-pi-session:163`). In a container that is the
  container hostname: either random per recreate (inboxes proliferate; each recreate re-mines into a fresh
  empty inbox) or identical across sibling devboxes (two containers writing one inbox). **Set
  `MEMPALACE_PI_DEVICE` explicitly per container.** It is a label, not a secret, so put it somewhere
  reviewable — a committed compose file — where duplicates are visible. That, not username hygiene in
  `.env`, is the discipline this design actually asks of you.

### 1.3 "Then why Pangolin at all, if the feeder uses SSH?"

Because they are not alternatives — they carry different traffic, and neither substitutes for the other.

| | Pangolin/newt (HTTPS) | SSH + rsync |
| --- | --- | --- |
| Carries | the **MCP tool surface**: `search`, `add_drawer`, `diary_write`, `kg_*` — every live tool call | **transcript files only**, once per session or cron run |
| Used by | the pi extension, opencode `type:remote`, any MCP client | the feeder, internally (`bin/mempalace-pi-session:677-680`) |
| Needed because | clients need one stable URL, reachable from wherever they are | `mempalace_mine` expands its source path **server-side**, so the server can only mine files on its own disk |

HTTPS alone is a palace you can query but cannot feed. SSH alone is files shipped with no live query API.
The rsync is not a transport preference; it is a workaround for *where `mine` resolves paths*.

**Could SSH replace Pangolin?** Partly, and it is worth being honest about it:
`ssh -L 8765:172.17.0.1:8765 synlig` yields a working local MCP endpoint with no public HTTPS at all.
Three reasons this runbook does not do that:

1. **Direction.** synlig dials *out* through newt. That we reached for a dial-out tunnel rather than a
   port-forward is itself the evidence that inbound was not available — a corporate host does not accept
   connections from a phone on a foreign network.
2. **MCP clients want a durable URL**, not a per-session forwarded port. opencode `type:remote` takes a
   URL; a forward that drops takes the tools down mid-session.
3. The forward must be up on **every device before every session**. Pangolin is up once.

**The weak point, stated plainly.** The rsync runs *client → synlig*, so it needs synlig's SSH reachable
**from the client**. Were that already true everywhere, no tunnel would be needed for MCP either. So the
honest expectation after Phase 1 is: **query and write from anywhere, mine only from devices that can
reach synlig's SSH** (corporate network / VPN / LAN). See §4 for the change that would remove that limit.

---

## 2. The bind trap, in full

RFC §6.2 and runbook §2.4 already say **do not bind loopback behind the tunnel**, because
`enforce_host_pin = _http_is_loopback(host)` (`mcp_server.py:5367`) makes a loopback bind reject the
proxy's forwarded `Host:` with a **403** that reads exactly like a Pangolin misconfiguration.

**Additional finding, 2026-08-12 — the same reflex also silently removes authentication.** Token
resolution in `cmd_serve` (`cli.py:1447-1450`) is:

```python
loopback = _server_is_loopback(host)
if not token and not loopback and not args.allow_insecure:
    token, token_created = _load_or_create_server_token(palace_path)
```

Auto-minting is gated on the bind being **non-loopback**. A loopback bind therefore starts with **no token
at all** — no error, no warning, `--allow-insecure` not required — because the server has concluded it is
only reachable locally, while the tunnel is serving it to the internet. Bind loopback behind newt and you
get a 403 wall *and*, the moment anything relaxes the Host pin, an unauthenticated palace.

Both failure modes have the same cure, already implemented in
`contrib/systemd/mempalace-serve.service`: **bind `172.17.0.1`**. Non-loopback, so the Host pin relaxes and
the token is mandatory; docker0-only, so newt reaches it and the LAN does not.

> Belt and braces: set `MEMPALACE_MCP_HTTP_TOKEN` explicitly in the unit rather than relying on
> auto-minting. Then no future bind change can quietly drop authentication.

---

## 3. Steps

Ordered so nothing is reachable before it is authenticated.

### 3.1 Start the primary (runbook §4.3 — one `sudo`, unit already staged)

```sh
sudo loginctl enable-linger ecsjper
cd ~/.config/systemd/user && mv mempalace-serve.service.staged mempalace-serve.service
systemctl --user daemon-reload && systemctl --user enable --now mempalace-serve

curl -s 172.17.0.1:8765/healthz     # expect ok
curl -s 127.0.0.1:8765/healthz      # expect NOTHING — connection refused, exit 7 (see below)
ss -ltnp | grep 8765                # expect 172.17.0.1:8765 only
```

⚠ **The loopback probe returns empty, not 403** — corrected 2026-08-12 against the real run. Nothing is
listening on `127.0.0.1`, so the connection is refused at TCP level and `curl -s` prints nothing; check it
with `-w '%{http_code}'` → `000` and `$?` → `7`. The 403 belongs to a *different* configuration: server
bound **to loopback**, receiving a proxy-forwarded foreign `Host:` (§2, verified 2026-08-10). With a
docker0-only bind you cannot get 403 from loopback, because you never get far enough to send a header.
Refusal is the stronger signal of the two: it proves the loopback and LAN surface is not listening at all.
If it *hangs* instead, or `ss` shows `0.0.0.0:8765`, stop — that is not this configuration.
```sh

### 3.2 Collect the shared token

```sh
cat ~/.mempalace/server/f5d849287f6d73f0141b29d7/token
```

Directory name is `sha256(realpath(palace))[:24]` — it changes if the palace path ever moves. Store via the
`.env.age` flow, 0600 (RFC §6.2).

### 3.3 newt on synlig — ✅ done 2026-08-12 (installed, connected to Pangolin)

synlig runs Docker (Gitea Actions runner + digikam) but **no tunnel client** — runbook §4.2. Pangolin on
nyvaken cannot dial in; synlig must dial out. Add a `newt` container with the credentials Pangolin issues
for a new site.

Because newt runs in Docker on this box, the docker0 bind is already correct for it: from inside the
container the primary is `172.17.0.1:8765`. Verify from *inside* newt's network namespace, not from the
host, before touching DNS.

> synlig has 7.8 GiB shared with a CI runner (runbook §1). newt is small, but do not colocate anything
> else here casually.

**Confirm next, now that newt is up.** "Connected to Pangolin" proves newt reached *nyvaken* — a different
claim from newt reaching *the palace*, and the two fail independently:

```sh
# from inside newt's namespace, not from the host
docker exec <newt-container> wget -qO- http://172.17.0.1:8765/healthz    # expect ok
```

If that hangs or refuses while the Pangolin dashboard shows the site online, the tunnel is fine and the
*target* is wrong — look at the resource's upstream address (§3.5), not at newt. Note this check needs
§3.1 done first: if `mempalace-serve` is not running yet, it fails for that reason alone.

### 3.4 DNS at the web hotel — ✅ done 2026-08-12

One CNAME: `mempalace` → **the same target your existing Pangolin resources use** (nyvaken's public
hostname). RFC §6.2 costed this as *"one DNS record per service on the web hotel is the whole setup cost."*
Done: `mempalace.jordbo.se` resolves and terminates TLS at Pangolin on nyvaken — the §6.2 estimate held.

⚠️ Not verified from here: nyvaken's public FQDN, and whether your web hotel permits a CNAME at that label
(some require an A record, or forbid CNAME where other records exist). Confirm before assuming a 5-minute job.

### 3.5 Pangolin resource

- Target: newt site → **`http://172.17.0.1:8765`** — path `/mcp` (plus `/healthz` for the external probe).
- ⚠️ **The scheme is `http`, not `https`.** The primary runs `serve --host 172.17.0.1 --port 8765` with no
  cert (`contrib/systemd/mempalace-serve.service`): TLS terminates **at Pangolin**, which is the entire
  point of the §6.2 decision. Point the resource at `https://172.17.0.1:8765` and Pangolin attempts a TLS
  handshake against a plaintext listener — you get a 502/Bad Gateway from outside while the server itself
  looks perfectly healthy on `curl 172.17.0.1:8765/healthz`. (Got this wrong on the first attempt
  2026-08-12, because this line used to omit the scheme.)
- **Auth: none at the Pangolin layer** (§1.1). TLS termination only.
- ⚠️ **If resource auth is left on, the signature is a `302`, not a 401 or 403** — hit for real 2026-08-12:
  ```
  HTTP/2 302
  location: https://pangolin.jordbo.se/auth/resource/<uuid>?redirect=https%3A%2F%2Fmempalace.jordbo.se%2Fhealthz
  content-length: 0
  ```
  This is §1.1's "browser-shaped auth" arriving as a concrete symptom: Pangolin sends the login redirect
  **before** proxying, so the palace never sees the request and its journal stays silent. `curl -s` shows
  an empty body and an MCP client sees non-JSON. Diagnose with `-D-` or
  `-w '%{http_code} %{redirect_url}'` — a `location:` pointing at `/auth/resource/…` means the fix is in
  the Pangolin UI (switch the site's authentication off), not in the palace, the unit, or newt.
  **A 302 is unambiguously good news:** DNS, TLS and routing all worked — only the auth layer intervened.
- Do **not** attach an `Origin`-injecting proxy or browser client: a *present* non-loopback `Origin` is a
  hard 403 with no override (runbook §2.4 B3).

**The server's entire HTTP surface is two exact paths**, so path-scoped rules cover it completely
(`mcp_server.py:5299-5318`, read 2026-08-12):

| Method + path | Auth | Notes |
| --- | --- | --- |
| `GET /healthz` | none (Host/Origin gated only) | the liveness probe; works with no creds by design |
| `POST /mcp` | `Authorization: Bearer <token>`, `hmac.compare_digest` on the exact string | the whole tool surface |
| anything else | — | `send_error(404)` from the palace itself |

Three consequences worth having in writing:

- **Path-scoped Pangolin rules are not a compromise here, they are tighter than a host-wide proxy** and
  lose nothing — there is no third endpoint to forget.
- **`/mcp` is matched exactly** (`if path != "/mcp"`), so a client URL with a trailing slash gets a 404
  from the palace. Configure clients as `https://mempalace.jordbo.se/mcp` — no trailing slash.
- **There is no `GET /mcp`, no SSE, no session id, no `DELETE`.** This is plain JSON-RPC over POST, not MCP
  streamable-HTTP. A strict client that opens with a `GET` handshake will see 404; pi's extension,
  opencode `type:remote` and the feeder all POST directly and are fine.

### 3.6 Verify end-to-end before flipping any client — ✅ passed 2026-08-12

```sh
curl -s https://mempalace.jordbo.se/healthz                      # ok        ✓ from devbox AND synlig
curl -s -o /dev/null -w '%{http_code}\n' -X POST \
     https://mempalace.jordbo.se/mcp                             # 401       ✓ the token is doing its job
TOKEN=$(cat ~/.mempalace/server/f5d849287f6d73f0141b29d7/token)  # on synlig
curl -s -X POST https://mempalace.jordbo.se/mcp \
     -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
     -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | head -c 300   # 36 tools
```

No `initialize` and no `Accept: text/event-stream` needed — see the surface table above; a bare
`tools/list` POST is a complete request. Run the 200 and the 401 from **two different networks** (a client
box and the primary itself): passing from only one leaves split-horizon DNS untested.

The 401 check matters as much as the 200: it is the only evidence that the thing you just published to the
internet is not open. Then, and only then, Phase 1 client flip — **one machine first** (RFC §8), and
remember opencode containers need the §4.1 sidecar merge before their `.env` takes effect.

### 3.7 Flipping a client: set three variables, or none

⚠ **`MEMPALACE_REMOTE_URL` on its own does not degrade to local feeding — it stops feeding.** `auto` mode
switches to `remote` the moment the URL is set, and remote mode then refuses to run without an SSH target:

```sh
auto) if [[ -n "$REMOTE_URL" ]]; then MODE="remote"; else MODE="local"; fi ;;   # :286
...
command -v rsync >/dev/null 2>&1 || { echo "error: rsync not found ..."; exit 3; }   # :297
if [[ -z "$SSH_TARGET" ]]; then
  echo "error: MEMPALACE_PI_SSH_TARGET unset (needed for --mode remote)" >&2; exit 1  # :298-300
fi
```

That exit happens **before anything is staged or filed**, and a cron-driven feeder will simply start
failing — the loudest symptom is silence, which is the hardest kind to notice. Two safe orders:

- **Both paths at once:** set `MEMPALACE_REMOTE_URL`, `MEMPALACE_REMOTE_TOKEN` **and**
  `MEMPALACE_PI_SSH_TARGET` (plus `MEMPALACE_PI_DEVICE`, §1.2) in the same edit.
- **HTTPS first, mining later:** set the URL and token, and pin the feeder to `--mode local` until the SSH
  target exists. Tools then read/write the shared palace while transcripts keep landing in the local one.

`MEMPALACE_REMOTE_URL` is the **full endpoint including `/mcp`, with no trailing slash** — the feeder POSTs
to it verbatim (`urllib.request.Request(url, data=payload, …)`, `:718`), and §3.5 shows the server matches
`/mcp` exactly. Matches the existing examples (`docker-compose.mempalace.yml:7`,
`MEMPALACE_REMOTE_URL=http://<reachable-host>:8765/mcp`). For this fleet:
`MEMPALACE_REMOTE_URL=https://mempalace.jordbo.se/mcp`

Either way, run the feeder once by hand and read its exit code before trusting the timer. This is
precisely the failure "one machine first" is meant to contain.

**The same misconfiguration has two different symptoms depending on who invokes the feeder** — checked in
the code 2026-08-12, and the quieter one is the trap:

| Caller | Behaviour with `REMOTE_URL` set and `SSH_TARGET` unset |
| --- | --- |
| direct run, session-end hook, cron | `exit 1` with `error: MEMPALACE_PI_SSH_TARGET unset` (`:298-300`) |
| **pi-devbox container start** | **silently skips — no error, no log file** (`entrypoint-user.sh:134`: `: # remote palace but no inbox configured — nothing we can ship to; skip quietly`) |

The entrypoint's skip happens *before* the subshell that writes
`~/.pi/agent/mempalace-catchup.log`, so there is not even an empty log to notice. Someone asking "why is
nothing from this container in the palace?" finds no artifact at all. The skip is deliberate — there is
genuinely nothing to ship to — but it is indistinguishable from a healthy run that had nothing to do.

→ On a pi-devbox container, confirm the feeder is actually alive after a flip rather than assuming:

```sh
mempalace-pi-session --reason manual-check; echo "exit=$?"   # exit=0 and a filed count, not silence
cat ~/.pi/agent/mempalace-catchup.log                        # missing file = the entrypoint skipped
```

---

## 4. Still open

- **Feeding without SSH — the upstream ask that would close §1.3's gap.** Have the feeder send *content*
  over MCP (`add_drawer` / `diary_write`, which it already calls) instead of asking the server to mine a
  path it must first rsync there. HTTPS would then be genuinely sufficient and mining would work from any
  network. Until then, mining is limited to devices that can reach synlig's SSH.
- **Per-device tokens** — Phase 4. Until then `origin_device` is advisory (§1.1).
- **§7.6 diary dedup** must be settled *before* the first §4.4 join; replay duplicates every entry.
- **§7.2**: never run `mempalace sync` against the shared palace. Doubly true now that the pi/opencode
  feeders stage *inside* the palace root, which puts staged sources in scope for a sync of the palace dir.
- **nyvaken's public FQDN and the web hotel's CNAME rules** — unverified (§3.4).