From 08e344b047047e1fad1c8f3c502aec4153e5af32 Mon Sep 17 00:00:00 2001 From: Joakim Persson Date: Thu, 13 Aug 2026 16:01:02 +0200 Subject: [PATCH] docs: proxy target is http:// not https://; loopback probe refuses, it does not 403 Both corrections come from the first real Phase 1 start on synlig (2026-08-12). 1. The Pangolin resource target was documented as bare `172.17.0.1:8765` with no scheme, and the obvious guess from that is `https://` -- which cannot work. `contrib/systemd/mempalace-serve.service` runs `serve --host 172.17.0.1 --port 8765` with no --tls-cert, so the primary speaks plaintext HTTP; TLS terminates at Pangolin, which is the whole point of the RFC 6.2 decision. Point a proxy at https:// and it attempts a TLS handshake against a plaintext listener: 502 from outside, while `curl 172.17.0.1:8765/healthz` on the box still says ok -- a confusing pair of symptoms. Now spelled `http://` with the failure mode named, in the runbook and in the unit's comments. 2. `curl -s 127.0.0.1:8765/healthz` was documented as "expect 403". Wrong: the real run returns empty. With a docker0-only bind nothing is listening on loopback, so the connection is refused at TCP level before any header is sent (%{http_code} -> 000, exit 7). The 403 is the *loopback-bind* case verified 2026-08-10 -- server on 127.0.0.1 answering a proxy-forwarded foreign Host. Two distinct behaviours had been collapsed into one expectation in three places (both runbooks and the unit). Worth stating why the correction matters rather than just fixing it: refusal is the *stronger* signal. A 403 proves only that a request was rejected; a refused connection proves the loopback and LAN surface is not listening at all. Someone who expected 403, saw silence, and "fixed" it by rebinding to 0.0.0.0 would have converted a correct configuration into an exposed one. The docs now also say what to do if it hangs, or if ss shows 0.0.0.0:8765. --- contrib/systemd/mempalace-serve.service | 13 ++++++++++++- docs/phase-1-exposure-runbook.md | 19 +++++++++++++++++-- docs/synlig-primary-runbook.md | 11 +++++++++-- 3 files changed, 38 insertions(+), 5 deletions(-) diff --git a/contrib/systemd/mempalace-serve.service b/contrib/systemd/mempalace-serve.service index beefad1..62ed58c 100644 --- a/contrib/systemd/mempalace-serve.service +++ b/contrib/systemd/mempalace-serve.service @@ -52,5 +52,16 @@ WantedBy=default.target # sudo loginctl enable-linger $USER # else the unit dies with your login # systemctl --user daemon-reload # systemctl --user enable --now mempalace-serve -# curl -s localhost:8765/healthz # -> 403 (expected! see bind note) # curl -s 172.17.0.1:8765/healthz # -> ok +# curl -s localhost:8765/healthz # -> nothing: connection refused, exit 7. +# Corrected 2026-08-12 (the earlier "-> 403" here was wrong). With this +# docker0-only bind nothing listens on loopback, so the connection is +# refused before any header is sent. The 403 above is the *loopback-bind* +# case: server on 127.0.0.1 receiving a forwarded foreign Host header. +# Refusal is the stronger signal -- it proves loopback/LAN isn't listening. +# +# ── Reverse-proxy target: use http://, not https:// ────────────────────────── +# There is no --tls-cert below, so this server speaks PLAINTEXT HTTP. Point +# the tunnel/proxy resource at http://172.17.0.1:8765. Targeting https:// makes +# the proxy attempt a TLS handshake against a plaintext listener: 502 from +# outside, while curl on 172.17.0.1 still says ok. TLS belongs at the proxy. diff --git a/docs/phase-1-exposure-runbook.md b/docs/phase-1-exposure-runbook.md index 2d1390a..38bcdc2 100644 --- a/docs/phase-1-exposure-runbook.md +++ b/docs/phase-1-exposure-runbook.md @@ -147,10 +147,19 @@ cd ~/.config/systemd/user && mv mempalace-serve.service.staged mempalace-serve.s systemctl --user daemon-reload && systemctl --user enable --now mempalace-serve curl -s 172.17.0.1:8765/healthz # expect ok -curl -s 127.0.0.1:8765/healthz # expect 403 — correct, not a bug (§2) +curl -s 127.0.0.1:8765/healthz # expect NOTHING — connection refused, exit 7 (see below) ss -ltnp | grep 8765 # expect 172.17.0.1:8765 only ``` +⚠ **The loopback probe returns empty, not 403** — corrected 2026-08-12 against the real run. Nothing is +listening on `127.0.0.1`, so the connection is refused at TCP level and `curl -s` prints nothing; check it +with `-w '%{http_code}'` → `000` and `$?` → `7`. The 403 belongs to a *different* configuration: server +bound **to loopback**, receiving a proxy-forwarded foreign `Host:` (§2, verified 2026-08-10). With a +docker0-only bind you cannot get 403 from loopback, because you never get far enough to send a header. +Refusal is the stronger signal of the two: it proves the loopback and LAN surface is not listening at all. +If it *hangs* instead, or `ss` shows `0.0.0.0:8765`, stop — that is not this configuration. +```sh + ### 3.2 Collect the shared token ```sh @@ -195,7 +204,13 @@ hostname). RFC §6.2 costed this as *"one DNS record per service on the web hote ### 3.5 Pangolin resource -- Target: newt site → `172.17.0.1:8765`, path `/mcp` (plus `/healthz` if you want the external probe). +- Target: newt site → **`http://172.17.0.1:8765`** — path `/mcp` (plus `/healthz` for the external probe). +- ⚠️ **The scheme is `http`, not `https`.** The primary runs `serve --host 172.17.0.1 --port 8765` with no + cert (`contrib/systemd/mempalace-serve.service`): TLS terminates **at Pangolin**, which is the entire + point of the §6.2 decision. Point the resource at `https://172.17.0.1:8765` and Pangolin attempts a TLS + handshake against a plaintext listener — you get a 502/Bad Gateway from outside while the server itself + looks perfectly healthy on `curl 172.17.0.1:8765/healthz`. (Got this wrong on the first attempt + 2026-08-12, because this line used to omit the scheme.) - **Auth: none at the Pangolin layer** (§1.1). TLS termination only. - Do **not** attach an `Origin`-injecting proxy or browser client: a *present* non-loopback `Origin` is a hard 403 with no override (runbook §2.4 B3). diff --git a/docs/synlig-primary-runbook.md b/docs/synlig-primary-runbook.md index b559507..31453d8 100644 --- a/docs/synlig-primary-runbook.md +++ b/docs/synlig-primary-runbook.md @@ -143,14 +143,21 @@ standing up a network-reachable service while you were asleep was not mine to de its own side — which is also why "connected" is not yet proof it can reach the palace: verify `172.17.0.1:8765/healthz` from *inside* newt's namespace, exposure runbook §3.3.) Since newt runs in Docker here, the docker0 bind above is already correct for it. -3. **One sudo, then start** (the unit is already staged; just drop the suffix): +3. **One sudo, then start** (the unit is already staged; just drop the suffix). ✅ **done 2026-08-12** — + linger enabled, unit enabled, `172.17.0.1:8765/healthz` → `ok`. ```sh sudo loginctl enable-linger ecsjper cd ~/.config/systemd/user && mv mempalace-serve.service.staged mempalace-serve.service systemctl --user daemon-reload && systemctl --user enable --now mempalace-serve curl -s 172.17.0.1:8765/healthz # ok - curl -s 127.0.0.1:8765/healthz # 403 — expected, not a bug (§2.4) + curl -s 127.0.0.1:8765/healthz # NOTHING — refused, exit 7 (not 403; see below) + ss -ltnp | grep 8765 # 172.17.0.1:8765 only ``` + ⚠ **Corrected 2026-08-12:** this line predicted `403`. The real run returned empty, which is *more* + reassuring. With the docker0-only bind nothing listens on loopback, so the connection is refused before + any header is sent (`%{http_code}` → `000`, `$?` → `7`). The 403 in §2.4 is the **loopback-bind** case: + a server on `127.0.0.1` answering a proxy-forwarded foreign `Host:`. Two different failures that were + collapsed into one expectation here. 4. **Collect the shared token** (auto-minted on first non-loopback start, stable across restarts): ```sh cat ~/.mempalace/server/f5d849287f6d73f0141b29d7/token