diff --git a/README.md b/README.md index 5cddad1..8392722 100644 --- a/README.md +++ b/README.md @@ -68,7 +68,7 @@ Start here if you are deciding **what to put where**, or running MemPalace on mo | [`docs/rfc-003-coordination-log.md`](docs/rfc-003-coordination-log.md) | The inter-agent / inter-device coordination log (`logstream`): storage, append and query semantics, delivery and latency, the trust model, and its landmines. | | [`docs/rfc-001-global-palace.md`](docs/rfc-001-global-palace.md) | Why and how a palace is centralised, what should *not* be global, and the Phase-0 landmines (including: never run `mempalace sync` against a shared palace). | | [`docs/rfc-002-joiner.md`](docs/rfc-002-joiner.md) | Replaying a second palace into a shared primary, and what dedupes what. | -| [`docs/phase-1-exposure-runbook.md`](docs/phase-1-exposure-runbook.md) | Exposing a palace over HTTP: the Host/Origin pin, tokens, TLS. | +| [`docs/phase-1-exposure-runbook.md`](docs/phase-1-exposure-runbook.md) | *(moved)* The exposure record for one deployment is now private operator data. The stub names the reusable mechanism it also carried — the `Host`/`Origin` pin above all — which is not yet published elsewhere. | | [`docs/backup-and-recovery.md`](docs/backup-and-recovery.md) | Why a palace needs its own backup procedure, and how to restore one. | | [`extensions/pi/README.md`](extensions/pi/README.md) | The pi-side client: provenance stamping at the edge, and the auto-delivered mailbox. | @@ -596,9 +596,12 @@ the palace host and asks the server to mine its own local copy. Requires `MEMPALACE_PI_REMOTE_PATH`, and `MEMPALACE_PI_DEVICE` in `--help` for the rest. `MEMPALACE_PI_DEVICE` also defaults `--agent` to `pi@` so each drawer's `added_by` records which harness and which machine produced it — -mempalace itself stores neither. Deploying that primary — newt, DNS, and why the -auth is a shared bearer token rather than per-device proxy users — is -[`docs/phase-1-exposure-runbook.md`](docs/phase-1-exposure-runbook.md). +mempalace itself stores neither. Deploying a primary — tunnel, DNS, token +custody — is deployment-specific operator data and lives in a private fleet +repository; [`docs/phase-1-exposure-runbook.md`](docs/phase-1-exposure-runbook.md) +records what moved and why. The *consequence* of choosing one shared fleet token +over per-device proxy users is public, in +[`docs/rfc-003-coordination-log.md`](docs/rfc-003-coordination-log.md) §6. --- diff --git a/contrib/README.md b/contrib/README.md index a031c75..c32b39f 100644 --- a/contrib/README.md +++ b/contrib/README.md @@ -48,9 +48,8 @@ The pi variants are drop-in copies of the opencode variants with script name and The odd one out in this directory: every other template *feeds* a palace on a schedule, this one **serves** a palace over HTTP so several machines can share it (RFC-001). **This unit currently runs -the fleet primary on `synlig`** — serving since 2026-08-12, seeded 2026-08-14, and the palace behind -it is the only copy. Read [`docs/rfc-001-global-palace.md`](../docs/rfc-001-global-palace.md) and -[`docs/phase-1-exposure-runbook.md`](../docs/phase-1-exposure-runbook.md) before installing a second one. +a fleet primary** — whose palace may be the only copy. Read [`docs/rfc-001-global-palace.md`](../docs/rfc-001-global-palace.md) +before installing a second one. It is a **user** unit (`systemctl --user`), so it dies with your login session unless lingering is enabled — that is the one `sudo` this recipe needs: diff --git a/docs/fleet-memory.md b/docs/fleet-memory.md index 3561967..0a9df2b 100644 --- a/docs/fleet-memory.md +++ b/docs/fleet-memory.md @@ -212,7 +212,7 @@ Small, and the whole value rests on them: - `docs/rfc-003-coordination-log.md` — the coordination log: storage, semantics, security model, landmines. - `docs/rfc-001-global-palace.md` — how and why a palace is centralised; §5 what should not be global; §7 the landmines, including the `sync` hazard. -- `docs/phase-1-exposure-runbook.md` — exposing a palace over HTTP. +- `docs/phase-1-exposure-runbook.md` — (moved) the site-specific exposure record is now private; the stub names the mechanism still awaiting extraction. - `docs/backup-and-recovery.md` — why a palace needs its own backup procedure. - `extensions/pi/README.md` — the pi-side client: provenance stamping and the auto-delivered mailbox. - `~/.agents/skills/mempalace/SKILL.md` — the protocol the agents themselves follow. diff --git a/docs/phase-1-exposure-runbook.md b/docs/phase-1-exposure-runbook.md index cf9aff8..cdcc5f3 100644 --- a/docs/phase-1-exposure-runbook.md +++ b/docs/phase-1-exposure-runbook.md @@ -1,463 +1,47 @@ -# Phase 1 exposure — newt on synlig, DNS, and the client auth model - -Companion to [`rfc-001-global-palace.md`](./rfc-001-global-palace.md) (design + decisions) and -[`synlig-primary-runbook.md`](./synlig-primary-runbook.md) (what is already installed on the primary). -This doc covers only the step the other two leave open: **making the primary reachable** — runbook §4 -items 2 and 5. - -> **Status 2026-08-14 — DONE. Exposed, seeded, and one client flipped.** -> `https://mempalace.jordbo.se/mcp` has been serving since 2026-08-12 (§3.3–§3.6 are all ✅ below). -> The palace was seeded 2026-08-14 15:07 from EMB-7KJ4VR4G (14,777 → 14,803 drawers) — whose palace was -> itself carried over from the previous work computer **EMB-X1JY06WJ** around 2026-07-06, so the -> primary's lineage is EMB-X1JY06WJ → EMB-7KJ4VR4G → synlig by two file-level copies (rfc-001 §4.4 -> Deviation) — and that machine's pi-devbox container is flipped and **verified end-to-end — see §3.8**, which is the -> verification procedure that did not exist when the first flip was performed. -> Still outstanding: the transcript feeder is **inert on every deployed image** (§3.7), and §7.6 is -> still a hard blocker for the *second* machine to join (§4). - -Original status, kept for the record — it contradicted this file's own ✅ section markers for two days, -which is the failure mode a file-top status block invites: - -**Status 2026-08-12 — Pangolin updated on nyvaken (done, yours). newt not yet installed on synlig. -Nothing exposed. No client `.env` flipped.** - -Read this before touching Pangolin: three of the four questions this step raises were **already decided** -in RFC §6.2 on 2026-08-09, and re-deciding them differently is how the fleet ends up in two states. - ---- - -## 1. The four questions, answered - -| Question | Answer | Where it was decided | -| --- | --- | --- | -| Which port? | **8765**, path **`/mcp`** (liveness: `/healthz`) | `cli.py:2141` default; runbook §2.4 | -| What does newt target? | **`172.17.0.1:8765`** (docker0), **never** `127.0.0.1` | RFC §6.2 Transport; runbook §2.4 | -| Open, or authenticated? | **Authenticated. The primary is never an open public resource.** | RFC §6.2 Network posture | -| Per-device credentials? | **No — Phase 1 ships the single shared bearer token.** Per-device tokens are Phase 4. | RFC §6.2 Authentication | - -### 1.1 Why not per-device users at the proxy - -The instinct — "create a Pangolin user per container, put the credentials in each `.env`, keep the -usernames distinct" — is the right *goal* (revocation, attribution) reached through the wrong *layer*, -twice over: - -1. **mempalace validates exactly one token.** `hmac.compare_digest(provided, f"Bearer {srv.auth_token}")` - (`mcp_server.py:5292-5295`) — there is no user table and no second credential. Per-device HTTP identity - is not a configuration you can express today; it is Phase 4 work (a server-side - `token → {device_id, scopes}` registry). RFC §6.2 chose the shared token for Phase 1 deliberately: - *"iterate more feature rich but more complex solutions over time."* - -2. **Pangolin's HTTP auth is browser-shaped; the clients are not.** SSO login, resource PIN and resource - password all assume something that can follow a redirect, render a form and hold a session cookie. - Every MemPalace client here is a headless JSON-RPC `POST` with an `Authorization` header — pi's - extension, opencode's `type:remote` MCP entry, and `mempalace-pi-session --mode remote`'s - `urllib.request.urlopen`. Point those at a user-authenticated resource and they receive a login page - where JSON should be. Enabling that protection breaks precisely the clients it is meant to protect. - -So: **Pangolin terminates TLS and nothing more** (RFC §6.2 Transport, decided 2026-08-09). The bearer token -is the authentication. This is not "unprotected" — an unauthenticated request to `/mcp` gets a 401 from -mempalace itself, verified A4/A5 in runbook §2.4. - -**Consequence to accept consciously** (RFC §6.2, §7.3.2): until Phase 4 the primary **cannot tell devices -apart**. `origin_device` is client-asserted and advisory — nothing load-bearing may depend on it, and -revoking one laptop means rotating the token everywhere. - -### 1.2 The one place per-device identity *does* exist today - -Remote mode is not only HTTP. `mempalace_mine` expands its source path in the **server** process, so a -client's staged transcripts must physically exist on the primary. The feeder therefore ships them over -SSH into a **per-device inbox** before asking the server to mine its own copy: - -```sh -rsync -a --update -e "$ssh_cmd" "$STAGE/" "${SSH_TARGET%/}/$DEVICE/" # bin/mempalace-pi-session:677-680 -``` - -That SSH key **is** per-device identity, and it is individually revocable (one line out of -`authorized_keys`) years before Phase 4 lands. It costs nothing extra, because the mining path needs SSH -regardless. - -Two implications people miss: - -- **`mempalace.jordbo.se` alone does not enable mining.** HTTPS covers the read/write tool surface - (`search`, `add_drawer`, `diary_write`, `kg_*`) — genuinely useful on its own, and the reason to do this - at all. But `--mode remote` also needs `MEMPALACE_PI_SSH_TARGET` reachable. Budget for both paths. -- **`DEVICE` defaults to `$(hostname)`** (`bin/mempalace-pi-session:163`). In a container that is the - container hostname: either random per recreate (inboxes proliferate; each recreate re-mines into a fresh - empty inbox) or identical across sibling devboxes (two containers writing one inbox). **Set - `MEMPALACE_PI_DEVICE` explicitly per container.** It is a label, not a secret, so put it somewhere - reviewable — a committed compose file — where duplicates are visible. That, not username hygiene in - `.env`, is the discipline this design actually asks of you. - -### 1.3 "Then why Pangolin at all, if the feeder uses SSH?" - -Because they are not alternatives — they carry different traffic, and neither substitutes for the other. - -| | Pangolin/newt (HTTPS) | SSH + rsync | -| --- | --- | --- | -| Carries | the **MCP tool surface**: `search`, `add_drawer`, `diary_write`, `kg_*` — every live tool call | **transcript files only**, once per session or cron run | -| Used by | the pi extension, opencode `type:remote`, any MCP client | the feeder, internally (`bin/mempalace-pi-session:677-680`) | -| Needed because | clients need one stable URL, reachable from wherever they are | `mempalace_mine` expands its source path **server-side**, so the server can only mine files on its own disk | - -HTTPS alone is a palace you can query but cannot feed. SSH alone is files shipped with no live query API. -The rsync is not a transport preference; it is a workaround for *where `mine` resolves paths*. - -**Could SSH replace Pangolin?** Partly, and it is worth being honest about it: -`ssh -L 8765:172.17.0.1:8765 synlig` yields a working local MCP endpoint with no public HTTPS at all. -Three reasons this runbook does not do that: - -1. **Direction.** synlig dials *out* through newt. That we reached for a dial-out tunnel rather than a - port-forward is itself the evidence that inbound was not available — a corporate host does not accept - connections from a phone on a foreign network. -2. **MCP clients want a durable URL**, not a per-session forwarded port. opencode `type:remote` takes a - URL; a forward that drops takes the tools down mid-session. -3. The forward must be up on **every device before every session**. Pangolin is up once. - -**The weak point, stated plainly.** The rsync runs *client → synlig*, so it needs synlig's SSH reachable -**from the client**. Were that already true everywhere, no tunnel would be needed for MCP either. So the -honest expectation after Phase 1 is: **query and write from anywhere, mine only from devices that can -reach synlig's SSH** (corporate network / VPN / LAN). See §4 for the change that would remove that limit. - ---- - -## 2. The bind trap, in full - -RFC §6.2 and runbook §2.4 already say **do not bind loopback behind the tunnel**, because -`enforce_host_pin = _http_is_loopback(host)` (`mcp_server.py:5367`) makes a loopback bind reject the -proxy's forwarded `Host:` with a **403** that reads exactly like a Pangolin misconfiguration. - -**Additional finding, 2026-08-12 — the same reflex also silently removes authentication.** Token -resolution in `cmd_serve` (`cli.py:1447-1450`) is: - -```python -loopback = _server_is_loopback(host) -if not token and not loopback and not args.allow_insecure: - token, token_created = _load_or_create_server_token(palace_path) -``` - -Auto-minting is gated on the bind being **non-loopback**. A loopback bind therefore starts with **no token -at all** — no error, no warning, `--allow-insecure` not required — because the server has concluded it is -only reachable locally, while the tunnel is serving it to the internet. Bind loopback behind newt and you -get a 403 wall *and*, the moment anything relaxes the Host pin, an unauthenticated palace. - -Both failure modes have the same cure, already implemented in -`contrib/systemd/mempalace-serve.service`: **bind `172.17.0.1`**. Non-loopback, so the Host pin relaxes and -the token is mandatory; docker0-only, so newt reaches it and the LAN does not. - -> Belt and braces: set `MEMPALACE_MCP_HTTP_TOKEN` explicitly in the unit rather than relying on -> auto-minting. Then no future bind change can quietly drop authentication. - ---- - -## 3. Steps - -Ordered so nothing is reachable before it is authenticated. - -### 3.1 Start the primary (runbook §4.3 — one `sudo`, unit already staged) - -```sh -sudo loginctl enable-linger ecsjper -cd ~/.config/systemd/user && mv mempalace-serve.service.staged mempalace-serve.service -systemctl --user daemon-reload && systemctl --user enable --now mempalace-serve - -curl -s 172.17.0.1:8765/healthz # expect ok -curl -s 127.0.0.1:8765/healthz # expect NOTHING — connection refused, exit 7 (see below) -ss -ltnp | grep 8765 # expect 172.17.0.1:8765 only -``` - -⚠ **The loopback probe returns empty, not 403** — corrected 2026-08-12 against the real run. Nothing is -listening on `127.0.0.1`, so the connection is refused at TCP level and `curl -s` prints nothing; check it -with `-w '%{http_code}'` → `000` and `$?` → `7`. The 403 belongs to a *different* configuration: server -bound **to loopback**, receiving a proxy-forwarded foreign `Host:` (§2, verified 2026-08-10). With a -docker0-only bind you cannot get 403 from loopback, because you never get far enough to send a header. -Refusal is the stronger signal of the two: it proves the loopback and LAN surface is not listening at all. -If it *hangs* instead, or `ss` shows `0.0.0.0:8765`, stop — that is not this configuration. - -### 3.2 Collect the shared token - -```sh -cat ~/.mempalace/server/f5d849287f6d73f0141b29d7/token -``` - -Directory name is `sha256(realpath(palace))[:24]` — it changes if the palace path ever moves. Store via the -`.env.age` flow, 0600 (RFC §6.2). - -### 3.3 newt on synlig — ✅ done 2026-08-12 (installed, connected to Pangolin) - -synlig runs Docker (Gitea Actions runner + digikam) but **no tunnel client** — runbook §4.2. Pangolin on -nyvaken cannot dial in; synlig must dial out. Add a `newt` container with the credentials Pangolin issues -for a new site. - -Because newt runs in Docker on this box, the docker0 bind is already correct for it: from inside the -container the primary is `172.17.0.1:8765`. Verify from *inside* newt's network namespace, not from the -host, before touching DNS. - -> synlig has 7.8 GiB shared with a CI runner (runbook §1). newt is small, but do not colocate anything -> else here casually. - -**Confirm next, now that newt is up.** "Connected to Pangolin" proves newt reached *nyvaken* — a different -claim from newt reaching *the palace*, and the two fail independently: - -```sh -# from inside newt's namespace, not from the host -docker exec wget -qO- http://172.17.0.1:8765/healthz # expect ok -``` - -If that hangs or refuses while the Pangolin dashboard shows the site online, the tunnel is fine and the -*target* is wrong — look at the resource's upstream address (§3.5), not at newt. Note this check needs -§3.1 done first: if `mempalace-serve` is not running yet, it fails for that reason alone. - -### 3.4 DNS at the web hotel — ✅ done 2026-08-12 - -One CNAME: `mempalace` → **the same target your existing Pangolin resources use** (nyvaken's public -hostname). RFC §6.2 costed this as *"one DNS record per service on the web hotel is the whole setup cost."* -Done: `mempalace.jordbo.se` resolves and terminates TLS at Pangolin on nyvaken — the §6.2 estimate held. - -⚠️ Not verified from here: nyvaken's public FQDN, and whether your web hotel permits a CNAME at that label -(some require an A record, or forbid CNAME where other records exist). Confirm before assuming a 5-minute job. - -### 3.5 Pangolin resource - -- Target: newt site → **`http://172.17.0.1:8765`** — path `/mcp` (plus `/healthz` for the external probe). -- ⚠️ **The scheme is `http`, not `https`.** The primary runs `serve --host 172.17.0.1 --port 8765` with no - cert (`contrib/systemd/mempalace-serve.service`): TLS terminates **at Pangolin**, which is the entire - point of the §6.2 decision. Point the resource at `https://172.17.0.1:8765` and Pangolin attempts a TLS - handshake against a plaintext listener — you get a 502/Bad Gateway from outside while the server itself - looks perfectly healthy on `curl 172.17.0.1:8765/healthz`. (Got this wrong on the first attempt - 2026-08-12, because this line used to omit the scheme.) -- **Auth: none at the Pangolin layer** (§1.1). TLS termination only. -- ⚠️ **If resource auth is left on, the signature is a `302`, not a 401 or 403** — hit for real 2026-08-12: - ``` - HTTP/2 302 - location: https://pangolin.jordbo.se/auth/resource/?redirect=https%3A%2F%2Fmempalace.jordbo.se%2Fhealthz - content-length: 0 - ``` - This is §1.1's "browser-shaped auth" arriving as a concrete symptom: Pangolin sends the login redirect - **before** proxying, so the palace never sees the request and its journal stays silent. `curl -s` shows - an empty body and an MCP client sees non-JSON. Diagnose with `-D-` or - `-w '%{http_code} %{redirect_url}'` — a `location:` pointing at `/auth/resource/…` means the fix is in - the Pangolin UI (switch the site's authentication off), not in the palace, the unit, or newt. - **A 302 is unambiguously good news:** DNS, TLS and routing all worked — only the auth layer intervened. -- Do **not** attach an `Origin`-injecting proxy or browser client: a *present* non-loopback `Origin` is a - hard 403 with no override (runbook §2.4 B3). - -**The server's entire HTTP surface is two exact paths**, so path-scoped rules cover it completely -(`mcp_server.py:5299-5318`, read 2026-08-12): - -| Method + path | Auth | Notes | -| --- | --- | --- | -| `GET /healthz` | none (Host/Origin gated only) | the liveness probe; works with no creds by design | -| `POST /mcp` | `Authorization: Bearer `, `hmac.compare_digest` on the exact string | the whole tool surface | -| anything else | — | `send_error(404)` from the palace itself | - -Three consequences worth having in writing: - -- **Path-scoped Pangolin rules are not a compromise here, they are tighter than a host-wide proxy** and - lose nothing — there is no third endpoint to forget. -- **`/mcp` is matched exactly** (`if path != "/mcp"`), so a client URL with a trailing slash gets a 404 - from the palace. Configure clients as `https://mempalace.jordbo.se/mcp` — no trailing slash. -- **There is no `GET /mcp`, no SSE, no session id, no `DELETE`.** This is plain JSON-RPC over POST, not MCP - streamable-HTTP. A strict client that opens with a `GET` handshake will see 404; pi's extension, - opencode `type:remote` and the feeder all POST directly and are fine. - -### 3.6 Verify end-to-end before flipping any client — ✅ passed 2026-08-12 - -```sh -curl -s https://mempalace.jordbo.se/healthz # ok ✓ from devbox AND synlig -curl -s -o /dev/null -w '%{http_code}\n' -X POST \ - https://mempalace.jordbo.se/mcp # 401 ✓ the token is doing its job -TOKEN=$(cat ~/.mempalace/server/f5d849287f6d73f0141b29d7/token) # on synlig -curl -s -X POST https://mempalace.jordbo.se/mcp \ - -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \ - -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | head -c 300 # 36 tools -``` - -No `initialize` and no `Accept: text/event-stream` needed — see the surface table above; a bare -`tools/list` POST is a complete request. Run the 200 and the 401 from **two different networks** (a client -box and the primary itself): passing from only one leaves split-horizon DNS untested. - -The 401 check matters as much as the 200: it is the only evidence that the thing you just published to the -internet is not open. Then, and only then, Phase 1 client flip — **one machine first** (RFC §8), and -remember opencode containers need the §4.1 sidecar merge before their `.env` takes effect. - -### 3.7 Flipping a client: set three variables, or none - -⚠ **`MEMPALACE_REMOTE_URL` on its own does not degrade to local feeding — it stops feeding.** `auto` mode -switches to `remote` the moment the URL is set, and remote mode then refuses to run without an SSH target: - -```sh -auto) if [[ -n "$REMOTE_URL" ]]; then MODE="remote"; else MODE="local"; fi ;; # :286 -... -command -v rsync >/dev/null 2>&1 || { echo "error: rsync not found ..."; exit 3; } # :297 -if [[ -z "$SSH_TARGET" ]]; then - echo "error: MEMPALACE_PI_SSH_TARGET unset (needed for --mode remote)" >&2; exit 1 # :298-300 -fi -``` - -That exit happens **before anything is staged or filed**, and a cron-driven feeder will simply start -failing — the loudest symptom is silence, which is the hardest kind to notice. Two safe orders: - -- **Both paths at once:** set `MEMPALACE_REMOTE_URL`, `MEMPALACE_REMOTE_TOKEN` **and** - `MEMPALACE_PI_SSH_TARGET` (plus `MEMPALACE_PI_DEVICE`, §1.2) in the same edit. -- **HTTPS first, mining later:** set the URL and token, and pin the feeder to `--mode local` until the SSH - target exists. Tools then read/write the shared palace while transcripts keep landing in the local one. - -`MEMPALACE_REMOTE_URL` is the **full endpoint including `/mcp`, with no trailing slash** — the feeder POSTs -to it verbatim (`urllib.request.Request(url, data=payload, …)`, `:718`), and §3.5 shows the server matches -`/mcp` exactly. Matches the existing examples (`docker-compose.mempalace.yml:7`, -`MEMPALACE_REMOTE_URL=http://:8765/mcp`). For this fleet: -`MEMPALACE_REMOTE_URL=https://mempalace.jordbo.se/mcp` - -Either way, run the feeder once by hand and read its exit code before trusting the timer. This is -precisely the failure "one machine first" is meant to contain. - -**The same misconfiguration has two different symptoms depending on who invokes the feeder** — checked in -the code 2026-08-12, and the quieter one is the trap: - -| Caller | Behaviour with `REMOTE_URL` set and `SSH_TARGET` unset | -| --- | --- | -| direct run, session-end hook, cron | `exit 1` with `error: MEMPALACE_PI_SSH_TARGET unset` (`:298-300`) | -| **pi-devbox container start, images ≤ v1.7.0** | **silently skips — no error, no log file** (`entrypoint-user.sh:134`: `: # remote palace but no inbox configured — nothing we can ship to; skip quietly`) | -| pi-devbox container start, images after `cbd7cf5` | prints `MemPalace catch-up skipped: remote palace with no transcript inbox` to the start output *and* to `mempalace-catchup.log`, naming both variables | - -The silent skip is fixed in pi-devbox (`cbd7cf5`), but the fix is in -`entrypoint-user.sh`, which is `COPY`d in `Dockerfile.base` — so **every container running an image built -before that base rebuild still skips silently.** That is the whole fleet today. Until the rebuild lands, -assume silence and check by hand. - -The old skip happened *before* the subshell that writes `~/.pi/agent/mempalace-catchup.log`, so there was -not even an empty log to notice. Someone asking "why is nothing from this container in the palace?" found -no artifact at all. The skip itself is correct — there is genuinely nothing to ship to — but it was -indistinguishable from a healthy run that had nothing to do, which is the worst property a memory system -can have: **the failure looks exactly like success.** - -→ On a pi-devbox container, confirm the feeder is actually alive after a flip rather than assuming: - -```sh -mempalace-pi-session --reason manual-check; echo "exit=$?" # exit=0 and a filed count, not silence -cat ~/.pi/agent/mempalace-catchup.log # missing file = the entrypoint skipped -``` - -### 3.8 Verify the flip actually took — ✅ done 2026-08-14 on EMB-7KJ4VR4G - -§3.6 verifies the *endpoint* before you flip. §3.7 tells you *how* to flip. Neither verifies the thing -you actually care about afterwards: **that the agent's palace tools are now talking to synlig.** That -gap is why the first flip was "done" for an hour before anyone could say whether it had worked. - -There are **three separate claims** here and they fail independently. Check them in order; each one is -cheap and rules out a different fault. - -**(a) Did the container receive the variable?** In a shell *inside* the container: - -```sh -env | grep MEMPALACE_REMOTE_URL # -> https://mempalace.jordbo.se/mcp -``` - -⚠ Do **not** run a bare `env | grep MEMPALACE` — that prints the bearer token into your scrollback. -If the variable is absent, the cause is almost always §3.7's edit not having been applied: `env_file` -is read at container **create** time and baked into the container config, so **`docker compose up -d` -is required; `docker compose restart` silently reuses the old config.** Evidence from 2026-08-14 — -running container config-hash `3a55e09ac19e0118` vs compose-computed `8a7e0cd4a677a9c1`; a differing -hash is what makes `up -d` recreate. `--force-recreate` is not needed. -**Never `docker compose down -v`** to "pick up" a change: on pi-devbox that destroys seven named -volumes, including `devbox-pi-config` (pi's config **and every session transcript**), `devbox-uv`, and -`devbox-chroma-cache` (a large embedding-model re-download). - -**(b) Is the server reachable and the token accepted?** Still inside the container — this proves the -network path and the credential, independently of any agent: - -```sh -curl -s -X POST "$MEMPALACE_REMOTE_URL" \ - -H "Authorization: Bearer $MEMPALACE_REMOTE_TOKEN" \ - -H 'Content-Type: application/json' \ - -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | head -c 200 -``` - -`401` = token problem (compare `md5sum` of the client value against synlig's token file — trailing -whitespace from an editor is the classic cause). Connection failure = DNS/tunnel, not auth. - -**(c) Did the agent's bridge actually switch?** This is the claim that matters and the one that cannot -be checked from bash — there is no log file. Ask the agent running in the container for -`mempalace_status` and read the palace path it reports: - -``` -sqlite_integrity.palace = /home/ecsjper/.mempalace/palace # ← proves remote -``` - -**This is the cheapest and strongest discriminator: that path cannot exist inside the container**, -whose user is `developer` with `HOME=/home/developer` and whose local palace is -`/home/developer/.mempalace/palace`. One call, no token in scrollback, no writes. If instead it -reports the local path, the bridge fell back: `createClient()` reads `MEMPALACE_REMOTE_URL` **once at -load**, so re-check (a) and restart the agent, not just the container. - -#### What does *not* verify the flip — read this before inventing your own check - -- **❌ A drawer count.** Central was seeded *from* the client's own palace, so both report ~14,777. - Counts cannot tell the two apart. Worse, `mempalace status` counts **chunk rows, not logical - drawers** (three drawers plus a 2-chunk diary presented as +9), so a delta does not even mean what - it looks like. Never reason about palace identity or contents from a count. -- **❌ Writing a drawer through the palace tools and reading it back through the same tools.** This - succeeds *identically whether or not the flip worked* — both palaces are healthy and were seeded - from the same source, so a write-then-read round-trips either way. It only becomes evidence if the - drawer is read back over a **different transport** (the `curl` in (b), by drawer id) or is proven - **absent** from the local sqlite. This is the same false-positive class as the CLI below, one layer - up, and it is an easy trap to fall into precisely because it feels like an end-to-end test. -- **❌ The `mempalace` CLI, in any form.** The CLI has **no remote support whatsoever** — its only - selector is `--palace ` — so it reads and writes the LOCAL on-disk archive via - `config.json`. After a flip that archive is dead, yet `mempalace status` / `mempalace search` will - cheerfully report ~14,777 drawers and look exactly like success. It is a false-positive machine. - **Corollary that bites later:** memories filed with the CLI after a flip land in the dead archive, - not in central, and a transcript backfill must therefore be mined **on synlig**, where the CLI's - local palace *is* the central one. - -#### Expected failure behaviour, so you can recognise it - -The bridge is **fail-closed, not fail-local.** If synlig is unreachable, DNS fails, or the token is -rejected, the extension retries a bounded number of times, prints `mempalace-mcp unavailable after -retries; continuing without palace tools`, and **does not register the palace tools at all**. It does -not silently write to the local palace; there is no dual-write and no local mirror, and in remote mode -no local `mempalace-mcp` process is spawned (verified: zero mempalace processes in the flipped -container). So **"the agent has no `mempalace_*` tools" is the expected symptom of a server, token, or -DNS fault** — not of a broken container. Diagnose with (b). - -#### Rollback — ~30 seconds, loses nothing - -Comment out `MEMPALACE_REMOTE_URL` in `.env`, then `docker compose up -d`. The bridge falls back to the -local stdio palace, which is intact. Note the local archive is **frozen, not empty**: it stops at the -moment of the flip, so anything the agent filed into central since then will not be there. - ---- - -## 4. Still open - -- **Feeding without SSH — the upstream ask that would close §1.3's gap.** Have the feeder send *content* - over MCP (`add_drawer` / `diary_write`, which it already calls) instead of asking the server to mine a - path it must first rsync there. HTTPS would then be genuinely sufficient and mining would work from any - network. Until then, mining is limited to devices that can reach synlig's SSH. -- **Per-device tokens** — Phase 4. Until then `origin_device` is advisory (§1.1). -- **§7.6 diary dedup** must be settled *before* the **next** §4.4 join; replay duplicates every entry. - ⚠ **2026-08-14 — the first join did not resolve this, it SIDESTEPPED it.** The seed was a file-level - copy of one palace, which replays no diaries and therefore cannot duplicate them. The success of - that join is *not* evidence the replay path is safe — it never exercised it. §7.6 remains a hard - blocker for the second machine, which is the one that will actually need merge semantics. -- **The transcript feeder is inert on every deployed image**, so nothing is being mined automatically - anywhere in the fleet (§3.7). Regaining it needs a **new tagged pi-devbox release** — its CI - publishes only on `v*` tags and the latest tag *is* the currently deployed image — built on - pi-devbox ≥ `7c00dd6` **and** mempalace-toolkit ≥ `29e660e`. Both are required: the first restores - the container-start catch-up, the second is where extension-side feeding was implemented at all. -- **Whether *native* pi on a flipped machine was also flipped** is a per-machine question. Answered for - **EMB-7KJ4VR4G (2026-08-14): native pi is not installed there *yet*** — no `pi`/`mempalace` on the host - PATH, no `~/.config/pi/`, no `~/.pi/agent/extensions/`; it is a replacement machine that has not had - native pi set up. So nothing further needs flipping there **today** — but this is a deferred hazard, - not a closed one: a native install resolves its palace to `~/.mempalace`, which on that host is the - bind-mounted **frozen archive**, so it would start writing there and split the machine's memory from - central silently. **Flip native pi at install time, not after.** **Still open for every other host** - running native pi beside a flipped container. Verify with the §3.8(c) path check, per machine. - Note native pi has **no `.env` to edit** — pi loads no dotenv file and has no `env` block in - `settings.json`, so the variables must come from the shell that launches it; recipe in - [`extensions/pi/README.md`](../extensions/pi/README.md#transport-local-vs-external) § Transport. On - EMB-7KJ4VR4G specifically, that tree's `config.json` also points at - `palace_path=/home/developer/.mempalace/palace` — a *container* path that does not exist on macOS — so - a native install must not inherit it unexamined. -- **§7.2**: never run `mempalace sync` against the shared palace. Doubly true now that the pi/opencode - feeders stage *inside* the palace root, which puts staged sources in scope for a sync of the palace dir. -- **nyvaken's public FQDN and the web hotel's CNAME rules** — unverified (§3.4). +# (moved) Phase 1 exposure runbook + +This file used to contain the Phase 1 exposure record for one specific +deployment — the primary host and tunnel host by name, the DNS registrar step, +the Pangolin/newt resource wiring, the shared-token handling, per-machine flip +dates, and a palace lineage naming three work machines. + +**That content now lives in a private repository**, for the same reason +[`synlig-primary-runbook.md`](synlig-primary-runbook.md) does: a host inventory +is operator data for one deployment, not part of the toolkit. This repository is +public and keeps only host-agnostic *mechanism*. + +What lives where: + +| Content | Home | +|---|---| +| Why a palace needs a special backup, and how to restore one | [`backup-and-recovery.md`](backup-and-recovery.md) (here) | +| The coordination log: semantics, delivery, trust model, landmines | [`rfc-003-coordination-log.md`](rfc-003-coordination-log.md) (here) | +| What the stores are for, and how a fleet shares one palace | [`fleet-memory.md`](fleet-memory.md) (here) | +| Unit/timer/plist templates | [`contrib/`](../contrib/) (here) | +| Which host is primary and which fronts the tunnel, at which addresses, as which user | private fleet repository | +| Registrar/DNS records, tunnel resource config, token custody | private fleet repository | +| Per-machine flip dates, seeding history, palace lineage | private fleet repository | + +## The mechanism this file also carried — not yet extracted + +Unlike the primary-host runbook, this file was **mixed**: several sections were +reusable mechanism that would be true of anyone's palace, and those are not +published anywhere else yet. Named here so the debt is visible rather than lost: + +- **The bind trap, in full.** Why binding a palace to a public interface is not + the same as exposing it, and the `Host`/`Origin` pin that makes an MCP + endpoint refuse requests that arrive with the wrong hostname — the single + most surprising failure in the whole exposure path. +- **Why not per-device users at the proxy.** The reasoning behind one shared + fleet token instead of per-device proxy credentials, and the consequence + documented in [`rfc-003-coordination-log.md`](rfc-003-coordination-log.md) §6: + the deployment authenticates the *fleet*, not the *agent*. +- **The one place per-device identity does exist**, and why that is the feeder + path rather than the HTTP path. +- **Flipping a client: three variables, or none.** The all-or-nothing shape of + pointing a machine at a remote palace, and how a half-flipped client fails. + +Until that extraction happens, the mechanism is readable only in the private +record. If you are standing up your own palace over HTTP, the two pieces you +must not skip are the `Host`/`Origin` pin and the fact that a client is flipped +by environment variables that travel as a set. diff --git a/docs/synlig-primary-runbook.md b/docs/synlig-primary-runbook.md index 333312a..a288f35 100644 --- a/docs/synlig-primary-runbook.md +++ b/docs/synlig-primary-runbook.md @@ -11,7 +11,7 @@ What lives where: | Content | Home | |---|---| -| How to expose a palace over HTTP, and the Host/Origin pin | `docs/phase-1-exposure-runbook.md` (here) | +| How this deployment was exposed over HTTP (tunnel, DNS, token custody) | private fleet repository — see [`phase-1-exposure-runbook.md`](phase-1-exposure-runbook.md), also moved | | Why a palace needs a special backup, and how to restore one | `docs/backup-and-recovery.md` (here) | | Unit/timer/plist templates | `contrib/` (here) | | Which machine is primary, its addresses, users, tunnels, offsite target | private fleet repository | diff --git a/extensions/pi/README.md b/extensions/pi/README.md index a1ce2d7..2685753 100644 --- a/extensions/pi/README.md +++ b/extensions/pi/README.md @@ -187,8 +187,8 @@ chosen at load time: transport is chosen once at extension load. Confirm the result the same way as a container flip: ask the agent for `mempalace_status` and check the reported palace path is the **remote** host's, not your own - `$HOME/.mempalace/palace` — see - [`docs/phase-1-exposure-runbook.md`](../../docs/phase-1-exposure-runbook.md) §3.8. + `$HOME/.mempalace/palace`. That one check is the whole verification: a + half-flipped client reports a local path while looking healthy. Serve such an endpoint with `mempalace serve --host 172.17.0.1 --port 8765` (the `pi-devbox` / `opencode-devbox` repos ship a @@ -207,8 +207,9 @@ chosen at load time: token auto-minting is gated on the bind being non-loopback, so it starts with **no authentication at all**, no warning. Bind the docker0 gateway (`172.17.0.1`): reachable from the host and its containers, not from the LAN. - See - [`docs/phase-1-exposure-runbook.md`](../../docs/phase-1-exposure-runbook.md). + Binding an interface is not the same as exposing a palace — an MCP endpoint + also pins `Host`/`Origin`, so a request arriving under the wrong hostname is + refused even when the port is open. Implementation note: the HTTP client (`RemoteMcpClient`) is **vendored** from [`pi-extensions`](https://gitea.jordbo.se/joakimp/pi-extensions)' @@ -234,9 +235,9 @@ local palace, so a remote outage can never scatter memories into a local copy nobody will look at again. The practical corollary, worth knowing before you debug the wrong layer: **"the agent has no `mempalace_*` tools" is the expected symptom of a server, token, or DNS fault**, not of a broken install. -Diagnose it with a direct `curl` to `MEMPALACE_REMOTE_URL` — see -[`docs/phase-1-exposure-runbook.md`](../../docs/phase-1-exposure-runbook.md) -§3.8. The design rationale for de-registering rather than degrading is in +Diagnose it with a direct `curl` to `MEMPALACE_REMOTE_URL`, and confirm the flip +with `mempalace_status` — the reported palace path must be the remote host's. +The design rationale for de-registering rather than degrading is in [`docs/rfc-001-global-palace.md`](../../docs/rfc-001-global-palace.md) §2 and §4.1. ## Identity