mailbox: join the owed set on hlc, not seq, before a second replica exists

deriveOwed decides "is this ask still owed?" by asking whether one of my own terminal replies is LATER than the ask. It compared `seq` — this database's arrival rowid. On a single hub that is global order, so it was correct; the comment above it already said hlc was the durable key "once mesh_peers reports actual peers". Making the switch now, while one replica means the two orderings agree, costs nothing; making it later means changing the rule while two machines already disagree about order. The bug being pre-empted is specific: with a second replica the same event gets a different `seq` in each database, because arrival order is not authorship order. A reply authored after its ask can arrive first and take the lower seq; the join then concludes "no later reply exists" and an already-answered ask reappears as owed — permanently, on that machine. isStrictlyAfter() prefers `hlc` when both events carry one and falls back to `seq` otherwise (a server predating the field, or an un-backfilled row). hlc is rendered fixed-width, <unix_ms:13 digits>-<counter:6 hex>-<replica_id>, so a plain string comparison IS the causal comparison, with the replica id as final tiebreak. Verified before writing the code that the field is actually on the wire — event_list returns it per event (logstream.py:593) — because a fallback that never fires would have made this a no-op dressed as a fix. `created_at` stays rejected, and the reason is now written down where the decision is: it is server-generated at second precision, so ties are routine, and a tie can suppress an UNANSWERED ask. Both remaining failure modes are on the noisy-but-visible side — an answered item resurfacing is annoying, an unanswered ask going silent defeats the mailbox. Tested: 18 cases against the extracted comparator — hlc later/earlier/equal, same-ms counter ties in hex (0x10 vs 0x9, which is where a non-padded format would break), cross-replica tiebreak, both mesh reorderings, every seq-fallback path, and degenerate input (null seq, numeric hlc, empty strings, no keys) which must never claim "answered". All pass. Real-data check on the positive-control pair: seq 26/27 carry hlc ...3071857/...3574085, so the orderings agree today and the switch is a no-op now and correct later. The cursor keeps the opposite ordering ON PURPOSE — since_event_id is local-arrival ordered so a tail consumer still sees late-arriving remote ops whose hlc is older (hlc.py:19-21). RFC 003 §9.3 now says so explicitly, because that asymmetry looks like a bug worth "fixing" and is not.
docs: take the Phase 1 exposure record private, leave a moved-note
2026-08-26 23:30:09 +02:00 · 2026-08-26 23:30:09 +02:00
8 changed files with 119 additions and 495 deletions
@@ -68,7 +68,7 @@ Start here if you are deciding **what to put where**, or running MemPalace on mo
 | [`docs/rfc-003-coordination-log.md`](docs/rfc-003-coordination-log.md) | The inter-agent / inter-device coordination log (`logstream`): storage, append and query semantics, delivery and latency, the trust model, and its landmines. |
 | [`docs/rfc-001-global-palace.md`](docs/rfc-001-global-palace.md) | Why and how a palace is centralised, what should *not* be global, and the Phase-0 landmines (including: never run `mempalace sync` against a shared palace). |
 | [`docs/rfc-002-joiner.md`](docs/rfc-002-joiner.md) | Replaying a second palace into a shared primary, and what dedupes what. |
-| [`docs/phase-1-exposure-runbook.md`](docs/phase-1-exposure-runbook.md) | Exposing a palace over HTTP: the Host/Origin pin, tokens, TLS. |
+| [`docs/phase-1-exposure-runbook.md`](docs/phase-1-exposure-runbook.md) | *(moved)* The exposure record for one deployment is now private operator data. The stub names the reusable mechanism it also carried — the `Host`/`Origin` pin above all — which is not yet published elsewhere. |
 | [`docs/backup-and-recovery.md`](docs/backup-and-recovery.md) | Why a palace needs its own backup procedure, and how to restore one. |
 | [`extensions/pi/README.md`](extensions/pi/README.md) | The pi-side client: provenance stamping at the edge, and the auto-delivered mailbox. |
@@ -596,9 +596,12 @@ the palace host and asks the server to mine its own local copy. Requires
 `MEMPALACE_PI_REMOTE_PATH`, and `MEMPALACE_PI_DEVICE` in `--help` for the
 rest. `MEMPALACE_PI_DEVICE` also defaults `--agent` to `pi@<device>` so each
 drawer's `added_by` records which harness and which machine produced it —
-mempalace itself stores neither. Deploying that primary — newt, DNS, and why the
+mempalace itself stores neither. Deploying a primary — tunnel, DNS, token
-auth is a shared bearer token rather than per-device proxy users — is
+custody — is deployment-specific operator data and lives in a private fleet
-[`docs/phase-1-exposure-runbook.md`](docs/phase-1-exposure-runbook.md).
+repository; [`docs/phase-1-exposure-runbook.md`](docs/phase-1-exposure-runbook.md)
 records what moved and why. The *consequence* of choosing one shared fleet token
 over per-device proxy users is public, in
 [`docs/rfc-003-coordination-log.md`](docs/rfc-003-coordination-log.md) §6.
 ---
@@ -48,9 +48,8 @@ The pi variants are drop-in copies of the opencode variants with script name and
 The odd one out in this directory: every other template *feeds* a palace on a schedule, this one
 **serves** a palace over HTTP so several machines can share it (RFC-001). **This unit currently runs
-the fleet primary on `synlig`** — serving since 2026-08-12, seeded 2026-08-14, and the palace behind
+a fleet primary** — whose palace may be the only copy. Read [`docs/rfc-001-global-palace.md`](../docs/rfc-001-global-palace.md)
-it is the only copy. Read [`docs/rfc-001-global-palace.md`](../docs/rfc-001-global-palace.md) and
+before installing a second one.
 [`docs/phase-1-exposure-runbook.md`](../docs/phase-1-exposure-runbook.md) before installing a second one.
 It is a **user** unit (`systemctl --user`), so it dies with your login session unless lingering is
 enabled — that is the one `sudo` this recipe needs:
@@ -212,7 +212,7 @@ Small, and the whole value rests on them:
 - `docs/rfc-003-coordination-log.md` — the coordination log: storage, semantics, security model, landmines.
 - `docs/rfc-001-global-palace.md` — how and why a palace is centralised; §5 what should not be global; §7 the landmines, including the `sync` hazard.
- `docs/phase-1-exposure-runbook.md` — exposing a palace over HTTP.
+- `docs/phase-1-exposure-runbook.md` — (moved) the site-specific exposure record is now private; the stub names the mechanism still awaiting extraction.
 - `docs/backup-and-recovery.md` — why a palace needs its own backup procedure.
 - `extensions/pi/README.md` — the pi-side client: provenance stamping and the auto-delivered mailbox.
 - `~/.agents/skills/mempalace/SKILL.md` — the protocol the agents themselves follow.
@@ -1,463 +1,47 @@
-# Phase 1 exposure — newt on synlig, DNS, and the client auth model
+# (moved) Phase 1 exposure runbook
-
+
-Companion to [`rfc-001-global-palace.md`](./rfc-001-global-palace.md) (design + decisions) and
+This file used to contain the Phase 1 exposure record for one specific
-[`synlig-primary-runbook.md`](./synlig-primary-runbook.md) (what is already installed on the primary).
+deployment — the primary host and tunnel host by name, the DNS registrar step,
-This doc covers only the step the other two leave open: **making the primary reachable** — runbook §4
+the Pangolin/newt resource wiring, the shared-token handling, per-machine flip
-items 2 and 5.
+dates, and a palace lineage naming three work machines.
-
+
-> **Status 2026-08-14 — DONE. Exposed, seeded, and one client flipped.**
+**That content now lives in a private repository**, for the same reason
-> `https://mempalace.jordbo.se/mcp` has been serving since 2026-08-12 (§3.3–§3.6 are all ✅ below).
+[`synlig-primary-runbook.md`](synlig-primary-runbook.md) does: a host inventory
-> The palace was seeded 2026-08-14 15:07 from EMB-7KJ4VR4G (14,777 → 14,803 drawers) — whose palace was
+is operator data for one deployment, not part of the toolkit. This repository is
-> itself carried over from the previous work computer **EMB-X1JY06WJ** around 2026-07-06, so the
+public and keeps only host-agnostic *mechanism*.
-> primary's lineage is EMB-X1JY06WJ → EMB-7KJ4VR4G → synlig by two file-level copies (rfc-001 §4.4
+
-> Deviation) — and that machine's pi-devbox container is flipped and **verified end-to-end — see §3.8**, which is the
+What lives where:
-> verification procedure that did not exist when the first flip was performed.
+
-> Still outstanding: the transcript feeder is **inert on every deployed image** (§3.7), and §7.6 is
+| Content | Home |
-> still a hard blocker for the *second* machine to join (§4).
+|---|---|
-
+| Why a palace needs a special backup, and how to restore one | [`backup-and-recovery.md`](backup-and-recovery.md) (here) |
-Original status, kept for the record — it contradicted this file's own ✅ section markers for two days,
+| The coordination log: semantics, delivery, trust model, landmines | [`rfc-003-coordination-log.md`](rfc-003-coordination-log.md) (here) |
-which is the failure mode a file-top status block invites:
+| What the stores are for, and how a fleet shares one palace | [`fleet-memory.md`](fleet-memory.md) (here) |
-
+| Unit/timer/plist templates | [`contrib/`](../contrib/) (here) |
-**Status 2026-08-12 — Pangolin updated on nyvaken (done, yours). newt not yet installed on synlig.
+| Which host is primary and which fronts the tunnel, at which addresses, as which user | private fleet repository |
-Nothing exposed. No client `.env` flipped.**
+| Registrar/DNS records, tunnel resource config, token custody | private fleet repository |
-
+| Per-machine flip dates, seeding history, palace lineage | private fleet repository |
-Read this before touching Pangolin: three of the four questions this step raises were **already decided**
+
-in RFC §6.2 on 2026-08-09, and re-deciding them differently is how the fleet ends up in two states.
+## The mechanism this file also carried — not yet extracted
-
+
---
+Unlike the primary-host runbook, this file was **mixed**: several sections were
-
+reusable mechanism that would be true of anyone's palace, and those are not
-## 1. The four questions, answered
+published anywhere else yet. Named here so the debt is visible rather than lost:
-
+
-| Question | Answer | Where it was decided |
+- **The bind trap, in full.** Why binding a palace to a public interface is not
-| --- | --- | --- |
+  the same as exposing it, and the `Host`/`Origin` pin that makes an MCP
-| Which port? | **8765**, path **`/mcp`** (liveness: `/healthz`) | `cli.py:2141` default; runbook §2.4 |
+  endpoint refuse requests that arrive with the wrong hostname — the single
-| What does newt target? | **`172.17.0.1:8765`** (docker0), **never** `127.0.0.1` | RFC §6.2 Transport; runbook §2.4 |
+  most surprising failure in the whole exposure path.
-| Open, or authenticated? | **Authenticated. The primary is never an open public resource.** | RFC §6.2 Network posture |
+- **Why not per-device users at the proxy.** The reasoning behind one shared
-| Per-device credentials? | **No — Phase 1 ships the single shared bearer token.** Per-device tokens are Phase 4. | RFC §6.2 Authentication |
+  fleet token instead of per-device proxy credentials, and the consequence
-
+  documented in [`rfc-003-coordination-log.md`](rfc-003-coordination-log.md) §6:
-### 1.1 Why not per-device users at the proxy
+  the deployment authenticates the *fleet*, not the *agent*.
-
+- **The one place per-device identity does exist**, and why that is the feeder
-The instinct — "create a Pangolin user per container, put the credentials in each `.env`, keep the
+  path rather than the HTTP path.
-usernames distinct" — is the right *goal* (revocation, attribution) reached through the wrong *layer*,
+- **Flipping a client: three variables, or none.** The all-or-nothing shape of
-twice over:
+  pointing a machine at a remote palace, and how a half-flipped client fails.
-
+
-1. **mempalace validates exactly one token.** `hmac.compare_digest(provided, f"Bearer {srv.auth_token}")`
+Until that extraction happens, the mechanism is readable only in the private
-   (`mcp_server.py:5292-5295`) — there is no user table and no second credential. Per-device HTTP identity
+record. If you are standing up your own palace over HTTP, the two pieces you
-   is not a configuration you can express today; it is Phase 4 work (a server-side
+must not skip are the `Host`/`Origin` pin and the fact that a client is flipped
-   `token → {device_id, scopes}` registry). RFC §6.2 chose the shared token for Phase 1 deliberately:
+by environment variables that travel as a set.
   *"iterate more feature rich but more complex solutions over time."*
 2. **Pangolin's HTTP auth is browser-shaped; the clients are not.** SSO login, resource PIN and resource
   password all assume something that can follow a redirect, render a form and hold a session cookie.
   Every MemPalace client here is a headless JSON-RPC `POST` with an `Authorization` header — pi's
   extension, opencode's `type:remote` MCP entry, and `mempalace-pi-session --mode remote`'s
   `urllib.request.urlopen`. Point those at a user-authenticated resource and they receive a login page
   where JSON should be. Enabling that protection breaks precisely the clients it is meant to protect.
 So: **Pangolin terminates TLS and nothing more** (RFC §6.2 Transport, decided 2026-08-09). The bearer token
 is the authentication. This is not "unprotected" — an unauthenticated request to `/mcp` gets a 401 from
 mempalace itself, verified A4/A5 in runbook §2.4.
 **Consequence to accept consciously** (RFC §6.2, §7.3.2): until Phase 4 the primary **cannot tell devices
 apart**. `origin_device` is client-asserted and advisory — nothing load-bearing may depend on it, and
 revoking one laptop means rotating the token everywhere.
 ### 1.2 The one place per-device identity *does* exist today
 Remote mode is not only HTTP. `mempalace_mine` expands its source path in the **server** process, so a
 client's staged transcripts must physically exist on the primary. The feeder therefore ships them over
 SSH into a **per-device inbox** before asking the server to mine its own copy:
 ```sh
 rsync -a --update -e "$ssh_cmd" "$STAGE/" "${SSH_TARGET%/}/$DEVICE/"   # bin/mempalace-pi-session:677-680
 ```
 That SSH key **is** per-device identity, and it is individually revocable (one line out of
 `authorized_keys`) years before Phase 4 lands. It costs nothing extra, because the mining path needs SSH
 regardless.
 Two implications people miss:
 - **`mempalace.jordbo.se` alone does not enable mining.** HTTPS covers the read/write tool surface
  (`search`, `add_drawer`, `diary_write`, `kg_*`) — genuinely useful on its own, and the reason to do this
  at all. But `--mode remote` also needs `MEMPALACE_PI_SSH_TARGET` reachable. Budget for both paths.
 - **`DEVICE` defaults to `$(hostname)`** (`bin/mempalace-pi-session:163`). In a container that is the
  container hostname: either random per recreate (inboxes proliferate; each recreate re-mines into a fresh
  empty inbox) or identical across sibling devboxes (two containers writing one inbox). **Set
  `MEMPALACE_PI_DEVICE` explicitly per container.** It is a label, not a secret, so put it somewhere
  reviewable — a committed compose file — where duplicates are visible. That, not username hygiene in
  `.env`, is the discipline this design actually asks of you.
 ### 1.3 "Then why Pangolin at all, if the feeder uses SSH?"
 Because they are not alternatives — they carry different traffic, and neither substitutes for the other.
 | | Pangolin/newt (HTTPS) | SSH + rsync |
 | --- | --- | --- |
 | Carries | the **MCP tool surface**: `search`, `add_drawer`, `diary_write`, `kg_*` — every live tool call | **transcript files only**, once per session or cron run |
 | Used by | the pi extension, opencode `type:remote`, any MCP client | the feeder, internally (`bin/mempalace-pi-session:677-680`) |
 | Needed because | clients need one stable URL, reachable from wherever they are | `mempalace_mine` expands its source path **server-side**, so the server can only mine files on its own disk |
 HTTPS alone is a palace you can query but cannot feed. SSH alone is files shipped with no live query API.
 The rsync is not a transport preference; it is a workaround for *where `mine` resolves paths*.
 **Could SSH replace Pangolin?** Partly, and it is worth being honest about it:
 `ssh -L 8765:172.17.0.1:8765 synlig` yields a working local MCP endpoint with no public HTTPS at all.
 Three reasons this runbook does not do that:
 1. **Direction.** synlig dials *out* through newt. That we reached for a dial-out tunnel rather than a
   port-forward is itself the evidence that inbound was not available — a corporate host does not accept
   connections from a phone on a foreign network.
 2. **MCP clients want a durable URL**, not a per-session forwarded port. opencode `type:remote` takes a
   URL; a forward that drops takes the tools down mid-session.
 3. The forward must be up on **every device before every session**. Pangolin is up once.
 **The weak point, stated plainly.** The rsync runs *client → synlig*, so it needs synlig's SSH reachable
 **from the client**. Were that already true everywhere, no tunnel would be needed for MCP either. So the
 honest expectation after Phase 1 is: **query and write from anywhere, mine only from devices that can
 reach synlig's SSH** (corporate network / VPN / LAN). See §4 for the change that would remove that limit.
 ---
 ## 2. The bind trap, in full
 RFC §6.2 and runbook §2.4 already say **do not bind loopback behind the tunnel**, because
 `enforce_host_pin = _http_is_loopback(host)` (`mcp_server.py:5367`) makes a loopback bind reject the
 proxy's forwarded `Host:` with a **403** that reads exactly like a Pangolin misconfiguration.
 **Additional finding, 2026-08-12 — the same reflex also silently removes authentication.** Token
 resolution in `cmd_serve` (`cli.py:1447-1450`) is:
 ```python
 loopback = _server_is_loopback(host)
 if not token and not loopback and not args.allow_insecure:
    token, token_created = _load_or_create_server_token(palace_path)
 ```
 Auto-minting is gated on the bind being **non-loopback**. A loopback bind therefore starts with **no token
 at all** — no error, no warning, `--allow-insecure` not required — because the server has concluded it is
 only reachable locally, while the tunnel is serving it to the internet. Bind loopback behind newt and you
 get a 403 wall *and*, the moment anything relaxes the Host pin, an unauthenticated palace.
 Both failure modes have the same cure, already implemented in
 `contrib/systemd/mempalace-serve.service`: **bind `172.17.0.1`**. Non-loopback, so the Host pin relaxes and
 the token is mandatory; docker0-only, so newt reaches it and the LAN does not.
 > Belt and braces: set `MEMPALACE_MCP_HTTP_TOKEN` explicitly in the unit rather than relying on
 > auto-minting. Then no future bind change can quietly drop authentication.
 ---
 ## 3. Steps
 Ordered so nothing is reachable before it is authenticated.
 ### 3.1 Start the primary (runbook §4.3 — one `sudo`, unit already staged)
 ```sh
 sudo loginctl enable-linger ecsjper
 cd ~/.config/systemd/user && mv mempalace-serve.service.staged mempalace-serve.service
 systemctl --user daemon-reload && systemctl --user enable --now mempalace-serve
 curl -s 172.17.0.1:8765/healthz     # expect ok
 curl -s 127.0.0.1:8765/healthz      # expect NOTHING — connection refused, exit 7 (see below)
 ss -ltnp | grep 8765                # expect 172.17.0.1:8765 only
 ```
 ⚠ **The loopback probe returns empty, not 403** — corrected 2026-08-12 against the real run. Nothing is
 listening on `127.0.0.1`, so the connection is refused at TCP level and `curl -s` prints nothing; check it
 with `-w '%{http_code}'` → `000` and `$?` → `7`. The 403 belongs to a *different* configuration: server
 bound **to loopback**, receiving a proxy-forwarded foreign `Host:` (§2, verified 2026-08-10). With a
 docker0-only bind you cannot get 403 from loopback, because you never get far enough to send a header.
 Refusal is the stronger signal of the two: it proves the loopback and LAN surface is not listening at all.
 If it *hangs* instead, or `ss` shows `0.0.0.0:8765`, stop — that is not this configuration.
 ### 3.2 Collect the shared token
 ```sh
 cat ~/.mempalace/server/f5d849287f6d73f0141b29d7/token
 ```
 Directory name is `sha256(realpath(palace))[:24]` — it changes if the palace path ever moves. Store via the
 `.env.age` flow, 0600 (RFC §6.2).
 ### 3.3 newt on synlig — ✅ done 2026-08-12 (installed, connected to Pangolin)
 synlig runs Docker (Gitea Actions runner + digikam) but **no tunnel client** — runbook §4.2. Pangolin on
 nyvaken cannot dial in; synlig must dial out. Add a `newt` container with the credentials Pangolin issues
 for a new site.
 Because newt runs in Docker on this box, the docker0 bind is already correct for it: from inside the
 container the primary is `172.17.0.1:8765`. Verify from *inside* newt's network namespace, not from the
 host, before touching DNS.
 > synlig has 7.8 GiB shared with a CI runner (runbook §1). newt is small, but do not colocate anything
 > else here casually.
 **Confirm next, now that newt is up.** "Connected to Pangolin" proves newt reached *nyvaken* — a different
 claim from newt reaching *the palace*, and the two fail independently:
 ```sh
 # from inside newt's namespace, not from the host
 docker exec <newt-container> wget -qO- http://172.17.0.1:8765/healthz    # expect ok
 ```
 If that hangs or refuses while the Pangolin dashboard shows the site online, the tunnel is fine and the
 *target* is wrong — look at the resource's upstream address (§3.5), not at newt. Note this check needs
 §3.1 done first: if `mempalace-serve` is not running yet, it fails for that reason alone.
 ### 3.4 DNS at the web hotel — ✅ done 2026-08-12
 One CNAME: `mempalace` → **the same target your existing Pangolin resources use** (nyvaken's public
 hostname). RFC §6.2 costed this as *"one DNS record per service on the web hotel is the whole setup cost."*
 Done: `mempalace.jordbo.se` resolves and terminates TLS at Pangolin on nyvaken — the §6.2 estimate held.
 ⚠️ Not verified from here: nyvaken's public FQDN, and whether your web hotel permits a CNAME at that label
 (some require an A record, or forbid CNAME where other records exist). Confirm before assuming a 5-minute job.
 ### 3.5 Pangolin resource
 - Target: newt site → **`http://172.17.0.1:8765`** — path `/mcp` (plus `/healthz` for the external probe).
 - ⚠️ **The scheme is `http`, not `https`.** The primary runs `serve --host 172.17.0.1 --port 8765` with no
  cert (`contrib/systemd/mempalace-serve.service`): TLS terminates **at Pangolin**, which is the entire
  point of the §6.2 decision. Point the resource at `https://172.17.0.1:8765` and Pangolin attempts a TLS
  handshake against a plaintext listener — you get a 502/Bad Gateway from outside while the server itself
  looks perfectly healthy on `curl 172.17.0.1:8765/healthz`. (Got this wrong on the first attempt
  2026-08-12, because this line used to omit the scheme.)
 - **Auth: none at the Pangolin layer** (§1.1). TLS termination only.
 - ⚠️ **If resource auth is left on, the signature is a `302`, not a 401 or 403** — hit for real 2026-08-12:
  ```
  HTTP/2 302
  location: https://pangolin.jordbo.se/auth/resource/<uuid>?redirect=https%3A%2F%2Fmempalace.jordbo.se%2Fhealthz
  content-length: 0
  ```
  This is §1.1's "browser-shaped auth" arriving as a concrete symptom: Pangolin sends the login redirect
  **before** proxying, so the palace never sees the request and its journal stays silent. `curl -s` shows
  an empty body and an MCP client sees non-JSON. Diagnose with `-D-` or
  `-w '%{http_code} %{redirect_url}'` — a `location:` pointing at `/auth/resource/…` means the fix is in
  the Pangolin UI (switch the site's authentication off), not in the palace, the unit, or newt.
  **A 302 is unambiguously good news:** DNS, TLS and routing all worked — only the auth layer intervened.
 - Do **not** attach an `Origin`-injecting proxy or browser client: a *present* non-loopback `Origin` is a
  hard 403 with no override (runbook §2.4 B3).
 **The server's entire HTTP surface is two exact paths**, so path-scoped rules cover it completely
 (`mcp_server.py:5299-5318`, read 2026-08-12):
 | Method + path | Auth | Notes |
 | --- | --- | --- |
 | `GET /healthz` | none (Host/Origin gated only) | the liveness probe; works with no creds by design |
 | `POST /mcp` | `Authorization: Bearer <token>`, `hmac.compare_digest` on the exact string | the whole tool surface |
 | anything else | — | `send_error(404)` from the palace itself |
 Three consequences worth having in writing:
 - **Path-scoped Pangolin rules are not a compromise here, they are tighter than a host-wide proxy** and
  lose nothing — there is no third endpoint to forget.
 - **`/mcp` is matched exactly** (`if path != "/mcp"`), so a client URL with a trailing slash gets a 404
  from the palace. Configure clients as `https://mempalace.jordbo.se/mcp` — no trailing slash.
 - **There is no `GET /mcp`, no SSE, no session id, no `DELETE`.** This is plain JSON-RPC over POST, not MCP
  streamable-HTTP. A strict client that opens with a `GET` handshake will see 404; pi's extension,
  opencode `type:remote` and the feeder all POST directly and are fine.
 ### 3.6 Verify end-to-end before flipping any client — ✅ passed 2026-08-12
 ```sh
 curl -s https://mempalace.jordbo.se/healthz                      # ok        ✓ from devbox AND synlig
 curl -s -o /dev/null -w '%{http_code}\n' -X POST \
     https://mempalace.jordbo.se/mcp                             # 401       ✓ the token is doing its job
 TOKEN=$(cat ~/.mempalace/server/f5d849287f6d73f0141b29d7/token)  # on synlig
 curl -s -X POST https://mempalace.jordbo.se/mcp \
     -H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
     -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | head -c 300   # 36 tools
 ```
 No `initialize` and no `Accept: text/event-stream` needed — see the surface table above; a bare
 `tools/list` POST is a complete request. Run the 200 and the 401 from **two different networks** (a client
 box and the primary itself): passing from only one leaves split-horizon DNS untested.
 The 401 check matters as much as the 200: it is the only evidence that the thing you just published to the
 internet is not open. Then, and only then, Phase 1 client flip — **one machine first** (RFC §8), and
 remember opencode containers need the §4.1 sidecar merge before their `.env` takes effect.
 ### 3.7 Flipping a client: set three variables, or none
 ⚠ **`MEMPALACE_REMOTE_URL` on its own does not degrade to local feeding — it stops feeding.** `auto` mode
 switches to `remote` the moment the URL is set, and remote mode then refuses to run without an SSH target:
 ```sh
 auto) if [[ -n "$REMOTE_URL" ]]; then MODE="remote"; else MODE="local"; fi ;;   # :286
 ...
 command -v rsync >/dev/null 2>&1 || { echo "error: rsync not found ..."; exit 3; }   # :297
 if [[ -z "$SSH_TARGET" ]]; then
  echo "error: MEMPALACE_PI_SSH_TARGET unset (needed for --mode remote)" >&2; exit 1  # :298-300
 fi
 ```
 That exit happens **before anything is staged or filed**, and a cron-driven feeder will simply start
 failing — the loudest symptom is silence, which is the hardest kind to notice. Two safe orders:
 - **Both paths at once:** set `MEMPALACE_REMOTE_URL`, `MEMPALACE_REMOTE_TOKEN` **and**
  `MEMPALACE_PI_SSH_TARGET` (plus `MEMPALACE_PI_DEVICE`, §1.2) in the same edit.
 - **HTTPS first, mining later:** set the URL and token, and pin the feeder to `--mode local` until the SSH
  target exists. Tools then read/write the shared palace while transcripts keep landing in the local one.
 `MEMPALACE_REMOTE_URL` is the **full endpoint including `/mcp`, with no trailing slash** — the feeder POSTs
 to it verbatim (`urllib.request.Request(url, data=payload, …)`, `:718`), and §3.5 shows the server matches
 `/mcp` exactly. Matches the existing examples (`docker-compose.mempalace.yml:7`,
 `MEMPALACE_REMOTE_URL=http://<reachable-host>:8765/mcp`). For this fleet:
 `MEMPALACE_REMOTE_URL=https://mempalace.jordbo.se/mcp`
 Either way, run the feeder once by hand and read its exit code before trusting the timer. This is
 precisely the failure "one machine first" is meant to contain.
 **The same misconfiguration has two different symptoms depending on who invokes the feeder** — checked in
 the code 2026-08-12, and the quieter one is the trap:
 | Caller | Behaviour with `REMOTE_URL` set and `SSH_TARGET` unset |
 | --- | --- |
 | direct run, session-end hook, cron | `exit 1` with `error: MEMPALACE_PI_SSH_TARGET unset` (`:298-300`) |
 | **pi-devbox container start, images ≤ v1.7.0** | **silently skips — no error, no log file** (`entrypoint-user.sh:134`: `: # remote palace but no inbox configured — nothing we can ship to; skip quietly`) |
 | pi-devbox container start, images after `cbd7cf5` | prints `MemPalace catch-up skipped: remote palace with no transcript inbox` to the start output *and* to `mempalace-catchup.log`, naming both variables |
 The silent skip is fixed in pi-devbox (`cbd7cf5`), but the fix is in
 `entrypoint-user.sh`, which is `COPY`d in `Dockerfile.base` — so **every container running an image built
 before that base rebuild still skips silently.** That is the whole fleet today. Until the rebuild lands,
 assume silence and check by hand.
 The old skip happened *before* the subshell that writes `~/.pi/agent/mempalace-catchup.log`, so there was
 not even an empty log to notice. Someone asking "why is nothing from this container in the palace?" found
 no artifact at all. The skip itself is correct — there is genuinely nothing to ship to — but it was
 indistinguishable from a healthy run that had nothing to do, which is the worst property a memory system
 can have: **the failure looks exactly like success.**
 → On a pi-devbox container, confirm the feeder is actually alive after a flip rather than assuming:
 ```sh
 mempalace-pi-session --reason manual-check; echo "exit=$?"   # exit=0 and a filed count, not silence
 cat ~/.pi/agent/mempalace-catchup.log                        # missing file = the entrypoint skipped
 ```
 ### 3.8 Verify the flip actually took — ✅ done 2026-08-14 on EMB-7KJ4VR4G
 §3.6 verifies the *endpoint* before you flip. §3.7 tells you *how* to flip. Neither verifies the thing
 you actually care about afterwards: **that the agent's palace tools are now talking to synlig.** That
 gap is why the first flip was "done" for an hour before anyone could say whether it had worked.
 There are **three separate claims** here and they fail independently. Check them in order; each one is
 cheap and rules out a different fault.
 **(a) Did the container receive the variable?** In a shell *inside* the container:
 ```sh
 env | grep MEMPALACE_REMOTE_URL      # -> https://mempalace.jordbo.se/mcp
 ```
 ⚠ Do **not** run a bare `env | grep MEMPALACE` — that prints the bearer token into your scrollback.
 If the variable is absent, the cause is almost always §3.7's edit not having been applied: `env_file`
 is read at container **create** time and baked into the container config, so **`docker compose up -d`
 is required; `docker compose restart` silently reuses the old config.** Evidence from 2026-08-14 —
 running container config-hash `3a55e09ac19e0118` vs compose-computed `8a7e0cd4a677a9c1`; a differing
 hash is what makes `up -d` recreate. `--force-recreate` is not needed.
 **Never `docker compose down -v`** to "pick up" a change: on pi-devbox that destroys seven named
 volumes, including `devbox-pi-config` (pi's config **and every session transcript**), `devbox-uv`, and
 `devbox-chroma-cache` (a large embedding-model re-download).
 **(b) Is the server reachable and the token accepted?** Still inside the container — this proves the
 network path and the credential, independently of any agent:
 ```sh
 curl -s -X POST "$MEMPALACE_REMOTE_URL" \
  -H "Authorization: Bearer $MEMPALACE_REMOTE_TOKEN" \
  -H 'Content-Type: application/json' \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | head -c 200
 ```
 `401` = token problem (compare `md5sum` of the client value against synlig's token file — trailing
 whitespace from an editor is the classic cause). Connection failure = DNS/tunnel, not auth.
 **(c) Did the agent's bridge actually switch?** This is the claim that matters and the one that cannot
 be checked from bash — there is no log file. Ask the agent running in the container for
 `mempalace_status` and read the palace path it reports:
 ```
 sqlite_integrity.palace = /home/ecsjper/.mempalace/palace     # ← proves remote
 ```
 **This is the cheapest and strongest discriminator: that path cannot exist inside the container**,
 whose user is `developer` with `HOME=/home/developer` and whose local palace is
 `/home/developer/.mempalace/palace`. One call, no token in scrollback, no writes. If instead it
 reports the local path, the bridge fell back: `createClient()` reads `MEMPALACE_REMOTE_URL` **once at
 load**, so re-check (a) and restart the agent, not just the container.
 #### What does *not* verify the flip — read this before inventing your own check
 - **❌ A drawer count.** Central was seeded *from* the client's own palace, so both report ~14,777.
  Counts cannot tell the two apart. Worse, `mempalace status` counts **chunk rows, not logical
  drawers** (three drawers plus a 2-chunk diary presented as +9), so a delta does not even mean what
  it looks like. Never reason about palace identity or contents from a count.
 - **❌ Writing a drawer through the palace tools and reading it back through the same tools.** This
  succeeds *identically whether or not the flip worked* — both palaces are healthy and were seeded
  from the same source, so a write-then-read round-trips either way. It only becomes evidence if the
  drawer is read back over a **different transport** (the `curl` in (b), by drawer id) or is proven
  **absent** from the local sqlite. This is the same false-positive class as the CLI below, one layer
  up, and it is an easy trap to fall into precisely because it feels like an end-to-end test.
 - **❌ The `mempalace` CLI, in any form.** The CLI has **no remote support whatsoever** — its only
  selector is `--palace <path>` — so it reads and writes the LOCAL on-disk archive via
  `config.json`. After a flip that archive is dead, yet `mempalace status` / `mempalace search` will
  cheerfully report ~14,777 drawers and look exactly like success. It is a false-positive machine.
  **Corollary that bites later:** memories filed with the CLI after a flip land in the dead archive,
  not in central, and a transcript backfill must therefore be mined **on synlig**, where the CLI's
  local palace *is* the central one.
 #### Expected failure behaviour, so you can recognise it
 The bridge is **fail-closed, not fail-local.** If synlig is unreachable, DNS fails, or the token is
 rejected, the extension retries a bounded number of times, prints `mempalace-mcp unavailable after
 retries; continuing without palace tools`, and **does not register the palace tools at all**. It does
 not silently write to the local palace; there is no dual-write and no local mirror, and in remote mode
 no local `mempalace-mcp` process is spawned (verified: zero mempalace processes in the flipped
 container). So **"the agent has no `mempalace_*` tools" is the expected symptom of a server, token, or
 DNS fault** — not of a broken container. Diagnose with (b).
 #### Rollback — ~30 seconds, loses nothing
 Comment out `MEMPALACE_REMOTE_URL` in `.env`, then `docker compose up -d`. The bridge falls back to the
 local stdio palace, which is intact. Note the local archive is **frozen, not empty**: it stops at the
 moment of the flip, so anything the agent filed into central since then will not be there.
 ---
 ## 4. Still open
 - **Feeding without SSH — the upstream ask that would close §1.3's gap.** Have the feeder send *content*
  over MCP (`add_drawer` / `diary_write`, which it already calls) instead of asking the server to mine a
  path it must first rsync there. HTTPS would then be genuinely sufficient and mining would work from any
  network. Until then, mining is limited to devices that can reach synlig's SSH.
 - **Per-device tokens** — Phase 4. Until then `origin_device` is advisory (§1.1).
 - **§7.6 diary dedup** must be settled *before* the **next** §4.4 join; replay duplicates every entry.
  ⚠ **2026-08-14 — the first join did not resolve this, it SIDESTEPPED it.** The seed was a file-level
  copy of one palace, which replays no diaries and therefore cannot duplicate them. The success of
  that join is *not* evidence the replay path is safe — it never exercised it. §7.6 remains a hard
  blocker for the second machine, which is the one that will actually need merge semantics.
 - **The transcript feeder is inert on every deployed image**, so nothing is being mined automatically
  anywhere in the fleet (§3.7). Regaining it needs a **new tagged pi-devbox release** — its CI
  publishes only on `v*` tags and the latest tag *is* the currently deployed image — built on
  pi-devbox ≥ `7c00dd6` **and** mempalace-toolkit ≥ `29e660e`. Both are required: the first restores
  the container-start catch-up, the second is where extension-side feeding was implemented at all.
 - **Whether *native* pi on a flipped machine was also flipped** is a per-machine question. Answered for
  **EMB-7KJ4VR4G (2026-08-14): native pi is not installed there *yet*** — no `pi`/`mempalace` on the host
  PATH, no `~/.config/pi/`, no `~/.pi/agent/extensions/`; it is a replacement machine that has not had
  native pi set up. So nothing further needs flipping there **today** — but this is a deferred hazard,
  not a closed one: a native install resolves its palace to `~/.mempalace`, which on that host is the
  bind-mounted **frozen archive**, so it would start writing there and split the machine's memory from
  central silently. **Flip native pi at install time, not after.** **Still open for every other host**
  running native pi beside a flipped container. Verify with the §3.8(c) path check, per machine.
  Note native pi has **no `.env` to edit** — pi loads no dotenv file and has no `env` block in
  `settings.json`, so the variables must come from the shell that launches it; recipe in
  [`extensions/pi/README.md`](../extensions/pi/README.md#transport-local-vs-external) § Transport. On
  EMB-7KJ4VR4G specifically, that tree's `config.json` also points at
  `palace_path=/home/developer/.mempalace/palace` — a *container* path that does not exist on macOS — so
  a native install must not inherit it unexamined.
 - **§7.2**: never run `mempalace sync` against the shared palace. Doubly true now that the pi/opencode
  feeders stage *inside* the palace root, which puts staged sources in scope for a sync of the palace dir.
 - **nyvaken's public FQDN and the web hotel's CNAME rules** — unverified (§3.4).
@@ -267,9 +267,11 @@ Covered in §3.3 and repeated here because it is the mistake most likely to be m
 ### 7.3 `seq` is local arrival order and is meaningless across replicas
-`seq` is the local `rowid`. The moment a second replica exists, a remote event that was *authored* earlier can arrive *later* and receive a higher `seq`. The owed-set derivation in §3.3 compares `seq`, so it is sound only on a single replica.
+`seq` is the local `rowid`. The moment a second replica exists, a remote event that was *authored* earlier can arrive *later* and receive a higher `seq`. An owed-set derivation that compares `seq` is therefore sound only on a single replica: on a mesh, a reply can land before the ask it answers, the join concludes "no later reply exists", and an already-answered ask reappears as owed — permanently, on that machine.
-**Action:** on a hub-and-spoke fleet (all clients on one replica) this is correct today. `hlc` is the field to switch to when `mesh_peers` reports actual peers — and that switch must happen *before* multi-replica, not after.
+**✅ Resolved 2026-08-26 (toolkit `extensions/pi/mempalace.ts`).** The derivation now compares `hlc` when both events carry one, falling back to `seq` only when either lacks it (a server predating the field, or an un-backfilled row). `hlc` is rendered fixed-width — `<unix_ms:13 digits>-<counter:6 hex>-<replica_id>` (`hlc.py:1-21`) — so a plain string comparison *is* the causal comparison, with the replica id as final tiebreak. The field is present on the wire: `event_list` returns it per event (`logstream.py:593`), verified against the live hub the same day.
 Measured on real data: the positive-control pair carries `seq` 26/27 and `hlc` `1787773071857-…`/`1787773574085-…`, so on today's single replica the two orderings agree and the switch is a **no-op now and correct later** — which is the whole reason to make it before a second replica exists rather than after. `created_at` remains the wrong key in both worlds: server-generated at second precision, so ties are routine and a tie can suppress an *unanswered* ask outright.
 ### 7.4 A broadcast reaches no mailbox, and `to_agent=NULL` reaches nobody at all
@@ -331,7 +333,7 @@ Replication exists in code — `version_vector()`, `list_ops()`, `apply_remote_e
 1. **Retention.** No TTL, compaction or pruning exists, and `mempalace sync` does not touch the log (§2). For a fleet log this is mostly a feature — nothing is lost by being offline for weeks — but every 4 MiB artifact is permanent. Decide a policy before the log outgrows a comfortable backup, or decide explicitly that permanence is the policy.
 2. **Idempotency key.** Should `event_append` accept an optional client-supplied dedupe key so a retried call is a no-op? This is the one change that would make §7.1 disappear.
-3. **`hlc` as the mailbox join key**, replacing `seq`. Required before a second replica (§7.3). Cheap now, breaking later.
+3. ~~**`hlc` as the mailbox join key**, replacing `seq`.~~ **Done 2026-08-26** — see §7.3. What remains open is the *reverse* direction: `mempalace_event_list`'s `since_event_id` cursor is deliberately local-arrival-ordered (`hlc.py:19-21` — "a tail consumer must see late-arriving remote ops even though their HLC is older"), so cursor semantics and join semantics use different orderings on purpose. That is correct, and worth stating loudly before someone "fixes" the cursor to match the join.
 4. **Per-agent identity.** Do we ever want `from_agent` to be authenticated (§6), or is "authenticates the fleet, not the agent" the permanent contract?
 5. **Should `GET /logstream/events` exist?** A read-only HTTP tail would let non-MCP consumers (dashboards, CI) follow a stream without an MCP client. Today they must hold SSE or speak MCP.
 6. **Undocumented limits.** The 64 KiB `metadata` cap and the lowercase-`type` regex are enforced server-side but absent from the tool schemas (§3.1). Document, or relax.
@@ -11,7 +11,7 @@ What lives where:
 | Content | Home |
 |---|---|
-| How to expose a palace over HTTP, and the Host/Origin pin | `docs/phase-1-exposure-runbook.md` (here) |
+| How this deployment was exposed over HTTP (tunnel, DNS, token custody) | private fleet repository — see [`phase-1-exposure-runbook.md`](phase-1-exposure-runbook.md), also moved |
 | Why a palace needs a special backup, and how to restore one | `docs/backup-and-recovery.md` (here) |
 | Unit/timer/plist templates | `contrib/` (here) |
 | Which machine is primary, its addresses, users, tunnels, offsite target | private fleet repository |
@@ -187,8 +187,8 @@ chosen at load time:
  transport is chosen once at extension load. Confirm the result the same way
  as a container flip: ask the agent for `mempalace_status` and check the
  reported palace path is the **remote** host's, not your own
-  `$HOME/.mempalace/palace` — see
+  `$HOME/.mempalace/palace`. That one check is the whole verification: a
-  [`docs/phase-1-exposure-runbook.md`](../../docs/phase-1-exposure-runbook.md) §3.8.
+  half-flipped client reports a local path while looking healthy.
  Serve such an endpoint with `mempalace serve --host 172.17.0.1 --port 8765`
  (the `pi-devbox` / `opencode-devbox` repos ship a
@@ -207,8 +207,9 @@ chosen at load time:
  token auto-minting is gated on the bind being non-loopback, so it starts with
  **no authentication at all**, no warning. Bind the docker0 gateway
  (`172.17.0.1`): reachable from the host and its containers, not from the LAN.
-  See
+  Binding an interface is not the same as exposing a palace — an MCP endpoint
-  [`docs/phase-1-exposure-runbook.md`](../../docs/phase-1-exposure-runbook.md).
+  also pins `Host`/`Origin`, so a request arriving under the wrong hostname is
  refused even when the port is open.
  Implementation note: the HTTP client (`RemoteMcpClient`) is **vendored** from
  [`pi-extensions`](https://gitea.jordbo.se/joakimp/pi-extensions)'
@@ -234,9 +235,9 @@ local palace, so a remote outage can never scatter memories into a local copy
 nobody will look at again. The practical corollary, worth knowing before you
 debug the wrong layer: **"the agent has no `mempalace_*` tools" is the
 expected symptom of a server, token, or DNS fault**, not of a broken install.
-Diagnose it with a direct `curl` to `MEMPALACE_REMOTE_URL` — see
+Diagnose it with a direct `curl` to `MEMPALACE_REMOTE_URL`, and confirm the flip
-[`docs/phase-1-exposure-runbook.md`](../../docs/phase-1-exposure-runbook.md)
+with `mempalace_status` — the reported palace path must be the remote host's.
-§3.8. The design rationale for de-registering rather than degrading is in
+The design rationale for de-registering rather than degrading is in
 [`docs/rfc-001-global-palace.md`](../../docs/rfc-001-global-palace.md) §2 and §4.1.
 ## Identity
@@ -319,12 +320,19 @@ Owed-ness is **derived, never read off a field**, because `event_ack` appends an
 `status` is written once: a directed `open` event matches the mailbox query
 *forever*, answered or not. Two calls (`to_agent=<me> status=open`, and
 `from_agent=<me>`), then a candidate counts as answered only when one of this
-device's own events has a **strictly higher `seq`**, joins via
+device's own events is **strictly later**, joins via
 `metadata.ack_of` or a shared `correlation_id`, *and* carries a terminal status
-(`applied`/`superseded`/`failed`/`blocked`). The `seq` test is load-bearing:
+(`applied`/`superseded`/`failed`/`blocked`). The ordering test is load-bearing:
 without it one terminal reply suppresses every later ask on that correlation
-forever. `seq` is replica-local, so `hlc` is the correct key once `mesh_peers`
+forever.
-reports actual peers.
+
 "Strictly later" means `hlc` when both events carry one — a hybrid logical clock
 rendered fixed-width, so a string comparison is a causal comparison across
 replicas — falling back to `seq` only when either side lacks an `hlc`. `seq` is
 this database's arrival `rowid`, so on a mesh the same event has a different
 `seq` per replica and a reply can arrive before its ask. Never `created_at`: it
 is second-precision, and a tie there can suppress an *unanswered* ask, which is
 the one failure this derivation exists to prevent.
 `*` broadcasts are excluded even though `to_agent=<me>` matches them, because the
 protocol says a broadcast owes nobody a reply — which also means broadcasting an
@@ -132,6 +132,7 @@ const num = (envVal: string | undefined, fallback: number): number => {
 type LogEvent = {
 	id?: string;
 	seq?: number;
 	hlc?: string;
 	type?: string;
 	status?: string;
 	from_agent?: string;
@@ -1007,6 +1008,40 @@ export default async function mempalaceExtension(pi: ExtensionAPI) {
 		}
 	};
 	/**
 	 * Is `later` strictly after `earlier` in the fleet's ordering?
 	 *
 	 * Prefer `hlc` — a hybrid logical clock rendered fixed-width
 	 * (`<unix_ms:13 digits>-<counter:6 hex>-<replica_id>`), so a plain string
 	 * comparison IS the causal comparison, and it stays correct once a second
 	 * replica exists. `seq` is the arrival rowid of THIS database, so on a mesh
 	 * the same event carries a different `seq` per replica and a reply can land
 	 * before the ask it answers.
 	 *
 	 * Fall back to `seq` only when either side lacks an `hlc` (a server predating
 	 * the field, or a row whose backfill did not run). On a single replica the two
 	 * agree, so the fallback is not a downgrade today — it is the single-replica
 	 * case still being handled after the mesh case became primary.
 	 *
 	 * NEVER compare `created_at`: it is server-generated at second precision, so
 	 * ties are routine, and a tie or a skew there can suppress an UNANSWERED ask
 	 * outright. Every failure mode here is deliberately kept on the
 	 * noisy-but-visible side — an answered item resurfacing is annoying, an
 	 * unanswered ask going silent defeats the mailbox.
 	 */
 	const isStrictlyAfter = (later: LogEvent, earlier: LogEvent): boolean => {
 		if (
 			typeof later.hlc === "string" &&
 			typeof earlier.hlc === "string" &&
 			later.hlc !== "" &&
 			earlier.hlc !== ""
 		) {
 			return later.hlc > earlier.hlc;
 		}
 		if (typeof later.seq !== "number" || typeof earlier.seq !== "number") return false;
 		return later.seq > earlier.seq;
 	};
 	/**
 	 * Has one of MY events closed this candidate?
 	 *
@@ -1023,15 +1058,8 @@ export default async function mempalaceExtension(pi: ExtensionAPI) {
 			// it one terminal reply suppresses every LATER ask on the same
 			// correlation_id forever — silently, permanently, and worst on exactly
 			// the long-running threads the correlation join is for.
-			//
+			// See isStrictlyAfter for why the key is `hlc` and not `seq`/`created_at`.
-			// Compare `seq`, NEVER `created_at`. `seq` is replica-local (it equals
+			if (!isStrictlyAfter(m, candidate)) return false;
 			// origin_seq only while a single replica authors for every machine); the
 			// durable key once `mempalace_mesh_peers` reports real peers is `hlc`,
 			// which is total and causally consistent. Local-seq skew can only make an
 			// ANSWERED item resurface (noise, visible), whereas a timestamp
 			// comparison can suppress an UNANSWERED ask outright.
 			if (typeof m.seq !== "number" || typeof candidate.seq !== "number") return false;
 			if (m.seq <= candidate.seq) return false;
 			// (b) join on the exact ack (written for us by event_ack), else on a
 			// shared correlation_id — which is why correlation_id is mandatory on a
 			// directed open: without it there is no key to join a reply back to.