joakimp/mempalace-toolkit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: take the Phase 1 exposure record private, leave a moved-note

Browse Source 
463 lines with 64 mentions of specific hosts, in a public repo: the primary and
tunnel hosts by name, the registrar/DNS step, the tunnel resource wiring, shared
token custody, per-machine flip dates, and a palace lineage naming three work
machines. Now in the private fleet repo (fleet-ops 093fb65); this file becomes a
moved-note in the shape docs/synlig-primary-runbook.md already established.

Git history keeps the old text, so this limits future exposure rather than
undoing it.

Unlike the primary-host runbook, this file was MIXED — and the stub says so
instead of quietly implying the toolkit still documents HTTP exposure. §1.1-§1.3
(why one shared fleet token rather than per-device proxy users, and the one place
per-device identity does exist), §2 (the bind trap and the Host/Origin pin) and
§3.7 (a client is flipped by three env vars that travel as a set) are reusable
mechanism now published nowhere else. Named in the stub so the extraction is
tracked debt rather than a silent loss, and named in the private copy too so
whoever extracts it can delete the duplicate.

Inbound references fixed rather than left pointing at content that moved: two
§3.8 pointers in extensions/pi/README.md are replaced by the instruction they
were pointing at (the palace path reported by mempalace_status must be the
remote host's — a half-flipped client looks healthy while reporting a local
path), and the bind-trap reference is replaced by the Host/Origin sentence
itself, so the extension README no longer depends on the moved file. contrib
also loses a hostname and a seeding date it did not need to make its point.
This commit is contained in:
Joakim Persson
2026-08-26 23:30:09 +02:00
parent d4d8bb6109
commit d2764bf78e
6 changed files with 66 additions and 479 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/phase-1-exposure-runbook.md
+47 -463
View File
@@ -1,463 +1,47 @@
# Phase 1 exposure — newt on synlig, DNS, and the client auth model
Companion to [`rfc-001-global-palace.md`](./rfc-001-global-palace.md) (design + decisions) and
[`synlig-primary-runbook.md`](./synlig-primary-runbook.md) (what is already installed on the primary).
This doc covers only the step the other two leave open: **making the primary reachable** — runbook §4
items 2 and 5.
> **Status 2026-08-14 — DONE. Exposed, seeded, and one client flipped.**
> `https://mempalace.jordbo.se/mcp` has been serving since 2026-08-12 (§3.3–§3.6 are all ✅ below).
> The palace was seeded 2026-08-14 15:07 from EMB-7KJ4VR4G (14,777 → 14,803 drawers) — whose palace was
> itself carried over from the previous work computer **EMB-X1JY06WJ** around 2026-07-06, so the
> primary's lineage is EMB-X1JY06WJ → EMB-7KJ4VR4G → synlig by two file-level copies (rfc-001 §4.4
> Deviation) — and that machine's pi-devbox container is flipped and **verified end-to-end — see §3.8**, which is the
> verification procedure that did not exist when the first flip was performed.
> Still outstanding: the transcript feeder is **inert on every deployed image** (§3.7), and §7.6 is
> still a hard blocker for the *second* machine to join (§4).
Original status, kept for the record — it contradicted this file's own ✅ section markers for two days,
which is the failure mode a file-top status block invites:
**Status 2026-08-12 — Pangolin updated on nyvaken (done, yours). newt not yet installed on synlig.
Nothing exposed. No client `.env` flipped.**
Read this before touching Pangolin: three of the four questions this step raises were **already decided**
in RFC §6.2 on 2026-08-09, and re-deciding them differently is how the fleet ends up in two states.
---
## 1. The four questions, answered
| Question | Answer | Where it was decided |
| --- | --- | --- |
| Which port? | **8765**, path **`/mcp`** (liveness: `/healthz`) | `cli.py:2141` default; runbook §2.4 |
| What does newt target? | **`172.17.0.1:8765`** (docker0), **never** `127.0.0.1` | RFC §6.2 Transport; runbook §2.4 |
| Open, or authenticated? | **Authenticated. The primary is never an open public resource.** | RFC §6.2 Network posture |
| Per-device credentials? | **No — Phase 1 ships the single shared bearer token.** Per-device tokens are Phase 4. | RFC §6.2 Authentication |
### 1.1 Why not per-device users at the proxy
The instinct — "create a Pangolin user per container, put the credentials in each `.env`, keep the
usernames distinct" — is the right *goal* (revocation, attribution) reached through the wrong *layer*,
twice over:
1. **mempalace validates exactly one token.** `hmac.compare_digest(provided, f"Bearer {srv.auth_token}")`
(`mcp_server.py:5292-5295`) — there is no user table and no second credential. Per-device HTTP identity
is not a configuration you can express today; it is Phase 4 work (a server-side
`token → {device_id, scopes}` registry). RFC §6.2 chose the shared token for Phase 1 deliberately:
*"iterate more feature rich but more complex solutions over time."*
2. **Pangolin's HTTP auth is browser-shaped; the clients are not.** SSO login, resource PIN and resource
password all assume something that can follow a redirect, render a form and hold a session cookie.
Every MemPalace client here is a headless JSON-RPC `POST` with an `Authorization` header — pi's
extension, opencode's `type:remote` MCP entry, and `mempalace-pi-session --mode remote`'s
`urllib.request.urlopen`. Point those at a user-authenticated resource and they receive a login page
where JSON should be. Enabling that protection breaks precisely the clients it is meant to protect.
So: **Pangolin terminates TLS and nothing more** (RFC §6.2 Transport, decided 2026-08-09). The bearer token
is the authentication. This is not "unprotected" — an unauthenticated request to `/mcp` gets a 401 from
mempalace itself, verified A4/A5 in runbook §2.4.
**Consequence to accept consciously** (RFC §6.2, §7.3.2): until Phase 4 the primary **cannot tell devices
apart**. `origin_device` is client-asserted and advisory — nothing load-bearing may depend on it, and
revoking one laptop means rotating the token everywhere.
### 1.2 The one place per-device identity *does* exist today
Remote mode is not only HTTP. `mempalace_mine` expands its source path in the **server** process, so a
client's staged transcripts must physically exist on the primary. The feeder therefore ships them over
SSH into a **per-device inbox** before asking the server to mine its own copy:
```sh
rsync -a --update -e "$ssh_cmd" "$STAGE/" "${SSH_TARGET%/}/$DEVICE/" # bin/mempalace-pi-session:677-680
```
That SSH key **is** per-device identity, and it is individually revocable (one line out of
`authorized_keys`) years before Phase 4 lands. It costs nothing extra, because the mining path needs SSH
regardless.
Two implications people miss:
- **`mempalace.jordbo.se` alone does not enable mining.** HTTPS covers the read/write tool surface
(`search`, `add_drawer`, `diary_write`, `kg_*`) — genuinely useful on its own, and the reason to do this
at all. But `--mode remote` also needs `MEMPALACE_PI_SSH_TARGET` reachable. Budget for both paths.
- **`DEVICE` defaults to `$(hostname)`** (`bin/mempalace-pi-session:163`). In a container that is the
container hostname: either random per recreate (inboxes proliferate; each recreate re-mines into a fresh
empty inbox) or identical across sibling devboxes (two containers writing one inbox). **Set
`MEMPALACE_PI_DEVICE` explicitly per container.** It is a label, not a secret, so put it somewhere
reviewable — a committed compose file — where duplicates are visible. That, not username hygiene in
`.env`, is the discipline this design actually asks of you.
### 1.3 "Then why Pangolin at all, if the feeder uses SSH?"
Because they are not alternatives — they carry different traffic, and neither substitutes for the other.
| | Pangolin/newt (HTTPS) | SSH + rsync |
| --- | --- | --- |
| Carries | the **MCP tool surface**: `search`, `add_drawer`, `diary_write`, `kg_*` — every live tool call | **transcript files only**, once per session or cron run |
| Used by | the pi extension, opencode `type:remote`, any MCP client | the feeder, internally (`bin/mempalace-pi-session:677-680`) |
| Needed because | clients need one stable URL, reachable from wherever they are | `mempalace_mine` expands its source path **server-side**, so the server can only mine files on its own disk |
HTTPS alone is a palace you can query but cannot feed. SSH alone is files shipped with no live query API.
The rsync is not a transport preference; it is a workaround for *where `mine` resolves paths*.
**Could SSH replace Pangolin?** Partly, and it is worth being honest about it:
`ssh -L 8765:172.17.0.1:8765 synlig` yields a working local MCP endpoint with no public HTTPS at all.
Three reasons this runbook does not do that:
1. **Direction.** synlig dials *out* through newt. That we reached for a dial-out tunnel rather than a
port-forward is itself the evidence that inbound was not available — a corporate host does not accept
connections from a phone on a foreign network.
2. **MCP clients want a durable URL**, not a per-session forwarded port. opencode `type:remote` takes a
URL; a forward that drops takes the tools down mid-session.
3. The forward must be up on **every device before every session**. Pangolin is up once.
**The weak point, stated plainly.** The rsync runs *client → synlig*, so it needs synlig's SSH reachable
**from the client**. Were that already true everywhere, no tunnel would be needed for MCP either. So the
honest expectation after Phase 1 is: **query and write from anywhere, mine only from devices that can
reach synlig's SSH** (corporate network / VPN / LAN). See §4 for the change that would remove that limit.
---
## 2. The bind trap, in full
RFC §6.2 and runbook §2.4 already say **do not bind loopback behind the tunnel**, because
`enforce_host_pin = _http_is_loopback(host)` (`mcp_server.py:5367`) makes a loopback bind reject the
proxy's forwarded `Host:` with a **403** that reads exactly like a Pangolin misconfiguration.
**Additional finding, 2026-08-12 — the same reflex also silently removes authentication.** Token
resolution in `cmd_serve` (`cli.py:1447-1450`) is:
```python
loopback = _server_is_loopback(host)
if not token and not loopback and not args.allow_insecure:
token, token_created = _load_or_create_server_token(palace_path)
```
Auto-minting is gated on the bind being **non-loopback**. A loopback bind therefore starts with **no token
at all** — no error, no warning, `--allow-insecure` not required — because the server has concluded it is
only reachable locally, while the tunnel is serving it to the internet. Bind loopback behind newt and you
get a 403 wall *and*, the moment anything relaxes the Host pin, an unauthenticated palace.
Both failure modes have the same cure, already implemented in
`contrib/systemd/mempalace-serve.service`: **bind `172.17.0.1`**. Non-loopback, so the Host pin relaxes and
the token is mandatory; docker0-only, so newt reaches it and the LAN does not.
> Belt and braces: set `MEMPALACE_MCP_HTTP_TOKEN` explicitly in the unit rather than relying on
> auto-minting. Then no future bind change can quietly drop authentication.
---
## 3. Steps
Ordered so nothing is reachable before it is authenticated.
### 3.1 Start the primary (runbook §4.3 — one `sudo`, unit already staged)
```sh
sudo loginctl enable-linger ecsjper
cd ~/.config/systemd/user && mv mempalace-serve.service.staged mempalace-serve.service
systemctl --user daemon-reload && systemctl --user enable --now mempalace-serve
curl -s 172.17.0.1:8765/healthz # expect ok
curl -s 127.0.0.1:8765/healthz # expect NOTHING — connection refused, exit 7 (see below)
ss -ltnp | grep 8765 # expect 172.17.0.1:8765 only
```
⚠ **The loopback probe returns empty, not 403** — corrected 2026-08-12 against the real run. Nothing is
listening on `127.0.0.1`, so the connection is refused at TCP level and `curl -s` prints nothing; check it
with `-w '%{http_code}'` → `000` and `$?` → `7`. The 403 belongs to a *different* configuration: server
bound **to loopback**, receiving a proxy-forwarded foreign `Host:` (§2, verified 2026-08-10). With a
docker0-only bind you cannot get 403 from loopback, because you never get far enough to send a header.
Refusal is the stronger signal of the two: it proves the loopback and LAN surface is not listening at all.
If it *hangs* instead, or `ss` shows `0.0.0.0:8765`, stop — that is not this configuration.
### 3.2 Collect the shared token
```sh
cat ~/.mempalace/server/f5d849287f6d73f0141b29d7/token
```
Directory name is `sha256(realpath(palace))[:24]` — it changes if the palace path ever moves. Store via the
`.env.age` flow, 0600 (RFC §6.2).
### 3.3 newt on synlig — ✅ done 2026-08-12 (installed, connected to Pangolin)
synlig runs Docker (Gitea Actions runner + digikam) but **no tunnel client** — runbook §4.2. Pangolin on
nyvaken cannot dial in; synlig must dial out. Add a `newt` container with the credentials Pangolin issues
for a new site.
Because newt runs in Docker on this box, the docker0 bind is already correct for it: from inside the
container the primary is `172.17.0.1:8765`. Verify from *inside* newt's network namespace, not from the
host, before touching DNS.
> synlig has 7.8 GiB shared with a CI runner (runbook §1). newt is small, but do not colocate anything
> else here casually.
**Confirm next, now that newt is up.** "Connected to Pangolin" proves newt reached *nyvaken* — a different
claim from newt reaching *the palace*, and the two fail independently:
```sh
# from inside newt's namespace, not from the host
docker exec <newt-container> wget -qO- http://172.17.0.1:8765/healthz # expect ok
```
If that hangs or refuses while the Pangolin dashboard shows the site online, the tunnel is fine and the
*target* is wrong — look at the resource's upstream address (§3.5), not at newt. Note this check needs
§3.1 done first: if `mempalace-serve` is not running yet, it fails for that reason alone.
### 3.4 DNS at the web hotel — ✅ done 2026-08-12
One CNAME: `mempalace` → **the same target your existing Pangolin resources use** (nyvaken's public
hostname). RFC §6.2 costed this as *"one DNS record per service on the web hotel is the whole setup cost."*
Done: `mempalace.jordbo.se` resolves and terminates TLS at Pangolin on nyvaken — the §6.2 estimate held.
⚠️ Not verified from here: nyvaken's public FQDN, and whether your web hotel permits a CNAME at that label
(some require an A record, or forbid CNAME where other records exist). Confirm before assuming a 5-minute job.
### 3.5 Pangolin resource
- Target: newt site → **`http://172.17.0.1:8765`** — path `/mcp` (plus `/healthz` for the external probe).
- ⚠️ **The scheme is `http`, not `https`.** The primary runs `serve --host 172.17.0.1 --port 8765` with no
cert (`contrib/systemd/mempalace-serve.service`): TLS terminates **at Pangolin**, which is the entire
point of the §6.2 decision. Point the resource at `https://172.17.0.1:8765` and Pangolin attempts a TLS
handshake against a plaintext listener — you get a 502/Bad Gateway from outside while the server itself
looks perfectly healthy on `curl 172.17.0.1:8765/healthz`. (Got this wrong on the first attempt
2026-08-12, because this line used to omit the scheme.)
- **Auth: none at the Pangolin layer** (§1.1). TLS termination only.
- ⚠️ **If resource auth is left on, the signature is a `302`, not a 401 or 403** — hit for real 2026-08-12:
```
HTTP/2 302
location: https://pangolin.jordbo.se/auth/resource/<uuid>?redirect=https%3A%2F%2Fmempalace.jordbo.se%2Fhealthz
content-length: 0
```
This is §1.1's "browser-shaped auth" arriving as a concrete symptom: Pangolin sends the login redirect
**before** proxying, so the palace never sees the request and its journal stays silent. `curl -s` shows
an empty body and an MCP client sees non-JSON. Diagnose with `-D-` or
`-w '%{http_code} %{redirect_url}'` — a `location:` pointing at `/auth/resource/…` means the fix is in
the Pangolin UI (switch the site's authentication off), not in the palace, the unit, or newt.
**A 302 is unambiguously good news:** DNS, TLS and routing all worked — only the auth layer intervened.
- Do **not** attach an `Origin`-injecting proxy or browser client: a *present* non-loopback `Origin` is a
hard 403 with no override (runbook §2.4 B3).
**The server's entire HTTP surface is two exact paths**, so path-scoped rules cover it completely
(`mcp_server.py:5299-5318`, read 2026-08-12):
| Method + path | Auth | Notes |
| --- | --- | --- |
| `GET /healthz` | none (Host/Origin gated only) | the liveness probe; works with no creds by design |
| `POST /mcp` | `Authorization: Bearer <token>`, `hmac.compare_digest` on the exact string | the whole tool surface |
| anything else | — | `send_error(404)` from the palace itself |
Three consequences worth having in writing:
- **Path-scoped Pangolin rules are not a compromise here, they are tighter than a host-wide proxy** and
lose nothing — there is no third endpoint to forget.
- **`/mcp` is matched exactly** (`if path != "/mcp"`), so a client URL with a trailing slash gets a 404
from the palace. Configure clients as `https://mempalace.jordbo.se/mcp` — no trailing slash.
- **There is no `GET /mcp`, no SSE, no session id, no `DELETE`.** This is plain JSON-RPC over POST, not MCP
streamable-HTTP. A strict client that opens with a `GET` handshake will see 404; pi's extension,
opencode `type:remote` and the feeder all POST directly and are fine.
### 3.6 Verify end-to-end before flipping any client — ✅ passed 2026-08-12
```sh
curl -s https://mempalace.jordbo.se/healthz # ok ✓ from devbox AND synlig
curl -s -o /dev/null -w '%{http_code}\n' -X POST \
https://mempalace.jordbo.se/mcp # 401 ✓ the token is doing its job
TOKEN=$(cat ~/.mempalace/server/f5d849287f6d73f0141b29d7/token) # on synlig
curl -s -X POST https://mempalace.jordbo.se/mcp \
-H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
-d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | head -c 300 # 36 tools
```
No `initialize` and no `Accept: text/event-stream` needed — see the surface table above; a bare
`tools/list` POST is a complete request. Run the 200 and the 401 from **two different networks** (a client
box and the primary itself): passing from only one leaves split-horizon DNS untested.
The 401 check matters as much as the 200: it is the only evidence that the thing you just published to the
internet is not open. Then, and only then, Phase 1 client flip — **one machine first** (RFC §8), and
remember opencode containers need the §4.1 sidecar merge before their `.env` takes effect.
### 3.7 Flipping a client: set three variables, or none
⚠ **`MEMPALACE_REMOTE_URL` on its own does not degrade to local feeding — it stops feeding.** `auto` mode
switches to `remote` the moment the URL is set, and remote mode then refuses to run without an SSH target:
```sh
auto) if [[ -n "$REMOTE_URL" ]]; then MODE="remote"; else MODE="local"; fi ;; # :286
...
command -v rsync >/dev/null 2>&1 || { echo "error: rsync not found ..."; exit 3; } # :297
if [[ -z "$SSH_TARGET" ]]; then
echo "error: MEMPALACE_PI_SSH_TARGET unset (needed for --mode remote)" >&2; exit 1 # :298-300
fi
```
That exit happens **before anything is staged or filed**, and a cron-driven feeder will simply start
failing — the loudest symptom is silence, which is the hardest kind to notice. Two safe orders:
- **Both paths at once:** set `MEMPALACE_REMOTE_URL`, `MEMPALACE_REMOTE_TOKEN` **and**
`MEMPALACE_PI_SSH_TARGET` (plus `MEMPALACE_PI_DEVICE`, §1.2) in the same edit.
- **HTTPS first, mining later:** set the URL and token, and pin the feeder to `--mode local` until the SSH
target exists. Tools then read/write the shared palace while transcripts keep landing in the local one.
`MEMPALACE_REMOTE_URL` is the **full endpoint including `/mcp`, with no trailing slash** — the feeder POSTs
to it verbatim (`urllib.request.Request(url, data=payload, …)`, `:718`), and §3.5 shows the server matches
`/mcp` exactly. Matches the existing examples (`docker-compose.mempalace.yml:7`,
`MEMPALACE_REMOTE_URL=http://<reachable-host>:8765/mcp`). For this fleet:
`MEMPALACE_REMOTE_URL=https://mempalace.jordbo.se/mcp`
Either way, run the feeder once by hand and read its exit code before trusting the timer. This is
precisely the failure "one machine first" is meant to contain.
**The same misconfiguration has two different symptoms depending on who invokes the feeder** — checked in
the code 2026-08-12, and the quieter one is the trap:
| Caller | Behaviour with `REMOTE_URL` set and `SSH_TARGET` unset |
| --- | --- |
| direct run, session-end hook, cron | `exit 1` with `error: MEMPALACE_PI_SSH_TARGET unset` (`:298-300`) |
| **pi-devbox container start, images ≤ v1.7.0** | **silently skips — no error, no log file** (`entrypoint-user.sh:134`: `: # remote palace but no inbox configured — nothing we can ship to; skip quietly`) |
| pi-devbox container start, images after `cbd7cf5` | prints `MemPalace catch-up skipped: remote palace with no transcript inbox` to the start output *and* to `mempalace-catchup.log`, naming both variables |
The silent skip is fixed in pi-devbox (`cbd7cf5`), but the fix is in
`entrypoint-user.sh`, which is `COPY`d in `Dockerfile.base` — so **every container running an image built
before that base rebuild still skips silently.** That is the whole fleet today. Until the rebuild lands,
assume silence and check by hand.
The old skip happened *before* the subshell that writes `~/.pi/agent/mempalace-catchup.log`, so there was
not even an empty log to notice. Someone asking "why is nothing from this container in the palace?" found
no artifact at all. The skip itself is correct — there is genuinely nothing to ship to — but it was
indistinguishable from a healthy run that had nothing to do, which is the worst property a memory system
can have: **the failure looks exactly like success.**
→ On a pi-devbox container, confirm the feeder is actually alive after a flip rather than assuming:
```sh
mempalace-pi-session --reason manual-check; echo "exit=$?" # exit=0 and a filed count, not silence
cat ~/.pi/agent/mempalace-catchup.log # missing file = the entrypoint skipped
```
### 3.8 Verify the flip actually took — ✅ done 2026-08-14 on EMB-7KJ4VR4G
§3.6 verifies the *endpoint* before you flip. §3.7 tells you *how* to flip. Neither verifies the thing
you actually care about afterwards: **that the agent's palace tools are now talking to synlig.** That
gap is why the first flip was "done" for an hour before anyone could say whether it had worked.
There are **three separate claims** here and they fail independently. Check them in order; each one is
cheap and rules out a different fault.
**(a) Did the container receive the variable?** In a shell *inside* the container:
```sh
env | grep MEMPALACE_REMOTE_URL # -> https://mempalace.jordbo.se/mcp
```
⚠ Do **not** run a bare `env | grep MEMPALACE` — that prints the bearer token into your scrollback.
If the variable is absent, the cause is almost always §3.7's edit not having been applied: `env_file`
is read at container **create** time and baked into the container config, so **`docker compose up -d`
is required; `docker compose restart` silently reuses the old config.** Evidence from 2026-08-14 —
running container config-hash `3a55e09ac19e0118` vs compose-computed `8a7e0cd4a677a9c1`; a differing
hash is what makes `up -d` recreate. `--force-recreate` is not needed.
**Never `docker compose down -v`** to "pick up" a change: on pi-devbox that destroys seven named
volumes, including `devbox-pi-config` (pi's config **and every session transcript**), `devbox-uv`, and
`devbox-chroma-cache` (a large embedding-model re-download).
**(b) Is the server reachable and the token accepted?** Still inside the container — this proves the
network path and the credential, independently of any agent:
```sh
curl -s -X POST "$MEMPALACE_REMOTE_URL" \
-H "Authorization: Bearer $MEMPALACE_REMOTE_TOKEN" \
-H 'Content-Type: application/json' \
-d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | head -c 200
```
`401` = token problem (compare `md5sum` of the client value against synlig's token file — trailing
whitespace from an editor is the classic cause). Connection failure = DNS/tunnel, not auth.
**(c) Did the agent's bridge actually switch?** This is the claim that matters and the one that cannot
be checked from bash — there is no log file. Ask the agent running in the container for
`mempalace_status` and read the palace path it reports:
```
sqlite_integrity.palace = /home/ecsjper/.mempalace/palace # ← proves remote
```
**This is the cheapest and strongest discriminator: that path cannot exist inside the container**,
whose user is `developer` with `HOME=/home/developer` and whose local palace is
`/home/developer/.mempalace/palace`. One call, no token in scrollback, no writes. If instead it
reports the local path, the bridge fell back: `createClient()` reads `MEMPALACE_REMOTE_URL` **once at
load**, so re-check (a) and restart the agent, not just the container.
#### What does *not* verify the flip — read this before inventing your own check
- **❌ A drawer count.** Central was seeded *from* the client's own palace, so both report ~14,777.
Counts cannot tell the two apart. Worse, `mempalace status` counts **chunk rows, not logical
drawers** (three drawers plus a 2-chunk diary presented as +9), so a delta does not even mean what
it looks like. Never reason about palace identity or contents from a count.
- **❌ Writing a drawer through the palace tools and reading it back through the same tools.** This
succeeds *identically whether or not the flip worked* — both palaces are healthy and were seeded
from the same source, so a write-then-read round-trips either way. It only becomes evidence if the
drawer is read back over a **different transport** (the `curl` in (b), by drawer id) or is proven
**absent** from the local sqlite. This is the same false-positive class as the CLI below, one layer
up, and it is an easy trap to fall into precisely because it feels like an end-to-end test.
- **❌ The `mempalace` CLI, in any form.** The CLI has **no remote support whatsoever** — its only
selector is `--palace <path>` — so it reads and writes the LOCAL on-disk archive via
`config.json`. After a flip that archive is dead, yet `mempalace status` / `mempalace search` will
cheerfully report ~14,777 drawers and look exactly like success. It is a false-positive machine.
**Corollary that bites later:** memories filed with the CLI after a flip land in the dead archive,
not in central, and a transcript backfill must therefore be mined **on synlig**, where the CLI's
local palace *is* the central one.
#### Expected failure behaviour, so you can recognise it
The bridge is **fail-closed, not fail-local.** If synlig is unreachable, DNS fails, or the token is
rejected, the extension retries a bounded number of times, prints `mempalace-mcp unavailable after
retries; continuing without palace tools`, and **does not register the palace tools at all**. It does
not silently write to the local palace; there is no dual-write and no local mirror, and in remote mode
no local `mempalace-mcp` process is spawned (verified: zero mempalace processes in the flipped
container). So **"the agent has no `mempalace_*` tools" is the expected symptom of a server, token, or
DNS fault** — not of a broken container. Diagnose with (b).
#### Rollback — ~30 seconds, loses nothing
Comment out `MEMPALACE_REMOTE_URL` in `.env`, then `docker compose up -d`. The bridge falls back to the
local stdio palace, which is intact. Note the local archive is **frozen, not empty**: it stops at the
moment of the flip, so anything the agent filed into central since then will not be there.
---
## 4. Still open
- **Feeding without SSH — the upstream ask that would close §1.3's gap.** Have the feeder send *content*
over MCP (`add_drawer` / `diary_write`, which it already calls) instead of asking the server to mine a
path it must first rsync there. HTTPS would then be genuinely sufficient and mining would work from any
network. Until then, mining is limited to devices that can reach synlig's SSH.
- **Per-device tokens** — Phase 4. Until then `origin_device` is advisory (§1.1).
- **§7.6 diary dedup** must be settled *before* the **next** §4.4 join; replay duplicates every entry.
⚠ **2026-08-14 — the first join did not resolve this, it SIDESTEPPED it.** The seed was a file-level
copy of one palace, which replays no diaries and therefore cannot duplicate them. The success of
that join is *not* evidence the replay path is safe — it never exercised it. §7.6 remains a hard
blocker for the second machine, which is the one that will actually need merge semantics.
- **The transcript feeder is inert on every deployed image**, so nothing is being mined automatically
anywhere in the fleet (§3.7). Regaining it needs a **new tagged pi-devbox release** — its CI
publishes only on `v*` tags and the latest tag *is* the currently deployed image — built on
pi-devbox ≥ `7c00dd6` **and** mempalace-toolkit ≥ `29e660e`. Both are required: the first restores
the container-start catch-up, the second is where extension-side feeding was implemented at all.
- **Whether *native* pi on a flipped machine was also flipped** is a per-machine question. Answered for
**EMB-7KJ4VR4G (2026-08-14): native pi is not installed there *yet*** — no `pi`/`mempalace` on the host
PATH, no `~/.config/pi/`, no `~/.pi/agent/extensions/`; it is a replacement machine that has not had
native pi set up. So nothing further needs flipping there **today** — but this is a deferred hazard,
not a closed one: a native install resolves its palace to `~/.mempalace`, which on that host is the
bind-mounted **frozen archive**, so it would start writing there and split the machine's memory from
central silently. **Flip native pi at install time, not after.** **Still open for every other host**
running native pi beside a flipped container. Verify with the §3.8(c) path check, per machine.
Note native pi has **no `.env` to edit** — pi loads no dotenv file and has no `env` block in
`settings.json`, so the variables must come from the shell that launches it; recipe in
[`extensions/pi/README.md`](../extensions/pi/README.md#transport-local-vs-external) § Transport. On
EMB-7KJ4VR4G specifically, that tree's `config.json` also points at
`palace_path=/home/developer/.mempalace/palace` — a *container* path that does not exist on macOS — so
a native install must not inherit it unexamined.
- **§7.2**: never run `mempalace sync` against the shared palace. Doubly true now that the pi/opencode
feeders stage *inside* the palace root, which puts staged sources in scope for a sync of the palace dir.
- **nyvaken's public FQDN and the web hotel's CNAME rules** — unverified (§3.4).
# (moved) Phase 1 exposure runbook
This file used to contain the Phase 1 exposure record for one specific
deployment — the primary host and tunnel host by name, the DNS registrar step,
the Pangolin/newt resource wiring, the shared-token handling, per-machine flip
dates, and a palace lineage naming three work machines.
**That content now lives in a private repository**, for the same reason
[`synlig-primary-runbook.md`](synlig-primary-runbook.md) does: a host inventory
is operator data for one deployment, not part of the toolkit. This repository is
public and keeps only host-agnostic *mechanism*.
What lives where:
| Content | Home |
|---|---|
| Why a palace needs a special backup, and how to restore one | [`backup-and-recovery.md`](backup-and-recovery.md) (here) |
| The coordination log: semantics, delivery, trust model, landmines | [`rfc-003-coordination-log.md`](rfc-003-coordination-log.md) (here) |
| What the stores are for, and how a fleet shares one palace | [`fleet-memory.md`](fleet-memory.md) (here) |
| Unit/timer/plist templates | [`contrib/`](../contrib/) (here) |
| Which host is primary and which fronts the tunnel, at which addresses, as which user | private fleet repository |
| Registrar/DNS records, tunnel resource config, token custody | private fleet repository |
| Per-machine flip dates, seeding history, palace lineage | private fleet repository |
## The mechanism this file also carried — not yet extracted
Unlike the primary-host runbook, this file was **mixed**: several sections were
reusable mechanism that would be true of anyone's palace, and those are not
published anywhere else yet. Named here so the debt is visible rather than lost:
- **The bind trap, in full.** Why binding a palace to a public interface is not
the same as exposing it, and the `Host`/`Origin` pin that makes an MCP
endpoint refuse requests that arrive with the wrong hostname — the single
most surprising failure in the whole exposure path.
- **Why not per-device users at the proxy.** The reasoning behind one shared
fleet token instead of per-device proxy credentials, and the consequence
documented in [`rfc-003-coordination-log.md`](rfc-003-coordination-log.md) §6:
the deployment authenticates the *fleet*, not the *agent*.
- **The one place per-device identity does exist**, and why that is the feeder
path rather than the HTTP path.
- **Flipping a client: three variables, or none.** The all-or-nothing shape of
pointing a machine at a remote palace, and how a half-flipped client fails.
Until that extraction happens, the mechanism is readable only in the private
record. If you are standing up your own palace over HTTP, the two pieces you
must not skip are the `Host`/`Origin` pin and the fact that a client is flipped
by environment variables that travel as a set.