|
|
@@ -1,463 +1,47 @@
|
|
|
|
# Phase 1 exposure — newt on synlig, DNS, and the client auth model
|
|
|
|
# (moved) Phase 1 exposure runbook
|
|
|
|
|
|
|
|
|
|
|
|
Companion to [`rfc-001-global-palace.md`](./rfc-001-global-palace.md) (design + decisions) and
|
|
|
|
This file used to contain the Phase 1 exposure record for one specific
|
|
|
|
[`synlig-primary-runbook.md`](./synlig-primary-runbook.md) (what is already installed on the primary).
|
|
|
|
deployment — the primary host and tunnel host by name, the DNS registrar step,
|
|
|
|
This doc covers only the step the other two leave open: **making the primary reachable** — runbook §4
|
|
|
|
the Pangolin/newt resource wiring, the shared-token handling, per-machine flip
|
|
|
|
items 2 and 5.
|
|
|
|
dates, and a palace lineage naming three work machines.
|
|
|
|
|
|
|
|
|
|
|
|
> **Status 2026-08-14 — DONE. Exposed, seeded, and one client flipped.**
|
|
|
|
**That content now lives in a private repository**, for the same reason
|
|
|
|
> `https://mempalace.jordbo.se/mcp` has been serving since 2026-08-12 (§3.3–§3.6 are all ✅ below).
|
|
|
|
[`synlig-primary-runbook.md`](synlig-primary-runbook.md) does: a host inventory
|
|
|
|
> The palace was seeded 2026-08-14 15:07 from EMB-7KJ4VR4G (14,777 → 14,803 drawers) — whose palace was
|
|
|
|
is operator data for one deployment, not part of the toolkit. This repository is
|
|
|
|
> itself carried over from the previous work computer **EMB-X1JY06WJ** around 2026-07-06, so the
|
|
|
|
public and keeps only host-agnostic *mechanism*.
|
|
|
|
> primary's lineage is EMB-X1JY06WJ → EMB-7KJ4VR4G → synlig by two file-level copies (rfc-001 §4.4
|
|
|
|
|
|
|
|
> Deviation) — and that machine's pi-devbox container is flipped and **verified end-to-end — see §3.8**, which is the
|
|
|
|
|
|
|
|
> verification procedure that did not exist when the first flip was performed.
|
|
|
|
|
|
|
|
> Still outstanding: the transcript feeder is **inert on every deployed image** (§3.7), and §7.6 is
|
|
|
|
|
|
|
|
> still a hard blocker for the *second* machine to join (§4).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Original status, kept for the record — it contradicted this file's own ✅ section markers for two days,
|
|
|
|
What lives where:
|
|
|
|
which is the failure mode a file-top status block invites:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Status 2026-08-12 — Pangolin updated on nyvaken (done, yours). newt not yet installed on synlig.
|
|
|
|
| Content | Home |
|
|
|
|
Nothing exposed. No client `.env` flipped.**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Read this before touching Pangolin: three of the four questions this step raises were **already decided**
|
|
|
|
|
|
|
|
in RFC §6.2 on 2026-08-09, and re-deciding them differently is how the fleet ends up in two states.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## 1. The four questions, answered
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| Question | Answer | Where it was decided |
|
|
|
|
|
|
|
|
| --- | --- | --- |
|
|
|
|
|
|
|
|
| Which port? | **8765**, path **`/mcp`** (liveness: `/healthz`) | `cli.py:2141` default; runbook §2.4 |
|
|
|
|
|
|
|
|
| What does newt target? | **`172.17.0.1:8765`** (docker0), **never** `127.0.0.1` | RFC §6.2 Transport; runbook §2.4 |
|
|
|
|
|
|
|
|
| Open, or authenticated? | **Authenticated. The primary is never an open public resource.** | RFC §6.2 Network posture |
|
|
|
|
|
|
|
|
| Per-device credentials? | **No — Phase 1 ships the single shared bearer token.** Per-device tokens are Phase 4. | RFC §6.2 Authentication |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### 1.1 Why not per-device users at the proxy
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The instinct — "create a Pangolin user per container, put the credentials in each `.env`, keep the
|
|
|
|
|
|
|
|
usernames distinct" — is the right *goal* (revocation, attribution) reached through the wrong *layer*,
|
|
|
|
|
|
|
|
twice over:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1. **mempalace validates exactly one token.** `hmac.compare_digest(provided, f"Bearer {srv.auth_token}")`
|
|
|
|
|
|
|
|
(`mcp_server.py:5292-5295`) — there is no user table and no second credential. Per-device HTTP identity
|
|
|
|
|
|
|
|
is not a configuration you can express today; it is Phase 4 work (a server-side
|
|
|
|
|
|
|
|
`token → {device_id, scopes}` registry). RFC §6.2 chose the shared token for Phase 1 deliberately:
|
|
|
|
|
|
|
|
*"iterate more feature rich but more complex solutions over time."*
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
2. **Pangolin's HTTP auth is browser-shaped; the clients are not.** SSO login, resource PIN and resource
|
|
|
|
|
|
|
|
password all assume something that can follow a redirect, render a form and hold a session cookie.
|
|
|
|
|
|
|
|
Every MemPalace client here is a headless JSON-RPC `POST` with an `Authorization` header — pi's
|
|
|
|
|
|
|
|
extension, opencode's `type:remote` MCP entry, and `mempalace-pi-session --mode remote`'s
|
|
|
|
|
|
|
|
`urllib.request.urlopen`. Point those at a user-authenticated resource and they receive a login page
|
|
|
|
|
|
|
|
where JSON should be. Enabling that protection breaks precisely the clients it is meant to protect.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
So: **Pangolin terminates TLS and nothing more** (RFC §6.2 Transport, decided 2026-08-09). The bearer token
|
|
|
|
|
|
|
|
is the authentication. This is not "unprotected" — an unauthenticated request to `/mcp` gets a 401 from
|
|
|
|
|
|
|
|
mempalace itself, verified A4/A5 in runbook §2.4.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Consequence to accept consciously** (RFC §6.2, §7.3.2): until Phase 4 the primary **cannot tell devices
|
|
|
|
|
|
|
|
apart**. `origin_device` is client-asserted and advisory — nothing load-bearing may depend on it, and
|
|
|
|
|
|
|
|
revoking one laptop means rotating the token everywhere.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### 1.2 The one place per-device identity *does* exist today
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Remote mode is not only HTTP. `mempalace_mine` expands its source path in the **server** process, so a
|
|
|
|
|
|
|
|
client's staged transcripts must physically exist on the primary. The feeder therefore ships them over
|
|
|
|
|
|
|
|
SSH into a **per-device inbox** before asking the server to mine its own copy:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
|
|
|
rsync -a --update -e "$ssh_cmd" "$STAGE/" "${SSH_TARGET%/}/$DEVICE/" # bin/mempalace-pi-session:677-680
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
That SSH key **is** per-device identity, and it is individually revocable (one line out of
|
|
|
|
|
|
|
|
`authorized_keys`) years before Phase 4 lands. It costs nothing extra, because the mining path needs SSH
|
|
|
|
|
|
|
|
regardless.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Two implications people miss:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- **`mempalace.jordbo.se` alone does not enable mining.** HTTPS covers the read/write tool surface
|
|
|
|
|
|
|
|
(`search`, `add_drawer`, `diary_write`, `kg_*`) — genuinely useful on its own, and the reason to do this
|
|
|
|
|
|
|
|
at all. But `--mode remote` also needs `MEMPALACE_PI_SSH_TARGET` reachable. Budget for both paths.
|
|
|
|
|
|
|
|
- **`DEVICE` defaults to `$(hostname)`** (`bin/mempalace-pi-session:163`). In a container that is the
|
|
|
|
|
|
|
|
container hostname: either random per recreate (inboxes proliferate; each recreate re-mines into a fresh
|
|
|
|
|
|
|
|
empty inbox) or identical across sibling devboxes (two containers writing one inbox). **Set
|
|
|
|
|
|
|
|
`MEMPALACE_PI_DEVICE` explicitly per container.** It is a label, not a secret, so put it somewhere
|
|
|
|
|
|
|
|
reviewable — a committed compose file — where duplicates are visible. That, not username hygiene in
|
|
|
|
|
|
|
|
`.env`, is the discipline this design actually asks of you.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### 1.3 "Then why Pangolin at all, if the feeder uses SSH?"
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Because they are not alternatives — they carry different traffic, and neither substitutes for the other.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| | Pangolin/newt (HTTPS) | SSH + rsync |
|
|
|
|
|
|
|
|
| --- | --- | --- |
|
|
|
|
|
|
|
|
| Carries | the **MCP tool surface**: `search`, `add_drawer`, `diary_write`, `kg_*` — every live tool call | **transcript files only**, once per session or cron run |
|
|
|
|
|
|
|
|
| Used by | the pi extension, opencode `type:remote`, any MCP client | the feeder, internally (`bin/mempalace-pi-session:677-680`) |
|
|
|
|
|
|
|
|
| Needed because | clients need one stable URL, reachable from wherever they are | `mempalace_mine` expands its source path **server-side**, so the server can only mine files on its own disk |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
HTTPS alone is a palace you can query but cannot feed. SSH alone is files shipped with no live query API.
|
|
|
|
|
|
|
|
The rsync is not a transport preference; it is a workaround for *where `mine` resolves paths*.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Could SSH replace Pangolin?** Partly, and it is worth being honest about it:
|
|
|
|
|
|
|
|
`ssh -L 8765:172.17.0.1:8765 synlig` yields a working local MCP endpoint with no public HTTPS at all.
|
|
|
|
|
|
|
|
Three reasons this runbook does not do that:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
1. **Direction.** synlig dials *out* through newt. That we reached for a dial-out tunnel rather than a
|
|
|
|
|
|
|
|
port-forward is itself the evidence that inbound was not available — a corporate host does not accept
|
|
|
|
|
|
|
|
connections from a phone on a foreign network.
|
|
|
|
|
|
|
|
2. **MCP clients want a durable URL**, not a per-session forwarded port. opencode `type:remote` takes a
|
|
|
|
|
|
|
|
URL; a forward that drops takes the tools down mid-session.
|
|
|
|
|
|
|
|
3. The forward must be up on **every device before every session**. Pangolin is up once.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**The weak point, stated plainly.** The rsync runs *client → synlig*, so it needs synlig's SSH reachable
|
|
|
|
|
|
|
|
**from the client**. Were that already true everywhere, no tunnel would be needed for MCP either. So the
|
|
|
|
|
|
|
|
honest expectation after Phase 1 is: **query and write from anywhere, mine only from devices that can
|
|
|
|
|
|
|
|
reach synlig's SSH** (corporate network / VPN / LAN). See §4 for the change that would remove that limit.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## 2. The bind trap, in full
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
RFC §6.2 and runbook §2.4 already say **do not bind loopback behind the tunnel**, because
|
|
|
|
|
|
|
|
`enforce_host_pin = _http_is_loopback(host)` (`mcp_server.py:5367`) makes a loopback bind reject the
|
|
|
|
|
|
|
|
proxy's forwarded `Host:` with a **403** that reads exactly like a Pangolin misconfiguration.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Additional finding, 2026-08-12 — the same reflex also silently removes authentication.** Token
|
|
|
|
|
|
|
|
resolution in `cmd_serve` (`cli.py:1447-1450`) is:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
```python
|
|
|
|
|
|
|
|
loopback = _server_is_loopback(host)
|
|
|
|
|
|
|
|
if not token and not loopback and not args.allow_insecure:
|
|
|
|
|
|
|
|
token, token_created = _load_or_create_server_token(palace_path)
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Auto-minting is gated on the bind being **non-loopback**. A loopback bind therefore starts with **no token
|
|
|
|
|
|
|
|
at all** — no error, no warning, `--allow-insecure` not required — because the server has concluded it is
|
|
|
|
|
|
|
|
only reachable locally, while the tunnel is serving it to the internet. Bind loopback behind newt and you
|
|
|
|
|
|
|
|
get a 403 wall *and*, the moment anything relaxes the Host pin, an unauthenticated palace.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Both failure modes have the same cure, already implemented in
|
|
|
|
|
|
|
|
`contrib/systemd/mempalace-serve.service`: **bind `172.17.0.1`**. Non-loopback, so the Host pin relaxes and
|
|
|
|
|
|
|
|
the token is mandatory; docker0-only, so newt reaches it and the LAN does not.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
> Belt and braces: set `MEMPALACE_MCP_HTTP_TOKEN` explicitly in the unit rather than relying on
|
|
|
|
|
|
|
|
> auto-minting. Then no future bind change can quietly drop authentication.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## 3. Steps
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Ordered so nothing is reachable before it is authenticated.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### 3.1 Start the primary (runbook §4.3 — one `sudo`, unit already staged)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
|
|
|
sudo loginctl enable-linger ecsjper
|
|
|
|
|
|
|
|
cd ~/.config/systemd/user && mv mempalace-serve.service.staged mempalace-serve.service
|
|
|
|
|
|
|
|
systemctl --user daemon-reload && systemctl --user enable --now mempalace-serve
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
curl -s 172.17.0.1:8765/healthz # expect ok
|
|
|
|
|
|
|
|
curl -s 127.0.0.1:8765/healthz # expect NOTHING — connection refused, exit 7 (see below)
|
|
|
|
|
|
|
|
ss -ltnp | grep 8765 # expect 172.17.0.1:8765 only
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
⚠ **The loopback probe returns empty, not 403** — corrected 2026-08-12 against the real run. Nothing is
|
|
|
|
|
|
|
|
listening on `127.0.0.1`, so the connection is refused at TCP level and `curl -s` prints nothing; check it
|
|
|
|
|
|
|
|
with `-w '%{http_code}'` → `000` and `$?` → `7`. The 403 belongs to a *different* configuration: server
|
|
|
|
|
|
|
|
bound **to loopback**, receiving a proxy-forwarded foreign `Host:` (§2, verified 2026-08-10). With a
|
|
|
|
|
|
|
|
docker0-only bind you cannot get 403 from loopback, because you never get far enough to send a header.
|
|
|
|
|
|
|
|
Refusal is the stronger signal of the two: it proves the loopback and LAN surface is not listening at all.
|
|
|
|
|
|
|
|
If it *hangs* instead, or `ss` shows `0.0.0.0:8765`, stop — that is not this configuration.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### 3.2 Collect the shared token
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
|
|
|
cat ~/.mempalace/server/f5d849287f6d73f0141b29d7/token
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Directory name is `sha256(realpath(palace))[:24]` — it changes if the palace path ever moves. Store via the
|
|
|
|
|
|
|
|
`.env.age` flow, 0600 (RFC §6.2).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### 3.3 newt on synlig — ✅ done 2026-08-12 (installed, connected to Pangolin)
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
synlig runs Docker (Gitea Actions runner + digikam) but **no tunnel client** — runbook §4.2. Pangolin on
|
|
|
|
|
|
|
|
nyvaken cannot dial in; synlig must dial out. Add a `newt` container with the credentials Pangolin issues
|
|
|
|
|
|
|
|
for a new site.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Because newt runs in Docker on this box, the docker0 bind is already correct for it: from inside the
|
|
|
|
|
|
|
|
container the primary is `172.17.0.1:8765`. Verify from *inside* newt's network namespace, not from the
|
|
|
|
|
|
|
|
host, before touching DNS.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
> synlig has 7.8 GiB shared with a CI runner (runbook §1). newt is small, but do not colocate anything
|
|
|
|
|
|
|
|
> else here casually.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**Confirm next, now that newt is up.** "Connected to Pangolin" proves newt reached *nyvaken* — a different
|
|
|
|
|
|
|
|
claim from newt reaching *the palace*, and the two fail independently:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
|
|
|
# from inside newt's namespace, not from the host
|
|
|
|
|
|
|
|
docker exec <newt-container> wget -qO- http://172.17.0.1:8765/healthz # expect ok
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
If that hangs or refuses while the Pangolin dashboard shows the site online, the tunnel is fine and the
|
|
|
|
|
|
|
|
*target* is wrong — look at the resource's upstream address (§3.5), not at newt. Note this check needs
|
|
|
|
|
|
|
|
§3.1 done first: if `mempalace-serve` is not running yet, it fails for that reason alone.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### 3.4 DNS at the web hotel — ✅ done 2026-08-12
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
One CNAME: `mempalace` → **the same target your existing Pangolin resources use** (nyvaken's public
|
|
|
|
|
|
|
|
hostname). RFC §6.2 costed this as *"one DNS record per service on the web hotel is the whole setup cost."*
|
|
|
|
|
|
|
|
Done: `mempalace.jordbo.se` resolves and terminates TLS at Pangolin on nyvaken — the §6.2 estimate held.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
⚠️ Not verified from here: nyvaken's public FQDN, and whether your web hotel permits a CNAME at that label
|
|
|
|
|
|
|
|
(some require an A record, or forbid CNAME where other records exist). Confirm before assuming a 5-minute job.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### 3.5 Pangolin resource
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- Target: newt site → **`http://172.17.0.1:8765`** — path `/mcp` (plus `/healthz` for the external probe).
|
|
|
|
|
|
|
|
- ⚠️ **The scheme is `http`, not `https`.** The primary runs `serve --host 172.17.0.1 --port 8765` with no
|
|
|
|
|
|
|
|
cert (`contrib/systemd/mempalace-serve.service`): TLS terminates **at Pangolin**, which is the entire
|
|
|
|
|
|
|
|
point of the §6.2 decision. Point the resource at `https://172.17.0.1:8765` and Pangolin attempts a TLS
|
|
|
|
|
|
|
|
handshake against a plaintext listener — you get a 502/Bad Gateway from outside while the server itself
|
|
|
|
|
|
|
|
looks perfectly healthy on `curl 172.17.0.1:8765/healthz`. (Got this wrong on the first attempt
|
|
|
|
|
|
|
|
2026-08-12, because this line used to omit the scheme.)
|
|
|
|
|
|
|
|
- **Auth: none at the Pangolin layer** (§1.1). TLS termination only.
|
|
|
|
|
|
|
|
- ⚠️ **If resource auth is left on, the signature is a `302`, not a 401 or 403** — hit for real 2026-08-12:
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
HTTP/2 302
|
|
|
|
|
|
|
|
location: https://pangolin.jordbo.se/auth/resource/<uuid>?redirect=https%3A%2F%2Fmempalace.jordbo.se%2Fhealthz
|
|
|
|
|
|
|
|
content-length: 0
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
This is §1.1's "browser-shaped auth" arriving as a concrete symptom: Pangolin sends the login redirect
|
|
|
|
|
|
|
|
**before** proxying, so the palace never sees the request and its journal stays silent. `curl -s` shows
|
|
|
|
|
|
|
|
an empty body and an MCP client sees non-JSON. Diagnose with `-D-` or
|
|
|
|
|
|
|
|
`-w '%{http_code} %{redirect_url}'` — a `location:` pointing at `/auth/resource/…` means the fix is in
|
|
|
|
|
|
|
|
the Pangolin UI (switch the site's authentication off), not in the palace, the unit, or newt.
|
|
|
|
|
|
|
|
**A 302 is unambiguously good news:** DNS, TLS and routing all worked — only the auth layer intervened.
|
|
|
|
|
|
|
|
- Do **not** attach an `Origin`-injecting proxy or browser client: a *present* non-loopback `Origin` is a
|
|
|
|
|
|
|
|
hard 403 with no override (runbook §2.4 B3).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**The server's entire HTTP surface is two exact paths**, so path-scoped rules cover it completely
|
|
|
|
|
|
|
|
(`mcp_server.py:5299-5318`, read 2026-08-12):
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| Method + path | Auth | Notes |
|
|
|
|
|
|
|
|
| --- | --- | --- |
|
|
|
|
|
|
|
|
| `GET /healthz` | none (Host/Origin gated only) | the liveness probe; works with no creds by design |
|
|
|
|
|
|
|
|
| `POST /mcp` | `Authorization: Bearer <token>`, `hmac.compare_digest` on the exact string | the whole tool surface |
|
|
|
|
|
|
|
|
| anything else | — | `send_error(404)` from the palace itself |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Three consequences worth having in writing:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- **Path-scoped Pangolin rules are not a compromise here, they are tighter than a host-wide proxy** and
|
|
|
|
|
|
|
|
lose nothing — there is no third endpoint to forget.
|
|
|
|
|
|
|
|
- **`/mcp` is matched exactly** (`if path != "/mcp"`), so a client URL with a trailing slash gets a 404
|
|
|
|
|
|
|
|
from the palace. Configure clients as `https://mempalace.jordbo.se/mcp` — no trailing slash.
|
|
|
|
|
|
|
|
- **There is no `GET /mcp`, no SSE, no session id, no `DELETE`.** This is plain JSON-RPC over POST, not MCP
|
|
|
|
|
|
|
|
streamable-HTTP. A strict client that opens with a `GET` handshake will see 404; pi's extension,
|
|
|
|
|
|
|
|
opencode `type:remote` and the feeder all POST directly and are fine.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### 3.6 Verify end-to-end before flipping any client — ✅ passed 2026-08-12
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
|
|
|
curl -s https://mempalace.jordbo.se/healthz # ok ✓ from devbox AND synlig
|
|
|
|
|
|
|
|
curl -s -o /dev/null -w '%{http_code}\n' -X POST \
|
|
|
|
|
|
|
|
https://mempalace.jordbo.se/mcp # 401 ✓ the token is doing its job
|
|
|
|
|
|
|
|
TOKEN=$(cat ~/.mempalace/server/f5d849287f6d73f0141b29d7/token) # on synlig
|
|
|
|
|
|
|
|
curl -s -X POST https://mempalace.jordbo.se/mcp \
|
|
|
|
|
|
|
|
-H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
|
|
|
|
|
|
|
|
-d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | head -c 300 # 36 tools
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
No `initialize` and no `Accept: text/event-stream` needed — see the surface table above; a bare
|
|
|
|
|
|
|
|
`tools/list` POST is a complete request. Run the 200 and the 401 from **two different networks** (a client
|
|
|
|
|
|
|
|
box and the primary itself): passing from only one leaves split-horizon DNS untested.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The 401 check matters as much as the 200: it is the only evidence that the thing you just published to the
|
|
|
|
|
|
|
|
internet is not open. Then, and only then, Phase 1 client flip — **one machine first** (RFC §8), and
|
|
|
|
|
|
|
|
remember opencode containers need the §4.1 sidecar merge before their `.env` takes effect.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
### 3.7 Flipping a client: set three variables, or none
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
⚠ **`MEMPALACE_REMOTE_URL` on its own does not degrade to local feeding — it stops feeding.** `auto` mode
|
|
|
|
|
|
|
|
switches to `remote` the moment the URL is set, and remote mode then refuses to run without an SSH target:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
|
|
|
auto) if [[ -n "$REMOTE_URL" ]]; then MODE="remote"; else MODE="local"; fi ;; # :286
|
|
|
|
|
|
|
|
...
|
|
|
|
|
|
|
|
command -v rsync >/dev/null 2>&1 || { echo "error: rsync not found ..."; exit 3; } # :297
|
|
|
|
|
|
|
|
if [[ -z "$SSH_TARGET" ]]; then
|
|
|
|
|
|
|
|
echo "error: MEMPALACE_PI_SSH_TARGET unset (needed for --mode remote)" >&2; exit 1 # :298-300
|
|
|
|
|
|
|
|
fi
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
That exit happens **before anything is staged or filed**, and a cron-driven feeder will simply start
|
|
|
|
|
|
|
|
failing — the loudest symptom is silence, which is the hardest kind to notice. Two safe orders:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- **Both paths at once:** set `MEMPALACE_REMOTE_URL`, `MEMPALACE_REMOTE_TOKEN` **and**
|
|
|
|
|
|
|
|
`MEMPALACE_PI_SSH_TARGET` (plus `MEMPALACE_PI_DEVICE`, §1.2) in the same edit.
|
|
|
|
|
|
|
|
- **HTTPS first, mining later:** set the URL and token, and pin the feeder to `--mode local` until the SSH
|
|
|
|
|
|
|
|
target exists. Tools then read/write the shared palace while transcripts keep landing in the local one.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
`MEMPALACE_REMOTE_URL` is the **full endpoint including `/mcp`, with no trailing slash** — the feeder POSTs
|
|
|
|
|
|
|
|
to it verbatim (`urllib.request.Request(url, data=payload, …)`, `:718`), and §3.5 shows the server matches
|
|
|
|
|
|
|
|
`/mcp` exactly. Matches the existing examples (`docker-compose.mempalace.yml:7`,
|
|
|
|
|
|
|
|
`MEMPALACE_REMOTE_URL=http://<reachable-host>:8765/mcp`). For this fleet:
|
|
|
|
|
|
|
|
`MEMPALACE_REMOTE_URL=https://mempalace.jordbo.se/mcp`
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Either way, run the feeder once by hand and read its exit code before trusting the timer. This is
|
|
|
|
|
|
|
|
precisely the failure "one machine first" is meant to contain.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**The same misconfiguration has two different symptoms depending on who invokes the feeder** — checked in
|
|
|
|
|
|
|
|
the code 2026-08-12, and the quieter one is the trap:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| Caller | Behaviour with `REMOTE_URL` set and `SSH_TARGET` unset |
|
|
|
|
|
|
|
|
|---|---|
|
|
|
|
|---|---|
|
|
|
|
| direct run, session-end hook, cron | `exit 1` with `error: MEMPALACE_PI_SSH_TARGET unset` (`:298-300`) |
|
|
|
|
| Why a palace needs a special backup, and how to restore one | [`backup-and-recovery.md`](backup-and-recovery.md) (here) |
|
|
|
|
| **pi-devbox container start, images ≤ v1.7.0** | **silently skips — no error, no log file** (`entrypoint-user.sh:134`: `: # remote palace but no inbox configured — nothing we can ship to; skip quietly`) |
|
|
|
|
| The coordination log: semantics, delivery, trust model, landmines | [`rfc-003-coordination-log.md`](rfc-003-coordination-log.md) (here) |
|
|
|
|
| pi-devbox container start, images after `cbd7cf5` | prints `MemPalace catch-up skipped: remote palace with no transcript inbox` to the start output *and* to `mempalace-catchup.log`, naming both variables |
|
|
|
|
| What the stores are for, and how a fleet shares one palace | [`fleet-memory.md`](fleet-memory.md) (here) |
|
|
|
|
|
|
|
|
| Unit/timer/plist templates | [`contrib/`](../contrib/) (here) |
|
|
|
|
|
|
|
|
| Which host is primary and which fronts the tunnel, at which addresses, as which user | private fleet repository |
|
|
|
|
|
|
|
|
| Registrar/DNS records, tunnel resource config, token custody | private fleet repository |
|
|
|
|
|
|
|
|
| Per-machine flip dates, seeding history, palace lineage | private fleet repository |
|
|
|
|
|
|
|
|
|
|
|
|
The silent skip is fixed in pi-devbox (`cbd7cf5`), but the fix is in
|
|
|
|
## The mechanism this file also carried — not yet extracted
|
|
|
|
`entrypoint-user.sh`, which is `COPY`d in `Dockerfile.base` — so **every container running an image built
|
|
|
|
|
|
|
|
before that base rebuild still skips silently.** That is the whole fleet today. Until the rebuild lands,
|
|
|
|
|
|
|
|
assume silence and check by hand.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The old skip happened *before* the subshell that writes `~/.pi/agent/mempalace-catchup.log`, so there was
|
|
|
|
Unlike the primary-host runbook, this file was **mixed**: several sections were
|
|
|
|
not even an empty log to notice. Someone asking "why is nothing from this container in the palace?" found
|
|
|
|
reusable mechanism that would be true of anyone's palace, and those are not
|
|
|
|
no artifact at all. The skip itself is correct — there is genuinely nothing to ship to — but it was
|
|
|
|
published anywhere else yet. Named here so the debt is visible rather than lost:
|
|
|
|
indistinguishable from a healthy run that had nothing to do, which is the worst property a memory system
|
|
|
|
|
|
|
|
can have: **the failure looks exactly like success.**
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
→ On a pi-devbox container, confirm the feeder is actually alive after a flip rather than assuming:
|
|
|
|
- **The bind trap, in full.** Why binding a palace to a public interface is not
|
|
|
|
|
|
|
|
the same as exposing it, and the `Host`/`Origin` pin that makes an MCP
|
|
|
|
|
|
|
|
endpoint refuse requests that arrive with the wrong hostname — the single
|
|
|
|
|
|
|
|
most surprising failure in the whole exposure path.
|
|
|
|
|
|
|
|
- **Why not per-device users at the proxy.** The reasoning behind one shared
|
|
|
|
|
|
|
|
fleet token instead of per-device proxy credentials, and the consequence
|
|
|
|
|
|
|
|
documented in [`rfc-003-coordination-log.md`](rfc-003-coordination-log.md) §6:
|
|
|
|
|
|
|
|
the deployment authenticates the *fleet*, not the *agent*.
|
|
|
|
|
|
|
|
- **The one place per-device identity does exist**, and why that is the feeder
|
|
|
|
|
|
|
|
path rather than the HTTP path.
|
|
|
|
|
|
|
|
- **Flipping a client: three variables, or none.** The all-or-nothing shape of
|
|
|
|
|
|
|
|
pointing a machine at a remote palace, and how a half-flipped client fails.
|
|
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
Until that extraction happens, the mechanism is readable only in the private
|
|
|
|
mempalace-pi-session --reason manual-check; echo "exit=$?" # exit=0 and a filed count, not silence
|
|
|
|
record. If you are standing up your own palace over HTTP, the two pieces you
|
|
|
|
cat ~/.pi/agent/mempalace-catchup.log # missing file = the entrypoint skipped
|
|
|
|
must not skip are the `Host`/`Origin` pin and the fact that a client is flipped
|
|
|
|
```
|
|
|
|
by environment variables that travel as a set.
|
|
|
|
|
|
|
|
|
|
|
|
### 3.8 Verify the flip actually took — ✅ done 2026-08-14 on EMB-7KJ4VR4G
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
§3.6 verifies the *endpoint* before you flip. §3.7 tells you *how* to flip. Neither verifies the thing
|
|
|
|
|
|
|
|
you actually care about afterwards: **that the agent's palace tools are now talking to synlig.** That
|
|
|
|
|
|
|
|
gap is why the first flip was "done" for an hour before anyone could say whether it had worked.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
There are **three separate claims** here and they fail independently. Check them in order; each one is
|
|
|
|
|
|
|
|
cheap and rules out a different fault.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**(a) Did the container receive the variable?** In a shell *inside* the container:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
|
|
|
env | grep MEMPALACE_REMOTE_URL # -> https://mempalace.jordbo.se/mcp
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
⚠ Do **not** run a bare `env | grep MEMPALACE` — that prints the bearer token into your scrollback.
|
|
|
|
|
|
|
|
If the variable is absent, the cause is almost always §3.7's edit not having been applied: `env_file`
|
|
|
|
|
|
|
|
is read at container **create** time and baked into the container config, so **`docker compose up -d`
|
|
|
|
|
|
|
|
is required; `docker compose restart` silently reuses the old config.** Evidence from 2026-08-14 —
|
|
|
|
|
|
|
|
running container config-hash `3a55e09ac19e0118` vs compose-computed `8a7e0cd4a677a9c1`; a differing
|
|
|
|
|
|
|
|
hash is what makes `up -d` recreate. `--force-recreate` is not needed.
|
|
|
|
|
|
|
|
**Never `docker compose down -v`** to "pick up" a change: on pi-devbox that destroys seven named
|
|
|
|
|
|
|
|
volumes, including `devbox-pi-config` (pi's config **and every session transcript**), `devbox-uv`, and
|
|
|
|
|
|
|
|
`devbox-chroma-cache` (a large embedding-model re-download).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**(b) Is the server reachable and the token accepted?** Still inside the container — this proves the
|
|
|
|
|
|
|
|
network path and the credential, independently of any agent:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
```sh
|
|
|
|
|
|
|
|
curl -s -X POST "$MEMPALACE_REMOTE_URL" \
|
|
|
|
|
|
|
|
-H "Authorization: Bearer $MEMPALACE_REMOTE_TOKEN" \
|
|
|
|
|
|
|
|
-H 'Content-Type: application/json' \
|
|
|
|
|
|
|
|
-d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | head -c 200
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
`401` = token problem (compare `md5sum` of the client value against synlig's token file — trailing
|
|
|
|
|
|
|
|
whitespace from an editor is the classic cause). Connection failure = DNS/tunnel, not auth.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**(c) Did the agent's bridge actually switch?** This is the claim that matters and the one that cannot
|
|
|
|
|
|
|
|
be checked from bash — there is no log file. Ask the agent running in the container for
|
|
|
|
|
|
|
|
`mempalace_status` and read the palace path it reports:
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
sqlite_integrity.palace = /home/ecsjper/.mempalace/palace # ← proves remote
|
|
|
|
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
**This is the cheapest and strongest discriminator: that path cannot exist inside the container**,
|
|
|
|
|
|
|
|
whose user is `developer` with `HOME=/home/developer` and whose local palace is
|
|
|
|
|
|
|
|
`/home/developer/.mempalace/palace`. One call, no token in scrollback, no writes. If instead it
|
|
|
|
|
|
|
|
reports the local path, the bridge fell back: `createClient()` reads `MEMPALACE_REMOTE_URL` **once at
|
|
|
|
|
|
|
|
load**, so re-check (a) and restart the agent, not just the container.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
#### What does *not* verify the flip — read this before inventing your own check
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- **❌ A drawer count.** Central was seeded *from* the client's own palace, so both report ~14,777.
|
|
|
|
|
|
|
|
Counts cannot tell the two apart. Worse, `mempalace status` counts **chunk rows, not logical
|
|
|
|
|
|
|
|
drawers** (three drawers plus a 2-chunk diary presented as +9), so a delta does not even mean what
|
|
|
|
|
|
|
|
it looks like. Never reason about palace identity or contents from a count.
|
|
|
|
|
|
|
|
- **❌ Writing a drawer through the palace tools and reading it back through the same tools.** This
|
|
|
|
|
|
|
|
succeeds *identically whether or not the flip worked* — both palaces are healthy and were seeded
|
|
|
|
|
|
|
|
from the same source, so a write-then-read round-trips either way. It only becomes evidence if the
|
|
|
|
|
|
|
|
drawer is read back over a **different transport** (the `curl` in (b), by drawer id) or is proven
|
|
|
|
|
|
|
|
**absent** from the local sqlite. This is the same false-positive class as the CLI below, one layer
|
|
|
|
|
|
|
|
up, and it is an easy trap to fall into precisely because it feels like an end-to-end test.
|
|
|
|
|
|
|
|
- **❌ The `mempalace` CLI, in any form.** The CLI has **no remote support whatsoever** — its only
|
|
|
|
|
|
|
|
selector is `--palace <path>` — so it reads and writes the LOCAL on-disk archive via
|
|
|
|
|
|
|
|
`config.json`. After a flip that archive is dead, yet `mempalace status` / `mempalace search` will
|
|
|
|
|
|
|
|
cheerfully report ~14,777 drawers and look exactly like success. It is a false-positive machine.
|
|
|
|
|
|
|
|
**Corollary that bites later:** memories filed with the CLI after a flip land in the dead archive,
|
|
|
|
|
|
|
|
not in central, and a transcript backfill must therefore be mined **on synlig**, where the CLI's
|
|
|
|
|
|
|
|
local palace *is* the central one.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
#### Expected failure behaviour, so you can recognise it
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
The bridge is **fail-closed, not fail-local.** If synlig is unreachable, DNS fails, or the token is
|
|
|
|
|
|
|
|
rejected, the extension retries a bounded number of times, prints `mempalace-mcp unavailable after
|
|
|
|
|
|
|
|
retries; continuing without palace tools`, and **does not register the palace tools at all**. It does
|
|
|
|
|
|
|
|
not silently write to the local palace; there is no dual-write and no local mirror, and in remote mode
|
|
|
|
|
|
|
|
no local `mempalace-mcp` process is spawned (verified: zero mempalace processes in the flipped
|
|
|
|
|
|
|
|
container). So **"the agent has no `mempalace_*` tools" is the expected symptom of a server, token, or
|
|
|
|
|
|
|
|
DNS fault** — not of a broken container. Diagnose with (b).
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
#### Rollback — ~30 seconds, loses nothing
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Comment out `MEMPALACE_REMOTE_URL` in `.env`, then `docker compose up -d`. The bridge falls back to the
|
|
|
|
|
|
|
|
local stdio palace, which is intact. Note the local archive is **frozen, not empty**: it stops at the
|
|
|
|
|
|
|
|
moment of the flip, so anything the agent filed into central since then will not be there.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
---
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
## 4. Still open
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- **Feeding without SSH — the upstream ask that would close §1.3's gap.** Have the feeder send *content*
|
|
|
|
|
|
|
|
over MCP (`add_drawer` / `diary_write`, which it already calls) instead of asking the server to mine a
|
|
|
|
|
|
|
|
path it must first rsync there. HTTPS would then be genuinely sufficient and mining would work from any
|
|
|
|
|
|
|
|
network. Until then, mining is limited to devices that can reach synlig's SSH.
|
|
|
|
|
|
|
|
- **Per-device tokens** — Phase 4. Until then `origin_device` is advisory (§1.1).
|
|
|
|
|
|
|
|
- **§7.6 diary dedup** must be settled *before* the **next** §4.4 join; replay duplicates every entry.
|
|
|
|
|
|
|
|
⚠ **2026-08-14 — the first join did not resolve this, it SIDESTEPPED it.** The seed was a file-level
|
|
|
|
|
|
|
|
copy of one palace, which replays no diaries and therefore cannot duplicate them. The success of
|
|
|
|
|
|
|
|
that join is *not* evidence the replay path is safe — it never exercised it. §7.6 remains a hard
|
|
|
|
|
|
|
|
blocker for the second machine, which is the one that will actually need merge semantics.
|
|
|
|
|
|
|
|
- **The transcript feeder is inert on every deployed image**, so nothing is being mined automatically
|
|
|
|
|
|
|
|
anywhere in the fleet (§3.7). Regaining it needs a **new tagged pi-devbox release** — its CI
|
|
|
|
|
|
|
|
publishes only on `v*` tags and the latest tag *is* the currently deployed image — built on
|
|
|
|
|
|
|
|
pi-devbox ≥ `7c00dd6` **and** mempalace-toolkit ≥ `29e660e`. Both are required: the first restores
|
|
|
|
|
|
|
|
the container-start catch-up, the second is where extension-side feeding was implemented at all.
|
|
|
|
|
|
|
|
- **Whether *native* pi on a flipped machine was also flipped** is a per-machine question. Answered for
|
|
|
|
|
|
|
|
**EMB-7KJ4VR4G (2026-08-14): native pi is not installed there *yet*** — no `pi`/`mempalace` on the host
|
|
|
|
|
|
|
|
PATH, no `~/.config/pi/`, no `~/.pi/agent/extensions/`; it is a replacement machine that has not had
|
|
|
|
|
|
|
|
native pi set up. So nothing further needs flipping there **today** — but this is a deferred hazard,
|
|
|
|
|
|
|
|
not a closed one: a native install resolves its palace to `~/.mempalace`, which on that host is the
|
|
|
|
|
|
|
|
bind-mounted **frozen archive**, so it would start writing there and split the machine's memory from
|
|
|
|
|
|
|
|
central silently. **Flip native pi at install time, not after.** **Still open for every other host**
|
|
|
|
|
|
|
|
running native pi beside a flipped container. Verify with the §3.8(c) path check, per machine.
|
|
|
|
|
|
|
|
Note native pi has **no `.env` to edit** — pi loads no dotenv file and has no `env` block in
|
|
|
|
|
|
|
|
`settings.json`, so the variables must come from the shell that launches it; recipe in
|
|
|
|
|
|
|
|
[`extensions/pi/README.md`](../extensions/pi/README.md#transport-local-vs-external) § Transport. On
|
|
|
|
|
|
|
|
EMB-7KJ4VR4G specifically, that tree's `config.json` also points at
|
|
|
|
|
|
|
|
`palace_path=/home/developer/.mempalace/palace` — a *container* path that does not exist on macOS — so
|
|
|
|
|
|
|
|
a native install must not inherit it unexamined.
|
|
|
|
|
|
|
|
- **§7.2**: never run `mempalace sync` against the shared palace. Doubly true now that the pi/opencode
|
|
|
|
|
|
|
|
feeders stage *inside* the palace root, which puts staged sources in scope for a sync of the palace dir.
|
|
|
|
|
|
|
|
- **nyvaken's public FQDN and the web hotel's CNAME rules** — unverified (§3.4).
|
|
|
|
|
|
|
|
