joakimp/mempalace-toolkit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feeders: stage beside the palace, not in ~/.cache; document Phase 1 exposure

Browse Source 
Staging default moves out of ~/.cache to <palace-root>/pi-stage (pi) and
<palace-root>/opencode-stage (opencode), resolved with mempalace's own
palace-path precedence ($MEMPALACE_PALACE_PATH -> $MEMPAL_PALACE_PATH ->
~/.mempalace/config.json -> ~/.mempalace/palace), then dirname.

Why: the convos miner keys dedup on the *staged* path, so a wiped stage plus a
sync scoped to include it prunes the drawers mined from those sources --
deleting memories, not a cache. Under ~/.cache that state was reachable by
anything treating a cache as disposable. Staging inside the palace makes the
coupling structural: the stage cannot be wiped without touching the palace
itself. Overrides ($MEMPALACE_PI_STAGE / $MEMPALACE_SESSION_STAGE, --stage) are
unchanged. Note the old default had never been created on any host, so this
closed a latent hazard, not a live one.

Measured, and the docs now claim only this much: sync prunes only within the
scope it is given -- wing-only, 1299 scanned / 1299 out of scope / 0 removed;
scoped at the palace root, 651 kept / 648 out of scope. The previous blanket
"sync prunes every drawer" wording overstated it, which is a liability: the next
reader disproves the overstatement and discards the real constraint with it.

Also in this change:
- cron log dir ~/.cache/mempalace-session -> ~/.cache/mempalace-logs. The stage
  left that namespace, so the old name now read as "the stage".
- AGENTS.md: the convos miner *does* check mtime (verified against upstream
  convo_miner.py); the previous "no mtime check" claim was wrong.
- smoke-test assertions use `mktemp -d` for --sessions-dir. One pointed at /tmp,
  which still held earlier synthetic transcripts, so a --dry-run exported a fake
  session into the real stage: --dry-run skips the mine, not the export.

docs/phase-1-exposure-runbook.md -- the newt/DNS/auth step that RFC 001 and the
synlig runbook leave open (runbook section 4, items 2 and 5). Port 8765 at /mcp,
newt targets 172.17.0.1, and the authentication is the single shared bearer
token (RFC 6.2, decided 2026-08-09) rather than per-device proxy users. The
latter cannot work today: mempalace validates exactly one token, and Pangolin's
SSO/PIN/password are browser-shaped while every client here is a headless
JSON-RPC POST -- enabling that protection breaks the clients it protects. The
per-device axis that *does* exist is the feeder's SSH key + per-device inbox.

New finding recorded there: a loopback bind does not merely 403 behind a tunnel
(already known, runbook 2.4) -- it also silently starts the server with no token
at all, because auto-minting is gated on the bind being non-loopback.

extensions/pi/README.md: the HTTP transport IS authenticated as of mempalace
3.6.0; the "sessionless and unauthenticated" note dated from the v1.3.0 era.
Closes the RFC section 8 Phase-0 hygiene item.
This commit is contained in:
Joakim Persson
2026-08-12 17:04:01 +02:00
parent 3626946013
commit 29e660e18f
15 changed files with 1019 additions and 66 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/phase-1-exposure-runbook.md
+185
View File
@@ -0,0 +1,185 @@
# Phase 1 exposure — newt on synlig, DNS, and the client auth model
Companion to [`rfc-001-global-palace.md`](./rfc-001-global-palace.md) (design + decisions) and
[`synlig-primary-runbook.md`](./synlig-primary-runbook.md) (what is already installed on the primary).
This doc covers only the step the other two leave open: **making the primary reachable** — runbook §4
items 2 and 5.
**Status 2026-08-12 — Pangolin updated on nyvaken (done, yours). newt not yet installed on synlig.
Nothing exposed. No client `.env` flipped.**
Read this before touching Pangolin: three of the four questions this step raises were **already decided**
in RFC §6.2 on 2026-08-09, and re-deciding them differently is how the fleet ends up in two states.
---
## 1. The four questions, answered
| Question | Answer | Where it was decided |
| --- | --- | --- |
| Which port? | **8765**, path **`/mcp`** (liveness: `/healthz`) | `cli.py:2141` default; runbook §2.4 |
| What does newt target? | **`172.17.0.1:8765`** (docker0), **never** `127.0.0.1` | RFC §6.2 Transport; runbook §2.4 |
| Open, or authenticated? | **Authenticated. The primary is never an open public resource.** | RFC §6.2 Network posture |
| Per-device credentials? | **No — Phase 1 ships the single shared bearer token.** Per-device tokens are Phase 4. | RFC §6.2 Authentication |
### 1.1 Why not per-device users at the proxy
The instinct — "create a Pangolin user per container, put the credentials in each `.env`, keep the
usernames distinct" — is the right *goal* (revocation, attribution) reached through the wrong *layer*,
twice over:
1. **mempalace validates exactly one token.** `hmac.compare_digest(provided, f"Bearer {srv.auth_token}")`
(`mcp_server.py:5292-5295`) — there is no user table and no second credential. Per-device HTTP identity
is not a configuration you can express today; it is Phase 4 work (a server-side
`token → {device_id, scopes}` registry). RFC §6.2 chose the shared token for Phase 1 deliberately:
*"iterate more feature rich but more complex solutions over time."*
2. **Pangolin's HTTP auth is browser-shaped; the clients are not.** SSO login, resource PIN and resource
password all assume something that can follow a redirect, render a form and hold a session cookie.
Every MemPalace client here is a headless JSON-RPC `POST` with an `Authorization` header — pi's
extension, opencode's `type:remote` MCP entry, and `mempalace-pi-session --mode remote`'s
`urllib.request.urlopen`. Point those at a user-authenticated resource and they receive a login page
where JSON should be. Enabling that protection breaks precisely the clients it is meant to protect.
So: **Pangolin terminates TLS and nothing more** (RFC §6.2 Transport, decided 2026-08-09). The bearer token
is the authentication. This is not "unprotected" — an unauthenticated request to `/mcp` gets a 401 from
mempalace itself, verified A4/A5 in runbook §2.4.
**Consequence to accept consciously** (RFC §6.2, §7.3.2): until Phase 4 the primary **cannot tell devices
apart**. `origin_device` is client-asserted and advisory — nothing load-bearing may depend on it, and
revoking one laptop means rotating the token everywhere.
### 1.2 The one place per-device identity *does* exist today
Remote mode is not only HTTP. `mempalace_mine` expands its source path in the **server** process, so a
client's staged transcripts must physically exist on the primary. The feeder therefore ships them over
SSH into a **per-device inbox** before asking the server to mine its own copy:
```sh
rsync -a --update -e "$ssh_cmd" "$STAGE/" "${SSH_TARGET%/}/$DEVICE/" # bin/mempalace-pi-session:677-680
```
That SSH key **is** per-device identity, and it is individually revocable (one line out of
`authorized_keys`) years before Phase 4 lands. It costs nothing extra, because the mining path needs SSH
regardless.
Two implications people miss:
- **`mempalace.jordbo.se` alone does not enable mining.** HTTPS covers the read/write tool surface
(`search`, `add_drawer`, `diary_write`, `kg_*`) — genuinely useful on its own, and the reason to do this
at all. But `--mode remote` also needs `MEMPALACE_PI_SSH_TARGET` reachable. Budget for both paths.
- **`DEVICE` defaults to `$(hostname)`** (`bin/mempalace-pi-session:163`). In a container that is the
container hostname: either random per recreate (inboxes proliferate; each recreate re-mines into a fresh
empty inbox) or identical across sibling devboxes (two containers writing one inbox). **Set
`MEMPALACE_PI_DEVICE` explicitly per container.** It is a label, not a secret, so put it somewhere
reviewable — a committed compose file — where duplicates are visible. That, not username hygiene in
`.env`, is the discipline this design actually asks of you.
---
## 2. The bind trap, in full
RFC §6.2 and runbook §2.4 already say **do not bind loopback behind the tunnel**, because
`enforce_host_pin = _http_is_loopback(host)` (`mcp_server.py:5367`) makes a loopback bind reject the
proxy's forwarded `Host:` with a **403** that reads exactly like a Pangolin misconfiguration.
**Additional finding, 2026-08-12 — the same reflex also silently removes authentication.** Token
resolution in `cmd_serve` (`cli.py:1447-1450`) is:
```python
loopback = _server_is_loopback(host)
if not token and not loopback and not args.allow_insecure:
token, token_created = _load_or_create_server_token(palace_path)
```
Auto-minting is gated on the bind being **non-loopback**. A loopback bind therefore starts with **no token
at all** — no error, no warning, `--allow-insecure` not required — because the server has concluded it is
only reachable locally, while the tunnel is serving it to the internet. Bind loopback behind newt and you
get a 403 wall *and*, the moment anything relaxes the Host pin, an unauthenticated palace.
Both failure modes have the same cure, already implemented in
`contrib/systemd/mempalace-serve.service`: **bind `172.17.0.1`**. Non-loopback, so the Host pin relaxes and
the token is mandatory; docker0-only, so newt reaches it and the LAN does not.
> Belt and braces: set `MEMPALACE_MCP_HTTP_TOKEN` explicitly in the unit rather than relying on
> auto-minting. Then no future bind change can quietly drop authentication.
---
## 3. Steps
Ordered so nothing is reachable before it is authenticated.
### 3.1 Start the primary (runbook §4.3 — one `sudo`, unit already staged)
```sh
sudo loginctl enable-linger ecsjper
cd ~/.config/systemd/user && mv mempalace-serve.service.staged mempalace-serve.service
systemctl --user daemon-reload && systemctl --user enable --now mempalace-serve
curl -s 172.17.0.1:8765/healthz # expect ok
curl -s 127.0.0.1:8765/healthz # expect 403 — correct, not a bug (§2)
ss -ltnp | grep 8765 # expect 172.17.0.1:8765 only
```
### 3.2 Collect the shared token
```sh
cat ~/.mempalace/server/f5d849287f6d73f0141b29d7/token
```
Directory name is `sha256(realpath(palace))[:24]` — it changes if the palace path ever moves. Store via the
`.env.age` flow, 0600 (RFC §6.2).
### 3.3 newt on synlig
synlig runs Docker (Gitea Actions runner + digikam) but **no tunnel client** — runbook §4.2. Pangolin on
nyvaken cannot dial in; synlig must dial out. Add a `newt` container with the credentials Pangolin issues
for a new site.
Because newt runs in Docker on this box, the docker0 bind is already correct for it: from inside the
container the primary is `172.17.0.1:8765`. Verify from *inside* newt's network namespace, not from the
host, before touching DNS.
> synlig has 7.8 GiB shared with a CI runner (runbook §1). newt is small, but do not colocate anything
> else here casually.
### 3.4 DNS at the web hotel
One CNAME: `mempalace` → **the same target your existing Pangolin resources use** (nyvaken's public
hostname). RFC §6.2 costed this as *"one DNS record per service on the web hotel is the whole setup cost."*
⚠️ Not verified from here: nyvaken's public FQDN, and whether your web hotel permits a CNAME at that label
(some require an A record, or forbid CNAME where other records exist). Confirm before assuming a 5-minute job.
### 3.5 Pangolin resource
- Target: newt site → `172.17.0.1:8765`, path `/mcp` (plus `/healthz` if you want the external probe).
- **Auth: none at the Pangolin layer** (§1.1). TLS termination only.
- Do **not** attach an `Origin`-injecting proxy or browser client: a *present* non-loopback `Origin` is a
hard 403 with no override (runbook §2.4 B3).
### 3.6 Verify end-to-end before flipping any client
```sh
curl -s https://mempalace.jordbo.se/healthz # ok
curl -s -o /dev/null -w '%{http_code}\n' -X POST \
https://mempalace.jordbo.se/mcp # 401 — the token is doing its job
curl -s -X POST https://mempalace.jordbo.se/mcp \
-H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
-d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | head -c 300 # 36 tools
```
The 401 check matters as much as the 200: it is the only evidence that the thing you just published to the
internet is not open. Then, and only then, Phase 1 client flip — **one machine first** (RFC §8), and
remember opencode containers need the §4.1 sidecar merge before their `.env` takes effect.
---
## 4. Still open
- **Per-device tokens** — Phase 4. Until then `origin_device` is advisory (§1.1).
- **§7.6 diary dedup** must be settled *before* the first §4.4 join; replay duplicates every entry.
- **§7.2**: never run `mempalace sync` against the shared palace. Doubly true now that the pi/opencode
feeders stage *inside* the palace root, which puts staged sources in scope for a sync of the palace dir.
- **nyvaken's public FQDN and the web hotel's CNAME rules** — unverified (§3.4).