29e660e18f
Staging default moves out of ~/.cache to <palace-root>/pi-stage (pi) and <palace-root>/opencode-stage (opencode), resolved with mempalace's own palace-path precedence ($MEMPALACE_PALACE_PATH -> $MEMPAL_PALACE_PATH -> ~/.mempalace/config.json -> ~/.mempalace/palace), then dirname. Why: the convos miner keys dedup on the *staged* path, so a wiped stage plus a sync scoped to include it prunes the drawers mined from those sources -- deleting memories, not a cache. Under ~/.cache that state was reachable by anything treating a cache as disposable. Staging inside the palace makes the coupling structural: the stage cannot be wiped without touching the palace itself. Overrides ($MEMPALACE_PI_STAGE / $MEMPALACE_SESSION_STAGE, --stage) are unchanged. Note the old default had never been created on any host, so this closed a latent hazard, not a live one. Measured, and the docs now claim only this much: sync prunes only within the scope it is given -- wing-only, 1299 scanned / 1299 out of scope / 0 removed; scoped at the palace root, 651 kept / 648 out of scope. The previous blanket "sync prunes every drawer" wording overstated it, which is a liability: the next reader disproves the overstatement and discards the real constraint with it. Also in this change: - cron log dir ~/.cache/mempalace-session -> ~/.cache/mempalace-logs. The stage left that namespace, so the old name now read as "the stage". - AGENTS.md: the convos miner *does* check mtime (verified against upstream convo_miner.py); the previous "no mtime check" claim was wrong. - smoke-test assertions use `mktemp -d` for --sessions-dir. One pointed at /tmp, which still held earlier synthetic transcripts, so a --dry-run exported a fake session into the real stage: --dry-run skips the mine, not the export. docs/phase-1-exposure-runbook.md -- the newt/DNS/auth step that RFC 001 and the synlig runbook leave open (runbook section 4, items 2 and 5). Port 8765 at /mcp, newt targets 172.17.0.1, and the authentication is the single shared bearer token (RFC 6.2, decided 2026-08-09) rather than per-device proxy users. The latter cannot work today: mempalace validates exactly one token, and Pangolin's SSO/PIN/password are browser-shaped while every client here is a headless JSON-RPC POST -- enabling that protection breaks the clients it protects. The per-device axis that *does* exist is the feeder's SSH key + per-device inbox. New finding recorded there: a loopback bind does not merely 403 behind a tunnel (already known, runbook 2.4) -- it also silently starts the server with no token at all, because auto-minting is gated on the bind being non-loopback. extensions/pi/README.md: the HTTP transport IS authenticated as of mempalace 3.6.0; the "sessionless and unauthenticated" note dated from the v1.3.0 era. Closes the RFC section 8 Phase-0 hygiene item.
186 lines
9.6 KiB
Markdown
186 lines
9.6 KiB
Markdown
# Phase 1 exposure — newt on synlig, DNS, and the client auth model
|
|
|
|
Companion to [`rfc-001-global-palace.md`](./rfc-001-global-palace.md) (design + decisions) and
|
|
[`synlig-primary-runbook.md`](./synlig-primary-runbook.md) (what is already installed on the primary).
|
|
This doc covers only the step the other two leave open: **making the primary reachable** — runbook §4
|
|
items 2 and 5.
|
|
|
|
**Status 2026-08-12 — Pangolin updated on nyvaken (done, yours). newt not yet installed on synlig.
|
|
Nothing exposed. No client `.env` flipped.**
|
|
|
|
Read this before touching Pangolin: three of the four questions this step raises were **already decided**
|
|
in RFC §6.2 on 2026-08-09, and re-deciding them differently is how the fleet ends up in two states.
|
|
|
|
---
|
|
|
|
## 1. The four questions, answered
|
|
|
|
| Question | Answer | Where it was decided |
|
|
| --- | --- | --- |
|
|
| Which port? | **8765**, path **`/mcp`** (liveness: `/healthz`) | `cli.py:2141` default; runbook §2.4 |
|
|
| What does newt target? | **`172.17.0.1:8765`** (docker0), **never** `127.0.0.1` | RFC §6.2 Transport; runbook §2.4 |
|
|
| Open, or authenticated? | **Authenticated. The primary is never an open public resource.** | RFC §6.2 Network posture |
|
|
| Per-device credentials? | **No — Phase 1 ships the single shared bearer token.** Per-device tokens are Phase 4. | RFC §6.2 Authentication |
|
|
|
|
### 1.1 Why not per-device users at the proxy
|
|
|
|
The instinct — "create a Pangolin user per container, put the credentials in each `.env`, keep the
|
|
usernames distinct" — is the right *goal* (revocation, attribution) reached through the wrong *layer*,
|
|
twice over:
|
|
|
|
1. **mempalace validates exactly one token.** `hmac.compare_digest(provided, f"Bearer {srv.auth_token}")`
|
|
(`mcp_server.py:5292-5295`) — there is no user table and no second credential. Per-device HTTP identity
|
|
is not a configuration you can express today; it is Phase 4 work (a server-side
|
|
`token → {device_id, scopes}` registry). RFC §6.2 chose the shared token for Phase 1 deliberately:
|
|
*"iterate more feature rich but more complex solutions over time."*
|
|
|
|
2. **Pangolin's HTTP auth is browser-shaped; the clients are not.** SSO login, resource PIN and resource
|
|
password all assume something that can follow a redirect, render a form and hold a session cookie.
|
|
Every MemPalace client here is a headless JSON-RPC `POST` with an `Authorization` header — pi's
|
|
extension, opencode's `type:remote` MCP entry, and `mempalace-pi-session --mode remote`'s
|
|
`urllib.request.urlopen`. Point those at a user-authenticated resource and they receive a login page
|
|
where JSON should be. Enabling that protection breaks precisely the clients it is meant to protect.
|
|
|
|
So: **Pangolin terminates TLS and nothing more** (RFC §6.2 Transport, decided 2026-08-09). The bearer token
|
|
is the authentication. This is not "unprotected" — an unauthenticated request to `/mcp` gets a 401 from
|
|
mempalace itself, verified A4/A5 in runbook §2.4.
|
|
|
|
**Consequence to accept consciously** (RFC §6.2, §7.3.2): until Phase 4 the primary **cannot tell devices
|
|
apart**. `origin_device` is client-asserted and advisory — nothing load-bearing may depend on it, and
|
|
revoking one laptop means rotating the token everywhere.
|
|
|
|
### 1.2 The one place per-device identity *does* exist today
|
|
|
|
Remote mode is not only HTTP. `mempalace_mine` expands its source path in the **server** process, so a
|
|
client's staged transcripts must physically exist on the primary. The feeder therefore ships them over
|
|
SSH into a **per-device inbox** before asking the server to mine its own copy:
|
|
|
|
```sh
|
|
rsync -a --update -e "$ssh_cmd" "$STAGE/" "${SSH_TARGET%/}/$DEVICE/" # bin/mempalace-pi-session:677-680
|
|
```
|
|
|
|
That SSH key **is** per-device identity, and it is individually revocable (one line out of
|
|
`authorized_keys`) years before Phase 4 lands. It costs nothing extra, because the mining path needs SSH
|
|
regardless.
|
|
|
|
Two implications people miss:
|
|
|
|
- **`mempalace.jordbo.se` alone does not enable mining.** HTTPS covers the read/write tool surface
|
|
(`search`, `add_drawer`, `diary_write`, `kg_*`) — genuinely useful on its own, and the reason to do this
|
|
at all. But `--mode remote` also needs `MEMPALACE_PI_SSH_TARGET` reachable. Budget for both paths.
|
|
- **`DEVICE` defaults to `$(hostname)`** (`bin/mempalace-pi-session:163`). In a container that is the
|
|
container hostname: either random per recreate (inboxes proliferate; each recreate re-mines into a fresh
|
|
empty inbox) or identical across sibling devboxes (two containers writing one inbox). **Set
|
|
`MEMPALACE_PI_DEVICE` explicitly per container.** It is a label, not a secret, so put it somewhere
|
|
reviewable — a committed compose file — where duplicates are visible. That, not username hygiene in
|
|
`.env`, is the discipline this design actually asks of you.
|
|
|
|
---
|
|
|
|
## 2. The bind trap, in full
|
|
|
|
RFC §6.2 and runbook §2.4 already say **do not bind loopback behind the tunnel**, because
|
|
`enforce_host_pin = _http_is_loopback(host)` (`mcp_server.py:5367`) makes a loopback bind reject the
|
|
proxy's forwarded `Host:` with a **403** that reads exactly like a Pangolin misconfiguration.
|
|
|
|
**Additional finding, 2026-08-12 — the same reflex also silently removes authentication.** Token
|
|
resolution in `cmd_serve` (`cli.py:1447-1450`) is:
|
|
|
|
```python
|
|
loopback = _server_is_loopback(host)
|
|
if not token and not loopback and not args.allow_insecure:
|
|
token, token_created = _load_or_create_server_token(palace_path)
|
|
```
|
|
|
|
Auto-minting is gated on the bind being **non-loopback**. A loopback bind therefore starts with **no token
|
|
at all** — no error, no warning, `--allow-insecure` not required — because the server has concluded it is
|
|
only reachable locally, while the tunnel is serving it to the internet. Bind loopback behind newt and you
|
|
get a 403 wall *and*, the moment anything relaxes the Host pin, an unauthenticated palace.
|
|
|
|
Both failure modes have the same cure, already implemented in
|
|
`contrib/systemd/mempalace-serve.service`: **bind `172.17.0.1`**. Non-loopback, so the Host pin relaxes and
|
|
the token is mandatory; docker0-only, so newt reaches it and the LAN does not.
|
|
|
|
> Belt and braces: set `MEMPALACE_MCP_HTTP_TOKEN` explicitly in the unit rather than relying on
|
|
> auto-minting. Then no future bind change can quietly drop authentication.
|
|
|
|
---
|
|
|
|
## 3. Steps
|
|
|
|
Ordered so nothing is reachable before it is authenticated.
|
|
|
|
### 3.1 Start the primary (runbook §4.3 — one `sudo`, unit already staged)
|
|
|
|
```sh
|
|
sudo loginctl enable-linger ecsjper
|
|
cd ~/.config/systemd/user && mv mempalace-serve.service.staged mempalace-serve.service
|
|
systemctl --user daemon-reload && systemctl --user enable --now mempalace-serve
|
|
|
|
curl -s 172.17.0.1:8765/healthz # expect ok
|
|
curl -s 127.0.0.1:8765/healthz # expect 403 — correct, not a bug (§2)
|
|
ss -ltnp | grep 8765 # expect 172.17.0.1:8765 only
|
|
```
|
|
|
|
### 3.2 Collect the shared token
|
|
|
|
```sh
|
|
cat ~/.mempalace/server/f5d849287f6d73f0141b29d7/token
|
|
```
|
|
|
|
Directory name is `sha256(realpath(palace))[:24]` — it changes if the palace path ever moves. Store via the
|
|
`.env.age` flow, 0600 (RFC §6.2).
|
|
|
|
### 3.3 newt on synlig
|
|
|
|
synlig runs Docker (Gitea Actions runner + digikam) but **no tunnel client** — runbook §4.2. Pangolin on
|
|
nyvaken cannot dial in; synlig must dial out. Add a `newt` container with the credentials Pangolin issues
|
|
for a new site.
|
|
|
|
Because newt runs in Docker on this box, the docker0 bind is already correct for it: from inside the
|
|
container the primary is `172.17.0.1:8765`. Verify from *inside* newt's network namespace, not from the
|
|
host, before touching DNS.
|
|
|
|
> synlig has 7.8 GiB shared with a CI runner (runbook §1). newt is small, but do not colocate anything
|
|
> else here casually.
|
|
|
|
### 3.4 DNS at the web hotel
|
|
|
|
One CNAME: `mempalace` → **the same target your existing Pangolin resources use** (nyvaken's public
|
|
hostname). RFC §6.2 costed this as *"one DNS record per service on the web hotel is the whole setup cost."*
|
|
|
|
⚠️ Not verified from here: nyvaken's public FQDN, and whether your web hotel permits a CNAME at that label
|
|
(some require an A record, or forbid CNAME where other records exist). Confirm before assuming a 5-minute job.
|
|
|
|
### 3.5 Pangolin resource
|
|
|
|
- Target: newt site → `172.17.0.1:8765`, path `/mcp` (plus `/healthz` if you want the external probe).
|
|
- **Auth: none at the Pangolin layer** (§1.1). TLS termination only.
|
|
- Do **not** attach an `Origin`-injecting proxy or browser client: a *present* non-loopback `Origin` is a
|
|
hard 403 with no override (runbook §2.4 B3).
|
|
|
|
### 3.6 Verify end-to-end before flipping any client
|
|
|
|
```sh
|
|
curl -s https://mempalace.jordbo.se/healthz # ok
|
|
curl -s -o /dev/null -w '%{http_code}\n' -X POST \
|
|
https://mempalace.jordbo.se/mcp # 401 — the token is doing its job
|
|
curl -s -X POST https://mempalace.jordbo.se/mcp \
|
|
-H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
|
|
-d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | head -c 300 # 36 tools
|
|
```
|
|
|
|
The 401 check matters as much as the 200: it is the only evidence that the thing you just published to the
|
|
internet is not open. Then, and only then, Phase 1 client flip — **one machine first** (RFC §8), and
|
|
remember opencode containers need the §4.1 sidecar merge before their `.env` takes effect.
|
|
|
|
---
|
|
|
|
## 4. Still open
|
|
|
|
- **Per-device tokens** — Phase 4. Until then `origin_device` is advisory (§1.1).
|
|
- **§7.6 diary dedup** must be settled *before* the first §4.4 join; replay duplicates every entry.
|
|
- **§7.2**: never run `mempalace sync` against the shared palace. Doubly true now that the pi/opencode
|
|
feeders stage *inside* the palace root, which puts staged sources in scope for a sync of the palace dir.
|
|
- **nyvaken's public FQDN and the web hotel's CNAME rules** — unverified (§3.4).
|