feeders: stage beside the palace, not in ~/.cache; document Phase 1 exposure
Staging default moves out of ~/.cache to <palace-root>/pi-stage (pi) and <palace-root>/opencode-stage (opencode), resolved with mempalace's own palace-path precedence ($MEMPALACE_PALACE_PATH -> $MEMPAL_PALACE_PATH -> ~/.mempalace/config.json -> ~/.mempalace/palace), then dirname. Why: the convos miner keys dedup on the *staged* path, so a wiped stage plus a sync scoped to include it prunes the drawers mined from those sources -- deleting memories, not a cache. Under ~/.cache that state was reachable by anything treating a cache as disposable. Staging inside the palace makes the coupling structural: the stage cannot be wiped without touching the palace itself. Overrides ($MEMPALACE_PI_STAGE / $MEMPALACE_SESSION_STAGE, --stage) are unchanged. Note the old default had never been created on any host, so this closed a latent hazard, not a live one. Measured, and the docs now claim only this much: sync prunes only within the scope it is given -- wing-only, 1299 scanned / 1299 out of scope / 0 removed; scoped at the palace root, 651 kept / 648 out of scope. The previous blanket "sync prunes every drawer" wording overstated it, which is a liability: the next reader disproves the overstatement and discards the real constraint with it. Also in this change: - cron log dir ~/.cache/mempalace-session -> ~/.cache/mempalace-logs. The stage left that namespace, so the old name now read as "the stage". - AGENTS.md: the convos miner *does* check mtime (verified against upstream convo_miner.py); the previous "no mtime check" claim was wrong. - smoke-test assertions use `mktemp -d` for --sessions-dir. One pointed at /tmp, which still held earlier synthetic transcripts, so a --dry-run exported a fake session into the real stage: --dry-run skips the mine, not the export. docs/phase-1-exposure-runbook.md -- the newt/DNS/auth step that RFC 001 and the synlig runbook leave open (runbook section 4, items 2 and 5). Port 8765 at /mcp, newt targets 172.17.0.1, and the authentication is the single shared bearer token (RFC 6.2, decided 2026-08-09) rather than per-device proxy users. The latter cannot work today: mempalace validates exactly one token, and Pangolin's SSO/PIN/password are browser-shaped while every client here is a headless JSON-RPC POST -- enabling that protection breaks the clients it protects. The per-device axis that *does* exist is the feeder's SSH key + per-device inbox. New finding recorded there: a loopback bind does not merely 403 behind a tunnel (already known, runbook 2.4) -- it also silently starts the server with no token at all, because auto-minting is gated on the bind being non-loopback. extensions/pi/README.md: the HTTP transport IS authenticated as of mempalace 3.6.0; the "sessionless and unauthenticated" note dated from the v1.3.0 era. Closes the RFC section 8 Phase-0 hygiene item.
This commit is contained in:
Joakim Persson
@@ -0,0 +1,185 @@
|
||||
# Phase 1 exposure — newt on synlig, DNS, and the client auth model
|
||||
|
||||
Companion to [`rfc-001-global-palace.md`](./rfc-001-global-palace.md) (design + decisions) and
|
||||
[`synlig-primary-runbook.md`](./synlig-primary-runbook.md) (what is already installed on the primary).
|
||||
This doc covers only the step the other two leave open: **making the primary reachable** — runbook §4
|
||||
items 2 and 5.
|
||||
|
||||
**Status 2026-08-12 — Pangolin updated on nyvaken (done, yours). newt not yet installed on synlig.
|
||||
Nothing exposed. No client `.env` flipped.**
|
||||
|
||||
Read this before touching Pangolin: three of the four questions this step raises were **already decided**
|
||||
in RFC §6.2 on 2026-08-09, and re-deciding them differently is how the fleet ends up in two states.
|
||||
|
||||
---
|
||||
|
||||
## 1. The four questions, answered
|
||||
|
||||
| Question | Answer | Where it was decided |
|
||||
| --- | --- | --- |
|
||||
| Which port? | **8765**, path **`/mcp`** (liveness: `/healthz`) | `cli.py:2141` default; runbook §2.4 |
|
||||
| What does newt target? | **`172.17.0.1:8765`** (docker0), **never** `127.0.0.1` | RFC §6.2 Transport; runbook §2.4 |
|
||||
| Open, or authenticated? | **Authenticated. The primary is never an open public resource.** | RFC §6.2 Network posture |
|
||||
| Per-device credentials? | **No — Phase 1 ships the single shared bearer token.** Per-device tokens are Phase 4. | RFC §6.2 Authentication |
|
||||
|
||||
### 1.1 Why not per-device users at the proxy
|
||||
|
||||
The instinct — "create a Pangolin user per container, put the credentials in each `.env`, keep the
|
||||
usernames distinct" — is the right *goal* (revocation, attribution) reached through the wrong *layer*,
|
||||
twice over:
|
||||
|
||||
1. **mempalace validates exactly one token.** `hmac.compare_digest(provided, f"Bearer {srv.auth_token}")`
|
||||
(`mcp_server.py:5292-5295`) — there is no user table and no second credential. Per-device HTTP identity
|
||||
is not a configuration you can express today; it is Phase 4 work (a server-side
|
||||
`token → {device_id, scopes}` registry). RFC §6.2 chose the shared token for Phase 1 deliberately:
|
||||
*"iterate more feature rich but more complex solutions over time."*
|
||||
|
||||
2. **Pangolin's HTTP auth is browser-shaped; the clients are not.** SSO login, resource PIN and resource
|
||||
password all assume something that can follow a redirect, render a form and hold a session cookie.
|
||||
Every MemPalace client here is a headless JSON-RPC `POST` with an `Authorization` header — pi's
|
||||
extension, opencode's `type:remote` MCP entry, and `mempalace-pi-session --mode remote`'s
|
||||
`urllib.request.urlopen`. Point those at a user-authenticated resource and they receive a login page
|
||||
where JSON should be. Enabling that protection breaks precisely the clients it is meant to protect.
|
||||
|
||||
So: **Pangolin terminates TLS and nothing more** (RFC §6.2 Transport, decided 2026-08-09). The bearer token
|
||||
is the authentication. This is not "unprotected" — an unauthenticated request to `/mcp` gets a 401 from
|
||||
mempalace itself, verified A4/A5 in runbook §2.4.
|
||||
|
||||
**Consequence to accept consciously** (RFC §6.2, §7.3.2): until Phase 4 the primary **cannot tell devices
|
||||
apart**. `origin_device` is client-asserted and advisory — nothing load-bearing may depend on it, and
|
||||
revoking one laptop means rotating the token everywhere.
|
||||
|
||||
### 1.2 The one place per-device identity *does* exist today
|
||||
|
||||
Remote mode is not only HTTP. `mempalace_mine` expands its source path in the **server** process, so a
|
||||
client's staged transcripts must physically exist on the primary. The feeder therefore ships them over
|
||||
SSH into a **per-device inbox** before asking the server to mine its own copy:
|
||||
|
||||
```sh
|
||||
rsync -a --update -e "$ssh_cmd" "$STAGE/" "${SSH_TARGET%/}/$DEVICE/" # bin/mempalace-pi-session:677-680
|
||||
```
|
||||
|
||||
That SSH key **is** per-device identity, and it is individually revocable (one line out of
|
||||
`authorized_keys`) years before Phase 4 lands. It costs nothing extra, because the mining path needs SSH
|
||||
regardless.
|
||||
|
||||
Two implications people miss:
|
||||
|
||||
- **`mempalace.jordbo.se` alone does not enable mining.** HTTPS covers the read/write tool surface
|
||||
(`search`, `add_drawer`, `diary_write`, `kg_*`) — genuinely useful on its own, and the reason to do this
|
||||
at all. But `--mode remote` also needs `MEMPALACE_PI_SSH_TARGET` reachable. Budget for both paths.
|
||||
- **`DEVICE` defaults to `$(hostname)`** (`bin/mempalace-pi-session:163`). In a container that is the
|
||||
container hostname: either random per recreate (inboxes proliferate; each recreate re-mines into a fresh
|
||||
empty inbox) or identical across sibling devboxes (two containers writing one inbox). **Set
|
||||
`MEMPALACE_PI_DEVICE` explicitly per container.** It is a label, not a secret, so put it somewhere
|
||||
reviewable — a committed compose file — where duplicates are visible. That, not username hygiene in
|
||||
`.env`, is the discipline this design actually asks of you.
|
||||
|
||||
---
|
||||
|
||||
## 2. The bind trap, in full
|
||||
|
||||
RFC §6.2 and runbook §2.4 already say **do not bind loopback behind the tunnel**, because
|
||||
`enforce_host_pin = _http_is_loopback(host)` (`mcp_server.py:5367`) makes a loopback bind reject the
|
||||
proxy's forwarded `Host:` with a **403** that reads exactly like a Pangolin misconfiguration.
|
||||
|
||||
**Additional finding, 2026-08-12 — the same reflex also silently removes authentication.** Token
|
||||
resolution in `cmd_serve` (`cli.py:1447-1450`) is:
|
||||
|
||||
```python
|
||||
loopback = _server_is_loopback(host)
|
||||
if not token and not loopback and not args.allow_insecure:
|
||||
token, token_created = _load_or_create_server_token(palace_path)
|
||||
```
|
||||
|
||||
Auto-minting is gated on the bind being **non-loopback**. A loopback bind therefore starts with **no token
|
||||
at all** — no error, no warning, `--allow-insecure` not required — because the server has concluded it is
|
||||
only reachable locally, while the tunnel is serving it to the internet. Bind loopback behind newt and you
|
||||
get a 403 wall *and*, the moment anything relaxes the Host pin, an unauthenticated palace.
|
||||
|
||||
Both failure modes have the same cure, already implemented in
|
||||
`contrib/systemd/mempalace-serve.service`: **bind `172.17.0.1`**. Non-loopback, so the Host pin relaxes and
|
||||
the token is mandatory; docker0-only, so newt reaches it and the LAN does not.
|
||||
|
||||
> Belt and braces: set `MEMPALACE_MCP_HTTP_TOKEN` explicitly in the unit rather than relying on
|
||||
> auto-minting. Then no future bind change can quietly drop authentication.
|
||||
|
||||
---
|
||||
|
||||
## 3. Steps
|
||||
|
||||
Ordered so nothing is reachable before it is authenticated.
|
||||
|
||||
### 3.1 Start the primary (runbook §4.3 — one `sudo`, unit already staged)
|
||||
|
||||
```sh
|
||||
sudo loginctl enable-linger ecsjper
|
||||
cd ~/.config/systemd/user && mv mempalace-serve.service.staged mempalace-serve.service
|
||||
systemctl --user daemon-reload && systemctl --user enable --now mempalace-serve
|
||||
|
||||
curl -s 172.17.0.1:8765/healthz # expect ok
|
||||
curl -s 127.0.0.1:8765/healthz # expect 403 — correct, not a bug (§2)
|
||||
ss -ltnp | grep 8765 # expect 172.17.0.1:8765 only
|
||||
```
|
||||
|
||||
### 3.2 Collect the shared token
|
||||
|
||||
```sh
|
||||
cat ~/.mempalace/server/f5d849287f6d73f0141b29d7/token
|
||||
```
|
||||
|
||||
Directory name is `sha256(realpath(palace))[:24]` — it changes if the palace path ever moves. Store via the
|
||||
`.env.age` flow, 0600 (RFC §6.2).
|
||||
|
||||
### 3.3 newt on synlig
|
||||
|
||||
synlig runs Docker (Gitea Actions runner + digikam) but **no tunnel client** — runbook §4.2. Pangolin on
|
||||
nyvaken cannot dial in; synlig must dial out. Add a `newt` container with the credentials Pangolin issues
|
||||
for a new site.
|
||||
|
||||
Because newt runs in Docker on this box, the docker0 bind is already correct for it: from inside the
|
||||
container the primary is `172.17.0.1:8765`. Verify from *inside* newt's network namespace, not from the
|
||||
host, before touching DNS.
|
||||
|
||||
> synlig has 7.8 GiB shared with a CI runner (runbook §1). newt is small, but do not colocate anything
|
||||
> else here casually.
|
||||
|
||||
### 3.4 DNS at the web hotel
|
||||
|
||||
One CNAME: `mempalace` → **the same target your existing Pangolin resources use** (nyvaken's public
|
||||
hostname). RFC §6.2 costed this as *"one DNS record per service on the web hotel is the whole setup cost."*
|
||||
|
||||
⚠️ Not verified from here: nyvaken's public FQDN, and whether your web hotel permits a CNAME at that label
|
||||
(some require an A record, or forbid CNAME where other records exist). Confirm before assuming a 5-minute job.
|
||||
|
||||
### 3.5 Pangolin resource
|
||||
|
||||
- Target: newt site → `172.17.0.1:8765`, path `/mcp` (plus `/healthz` if you want the external probe).
|
||||
- **Auth: none at the Pangolin layer** (§1.1). TLS termination only.
|
||||
- Do **not** attach an `Origin`-injecting proxy or browser client: a *present* non-loopback `Origin` is a
|
||||
hard 403 with no override (runbook §2.4 B3).
|
||||
|
||||
### 3.6 Verify end-to-end before flipping any client
|
||||
|
||||
```sh
|
||||
curl -s https://mempalace.jordbo.se/healthz # ok
|
||||
curl -s -o /dev/null -w '%{http_code}\n' -X POST \
|
||||
https://mempalace.jordbo.se/mcp # 401 — the token is doing its job
|
||||
curl -s -X POST https://mempalace.jordbo.se/mcp \
|
||||
-H "Authorization: Bearer $TOKEN" -H 'Content-Type: application/json' \
|
||||
-d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | head -c 300 # 36 tools
|
||||
```
|
||||
|
||||
The 401 check matters as much as the 200: it is the only evidence that the thing you just published to the
|
||||
internet is not open. Then, and only then, Phase 1 client flip — **one machine first** (RFC §8), and
|
||||
remember opencode containers need the §4.1 sidecar merge before their `.env` takes effect.
|
||||
|
||||
---
|
||||
|
||||
## 4. Still open
|
||||
|
||||
- **Per-device tokens** — Phase 4. Until then `origin_device` is advisory (§1.1).
|
||||
- **§7.6 diary dedup** must be settled *before* the first §4.4 join; replay duplicates every entry.
|
||||
- **§7.2**: never run `mempalace sync` against the shared palace. Doubly true now that the pi/opencode
|
||||
feeders stage *inside* the palace root, which puts staged sources in scope for a sync of the palace dir.
|
||||
- **nyvaken's public FQDN and the web hotel's CNAME rules** — unverified (§3.4).
|
||||
@@ -100,6 +100,16 @@ Two of the three pieces already exist. This is not greenfield.
|
||||
> `pi-devbox/.env.example:21` still advertises `mempalace-mcp --transport http --host 0.0.0.0 --port
|
||||
> 8765` as the way to serve a shared palace — replace with `mempalace serve --token … --tls-cert …`,
|
||||
> and change the example URL from `http://mempalace.lan:8765/mcp` to `https://`.
|
||||
>
|
||||
> **✅ Done 2026-08-12** — `pi-devbox/.env.example` and
|
||||
> `mempalace-toolkit/extensions/pi/README.md` both now recommend `mempalace serve`, bind docker0 rather
|
||||
> than `0.0.0.0`/loopback, use an `https://` example URL, and state that the transport *is*
|
||||
> authenticated. `docker-compose.mempalace.yml` audited too, and it was worse than stale — it was
|
||||
> **broken on 3.6.0 in both directions**: `--host 0.0.0.0` with no token in the environment makes the
|
||||
> server refuse to start (crash-looping under `restart: unless-stopped`), and once a token *is* supplied
|
||||
> the healthcheck's unauthenticated `tools/list` POST 401s, marking a healthy server unhealthy forever.
|
||||
> Fixed: token now required via `${MEMPALACE_REMOTE_TOKEN:?}` (fails fast at `up`), healthcheck switched
|
||||
> to the token-free `/healthz`.
|
||||
|
||||
### Two things that sound like the feature and are not
|
||||
|
||||
@@ -682,7 +692,7 @@ join replays (§4.4).
|
||||
|
||||
| Phase | Effort | Deliverable |
|
||||
| --- | --- | --- |
|
||||
| **0 — hygiene** | hours | §7 runbook: converge the KG/entities store paths **on synlig before first `serve`** (§7.1 — **done 2026-08-10**, runbook §2.3), ban `sync` on shared palaces (§7.2), fix stale "unauthenticated" docs (incl. `pi-devbox/.env.example:21`). Added 2026-08-09: settle the **diary dedup** approach and file its upstream ask (§7.6), and **dry-run the join from one palace only** (§4.4). **No provenance work here** — it is not backfill-critical (§7.3.3) and belongs to the stamper, not the agent |
|
||||
| **0 — hygiene** | hours | §7 runbook: converge the KG/entities store paths **on synlig before first `serve`** (§7.1 — **done 2026-08-10**, runbook §2.3), ban `sync` on shared palaces (§7.2), ~~fix stale "unauthenticated" docs (incl. `pi-devbox/.env.example:21`)~~ — **done 2026-08-12**, and `docker-compose.mempalace.yml` turned out to be outright broken on 3.6.0, now fixed. Added 2026-08-09: settle the **diary dedup** approach and file its upstream ask (§7.6), and **dry-run the join from one palace only** (§4.4). **No provenance work here** — it is not backfill-critical (§7.3.3) and belongs to the stamper, not the agent |
|
||||
| **1 — primary up** | hours, **no code** | `mempalace serve --token --tls-cert` on a private-net host (reuse `docker-compose.mempalace.yml` — keep it a **separate standalone project**, R4 — and mind port 8765 vs pi-studio; tor-ms22 already moved to 8766). Repoint pi clients via `MEMPALACE_REMOTE_URL`/`MEMPALACE_REMOTE_TOKEN`. **opencode clients can be repointed in the same breath** — remote MCP is supported and `generate-config.py` already emits it (§9.1, §9.6), subject to the sidecar caveat in §4.1. **Shared memory today, no offline.** **Decided 2026-08-09: primary = synlig, TLS at Pangolin, single shared token (§8.1)** — mind the loopback Host-pin trap in §6.2. |
|
||||
| **1.5 — opencode env propagation** | hours | Make the `mcp.mempalace` subtree env-authoritative in `generate-config.py`, gated by a generated-value fingerprint (§4.1). Independent of the rest of this RFC. Without it, adopting *or reverting* the opt-in on an existing opencode container needs a manual sidecar merge or a `docker volume rm` — which also blocks R6 reversibility |
|
||||
| **2 — `mempalace-edge`** | ~1 week | The actual ask: local-first writes + outbox flush + merged reads + per-wing policy. **Not** "fixes opencode" — opencode's *transport* is already fine after Phase 1; what edge adds there is offline/local-first, since `generate-config.py`'s switch is remote **or** local with no failover. **Ships with the §1.2 opt-in wiring (compose + `.env.example` + a third branch in the existing `generate-config.py`) and must pass the R1 acceptance test.** |
|
||||
|
||||
@@ -128,7 +128,13 @@ standing up a network-reachable service while you were asleep was not mine to de
|
||||
|
||||
## 4. Tomorrow, in order
|
||||
|
||||
1. **Pangolin update on nyvaken** (yours).
|
||||
> **2026-08-12: items 1–2 and 5 now have their own runbook —**
|
||||
> [`phase-1-exposure-runbook.md`](./phase-1-exposure-runbook.md). Pangolin on nyvaken is updated (done).
|
||||
> newt is still missing on synlig. That doc also records why per-device Pangolin users are the wrong layer,
|
||||
> and an additional loopback finding: a loopback bind does not merely 403, it also silently starts the
|
||||
> server with **no token at all** (auto-minting is gated on the bind being non-loopback).
|
||||
|
||||
1. **Pangolin update on nyvaken** (yours). ✅ done 2026-08-12.
|
||||
2. **⚠️ synlig has no tunnel client.** `docker ps` shows only the Gitea runner and digikam — no `newt`.
|
||||
Pangolin on nyvaken cannot reach synlig by itself; synlig needs a newt container (or an equivalent
|
||||
outbound tunnel) added. Easy to miss because Pangolin will look healthy on its own side. If newt runs
|
||||
|
||||
Reference in New Issue
Block a user