joakimp/mempalace-toolkit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: reconcile the RFC-001 docs with what is actually deployed

Browse Source 
Audit of every doc touching the global-palace rollout against the running
fleet. Each correction below was verified against the filesystem or the host,
not against another doc:

- synlig-primary-runbook: the decommission `rm -rf ~/.mempalace` now carries a
  STOP block. That tree holds the fleet palace *and* the only copy of the
  bearer token every client authenticates with; the old "empty today" comment
  stopped being true when the palace was seeded on 2026-08-14. Adds an ordered
  safe decommission, and drops count-based join verification.

- phase-1-exposure-runbook: new S3.8, how to verify a flip actually took --
  the procedure that until now existed only in an untracked handover file.
  Three claims that fail independently (env var / curl / the palace-path
  discriminator) plus an explicit list of checks that produce FALSE POSITIVES:
  drawer counts (both sides were seeded from the same palace, and `status`
  counts chunks not drawers), write-then-read through the same transport, and
  the `mempalace` CLI -- which has no remote support at all, so post-flip it
  reads the dead local archive and reports success.

- rfc-001: status Draft -> Phases 0-1 implemented. Records that the join was a
  file-level copy, which SIDESTEPPED the S7.6 diary-dedup question rather than
  answering it -- so S7.6 remains a hard blocker for the second machine, which
  is the one that will actually exercise merge semantics.

- ARCHITECTURE, SKILL, contrib/README, extensions/pi/README all claimed pi
  feeds the palace automatically, unconditionally. That is gated on
  mempalace-toolkit >= 29e660e and every deployed image predates it, so the
  claim is currently false fleet-wide. Each site now states the gate plus a
  check that inspects the *deployed* file rather than repo HEAD.

- extensions/pi/README: plaintext http://mempalace.lan example -> https
  endpoint; the two transports are either/or (no dual-write, no local mirror);
  the bridge fails CLOSED, so "the agent has no mempalace_* tools" is the
  expected symptom of a server/token/DNS fault, not of a broken install.

- contrib/README: documents mempalace-serve.service, which this directory has
  shipped since day one without explaining it (linger, the load-bearing
  172.17.0.1 bind and why loopback is the unsafe-looking-safe option, the
  token path, and an uninstall warning).

- Fixes a pre-existing stray ```sh fence that was swallowing S3.2's heading and
  the token command into a code block.

Docs only; no behaviour change.
This commit is contained in:
Joakim Persson
2026-08-14 22:57:22 +02:00
parent 2293f1c89b
commit ec436ed3ad
7 changed files with 443 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/phase-1-exposure-runbook.md
+112 -2
View File
@@ -5,6 +5,17 @@ Companion to [`rfc-001-global-palace.md`](./rfc-001-global-palace.md) (design +
This doc covers only the step the other two leave open: **making the primary reachable** — runbook §4
items 2 and 5.
> **Status 2026-08-14 — DONE. Exposed, seeded, and one client flipped.**
> `https://mempalace.jordbo.se/mcp` has been serving since 2026-08-12 (§3.3–§3.6 are all ✅ below).
> The palace was seeded 2026-08-14 15:07 from EMB-7KJ4VR4G (14,777 → 14,803 drawers), and that
> machine's pi-devbox container is flipped and **verified end-to-end — see §3.8**, which is the
> verification procedure that did not exist when the first flip was performed.
> Still outstanding: the transcript feeder is **inert on every deployed image** (§3.7), and §7.6 is
> still a hard blocker for the *second* machine to join (§4).
Original status, kept for the record — it contradicted this file's own ✅ section markers for two days,
which is the failure mode a file-top status block invites:
**Status 2026-08-12 — Pangolin updated on nyvaken (done, yours). newt not yet installed on synlig.
Nothing exposed. No client `.env` flipped.**
@@ -158,7 +169,6 @@ bound **to loopback**, receiving a proxy-forwarded foreign `Host:` (§2, verifie
docker0-only bind you cannot get 403 from loopback, because you never get far enough to send a header.
Refusal is the stronger signal of the two: it proves the loopback and LAN surface is not listening at all.
If it *hangs* instead, or `ss` shows `0.0.0.0:8765`, stop — that is not this configuration.
```sh
### 3.2 Collect the shared token
@@ -325,6 +335,94 @@ mempalace-pi-session --reason manual-check; echo "exit=$?" # exit=0 and a file
cat ~/.pi/agent/mempalace-catchup.log # missing file = the entrypoint skipped
```
### 3.8 Verify the flip actually took — ✅ done 2026-08-14 on EMB-7KJ4VR4G
§3.6 verifies the *endpoint* before you flip. §3.7 tells you *how* to flip. Neither verifies the thing
you actually care about afterwards: **that the agent's palace tools are now talking to synlig.** That
gap is why the first flip was "done" for an hour before anyone could say whether it had worked.
There are **three separate claims** here and they fail independently. Check them in order; each one is
cheap and rules out a different fault.
**(a) Did the container receive the variable?** In a shell *inside* the container:
```sh
env | grep MEMPALACE_REMOTE_URL # -> https://mempalace.jordbo.se/mcp
```
⚠ Do **not** run a bare `env | grep MEMPALACE` — that prints the bearer token into your scrollback.
If the variable is absent, the cause is almost always §3.7's edit not having been applied: `env_file`
is read at container **create** time and baked into the container config, so **`docker compose up -d`
is required; `docker compose restart` silently reuses the old config.** Evidence from 2026-08-14 —
running container config-hash `3a55e09ac19e0118` vs compose-computed `8a7e0cd4a677a9c1`; a differing
hash is what makes `up -d` recreate. `--force-recreate` is not needed.
**Never `docker compose down -v`** to "pick up" a change: on pi-devbox that destroys seven named
volumes, including `devbox-pi-config` (pi's config **and every session transcript**), `devbox-uv`, and
`devbox-chroma-cache` (a large embedding-model re-download).
**(b) Is the server reachable and the token accepted?** Still inside the container — this proves the
network path and the credential, independently of any agent:
```sh
curl -s -X POST "$MEMPALACE_REMOTE_URL" \
-H "Authorization: Bearer $MEMPALACE_REMOTE_TOKEN" \
-H 'Content-Type: application/json' \
-d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | head -c 200
```
`401` = token problem (compare `md5sum` of the client value against synlig's token file — trailing
whitespace from an editor is the classic cause). Connection failure = DNS/tunnel, not auth.
**(c) Did the agent's bridge actually switch?** This is the claim that matters and the one that cannot
be checked from bash — there is no log file. Ask the agent running in the container for
`mempalace_status` and read the palace path it reports:
```
sqlite_integrity.palace = /home/ecsjper/.mempalace/palace # ← proves remote
```
**This is the cheapest and strongest discriminator: that path cannot exist inside the container**,
whose user is `developer` with `HOME=/home/developer` and whose local palace is
`/home/developer/.mempalace/palace`. One call, no token in scrollback, no writes. If instead it
reports the local path, the bridge fell back: `createClient()` reads `MEMPALACE_REMOTE_URL` **once at
load**, so re-check (a) and restart the agent, not just the container.
#### What does *not* verify the flip — read this before inventing your own check
- **❌ A drawer count.** Central was seeded *from* the client's own palace, so both report ~14,777.
Counts cannot tell the two apart. Worse, `mempalace status` counts **chunk rows, not logical
drawers** (three drawers plus a 2-chunk diary presented as +9), so a delta does not even mean what
it looks like. Never reason about palace identity or contents from a count.
- **❌ Writing a drawer through the palace tools and reading it back through the same tools.** This
succeeds *identically whether or not the flip worked* — both palaces are healthy and were seeded
from the same source, so a write-then-read round-trips either way. It only becomes evidence if the
drawer is read back over a **different transport** (the `curl` in (b), by drawer id) or is proven
**absent** from the local sqlite. This is the same false-positive class as the CLI below, one layer
up, and it is an easy trap to fall into precisely because it feels like an end-to-end test.
- **❌ The `mempalace` CLI, in any form.** The CLI has **no remote support whatsoever** — its only
selector is `--palace <path>` — so it reads and writes the LOCAL on-disk archive via
`config.json`. After a flip that archive is dead, yet `mempalace status` / `mempalace search` will
cheerfully report ~14,777 drawers and look exactly like success. It is a false-positive machine.
**Corollary that bites later:** memories filed with the CLI after a flip land in the dead archive,
not in central, and a transcript backfill must therefore be mined **on synlig**, where the CLI's
local palace *is* the central one.
#### Expected failure behaviour, so you can recognise it
The bridge is **fail-closed, not fail-local.** If synlig is unreachable, DNS fails, or the token is
rejected, the extension retries a bounded number of times, prints `mempalace-mcp unavailable after
retries; continuing without palace tools`, and **does not register the palace tools at all**. It does
not silently write to the local palace; there is no dual-write and no local mirror, and in remote mode
no local `mempalace-mcp` process is spawned (verified: zero mempalace processes in the flipped
container). So **"the agent has no `mempalace_*` tools" is the expected symptom of a server, token, or
DNS fault** — not of a broken container. Diagnose with (b).
#### Rollback — ~30 seconds, loses nothing
Comment out `MEMPALACE_REMOTE_URL` in `.env`, then `docker compose up -d`. The bridge falls back to the
local stdio palace, which is intact. Note the local archive is **frozen, not empty**: it stops at the
moment of the flip, so anything the agent filed into central since then will not be there.
---
## 4. Still open
@@ -334,7 +432,19 @@ cat ~/.pi/agent/mempalace-catchup.log # missing file = th
path it must first rsync there. HTTPS would then be genuinely sufficient and mining would work from any
network. Until then, mining is limited to devices that can reach synlig's SSH.
- **Per-device tokens** — Phase 4. Until then `origin_device` is advisory (§1.1).
- **§7.6 diary dedup** must be settled *before* the first §4.4 join; replay duplicates every entry.
- **§7.6 diary dedup** must be settled *before* the **next** §4.4 join; replay duplicates every entry.
⚠ **2026-08-14 — the first join did not resolve this, it SIDESTEPPED it.** The seed was a file-level
copy of one palace, which replays no diaries and therefore cannot duplicate them. The success of
that join is *not* evidence the replay path is safe — it never exercised it. §7.6 remains a hard
blocker for the second machine, which is the one that will actually need merge semantics.
- **The transcript feeder is inert on every deployed image**, so nothing is being mined automatically
anywhere in the fleet (§3.7). Regaining it needs a **new tagged pi-devbox release** — its CI
publishes only on `v*` tags and the latest tag *is* the currently deployed image — built on
pi-devbox ≥ `7c00dd6` **and** mempalace-toolkit ≥ `29e660e`. Both are required: the first restores
the container-start catch-up, the second is where extension-side feeding was implemented at all.
- **Whether *native* pi on a flipped machine was also flipped** is a per-machine question nobody has
checked. If the container is flipped and native pi is not, that machine writes into two different
palaces and its memory silently splits in two. Verify per machine with the §3.8(c) path check.
- **§7.2**: never run `mempalace sync` against the shared palace. Doubly true now that the pi/opencode
feeders stage *inside* the palace root, which puts staged sources in scope for a sync of the palace dir.
- **nyvaken's public FQDN and the web hotel's CNAME rules** — unverified (§3.4).