joakimp/mempalace-toolkit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: reconcile the RFC-001 docs with what is actually deployed

Browse Source 
Audit of every doc touching the global-palace rollout against the running
fleet. Each correction below was verified against the filesystem or the host,
not against another doc:

- synlig-primary-runbook: the decommission `rm -rf ~/.mempalace` now carries a
  STOP block. That tree holds the fleet palace *and* the only copy of the
  bearer token every client authenticates with; the old "empty today" comment
  stopped being true when the palace was seeded on 2026-08-14. Adds an ordered
  safe decommission, and drops count-based join verification.

- phase-1-exposure-runbook: new S3.8, how to verify a flip actually took --
  the procedure that until now existed only in an untracked handover file.
  Three claims that fail independently (env var / curl / the palace-path
  discriminator) plus an explicit list of checks that produce FALSE POSITIVES:
  drawer counts (both sides were seeded from the same palace, and `status`
  counts chunks not drawers), write-then-read through the same transport, and
  the `mempalace` CLI -- which has no remote support at all, so post-flip it
  reads the dead local archive and reports success.

- rfc-001: status Draft -> Phases 0-1 implemented. Records that the join was a
  file-level copy, which SIDESTEPPED the S7.6 diary-dedup question rather than
  answering it -- so S7.6 remains a hard blocker for the second machine, which
  is the one that will actually exercise merge semantics.

- ARCHITECTURE, SKILL, contrib/README, extensions/pi/README all claimed pi
  feeds the palace automatically, unconditionally. That is gated on
  mempalace-toolkit >= 29e660e and every deployed image predates it, so the
  claim is currently false fleet-wide. Each site now states the gate plus a
  check that inspects the *deployed* file rather than repo HEAD.

- extensions/pi/README: plaintext http://mempalace.lan example -> https
  endpoint; the two transports are either/or (no dual-write, no local mirror);
  the bridge fails CLOSED, so "the agent has no mempalace_* tools" is the
  expected symptom of a server/token/DNS fault, not of a broken install.

- contrib/README: documents mempalace-serve.service, which this directory has
  shipped since day one without explaining it (linger, the load-bearing
  172.17.0.1 bind and why loopback is the unsafe-looking-safe option, the
  token path, and an uninstall warning).

- Fixes a pre-existing stray ```sh fence that was swallowing S3.2's heading and
  the token command into a code block.

Docs only; no behaviour change.
This commit is contained in:
Joakim Persson
2026-08-14 22:57:22 +02:00
parent 2293f1c89b
commit ec436ed3ad
7 changed files with 443 additions and 35 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/rfc-001-global-palace.md
+100 -12
View File
@@ -2,11 +2,12 @@
| | |
| --- | --- |
| **Status** | Draft — design agreed, not implemented |
| **Status** | **Phases 0–1 implemented and verified — 2026-08-10 / 08-12 / 08-14.** Primary live + seeded, first client flipped. Phases 1.5, 2 and 4 outstanding; Phase 3 deferred by decision. |
| **Created** | 2026-08-08 |
| **Applies to** | mempalace 3.6.0, mempalace-toolkit @ `96699f2`, pi-devbox ≥ v1.3.0 |
| **Decision** | Phases 0–2 + 4 in scope. **Phase 3 (full pull replication) explicitly deferred** — "a laptop that can reach its own stuff plus whatever it can reach" is good enough. **Centralization is strictly opt-in — solitary devbox operation remains the default and must not change (§1.1).** |
| **Recon update** | **2026-08-09 — §9 Q1 and Q6 are RESOLVED, both in the permissive direction** (opencode supports remote MCP; opencode-devbox already templates the mempalace entry). Neither is a blocker. See those entries for evidence; §2, §4.1 and R5 were corrected accordingly. |
| **Rollout update** | **2026-08-14 — Phase 1 is live.** Primary serving at `https://mempalace.jordbo.se/mcp` since 2026-08-12 (synlig, `mempalace-serve.service` under `systemctl --user`, palace `/home/ecsjper/.mempalace/palace`); **seeded 2026-08-14 15:07** from EMB-7KJ4VR4G's palace — 14,777 drawers / 9 wings / 16,337 embeddings / KG 46 entities, 34 triples (14,803 drawers by 17:00); **first client flipped and verified end-to-end** the same afternoon. ⚠️ Two things not to misread: the seed was a **file-level copy of one palace**, *not* the §4.4 MCP replay (see the Deviation note in §4.4), and it therefore **sidestepped §7.6 rather than resolving it — §7.6 remains a hard blocker for the *second* joiner (§8 Phase 0).** §2's and §4.1's predictions about the pi client were confirmed in production; §4.4's count-based verification advice was **wrong** and has been corrected. |
**Read this first if you are asked to "centralize MemPalace" / "sync palaces between machines".** Most of the
hard-won facts below are non-obvious and two of them are actively destructive if you guess wrong
@@ -88,8 +89,9 @@ Two of the three pieces already exist. This is not greenfield.
| Piece | State | Evidence |
| --- | --- | --- |
| **Primary server** | ✅ **Exists.** `mempalace serve --host --port --token --tls-cert --tls-key --read-only --allow-insecure`. Bearer token compared with `hmac.compare_digest`, **mandatory** on non-loopback binds (unless `--allow-insecure`), TLS 1.2+ resolved *before* bind, Host-header pinning + `Origin` allowlist (anti-DNS-rebinding), 16 MiB body cap, token-free `/healthz`. | `cli.py:cmd_serve` (~1448); `mcp_server.py:5205-5215`, `5284-5289` |
| **Remote client** | ⚠️ **Exists for pi only, and it is either/or.** `createClient()` picks stdio *or* HTTP once at process start. | `extensions/pi/mempalace.ts:629-641` |
| **Fallback + resync** | ❌ **Absent everywhere.** On remote failure the pi bridge re-handshakes the same URL, then **de-registers all palace tools** and runs blind. | `extensions/pi/mempalace.ts:665-673` |
| **Remote client** | ⚠️ **Exists for pi only, and it is either/or.** `createClient()` picks stdio *or* HTTP once at process start. ✅ **Confirmed in production 2026-08-14** on the first flipped client: the branch is a pure transport swap, and in remote mode **no local `mempalace-mcp` child is spawned at all** (zero mempalace processes in the flipped container). So **writes go only to the primary — no dual-write, no local mirror.** | `extensions/pi/mempalace.ts:629-641` |
| **Fallback + resync** | ❌ **Absent everywhere.** On remote failure the pi bridge re-handshakes the same URL, then **de-registers all palace tools** and runs blind. ✅ **Confirmed 2026-08-14 — and the precise word is fail-*closed*, not fail-local.** After bounded retries it prints `mempalace-mcp unavailable after retries; continuing without palace tools` and registers nothing; it never silently falls back to the local palace, so a write cannot land in the wrong store. Operational corollary worth stating once: **"the palace tools vanished" is the expected symptom of a server / token / DNS fault**, not of a broken client. | `extensions/pi/mempalace.ts:665-673` |
| **`mempalace` CLI** | ❌ **No remote support whatsoever.** Its only palace selector is `--palace <path>`; otherwise it resolves `palace_path` from the local `config.json`. **Verified 2026-08-14: on a flipped client the CLI still reads and writes the now-dead local archive — and cheerfully reports ~14,777 drawers while doing so.** It is a false-positive machine: never verify a flip with `mempalace status`/`search`, and never file memories with the CLI post-flip (they land in the archive, not the primary). Corollary: a **transcript backfill must be mined on the primary host**, where the CLI's local palace *is* the primary. | `cli.py` argument surface; verified on EMB-7KJ4VR4G |
| **opencode client** | ✅ **Remote is supported and already wired.** opencode's published schema (`https://opencode.ai/config.json`, `$defs.McpRemoteConfig`) makes `{"type":"remote","url","headers","oauth"}` a first-class sibling of `McpLocalConfig`, `headers` being a free string→string map (so bearer is a convention, not a constraint). `opencode-devbox` already emits exactly that entry when `MEMPALACE_REMOTE_URL` is set. The all-`type:local` configs in `myconfigs` are a *deployment* fact, not a capability limit. | `generate-config.py:107-118`; hook at `entrypoint-user.sh:117`; schema `$defs.McpRemoteConfig` |
| **Server compose** | ✅ Exists: `pi-devbox/docker-compose.mempalace.yml` (canonical) + a tor-ms22 derivative (`docker-compose-repo/tor-ms22/pi-devbox/`, `df2c2ae`, port 8766, uid 1000, binds the real `~/.mempalace`). Not enabled. | pi-devbox CHANGELOG v1.3.0 (2026-07-02) |
@@ -103,13 +105,19 @@ Two of the three pieces already exist. This is not greenfield.
>
> **✅ Done 2026-08-12** — `pi-devbox/.env.example` and
> `mempalace-toolkit/extensions/pi/README.md` both now recommend `mempalace serve`, bind docker0 rather
> than `0.0.0.0`/loopback, use an `https://` example URL, and state that the transport *is*
> than `0.0.0.0`/loopback, and state that the transport *is*
> authenticated. `docker-compose.mempalace.yml` audited too, and it was worse than stale — it was
> **broken on 3.6.0 in both directions**: `--host 0.0.0.0` with no token in the environment makes the
> server refuse to start (crash-looping under `restart: unless-stopped`), and once a token *is* supplied
> the healthcheck's unauthenticated `tools/list` POST 401s, marking a healthy server unhealthy forever.
> Fixed: token now required via `${MEMPALACE_REMOTE_TOKEN:?}` (fails fast at `up`), healthcheck switched
> to the token-free `/healthz`.
>
> ⚠️ **Corrected 2026-08-14 — one sub-claim above was wrong for two days.** The `https://` example URL
> landed in `pi-devbox/.env.example` only; `extensions/pi/README.md` still carried
> `http://mempalace.lan:8765/mcp` until it was fixed on **2026-08-14**. Everything else in this block
> checked out. Worth naming as a pattern, because it is cheap to repeat: **a doc's own ✅ is not evidence
> the work landed — verify it per sub-claim and per file.**
### Two things that sound like the feature and are not
@@ -133,6 +141,11 @@ independently disqualifying:
vectors are written to Postgres.*" Cross-architecture ONNX determinism (arm64 macOS vs x86-64
Linux, different execution providers) is not guaranteed. → **Ship text + metadata, always re-embed
on receipt.** Cheap and safe: the raw text is always stored as the chroma document.
⚠️ **Not absolute — see the 2026-08-14 deviation in §4.4.** A whole-palace file-level copy arm64 macOS →
x86-64 Linux *did* preserve working search and an intact HNSW index. Re-embedding on receipt remains the
rule for **merging** operations into an existing store; it is not a prohibition on cloning one palace
wholesale, provided the embedder sidecar travels with it (§7.4).
2. **KG rows cannot be copied.** `triples` has **no** `UNIQUE(subject,predicate,object,valid_from)` —
only `id` is unique, and `make_triple_id` embeds `datetime.now()`. A row copy therefore duplicates
every fact. But `add_triple()` guards at the application level (`SELECT id … WHERE subject=? AND
@@ -315,6 +328,32 @@ precisely to be extended.
container joins the same way, at any time, repeatedly — so a join is **idempotent replay of local
history**, and the only real question per record type is *what dedupes it*.
> ⚠️ **Deviation (2026-08-14) — the first join did not follow this section.** It was precisely the thing
> the framing correction says this is *not*: a **file-level copy of one chosen palace**, arm64 macOS →
> x86-64 Linux, bypassing MCP replay entirely. It worked — search verified live on the primary, HNSW
> index intact, KG intact (46 entities / 34 triples). Recorded here so the next operator neither repeats
> it blind nor believes it is forbidden. **Four conditions made it safe, and all four must hold:**
>
> 1. **Same mempalace version (3.6.0) at both ends**, so the on-disk chroma/HNSW layout matched.
> 2. **`mempalace_embedder.json` travelled with the palace**, so embedder identity matched. This is the
> load-bearing one: a mismatched embedding model does **not** raise — it silently returns garbage
> search results, and §7.4 explains why the guard cannot catch it (only the model *name* is compared,
> and `dimension: 0` is skipped as unknown).
> 3. **A single source palace, so no merge semantics were exercised at all.** This is exactly what makes
> the method inapplicable to the second joiner.
> 4. **Python's `sqlite3` online-backup API** (`src.backup(dst)`) for the two sqlite DBs — WAL-safe with
> a live writer, and the only option available because **neither host has the `sqlite3` CLI**. The
> rest of `palace/` was `rsync`ed with `--exclude 'chroma.sqlite3*'`.
>
> **This sidesteps the merge problem; it does not solve it.** The second machine to join still needs the
> replay path and the dedupe keys tabulated below, and still needs §7.6 settled. Three traps found while
> doing it: **never `rsync --delete` into `~/.mempalace`** — the server's bearer token lives *inside* that
> tree at `~/.mempalace/server/<hash>/token` (§10) and there is no second copy; **palace paths cannot
> move**, because the palace directory name is a sha256 prefix of its own path; and compare sizes with
> **`stat -c%s`, not `du`** (APFS and ext4 disagree on block accounting, so `du` shows a spurious delta).
> Quiesce the writer first — the source palace's own MCP server was live throughout, which is how a write
> landed on the far side of the snapshot boundary and produced the false alarm described below.
| Record type | Dedupe on replay | Client work needed |
| --- | --- | --- |
| `add_drawer` / `checkpoint` drawers | **Server-side**: content-addressed id + pre-write `col.get` probe → `{"reason":"already_exists"}`, no write (`mcp_server.py:2593-2600`) | **None.** Just replay |
@@ -342,9 +381,32 @@ for everything except diaries. Belt and braces:
pattern as the outbox (§4.3). It must **not** live in a container-only path: `~/.mempalace` is not
preserved by default for solitary users (§1.2), which is exactly why the file belongs to the palace
directory.
- **First join is a dry run.** Bootstrap one palace, verify counts (`status`, `kg_stats`, per-wing
`list_drawers`) against expectations, *then* let the rest join. Ordering matters only because of diaries
and closed facts; everything else is order-free.
- **First join is a dry run.** Bootstrap one palace, verify it, *then* let the rest join. Ordering matters
only because of diaries and closed facts; everything else is order-free.
> ⚠️ **Corrected 2026-08-14. This bullet used to say "verify counts (`status`, `kg_stats`, per-wing
> `list_drawers`) against expectations". Do not verify a join by counts.** Two independent reasons,
> both learned on the first real seed:
>
> 1. **`mempalace status` counts chunk rows, not logical drawers.** Three drawers plus one 2-chunk diary
> presented as **+9**. A count delta cannot even tell you how many *records* moved.
> 2. **Chunk counts and chunk-id sets legitimately differ between two palaces** whenever a drawer was
> updated on either side: an update preserves `drawer_id`, **re-chunks to the new length, and deletes
> the surplus chunk rows** (§9.3). A `…_chunk_000007` present on one side and absent on the other is
> therefore the ordinary signature of an **edit**, not of loss.
>
> On 2026-08-14 that second mechanism produced a confident "the seed lost a chunk" conclusion that was
> filed as a finding before being retracted. Counts **hid** the difference (14,777 vs 14,778 looked like
> one lost row); an id-set diff **over-reported** it. The two methods fail in *opposite* directions, so
> agreeing with either one alone proves nothing.
>
> **Verify by content.** For a sample of drawers, `get_drawer(<parent id>)` on *both* palaces and compare
> the **reassembled `content`**. An id-set diff is a fine first pass to *find* candidates, but it must be
> adjudicated by a content comparison before anyone concludes loss. Two cheap decisive checks once a
> difference is real: **is one side a prefix of the other** (that, and only that, is truncation), and
> **does the final chunk's length equal `len(content) - chunk_size * (n_chunks - 1)`** (800 in 3.6.0)? If
> both sides satisfy the arithmetic for their own content, both are complete and you are looking at two
> revisions, not damage.
The genuinely hard case is **two different palaces holding overlapping mined content** — the same repo
mined on a laptop and a workstation under different absolute paths. §5 excludes it by keeping mined wings
@@ -464,10 +526,20 @@ primary, where there is nothing to move):
```sh
ln -sfn palace/knowledge_graph.sqlite3 ~/.mempalace/knowledge_graph.sqlite3
ln -sfn palace/known_entities.json ~/.mempalace/known_entities.json
# hallways.json deliberately NOT symlinked: already palace-derived, and its HOME
# path is a warning-only legacy probe that never auto-migrates (hallways.py:73-95)
ln -sfn palace/hallways.json ~/.mempalace/hallways.json # added 2026-08-14, see note
```
> **Updated 2026-08-14 — `hallways.json` is symlinked too now.** This block previously said it was
> *deliberately* not symlinked, on the narrow grounds that it is already palace-derived and its HOME path
> is a warning-only legacy probe that never auto-migrates (`hallways.py:73-95`). Both facts still hold —
> but the goal changed from "symlink only what the code demands" to **"all real state lives under
> `palace/`, so one copy of `palace/` is a complete copy"**, which is what let the 2026-08-14 file-level
> seed (§4.4) treat the palace as a single self-contained unit. With all three links in place every
> parent-level path resolves, and 3.6.0's *three different* resolution rules — palace-relative for the
> served KG, HOME for the CLI KG, `dirname(palace_path)` for hallways, hardcoded HOME for
> `known_entities.json` — converge on one set of files. Revert by deleting the symlink if it ever causes
> trouble.
Verified on synlig 2026-08-10, because the WAL behaviour was the load-bearing assumption: a **dangling**
symlink is created on first `sqlite3.connect`; `-wal`/`-shm` land next to the **target** (inside the palace
dir, so the palace stays a self-contained backup/bind-mount unit) and *not* beside the symlink; a write
@@ -675,6 +747,13 @@ duplication.
This matters more than it sounds, because diaries are `replicated` (§5) and are precisely the content a
join replays (§4.4).
> ⚠️ **Still open as of 2026-08-14 — and the first join did not test it.** The seed was a file-level copy
> (§4.4 Deviation), which replays no diaries and therefore *cannot* duplicate them. **§7.6 was sidestepped
> by method choice, not resolved.** Neither action below has been built. The moment a *second* palace
> joins — by any replay-based route — this becomes live again, and it is the single most likely thing for a
> future operator to get wrong, because the first join appears to have proved the path safe. It did not:
> it avoided the path.
**Actions:**
1. **Client-side, now:** before replay, list the target's diary drawers for the wing, extract the 12-hex
@@ -692,8 +771,8 @@ join replays (§4.4).
| Phase | Effort | Deliverable |
| --- | --- | --- |
| **0 — hygiene** | hours | §7 runbook: converge the KG/entities store paths **on synlig before first `serve`** (§7.1 — **done 2026-08-10**, runbook §2.3), ban `sync` on shared palaces (§7.2), ~~fix stale "unauthenticated" docs (incl. `pi-devbox/.env.example:21`)~~ — **done 2026-08-12**, and `docker-compose.mempalace.yml` turned out to be outright broken on 3.6.0, now fixed. Added 2026-08-09: settle the **diary dedup** approach and file its upstream ask (§7.6), and **dry-run the join from one palace only** (§4.4). **No provenance work here** — it is not backfill-critical (§7.3.3) and belongs to the stamper, not the agent |
| **1 — primary up** | hours, **no code** | `mempalace serve --token --tls-cert` on a private-net host (reuse `docker-compose.mempalace.yml` — keep it a **separate standalone project**, R4 — and mind port 8765 vs pi-studio; tor-ms22 already moved to 8766). Repoint pi clients via `MEMPALACE_REMOTE_URL`/`MEMPALACE_REMOTE_TOKEN`. **opencode clients can be repointed in the same breath** — remote MCP is supported and `generate-config.py` already emits it (§9.1, §9.6), subject to the sidecar caveat in §4.1. **Shared memory today, no offline.** **Decided 2026-08-09: primary = synlig, TLS at Pangolin, single shared token (§8.1)** — mind the loopback Host-pin trap in §6.2. |
| **0 — hygiene** | hours | §7 runbook: converge the KG/entities store paths **on synlig before first `serve`** (§7.1 — **done 2026-08-10**, runbook §2.3), ban `sync` on shared palaces (§7.2), ~~fix stale "unauthenticated" docs (incl. `pi-devbox/.env.example:21`)~~ — **done 2026-08-12**, and `docker-compose.mempalace.yml` turned out to be outright broken on 3.6.0, now fixed. Added 2026-08-09: settle the **diary dedup** approach and file its upstream ask (§7.6), and **dry-run the join from one palace only** (§4.4). ✅ **Both done 2026-08-14, with one asterisk that matters:** the join was dry-run *and* executed from one palace only — but **§7.6 was sidestepped, not settled** (a file-level copy replays no diaries, so it cannot duplicate them). The §7.6 client-side dedupe and its upstream ask are **still unbuilt and are hard blockers for the second joiner.** **No provenance work here** — it is not backfill-critical (§7.3.3) and belongs to the stamper, not the agent |
| **1 — primary up** ✅ **done 2026-08-14** | hours, **no code** | `mempalace serve --token --tls-cert` on a private-net host (reuse `docker-compose.mempalace.yml` — keep it a **separate standalone project**, R4 — and mind port 8765 vs pi-studio; tor-ms22 already moved to 8766). Repoint pi clients via `MEMPALACE_REMOTE_URL`/`MEMPALACE_REMOTE_TOKEN`. **opencode clients can be repointed in the same breath** — remote MCP is supported and `generate-config.py` already emits it (§9.1, §9.6), subject to the sidecar caveat in §4.1. **Shared memory today, no offline.** **Decided 2026-08-09: primary = synlig, TLS at Pangolin, single shared token (§8.1)** — mind the loopback Host-pin trap in §6.2. |
| **1.5 — opencode env propagation** | hours | Make the `mcp.mempalace` subtree env-authoritative in `generate-config.py`, gated by a generated-value fingerprint (§4.1). Independent of the rest of this RFC. Without it, adopting *or reverting* the opt-in on an existing opencode container needs a manual sidecar merge or a `docker volume rm` — which also blocks R6 reversibility |
| **2 — `mempalace-edge`** | ~1 week | The actual ask: local-first writes + outbox flush + merged reads + per-wing policy. **Not** "fixes opencode" — opencode's *transport* is already fine after Phase 1; what edge adds there is offline/local-first, since `generate-config.py`'s switch is remote **or** local with no failover. **Ships with the §1.2 opt-in wiring (compose + `.env.example` + a third branch in the existing `generate-config.py`) and must pass the R1 acceptance test.** |
| **3 — pull replication** | ~1 week | **DEFERRED (2026-08-08).** Server-side op-log with monotonic seq → each edge a full offline replica. Only needed if a laptop must hold *everything* offline. Accepted trade-off: offline recall = own writes + last-reachable state. |
@@ -742,7 +821,16 @@ for online clients.
re-call with identical oversized content cannot duplicate a legacy pre-#1539 single-row write. The
normal path is therefore safe by construction. What remains untested is the abnormal one: a chunk set
left partial by a crash or kill *mid-upsert*, which would then look present and stay truncated. Worth a
single fault-injection test before trusting bulk replay of oversized drawers.
single fault-injection test before trusting bulk replay of oversized drawers. **Still untested as of
2026-08-14** — the first join did *not* exercise it.
⚠️ **But 2026-08-14 surfaced a benign mechanism that mimics it, and it will generate false alarms during
any join verification (§4.4).** Updating a drawer preserves `drawer_id`, **re-chunks the new content, and
deletes the surplus chunk rows**. So two palaces holding two *revisions* of one drawer legitimately
differ in chunk count and chunk-id set, with zero data loss — and the shorter side returns a clean
"not found" for the chunk ids it no longer needs, which looks exactly like the truncation this entry
warns about. **The absence of a derived chunk id is not evidence of loss.** Distinguish them by prefix
test and final-chunk arithmetic per §4.4, not by id-set membership.
4. **`migrate.py` as a bootstrap tool.** `extract_drawers_from_sqlite()` reads `{id, document,
metadata}` straight out of chroma's SQLite (bypassing the chromadb API) and re-`add`s them with ids
and metadata preserved and **embeddings recomputed** (`migrate.py:326`) — the right shape for a