diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index 0c52fe7..2ab2f9b 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -32,10 +32,15 @@ So on a machine using opencode + the "docs-first palace hygiene" policy, three g The two wrappers in `bin/` close gaps **1** and **2**. Gap **3** is upstream work (see §6). (Pi is a different story on gap 3: `bin/mempalace-pi-session` closes pi's -version of gap 2 the same way `mempalace-session` closes opencode's, but pi's -gap 3 is already closed **without** any upstream dependency — its bridge +version of gap 2 the same way `mempalace-session` closes opencode's, and pi's +gap 3 is closed **without** any upstream dependency — its bridge extension self-triggers the feed on `session_shutdown` and a debounced -`agent_settled`. See §3's `mempalace-pi-session` subsection and +`agent_settled`. **Version gate: extension-side feeding landed in +mempalace-toolkit `29e660e` (2026-08-12), so a copy baked into an older +container image cannot feed at all** — check the deployed file, not repo HEAD: +`grep -c MEMPALACE_FEED "$(readlink -f ~/.pi/agent/extensions/mempalace.ts)"`. +Zero means that machine still needs `contrib/`'s scheduled recipes. See §3's +`mempalace-pi-session` subsection and [`extensions/pi/README.md`](extensions/pi/README.md).) --- @@ -433,7 +438,7 @@ These gaps should ideally close upstream, making the wrappers thinner or obsolet 3. **Opencode harness in `hooks_cli.py`** — mempalace's hooks CLI only knows `claude-code` + `codex` today. Adding `opencode` would let the auto-save diary path work on opencode too. Pairs with #2 above. 4. **SQLite mode for `mempalace mine --mode convos`** — if upstream ever adds direct SQLite ingest for opencode, `mempalace-session` loses its reason to exist (the export-to-JSONL dance goes away). -When #1 merges, retire `mempalace-docs` to a thin shim. When #2 + #3 land together, `mempalace-session` becomes a manual-only fallback (cron / backfill) while hooks handle live saves. (`mempalace-pi-session` has no equivalent entry here: pi's bridge extension already self-triggers the feed with no upstream dependency — see §3.) +When #1 merges, retire `mempalace-docs` to a thin shim. When #2 + #3 land together, `mempalace-session` becomes a manual-only fallback (cron / backfill) while hooks handle live saves. (`mempalace-pi-session` has no equivalent entry here: pi's bridge extension already self-triggers the feed with no upstream dependency — see §3 — **provided the deployed copy is ≥ `29e660e` (2026-08-12); older baked images cannot feed, so `contrib/`'s schedulers remain load-bearing there**.) Separately tracked in [`docs/rfc-001-global-palace.md`](docs/rfc-001-global-palace.md): moving from one palace *per machine* to a **single primary palace with per-machine local fallback** (`mempalace-edge`). That RFC also records upstream items of its own — server-side `origin_device` provenance stamped from a per-device credential, per-wing ACLs, a `mempalace_kg_supersede` tool-classification fix, and a guard against running `mempalace sync` on a shared palace. diff --git a/SKILL.md b/SKILL.md index 7e2f1a3..9a6f315 100644 --- a/SKILL.md +++ b/SKILL.md @@ -20,11 +20,17 @@ The `mempalace` skill covers *using* the palace (search, diary, KG). This skill Both follow the same **stage-to-cache-then-mine** idiom — they curate input into `~/.cache/…//`, then delegate to `mempalace mine`. -**Pi is out of scope for this skill.** The pi coding-agent has its own -wrapper, `mempalace-pi-session`, but — unlike opencode — pi's bridge +**Pi is out of scope for this skill — conditionally.** The pi coding-agent has +its own wrapper, `mempalace-pi-session`, but — unlike opencode — pi's bridge extension (`extensions/pi/mempalace.ts`) invokes it automatically on session -shutdown and a debounced mid-session tick, so a pi machine with that -extension installed needs none of this skill's manual/scheduled recipe. See +shutdown and a debounced mid-session tick, so a pi machine running +**mempalace-toolkit ≥ `29e660e` (2026-08-12)** needs none of this skill's +manual/scheduled recipe. Older installs — and any container image baked before +that date — **do** still need it: having the extension is necessary but not +sufficient, because the pre-`29e660e` extension has no feed path at all. +Check the *deployed* file rather than the repo: +`grep -c MEMPALACE_FEED "$(readlink -f ~/.pi/agent/extensions/mempalace.ts)"` +— zero means fall back to the recipes below. See `extensions/pi/README.md` § Automatic transcript feeding in the same repo. ## When to Load This Skill diff --git a/contrib/README.md b/contrib/README.md index fd3fecb..618d0dd 100644 --- a/contrib/README.md +++ b/contrib/README.md @@ -1,11 +1,18 @@ # contrib/ — automation recipes for `mempalace-session` and `mempalace-pi-session` -Manual invocation of the session-mining wrappers is fine on a machine you actively drive. For long-running devboxes, a weekly automated mine keeps the palace fresh without thinking about it. This directory ships ready-to-use templates for two common scheduling mechanisms, for each wrapper. +Manual invocation of the session-mining wrappers is fine on a machine you actively drive. For long-running devboxes, a weekly automated mine keeps the palace fresh without thinking about it. This directory ships ready-to-use templates for two common scheduling mechanisms, for each wrapper — plus `systemd/mempalace-serve.service`, which is not a mining job at all but the **shared-palace server** ([its own section below](#mempalace-serveservice--the-shared-palace-server)). > **pi machines: check whether you need this at all.** If the pi bridge -> extension (`extensions/pi/mempalace.ts`) is installed, it already feeds the +> extension (`extensions/pi/mempalace.ts`) is installed **and is ≥ `29e660e` +> (2026-08-12)**, it already feeds the > palace by itself on `session_shutdown` and a debounced `agent_settled` — > see [`extensions/pi/README.md` § Automatic transcript feeding](../extensions/pi/README.md#automatic-transcript-feeding). +> ⚠️ **"Installed" is not enough** — the pre-`29e660e` extension has no feed +> path at all, and a container image baked before that date ships exactly that +> copy. Check the *deployed* file, not the repo clone: +> `grep -c MEMPALACE_FEED "$(readlink -f ~/.pi/agent/extensions/mempalace.ts)"` +> — zero means the templates below are **not** optional on that machine. As of +> 2026-08-14 the entire pi-devbox fleet returns zero. > The templates below were written when scheduling was the *only* path for > both harnesses; that's still true for **opencode** (no such extension > exists), but for pi they're now a fallback — useful for a bare pi install @@ -29,11 +36,78 @@ Pick **one scheduler** (systemd *or* launchd *or* cron). The opencode and pi job | `cron/mempalace-session.cron` | opencode → palace | Mon 03:00 | | `cron/mempalace-pi-session.cron` | pi → palace | Tue 03:00 | | `cron/mempalace-session-devbox.cron` | opencode (devbox) → palace | Mon 03:00 | +| `systemd/mempalace-serve.service` | **not a mining job** — runs the shared palace *server* | always-on | The pi variants are drop-in copies of the opencode variants with script name and schedule updated; the install recipes below apply equally — just swap `mempalace-session` for `mempalace-pi-session` and the schedule day. --- +## `mempalace-serve.service` — the shared palace server + +The odd one out in this directory: every other template *feeds* a palace on a schedule, this one +**serves** a palace over HTTP so several machines can share it (RFC-001). **This unit currently runs +the fleet primary on `synlig`** — serving since 2026-08-12, seeded 2026-08-14, and the palace behind +it is the only copy. Read [`docs/rfc-001-global-palace.md`](../docs/rfc-001-global-palace.md) and +[`docs/phase-1-exposure-runbook.md`](../docs/phase-1-exposure-runbook.md) before installing a second one. + +It is a **user** unit (`systemctl --user`), so it dies with your login session unless lingering is +enabled — that is the one `sudo` this recipe needs: + +```sh +# Install +mkdir -p ~/.config/systemd/user +cp contrib/systemd/mempalace-serve.service ~/.config/systemd/user/ + +sudo loginctl enable-linger "$USER" # else the server stops when you log out +systemctl --user daemon-reload +systemctl --user enable --now mempalace-serve + +# Verify — both lines matter +curl -s 172.17.0.1:8765/healthz # -> ok +curl -s localhost:8765/healthz # -> connection refused (exit 7), and that is CORRECT + +# Logs +journalctl --user -u mempalace-serve -f +``` + +The unit refuses to start if `~/.mempalace/palace` does not exist (`ConditionPathExists`) — better a +clear failure than a server quietly creating an empty palace somewhere unexpected. + +**Why it binds `172.17.0.1` and not loopback**, since this looks backwards and is the single most +load-bearing line in the file: mempalace pins the HTTP `Host` header to loopback literals *only on a +loopback bind*, so a `127.0.0.1` server behind a reverse proxy 403s every proxied request — **and** +token auto-minting is gated on the bind being non-loopback, so the "safe-looking" loopback bind starts +with **no authentication at all and no warning**. `172.17.0.1` is the docker0 gateway: non-loopback +(so the pin relaxes and a token is minted), reachable from this host and its containers, not from the +LAN. Point the tunnel/proxy at **`http://172.17.0.1:8765`** — not `https://` — because there is no +`--tls-cert` here; TLS belongs at the proxy. Targeting `https://` yields 502 from outside while local +curl still says `ok`. + +**The token.** There is deliberately no `--token` in the unit (units are world-readable). `serve` +mints or reuses a `0600` token at `~/.mempalace/server//token`, stable +across restarts. Read it from there to configure clients. + +> ⚠️ **Uninstall is where this unit differs from every other template here.** Stopping it is safe; +> deleting its data is not. +> +> ```sh +> systemctl --user disable --now mempalace-serve +> rm ~/.config/systemd/user/mempalace-serve.service && systemctl --user daemon-reload +> ``` +> +> **Do not `rm -rf ~/.mempalace` on a host that has served the fleet.** That tree holds the shared +> palace *and* the only copy of the bearer token every client authenticates with. For the same reason, +> **never `rsync --delete` into `~/.mempalace`** — the token lives inside the tree you would be +> syncing. And note palace directories cannot simply be moved: the directory name is a sha256 prefix +> of its own path. +> +> Clients **fail closed** when this unit is down — they lose their palace tools entirely rather than +> falling back to a local palace — so stopping it is visible, reversible, and loses no data. + +Operational note: the server serializes every request behind one lock, so a wedged process is a +fleet-wide outage. Hence `Restart=on-failure` with `TimeoutStopSec=30` — fail fast and let systemd +recover it. + ## systemd user timer (recommended on modern Linux) **Why:** runs without the user logged in (with `loginctl enable-linger`), survives reboots, logs to `journalctl`, Persistent=true catches missed runs after the machine was off. No root required — it's a *user* unit. diff --git a/docs/phase-1-exposure-runbook.md b/docs/phase-1-exposure-runbook.md index 7a46e45..118695f 100644 --- a/docs/phase-1-exposure-runbook.md +++ b/docs/phase-1-exposure-runbook.md @@ -5,6 +5,17 @@ Companion to [`rfc-001-global-palace.md`](./rfc-001-global-palace.md) (design + This doc covers only the step the other two leave open: **making the primary reachable** — runbook §4 items 2 and 5. +> **Status 2026-08-14 — DONE. Exposed, seeded, and one client flipped.** +> `https://mempalace.jordbo.se/mcp` has been serving since 2026-08-12 (§3.3–§3.6 are all ✅ below). +> The palace was seeded 2026-08-14 15:07 from EMB-7KJ4VR4G (14,777 → 14,803 drawers), and that +> machine's pi-devbox container is flipped and **verified end-to-end — see §3.8**, which is the +> verification procedure that did not exist when the first flip was performed. +> Still outstanding: the transcript feeder is **inert on every deployed image** (§3.7), and §7.6 is +> still a hard blocker for the *second* machine to join (§4). + +Original status, kept for the record — it contradicted this file's own ✅ section markers for two days, +which is the failure mode a file-top status block invites: + **Status 2026-08-12 — Pangolin updated on nyvaken (done, yours). newt not yet installed on synlig. Nothing exposed. No client `.env` flipped.** @@ -158,7 +169,6 @@ bound **to loopback**, receiving a proxy-forwarded foreign `Host:` (§2, verifie docker0-only bind you cannot get 403 from loopback, because you never get far enough to send a header. Refusal is the stronger signal of the two: it proves the loopback and LAN surface is not listening at all. If it *hangs* instead, or `ss` shows `0.0.0.0:8765`, stop — that is not this configuration. -```sh ### 3.2 Collect the shared token @@ -325,6 +335,94 @@ mempalace-pi-session --reason manual-check; echo "exit=$?" # exit=0 and a file cat ~/.pi/agent/mempalace-catchup.log # missing file = the entrypoint skipped ``` +### 3.8 Verify the flip actually took — ✅ done 2026-08-14 on EMB-7KJ4VR4G + +§3.6 verifies the *endpoint* before you flip. §3.7 tells you *how* to flip. Neither verifies the thing +you actually care about afterwards: **that the agent's palace tools are now talking to synlig.** That +gap is why the first flip was "done" for an hour before anyone could say whether it had worked. + +There are **three separate claims** here and they fail independently. Check them in order; each one is +cheap and rules out a different fault. + +**(a) Did the container receive the variable?** In a shell *inside* the container: + +```sh +env | grep MEMPALACE_REMOTE_URL # -> https://mempalace.jordbo.se/mcp +``` + +⚠ Do **not** run a bare `env | grep MEMPALACE` — that prints the bearer token into your scrollback. +If the variable is absent, the cause is almost always §3.7's edit not having been applied: `env_file` +is read at container **create** time and baked into the container config, so **`docker compose up -d` +is required; `docker compose restart` silently reuses the old config.** Evidence from 2026-08-14 — +running container config-hash `3a55e09ac19e0118` vs compose-computed `8a7e0cd4a677a9c1`; a differing +hash is what makes `up -d` recreate. `--force-recreate` is not needed. +**Never `docker compose down -v`** to "pick up" a change: on pi-devbox that destroys seven named +volumes, including `devbox-pi-config` (pi's config **and every session transcript**), `devbox-uv`, and +`devbox-chroma-cache` (a large embedding-model re-download). + +**(b) Is the server reachable and the token accepted?** Still inside the container — this proves the +network path and the credential, independently of any agent: + +```sh +curl -s -X POST "$MEMPALACE_REMOTE_URL" \ + -H "Authorization: Bearer $MEMPALACE_REMOTE_TOKEN" \ + -H 'Content-Type: application/json' \ + -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}' | head -c 200 +``` + +`401` = token problem (compare `md5sum` of the client value against synlig's token file — trailing +whitespace from an editor is the classic cause). Connection failure = DNS/tunnel, not auth. + +**(c) Did the agent's bridge actually switch?** This is the claim that matters and the one that cannot +be checked from bash — there is no log file. Ask the agent running in the container for +`mempalace_status` and read the palace path it reports: + +``` +sqlite_integrity.palace = /home/ecsjper/.mempalace/palace # ← proves remote +``` + +**This is the cheapest and strongest discriminator: that path cannot exist inside the container**, +whose user is `developer` with `HOME=/home/developer` and whose local palace is +`/home/developer/.mempalace/palace`. One call, no token in scrollback, no writes. If instead it +reports the local path, the bridge fell back: `createClient()` reads `MEMPALACE_REMOTE_URL` **once at +load**, so re-check (a) and restart the agent, not just the container. + +#### What does *not* verify the flip — read this before inventing your own check + +- **❌ A drawer count.** Central was seeded *from* the client's own palace, so both report ~14,777. + Counts cannot tell the two apart. Worse, `mempalace status` counts **chunk rows, not logical + drawers** (three drawers plus a 2-chunk diary presented as +9), so a delta does not even mean what + it looks like. Never reason about palace identity or contents from a count. +- **❌ Writing a drawer through the palace tools and reading it back through the same tools.** This + succeeds *identically whether or not the flip worked* — both palaces are healthy and were seeded + from the same source, so a write-then-read round-trips either way. It only becomes evidence if the + drawer is read back over a **different transport** (the `curl` in (b), by drawer id) or is proven + **absent** from the local sqlite. This is the same false-positive class as the CLI below, one layer + up, and it is an easy trap to fall into precisely because it feels like an end-to-end test. +- **❌ The `mempalace` CLI, in any form.** The CLI has **no remote support whatsoever** — its only + selector is `--palace ` — so it reads and writes the LOCAL on-disk archive via + `config.json`. After a flip that archive is dead, yet `mempalace status` / `mempalace search` will + cheerfully report ~14,777 drawers and look exactly like success. It is a false-positive machine. + **Corollary that bites later:** memories filed with the CLI after a flip land in the dead archive, + not in central, and a transcript backfill must therefore be mined **on synlig**, where the CLI's + local palace *is* the central one. + +#### Expected failure behaviour, so you can recognise it + +The bridge is **fail-closed, not fail-local.** If synlig is unreachable, DNS fails, or the token is +rejected, the extension retries a bounded number of times, prints `mempalace-mcp unavailable after +retries; continuing without palace tools`, and **does not register the palace tools at all**. It does +not silently write to the local palace; there is no dual-write and no local mirror, and in remote mode +no local `mempalace-mcp` process is spawned (verified: zero mempalace processes in the flipped +container). So **"the agent has no `mempalace_*` tools" is the expected symptom of a server, token, or +DNS fault** — not of a broken container. Diagnose with (b). + +#### Rollback — ~30 seconds, loses nothing + +Comment out `MEMPALACE_REMOTE_URL` in `.env`, then `docker compose up -d`. The bridge falls back to the +local stdio palace, which is intact. Note the local archive is **frozen, not empty**: it stops at the +moment of the flip, so anything the agent filed into central since then will not be there. + --- ## 4. Still open @@ -334,7 +432,19 @@ cat ~/.pi/agent/mempalace-catchup.log # missing file = th path it must first rsync there. HTTPS would then be genuinely sufficient and mining would work from any network. Until then, mining is limited to devices that can reach synlig's SSH. - **Per-device tokens** — Phase 4. Until then `origin_device` is advisory (§1.1). -- **§7.6 diary dedup** must be settled *before* the first §4.4 join; replay duplicates every entry. +- **§7.6 diary dedup** must be settled *before* the **next** §4.4 join; replay duplicates every entry. + ⚠ **2026-08-14 — the first join did not resolve this, it SIDESTEPPED it.** The seed was a file-level + copy of one palace, which replays no diaries and therefore cannot duplicate them. The success of + that join is *not* evidence the replay path is safe — it never exercised it. §7.6 remains a hard + blocker for the second machine, which is the one that will actually need merge semantics. +- **The transcript feeder is inert on every deployed image**, so nothing is being mined automatically + anywhere in the fleet (§3.7). Regaining it needs a **new tagged pi-devbox release** — its CI + publishes only on `v*` tags and the latest tag *is* the currently deployed image — built on + pi-devbox ≥ `7c00dd6` **and** mempalace-toolkit ≥ `29e660e`. Both are required: the first restores + the container-start catch-up, the second is where extension-side feeding was implemented at all. +- **Whether *native* pi on a flipped machine was also flipped** is a per-machine question nobody has + checked. If the container is flipped and native pi is not, that machine writes into two different + palaces and its memory silently splits in two. Verify per machine with the §3.8(c) path check. - **§7.2**: never run `mempalace sync` against the shared palace. Doubly true now that the pi/opencode feeders stage *inside* the palace root, which puts staged sources in scope for a sync of the palace dir. - **nyvaken's public FQDN and the web hotel's CNAME rules** — unverified (§3.4). diff --git a/docs/rfc-001-global-palace.md b/docs/rfc-001-global-palace.md index 159698c..52e5441 100644 --- a/docs/rfc-001-global-palace.md +++ b/docs/rfc-001-global-palace.md @@ -2,11 +2,12 @@ | | | | --- | --- | -| **Status** | Draft — design agreed, not implemented | +| **Status** | **Phases 0–1 implemented and verified — 2026-08-10 / 08-12 / 08-14.** Primary live + seeded, first client flipped. Phases 1.5, 2 and 4 outstanding; Phase 3 deferred by decision. | | **Created** | 2026-08-08 | | **Applies to** | mempalace 3.6.0, mempalace-toolkit @ `96699f2`, pi-devbox ≥ v1.3.0 | | **Decision** | Phases 0–2 + 4 in scope. **Phase 3 (full pull replication) explicitly deferred** — "a laptop that can reach its own stuff plus whatever it can reach" is good enough. **Centralization is strictly opt-in — solitary devbox operation remains the default and must not change (§1.1).** | | **Recon update** | **2026-08-09 — §9 Q1 and Q6 are RESOLVED, both in the permissive direction** (opencode supports remote MCP; opencode-devbox already templates the mempalace entry). Neither is a blocker. See those entries for evidence; §2, §4.1 and R5 were corrected accordingly. | +| **Rollout update** | **2026-08-14 — Phase 1 is live.** Primary serving at `https://mempalace.jordbo.se/mcp` since 2026-08-12 (synlig, `mempalace-serve.service` under `systemctl --user`, palace `/home/ecsjper/.mempalace/palace`); **seeded 2026-08-14 15:07** from EMB-7KJ4VR4G's palace — 14,777 drawers / 9 wings / 16,337 embeddings / KG 46 entities, 34 triples (14,803 drawers by 17:00); **first client flipped and verified end-to-end** the same afternoon. ⚠️ Two things not to misread: the seed was a **file-level copy of one palace**, *not* the §4.4 MCP replay (see the Deviation note in §4.4), and it therefore **sidestepped §7.6 rather than resolving it — §7.6 remains a hard blocker for the *second* joiner (§8 Phase 0).** §2's and §4.1's predictions about the pi client were confirmed in production; §4.4's count-based verification advice was **wrong** and has been corrected. | **Read this first if you are asked to "centralize MemPalace" / "sync palaces between machines".** Most of the hard-won facts below are non-obvious and two of them are actively destructive if you guess wrong @@ -88,8 +89,9 @@ Two of the three pieces already exist. This is not greenfield. | Piece | State | Evidence | | --- | --- | --- | | **Primary server** | ✅ **Exists.** `mempalace serve --host --port --token --tls-cert --tls-key --read-only --allow-insecure`. Bearer token compared with `hmac.compare_digest`, **mandatory** on non-loopback binds (unless `--allow-insecure`), TLS 1.2+ resolved *before* bind, Host-header pinning + `Origin` allowlist (anti-DNS-rebinding), 16 MiB body cap, token-free `/healthz`. | `cli.py:cmd_serve` (~1448); `mcp_server.py:5205-5215`, `5284-5289` | -| **Remote client** | ⚠️ **Exists for pi only, and it is either/or.** `createClient()` picks stdio *or* HTTP once at process start. | `extensions/pi/mempalace.ts:629-641` | -| **Fallback + resync** | ❌ **Absent everywhere.** On remote failure the pi bridge re-handshakes the same URL, then **de-registers all palace tools** and runs blind. | `extensions/pi/mempalace.ts:665-673` | +| **Remote client** | ⚠️ **Exists for pi only, and it is either/or.** `createClient()` picks stdio *or* HTTP once at process start. ✅ **Confirmed in production 2026-08-14** on the first flipped client: the branch is a pure transport swap, and in remote mode **no local `mempalace-mcp` child is spawned at all** (zero mempalace processes in the flipped container). So **writes go only to the primary — no dual-write, no local mirror.** | `extensions/pi/mempalace.ts:629-641` | +| **Fallback + resync** | ❌ **Absent everywhere.** On remote failure the pi bridge re-handshakes the same URL, then **de-registers all palace tools** and runs blind. ✅ **Confirmed 2026-08-14 — and the precise word is fail-*closed*, not fail-local.** After bounded retries it prints `mempalace-mcp unavailable after retries; continuing without palace tools` and registers nothing; it never silently falls back to the local palace, so a write cannot land in the wrong store. Operational corollary worth stating once: **"the palace tools vanished" is the expected symptom of a server / token / DNS fault**, not of a broken client. | `extensions/pi/mempalace.ts:665-673` | +| **`mempalace` CLI** | ❌ **No remote support whatsoever.** Its only palace selector is `--palace `; otherwise it resolves `palace_path` from the local `config.json`. **Verified 2026-08-14: on a flipped client the CLI still reads and writes the now-dead local archive — and cheerfully reports ~14,777 drawers while doing so.** It is a false-positive machine: never verify a flip with `mempalace status`/`search`, and never file memories with the CLI post-flip (they land in the archive, not the primary). Corollary: a **transcript backfill must be mined on the primary host**, where the CLI's local palace *is* the primary. | `cli.py` argument surface; verified on EMB-7KJ4VR4G | | **opencode client** | ✅ **Remote is supported and already wired.** opencode's published schema (`https://opencode.ai/config.json`, `$defs.McpRemoteConfig`) makes `{"type":"remote","url","headers","oauth"}` a first-class sibling of `McpLocalConfig`, `headers` being a free string→string map (so bearer is a convention, not a constraint). `opencode-devbox` already emits exactly that entry when `MEMPALACE_REMOTE_URL` is set. The all-`type:local` configs in `myconfigs` are a *deployment* fact, not a capability limit. | `generate-config.py:107-118`; hook at `entrypoint-user.sh:117`; schema `$defs.McpRemoteConfig` | | **Server compose** | ✅ Exists: `pi-devbox/docker-compose.mempalace.yml` (canonical) + a tor-ms22 derivative (`docker-compose-repo/tor-ms22/pi-devbox/`, `df2c2ae`, port 8766, uid 1000, binds the real `~/.mempalace`). Not enabled. | pi-devbox CHANGELOG v1.3.0 (2026-07-02) | @@ -103,13 +105,19 @@ Two of the three pieces already exist. This is not greenfield. > > **✅ Done 2026-08-12** — `pi-devbox/.env.example` and > `mempalace-toolkit/extensions/pi/README.md` both now recommend `mempalace serve`, bind docker0 rather -> than `0.0.0.0`/loopback, use an `https://` example URL, and state that the transport *is* +> than `0.0.0.0`/loopback, and state that the transport *is* > authenticated. `docker-compose.mempalace.yml` audited too, and it was worse than stale — it was > **broken on 3.6.0 in both directions**: `--host 0.0.0.0` with no token in the environment makes the > server refuse to start (crash-looping under `restart: unless-stopped`), and once a token *is* supplied > the healthcheck's unauthenticated `tools/list` POST 401s, marking a healthy server unhealthy forever. > Fixed: token now required via `${MEMPALACE_REMOTE_TOKEN:?}` (fails fast at `up`), healthcheck switched > to the token-free `/healthz`. +> +> ⚠️ **Corrected 2026-08-14 — one sub-claim above was wrong for two days.** The `https://` example URL +> landed in `pi-devbox/.env.example` only; `extensions/pi/README.md` still carried +> `http://mempalace.lan:8765/mcp` until it was fixed on **2026-08-14**. Everything else in this block +> checked out. Worth naming as a pattern, because it is cheap to repeat: **a doc's own ✅ is not evidence +> the work landed — verify it per sub-claim and per file.** ### Two things that sound like the feature and are not @@ -133,6 +141,11 @@ independently disqualifying: vectors are written to Postgres.*" Cross-architecture ONNX determinism (arm64 macOS vs x86-64 Linux, different execution providers) is not guaranteed. → **Ship text + metadata, always re-embed on receipt.** Cheap and safe: the raw text is always stored as the chroma document. + + ⚠️ **Not absolute — see the 2026-08-14 deviation in §4.4.** A whole-palace file-level copy arm64 macOS → + x86-64 Linux *did* preserve working search and an intact HNSW index. Re-embedding on receipt remains the + rule for **merging** operations into an existing store; it is not a prohibition on cloning one palace + wholesale, provided the embedder sidecar travels with it (§7.4). 2. **KG rows cannot be copied.** `triples` has **no** `UNIQUE(subject,predicate,object,valid_from)` — only `id` is unique, and `make_triple_id` embeds `datetime.now()`. A row copy therefore duplicates every fact. But `add_triple()` guards at the application level (`SELECT id … WHERE subject=? AND @@ -315,6 +328,32 @@ precisely to be extended. container joins the same way, at any time, repeatedly — so a join is **idempotent replay of local history**, and the only real question per record type is *what dedupes it*. +> ⚠️ **Deviation (2026-08-14) — the first join did not follow this section.** It was precisely the thing +> the framing correction says this is *not*: a **file-level copy of one chosen palace**, arm64 macOS → +> x86-64 Linux, bypassing MCP replay entirely. It worked — search verified live on the primary, HNSW +> index intact, KG intact (46 entities / 34 triples). Recorded here so the next operator neither repeats +> it blind nor believes it is forbidden. **Four conditions made it safe, and all four must hold:** +> +> 1. **Same mempalace version (3.6.0) at both ends**, so the on-disk chroma/HNSW layout matched. +> 2. **`mempalace_embedder.json` travelled with the palace**, so embedder identity matched. This is the +> load-bearing one: a mismatched embedding model does **not** raise — it silently returns garbage +> search results, and §7.4 explains why the guard cannot catch it (only the model *name* is compared, +> and `dimension: 0` is skipped as unknown). +> 3. **A single source palace, so no merge semantics were exercised at all.** This is exactly what makes +> the method inapplicable to the second joiner. +> 4. **Python's `sqlite3` online-backup API** (`src.backup(dst)`) for the two sqlite DBs — WAL-safe with +> a live writer, and the only option available because **neither host has the `sqlite3` CLI**. The +> rest of `palace/` was `rsync`ed with `--exclude 'chroma.sqlite3*'`. +> +> **This sidesteps the merge problem; it does not solve it.** The second machine to join still needs the +> replay path and the dedupe keys tabulated below, and still needs §7.6 settled. Three traps found while +> doing it: **never `rsync --delete` into `~/.mempalace`** — the server's bearer token lives *inside* that +> tree at `~/.mempalace/server//token` (§10) and there is no second copy; **palace paths cannot +> move**, because the palace directory name is a sha256 prefix of its own path; and compare sizes with +> **`stat -c%s`, not `du`** (APFS and ext4 disagree on block accounting, so `du` shows a spurious delta). +> Quiesce the writer first — the source palace's own MCP server was live throughout, which is how a write +> landed on the far side of the snapshot boundary and produced the false alarm described below. + | Record type | Dedupe on replay | Client work needed | | --- | --- | --- | | `add_drawer` / `checkpoint` drawers | **Server-side**: content-addressed id + pre-write `col.get` probe → `{"reason":"already_exists"}`, no write (`mcp_server.py:2593-2600`) | **None.** Just replay | @@ -342,9 +381,32 @@ for everything except diaries. Belt and braces: pattern as the outbox (§4.3). It must **not** live in a container-only path: `~/.mempalace` is not preserved by default for solitary users (§1.2), which is exactly why the file belongs to the palace directory. -- **First join is a dry run.** Bootstrap one palace, verify counts (`status`, `kg_stats`, per-wing - `list_drawers`) against expectations, *then* let the rest join. Ordering matters only because of diaries - and closed facts; everything else is order-free. +- **First join is a dry run.** Bootstrap one palace, verify it, *then* let the rest join. Ordering matters + only because of diaries and closed facts; everything else is order-free. + + > ⚠️ **Corrected 2026-08-14. This bullet used to say "verify counts (`status`, `kg_stats`, per-wing + > `list_drawers`) against expectations". Do not verify a join by counts.** Two independent reasons, + > both learned on the first real seed: + > + > 1. **`mempalace status` counts chunk rows, not logical drawers.** Three drawers plus one 2-chunk diary + > presented as **+9**. A count delta cannot even tell you how many *records* moved. + > 2. **Chunk counts and chunk-id sets legitimately differ between two palaces** whenever a drawer was + > updated on either side: an update preserves `drawer_id`, **re-chunks to the new length, and deletes + > the surplus chunk rows** (§9.3). A `…_chunk_000007` present on one side and absent on the other is + > therefore the ordinary signature of an **edit**, not of loss. + > + > On 2026-08-14 that second mechanism produced a confident "the seed lost a chunk" conclusion that was + > filed as a finding before being retracted. Counts **hid** the difference (14,777 vs 14,778 looked like + > one lost row); an id-set diff **over-reported** it. The two methods fail in *opposite* directions, so + > agreeing with either one alone proves nothing. + > + > **Verify by content.** For a sample of drawers, `get_drawer()` on *both* palaces and compare + > the **reassembled `content`**. An id-set diff is a fine first pass to *find* candidates, but it must be + > adjudicated by a content comparison before anyone concludes loss. Two cheap decisive checks once a + > difference is real: **is one side a prefix of the other** (that, and only that, is truncation), and + > **does the final chunk's length equal `len(content) - chunk_size * (n_chunks - 1)`** (800 in 3.6.0)? If + > both sides satisfy the arithmetic for their own content, both are complete and you are looking at two + > revisions, not damage. The genuinely hard case is **two different palaces holding overlapping mined content** — the same repo mined on a laptop and a workstation under different absolute paths. §5 excludes it by keeping mined wings @@ -464,10 +526,20 @@ primary, where there is nothing to move): ```sh ln -sfn palace/knowledge_graph.sqlite3 ~/.mempalace/knowledge_graph.sqlite3 ln -sfn palace/known_entities.json ~/.mempalace/known_entities.json -# hallways.json deliberately NOT symlinked: already palace-derived, and its HOME -# path is a warning-only legacy probe that never auto-migrates (hallways.py:73-95) +ln -sfn palace/hallways.json ~/.mempalace/hallways.json # added 2026-08-14, see note ``` +> **Updated 2026-08-14 — `hallways.json` is symlinked too now.** This block previously said it was +> *deliberately* not symlinked, on the narrow grounds that it is already palace-derived and its HOME path +> is a warning-only legacy probe that never auto-migrates (`hallways.py:73-95`). Both facts still hold — +> but the goal changed from "symlink only what the code demands" to **"all real state lives under +> `palace/`, so one copy of `palace/` is a complete copy"**, which is what let the 2026-08-14 file-level +> seed (§4.4) treat the palace as a single self-contained unit. With all three links in place every +> parent-level path resolves, and 3.6.0's *three different* resolution rules — palace-relative for the +> served KG, HOME for the CLI KG, `dirname(palace_path)` for hallways, hardcoded HOME for +> `known_entities.json` — converge on one set of files. Revert by deleting the symlink if it ever causes +> trouble. + Verified on synlig 2026-08-10, because the WAL behaviour was the load-bearing assumption: a **dangling** symlink is created on first `sqlite3.connect`; `-wal`/`-shm` land next to the **target** (inside the palace dir, so the palace stays a self-contained backup/bind-mount unit) and *not* beside the symlink; a write @@ -675,6 +747,13 @@ duplication. This matters more than it sounds, because diaries are `replicated` (§5) and are precisely the content a join replays (§4.4). +> ⚠️ **Still open as of 2026-08-14 — and the first join did not test it.** The seed was a file-level copy +> (§4.4 Deviation), which replays no diaries and therefore *cannot* duplicate them. **§7.6 was sidestepped +> by method choice, not resolved.** Neither action below has been built. The moment a *second* palace +> joins — by any replay-based route — this becomes live again, and it is the single most likely thing for a +> future operator to get wrong, because the first join appears to have proved the path safe. It did not: +> it avoided the path. + **Actions:** 1. **Client-side, now:** before replay, list the target's diary drawers for the wing, extract the 12-hex @@ -692,8 +771,8 @@ join replays (§4.4). | Phase | Effort | Deliverable | | --- | --- | --- | -| **0 — hygiene** | hours | §7 runbook: converge the KG/entities store paths **on synlig before first `serve`** (§7.1 — **done 2026-08-10**, runbook §2.3), ban `sync` on shared palaces (§7.2), ~~fix stale "unauthenticated" docs (incl. `pi-devbox/.env.example:21`)~~ — **done 2026-08-12**, and `docker-compose.mempalace.yml` turned out to be outright broken on 3.6.0, now fixed. Added 2026-08-09: settle the **diary dedup** approach and file its upstream ask (§7.6), and **dry-run the join from one palace only** (§4.4). **No provenance work here** — it is not backfill-critical (§7.3.3) and belongs to the stamper, not the agent | -| **1 — primary up** | hours, **no code** | `mempalace serve --token --tls-cert` on a private-net host (reuse `docker-compose.mempalace.yml` — keep it a **separate standalone project**, R4 — and mind port 8765 vs pi-studio; tor-ms22 already moved to 8766). Repoint pi clients via `MEMPALACE_REMOTE_URL`/`MEMPALACE_REMOTE_TOKEN`. **opencode clients can be repointed in the same breath** — remote MCP is supported and `generate-config.py` already emits it (§9.1, §9.6), subject to the sidecar caveat in §4.1. **Shared memory today, no offline.** **Decided 2026-08-09: primary = synlig, TLS at Pangolin, single shared token (§8.1)** — mind the loopback Host-pin trap in §6.2. | +| **0 — hygiene** | hours | §7 runbook: converge the KG/entities store paths **on synlig before first `serve`** (§7.1 — **done 2026-08-10**, runbook §2.3), ban `sync` on shared palaces (§7.2), ~~fix stale "unauthenticated" docs (incl. `pi-devbox/.env.example:21`)~~ — **done 2026-08-12**, and `docker-compose.mempalace.yml` turned out to be outright broken on 3.6.0, now fixed. Added 2026-08-09: settle the **diary dedup** approach and file its upstream ask (§7.6), and **dry-run the join from one palace only** (§4.4). ✅ **Both done 2026-08-14, with one asterisk that matters:** the join was dry-run *and* executed from one palace only — but **§7.6 was sidestepped, not settled** (a file-level copy replays no diaries, so it cannot duplicate them). The §7.6 client-side dedupe and its upstream ask are **still unbuilt and are hard blockers for the second joiner.** **No provenance work here** — it is not backfill-critical (§7.3.3) and belongs to the stamper, not the agent | +| **1 — primary up** ✅ **done 2026-08-14** | hours, **no code** | `mempalace serve --token --tls-cert` on a private-net host (reuse `docker-compose.mempalace.yml` — keep it a **separate standalone project**, R4 — and mind port 8765 vs pi-studio; tor-ms22 already moved to 8766). Repoint pi clients via `MEMPALACE_REMOTE_URL`/`MEMPALACE_REMOTE_TOKEN`. **opencode clients can be repointed in the same breath** — remote MCP is supported and `generate-config.py` already emits it (§9.1, §9.6), subject to the sidecar caveat in §4.1. **Shared memory today, no offline.** **Decided 2026-08-09: primary = synlig, TLS at Pangolin, single shared token (§8.1)** — mind the loopback Host-pin trap in §6.2. | | **1.5 — opencode env propagation** | hours | Make the `mcp.mempalace` subtree env-authoritative in `generate-config.py`, gated by a generated-value fingerprint (§4.1). Independent of the rest of this RFC. Without it, adopting *or reverting* the opt-in on an existing opencode container needs a manual sidecar merge or a `docker volume rm` — which also blocks R6 reversibility | | **2 — `mempalace-edge`** | ~1 week | The actual ask: local-first writes + outbox flush + merged reads + per-wing policy. **Not** "fixes opencode" — opencode's *transport* is already fine after Phase 1; what edge adds there is offline/local-first, since `generate-config.py`'s switch is remote **or** local with no failover. **Ships with the §1.2 opt-in wiring (compose + `.env.example` + a third branch in the existing `generate-config.py`) and must pass the R1 acceptance test.** | | **3 — pull replication** | ~1 week | **DEFERRED (2026-08-08).** Server-side op-log with monotonic seq → each edge a full offline replica. Only needed if a laptop must hold *everything* offline. Accepted trade-off: offline recall = own writes + last-reachable state. | @@ -742,7 +821,16 @@ for online clients. re-call with identical oversized content cannot duplicate a legacy pre-#1539 single-row write. The normal path is therefore safe by construction. What remains untested is the abnormal one: a chunk set left partial by a crash or kill *mid-upsert*, which would then look present and stay truncated. Worth a - single fault-injection test before trusting bulk replay of oversized drawers. + single fault-injection test before trusting bulk replay of oversized drawers. **Still untested as of + 2026-08-14** — the first join did *not* exercise it. + + ⚠️ **But 2026-08-14 surfaced a benign mechanism that mimics it, and it will generate false alarms during + any join verification (§4.4).** Updating a drawer preserves `drawer_id`, **re-chunks the new content, and + deletes the surplus chunk rows**. So two palaces holding two *revisions* of one drawer legitimately + differ in chunk count and chunk-id set, with zero data loss — and the shorter side returns a clean + "not found" for the chunk ids it no longer needs, which looks exactly like the truncation this entry + warns about. **The absence of a derived chunk id is not evidence of loss.** Distinguish them by prefix + test and final-chunk arithmetic per §4.4, not by id-set membership. 4. **`migrate.py` as a bootstrap tool.** `extract_drawers_from_sqlite()` reads `{id, document, metadata}` straight out of chroma's SQLite (bypassing the chromadb API) and re-`add`s them with ids and metadata preserved and **embeddings recomputed** (`migrate.py:326`) — the right shape for a diff --git a/docs/synlig-primary-runbook.md b/docs/synlig-primary-runbook.md index 31453d8..5a00bba 100644 --- a/docs/synlig-primary-runbook.md +++ b/docs/synlig-primary-runbook.md @@ -3,6 +3,18 @@ Companion to [`rfc-001-global-palace.md`](./rfc-001-global-palace.md). Records what was actually done on the primary, with verified evidence, so the next session (or the next machine) does not re-derive it. +> **Status 2026-08-14 17:00 — SUPERSEDED IN PART. The primary is live, exposed, and seeded.** +> Serving since 2026-08-12 at `https://mempalace.jordbo.se/mcp`. Seeded 2026-08-14 15:07 from +> EMB-7KJ4VR4G's palace — 14,777 drawers / 9 wings / 16,337 embeddings / KG 46 entities, 34 triples, +> now 14,803 drawers. One client (EMB-7KJ4VR4G's pi-devbox container) is flipped and verified +> end-to-end. Both Phase 0 blockers below are cleared. +> +> **Read §3 "Deliberately NOT done" as a record of the 2026-08-10 state, not of today's** — every +> item in it has since been done. And before running anything in §5 Rollback, read the warning at the +> top of it: `~/.mempalace` on synlig is no longer disposable. + +Original status, kept for the record: + **Status 2026-08-10 00:30 — Phase 0 prep complete. Not serving. Nothing exposed.** Blocked on two things, both deliberately left to Joakim: the Pangolin update on nyvaken, and one `sudo`. @@ -62,10 +74,21 @@ ln -sfn palace/knowledge_graph.sqlite3 ~/.mempalace/knowledge_graph.sqlite3 ln -sfn palace/known_entities.json ~/.mempalace/known_entities.json ``` -Relative targets, so a home-directory move survives. `hallways.json` is deliberately **not** symlinked: +Relative targets, so a home-directory move survives. `hallways.json` was originally left **un**symlinked: it is already palace-derived, and its HOME path is a warning-only legacy probe (`hallways.py:73-95`) that never auto-migrates. +> **Update 2026-08-14 — `hallways.json` is now symlinked too**, during the seeding session: +> ```sh +> ln -sfn palace/hallways.json ~/.mempalace/hallways.json +> ``` +> Rationale changed: the point is no longer "only symlink what the code demands" but *all real state +> lives under `palace/` as a single backup unit*, so one `palace/` copy is a complete copy. All three +> parent-level paths now resolve, which matters because mempalace 3.6.0 resolves these three paths +> inconsistently (MCP server: palace-relative; KG CLI default: `~/.mempalace`; `hallways.json`: +> `dirname(palace_path)`; `known_entities.json`: hardcoded `~`). Revert by deleting the symlink if it +> ever causes trouble. + Verified the symlink assumption rather than trusting it (`python3 sqlite3` on synlig, temp dir): | Check | Result | @@ -119,10 +142,21 @@ standing up a network-reachable service while you were asleep was not mine to de ## 3. Deliberately NOT done +> **⚠️ Historical — this section describes 2026-08-10 and is no longer true.** All five items were +> done between 2026-08-12 and 2026-08-14. Kept because the *reasoning* for deferring them is still +> the record of why the order was chosen. Current state per item is inlined below. + - **Nothing is serving.** No listener on 8765; no mempalace process. Re-verified at the end of the run. + → **Now serving** since 2026-08-12 (`mempalace-serve.service`, `systemctl --user`), reachable at + `https://mempalace.jordbo.se/mcp` via newt/Pangolin. - **No client `.env` was touched.** Your working setup is exactly as you left it (R6: reversible). + → **One client flipped 2026-08-14**: four variables on EMB-7KJ4VR4G, `docker-compose.yaml` unchanged. + Still reversible in ~30s (§3.8 of [`phase-1-exposure-runbook.md`](./phase-1-exposure-runbook.md)). - **No data joined.** The palace is empty. The §4.4 join needs the diary-dedup decision (§7.6) first — replaying diaries today duplicates them, and the primary is the one place that must stay clean. + → **Seeded 2026-08-14** from *one* palace by file-level copy. This sidestepped §7.6 rather than + solving it: a file-level copy replays no diaries, so it cannot duplicate them. **§7.6 is still a + hard blocker for the second machine to join.** - **nyvaken untouched.** Read nothing, changed nothing. - **No sudo.** `sudo -n` on synlig requires a password. @@ -167,13 +201,48 @@ standing up a network-reachable service while you were asleep was not mine to de pointing any client at it. 6. **Then, and only then**, Phase 1 client flip — one machine first, and remember opencode containers need the §4.1 sidecar merge (or Phase 1.5) before the `.env` takes effect. -7. **Before the first join:** settle §7.6 diary dedup, then dry-run §4.4 from **one** palace and check - counts. +7. **Before the first join:** settle §7.6 diary dedup, then dry-run §4.4 from **one** palace. + > **Correction 2026-08-14 — do NOT verify a join "by checking counts", which is what this item + > originally said.** Counts are not evidence, in either direction. `mempalace status` counts + > **chunk rows**, not logical drawers (3 drawers plus one 2-chunk diary presented as +9), and chunk + > counts legitimately differ between two palaces whenever a drawer was updated on either side, + > because an update re-chunks to the new length and deletes the surplus rows. Diffing chunk-id sets + > is a useful first pass but **over**-reports: a chunk id present on one side only is the ordinary + > signature of an edit, not of loss. This cost real time on 2026-08-14 — a missing + > `chunk_000007` was read as a truncated seed, when in fact the two palaces held two revisions of + > one drawer and nothing was lost. **Adjudicate by fetching the parent drawer on both sides and + > comparing the reassembled `content`.** ## 5. Rollback +> **⚠️ STOP — 2026-08-14. Do not run this block as it was originally written.** `~/.mempalace` on +> synlig is now the fleet primary. That tree holds the only central palace (14,803 drawers, seeded +> from EMB-7KJ4VR4G) **and the server's bearer token** at `~/.mempalace/server//token` — the +> single credential every flipped client authenticates with, of which there is no second copy. +> `rm -rf ~/.mempalace` destroys both. The original comment ("empty today — check before running once +> it isn't") is far too soft for a destructive command someone runs under pressure, which is exactly +> why it is being replaced rather than amended. + +Stopping the service is safe and reversible on its own, and is the whole of what "rollback" should +normally mean now: + ```sh -systemctl --user disable --now mempalace-serve # if enabled -~/.local/bin/uv tool uninstall mempalace -rm -rf ~/.mempalace # empty today — check before running once it isn't +systemctl --user disable --now mempalace-serve # clients fail CLOSED — they lose the palace + # tools; they do NOT fall back to a local palace +``` + +To genuinely decommission the primary, in this order: + +1. Flip every client back first (§3.8 of [`phase-1-exposure-runbook.md`](./phase-1-exposure-runbook.md), + in reverse) so nothing is pointed at a host that is about to lose its palace. +2. Copy `~/.mempalace/palace/` **and** the token file off the host, and verify the copy by comparing + reassembled drawer `content`, not counts (see §4 item 7). +3. Only then remove anything. Never `rsync --delete` into `~/.mempalace` — the token lives inside it. + +The two destructive steps below were written on 2026-08-10, when `~/.mempalace` was genuinely empty. +Kept for the record; **must not be run while the primary is live**: + +```sh +~/.local/bin/uv tool uninstall mempalace +rm -rf ~/.mempalace # ⚠️ DESTROYS THE FLEET PALACE AND THE ONLY TOKEN ``` diff --git a/extensions/pi/README.md b/extensions/pi/README.md index 92a24d8..e62f2f4 100644 --- a/extensions/pi/README.md +++ b/extensions/pi/README.md @@ -40,7 +40,10 @@ dependencies (~300 MB). describes. Skipped on resume/fork (context is already in the thread). 4. **Automatic transcript feeding** (`session_shutdown`, and a debounced `agent_settled`): stages + mines this pi installation's own session - transcripts into the palace with no user action needed. Unlike the diary + transcripts into the palace with no user action needed — **as of + mempalace-toolkit `29e660e` (2026-08-12); see the version gate below, + because "the extension is installed" does not imply "this copy can feed"**. + Unlike the diary below, this needs no LLM turn — it's a subprocess + a tool call — so it *can* run on `session_shutdown` where the diary cannot. See [Automatic transcript feeding](#automatic-transcript-feeding). @@ -53,6 +56,26 @@ dependencies (~300 MB). ## Automatic transcript feeding +> **⚠️ Version gate — requires mempalace-toolkit ≥ `29e660e` (2026-08-12), and "installed" is not +> the same question as "capable".** Feeding was added to this extension on 2026-08-12. A copy baked +> into a container image built before that date has *no* feed path at all — its entire +> `session_shutdown` handler is `client.stop()` — and it fails the only way a memory system must not: +> silently, looking exactly like a healthy run with nothing to do. +> +> **Check the deployed artifact, never the repo.** `/opt/*` in an image is baked at build time and can +> be days behind a bind-mounted clone, and `~/.pi/agent/extensions/mempalace.ts` is usually a symlink +> *into* that baked copy: +> +> ```sh +> grep -c MEMPALACE_FEED "$(readlink -f ~/.pi/agent/extensions/mempalace.ts)" # 0 = cannot feed +> ``` +> +> Zero hits means this machine needs the fallback recipes in +> [`contrib/`](../../contrib/README.md) until it is rebuilt, regardless of what the toolkit repo's HEAD +> looks like. Date the deployed copy with `stat` plus that content probe — not `git log`, which fails +> with *"detected dubious ownership"* inside a root-owned `/opt` tree. As of 2026-08-14 the whole +> pi-devbox fleet fails this check. + The bridge feeds this pi installation's own session transcripts into the palace by itself — no scheduler, no cron, no manual invocation. It fires on `session_shutdown` (covers quit, `/new`, `/resume`, `/fork`) and on a @@ -117,10 +140,27 @@ chosen at load time: lives wherever that process opens it (default `~/.mempalace`). This is the hardened path with per-request timeouts and respawn/self-heal (below). - **External** — set `MEMPALACE_REMOTE_URL` to a MemPalace HTTP endpoint (e.g. - `http://mempalace.lan:8765/mcp`) and the bridge connects over HTTP instead, - spawning no local process. Use this to share **one** palace across several - harnesses/containers (pi + opencode + native). `MEMPALACE_REMOTE_TOKEN`, if - set, is sent as `Authorization: Bearer `. + `https://mempalace.jordbo.se/mcp`, the live fleet primary — full path + including `/mcp`, no trailing slash) and the bridge connects over HTTP + instead, spawning no local process. Use this to share **one** palace across + several harnesses/containers (pi + opencode + native). + `MEMPALACE_REMOTE_TOKEN`, if set, is sent as `Authorization: Bearer `. + Use `https://` for anything crossing a network — the plaintext `http://` + example that stood here until 2026-08-14 predated the reverse proxy. + + The two transports are **either/or**, decided once at load time: with the URL + set, writes go **only** to the remote palace. There is no dual-write, no + local mirror, and no local `mempalace-mcp` process at all. + + ⚠️ **Consequence: once `MEMPALACE_REMOTE_URL` is set, the `mempalace` CLI on + that machine is no longer a valid way to inspect or feed the palace the agent + is using.** The CLI has no remote support whatsoever — its only selector is + `--palace ` — so it reads and writes the LOCAL on-disk archive. After a + flip that archive is frozen, yet `mempalace status` / `mempalace search` + still report a plausible drawer count and look exactly like success: a + false-positive machine. Memories filed with the CLI post-flip land in the + dead archive, not in the shared palace. Use the agent's own palace tools + (which go over HTTP), and mine backfills **on the palace host**. Serve such an endpoint with `mempalace serve --host 172.17.0.1 --port 8765` (the `pi-devbox` / `opencode-devbox` repos ship a @@ -155,6 +195,22 @@ If `mempalace-mcp` can't be spawned (PATH missing, binary crashes at startup, …) the extension logs to stderr and returns early. pi keeps working without palace tools rather than refusing to start. +**In remote mode the triggers differ but the outcome is identical.** An +unreachable server, a DNS failure, or an HTTP 401 from a wrong/expired token +all end the same way: after bounded retries the extension prints +`mempalace-mcp unavailable after retries; continuing without palace tools` and +**does not register the palace tools**. + +It is **fail-closed, not fail-local**: it does *not* quietly fall back to the +local palace, so a remote outage can never scatter memories into a local copy +nobody will look at again. The practical corollary, worth knowing before you +debug the wrong layer: **"the agent has no `mempalace_*` tools" is the +expected symptom of a server, token, or DNS fault**, not of a broken install. +Diagnose it with a direct `curl` to `MEMPALACE_REMOTE_URL` — see +[`docs/phase-1-exposure-runbook.md`](../../docs/phase-1-exposure-runbook.md) +§3.8. The design rationale for de-registering rather than degrading is in +[`docs/rfc-001-global-palace.md`](../../docs/rfc-001-global-palace.md) §2 and §4.1. + ## Identity `agent_name` for diary calls comes from `$MEMPALACE_AGENT_NAME`, defaulting