pi-devbox/docs/mempalace-broker-design.md

# Design: single-writer MemPalace broker (cross-host serialization)

> **Status:** DRAFT / RFC — not yet implemented. Captures the design so it can be
> picked up later. Authored 2026-06-14.
> **Owner:** unassigned. **Tracking:** queue item #4 ("host-side mempalace-mcp
> daemon over a UNIX/shared socket").

## Problem

The pi-devbox container's `~/.mempalace` (`/home/developer/.mempalace`) is a
**virtiofs bind-mount of the host's `/Users/joakim/.mempalace`** (verified
2026-06-14 via `/proc/mounts`: `mac /home/developer/.mempalace virtiofs rw`).
Container pi and host-native pi therefore **read and write ONE shared palace** —
full memory parity already exists; nothing needs to be built to *enable* sharing.

The actual hazard is the opposite of sharing: **concurrency**. Two pi processes
(one native on the host, one in the container) can open the same
`chroma.sqlite3` / `knowledge_graph.sqlite3` and write at the same time. The
palace directory already shows the scars of this:

- `chroma.sqlite3.broken-20260505`
- many `*.corrupt-20260528`
- a long run of `*.drift-2026*`
- `locks/` with `mine_palace_*.lock` files, including a **stale** one.

These are mempalace's defensive lock + auto-snapshot/repair machinery firing
under concurrent access.

### Why a shared lock file is NOT sufficient

The container runs inside a Linux VM (OrbStack / Docker Desktop on macOS); the
palace bytes live on the macOS host, surfaced into the VM via virtiofs.
Consequences:

- A **UNIX-domain socket file** visible at `~/.mempalace/broker.sock` inside the
  container is a *host-kernel* object. The container's kernel can see the inode
  but **cannot connect to it** across the VM boundary.
- **flock / advisory lockfiles are not coherent across the host↔VM boundary.**
  A lock taken on the host is not reliably seen in the container and vice-versa.
  (The stale `mine_palace_*.lock` is direct evidence the existing lock scheme is
  not bulletproof across this boundary.)

**Therefore the only trustworthy serialization is to route every write through a
single process.** That single process is the broker. The design question is *not*
"how do we lock" — it's "**where does the one writer live, and how does every pi
(host or container) reach it across the VM boundary?**"

## Goals

1. Exactly one process opens the palace SQLite files at any time (single writer;
   concurrent reads are fine).
2. Works in all three topologies on a given host:
   - native pi only,
   - native pi + container pi,
   - container pi only.
3. pi configuration is **identical** in every topology (no per-environment MCP
   config divergence).
4. No new corruption pathway introduced; degrade safely when the broker is
   genuinely unreachable and there are no peers.

### Non-goals (for this iteration)

- opencode / opencode-devbox co-existence (see "Co-existence with opencode"
  below — deferred until the pi case is solved).
- Multi-host palace replication. This is about one host's local palace.
- Changing mempalace's on-disk format or its public MCP tool surface.

## Architecture

```
pi (host)  ─stdio─►  mp-shim ─┐
                              ├─►  mempalace-broker  ─►  chroma.sqlite3
pi (ctr)   ─stdio─►  mp-shim ─┘     (SINGLE owner;        knowledge_graph.sqlite3
                                    serialized writer,    + in-memory HNSW index
                                    concurrent readers)
```

### `mempalace-broker`

A long-lived process that is the **only** opener of the palace SQLite files. It:

- runs the real mempalace engine,
- holds the HNSW index in memory,
- pushes all mutations through a single writer queue (reads may fan out),
- exposes the mempalace MCP JSON-RPC surface over one or more transports,
- is the canonical owner of palace state for the lifetime of the host session.

**Bonus:** a single always-resident owner also eliminates the stale-HNSW-index
problem that `mempalace_reconnect` exists to work around — there is never an
external writer to desync the in-memory index against.

### `mp-shim`

A tiny stdio↔transport adapter. pi's mempalace MCP config points at the shim
**everywhere, unchanged**. pi still believes it is speaking stdio MCP to a local
server; the shim forwards JSON-RPC to the broker over whichever transport is
available, and handles all discovery / startup / election complexity. Keeping
pi's config identical across topologies is a hard requirement (goal #3) and the
shim is what makes it possible.

## Canonical owner = the host

The broker's home is **always the host**, because:

1. The palace bytes physically live there (`/Users/joakim/.mempalace`).
2. The host outlives any container — ownership does not evaporate on
   `docker compose down`.
3. Containers already have a route back to it (`host.docker.internal` and the
   verified dssh ControlMaster bridge).

The broker binds **two listeners feeding one queue**:

- **AF_UNIX** at `$MEMPALACE_PATH/broker.sock` — for host-native pi (fast,
  filesystem-perms-secured).
- a **cross-boundary** transport for container clients (below).

## Transport matrix

| Topology | Broker runs on | Host pi reaches it via | Container pi reaches it via |
|---|---|---|---|
| native only | host | AF_UNIX socket | — |
| native + container | host | AF_UNIX socket | SSH-forwarded socket (preferred) or TCP |
| container only | host (started via bridge) | — | SSH-forwarded socket or TCP |

### Cross-boundary transport options

**(a) SSH-forwarded UNIX socket over the existing dssh ControlMaster — PREFERRED.**
The container's `setup-lan-access.sh` already establishes a ControlMaster to the
host with `ControlPersist 4h`. The container shim forwards the host broker socket
over that master:

```
ssh -F ~/.ssh-local/config \
    -L "$XDG_RUNTIME_DIR/mp.sock:$HOME/.mempalace/broker.sock" host
```

then connects to the local forwarded socket. Auth = SSH key; nothing is
LAN-exposed; no extra shared secret needed; rides the persistent master so setup
cost is near-zero. Most portable across non-OrbStack hosts.

**(b) TCP on `host.docker.internal:PORT` — fallback.** Simpler, but the broker
must bind a routable interface (not just `127.0.0.1`), which requires a
**shared-secret token** to prevent other local/LAN processes from talking to it.
The token is written to `broker.json` in the virtiofs-mounted palace dir
(readable from both sides). More care required to get the bind + auth right.

## Discovery + on-demand start (the shim's algorithm)

Run by the shim on every pi session start, so it is correct regardless of who is
already running:

```
1. If $MEMPALACE_BROKER is set        → use it verbatim (escape hatch).
2. Read $MEMPALACE_PATH/broker.json   → endpoint + pid + token.
   Try to connect (UNIX if host; forwarded-sock / TCP if container).
   If connected & healthy             → done.
3. Broker not reachable → START IT:
   - On host:      flock($MEMPALACE_PATH/broker.lock, non-blocking)
                     win  → exec broker, wait for broker.json, connect.
                     lose → someone else is starting it; backoff + retry connect.
   - In container: run `ssh host 'mempalace-broker --ensure'` (idempotent;
                   performs the SAME flock election ON THE HOST), then forward +
                   connect.
4. Last-resort fallback (no broker, cannot start one):
   open the palace DIRECTLY — but ONLY after asserting this process is the sole
   writer (no other live broker/pid recorded in broker.json). Degrades to
   today's behaviour for the genuinely-alone case; never used when a broker
   exists.
```

**Key trick:** host-side election uses `flock` on the host, where it is coherent
(same kernel) — bulletproof. The cross-boundary case **never relies on cross-VM
locking**; it relies on `ssh host 'broker --ensure'`, which runs the election on
the host where flock works. That is what makes the design topology-independent.

### Lifecycle

- Broker writes `broker.json` (endpoint + pid + token) **atomically** after
  binding.
- Broker holds `broker.lock` for its entire lifetime → at most one host broker.
- Idle-exit after N minutes with no connected clients; the next client
  re-elects. (Or keep-alive; idle-exit is friendlier on resources.)
- Clients reclaim a stale lock if the pid recorded in `broker.json` is dead.
- Clients retry with backoff while a broker is mid-startup.

## Engine vs. shim — what the image must still ship

The component bundled in the images today is really **two separable pieces**:

- the **mempalace engine** — opens the SQLite files, computes embeddings, owns
  the HNSW index (the heavy part: chromadb, embedding model, etc.), and
- the thin client surface pi actually talks to.

In the brokered design these split cleanly:

- the **broker** is the only thing that runs the *engine*;
- the **shim** is **engine-free** — it just forwards MCP JSON-RPC. It needs no
  chromadb, no embedding model, no heavy deps. Embeddings/search happen
  broker-side. (Potential image-slimming opportunity, though see below for why
  we keep the engine bundled anyway.)

Whether the bundled engine is "used as-is" or merely fronted by the broker
**depends on who owns the broker**:

**A) Host runs the broker (native, or native+container — the common case).**
The *host's* engine is authoritative and used as-is. The broker is purely an
intermediate step so writes can't collide; the host engine does the read/write.
The container's **bundled engine is dormant** — the container uses only its shim
to reach the host broker. The engine in the image is not needed for this path.

**B) Container lands on a host with no mempalace (fresh-host case).**
The bundled engine earns its keep — you cannot conjure an engine onto the host
without installing one. Either the container runs the broker *itself*
(in-container ownership, bundled engine used as-is) or it falls back to degraded
direct mode (single writer, bundled engine used directly).

**Decision: keep shipping the engine in the images** — but for three specific
reasons, not because the brokered path needs it:

1. **Self-containedness** — pi-devbox's promise is "works on any host." A
   container with no memory unless the host pre-installed mempalace breaks that,
   especially for the Docker Hub audience.
2. **Fresh-host bootstrap** (case B) — no host engine to borrow.
3. **Degraded fallback** — the no-broker-reachable path opens the DB locally and
   needs the engine present.

In the host-managed common case the bundled engine is just dormant insurance;
the shim is the only piece the container actively uses.

### Version-coherence note

Because **only the broker's engine ever writes**, its version defines the
on-disk format. Host-vs-bundled engine version skew is therefore **harmless in
the brokered path** (only one engine ever touches the bytes). Skew only bites in
**degraded direct mode**, where the container writes with a possibly-different
engine version than the host would. This argues for the broker pinning/owning
the authoritative engine version and treating the bundled engine as
fallback-only.

> Partially resolves the "where the broker binary ships" open question below:
> the **shim** must ship on both sides; the **engine** must ship on the host
> (to run the broker) and stays bundled in the image as fallback/bootstrap
> insurance, not as the authoritative writer in the common case.

## The genuinely hard case

**Container-only with no SSH bridge configured** (e.g. plain Linux Docker,
`HOST_SSH_USER` unset, no `host.docker.internal`). The container cannot start or
reach a host broker. Options, none free:

1. **Require the bridge** for multi-writer container setups, and document it as a
   precondition. Reasonable: pi-devbox already ships `setup-lan-access.sh` and
   the bridge is the supported path.
2. **Run the broker inside the container**, publishing a Docker port the host can
   later reach. Works, but inverts ownership and the broker dies with the
   container — only acceptable if containers are the *sole* writers on that host.
3. **Accept degraded mode** (algorithm step 4): a lone container with no peers
   has no concurrency, so direct access is safe *as long as* nothing else opens
   the palace concurrently. The host shim also checks `broker.json` before
   opening directly, so a later host pi will not silently start a second
   uncoordinated writer.

**Summary:** fully robust for native-only, native+container, and
container-only-with-bridge. The only residual sharp edge is container-only
*without* a bridge *and* a future concurrent host writer — intrinsic (no shared
coherent lock exists across that boundary), best handled by mandating the bridge
rather than pretending file locks work.

## Co-existence with opencode / opencode-devbox (DEFERRED — context only)

The palace is shared by more than pi. opencode (native) and opencode-devbox
(container) also write to the same `~/.mempalace`. **Assumption to verify:**
opencode sessions write to **different wings** than pi sessions (pi uses
`wing_pi`, diaries per-agent, etc.), so cross-tool intermixing into the *same*
destination may be a non-issue at the application level.

However, the corruption risk here is at the **SQLite-file level, not the wing
level** — two processes writing different wings of the *same* `chroma.sqlite3`
concurrently is still a concurrent write to one file. So the broker, once it
exists, is the right serialization point for opencode too: opencode's mempalace
client would route through the same broker via the same shim mechanism.

**Decision:** do not design for opencode co-existence yet. Resolve the pi case
first; then revisit whether opencode clients adopt the same shim. The residual
risk in the interim is native + container *opencode* sessions writing the same
palace simultaneously — explicitly deferred ("cross that bridge later").

## Open questions / TODO before implementation

- Does the mempalace engine expose an embeddable entrypoint suitable for running
  inside a long-lived broker, or does the broker wrap the existing MCP server
  binary and multiplex stdio clients onto it? (Affects whether reads can truly
  fan out or are also serialized.)
- Idle-exit timeout default + whether to expose it via env.
- `broker.json` schema + atomic-write + stale-pid-reclaim details.
- TCP-path token handling and safe bind interface selection on Linux Docker
  (`--add-host=host.docker.internal:host-gateway`).
- Where the broker binary ships: baked into `Dockerfile.base`? host install via
  pi-toolkit / mempalace-toolkit? Both, since both sides need the shim and the
  host needs the broker.
- Smoke-test plan: prove single-writer invariant under a deliberate concurrent
  host+container write storm (should produce zero `.corrupt`/`.drift` snapshots).