docs(mempalace-broker): clarify engine vs shim — what the image must still ship

This commit is contained in:
pi
2026-06-14 18:20:39 +02:00
parent b17dc1fa1f
commit f8da7890df
+59
View File
@@ -183,6 +183,65 @@ the host where flock works. That is what makes the design topology-independent.
- Clients reclaim a stale lock if the pid recorded in `broker.json` is dead.
- Clients retry with backoff while a broker is mid-startup.
## Engine vs. shim — what the image must still ship
The component bundled in the images today is really **two separable pieces**:
- the **mempalace engine** — opens the SQLite files, computes embeddings, owns
the HNSW index (the heavy part: chromadb, embedding model, etc.), and
- the thin client surface pi actually talks to.
In the brokered design these split cleanly:
- the **broker** is the only thing that runs the *engine*;
- the **shim** is **engine-free** — it just forwards MCP JSON-RPC. It needs no
chromadb, no embedding model, no heavy deps. Embeddings/search happen
broker-side. (Potential image-slimming opportunity, though see below for why
we keep the engine bundled anyway.)
Whether the bundled engine is "used as-is" or merely fronted by the broker
**depends on who owns the broker**:
**A) Host runs the broker (native, or native+container — the common case).**
The *host's* engine is authoritative and used as-is. The broker is purely an
intermediate step so writes can't collide; the host engine does the read/write.
The container's **bundled engine is dormant** — the container uses only its shim
to reach the host broker. The engine in the image is not needed for this path.
**B) Container lands on a host with no mempalace (fresh-host case).**
The bundled engine earns its keep — you cannot conjure an engine onto the host
without installing one. Either the container runs the broker *itself*
(in-container ownership, bundled engine used as-is) or it falls back to degraded
direct mode (single writer, bundled engine used directly).
**Decision: keep shipping the engine in the images** — but for three specific
reasons, not because the brokered path needs it:
1. **Self-containedness** — pi-devbox's promise is "works on any host." A
container with no memory unless the host pre-installed mempalace breaks that,
especially for the Docker Hub audience.
2. **Fresh-host bootstrap** (case B) — no host engine to borrow.
3. **Degraded fallback** — the no-broker-reachable path opens the DB locally and
needs the engine present.
In the host-managed common case the bundled engine is just dormant insurance;
the shim is the only piece the container actively uses.
### Version-coherence note
Because **only the broker's engine ever writes**, its version defines the
on-disk format. Host-vs-bundled engine version skew is therefore **harmless in
the brokered path** (only one engine ever touches the bytes). Skew only bites in
**degraded direct mode**, where the container writes with a possibly-different
engine version than the host would. This argues for the broker pinning/owning
the authoritative engine version and treating the bundled engine as
fallback-only.
> Partially resolves the "where the broker binary ships" open question below:
> the **shim** must ship on both sides; the **engine** must ship on the host
> (to run the broker) and stays bundled in the image as fallback/bootstrap
> insurance, not as the authoritative writer in the common case.
## The genuinely hard case
**Container-only with no SSH bridge configured** (e.g. plain Linux Docker,