rfc-001: join protocol, diary non-idempotency, deployment decisions
Second planning round. Three findings from verification, and the decisions that were blocking phasing. Verified in mempalace 3.6.0 and written up: - NEW §7.6 — diary_write has no idempotency guard at all. The id embeds datetime.now() at microsecond resolution and the write is a bare col.add (mcp_server.py:3546) with no col.get probe, in direct contrast to add_drawer's probe 900 lines earlier (:2593-2600). So §3's "log the intent, replay the intent" is false for diaries — replay duplicates. This lands on the critical path because diaries are `replicated` and are exactly what a join replays. - NEW §4.4 — join/bootstrap protocol, which the RFC simply lacked. Not "seed the primary from one palace": every container joins the same way, repeatedly, so a join is idempotent replay and the only question per record type is what dedupes it. Drawers and open KG facts need zero client bookkeeping; closed KG facts and diaries need client-side keys. Join state belongs in the palace dir, not the container (~/.mempalace is not preserved by default), which also makes two containers sharing one host's palace the easy case rather than a double-upload hazard. - §3.2 — the add_triple guard is scoped to `valid_to IS NULL`, so closed historical facts are unguarded. §3 read as unconditionally idempotent. - §7.3 — drawer/diary metadata is exhaustive: no session, PID or conversation field. Two concurrent pi sessions are indistinguishable, and pi-vs-opencode is only accidentally distinguishable because the pi extension never sets added_by (it sets identity for diaries only, mempalace.ts:758). Corrects the §7.3.3 bullet that claimed multi-harness attribution was already solved — the field is the right home, but nothing populates it. Design rule stated: device + agent, never session; provenance granularity equals token granularity. - §6.2 — Host-pinning is coupled to the bind address (enforce_host_pin = _http_is_loopback(host), :5367). The trap is the RFC's own "bind loopback" reflex: behind a proxy that forwards a public Host, a loopback bind 403s, while a non-loopback bind deliberately relaxes the pin. The Origin check is never relaxed. This replaces the warning I first wrote, which had it backwards. - §9.3 — largely resolved: the last-chunk-only probe is deliberate (batched upsert is all-or-nothing, :2586-2592). Only a crash mid-upsert remains untested. - §4.1 — fix shape for opencode's stale-config asymmetry: make the mcp.mempalace subtree env-authoritative, with a fingerprint so hand-edits still win. pi-devbox/entrypoint-user.sh:131-162 already has the jq deep-merge pattern to port; no OPENCODE_CONFIG* layer exists upstream, so the real file must be written. Now Phase 1.5. Decisions (new §8.1): primary = synlig; TLS at Pangolin; single shared token for now (so origin_device stays advisory); diaries replicated. Work/personal is per-wing, not per-device — two primaries split along machine lines is rejected, because pi-devbox/opencode-devbox are simultaneously work and home projects and the device where work happened cannot classify the project. MEMPALACE_REMOTE_URL stays scalar so multi-store remains additive later (new §9.7 keeps the placement question open).
This commit is contained in:
Joakim Persson
+202
-14
@@ -128,6 +128,11 @@ independently disqualifying:
|
||||
every fact. But `add_triple()` guards at the application level (`SELECT id … WHERE subject=? AND
|
||||
predicate=? AND object=? AND valid_to IS NULL` → returns the existing id), so **replaying `kg_add`
|
||||
is idempotent for open facts** (`knowledge_graph.py:163-178`, `305-313`).
|
||||
⚠️ **Scoped to open facts only** (verified 2026-08-09). The guard's `WHERE … valid_to IS NULL` means an
|
||||
already-**closed** historical fact — one written with `valid_to`, or closed later by
|
||||
`kg_invalidate`/`kg_supersede` — has no guard at all, so replaying it inserts a duplicate row every
|
||||
time. A bootstrap that replays full KG *history* rather than just currently-open facts must dedupe
|
||||
closed facts client-side on `(s,p,o,valid_from,valid_to)` (§4.4).
|
||||
3. **Only one write path has deterministic IDs.**
|
||||
|
||||
| Write path | ID recipe | Same content on 2 hosts → same ID? |
|
||||
@@ -135,7 +140,7 @@ independently disqualifying:
|
||||
| `add_drawer` / `checkpoint` | `drawer_{wing}_{room}_{sha256(wing\|room\|content)[:24]}` | **YES** — content-addressed, merges for free |
|
||||
| project/format miner | `…sha256(source_file\|chunk_index)` | **NO** — absolute path (`/Users/joakim/x` vs `/workspace/x`) |
|
||||
| convo miner | `…sha256(source_file\|extract_mode\|chunk_index)` | **NO** — same reason |
|
||||
| diary | `diary_{wing}_{YYYYmmdd_HHMMSSffffff}_{sha256(entry)[:12]}` | **NO** (µs timestamp) — but the 12-hex suffix is a usable content dedup key |
|
||||
| diary | `diary_{wing}_{YYYYmmdd_HHMMSSffffff}_{sha256(entry)[:12]}` | **NO** (µs timestamp) — and the write is a bare `col.add` with **no pre-write probe at all** (`mcp_server.py:3546`), unlike `add_drawer`. The 12-hex suffix is the only usable content dedup key, and it must be applied client-side (§7.6) |
|
||||
| KG triple | `t_{s}_{p}_{o}_{sha256(valid_from\|recorded_at)[:12]}` | **NO** (`recorded_at = now()`) |
|
||||
|
||||
(`ids.py:56,71` — `ID_RECIPE = "v3"`, length-prefixed delimited hashing.)
|
||||
@@ -148,14 +153,20 @@ independently disqualifying:
|
||||
vocabulary. Log the *intent*, replay the *intent*. That sidesteps chroma internals, chunking,
|
||||
embedder drift and vector portability in one move.
|
||||
|
||||
**One exception, and it is not cosmetic:** `diary_write` has no idempotency guard whatsoever, so for
|
||||
diaries "replay the intent" *duplicates* rather than merges (§7.6). Every other write path either
|
||||
content-addresses or guards. §4.4 carries the per-record-type dedup keys a joining client must
|
||||
therefore bring with it.
|
||||
|
||||
---
|
||||
|
||||
## 4. Design: `mempalace-edge`, a local MCP proxy
|
||||
|
||||
### 4.1 Shape
|
||||
|
||||
Do **not** put fallback logic in `extensions/pi/mempalace.ts` — opencode would need it again, and it
|
||||
may not support remote MCP at all (§9.1). Instead: a sidecar that *is* an MCP server.
|
||||
Do **not** put fallback logic in `extensions/pi/mempalace.ts` — opencode would need it again — not
|
||||
because opencode lacks remote MCP (it has it, §9.1), but because *offline-first writes* are a client
|
||||
concern that every harness would otherwise reimplement. Instead: a sidecar that *is* an MCP server.
|
||||
|
||||
```
|
||||
pi (mempalace.ts, stdio) ─┐
|
||||
@@ -206,6 +217,32 @@ image (it costs nothing unused) and inserted into the path only when `MEMPALACE_
|
||||
> (reversibility). This asymmetry with pi — whose `createClient()` re-reads env every start — is the
|
||||
> single biggest behavioural difference between the two harnesses under this RFC.
|
||||
|
||||
> ✅ **Fix shape (decided 2026-08-09): make the `mcp.mempalace` subtree env-authoritative.** Four
|
||||
> verified inputs:
|
||||
> 1. **pi is immune for a *structural* reason worth naming:** its MemPalace wiring is **code, not
|
||||
> config** — `createClient()` reads env at every startup, with no generated file in the path. pi's
|
||||
> own `settings.json` *does* sit on a preserved volume and *does* go stale, and pi-devbox already
|
||||
> solved that properly at `pi-devbox/entrypoint-user.sh:131-162`: `jq -s '.[0] * .[1]'` deep-merge
|
||||
> with **template first, live second** so the user's values always win and only *missing* keys are
|
||||
> filled; arrays treated as leaves (a deliberately removed model is not re-added); rewrite only when
|
||||
> the merge changes something; timestamped `.bak` first; `PI_SETTINGS_MERGE=0` to disable; invalid
|
||||
> JSON on either side → skip, never clobber. **That is the pattern to port.**
|
||||
> 2. **The precedence must be inverted for this one subtree.** pi's "live wins" is right for *adding*
|
||||
> new keys and wrong for a *changed* env value — and a changed `MEMPALACE_REMOTE_URL` is the whole
|
||||
> problem. So `mcp.mempalace` needs env-wins, which is only safe with a **fingerprint**: store a
|
||||
> hash of what was last auto-generated and refresh only while the live value still matches it;
|
||||
> otherwise fall back to the `.proposed` sidecar for that key. `write_proposed`
|
||||
> (`generate-config.py:203-244`) already diffs rendered config against the live file, so this is an
|
||||
> extension of existing logic at narrower granularity, not new machinery.
|
||||
> 3. **No higher-precedence layer exists to hide in.** There is no `OPENCODE_CONFIG*` env override in
|
||||
> opencode's published schema (or anywhere in `opencode-devbox`/`myconfigs`), and MCP registration
|
||||
> is global rather than project-scoped, so the real file must be written.
|
||||
> 4. **The fingerprint is the one genuinely new mechanism** — neither repo has a managed-marker
|
||||
> convention for JSON; the only prior art is the markdown `<!-- pi-devbox:managed-block -->` idiom.
|
||||
>
|
||||
> This is a **self-contained opencode-devbox change, independent of the rest of this RFC**, and it is
|
||||
> what turns Phase 1 into "flip `.env`, restart" for *both* harnesses — hence Phase 1.5 in §8.
|
||||
|
||||
### 4.2 Routing policy
|
||||
|
||||
`service.py:29,54,72` already ships the exact three-way split we need — `READ_TOOLS`,
|
||||
@@ -262,6 +299,47 @@ backoff, resume on reconnect. The primary de-dupes on `op_id`; `add_drawer`'s ow
|
||||
gain a `REMOTE` target. The module says it "*changes no caller defaults by itself*" — i.e. it exists
|
||||
precisely to be extended.
|
||||
|
||||
### 4.4 Joining an existing primary (bootstrap)
|
||||
|
||||
**Framing correction (2026-08-09).** This is *not* "seed the primary from one chosen palace". Every
|
||||
container joins the same way, at any time, repeatedly — so a join is **idempotent replay of local
|
||||
history**, and the only real question per record type is *what dedupes it*.
|
||||
|
||||
| Record type | Dedupe on replay | Client work needed |
|
||||
| --- | --- | --- |
|
||||
| `add_drawer` / `checkpoint` drawers | **Server-side**: content-addressed id + pre-write `col.get` probe → `{"reason":"already_exists"}`, no write (`mcp_server.py:2593-2600`) | **None.** Just replay |
|
||||
| KG **open** facts | **Server-side**: `add_triple` guard on `(s,p,o) WHERE valid_to IS NULL` | **None.** Just replay |
|
||||
| KG **closed** facts | **None** — the guard is scoped to open facts (§3.2) | Dedupe on `(s,p,o,valid_from,valid_to)` before sending |
|
||||
| **Diary entries** | **None whatsoever** (§7.6) | Skip any local entry whose `sha256(entry)[:12]` suffix already exists remotely |
|
||||
| Mined drawers | id is path-dependent → same content from two hosts = two rows | Out of scope: mined wings are `local` (§5) |
|
||||
|
||||
Existence checks available today, with **no new server code**:
|
||||
|
||||
| Tool | Kind | Fit |
|
||||
| --- | --- | --- |
|
||||
| `get_drawer(id)` | exact id, clean not-found (`mcp_server.py:3103-3114`) | The right check wherever ids are deterministic |
|
||||
| `list_drawers(wing, room, since, before)` | metadata page | Bulk "what does this wing already hold" — the cheap way to collect existing diary id suffixes. Note `since`/`before` filter in **Python**, not in the backend `where` (chroma 1.5.7 rejects string `$gte`/`$lt`) |
|
||||
| `check_duplicate(content, threshold)` | semantic, **whole collection, no wing/room scoping** | One embed + one HNSW query *per call* → the cost driver at thousands-of-records scale. Reserve it for diaries, where nothing cheaper works |
|
||||
| `kg_query` | fact lookup | Redundant for open facts (the guard covers them); useful for closed ones |
|
||||
|
||||
**Two containers on one host are the *easy* case, not the hard one.** They share a bind-mounted palace,
|
||||
so it is *one* palace joining once, and content-addressing makes even a concurrent double-join harmless
|
||||
for everything except diaries. Belt and braces:
|
||||
|
||||
- **Keep join state in the shared palace, not in the container** — e.g. `<palace>/edge/bootstrap.json`
|
||||
holding `{target_url: {joined_at, high_water_local_seq}}`. Both containers then see "this palace has
|
||||
already joined", a recreate does not repeat the work, and it is the same durable-state-next-to-the-data
|
||||
pattern as the outbox (§4.3). It must **not** live in a container-only path: `~/.mempalace` is not
|
||||
preserved by default for solitary users (§1.2), which is exactly why the file belongs to the palace
|
||||
directory.
|
||||
- **First join is a dry run.** Bootstrap one palace, verify counts (`status`, `kg_stats`, per-wing
|
||||
`list_drawers`) against expectations, *then* let the rest join. Ordering matters only because of diaries
|
||||
and closed facts; everything else is order-free.
|
||||
|
||||
The genuinely hard case is **two different palaces holding overlapping mined content** — the same repo
|
||||
mined on a laptop and a workstation under different absolute paths. §5 excludes it by keeping mined wings
|
||||
`local`; that exclusion is load-bearing, not tidiness.
|
||||
|
||||
---
|
||||
|
||||
## 5. Not everything should be global
|
||||
@@ -277,6 +355,25 @@ This shrinks the hard problem to the layer that is already safe, and it is why *
|
||||
remote**: the primary serializes *every* request behind one lock (§7.5), so a bulk remote mine would
|
||||
stall every other agent in the fleet.
|
||||
|
||||
**Decided 2026-08-09: diaries are `replicated`.** Cross-machine continuity is the entire point, and
|
||||
diaries are the highest-value content in the palace to share ("*given their value, I say go with (a)*").
|
||||
Accepted consequence, stated plainly because it follows from the primary being **synlig, a work VM**
|
||||
(§8.1): personal diaries will live on employer infrastructure. They already quote internal hostnames,
|
||||
paths, moods and the occasional token prefix (§6.1 threat 2) — so "don't put secrets in memory" stops
|
||||
being advice and becomes a precondition.
|
||||
|
||||
**The work/personal boundary is a property of the wing, not of the device.** Rejected design
|
||||
(2026-08-09): splitting the fleet into a work primary and a personal primary along machine lines. The
|
||||
reasoning is decisive — pi-devbox/opencode-devbox are *simultaneously* work and home projects, and
|
||||
personal machines get used for work-adjacent work, so **the device where the work happened cannot
|
||||
classify the project**. That is precisely the axis this table already encodes, which is why
|
||||
`replicated`/`local` per wing is the right knob and a second primary is not needed to express it.
|
||||
|
||||
**Shape the config so multiple stores stay possible without paying for them now.** Phase 1 keeps
|
||||
`MEMPALACE_REMOTE_URL` a **scalar** (one primary). Per-wing *targets* — this same policy column, plus a
|
||||
destination — belong to the edge config in Phase 2+, so nothing in the `.env` contract has to be
|
||||
un-designed later.
|
||||
|
||||
---
|
||||
|
||||
## 6. Security model
|
||||
@@ -300,9 +397,9 @@ The transport is largely solved; **the gap is authorization, not cryptography.**
|
||||
|
||||
| Control | Decision |
|
||||
| --- | --- |
|
||||
| **Network posture** | Primary **never** internet-exposed. Bind loopback in the container; publish only onto the private overlay / existing tunnel (Pangolin/newt). |
|
||||
| **Transport** | TLS via `serve --tls-cert/--tls-key`, or terminate TLS **+ mTLS** in the reverse proxy (cheaper than patching Python's TLS surface). |
|
||||
| **Authentication** | **Per-device bearer tokens**, not the one shared token. Server-side registry `token → {device_id, scopes}`. Store in the existing `.env.age` flow, 0600 on disk. Enables revoking one laptop and rotating without a fleet outage. |
|
||||
| **Network posture** | Primary **never** internet-exposed. Publish only onto the private overlay / existing tunnel (Pangolin/newt). ⚠️ **The reflexive "bind loopback in the container" is the failure mode here — see Transport.** |
|
||||
| **Transport** | **Decided 2026-08-09: terminate TLS in the existing Pangolin/newt tunnel**, not in `serve` (cheaper than patching Python's TLS surface; one DNS record per service on the web hotel is the whole setup cost). ⚠️ **Host-pinning is coupled to the bind address**, verified: `enforce_host_pin = _http_is_loopback(host)` (`mcp_server.py:5367`). On a **loopback** bind, `Host` is pinned to loopback literals + the bound host, so a proxy forwarding `Host: palace.example.com` gets **403 Forbidden** — either make the proxy rewrite `Host` to `127.0.0.1:<port>`, or bind the private interface instead. On a **non-loopback** bind the pin is deliberately **relaxed** ("*may sit behind a proxy that rewrites Host … lean on the Origin check + optional token instead*", `:5362-5365`), which is also why `cli.py:1450` makes a tokenless non-loopback bind require `--allow-insecure`. **The `Origin` check is never relaxed:** an absent `Origin` is allowed (every non-browser MCP client, incl. pi and opencode), but a *present* non-loopback `Origin` is 403 with no override — so keep browser-based clients and `Origin`-injecting proxies out of the path. |
|
||||
| **Authentication** | Target: **per-device bearer tokens** with a server-side `token → {device_id, scopes}` registry — revoke one laptop, rotate without a fleet outage. **Decided 2026-08-09: Phase 1 ships the single shared token** ("*iterate more feature rich but more complex solutions over time*"), so per-device lands with Phase 4. Store in the existing `.env.age` flow, 0600 on disk. **Consequence: until then the primary cannot tell devices apart, so `origin_device` stays client-asserted and advisory — nothing load-bearing may depend on it (§7.3.2).** |
|
||||
| **Authorization** | Per-device read/write **wing globs**. Server-side refusal of `mine`/`sync`/`delete_*` except for an admin device. `--read-only` gives a free observer tier. |
|
||||
| **Provenance/audit** | `origin_device` (+ optional `origin_label`) + `op_id` on every record, stamped **server-side from the authenticated credential** — a client-asserted origin is a hint, not a fact (§7.3.2); server-side append-only op log. |
|
||||
| **Recovery** | Deletes as tombstones; server-side backups with `backups.py` retention (`MEMPALACE_MAX_BACKUPS`). |
|
||||
@@ -312,7 +409,8 @@ The transport is largely solved; **the gap is authorization, not cryptography.**
|
||||
|
||||
## 7. Landmines — Phase 0 runbook
|
||||
|
||||
Do these **before** any cutover. 7.1 and 7.2 cause silent data loss.
|
||||
Do these **before** any cutover. 7.1 and 7.2 cause silent data loss; 7.6 causes silent *duplication*
|
||||
the first time a palace joins (§4.4).
|
||||
|
||||
### 7.1 `MEMPALACE_PALACE_PATH` ≠ `--palace` (silent empty KG)
|
||||
|
||||
@@ -357,6 +455,27 @@ Today every drawer carries exactly `{wing, room, source_file, added_by, filed_at
|
||||
is the *agent* name (`pi`, `mcp`, `checkpoint`) — **never the machine**. So in a merged store you cannot
|
||||
tell which host wrote a record, cannot audit, and cannot compute per-device high-water marks.
|
||||
|
||||
**Verified 2026-08-09 (`mcp_server.py:2578-2585` drawers, `3527-3536` diaries): those lists are
|
||||
exhaustive — there is no session, PID or conversation field either.** Two consequences, because both are
|
||||
natural questions:
|
||||
|
||||
- **Two concurrent pi sessions** (the tmux pattern pi's own docs suggest) writing to one palace are
|
||||
**indistinguishable**. Nothing records which session produced which record.
|
||||
- **pi vs opencode is only *accidentally* distinguishable.** `added_by` is a free-form optional string
|
||||
(default `"mcp"`; `checkpoint` resolves explicit arg → diary `agent_name` → `"checkpoint"`), and
|
||||
`extensions/pi/mempalace.ts` **never sets it** for `add_drawer`/`checkpoint` — it sets identity only for
|
||||
diaries (`agent_name` from `$MEMPALACE_AGENT_NAME`, default `pi`, `:758`). `kg_add` has no attribution
|
||||
field at all. So the *only* real harness attribution today is the diary wing, and for drawers the value
|
||||
is whatever string an LLM happened to pass.
|
||||
|
||||
**Design consequence: device + agent, never session.** Provenance has exactly three consumers — poisoning
|
||||
triage ("which box planted this?"), per-device high-water marks, and revocation — and none of them needs
|
||||
session granularity; adding it would put a field on every record with no reader. Under §6's per-device
|
||||
tokens, **provenance granularity equals token granularity**: one token per host makes two containers on
|
||||
that host a single origin, while a token per container separates them. §7.3.4's `<agent>/<label>/<device>`
|
||||
encoding keeps the two axes orthogonal, so pi-vs-opencode stays recoverable — but only if the **edge**
|
||||
populates the agent segment, never the LLM.
|
||||
|
||||
**Earlier drafts of this section said "stamp it everywhere, now, because it cannot be backfilled." That
|
||||
was wrong on both counts.** See §7.3.3.
|
||||
|
||||
@@ -418,8 +537,11 @@ from *multiple* origins are interleaved in one store, which is exactly and only
|
||||
carry metadata for a feature they had opted out of.
|
||||
- **Migration is unaffected**: bulk attribution at import is *more* reliable than per-record stamping,
|
||||
because it cannot be partially applied.
|
||||
- **Multi-harness on one host stays solved** by `added_by` = agent name (`pi` vs `opencode`) — that is
|
||||
what the field is for, and it needs no device component.
|
||||
- **Multi-harness on one host is solved *in principle*** by `added_by` = agent name (`pi` vs `opencode`) —
|
||||
that is what the field is for, and it needs no device component. ⚠️ **Corrected 2026-08-09: nothing
|
||||
actually sets it.** The pi extension leaves `added_by` at its default for `add_drawer`/`checkpoint`, so
|
||||
in practice the value is whatever an LLM passed. The field is the right home; the client-side write that
|
||||
populates it is missing, and it belongs to the edge (the ⚠️ row of §7.3.2) — not to a skill instruction.
|
||||
- **A palace on a shared host bind-mount** (as tor-ms22's compose does) is still single-*device* under
|
||||
the "host owns the palace" model, so it too imports as one origin.
|
||||
|
||||
@@ -501,14 +623,44 @@ across models silently degrades recall.
|
||||
NFS/SMB is unreliable, and the lock wouldn't be shared anyway. One process owning the files and
|
||||
serving HTTP is the supported model — which is exactly why `serve` exists.
|
||||
|
||||
### 7.6 `diary_write` has no idempotency guard — replay duplicates every entry
|
||||
|
||||
```python
|
||||
# mcp_server.py:3511-3513, then 3546
|
||||
entry_id = f"diary_{wing}_{now:%Y%m%d_%H%M%S%f}_{hashlib.sha256(entry.encode()).hexdigest()[:12]}"
|
||||
...
|
||||
col.add(ids=[entry_id], ...) # not upsert — and no col.get probe anywhere in tool_diary_write
|
||||
```
|
||||
|
||||
Compare `add_drawer` 900 lines earlier, which probes `col.get(ids=idempotency_probe_ids)` and returns
|
||||
`{"reason": "already_exists"}` without writing (`:2593-2600`). The contrast *is* the finding: **the µs
|
||||
timestamp makes every diary id unique by construction, and nothing checks the content.** So §3's
|
||||
conclusion — log the intent, replay the intent — is **false for diaries**; for them, replay is
|
||||
duplication.
|
||||
|
||||
This matters more than it sounds, because diaries are `replicated` (§5) and are precisely the content a
|
||||
join replays (§4.4).
|
||||
|
||||
**Actions:**
|
||||
|
||||
1. **Client-side, now:** before replay, list the target's diary drawers for the wing, extract the 12-hex
|
||||
`sha256(entry)[:12]` suffix from each existing id, and skip any local entry whose suffix already
|
||||
appears. The suffix is already content-addressed — no new key needs inventing.
|
||||
2. **Upstream ask:** make `diary_write` probe-and-skip on that suffix the way `add_drawer` does on its
|
||||
content hash. A small, self-contained patch in the sibling function's own idiom, and it closes the gap
|
||||
for *every* future writer instead of for one migration script.
|
||||
3. **Do not "fix" this by making the agent dedupe.** An agent cannot know what the primary holds without a
|
||||
call, and per-call boilerplate is exactly the failure mode §7.3.2 rejects.
|
||||
|
||||
---
|
||||
|
||||
## 8. Phasing
|
||||
|
||||
| Phase | Effort | Deliverable |
|
||||
| --- | --- | --- |
|
||||
| **0 — hygiene** | hours | §7 runbook: move KG/hallways/entities, ban `sync` on shared palaces, fix stale "unauthenticated" docs (incl. `pi-devbox/.env.example:21`). **No provenance work here** — it is not backfill-critical (§7.3.3) and belongs to the stamper, not the agent |
|
||||
| **1 — primary up** | hours, **no code** | `mempalace serve --token --tls-cert` on a private-net host (reuse `docker-compose.mempalace.yml` — keep it a **separate standalone project**, R4 — and mind port 8765 vs pi-studio; tor-ms22 already moved to 8766). Repoint pi clients via `MEMPALACE_REMOTE_URL`/`MEMPALACE_REMOTE_TOKEN`. **opencode clients can be repointed in the same breath** — remote MCP is supported and `generate-config.py` already emits it (§9.1, §9.6), subject to the sidecar caveat in §4.1. **Shared memory today, no offline.** |
|
||||
| **0 — hygiene** | hours | §7 runbook: move KG/hallways/entities **on synlig, before first `serve`** (§7.1), ban `sync` on shared palaces (§7.2), fix stale "unauthenticated" docs (incl. `pi-devbox/.env.example:21`). Added 2026-08-09: settle the **diary dedup** approach and file its upstream ask (§7.6), and **dry-run the join from one palace only** (§4.4). **No provenance work here** — it is not backfill-critical (§7.3.3) and belongs to the stamper, not the agent |
|
||||
| **1 — primary up** | hours, **no code** | `mempalace serve --token --tls-cert` on a private-net host (reuse `docker-compose.mempalace.yml` — keep it a **separate standalone project**, R4 — and mind port 8765 vs pi-studio; tor-ms22 already moved to 8766). Repoint pi clients via `MEMPALACE_REMOTE_URL`/`MEMPALACE_REMOTE_TOKEN`. **opencode clients can be repointed in the same breath** — remote MCP is supported and `generate-config.py` already emits it (§9.1, §9.6), subject to the sidecar caveat in §4.1. **Shared memory today, no offline.** **Decided 2026-08-09: primary = synlig, TLS at Pangolin, single shared token (§8.1)** — mind the loopback Host-pin trap in §6.2. |
|
||||
| **1.5 — opencode env propagation** | hours | Make the `mcp.mempalace` subtree env-authoritative in `generate-config.py`, gated by a generated-value fingerprint (§4.1). Independent of the rest of this RFC. Without it, adopting *or reverting* the opt-in on an existing opencode container needs a manual sidecar merge or a `docker volume rm` — which also blocks R6 reversibility |
|
||||
| **2 — `mempalace-edge`** | ~1 week | The actual ask: local-first writes + outbox flush + merged reads + per-wing policy. **Not** "fixes opencode" — opencode's *transport* is already fine after Phase 1; what edge adds there is offline/local-first, since `generate-config.py`'s switch is remote **or** local with no failover. **Ships with the §1.2 opt-in wiring (compose + `.env.example` + a third branch in the existing `generate-config.py`) and must pass the R1 acceptance test.** |
|
||||
| **3 — pull replication** | ~1 week | **DEFERRED (2026-08-08).** Server-side op-log with monotonic seq → each edge a full offline replica. Only needed if a laptop must hold *everything* offline. Accepted trade-off: offline recall = own writes + last-reachable state. |
|
||||
| **4 — authz** | days | Per-wing ACL, per-device scopes, audit log, token rotation |
|
||||
@@ -516,6 +668,17 @@ across models silently degrades recall.
|
||||
Phase 1 is worth doing on its own — it is pure configuration and immediately ends KG fragmentation
|
||||
for online clients.
|
||||
|
||||
### 8.1 Deployment decisions (2026-08-09)
|
||||
|
||||
| Decision | Value | Note |
|
||||
| --- | --- | --- |
|
||||
| **Primary host** | **synlig** — a VM in Xerces (work OpenStack cloud) | Chosen for always-on + good connectivity, *not* for work/personal reasons. Consequence: replicated content, diaries included, lives on employer infrastructure (§5, §6.1 threat 2) |
|
||||
| **TLS / ingress** | **Pangolin/newt**, one DNS record per service on the web hotel | Not `serve --tls-cert`. See the bind-address/Host-pin coupling in §6.2 |
|
||||
| **Authentication** | **Single shared token** for now | Per-device deferred to Phase 4; `origin_device` stays advisory until then |
|
||||
| **Diaries** | **`replicated`** | Highest-value cross-machine content; accepted placement consequence (§5) |
|
||||
| **Work/personal split** | **per-wing, not per-device** | Rejects two primaries split along machine lines — the device cannot classify the project (§5) |
|
||||
| **Multi-store** | Possible later, not now | `MEMPALACE_REMOTE_URL` stays a scalar in Phase 1; per-wing targets are an edge-config concern (Phase 2+) |
|
||||
|
||||
---
|
||||
|
||||
## 9. Open questions
|
||||
@@ -536,19 +699,30 @@ for online clients.
|
||||
(`bin/` + `extensions/`). But `origin_device` stamping from the authenticated token, per-wing ACL, the `kg_supersede`
|
||||
classification fix, and a `sync --refuse-shared` guard all want to go **upstream**
|
||||
(`github.com/MemPalace/mempalace`).
|
||||
3. **Chunked drawers under merge.** Oversized content splits into `{drawer_id}_chunk_NNNNNN` with
|
||||
3. **Chunked drawers under merge.** ~~Oversized content splits into `{drawer_id}_chunk_NNNNNN` with
|
||||
`parent_drawer_id`, and the add-time idempotency probe checks only the *last* chunk id — a
|
||||
partially transferred chunk set may look "already present" and stay truncated. Unverified; test
|
||||
before trusting bulk replay.
|
||||
partially transferred chunk set may look "already present" and stay truncated.~~ **Largely resolved
|
||||
2026-08-09: the last-chunk-only probe is deliberate, not an oversight.** The code's own comment
|
||||
(`mcp_server.py:2586-2592`) states that the last chunk's presence implies every earlier one landed
|
||||
*because the batched `upsert` is all-or-nothing*, and it additionally probes `drawer_id` alongside so a
|
||||
re-call with identical oversized content cannot duplicate a legacy pre-#1539 single-row write. The
|
||||
normal path is therefore safe by construction. What remains untested is the abnormal one: a chunk set
|
||||
left partial by a crash or kill *mid-upsert*, which would then look present and stay truncated. Worth a
|
||||
single fault-injection test before trusting bulk replay of oversized drawers.
|
||||
4. **`migrate.py` as a bootstrap tool.** `extract_drawers_from_sqlite()` reads `{id, document,
|
||||
metadata}` straight out of chroma's SQLite (bypassing the chromadb API) and re-`add`s them with ids
|
||||
and metadata preserved and **embeddings recomputed** (`migrate.py:326`) — the right shape for a
|
||||
one-shot "seed the primary from an existing palace". Unverified: `col.add` behaviour on **id
|
||||
collision into a non-empty target**. Test before relying on it. (`exporter.py` is markdown-only,
|
||||
lossy, and has **zero callers** — not a transfer format. `backups.py` is retention pruning only.)
|
||||
**Reframed 2026-08-09:** §4.4 makes **MCP-level replay** the primary join path — it reuses the
|
||||
server's own idempotency guards instead of trusting chroma-level `add` semantics — so `migrate.py` is
|
||||
now a *fallback* for bulk transfer rather than the plan, and its untested collision behaviour only
|
||||
matters if we reach for it.
|
||||
5. **Does `_HTTP_REQUEST_LOCK` stay held for the duration of an MCP-triggered `mine`?** Strongly
|
||||
suggested by the code shape; if yes, a remote mine makes the primary unusable for its duration
|
||||
(another argument for §5).
|
||||
|
||||
6. ~~**How does opencode-devbox learn the opt-in?**~~ **RESOLVED 2026-08-09: it already does.** The
|
||||
image ships `rootfs/usr/local/lib/opencode-devbox/generate-config.py`, invoked from
|
||||
`entrypoint-user.sh:117` on every start, which auto-registers the `mempalace` MCP server —
|
||||
@@ -563,6 +737,13 @@ for online clients.
|
||||
this script over inventing a parallel mechanism — treat it as the reference implementation, and
|
||||
check it before designing any pi-devbox-side mechanism with an opencode counterpart.
|
||||
|
||||
7. **Is a work VM the right long-term home for personal memory?** Accepted with eyes open for Phase 1
|
||||
(§8.1): synlig wins on availability and connectivity, and the alternative — splitting by machine — was
|
||||
rejected for sound reasons (§5). But the placement question survives the routing question: diaries are
|
||||
`replicated` (decided), diaries are personal, and synlig is employer infrastructure. Revisit when
|
||||
per-wing *targets* exist (Phase 2+), at which point a second personal primary becomes an additive
|
||||
config change rather than a re-architecture. Until then, treat §6.1 threat 2 as live.
|
||||
|
||||
---
|
||||
|
||||
## 10. Evidence index
|
||||
@@ -577,6 +758,13 @@ Re-verify without re-reading the package. Paths relative to
|
||||
| KG path follows `--palace` flag only | `mcp_server.py:325`, `708-711`; `knowledge_graph.py:49` |
|
||||
| KG DDL, no `(s,p,o)` uniqueness, open-fact guard | `knowledge_graph.py:163-178`, `305-313` |
|
||||
| ID recipes | `ids.py:25`, `56`, `71`; diary at `mcp_server.py:3511-3514` |
|
||||
| **Diary write has no idempotency probe** | `mcp_server.py:3546` (`col.add`, no `col.get`) vs `add_drawer`'s probe at `2593-2600` |
|
||||
| Chunk probe is deliberate (all-or-nothing upsert) | `mcp_server.py:2586-2592` |
|
||||
| Metadata is exhaustive — no session/PID/conversation field | `mcp_server.py:2578-2585` (drawers), `3527-3536` (diaries) |
|
||||
| pi extension sets identity for **diaries only** | `extensions/pi/mempalace.ts:758` (`agent_name`); no `added_by` set anywhere in the file |
|
||||
| Host-pin coupled to bind address; Origin never relaxed | `mcp_server.py:5160-5194`, `5276-5290`, `5362-5367`; `cli.py:1450` |
|
||||
| pi-devbox non-destructive settings merge (pattern to port) | `pi-devbox/entrypoint-user.sh:131-162` (`jq -s '.[0] * .[1]'`, `.bak`, `PI_SETTINGS_MERGE=0`) |
|
||||
| No `OPENCODE_CONFIG*` env layer upstream | absent from `https://opencode.ai/config.json`; zero hits in `opencode-devbox`, `myconfigs` |
|
||||
| Tool classification (+ `kg_supersede` gap) | `service.py:29`, `54`, `72-84` |
|
||||
| Write-routing seam | `write_routing.py:1-60` |
|
||||
| Outbox prior art (queue DDL, token, retries) | `daemon.py` (`_init_db`, `ensure_token`, `MAX_ATTEMPTS`) |
|
||||
|
||||
Reference in New Issue
Block a user