diff --git a/docs/rfc-002-joiner.md b/docs/rfc-002-joiner.md new file mode 100644 index 0000000..89f5308 --- /dev/null +++ b/docs/rfc-002-joiner.md @@ -0,0 +1,147 @@ +# RFC 002 — The joiner: replaying a second palace into the shared primary + +**Status:** scoping. No code written yet. +**Author:** pi (agent), 2026-08-15. +**Context:** RFC 001 §4.4 designs a join as "idempotent replay of local history" but no replay tool +exists. The first two joins were whole-palace *file copies* (§4.4 deviation note), which work only for a +single source and cannot merge. tor-ms22 and MBP-M1-2020 each hold a substantial local palace whose +content should reach the primary. This document scopes the tool that does that. + +> **Every mechanism below was read out of mempalace 3.6.0's own source** at +> `/opt/uv-tools/mempalace/lib/python3.13/site-packages/mempalace/`, with file:line in the appendix. +> An earlier attempt to gather these facts via a delegated subagent returned confident, fabricated code +> for a package path that does not exist on this machine. **Do not trust any claim in this document that +> the appendix does not cite.** + +--- + +## 1. The finding that drives the design + +**MCP replay cannot preserve `filed_at`.** `add_drawer` stamps +`"filed_at": datetime.now().isoformat()` server-side (`mcp_server.py:2580`) and exposes no override +parameter. `diary_write` is the same: it builds its own `now`-based id (`mcp_server.py:3510-3513`). + +That is not a detail. This palace's value *is* its chronology — the primary's history runs from +2026-05-04, `list_drawers` filters on `since`/`before` against `filed_at`, and the diary is read in +order. A pure MCP replay of tor-ms22's palace would stamp **every** record with the join date, +collapsing months of history into one instant. RFC 001 §4.4 does not mention this, and it is the single +most important thing to decide before writing code. + +`kg_add` is the exception: it accepts `valid_from`/`valid_to`, so **fact validity windows survive** even +though a triple's own id embeds `recorded_at`. + +### Two write regimes + +| | **A — direct disk write** (run on synlig) | **B — MCP replay** (run anywhere) | +| --- | --- | --- | +| Preserves `filed_at` / original ids | **Yes** — `col.add(ids=…, documents=…, metadatas=…)` | **No** — always `now()` | +| Server-side dedup guards | **Bypassed** — joiner owns all dedup | **Available** (see §2) | +| Requires quiescing `mempalace-serve` | Yes (palace is single-writer) | No | +| Requires source palace present on synlig | Yes (rsync it first) | No | +| Precedent in-tree | **`migrate.py`** already does exactly this: reads drawers+metadata straight from the palace's sqlite, then re-adds them into a fresh palace preserving ids, documents and metadata (`migrate.py:326-330`) | none | + +**Recommendation:** regime **A** for the historical bulk (it is the only one that keeps the timeline, and +`migrate.py` is a working model to copy), regime **B** for small incremental top-ups where flattened +timestamps are acceptable. Do not build B first and discover the chronology loss afterwards. + +--- + +## 2. What must move, and what dedupes it + +Verified dedup keys — this supersedes nothing in RFC 001 §4.4 but makes each key concrete: + +| Class | How to identify it in the source palace | Dedup key (verified) | Regime A work | Regime B work | +| --- | --- | --- | --- | --- | +| **Agent-authored drawers** (`add_drawer`/`checkpoint`) | `source_file` empty | id is `sha256("{wing}\|{room}\|{content}")[:24]` (`ids.py:80`, used at `mcp_server.py:2560`) — **fully content-deterministic** | recompute id, skip if present | none: server probes `[drawer_id, last_chunk_id]` and returns `{"success": True, "reason": "already_exists"}` (`mcp_server.py:2593-2604`) | +| **Mined drawers** | `source_file` set | id is `sha256("{source_file}\|{chunk_index}")[:24]` (`ids.py:67`) — **path-dependent**, so the same content from two machines yields two rows | — | **do not replay.** Re-mine on synlig instead (§4) | +| **Diary entries** | id prefix `diary_`, `room="diary"` | id is `diary_{wing}_{now:%Y%m%d_%H%M%S%f}_{sha256(entry)[:12]}` (`mcp_server.py:3510-3513`). Timestamp is wall-clock, so **full ids never repeat** — match on the 12-hex suffix only | copy row verbatim (id and all) | build the target's suffix set, skip matches — this is RFC 001 §7.6 | +| **KG open facts** (`valid_to IS NULL`) | `triples.valid_to IS NULL` | server guard `WHERE subject=? AND predicate=? AND object=? AND valid_to IS NULL` returns the existing id (`knowledge_graph.py:305-311`) | pre-query same key | none: just replay | +| **KG closed facts** (`valid_to` set) | `triples.valid_to IS NOT NULL` | **no server guard at all** — the guard above is scoped to open facts, so every replay inserts a fresh row | pre-query `(s,p,o,valid_from,valid_to)` | same pre-query, client-side | +| **Entities** | `entities` table | — | needed as FK targets for triples | `kg_add` creates them implicitly | + +Chunking is deterministic and replay-safe: `DEFAULT_CHUNK_SIZE = 800` (`config.py:274`), chunk ids are +`f"{drawer_id}_chunk_{i:06d}"`, so nothing about chunking needs special handling. + +### Not carried by either regime + +**Hallways, `known_entities.json`, and the palace graph are built at mine time**, not by `add_drawer`. +Replayed drawers therefore arrive with no hallway/co-occurrence edges, so `list_hallways` and `traverse` +will under-report for joined content. These are derived artifacts — rebuildable by re-mining, or +acceptably degraded. Decide which; do not discover it later. (`hallways.py`, `entities.py`, +`palace_graph.py` — none referenced from the `add_drawer` write path.) + +--- + +## 3. Phases and deliverables + +**Phase A — census (read-only, no writes anywhere).** Point a script at a palace on disk, enumerate every +parent drawer from `chroma.sqlite3` (`embeddings` ⋈ `embedding_metadata`) plus every row of +`knowledge_graph.sqlite3`, classify each into the §2 rows, and emit a manifest JSON + counts. +*Deliverable: "tor-ms22 holds N agent-authored drawers, M diary entries, K closed facts, and X mined +drawers we will re-mine instead."* This is the number that sizes everything else, and it is the piece to +run the moment tor-ms22 is reachable. **Build this first and independently** — it is useful even if the +writer is never written, and it cannot break anything. + +**Phase B — target index.** Given the primary, build the three lookup sets Phase C needs: content-drawer +ids, diary `sha256(entry)[:12]` suffixes, and `(s,p,o,valid_from,valid_to)` tuples. Over MCP this is +`list_drawers` pagination + `kg_timeline`; on synlig it is two sqlite queries. Cheap either way. + +**Phase C — writer**, with `--dry-run` as the default and an explicit `--apply`, mirroring `sync`'s +contract. Two backends behind one interface: +- `--mode direct` (regime A): stop `mempalace-serve`, `col.add()` with original ids/metadata, restart. + Model on `migrate.py`. Must refuse to run if the server is up. +- `--mode mcp` (regime B): `add_drawer`/`diary_write`/`kg_add` over HTTP, accepting `filed_at` loss. + +**Phase D — verification.** Counts before/after per class, a real `search` against known joined content +(proves the HNSW index absorbed it — this is how the first seed was verified), `kg_stats`, and a spot +`get_drawer` on a known id. Plus: re-run the Phase A census against the *target* and diff. + +--- + +## 4. Explicitly out of scope + +- **Mined wings.** RFC 001 §5 makes them `local`; their ids are path-dependent, so replay produces + duplicates rather than dedup. Re-mine on synlig from sources present there. This is also the bulk by + count — on the primary, only **438 of 14,829** drawers have no `source_file`, so the genuinely + replay-only surface is hundreds, not tens of thousands. +- **Merging two palaces into a third.** Every join targets the existing primary. +- **`mempalace sync` interaction.** See RFC 001 §7.2 — settle the guard separately; a joiner must never + call it. + +## 5. Open decisions for ALC + +1. **Chronology: keep it or flatten it?** Regime A keeps it and costs a service stop plus an rsync of the + source palace onto synlig. Regime B is simpler and loses it. This is the fork in the road. +2. **Hallways for joined content:** re-mine to rebuild, or accept degraded `traverse`? +3. **Do MBP-M1-2020 and tor-ms22 keep their palaces on persistent storage?** If either is a Docker named + volume rather than a bind mount, its un-migrated content dies on the next container recreate — so the + census (Phase A) is time-sensitive there, and flipping before censusing is risky. +4. **Does §7.6 get fixed client-side (in the joiner) or upstream (probe-and-skip in `diary_write`)?** The + joiner needs the suffix skip either way; upstream would fix it for every writer. + +## 6. Effort + +Phase A is the bulk of the value and is small — two sqlite readers and a classifier. Phase B is trivial. +Phase C is where the risk lives, and regime A must be written defensively (refuse on a live server, +back up first, `--dry-run` default). Phase D is mostly assertions. Nothing here needs new server code, +which is the point of RFC 001 §4.4's "existence checks available today". + +--- + +## Appendix — verified source references + +mempalace 3.6.0, `/opt/uv-tools/mempalace/lib/python3.13/site-packages/mempalace/`: + +| Claim | Location | +| --- | --- | +| Content-addressed drawer id, 24 hex over `wing\|room\|content` | `ids.py:80` (`make_drawer_id_from_content`), `_HASH_TRUNC_DRAWER = 24` at `ids.py:36` | +| MCP `add_drawer` uses that recipe | `mcp_server.py:2560` | +| Miner id over `source_file\|chunk_index` | `ids.py:67` (`make_drawer_id_from_chunk`), used `miner.py:1381`, `format_miner.py:645` | +| Idempotency probe + `already_exists` (probes parent **and** last chunk) | `mcp_server.py:2593-2604` | +| `filed_at` stamped server-side, no override | `mcp_server.py:2580` | +| Diary id recipe, suffix = `sha256(entry)[:12]` | `mcp_server.py:3510-3513` | +| Triple id `t_{s}_{p}_{o}_{sha256(valid_from\|recorded_at)[:12]}` | `ids.py:111-131`, `_HASH_TRUNC_TRIPLE = 12` at `ids.py:37` | +| `add_triple` guard scoped to open facts | `knowledge_graph.py:305-311` | +| Chunk size 800 | `config.py:274` | +| Direct-write precedent preserving ids+metadata | `migrate.py:326-330` | +| No join/replay/import tooling exists | `cli.py` subcommands; `exporter.py` emits markdown (lossy, not a join primitive); `diary_ingest.py` ingests *daily-summary files*, unrelated to agent diaries; `migrate.py` is single-palace chromadb recovery; `dedup.py` is cosine-similarity pruning within one `source_file` |