mempalace-toolkit/docs/rfc-002-joiner.md

RFC 002 — The joiner: replaying a second palace into the shared primary

Status: scoping. No code written yet. Author: pi (agent), 2026-08-15. Context: RFC 001 §4.4 designs a join as "idempotent replay of local history" but no replay tool exists. The first two joins were whole-palace file copies (§4.4 deviation note), which work only for a single source and cannot merge. tor-ms22 and MBP-M1-2020 each hold a substantial local palace whose content should reach the primary. This document scopes the tool that does that.

Every mechanism below was read out of mempalace 3.6.0's own source at /opt/uv-tools/mempalace/lib/python3.13/site-packages/mempalace/, with file:line in the appendix. An earlier attempt to gather these facts via a delegated subagent returned confident, fabricated code for a package path that does not exist on this machine. Do not trust any claim in this document that the appendix does not cite.

---

1. The finding that drives the design

MCP replay cannot preserve filed_at. add_drawer stamps "filed_at": datetime.now().isoformat() server-side (mcp_server.py:2580) and exposes no override parameter. diary_write is the same: it builds its own now-based id (mcp_server.py:3510-3513).

That is not a detail. This palace's value is its chronology — the primary's history runs from 2026-05-04, list_drawers filters on since/before against filed_at, and the diary is read in order. A pure MCP replay of tor-ms22's palace would stamp every record with the join date, collapsing months of history into one instant. RFC 001 §4.4 does not mention this, and it is the single most important thing to decide before writing code.

kg_add is the exception: it accepts valid_from/valid_to, so fact validity windows survive even though a triple's own id embeds recorded_at.

Two write regimes

	A — direct disk write (run on synlig)	B — MCP replay (run anywhere)
Preserves filed_at / original ids	Yes — col.add(ids=…, documents=…, metadatas=…)	No — always now()
Server-side dedup guards	Bypassed — joiner owns all dedup	Available (see §2)
Requires quiescing mempalace-serve	Yes (palace is single-writer)	No
Requires source palace present on synlig	Yes (rsync it first)	No
Precedent in-tree	migrate.py already does exactly this: reads drawers+metadata straight from the palace's sqlite, then re-adds them into a fresh palace preserving ids, documents and metadata (migrate.py:326-330)	none

Recommendation: regime A for the historical bulk (it is the only one that keeps the timeline, and migrate.py is a working model to copy), regime B for small incremental top-ups where flattened timestamps are acceptable. Do not build B first and discover the chronology loss afterwards.

---

2. What must move, and what dedupes it

Verified dedup keys — this supersedes nothing in RFC 001 §4.4 but makes each key concrete:

Class	How to identify it in the source palace	Dedup key (verified)	Regime A work	Regime B work
Agent-authored drawers (add_drawer/checkpoint)	source_file empty	id is sha256("{wing}|{room}|{content}")[:24] (ids.py:80, used at mcp_server.py:2560) — fully content-deterministic	recompute id, skip if present	none: server probes [drawer_id, last_chunk_id] and returns {"success": True, "reason": "already_exists"} (mcp_server.py:2593-2604)
Mined drawers	source_file set	id is sha256("{source_file}|{chunk_index}")[:24] (ids.py:67) — path-dependent, so the same content from two machines yields two rows	—	do not replay. Re-mine on synlig instead (§4)
Diary entries	id prefix diary_, room="diary"	id is diary_{wing}_{now:%Y%m%d_%H%M%S%f}_{sha256(entry)[:12]} (mcp_server.py:3510-3513). Timestamp is wall-clock, so full ids never repeat — match on the 12-hex suffix only	copy row verbatim (id and all)	build the target's suffix set, skip matches — this is RFC 001 §7.6
KG open facts (valid_to IS NULL)	triples.valid_to IS NULL	server guard WHERE subject=? AND predicate=? AND object=? AND valid_to IS NULL returns the existing id (knowledge_graph.py:305-311)	pre-query same key	none: just replay
KG closed facts (valid_to set)	triples.valid_to IS NOT NULL	no server guard at all — the guard above is scoped to open facts, so every replay inserts a fresh row	pre-query (s,p,o,valid_from,valid_to)	same pre-query, client-side
Entities	entities table	—	needed as FK targets for triples	kg_add creates them implicitly

Chunking is deterministic and replay-safe: DEFAULT_CHUNK_SIZE = 800 (config.py:274), chunk ids are f"{drawer_id}_chunk_{i:06d}", so nothing about chunking needs special handling.

Not carried by either regime

Hallways, known_entities.json, the palace graph, and closets are built at mine time, not by add_drawer. Replayed drawers therefore arrive with no hallway/co-occurrence edges and no closets, so list_hallways and traverse will under-report for joined content, and joined drawers get no closet_boost in search ranking. All are derived artifacts — rebuilt by re-mining, or acceptably degraded. Decide which; do not discover it later.

Closets specifically: mempalace_closets is a second Chroma collection (1,560 rows in the archive, ~10% of the palace), and there is no MCP tool that writes a closet — mcp_server.py only exposes _purge_source_closets. closet_llm.py states it plainly: "Regex closets are always created by the miner", with the LLM path an opt-in regeneration afterwards. So closets cannot be replayed even in principle; they come back only by re-mining. Since ~99% of a devbox palace is mined content that gets re-mined on synlig anyway (§4), this resolves itself for the bulk.

Census gotcha — filter by collection. chroma.sqlite3 holds both collections. A naive embeddings-wide query over-counts: the archive yields 16,338 rows total, which is mempalace_drawers 14,778 + mempalace_closets 1,560. Join through segments→collections and keep mempalace_drawers, or the census inflates by ~10%. (14,778 also reconciles exactly with the 14,777 seeded to the primary plus the one known post-snapshot chunk.)

---

3. Phases and deliverables

Phase A — census (read-only, no writes anywhere). Point a script at a palace on disk, enumerate every parent drawer from chroma.sqlite3 (embeddings ⋈ embedding_metadata) plus every row of knowledge_graph.sqlite3, classify each into the §2 rows, and emit a manifest JSON + counts. Deliverable: "tor-ms22 holds N agent-authored drawers, M diary entries, K closed facts, and X mined drawers we will re-mine instead." This is the number that sizes everything else, and it is the piece to run the moment tor-ms22 is reachable. Build this first and independently — it is useful even if the writer is never written, and it cannot break anything.

Phase B — target index. Given the primary, build the three lookup sets Phase C needs: content-drawer ids, diary sha256(entry)[:12] suffixes, and (s,p,o,valid_from,valid_to) tuples. Over MCP this is list_drawers pagination + kg_timeline; on synlig it is two sqlite queries. Cheap either way.

Phase C — writer, with --dry-run as the default and an explicit --apply, mirroring sync's contract. Two backends behind one interface:

• --mode direct (regime A): stop mempalace-serve, col.add() with original ids/metadata, restart. Model on migrate.py. Must refuse to run if the server is up.
• --mode mcp (regime B): add_drawer/diary_write/kg_add over HTTP, accepting filed_at loss.

Phase D — verification. Counts before/after per class, a real search against known joined content (proves the HNSW index absorbed it — this is how the first seed was verified), kg_stats, and a spot get_drawer on a known id. Plus: re-run the Phase A census against the target and diff.

---

4. Explicitly out of scope

• Mined wings. RFC 001 §5 makes them local; their ids are path-dependent, so replay produces duplicates rather than dedup. Re-mine on synlig from sources present there.
• Merging two palaces into a third. Every join targets the existing primary.
• mempalace sync interaction. See RFC 001 §7.2 — settle the guard separately; a joiner must never call it.

4.1 The census, run for real — the replay surface is tiny

Phase A's classifier was run against the frozen EMB-7KJ4VR4G archive (2026-08-15). Since the fleet shares one devbox image, this is a reasonable prior for what tor-ms22 and MBP-M1-2020 hold:

Class	Count	Share	Joiner action
Mined (source_file set)	15,949	98.9%	re-mine on synlig; never replay
Diary entries	116	0.7%	replay + §7.6 suffix skip
Agent-authored drawers	60	0.4%	replay, idempotent by content id
KG open facts	34	—	replay, server guard dedupes
KG closed facts	0	—	nothing to do — see below

Two consequences that shrink this project sharply. First, the genuinely replay-only surface is 176 records, not thousands — so Phase C's writer is a small job, and the §7.6 diary guard that has been treated as the blocker governs 116 records. Second, there are zero closed KG facts, so the unguarded-closed-fact gap (§2) is real in the code but currently empty in the data; it needs handling for correctness, not for this join.

filed_at spread in the same archive — 12 records in May, 52 in June, 15,174 in July, 887 in August — is the concrete case for regime A: an MCP replay would restamp all of it to the join date.

5. Open decisions for ALC

1. Chronology: keep it or flatten it? Regime A keeps it and costs a service stop plus an rsync of the source palace onto synlig. Regime B is simpler and loses it. This is the fork in the road.
2. Hallways for joined content: re-mine to rebuild, or accept degraded traverse?
3. Do MBP-M1-2020 and tor-ms22 keep their palaces on persistent storage? If either is a Docker named volume rather than a bind mount, its un-migrated content dies on the next container recreate — so the census (Phase A) is time-sensitive there, and flipping before censusing is risky.
4. Does §7.6 get fixed client-side (in the joiner) or upstream (probe-and-skip in diary_write)? The joiner needs the suffix skip either way; upstream would fix it for every writer.

6. Effort

Phase A is the bulk of the value and is small — two sqlite readers and a classifier. Phase B is trivial. Phase C is where the risk lives, and regime A must be written defensively (refuse on a live server, back up first, --dry-run default). Phase D is mostly assertions. Nothing here needs new server code, which is the point of RFC 001 §4.4's "existence checks available today".

---

Appendix — verified source references

mempalace 3.6.0, /opt/uv-tools/mempalace/lib/python3.13/site-packages/mempalace/:

Claim	Location
Content-addressed drawer id, 24 hex over wing|room|content	ids.py:80 (make_drawer_id_from_content), _HASH_TRUNC_DRAWER = 24 at ids.py:36
MCP add_drawer uses that recipe	mcp_server.py:2560
Miner id over source_file|chunk_index	ids.py:67 (make_drawer_id_from_chunk), used miner.py:1381, format_miner.py:645
Idempotency probe + already_exists (probes parent and last chunk)	mcp_server.py:2593-2604
filed_at stamped server-side, no override	mcp_server.py:2580
Diary id recipe, suffix = sha256(entry)[:12]	mcp_server.py:3510-3513
Triple id t_{s}_{p}_{o}_{sha256(valid_from|recorded_at)[:12]}	ids.py:111-131, _HASH_TRUNC_TRIPLE = 12 at ids.py:37
add_triple guard scoped to open facts	knowledge_graph.py:305-311
Chunk size 800	config.py:274
Direct-write precedent preserving ids+metadata	migrate.py:326-330
No join/replay/import tooling exists	cli.py subcommands; exporter.py emits markdown (lossy, not a join primitive); diary_ingest.py ingests daily-summary files, unrelated to agent diaries; migrate.py is single-palace chromadb recovery; dedup.py is cosine-similarity pruning within one source_file
Closets are miner-derived, no MCP write path	closet_llm.py:8-10 ("Regex closets are always created by the miner"); mcp_server.py:2913 exposes only _purge_source_closets
Two collections in one sqlite file	mempalace_drawers (14,778 rows) and mempalace_closets (1,560) in the archive, via segments→collections