Joakim Persson
a25e22922b
RFC 001 §4.4 designs a join as "idempotent replay of local history", but no
replay tool exists and the two joins done so far were whole-palace file copies
that cannot merge. tor-ms22 and MBP-M1-2020 each hold a palace that needs to
reach the primary, so this scopes the tool.
The finding that drives the design: MCP replay CANNOT preserve filed_at.
add_drawer stamps it server-side (mcp_server.py:2580) with no override, and
diary_write builds its own now()-based id. A pure MCP replay would therefore
collapse months of history into the join instant -- on a palace whose value IS
its chronology, and where list_drawers filters on filed_at. RFC 001 does not
mention this. kg_add is the exception: it takes valid_from/valid_to, so fact
windows survive.
Hence two regimes, and a recommendation to build the history-preserving one
first: direct disk write on synlig (preserves ids + filed_at, bypasses the
server guards, needs the service stopped) vs MCP replay (guards work, timestamps
flatten). migrate.py is already a working model for the direct path -- it reads
drawers straight from the palace sqlite and re-adds them preserving ids,
documents and metadata.
Also concretised: every dedup key verified against mempalace 3.6.0 source rather
than assumed --
- agent-authored drawers: sha256(wing|room|content)[:24], fully deterministic
- mined drawers: sha256(source_file|chunk_index)[:24], PATH-dependent, so
replay duplicates instead of deduping -> re-mine on synlig, do not replay
- diaries: full ids never repeat (wall-clock component); match the
sha256(entry)[:12] suffix only -- this is §7.6
- KG closed facts: no server guard at all (it is scoped to valid_to IS NULL)
- hallways/known_entities/palace-graph are built at MINE time, so replayed
drawers arrive with no co-occurrence edges and traverse under-reports
Scope is phased so the useful half lands first: Phase A is a read-only census
that sizes the job and cannot break anything, and is the piece to run the moment
tor-ms22 is reachable. The replay-only surface is small -- only 438 of 14,829
drawers on the primary have no source_file.
Method note recorded in the doc: an attempt to gather these mechanisms via a
delegated subagent returned confident, fabricated code for a package path that
does not exist on this machine. Everything here is cited to file:line and was
re-read directly.
10 KiB
RFC 002 — The joiner: replaying a second palace into the shared primary
Status: scoping. No code written yet. Author: pi (agent), 2026-08-15. Context: RFC 001 §4.4 designs a join as "idempotent replay of local history" but no replay tool exists. The first two joins were whole-palace file copies (§4.4 deviation note), which work only for a single source and cannot merge. tor-ms22 and MBP-M1-2020 each hold a substantial local palace whose content should reach the primary. This document scopes the tool that does that.
Every mechanism below was read out of mempalace 3.6.0's own source at
/opt/uv-tools/mempalace/lib/python3.13/site-packages/mempalace/, with file:line in the appendix. An earlier attempt to gather these facts via a delegated subagent returned confident, fabricated code for a package path that does not exist on this machine. Do not trust any claim in this document that the appendix does not cite.
1. The finding that drives the design
MCP replay cannot preserve filed_at. add_drawer stamps
"filed_at": datetime.now().isoformat() server-side (mcp_server.py:2580) and exposes no override
parameter. diary_write is the same: it builds its own now-based id (mcp_server.py:3510-3513).
That is not a detail. This palace's value is its chronology — the primary's history runs from
2026-05-04, list_drawers filters on since/before against filed_at, and the diary is read in
order. A pure MCP replay of tor-ms22's palace would stamp every record with the join date,
collapsing months of history into one instant. RFC 001 §4.4 does not mention this, and it is the single
most important thing to decide before writing code.
kg_add is the exception: it accepts valid_from/valid_to, so fact validity windows survive even
though a triple's own id embeds recorded_at.
Two write regimes
| A — direct disk write (run on synlig) | B — MCP replay (run anywhere) | |
|---|---|---|
Preserves filed_at / original ids |
Yes — col.add(ids=…, documents=…, metadatas=…) |
No — always now() |
| Server-side dedup guards | Bypassed — joiner owns all dedup | Available (see §2) |
Requires quiescing mempalace-serve |
Yes (palace is single-writer) | No |
| Requires source palace present on synlig | Yes (rsync it first) | No |
| Precedent in-tree | migrate.py already does exactly this: reads drawers+metadata straight from the palace's sqlite, then re-adds them into a fresh palace preserving ids, documents and metadata (migrate.py:326-330) |
none |
Recommendation: regime A for the historical bulk (it is the only one that keeps the timeline, and
migrate.py is a working model to copy), regime B for small incremental top-ups where flattened
timestamps are acceptable. Do not build B first and discover the chronology loss afterwards.
2. What must move, and what dedupes it
Verified dedup keys — this supersedes nothing in RFC 001 §4.4 but makes each key concrete:
| Class | How to identify it in the source palace | Dedup key (verified) | Regime A work | Regime B work |
|---|---|---|---|---|
Agent-authored drawers (add_drawer/checkpoint) |
source_file empty |
id is sha256("{wing}|{room}|{content}")[:24] (ids.py:80, used at mcp_server.py:2560) — fully content-deterministic |
recompute id, skip if present | none: server probes [drawer_id, last_chunk_id] and returns {"success": True, "reason": "already_exists"} (mcp_server.py:2593-2604) |
| Mined drawers | source_file set |
id is sha256("{source_file}|{chunk_index}")[:24] (ids.py:67) — path-dependent, so the same content from two machines yields two rows |
— | do not replay. Re-mine on synlig instead (§4) |
| Diary entries | id prefix diary_, room="diary" |
id is diary_{wing}_{now:%Y%m%d_%H%M%S%f}_{sha256(entry)[:12]} (mcp_server.py:3510-3513). Timestamp is wall-clock, so full ids never repeat — match on the 12-hex suffix only |
copy row verbatim (id and all) | build the target's suffix set, skip matches — this is RFC 001 §7.6 |
KG open facts (valid_to IS NULL) |
triples.valid_to IS NULL |
server guard WHERE subject=? AND predicate=? AND object=? AND valid_to IS NULL returns the existing id (knowledge_graph.py:305-311) |
pre-query same key | none: just replay |
KG closed facts (valid_to set) |
triples.valid_to IS NOT NULL |
no server guard at all — the guard above is scoped to open facts, so every replay inserts a fresh row | pre-query (s,p,o,valid_from,valid_to) |
same pre-query, client-side |
| Entities | entities table |
— | needed as FK targets for triples | kg_add creates them implicitly |
Chunking is deterministic and replay-safe: DEFAULT_CHUNK_SIZE = 800 (config.py:274), chunk ids are
f"{drawer_id}_chunk_{i:06d}", so nothing about chunking needs special handling.
Not carried by either regime
Hallways, known_entities.json, and the palace graph are built at mine time, not by add_drawer.
Replayed drawers therefore arrive with no hallway/co-occurrence edges, so list_hallways and traverse
will under-report for joined content. These are derived artifacts — rebuildable by re-mining, or
acceptably degraded. Decide which; do not discover it later. (hallways.py, entities.py,
palace_graph.py — none referenced from the add_drawer write path.)
3. Phases and deliverables
Phase A — census (read-only, no writes anywhere). Point a script at a palace on disk, enumerate every
parent drawer from chroma.sqlite3 (embeddings ⋈ embedding_metadata) plus every row of
knowledge_graph.sqlite3, classify each into the §2 rows, and emit a manifest JSON + counts.
Deliverable: "tor-ms22 holds N agent-authored drawers, M diary entries, K closed facts, and X mined
drawers we will re-mine instead." This is the number that sizes everything else, and it is the piece to
run the moment tor-ms22 is reachable. Build this first and independently — it is useful even if the
writer is never written, and it cannot break anything.
Phase B — target index. Given the primary, build the three lookup sets Phase C needs: content-drawer
ids, diary sha256(entry)[:12] suffixes, and (s,p,o,valid_from,valid_to) tuples. Over MCP this is
list_drawers pagination + kg_timeline; on synlig it is two sqlite queries. Cheap either way.
Phase C — writer, with --dry-run as the default and an explicit --apply, mirroring sync's
contract. Two backends behind one interface:
--mode direct(regime A): stopmempalace-serve,col.add()with original ids/metadata, restart. Model onmigrate.py. Must refuse to run if the server is up.--mode mcp(regime B):add_drawer/diary_write/kg_addover HTTP, acceptingfiled_atloss.
Phase D — verification. Counts before/after per class, a real search against known joined content
(proves the HNSW index absorbed it — this is how the first seed was verified), kg_stats, and a spot
get_drawer on a known id. Plus: re-run the Phase A census against the target and diff.
4. Explicitly out of scope
- Mined wings. RFC 001 §5 makes them
local; their ids are path-dependent, so replay produces duplicates rather than dedup. Re-mine on synlig from sources present there. This is also the bulk by count — on the primary, only 438 of 14,829 drawers have nosource_file, so the genuinely replay-only surface is hundreds, not tens of thousands. - Merging two palaces into a third. Every join targets the existing primary.
mempalace syncinteraction. See RFC 001 §7.2 — settle the guard separately; a joiner must never call it.
5. Open decisions for ALC
- Chronology: keep it or flatten it? Regime A keeps it and costs a service stop plus an rsync of the source palace onto synlig. Regime B is simpler and loses it. This is the fork in the road.
- Hallways for joined content: re-mine to rebuild, or accept degraded
traverse? - Do MBP-M1-2020 and tor-ms22 keep their palaces on persistent storage? If either is a Docker named volume rather than a bind mount, its un-migrated content dies on the next container recreate — so the census (Phase A) is time-sensitive there, and flipping before censusing is risky.
- Does §7.6 get fixed client-side (in the joiner) or upstream (probe-and-skip in
diary_write)? The joiner needs the suffix skip either way; upstream would fix it for every writer.
6. Effort
Phase A is the bulk of the value and is small — two sqlite readers and a classifier. Phase B is trivial.
Phase C is where the risk lives, and regime A must be written defensively (refuse on a live server,
back up first, --dry-run default). Phase D is mostly assertions. Nothing here needs new server code,
which is the point of RFC 001 §4.4's "existence checks available today".
Appendix — verified source references
mempalace 3.6.0, /opt/uv-tools/mempalace/lib/python3.13/site-packages/mempalace/:
| Claim | Location |
|---|---|
Content-addressed drawer id, 24 hex over wing|room|content |
ids.py:80 (make_drawer_id_from_content), _HASH_TRUNC_DRAWER = 24 at ids.py:36 |
MCP add_drawer uses that recipe |
mcp_server.py:2560 |
Miner id over source_file|chunk_index |
ids.py:67 (make_drawer_id_from_chunk), used miner.py:1381, format_miner.py:645 |
Idempotency probe + already_exists (probes parent and last chunk) |
mcp_server.py:2593-2604 |
filed_at stamped server-side, no override |
mcp_server.py:2580 |
Diary id recipe, suffix = sha256(entry)[:12] |
mcp_server.py:3510-3513 |
Triple id t_{s}_{p}_{o}_{sha256(valid_from|recorded_at)[:12]} |
ids.py:111-131, _HASH_TRUNC_TRIPLE = 12 at ids.py:37 |
add_triple guard scoped to open facts |
knowledge_graph.py:305-311 |
| Chunk size 800 | config.py:274 |
| Direct-write precedent preserving ids+metadata | migrate.py:326-330 |
| No join/replay/import tooling exists | cli.py subcommands; exporter.py emits markdown (lossy, not a join primitive); diary_ingest.py ingests daily-summary files, unrelated to agent diaries; migrate.py is single-palace chromadb recovery; dedup.py is cosine-similarity pruning within one source_file |