docs: scope the joiner (RFC 002) — and the chronology loss nobody had costed
RFC 001 §4.4 designs a join as "idempotent replay of local history", but no
replay tool exists and the two joins done so far were whole-palace file copies
that cannot merge. tor-ms22 and MBP-M1-2020 each hold a palace that needs to
reach the primary, so this scopes the tool.
The finding that drives the design: MCP replay CANNOT preserve filed_at.
add_drawer stamps it server-side (mcp_server.py:2580) with no override, and
diary_write builds its own now()-based id. A pure MCP replay would therefore
collapse months of history into the join instant -- on a palace whose value IS
its chronology, and where list_drawers filters on filed_at. RFC 001 does not
mention this. kg_add is the exception: it takes valid_from/valid_to, so fact
windows survive.
Hence two regimes, and a recommendation to build the history-preserving one
first: direct disk write on synlig (preserves ids + filed_at, bypasses the
server guards, needs the service stopped) vs MCP replay (guards work, timestamps
flatten). migrate.py is already a working model for the direct path -- it reads
drawers straight from the palace sqlite and re-adds them preserving ids,
documents and metadata.
Also concretised: every dedup key verified against mempalace 3.6.0 source rather
than assumed --
- agent-authored drawers: sha256(wing|room|content)[:24], fully deterministic
- mined drawers: sha256(source_file|chunk_index)[:24], PATH-dependent, so
replay duplicates instead of deduping -> re-mine on synlig, do not replay
- diaries: full ids never repeat (wall-clock component); match the
sha256(entry)[:12] suffix only -- this is §7.6
- KG closed facts: no server guard at all (it is scoped to valid_to IS NULL)
- hallways/known_entities/palace-graph are built at MINE time, so replayed
drawers arrive with no co-occurrence edges and traverse under-reports
Scope is phased so the useful half lands first: Phase A is a read-only census
that sizes the job and cannot break anything, and is the piece to run the moment
tor-ms22 is reachable. The replay-only surface is small -- only 438 of 14,829
drawers on the primary have no source_file.
Method note recorded in the doc: an attempt to gather these mechanisms via a
delegated subagent returned confident, fabricated code for a package path that
does not exist on this machine. Everything here is cited to file:line and was
re-read directly.
This commit is contained in:
Joakim Persson
@@ -0,0 +1,147 @@
|
||||
# RFC 002 — The joiner: replaying a second palace into the shared primary
|
||||
|
||||
**Status:** scoping. No code written yet.
|
||||
**Author:** pi (agent), 2026-08-15.
|
||||
**Context:** RFC 001 §4.4 designs a join as "idempotent replay of local history" but no replay tool
|
||||
exists. The first two joins were whole-palace *file copies* (§4.4 deviation note), which work only for a
|
||||
single source and cannot merge. tor-ms22 and MBP-M1-2020 each hold a substantial local palace whose
|
||||
content should reach the primary. This document scopes the tool that does that.
|
||||
|
||||
> **Every mechanism below was read out of mempalace 3.6.0's own source** at
|
||||
> `/opt/uv-tools/mempalace/lib/python3.13/site-packages/mempalace/`, with file:line in the appendix.
|
||||
> An earlier attempt to gather these facts via a delegated subagent returned confident, fabricated code
|
||||
> for a package path that does not exist on this machine. **Do not trust any claim in this document that
|
||||
> the appendix does not cite.**
|
||||
|
||||
---
|
||||
|
||||
## 1. The finding that drives the design
|
||||
|
||||
**MCP replay cannot preserve `filed_at`.** `add_drawer` stamps
|
||||
`"filed_at": datetime.now().isoformat()` server-side (`mcp_server.py:2580`) and exposes no override
|
||||
parameter. `diary_write` is the same: it builds its own `now`-based id (`mcp_server.py:3510-3513`).
|
||||
|
||||
That is not a detail. This palace's value *is* its chronology — the primary's history runs from
|
||||
2026-05-04, `list_drawers` filters on `since`/`before` against `filed_at`, and the diary is read in
|
||||
order. A pure MCP replay of tor-ms22's palace would stamp **every** record with the join date,
|
||||
collapsing months of history into one instant. RFC 001 §4.4 does not mention this, and it is the single
|
||||
most important thing to decide before writing code.
|
||||
|
||||
`kg_add` is the exception: it accepts `valid_from`/`valid_to`, so **fact validity windows survive** even
|
||||
though a triple's own id embeds `recorded_at`.
|
||||
|
||||
### Two write regimes
|
||||
|
||||
| | **A — direct disk write** (run on synlig) | **B — MCP replay** (run anywhere) |
|
||||
| --- | --- | --- |
|
||||
| Preserves `filed_at` / original ids | **Yes** — `col.add(ids=…, documents=…, metadatas=…)` | **No** — always `now()` |
|
||||
| Server-side dedup guards | **Bypassed** — joiner owns all dedup | **Available** (see §2) |
|
||||
| Requires quiescing `mempalace-serve` | Yes (palace is single-writer) | No |
|
||||
| Requires source palace present on synlig | Yes (rsync it first) | No |
|
||||
| Precedent in-tree | **`migrate.py`** already does exactly this: reads drawers+metadata straight from the palace's sqlite, then re-adds them into a fresh palace preserving ids, documents and metadata (`migrate.py:326-330`) | none |
|
||||
|
||||
**Recommendation:** regime **A** for the historical bulk (it is the only one that keeps the timeline, and
|
||||
`migrate.py` is a working model to copy), regime **B** for small incremental top-ups where flattened
|
||||
timestamps are acceptable. Do not build B first and discover the chronology loss afterwards.
|
||||
|
||||
---
|
||||
|
||||
## 2. What must move, and what dedupes it
|
||||
|
||||
Verified dedup keys — this supersedes nothing in RFC 001 §4.4 but makes each key concrete:
|
||||
|
||||
| Class | How to identify it in the source palace | Dedup key (verified) | Regime A work | Regime B work |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| **Agent-authored drawers** (`add_drawer`/`checkpoint`) | `source_file` empty | id is `sha256("{wing}\|{room}\|{content}")[:24]` (`ids.py:80`, used at `mcp_server.py:2560`) — **fully content-deterministic** | recompute id, skip if present | none: server probes `[drawer_id, last_chunk_id]` and returns `{"success": True, "reason": "already_exists"}` (`mcp_server.py:2593-2604`) |
|
||||
| **Mined drawers** | `source_file` set | id is `sha256("{source_file}\|{chunk_index}")[:24]` (`ids.py:67`) — **path-dependent**, so the same content from two machines yields two rows | — | **do not replay.** Re-mine on synlig instead (§4) |
|
||||
| **Diary entries** | id prefix `diary_`, `room="diary"` | id is `diary_{wing}_{now:%Y%m%d_%H%M%S%f}_{sha256(entry)[:12]}` (`mcp_server.py:3510-3513`). Timestamp is wall-clock, so **full ids never repeat** — match on the 12-hex suffix only | copy row verbatim (id and all) | build the target's suffix set, skip matches — this is RFC 001 §7.6 |
|
||||
| **KG open facts** (`valid_to IS NULL`) | `triples.valid_to IS NULL` | server guard `WHERE subject=? AND predicate=? AND object=? AND valid_to IS NULL` returns the existing id (`knowledge_graph.py:305-311`) | pre-query same key | none: just replay |
|
||||
| **KG closed facts** (`valid_to` set) | `triples.valid_to IS NOT NULL` | **no server guard at all** — the guard above is scoped to open facts, so every replay inserts a fresh row | pre-query `(s,p,o,valid_from,valid_to)` | same pre-query, client-side |
|
||||
| **Entities** | `entities` table | — | needed as FK targets for triples | `kg_add` creates them implicitly |
|
||||
|
||||
Chunking is deterministic and replay-safe: `DEFAULT_CHUNK_SIZE = 800` (`config.py:274`), chunk ids are
|
||||
`f"{drawer_id}_chunk_{i:06d}"`, so nothing about chunking needs special handling.
|
||||
|
||||
### Not carried by either regime
|
||||
|
||||
**Hallways, `known_entities.json`, and the palace graph are built at mine time**, not by `add_drawer`.
|
||||
Replayed drawers therefore arrive with no hallway/co-occurrence edges, so `list_hallways` and `traverse`
|
||||
will under-report for joined content. These are derived artifacts — rebuildable by re-mining, or
|
||||
acceptably degraded. Decide which; do not discover it later. (`hallways.py`, `entities.py`,
|
||||
`palace_graph.py` — none referenced from the `add_drawer` write path.)
|
||||
|
||||
---
|
||||
|
||||
## 3. Phases and deliverables
|
||||
|
||||
**Phase A — census (read-only, no writes anywhere).** Point a script at a palace on disk, enumerate every
|
||||
parent drawer from `chroma.sqlite3` (`embeddings` ⋈ `embedding_metadata`) plus every row of
|
||||
`knowledge_graph.sqlite3`, classify each into the §2 rows, and emit a manifest JSON + counts.
|
||||
*Deliverable: "tor-ms22 holds N agent-authored drawers, M diary entries, K closed facts, and X mined
|
||||
drawers we will re-mine instead."* This is the number that sizes everything else, and it is the piece to
|
||||
run the moment tor-ms22 is reachable. **Build this first and independently** — it is useful even if the
|
||||
writer is never written, and it cannot break anything.
|
||||
|
||||
**Phase B — target index.** Given the primary, build the three lookup sets Phase C needs: content-drawer
|
||||
ids, diary `sha256(entry)[:12]` suffixes, and `(s,p,o,valid_from,valid_to)` tuples. Over MCP this is
|
||||
`list_drawers` pagination + `kg_timeline`; on synlig it is two sqlite queries. Cheap either way.
|
||||
|
||||
**Phase C — writer**, with `--dry-run` as the default and an explicit `--apply`, mirroring `sync`'s
|
||||
contract. Two backends behind one interface:
|
||||
- `--mode direct` (regime A): stop `mempalace-serve`, `col.add()` with original ids/metadata, restart.
|
||||
Model on `migrate.py`. Must refuse to run if the server is up.
|
||||
- `--mode mcp` (regime B): `add_drawer`/`diary_write`/`kg_add` over HTTP, accepting `filed_at` loss.
|
||||
|
||||
**Phase D — verification.** Counts before/after per class, a real `search` against known joined content
|
||||
(proves the HNSW index absorbed it — this is how the first seed was verified), `kg_stats`, and a spot
|
||||
`get_drawer` on a known id. Plus: re-run the Phase A census against the *target* and diff.
|
||||
|
||||
---
|
||||
|
||||
## 4. Explicitly out of scope
|
||||
|
||||
- **Mined wings.** RFC 001 §5 makes them `local`; their ids are path-dependent, so replay produces
|
||||
duplicates rather than dedup. Re-mine on synlig from sources present there. This is also the bulk by
|
||||
count — on the primary, only **438 of 14,829** drawers have no `source_file`, so the genuinely
|
||||
replay-only surface is hundreds, not tens of thousands.
|
||||
- **Merging two palaces into a third.** Every join targets the existing primary.
|
||||
- **`mempalace sync` interaction.** See RFC 001 §7.2 — settle the guard separately; a joiner must never
|
||||
call it.
|
||||
|
||||
## 5. Open decisions for ALC
|
||||
|
||||
1. **Chronology: keep it or flatten it?** Regime A keeps it and costs a service stop plus an rsync of the
|
||||
source palace onto synlig. Regime B is simpler and loses it. This is the fork in the road.
|
||||
2. **Hallways for joined content:** re-mine to rebuild, or accept degraded `traverse`?
|
||||
3. **Do MBP-M1-2020 and tor-ms22 keep their palaces on persistent storage?** If either is a Docker named
|
||||
volume rather than a bind mount, its un-migrated content dies on the next container recreate — so the
|
||||
census (Phase A) is time-sensitive there, and flipping before censusing is risky.
|
||||
4. **Does §7.6 get fixed client-side (in the joiner) or upstream (probe-and-skip in `diary_write`)?** The
|
||||
joiner needs the suffix skip either way; upstream would fix it for every writer.
|
||||
|
||||
## 6. Effort
|
||||
|
||||
Phase A is the bulk of the value and is small — two sqlite readers and a classifier. Phase B is trivial.
|
||||
Phase C is where the risk lives, and regime A must be written defensively (refuse on a live server,
|
||||
back up first, `--dry-run` default). Phase D is mostly assertions. Nothing here needs new server code,
|
||||
which is the point of RFC 001 §4.4's "existence checks available today".
|
||||
|
||||
---
|
||||
|
||||
## Appendix — verified source references
|
||||
|
||||
mempalace 3.6.0, `/opt/uv-tools/mempalace/lib/python3.13/site-packages/mempalace/`:
|
||||
|
||||
| Claim | Location |
|
||||
| --- | --- |
|
||||
| Content-addressed drawer id, 24 hex over `wing\|room\|content` | `ids.py:80` (`make_drawer_id_from_content`), `_HASH_TRUNC_DRAWER = 24` at `ids.py:36` |
|
||||
| MCP `add_drawer` uses that recipe | `mcp_server.py:2560` |
|
||||
| Miner id over `source_file\|chunk_index` | `ids.py:67` (`make_drawer_id_from_chunk`), used `miner.py:1381`, `format_miner.py:645` |
|
||||
| Idempotency probe + `already_exists` (probes parent **and** last chunk) | `mcp_server.py:2593-2604` |
|
||||
| `filed_at` stamped server-side, no override | `mcp_server.py:2580` |
|
||||
| Diary id recipe, suffix = `sha256(entry)[:12]` | `mcp_server.py:3510-3513` |
|
||||
| Triple id `t_{s}_{p}_{o}_{sha256(valid_from\|recorded_at)[:12]}` | `ids.py:111-131`, `_HASH_TRUNC_TRIPLE = 12` at `ids.py:37` |
|
||||
| `add_triple` guard scoped to open facts | `knowledge_graph.py:305-311` |
|
||||
| Chunk size 800 | `config.py:274` |
|
||||
| Direct-write precedent preserving ids+metadata | `migrate.py:326-330` |
|
||||
| No join/replay/import tooling exists | `cli.py` subcommands; `exporter.py` emits markdown (lossy, not a join primitive); `diary_ingest.py` ingests *daily-summary files*, unrelated to agent diaries; `migrate.py` is single-palace chromadb recovery; `dedup.py` is cosine-similarity pruning within one `source_file` |
|
||||
Reference in New Issue
Block a user