Joakim Persson
2f9170428c
Ran Phase A's classifier against the frozen EMB-7KJ4VR4G archive. Since the fleet shares one devbox image this is a reasonable prior for tor-ms22 and MBP-M1-2020: mined (source_file set) 15949 98.9% re-mine, never replay diary entries 116 0.7% replay + §7.6 suffix skip agent-authored drawers 60 0.4% replay, idempotent KG open facts 34 -- server guard dedupes KG closed facts 0 -- nothing to do Two consequences that shrink this project: the replay-only surface is 176 records, not thousands, so the writer is a small job and the §7.6 diary guard treated as the blocker governs 116 records; and there are ZERO closed KG facts, so the unguarded-closed-fact gap is real in the code but empty in the data. filed_at spread in the same archive -- 12 in May, 52 in June, 15174 in July, 887 in August -- is the concrete argument for the history-preserving regime. Two corrections to the first draft: 1. CLOSETS were omitted entirely. mempalace_closets is a second Chroma collection (1560 rows, ~10% of the palace) and there is NO MCP tool that writes one -- mcp_server.py exposes only _purge_source_closets, and closet_llm.py says outright that regex closets are always created by the miner. So closets cannot be replayed even in principle; they return only by re-mining. Same category as hallways/known_entities/palace-graph, and it resolves itself for the ~99% that gets re-mined anyway. 2. Census gotcha: chroma.sqlite3 holds BOTH collections, so an embeddings-wide query over-counts by ~10%. Must join segments->collections and keep mempalace_drawers. Doing so reconciles exactly: 14,778 = the 14,777 seeded to the primary + the one known post-snapshot chunk. Also drops the earlier "438 of 14,829" figure, which conflated central's no_source count (which includes its own later agent writes) with the archive's actual replay surface.
183 lines
13 KiB
Markdown
183 lines
13 KiB
Markdown
# RFC 002 — The joiner: replaying a second palace into the shared primary
|
|
|
|
**Status:** scoping. No code written yet.
|
|
**Author:** pi (agent), 2026-08-15.
|
|
**Context:** RFC 001 §4.4 designs a join as "idempotent replay of local history" but no replay tool
|
|
exists. The first two joins were whole-palace *file copies* (§4.4 deviation note), which work only for a
|
|
single source and cannot merge. tor-ms22 and MBP-M1-2020 each hold a substantial local palace whose
|
|
content should reach the primary. This document scopes the tool that does that.
|
|
|
|
> **Every mechanism below was read out of mempalace 3.6.0's own source** at
|
|
> `/opt/uv-tools/mempalace/lib/python3.13/site-packages/mempalace/`, with file:line in the appendix.
|
|
> An earlier attempt to gather these facts via a delegated subagent returned confident, fabricated code
|
|
> for a package path that does not exist on this machine. **Do not trust any claim in this document that
|
|
> the appendix does not cite.**
|
|
|
|
---
|
|
|
|
## 1. The finding that drives the design
|
|
|
|
**MCP replay cannot preserve `filed_at`.** `add_drawer` stamps
|
|
`"filed_at": datetime.now().isoformat()` server-side (`mcp_server.py:2580`) and exposes no override
|
|
parameter. `diary_write` is the same: it builds its own `now`-based id (`mcp_server.py:3510-3513`).
|
|
|
|
That is not a detail. This palace's value *is* its chronology — the primary's history runs from
|
|
2026-05-04, `list_drawers` filters on `since`/`before` against `filed_at`, and the diary is read in
|
|
order. A pure MCP replay of tor-ms22's palace would stamp **every** record with the join date,
|
|
collapsing months of history into one instant. RFC 001 §4.4 does not mention this, and it is the single
|
|
most important thing to decide before writing code.
|
|
|
|
`kg_add` is the exception: it accepts `valid_from`/`valid_to`, so **fact validity windows survive** even
|
|
though a triple's own id embeds `recorded_at`.
|
|
|
|
### Two write regimes
|
|
|
|
| | **A — direct disk write** (run on synlig) | **B — MCP replay** (run anywhere) |
|
|
| --- | --- | --- |
|
|
| Preserves `filed_at` / original ids | **Yes** — `col.add(ids=…, documents=…, metadatas=…)` | **No** — always `now()` |
|
|
| Server-side dedup guards | **Bypassed** — joiner owns all dedup | **Available** (see §2) |
|
|
| Requires quiescing `mempalace-serve` | Yes (palace is single-writer) | No |
|
|
| Requires source palace present on synlig | Yes (rsync it first) | No |
|
|
| Precedent in-tree | **`migrate.py`** already does exactly this: reads drawers+metadata straight from the palace's sqlite, then re-adds them into a fresh palace preserving ids, documents and metadata (`migrate.py:326-330`) | none |
|
|
|
|
**Recommendation:** regime **A** for the historical bulk (it is the only one that keeps the timeline, and
|
|
`migrate.py` is a working model to copy), regime **B** for small incremental top-ups where flattened
|
|
timestamps are acceptable. Do not build B first and discover the chronology loss afterwards.
|
|
|
|
---
|
|
|
|
## 2. What must move, and what dedupes it
|
|
|
|
Verified dedup keys — this supersedes nothing in RFC 001 §4.4 but makes each key concrete:
|
|
|
|
| Class | How to identify it in the source palace | Dedup key (verified) | Regime A work | Regime B work |
|
|
| --- | --- | --- | --- | --- |
|
|
| **Agent-authored drawers** (`add_drawer`/`checkpoint`) | `source_file` empty | id is `sha256("{wing}\|{room}\|{content}")[:24]` (`ids.py:80`, used at `mcp_server.py:2560`) — **fully content-deterministic** | recompute id, skip if present | none: server probes `[drawer_id, last_chunk_id]` and returns `{"success": True, "reason": "already_exists"}` (`mcp_server.py:2593-2604`) |
|
|
| **Mined drawers** | `source_file` set | id is `sha256("{source_file}\|{chunk_index}")[:24]` (`ids.py:67`) — **path-dependent**, so the same content from two machines yields two rows | — | **do not replay.** Re-mine on synlig instead (§4) |
|
|
| **Diary entries** | id prefix `diary_`, `room="diary"` | id is `diary_{wing}_{now:%Y%m%d_%H%M%S%f}_{sha256(entry)[:12]}` (`mcp_server.py:3510-3513`). Timestamp is wall-clock, so **full ids never repeat** — match on the 12-hex suffix only | copy row verbatim (id and all) | build the target's suffix set, skip matches — this is RFC 001 §7.6 |
|
|
| **KG open facts** (`valid_to IS NULL`) | `triples.valid_to IS NULL` | server guard `WHERE subject=? AND predicate=? AND object=? AND valid_to IS NULL` returns the existing id (`knowledge_graph.py:305-311`) | pre-query same key | none: just replay |
|
|
| **KG closed facts** (`valid_to` set) | `triples.valid_to IS NOT NULL` | **no server guard at all** — the guard above is scoped to open facts, so every replay inserts a fresh row | pre-query `(s,p,o,valid_from,valid_to)` | same pre-query, client-side |
|
|
| **Entities** | `entities` table | — | needed as FK targets for triples | `kg_add` creates them implicitly |
|
|
|
|
Chunking is deterministic and replay-safe: `DEFAULT_CHUNK_SIZE = 800` (`config.py:274`), chunk ids are
|
|
`f"{drawer_id}_chunk_{i:06d}"`, so nothing about chunking needs special handling.
|
|
|
|
### Not carried by either regime
|
|
|
|
**Hallways, `known_entities.json`, the palace graph, and closets are built at mine time**, not by
|
|
`add_drawer`. Replayed drawers therefore arrive with no hallway/co-occurrence edges and no closets, so
|
|
`list_hallways` and `traverse` will under-report for joined content, and joined drawers get no
|
|
`closet_boost` in search ranking. All are derived artifacts — rebuilt by re-mining, or acceptably
|
|
degraded. Decide which; do not discover it later.
|
|
|
|
**Closets specifically:** `mempalace_closets` is a second Chroma collection (1,560 rows in the archive,
|
|
~10% of the palace), and **there is no MCP tool that writes a closet** — `mcp_server.py` only exposes
|
|
`_purge_source_closets`. `closet_llm.py` states it plainly: *"Regex closets are always created by the
|
|
miner"*, with the LLM path an opt-in regeneration afterwards. So closets cannot be replayed even in
|
|
principle; they come back only by re-mining. Since ~99% of a devbox palace is mined content that gets
|
|
re-mined on synlig anyway (§4), this resolves itself for the bulk.
|
|
|
|
> **Census gotcha — filter by collection.** `chroma.sqlite3` holds *both* collections. A naive
|
|
> `embeddings`-wide query over-counts: the archive yields 16,338 rows total, which is
|
|
> `mempalace_drawers` 14,778 + `mempalace_closets` 1,560. Join through
|
|
> `segments`→`collections` and keep `mempalace_drawers`, or the census inflates by ~10%. (14,778 also
|
|
> reconciles exactly with the 14,777 seeded to the primary plus the one known post-snapshot chunk.)
|
|
|
|
---
|
|
|
|
## 3. Phases and deliverables
|
|
|
|
**Phase A — census (read-only, no writes anywhere).** Point a script at a palace on disk, enumerate every
|
|
parent drawer from `chroma.sqlite3` (`embeddings` ⋈ `embedding_metadata`) plus every row of
|
|
`knowledge_graph.sqlite3`, classify each into the §2 rows, and emit a manifest JSON + counts.
|
|
*Deliverable: "tor-ms22 holds N agent-authored drawers, M diary entries, K closed facts, and X mined
|
|
drawers we will re-mine instead."* This is the number that sizes everything else, and it is the piece to
|
|
run the moment tor-ms22 is reachable. **Build this first and independently** — it is useful even if the
|
|
writer is never written, and it cannot break anything.
|
|
|
|
**Phase B — target index.** Given the primary, build the three lookup sets Phase C needs: content-drawer
|
|
ids, diary `sha256(entry)[:12]` suffixes, and `(s,p,o,valid_from,valid_to)` tuples. Over MCP this is
|
|
`list_drawers` pagination + `kg_timeline`; on synlig it is two sqlite queries. Cheap either way.
|
|
|
|
**Phase C — writer**, with `--dry-run` as the default and an explicit `--apply`, mirroring `sync`'s
|
|
contract. Two backends behind one interface:
|
|
- `--mode direct` (regime A): stop `mempalace-serve`, `col.add()` with original ids/metadata, restart.
|
|
Model on `migrate.py`. Must refuse to run if the server is up.
|
|
- `--mode mcp` (regime B): `add_drawer`/`diary_write`/`kg_add` over HTTP, accepting `filed_at` loss.
|
|
|
|
**Phase D — verification.** Counts before/after per class, a real `search` against known joined content
|
|
(proves the HNSW index absorbed it — this is how the first seed was verified), `kg_stats`, and a spot
|
|
`get_drawer` on a known id. Plus: re-run the Phase A census against the *target* and diff.
|
|
|
|
---
|
|
|
|
## 4. Explicitly out of scope
|
|
|
|
- **Mined wings.** RFC 001 §5 makes them `local`; their ids are path-dependent, so replay produces
|
|
duplicates rather than dedup. Re-mine on synlig from sources present there.
|
|
- **Merging two palaces into a third.** Every join targets the existing primary.
|
|
- **`mempalace sync` interaction.** See RFC 001 §7.2 — settle the guard separately; a joiner must never
|
|
call it.
|
|
|
|
### 4.1 The census, run for real — the replay surface is tiny
|
|
|
|
Phase A's classifier was run against the frozen EMB-7KJ4VR4G archive (2026-08-15). Since the fleet
|
|
shares one devbox image, this is a reasonable prior for what tor-ms22 and MBP-M1-2020 hold:
|
|
|
|
| Class | Count | Share | Joiner action |
|
|
| --- | --- | --- | --- |
|
|
| Mined (`source_file` set) | 15,949 | **98.9%** | re-mine on synlig; **never replay** |
|
|
| Diary entries | 116 | 0.7% | replay + §7.6 suffix skip |
|
|
| Agent-authored drawers | 60 | 0.4% | replay, idempotent by content id |
|
|
| KG open facts | 34 | — | replay, server guard dedupes |
|
|
| KG **closed** facts | **0** | — | nothing to do — see below |
|
|
|
|
**Two consequences that shrink this project sharply.** First, the genuinely replay-only surface is
|
|
**176 records**, not thousands — so Phase C's writer is a small job, and the §7.6 diary guard that has
|
|
been treated as the blocker governs *116 records*. Second, **there are zero closed KG facts**, so the
|
|
unguarded-closed-fact gap (§2) is real in the code but currently empty in the data; it needs handling for
|
|
correctness, not for this join.
|
|
|
|
`filed_at` spread in the same archive — 12 records in May, 52 in June, 15,174 in July, 887 in August — is
|
|
the concrete case for regime A: an MCP replay would restamp all of it to the join date.
|
|
|
|
## 5. Open decisions for ALC
|
|
|
|
1. **Chronology: keep it or flatten it?** Regime A keeps it and costs a service stop plus an rsync of the
|
|
source palace onto synlig. Regime B is simpler and loses it. This is the fork in the road.
|
|
2. **Hallways for joined content:** re-mine to rebuild, or accept degraded `traverse`?
|
|
3. **Do MBP-M1-2020 and tor-ms22 keep their palaces on persistent storage?** If either is a Docker named
|
|
volume rather than a bind mount, its un-migrated content dies on the next container recreate — so the
|
|
census (Phase A) is time-sensitive there, and flipping before censusing is risky.
|
|
4. **Does §7.6 get fixed client-side (in the joiner) or upstream (probe-and-skip in `diary_write`)?** The
|
|
joiner needs the suffix skip either way; upstream would fix it for every writer.
|
|
|
|
## 6. Effort
|
|
|
|
Phase A is the bulk of the value and is small — two sqlite readers and a classifier. Phase B is trivial.
|
|
Phase C is where the risk lives, and regime A must be written defensively (refuse on a live server,
|
|
back up first, `--dry-run` default). Phase D is mostly assertions. Nothing here needs new server code,
|
|
which is the point of RFC 001 §4.4's "existence checks available today".
|
|
|
|
---
|
|
|
|
## Appendix — verified source references
|
|
|
|
mempalace 3.6.0, `/opt/uv-tools/mempalace/lib/python3.13/site-packages/mempalace/`:
|
|
|
|
| Claim | Location |
|
|
| --- | --- |
|
|
| Content-addressed drawer id, 24 hex over `wing\|room\|content` | `ids.py:80` (`make_drawer_id_from_content`), `_HASH_TRUNC_DRAWER = 24` at `ids.py:36` |
|
|
| MCP `add_drawer` uses that recipe | `mcp_server.py:2560` |
|
|
| Miner id over `source_file\|chunk_index` | `ids.py:67` (`make_drawer_id_from_chunk`), used `miner.py:1381`, `format_miner.py:645` |
|
|
| Idempotency probe + `already_exists` (probes parent **and** last chunk) | `mcp_server.py:2593-2604` |
|
|
| `filed_at` stamped server-side, no override | `mcp_server.py:2580` |
|
|
| Diary id recipe, suffix = `sha256(entry)[:12]` | `mcp_server.py:3510-3513` |
|
|
| Triple id `t_{s}_{p}_{o}_{sha256(valid_from\|recorded_at)[:12]}` | `ids.py:111-131`, `_HASH_TRUNC_TRIPLE = 12` at `ids.py:37` |
|
|
| `add_triple` guard scoped to open facts | `knowledge_graph.py:305-311` |
|
|
| Chunk size 800 | `config.py:274` |
|
|
| Direct-write precedent preserving ids+metadata | `migrate.py:326-330` |
|
|
| No join/replay/import tooling exists | `cli.py` subcommands; `exporter.py` emits markdown (lossy, not a join primitive); `diary_ingest.py` ingests *daily-summary files*, unrelated to agent diaries; `migrate.py` is single-palace chromadb recovery; `dedup.py` is cosine-similarity pruning within one `source_file` |
|
|
| Closets are miner-derived, no MCP write path | `closet_llm.py:8-10` ("Regex closets are always created by the miner"); `mcp_server.py:2913` exposes only `_purge_source_closets` |
|
|
| Two collections in one sqlite file | `mempalace_drawers` (14,778 rows) and `mempalace_closets` (1,560) in the archive, via `segments`→`collections` |
|