Joakim Persson
f60cf9c732
bin/mempalace-census classifies a palace on disk into the RFC 002 §2 classes
(mined / diary / agent-authored) and emits a human report or a --json manifest
that feeds Phases B/C. Read-only: every sqlite handle is opened mode=ro, no
-wal/-shm is created, safe against a live mempalace-serve. Reads LOCAL DISK only
and warns if MEMPALACE_REMOTE_URL is set, so a local census can't be mistaken
for a remote one.
The design point is that it SELF-VERIFIES instead of trusting ids.py's
docstrings: for every replayable drawer it reassembles content from chunks,
recomputes the upstream id and compares to the stored id. That single check
covers the id recipe, the chunk reassembly order and the classifier at once --
176/176 accounted for on the reference palace -- and it falsified three things
the RFC previously asserted from a docs-only reading (now RFC 002 §2.1):
(a) The hash input is LENGTH-PREFIXED, not "|"-joined. ids.py:31 defines
_DELIM = "|" and the make_* docstrings describe f"{wing}|{room}|{content}",
but _DELIM is dead code and _delimited_sha256 builds
"".join(f"{len(part)}:{part}"). Measured: length-prefixed reproduces real
ids 5/5, pipe-joined 0/5. Diary ids differ again -- a PLAIN sha256.
(b) id_recipe is NOT a mined-only marker. It looked like a clean
discriminator (same 14,586 count as source_file) but the server stamps
'v3' on content ids too, so classifying on `source_file OR id_recipe`
swallowed all 60 agent-authored drawers into MINED -- the dangerous
direction, since Phase C would try to re-mine drawers that have no source
file and silently drop them. Discriminator is a TRUTHY source_file (the
writer stores "" rather than omitting the key), cross-checked against the
miner-only keys source_mtime / normalize_version; disagreement is now a
first-class warning.
(c) Content ids DRIFT: 9 of 60 agent-authored drawers no longer reproduce
their own id, because update_drawer preserves the id while rewriting and
re-chunking. So "recompute the content id and skip if present" -- the
strategy this RFC specified for regime A -- misses every drifted drawer
and duplicates it. Phase C must key on the STORED id. Flagged as
edited_since_filing in the manifest so Phase C can be tested on them.
Also corrects §4.1's headline number: 15,949 mined / 98.9% was reconstructible
exactly as 16,338 (all embeddings rows) - 192 (diary rows) - 197 (agent rows),
i.e. it counted rows rather than parent drawers AND spanned both collections,
absorbing all 1,560 non-joinable mempalace_closets rows into the mined total.
Correct figures: 14,389 mined / 116 diary / 60 agent-authored = 14,565 parents,
replay surface 176 (1.2%). Two rules now enforced in the tool: always filter by
collection (one sqlite file holds both), and always say whether a count is rows
or parent drawers -- a chunked drawer contributes N rows and no parent row.
One implementation trap worth recording: Chroma splits metadata across
string_value and int_value, so reading only string_value nulls every numeric key
(source_mtime, chunk_index, line_start) -- which made the miner-marker
cross-check report 100% conflict until the loader coalesced the two columns.
225 lines
17 KiB
Markdown
225 lines
17 KiB
Markdown
# RFC 002 — The joiner: replaying a second palace into the shared primary
|
||
|
||
**Status:** scoping. No code written yet.
|
||
**Author:** pi (agent), 2026-08-15.
|
||
**Context:** RFC 001 §4.4 designs a join as "idempotent replay of local history" but no replay tool
|
||
exists. The first two joins were whole-palace *file copies* (§4.4 deviation note), which work only for a
|
||
single source and cannot merge. tor-ms22 and MBP-M1-2020 each hold a substantial local palace whose
|
||
content should reach the primary. This document scopes the tool that does that.
|
||
|
||
> **Every mechanism below was read out of mempalace 3.6.0's own source** at
|
||
> `/opt/uv-tools/mempalace/lib/python3.13/site-packages/mempalace/`, with file:line in the appendix.
|
||
> An earlier attempt to gather these facts via a delegated subagent returned confident, fabricated code
|
||
> for a package path that does not exist on this machine. **Do not trust any claim in this document that
|
||
> the appendix does not cite.**
|
||
|
||
---
|
||
|
||
## 1. The finding that drives the design
|
||
|
||
**MCP replay cannot preserve `filed_at`.** `add_drawer` stamps
|
||
`"filed_at": datetime.now().isoformat()` server-side (`mcp_server.py:2580`) and exposes no override
|
||
parameter. `diary_write` is the same: it builds its own `now`-based id (`mcp_server.py:3510-3513`).
|
||
|
||
That is not a detail. This palace's value *is* its chronology — the primary's history runs from
|
||
2026-05-04, `list_drawers` filters on `since`/`before` against `filed_at`, and the diary is read in
|
||
order. A pure MCP replay of tor-ms22's palace would stamp **every** record with the join date,
|
||
collapsing months of history into one instant. RFC 001 §4.4 does not mention this, and it is the single
|
||
most important thing to decide before writing code.
|
||
|
||
`kg_add` is the exception: it accepts `valid_from`/`valid_to`, so **fact validity windows survive** even
|
||
though a triple's own id embeds `recorded_at`.
|
||
|
||
### Two write regimes
|
||
|
||
| | **A — direct disk write** (run on synlig) | **B — MCP replay** (run anywhere) |
|
||
| --- | --- | --- |
|
||
| Preserves `filed_at` / original ids | **Yes** — `col.add(ids=…, documents=…, metadatas=…)` | **No** — always `now()` |
|
||
| Server-side dedup guards | **Bypassed** — joiner owns all dedup | **Available** (see §2) |
|
||
| Requires quiescing `mempalace-serve` | Yes (palace is single-writer) | No |
|
||
| Requires source palace present on synlig | Yes (rsync it first) | No |
|
||
| Precedent in-tree | **`migrate.py`** already does exactly this: reads drawers+metadata straight from the palace's sqlite, then re-adds them into a fresh palace preserving ids, documents and metadata (`migrate.py:326-330`) | none |
|
||
|
||
**Recommendation:** regime **A** for the historical bulk (it is the only one that keeps the timeline, and
|
||
`migrate.py` is a working model to copy), regime **B** for small incremental top-ups where flattened
|
||
timestamps are acceptable. Do not build B first and discover the chronology loss afterwards.
|
||
|
||
---
|
||
|
||
## 2. What must move, and what dedupes it
|
||
|
||
Verified dedup keys — this supersedes nothing in RFC 001 §4.4 but makes each key concrete:
|
||
|
||
| Class | How to identify it in the source palace | Dedup key (verified) | Regime A work | Regime B work |
|
||
| --- | --- | --- | --- | --- |
|
||
| **Agent-authored drawers** (`add_drawer`/`checkpoint`) | `source_file` empty — note the writer stores `""` rather than omitting the key, so test truthiness, not presence. `id_recipe` is **not** a discriminator: the server stamps `v3` on content ids too (§2.1) | `drawer_{wing}_{room}_{H(wing,room,content)[:24]}`, where `H` **length-prefixes** each part — *not* `\|`-joined (`ids.py:40-53`; `make_drawer_id_from_content` at `ids.py:80`, used at `mcp_server.py:2560`). Content-deterministic **unless the drawer was later edited** (§2.1) | replay by **stored** id — recompute-and-skip is unsafe (§2.1) | none: server probes `[drawer_id, last_chunk_id]` and returns `{"success": True, "reason": "already_exists"}` (`mcp_server.py:2593-2604`) |
|
||
| **Mined drawers** | `source_file` non-empty; corroborated by the miner-only keys `source_mtime` / `normalize_version` | `sha256(H(source_file, chunk_index))[:24]` (`ids.py:67`), same length-prefixed helper — **path-dependent**, so the same content from two machines yields two rows | — | **do not replay.** Re-mine on synlig instead (§4) |
|
||
| **Diary entries** | metadata `type="diary_entry"`; id prefix `diary_` | id is `diary_{wing}_{now:%Y%m%d_%H%M%S%f}_{sha256(entry)[:12]}` (`mcp_server.py:3510-3513`) — a **plain** sha256 of the entry, *not* the length-prefixed helper. Timestamp is wall-clock, so **full ids never repeat** — match on the 12-hex suffix only | copy row verbatim (id and all) | build the target's suffix set, skip matches — this is RFC 001 §7.6 |
|
||
| **KG open facts** (`valid_to IS NULL`) | `triples.valid_to IS NULL` | server guard `WHERE subject=? AND predicate=? AND object=? AND valid_to IS NULL` returns the existing id (`knowledge_graph.py:305-311`) | pre-query same key | none: just replay |
|
||
| **KG closed facts** (`valid_to` set) | `triples.valid_to IS NOT NULL` | **no server guard at all** — the guard above is scoped to open facts, so every replay inserts a fresh row | pre-query `(s,p,o,valid_from,valid_to)` | same pre-query, client-side |
|
||
| **Entities** | `entities` table | — | needed as FK targets for triples | `kg_add` creates them implicitly |
|
||
|
||
Chunking is deterministic and replay-safe: `DEFAULT_CHUNK_SIZE = 800` (`config.py:274`), chunk ids are
|
||
`f"{drawer_id}_chunk_{i:06d}"`, so nothing about chunking needs special handling. One structural note the
|
||
census depends on: an oversized drawer has **no parent row** — only `_chunk_NNNNNN` rows — so the parent
|
||
id must be recovered by stripping the suffix, and content by concatenating in `chunk_index` order.
|
||
|
||
### 2.1 Three corrections this table needed — found by building the census, not by reading
|
||
|
||
Every recipe above is now **empirically verified**: `bin/mempalace-census` recomputes each id from
|
||
reassembled content and compares it to the stored id (176/176 accounted for on the reference palace).
|
||
Building that check falsified three things an earlier docs-only reading of `ids.py` had asserted.
|
||
|
||
**(a) The hash input is length-prefixed, not `\|`-joined.** `ids.py:31` defines `_DELIM = "|"` and the
|
||
`make_*` docstrings describe the input as `f"{wing}|{room}|{content}"`. Both are misleading: `_DELIM` is
|
||
**dead code** (defined, never referenced) and `_delimited_sha256()` actually builds
|
||
`"".join(f"{len(part)}:{part}")`. Length-prefixing is the better scheme — it is unambiguous where a bare
|
||
delimiter is not — but the docstrings never caught up. Measured on real drawers: length-prefixed
|
||
reproduces stored ids **5/5**, pipe-joined **0/5**. *Verify id recipes against the implementation and a
|
||
real row; a docstring is a claim, not evidence.*
|
||
|
||
**(b) `id_recipe` is not a mined-only marker.** It looked like a clean discriminator (14,586 rows, the
|
||
same count as `source_file`), but the server stamps `v3` on **every** v3 id including content-hashed
|
||
ones, and the sets differ by exactly the 60 agent-authored drawers. Classifying on
|
||
`source_file OR id_recipe` therefore swallowed all 60 into *mined* — the dangerous direction, since
|
||
Phase C would try to re-mine drawers that have no source file at all and silently drop them from the
|
||
join. The discriminator is a **truthy** `source_file`, cross-checked against the miner-only keys
|
||
`source_mtime` / `normalize_version`. The census now reports any disagreement between the two as a
|
||
first-class warning rather than trusting one signal.
|
||
|
||
**(c) Content ids drift, so "recompute the id and skip if present" is unsafe.** 9 of the 60
|
||
agent-authored drawers (15%) no longer reproduce their own id: `update_drawer` **preserves the id** while
|
||
rewriting and re-chunking content, so an edited drawer's content hash stops matching. This is expected
|
||
behaviour, but it breaks the obvious Regime A strategy — recomputing the content id and probing the
|
||
target for it **misses every drifted drawer and duplicates it**. Phase C must key on the **stored** id.
|
||
The census flags these as `edited_since_filing` in the manifest so Phase C can be tested against them.
|
||
|
||
### Not carried by either regime
|
||
|
||
**Hallways, `known_entities.json`, the palace graph, and closets are built at mine time**, not by
|
||
`add_drawer`. Replayed drawers therefore arrive with no hallway/co-occurrence edges and no closets, so
|
||
`list_hallways` and `traverse` will under-report for joined content, and joined drawers get no
|
||
`closet_boost` in search ranking. All are derived artifacts — rebuilt by re-mining, or acceptably
|
||
degraded. Decide which; do not discover it later.
|
||
|
||
**Closets specifically:** `mempalace_closets` is a second Chroma collection (1,560 rows in the archive,
|
||
~10% of the palace), and **there is no MCP tool that writes a closet** — `mcp_server.py` only exposes
|
||
`_purge_source_closets`. `closet_llm.py` states it plainly: *"Regex closets are always created by the
|
||
miner"*, with the LLM path an opt-in regeneration afterwards. So closets cannot be replayed even in
|
||
principle; they come back only by re-mining. Since ~99% of a devbox palace is mined content that gets
|
||
re-mined on synlig anyway (§4), this resolves itself for the bulk.
|
||
|
||
> **Census gotcha — filter by collection.** `chroma.sqlite3` holds *both* collections. A naive
|
||
> `embeddings`-wide query over-counts: the archive yields 16,338 rows total, which is
|
||
> `mempalace_drawers` 14,778 + `mempalace_closets` 1,560. Join through
|
||
> `segments`→`collections` and keep `mempalace_drawers`, or the census inflates by ~10%. (14,778 also
|
||
> reconciles exactly with the 14,777 seeded to the primary plus the one known post-snapshot chunk.)
|
||
|
||
---
|
||
|
||
## 3. Phases and deliverables
|
||
|
||
**Phase A — census (read-only, no writes anywhere).** ✅ **Built: `bin/mempalace-census`.** Point it at a
|
||
palace on disk; it enumerates every parent drawer from `chroma.sqlite3` (`embeddings ⋈ embedding_metadata`,
|
||
filtered by collection) plus every row of `knowledge_graph.sqlite3`, classifies each into the §2 rows, and
|
||
emits `--json` (a manifest that feeds Phases B/C) or a human report. It opens every file `mode=ro` and is
|
||
safe to run against a live palace. It also **self-verifies** — recomputing each id from reassembled content
|
||
and comparing to the stored id, which checks the recipe, the chunk reassembly order and the classifier in
|
||
one pass. That check is what produced the §2.1 corrections. *Deliverable, realised: see §4.1.*
|
||
|
||
**Phase B — target index.** Given the primary, build the three lookup sets Phase C needs: content-drawer
|
||
ids, diary `sha256(entry)[:12]` suffixes, and `(s,p,o,valid_from,valid_to)` tuples. Over MCP this is
|
||
`list_drawers` pagination + `kg_timeline`; on synlig it is two sqlite queries. Cheap either way.
|
||
|
||
**Phase C — writer**, with `--dry-run` as the default and an explicit `--apply`, mirroring `sync`'s
|
||
contract. Two backends behind one interface:
|
||
- `--mode direct` (regime A): stop `mempalace-serve`, `col.add()` with original ids/metadata, restart.
|
||
Model on `migrate.py`. Must refuse to run if the server is up.
|
||
- `--mode mcp` (regime B): `add_drawer`/`diary_write`/`kg_add` over HTTP, accepting `filed_at` loss.
|
||
|
||
**Phase D — verification.** Counts before/after per class, a real `search` against known joined content
|
||
(proves the HNSW index absorbed it — this is how the first seed was verified), `kg_stats`, and a spot
|
||
`get_drawer` on a known id. Plus: re-run the Phase A census against the *target* and diff.
|
||
|
||
---
|
||
|
||
## 4. Explicitly out of scope
|
||
|
||
- **Mined wings.** RFC 001 §5 makes them `local`; their ids are path-dependent, so replay produces
|
||
duplicates rather than dedup. Re-mine on synlig from sources present there.
|
||
- **Merging two palaces into a third.** Every join targets the existing primary.
|
||
- **`mempalace sync` interaction.** See RFC 001 §7.2 — settle the guard separately; a joiner must never
|
||
call it.
|
||
|
||
### 4.1 The census, run for real — the replay surface is tiny
|
||
|
||
Produced by `bin/mempalace-census` against the container-local palace (2026-08-15). Since the fleet shares
|
||
one devbox image, this is a reasonable prior for what tor-ms22 and MBP-M1-2020 hold. Counts are **parent
|
||
drawers in the `mempalace_drawers` collection** — see the correction note below, which is the whole reason
|
||
to state the unit:
|
||
|
||
| Class | Count | Share | Joiner action |
|
||
| --- | --- | --- | --- |
|
||
| Mined (`source_file` truthy) | 14,389 | **98.8%** | re-mine on synlig; **never replay** |
|
||
| Diary entries | 116 | 0.8% | replay + §7.6 suffix skip |
|
||
| Agent-authored drawers | 60 | 0.4% | replay by stored id (9 are `edited_since_filing`, §2.1c) |
|
||
| KG open facts | 34 | — | replay, server guard dedupes |
|
||
| KG **closed** facts | **0** | — | nothing to do — see below |
|
||
|
||
**Corrected figure.** An earlier pass reported 15,949 mined / 98.9%. That number was reconstructible
|
||
exactly as `16,338 (all embeddings rows) − 192 (diary rows) − 197 (agent rows) = 15,949`: it counted
|
||
**rows, not parent drawers**, and it counted them across **both collections**, so it silently absorbed all
|
||
1,560 `mempalace_closets` rows into the mined total. Closets are derived at mine time and are not joinable
|
||
at all. Two lessons, both now enforced in the census: **always filter by collection** (`embeddings ⋈
|
||
segments ⋈ collections`, since one sqlite file holds both), and **always state whether a count is rows or
|
||
parent drawers** — a chunked drawer contributes N rows and no parent row.
|
||
|
||
**Two consequences that shrink this project sharply.** First, the genuinely replay-only surface is
|
||
**176 records**, not thousands — so Phase C's writer is a small job, and the §7.6 diary guard that has
|
||
been treated as the blocker governs *116 records*. Second, **there are zero closed KG facts**, so the
|
||
unguarded-closed-fact gap (§2) is real in the code but currently empty in the data; it needs handling for
|
||
correctness, not for this join.
|
||
|
||
`filed_at` spread over the same parent drawers — 12 in May, 52 in June, 13,619 in July, 882 in August — is
|
||
the concrete case for regime A: an MCP replay would restamp all of it to the join date.
|
||
|
||
## 5. Open decisions for ALC
|
||
|
||
1. **Chronology: keep it or flatten it?** Regime A keeps it and costs a service stop plus an rsync of the
|
||
source palace onto synlig. Regime B is simpler and loses it. This is the fork in the road.
|
||
2. **Hallways for joined content:** re-mine to rebuild, or accept degraded `traverse`?
|
||
3. **Do MBP-M1-2020 and tor-ms22 keep their palaces on persistent storage?** If either is a Docker named
|
||
volume rather than a bind mount, its un-migrated content dies on the next container recreate — so the
|
||
census (Phase A) is time-sensitive there, and flipping before censusing is risky.
|
||
4. **Does §7.6 get fixed client-side (in the joiner) or upstream (probe-and-skip in `diary_write`)?** The
|
||
joiner needs the suffix skip either way; upstream would fix it for every writer.
|
||
|
||
## 6. Effort
|
||
|
||
Phase A is the bulk of the value and is small — two sqlite readers and a classifier. Phase B is trivial.
|
||
Phase C is where the risk lives, and regime A must be written defensively (refuse on a live server,
|
||
back up first, `--dry-run` default). Phase D is mostly assertions. Nothing here needs new server code,
|
||
which is the point of RFC 001 §4.4's "existence checks available today".
|
||
|
||
---
|
||
|
||
## Appendix — verified source references
|
||
|
||
mempalace 3.6.0, `/opt/uv-tools/mempalace/lib/python3.13/site-packages/mempalace/`:
|
||
|
||
| Claim | Location |
|
||
| --- | --- |
|
||
| Content-addressed drawer id, 24 hex over `wing\|room\|content` | `ids.py:80` (`make_drawer_id_from_content`), `_HASH_TRUNC_DRAWER = 24` at `ids.py:36` |
|
||
| MCP `add_drawer` uses that recipe | `mcp_server.py:2560` |
|
||
| Miner id over `source_file\|chunk_index` | `ids.py:67` (`make_drawer_id_from_chunk`), used `miner.py:1381`, `format_miner.py:645` |
|
||
| Idempotency probe + `already_exists` (probes parent **and** last chunk) | `mcp_server.py:2593-2604` |
|
||
| `filed_at` stamped server-side, no override | `mcp_server.py:2580` |
|
||
| Diary id recipe, suffix = `sha256(entry)[:12]` | `mcp_server.py:3510-3513` |
|
||
| Triple id `t_{s}_{p}_{o}_{sha256(valid_from\|recorded_at)[:12]}` | `ids.py:111-131`, `_HASH_TRUNC_TRIPLE = 12` at `ids.py:37` |
|
||
| `add_triple` guard scoped to open facts | `knowledge_graph.py:305-311` |
|
||
| Chunk size 800 | `config.py:274` |
|
||
| Direct-write precedent preserving ids+metadata | `migrate.py:326-330` |
|
||
| No join/replay/import tooling exists | `cli.py` subcommands; `exporter.py` emits markdown (lossy, not a join primitive); `diary_ingest.py` ingests *daily-summary files*, unrelated to agent diaries; `migrate.py` is single-palace chromadb recovery; `dedup.py` is cosine-similarity pruning within one `source_file` |
|
||
| Closets are miner-derived, no MCP write path | `closet_llm.py:8-10` ("Regex closets are always created by the miner"); `mcp_server.py:2913` exposes only `_purge_source_closets` |
|
||
| Two collections in one sqlite file | `mempalace_drawers` (14,778 rows) and `mempalace_closets` (1,560) in the archive, via `segments`→`collections` |
|