joakimp/mempalace-toolkit
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

feat(census): RFC 002 Phase A — read-only join census, and three RFC corrections it found

Browse Source 
bin/mempalace-census classifies a palace on disk into the RFC 002 §2 classes
(mined / diary / agent-authored) and emits a human report or a --json manifest
that feeds Phases B/C. Read-only: every sqlite handle is opened mode=ro, no
-wal/-shm is created, safe against a live mempalace-serve. Reads LOCAL DISK only
and warns if MEMPALACE_REMOTE_URL is set, so a local census can't be mistaken
for a remote one.

The design point is that it SELF-VERIFIES instead of trusting ids.py's
docstrings: for every replayable drawer it reassembles content from chunks,
recomputes the upstream id and compares to the stored id. That single check
covers the id recipe, the chunk reassembly order and the classifier at once --
176/176 accounted for on the reference palace -- and it falsified three things
the RFC previously asserted from a docs-only reading (now RFC 002 §2.1):

  (a) The hash input is LENGTH-PREFIXED, not "|"-joined. ids.py:31 defines
      _DELIM = "|" and the make_* docstrings describe f"{wing}|{room}|{content}",
      but _DELIM is dead code and _delimited_sha256 builds
      "".join(f"{len(part)}:{part}"). Measured: length-prefixed reproduces real
      ids 5/5, pipe-joined 0/5. Diary ids differ again -- a PLAIN sha256.

  (b) id_recipe is NOT a mined-only marker. It looked like a clean
      discriminator (same 14,586 count as source_file) but the server stamps
      'v3' on content ids too, so classifying on `source_file OR id_recipe`
      swallowed all 60 agent-authored drawers into MINED -- the dangerous
      direction, since Phase C would try to re-mine drawers that have no source
      file and silently drop them. Discriminator is a TRUTHY source_file (the
      writer stores "" rather than omitting the key), cross-checked against the
      miner-only keys source_mtime / normalize_version; disagreement is now a
      first-class warning.

  (c) Content ids DRIFT: 9 of 60 agent-authored drawers no longer reproduce
      their own id, because update_drawer preserves the id while rewriting and
      re-chunking. So "recompute the content id and skip if present" -- the
      strategy this RFC specified for regime A -- misses every drifted drawer
      and duplicates it. Phase C must key on the STORED id. Flagged as
      edited_since_filing in the manifest so Phase C can be tested on them.

Also corrects §4.1's headline number: 15,949 mined / 98.9% was reconstructible
exactly as 16,338 (all embeddings rows) - 192 (diary rows) - 197 (agent rows),
i.e. it counted rows rather than parent drawers AND spanned both collections,
absorbing all 1,560 non-joinable mempalace_closets rows into the mined total.
Correct figures: 14,389 mined / 116 diary / 60 agent-authored = 14,565 parents,
replay surface 176 (1.2%). Two rules now enforced in the tool: always filter by
collection (one sqlite file holds both), and always say whether a count is rows
or parent drawers -- a chunked drawer contributes N rows and no parent row.

One implementation trap worth recording: Chroma splits metadata across
string_value and int_value, so reading only string_value nulls every numeric key
(source_mtime, chunk_index, line_start) -- which made the miner-marker
cross-check report 100% conflict until the loader coalesced the two columns.
This commit is contained in:
Joakim Persson
2026-08-15 09:06:40 +02:00
parent 2f9170428c
commit f60cf9c732
4 changed files with 423 additions and 17 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/rfc-002-joiner.md
+59 -17
View File
@@ -52,15 +52,47 @@ Verified dedup keys — this supersedes nothing in RFC 001 §4.4 but makes each
| Class | How to identify it in the source palace | Dedup key (verified) | Regime A work | Regime B work |
| --- | --- | --- | --- | --- |
| **Agent-authored drawers** (`add_drawer`/`checkpoint`) | `source_file` empty | id is `sha256("{wing}\|{room}\|{content}")[:24]` (`ids.py:80`, used at `mcp_server.py:2560`) — **fully content-deterministic** | recompute id, skip if present | none: server probes `[drawer_id, last_chunk_id]` and returns `{"success": True, "reason": "already_exists"}` (`mcp_server.py:2593-2604`) |
| **Mined drawers** | `source_file` set | id is `sha256("{source_file}\|{chunk_index}")[:24]` (`ids.py:67`) — **path-dependent**, so the same content from two machines yields two rows | — | **do not replay.** Re-mine on synlig instead (§4) |
| **Diary entries** | id prefix `diary_`, `room="diary"` | id is `diary_{wing}_{now:%Y%m%d_%H%M%S%f}_{sha256(entry)[:12]}` (`mcp_server.py:3510-3513`). Timestamp is wall-clock, so **full ids never repeat** — match on the 12-hex suffix only | copy row verbatim (id and all) | build the target's suffix set, skip matches — this is RFC 001 §7.6 |
| **Agent-authored drawers** (`add_drawer`/`checkpoint`) | `source_file` empty — note the writer stores `""` rather than omitting the key, so test truthiness, not presence. `id_recipe` is **not** a discriminator: the server stamps `v3` on content ids too (§2.1) | `drawer_{wing}_{room}_{H(wing,room,content)[:24]}`, where `H` **length-prefixes** each part — *not* `\|`-joined (`ids.py:40-53`; `make_drawer_id_from_content` at `ids.py:80`, used at `mcp_server.py:2560`). Content-deterministic **unless the drawer was later edited** (§2.1) | replay by **stored** id — recompute-and-skip is unsafe (§2.1) | none: server probes `[drawer_id, last_chunk_id]` and returns `{"success": True, "reason": "already_exists"}` (`mcp_server.py:2593-2604`) |
| **Mined drawers** | `source_file` non-empty; corroborated by the miner-only keys `source_mtime` / `normalize_version` | `sha256(H(source_file, chunk_index))[:24]` (`ids.py:67`), same length-prefixed helper — **path-dependent**, so the same content from two machines yields two rows | — | **do not replay.** Re-mine on synlig instead (§4) |
| **Diary entries** | metadata `type="diary_entry"`; id prefix `diary_` | id is `diary_{wing}_{now:%Y%m%d_%H%M%S%f}_{sha256(entry)[:12]}` (`mcp_server.py:3510-3513`) — a **plain** sha256 of the entry, *not* the length-prefixed helper. Timestamp is wall-clock, so **full ids never repeat** — match on the 12-hex suffix only | copy row verbatim (id and all) | build the target's suffix set, skip matches — this is RFC 001 §7.6 |
| **KG open facts** (`valid_to IS NULL`) | `triples.valid_to IS NULL` | server guard `WHERE subject=? AND predicate=? AND object=? AND valid_to IS NULL` returns the existing id (`knowledge_graph.py:305-311`) | pre-query same key | none: just replay |
| **KG closed facts** (`valid_to` set) | `triples.valid_to IS NOT NULL` | **no server guard at all** — the guard above is scoped to open facts, so every replay inserts a fresh row | pre-query `(s,p,o,valid_from,valid_to)` | same pre-query, client-side |
| **Entities** | `entities` table | — | needed as FK targets for triples | `kg_add` creates them implicitly |
Chunking is deterministic and replay-safe: `DEFAULT_CHUNK_SIZE = 800` (`config.py:274`), chunk ids are
`f"{drawer_id}_chunk_{i:06d}"`, so nothing about chunking needs special handling.
`f"{drawer_id}_chunk_{i:06d}"`, so nothing about chunking needs special handling. One structural note the
census depends on: an oversized drawer has **no parent row** — only `_chunk_NNNNNN` rows — so the parent
id must be recovered by stripping the suffix, and content by concatenating in `chunk_index` order.
### 2.1 Three corrections this table needed — found by building the census, not by reading
Every recipe above is now **empirically verified**: `bin/mempalace-census` recomputes each id from
reassembled content and compares it to the stored id (176/176 accounted for on the reference palace).
Building that check falsified three things an earlier docs-only reading of `ids.py` had asserted.
**(a) The hash input is length-prefixed, not `\|`-joined.** `ids.py:31` defines `_DELIM = "|"` and the
`make_*` docstrings describe the input as `f"{wing}|{room}|{content}"`. Both are misleading: `_DELIM` is
**dead code** (defined, never referenced) and `_delimited_sha256()` actually builds
`"".join(f"{len(part)}:{part}")`. Length-prefixing is the better scheme — it is unambiguous where a bare
delimiter is not — but the docstrings never caught up. Measured on real drawers: length-prefixed
reproduces stored ids **5/5**, pipe-joined **0/5**. *Verify id recipes against the implementation and a
real row; a docstring is a claim, not evidence.*
**(b) `id_recipe` is not a mined-only marker.** It looked like a clean discriminator (14,586 rows, the
same count as `source_file`), but the server stamps `v3` on **every** v3 id including content-hashed
ones, and the sets differ by exactly the 60 agent-authored drawers. Classifying on
`source_file OR id_recipe` therefore swallowed all 60 into *mined* — the dangerous direction, since
Phase C would try to re-mine drawers that have no source file at all and silently drop them from the
join. The discriminator is a **truthy** `source_file`, cross-checked against the miner-only keys
`source_mtime` / `normalize_version`. The census now reports any disagreement between the two as a
first-class warning rather than trusting one signal.
**(c) Content ids drift, so "recompute the id and skip if present" is unsafe.** 9 of the 60
agent-authored drawers (15%) no longer reproduce their own id: `update_drawer` **preserves the id** while
rewriting and re-chunking content, so an edited drawer's content hash stops matching. This is expected
behaviour, but it breaks the obvious Regime A strategy — recomputing the content id and probing the
target for it **misses every drifted drawer and duplicates it**. Phase C must key on the **stored** id.
The census flags these as `edited_since_filing` in the manifest so Phase C can be tested against them.
### Not carried by either regime
@@ -87,13 +119,13 @@ re-mined on synlig anyway (§4), this resolves itself for the bulk.
## 3. Phases and deliverables
**Phase A — census (read-only, no writes anywhere).** Point a script at a palace on disk, enumerate every
parent drawer from `chroma.sqlite3` (`embeddings` ⋈ `embedding_metadata`) plus every row of
`knowledge_graph.sqlite3`, classify each into the §2 rows, and emit a manifest JSON + counts.
*Deliverable: "tor-ms22 holds N agent-authored drawers, M diary entries, K closed facts, and X mined
drawers we will re-mine instead."* This is the number that sizes everything else, and it is the piece to
run the moment tor-ms22 is reachable. **Build this first and independently** — it is useful even if the
writer is never written, and it cannot break anything.
**Phase A — census (read-only, no writes anywhere).** ✅ **Built: `bin/mempalace-census`.** Point it at a
palace on disk; it enumerates every parent drawer from `chroma.sqlite3` (`embeddings ⋈ embedding_metadata`,
filtered by collection) plus every row of `knowledge_graph.sqlite3`, classifies each into the §2 rows, and
emits `--json` (a manifest that feeds Phases B/C) or a human report. It opens every file `mode=ro` and is
safe to run against a live palace. It also **self-verifies** — recomputing each id from reassembled content
and comparing to the stored id, which checks the recipe, the chunk reassembly order and the classifier in
one pass. That check is what produced the §2.1 corrections. *Deliverable, realised: see §4.1.*
**Phase B — target index.** Given the primary, build the three lookup sets Phase C needs: content-drawer
ids, diary `sha256(entry)[:12]` suffixes, and `(s,p,o,valid_from,valid_to)` tuples. Over MCP this is
@@ -121,24 +153,34 @@ contract. Two backends behind one interface:
### 4.1 The census, run for real — the replay surface is tiny
Phase A's classifier was run against the frozen EMB-7KJ4VR4G archive (2026-08-15). Since the fleet
shares one devbox image, this is a reasonable prior for what tor-ms22 and MBP-M1-2020 hold:
Produced by `bin/mempalace-census` against the container-local palace (2026-08-15). Since the fleet shares
one devbox image, this is a reasonable prior for what tor-ms22 and MBP-M1-2020 hold. Counts are **parent
drawers in the `mempalace_drawers` collection** — see the correction note below, which is the whole reason
to state the unit:
| Class | Count | Share | Joiner action |
| --- | --- | --- | --- |
| Mined (`source_file` set) | 15,949 | **98.9%** | re-mine on synlig; **never replay** |
| Diary entries | 116 | 0.7% | replay + §7.6 suffix skip |
| Agent-authored drawers | 60 | 0.4% | replay, idempotent by content id |
| Mined (`source_file` truthy) | 14,389 | **98.8%** | re-mine on synlig; **never replay** |
| Diary entries | 116 | 0.8% | replay + §7.6 suffix skip |
| Agent-authored drawers | 60 | 0.4% | replay by stored id (9 are `edited_since_filing`, §2.1c) |
| KG open facts | 34 | — | replay, server guard dedupes |
| KG **closed** facts | **0** | — | nothing to do — see below |
**Corrected figure.** An earlier pass reported 15,949 mined / 98.9%. That number was reconstructible
exactly as `16,338 (all embeddings rows) − 192 (diary rows) − 197 (agent rows) = 15,949`: it counted
**rows, not parent drawers**, and it counted them across **both collections**, so it silently absorbed all
1,560 `mempalace_closets` rows into the mined total. Closets are derived at mine time and are not joinable
at all. Two lessons, both now enforced in the census: **always filter by collection** (`embeddings ⋈
segments ⋈ collections`, since one sqlite file holds both), and **always state whether a count is rows or
parent drawers** — a chunked drawer contributes N rows and no parent row.
**Two consequences that shrink this project sharply.** First, the genuinely replay-only surface is
**176 records**, not thousands — so Phase C's writer is a small job, and the §7.6 diary guard that has
been treated as the blocker governs *116 records*. Second, **there are zero closed KG facts**, so the
unguarded-closed-fact gap (§2) is real in the code but currently empty in the data; it needs handling for
correctness, not for this join.
`filed_at` spread in the same archive — 12 records in May, 52 in June, 15,174 in July, 887 in August — is
`filed_at` spread over the same parent drawers — 12 in May, 52 in June, 13,619 in July, 882 in August — is
the concrete case for regime A: an MCP replay would restamp all of it to the join date.
## 5. Open decisions for ALC