bin/mempalace-census classifies a palace on disk into the RFC 002 §2 classes
(mined / diary / agent-authored) and emits a human report or a --json manifest
that feeds Phases B/C. Read-only: every sqlite handle is opened mode=ro, no
-wal/-shm is created, safe against a live mempalace-serve. Reads LOCAL DISK only
and warns if MEMPALACE_REMOTE_URL is set, so a local census can't be mistaken
for a remote one.
The design point is that it SELF-VERIFIES instead of trusting ids.py's
docstrings: for every replayable drawer it reassembles content from chunks,
recomputes the upstream id and compares to the stored id. That single check
covers the id recipe, the chunk reassembly order and the classifier at once --
176/176 accounted for on the reference palace -- and it falsified three things
the RFC previously asserted from a docs-only reading (now RFC 002 §2.1):
(a) The hash input is LENGTH-PREFIXED, not "|"-joined. ids.py:31 defines
_DELIM = "|" and the make_* docstrings describe f"{wing}|{room}|{content}",
but _DELIM is dead code and _delimited_sha256 builds
"".join(f"{len(part)}:{part}"). Measured: length-prefixed reproduces real
ids 5/5, pipe-joined 0/5. Diary ids differ again -- a PLAIN sha256.
(b) id_recipe is NOT a mined-only marker. It looked like a clean
discriminator (same 14,586 count as source_file) but the server stamps
'v3' on content ids too, so classifying on `source_file OR id_recipe`
swallowed all 60 agent-authored drawers into MINED -- the dangerous
direction, since Phase C would try to re-mine drawers that have no source
file and silently drop them. Discriminator is a TRUTHY source_file (the
writer stores "" rather than omitting the key), cross-checked against the
miner-only keys source_mtime / normalize_version; disagreement is now a
first-class warning.
(c) Content ids DRIFT: 9 of 60 agent-authored drawers no longer reproduce
their own id, because update_drawer preserves the id while rewriting and
re-chunking. So "recompute the content id and skip if present" -- the
strategy this RFC specified for regime A -- misses every drifted drawer
and duplicates it. Phase C must key on the STORED id. Flagged as
edited_since_filing in the manifest so Phase C can be tested on them.
Also corrects §4.1's headline number: 15,949 mined / 98.9% was reconstructible
exactly as 16,338 (all embeddings rows) - 192 (diary rows) - 197 (agent rows),
i.e. it counted rows rather than parent drawers AND spanned both collections,
absorbing all 1,560 non-joinable mempalace_closets rows into the mined total.
Correct figures: 14,389 mined / 116 diary / 60 agent-authored = 14,565 parents,
replay surface 176 (1.2%). Two rules now enforced in the tool: always filter by
collection (one sqlite file holds both), and always say whether a count is rows
or parent drawers -- a chunked drawer contributes N rows and no parent row.
One implementation trap worth recording: Chroma splits metadata across
string_value and int_value, so reading only string_value nulls every numeric key
(source_mtime, chunk_index, line_start) -- which made the miner-marker
cross-check report 100% conflict until the loader coalesced the two columns.
Joakim Persson
f60cf9c732