feat(census): RFC 002 Phase A — read-only join census, and three RFC corrections it found

bin/mempalace-census classifies a palace on disk into the RFC 002 §2 classes
(mined / diary / agent-authored) and emits a human report or a --json manifest
that feeds Phases B/C. Read-only: every sqlite handle is opened mode=ro, no
-wal/-shm is created, safe against a live mempalace-serve. Reads LOCAL DISK only
and warns if MEMPALACE_REMOTE_URL is set, so a local census can't be mistaken
for a remote one.

The design point is that it SELF-VERIFIES instead of trusting ids.py's
docstrings: for every replayable drawer it reassembles content from chunks,
recomputes the upstream id and compares to the stored id. That single check
covers the id recipe, the chunk reassembly order and the classifier at once --
176/176 accounted for on the reference palace -- and it falsified three things
the RFC previously asserted from a docs-only reading (now RFC 002 §2.1):

  (a) The hash input is LENGTH-PREFIXED, not "|"-joined. ids.py:31 defines
      _DELIM = "|" and the make_* docstrings describe f"{wing}|{room}|{content}",
      but _DELIM is dead code and _delimited_sha256 builds
      "".join(f"{len(part)}:{part}"). Measured: length-prefixed reproduces real
      ids 5/5, pipe-joined 0/5. Diary ids differ again -- a PLAIN sha256.

  (b) id_recipe is NOT a mined-only marker. It looked like a clean
      discriminator (same 14,586 count as source_file) but the server stamps
      'v3' on content ids too, so classifying on `source_file OR id_recipe`
      swallowed all 60 agent-authored drawers into MINED -- the dangerous
      direction, since Phase C would try to re-mine drawers that have no source
      file and silently drop them. Discriminator is a TRUTHY source_file (the
      writer stores "" rather than omitting the key), cross-checked against the
      miner-only keys source_mtime / normalize_version; disagreement is now a
      first-class warning.

  (c) Content ids DRIFT: 9 of 60 agent-authored drawers no longer reproduce
      their own id, because update_drawer preserves the id while rewriting and
      re-chunking. So "recompute the content id and skip if present" -- the
      strategy this RFC specified for regime A -- misses every drifted drawer
      and duplicates it. Phase C must key on the STORED id. Flagged as
      edited_since_filing in the manifest so Phase C can be tested on them.

Also corrects §4.1's headline number: 15,949 mined / 98.9% was reconstructible
exactly as 16,338 (all embeddings rows) - 192 (diary rows) - 197 (agent rows),
i.e. it counted rows rather than parent drawers AND spanned both collections,
absorbing all 1,560 non-joinable mempalace_closets rows into the mined total.
Correct figures: 14,389 mined / 116 diary / 60 agent-authored = 14,565 parents,
replay surface 176 (1.2%). Two rules now enforced in the tool: always filter by
collection (one sqlite file holds both), and always say whether a count is rows
or parent drawers -- a chunked drawer contributes N rows and no parent row.

One implementation trap worth recording: Chroma splits metadata across
string_value and int_value, so reading only string_value nulls every numeric key
(source_mtime, chunk_index, line_start) -- which made the miner-marker
cross-check report 100% conflict until the loader coalesced the two columns.
This commit is contained in:
Joakim Persson
2026-08-15 09:06:40 +02:00
parent 2f9170428c
commit f60cf9c732
4 changed files with 423 additions and 17 deletions
+28
View File
@@ -7,6 +7,7 @@ Producer-side tooling for [MemPalace](https://github.com/MemPalace/mempalace)
- `bin/mempalace-session` — exports [opencode](https://github.com/anomalyco/opencode) session history from its local SQLite DB to Claude Code JSONL, then mines it via `mempalace mine --mode convos`.
- `bin/mempalace-pi-session` — the same idea for the [pi coding-agent](https://github.com/earendil-works/pi): exports its native JSONL session transcripts and mines them the same way. Unlike `mempalace-session`, this one is normally invoked *for you* — the pi bridge extension (below) runs it automatically on `session_shutdown` and a debounced `agent_settled`, so most pi machines never need the `contrib/` scheduling templates at all. See [`mempalace-pi-session`](#mempalace-pi-session) below and [`extensions/pi/README.md` § Automatic transcript feeding](extensions/pi/README.md#automatic-transcript-feeding).
- `bin/mempalace-docs` — mines project directories into MemPalace while excluding source code, keeping the palace signal-dense.
- `bin/mempalace-census` — read-only census of a palace **on disk**: classifies every drawer as mined / diary / agent-authored, so you know what a cross-machine join would actually move (and what dedupes it) before writing anything. Implements [RFC 002](docs/rfc-002-joiner.md) Phase A. Never writes — safe against a live palace.
- [`ARCHITECTURE.md`](ARCHITECTURE.md) — **canonical spec**: architecture diagram, component details, setup recipe, operational notes, upstream-retirement roadmap.
- [`SKILL.md`](SKILL.md) — the companion agent skill, symlinked into `~/.agents/skills/opencode-mempalace-bridge/` on install.
- [`extensions/pi/`](extensions/pi/) — the pi↔mempalace MCP bridge (a TypeScript extension symlinked into `~/.pi/agent/extensions/`). Pi's own base config (keybindings, env loader, settings template) is in the sibling [`pi-toolkit`](https://gitea.jordbo.se/joakimp/pi-toolkit) repo — split out 2026-05-05 so `opencode-devbox` can build slim containers without mempalace.
@@ -408,6 +409,33 @@ mempalace-docs --help
---
## `mempalace-census`
Read-only join census of a palace **on local disk**. Answers *"what would joining this palace into the primary actually move, and what dedupes it?"* — [RFC 002](docs/rfc-002-joiner.md) Phase A.
```bash
mempalace-census # default palace, human report
mempalace-census --palace /mnt/tor-ms22/palace # a palace rsynced from another machine
mempalace-census --kg /path/knowledge_graph.sqlite3
mempalace-census --json > manifest.json # machine-readable, feeds Phase B/C
mempalace-census --no-verify-ids # skip id recompute (faster on huge palaces)
mempalace-census --help
```
**Classification**, by descending signal strength:
| Class | Identified by | Join action |
| --- | --- | --- |
| `DIARY` | metadata `type='diary_entry'` | replay + RFC 001 §7.6 suffix skip |
| `MINED` | truthy `source_file` | **re-mine on the target, never replay** — ids are path-derived, so replay duplicates |
| `AGENT-AUTHORED` | neither | replay, idempotent by content id |
**It self-verifies rather than trusting the docs.** For every replayable drawer it reassembles content from chunks and recomputes the upstream id, comparing against the stored one — checking the id recipe, the chunk reassembly order and the classifier in a single pass. On the reference palace: 176/176 accounted for. This check is what caught three wrong assumptions now written up in [RFC 002 §2.1](docs/rfc-002-joiner.md) — most usefully that `id_recipe` is *not* a mined-only marker, and that `update_drawer` preserves a drawer's id while rewriting content, so 15% of agent-authored drawers no longer reproduce their own content hash (`edited_since_filing` in the manifest).
**Safety:** opens every sqlite file with `mode=ro` and never writes — no `-wal`/`-shm` files are created, so it is safe to run while `mempalace-serve` is live. It reads **local disk only** and deliberately ignores `MEMPALACE_REMOTE_URL` (it warns if that is set, so you don't mistake a local census for a remote one).
---
## `mempalace-session`
Opencode → MemPalace session bridge. Reads `~/.local/share/opencode/opencode.db`, transforms each session into Claude Code JSONL, and files via `mempalace mine --mode convos`.