From f60cf9c73205656e5ff0e9b85cb5bb817655507b Mon Sep 17 00:00:00 2001 From: Joakim Persson Date: Sat, 15 Aug 2026 09:06:40 +0200 Subject: [PATCH] =?UTF-8?q?feat(census):=20RFC=20002=20Phase=20A=20?= =?UTF-8?q?=E2=80=94=20read-only=20join=20census,=20and=20three=20RFC=20co?= =?UTF-8?q?rrections=20it=20found?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit bin/mempalace-census classifies a palace on disk into the RFC 002 §2 classes (mined / diary / agent-authored) and emits a human report or a --json manifest that feeds Phases B/C. Read-only: every sqlite handle is opened mode=ro, no -wal/-shm is created, safe against a live mempalace-serve. Reads LOCAL DISK only and warns if MEMPALACE_REMOTE_URL is set, so a local census can't be mistaken for a remote one. The design point is that it SELF-VERIFIES instead of trusting ids.py's docstrings: for every replayable drawer it reassembles content from chunks, recomputes the upstream id and compares to the stored id. That single check covers the id recipe, the chunk reassembly order and the classifier at once -- 176/176 accounted for on the reference palace -- and it falsified three things the RFC previously asserted from a docs-only reading (now RFC 002 §2.1): (a) The hash input is LENGTH-PREFIXED, not "|"-joined. ids.py:31 defines _DELIM = "|" and the make_* docstrings describe f"{wing}|{room}|{content}", but _DELIM is dead code and _delimited_sha256 builds "".join(f"{len(part)}:{part}"). Measured: length-prefixed reproduces real ids 5/5, pipe-joined 0/5. Diary ids differ again -- a PLAIN sha256. (b) id_recipe is NOT a mined-only marker. It looked like a clean discriminator (same 14,586 count as source_file) but the server stamps 'v3' on content ids too, so classifying on `source_file OR id_recipe` swallowed all 60 agent-authored drawers into MINED -- the dangerous direction, since Phase C would try to re-mine drawers that have no source file and silently drop them. Discriminator is a TRUTHY source_file (the writer stores "" rather than omitting the key), cross-checked against the miner-only keys source_mtime / normalize_version; disagreement is now a first-class warning. (c) Content ids DRIFT: 9 of 60 agent-authored drawers no longer reproduce their own id, because update_drawer preserves the id while rewriting and re-chunking. So "recompute the content id and skip if present" -- the strategy this RFC specified for regime A -- misses every drifted drawer and duplicates it. Phase C must key on the STORED id. Flagged as edited_since_filing in the manifest so Phase C can be tested on them. Also corrects §4.1's headline number: 15,949 mined / 98.9% was reconstructible exactly as 16,338 (all embeddings rows) - 192 (diary rows) - 197 (agent rows), i.e. it counted rows rather than parent drawers AND spanned both collections, absorbing all 1,560 non-joinable mempalace_closets rows into the mined total. Correct figures: 14,389 mined / 116 diary / 60 agent-authored = 14,565 parents, replay surface 176 (1.2%). Two rules now enforced in the tool: always filter by collection (one sqlite file holds both), and always say whether a count is rows or parent drawers -- a chunked drawer contributes N rows and no parent row. One implementation trap worth recording: Chroma splits metadata across string_value and int_value, so reading only string_value nulls every numeric key (source_mtime, chunk_index, line_start) -- which made the miner-marker cross-check report 100% conflict until the loader coalesced the two columns. --- AGENTS.md | 4 + README.md | 28 ++++ bin/mempalace-census | 332 +++++++++++++++++++++++++++++++++++++++++ docs/rfc-002-joiner.md | 76 +++++++--- 4 files changed, 423 insertions(+), 17 deletions(-) create mode 100755 bin/mempalace-census diff --git a/AGENTS.md b/AGENTS.md index 918d9c1..31288e7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -15,10 +15,14 @@ README.md # Human-facing quickstart + per-tool usage reference SKILL.md # Agent skill (symlinked into ~/.agents/skills/ on install) docs/ rfc-001-global-palace.md # Design: one primary palace + per-machine local fallback (mempalace-edge) + rfc-002-joiner.md # Design: joining a machine-local palace into the primary bin/ mempalace-docs # Docs-only MemPalace miner (bash wrapper) mempalace-session # Opencode session → MemPalace bridge (bash + inline Python) mempalace-pi-session # pi session → MemPalace bridge (bash + inline Python) + mempalace-census # RFC 002 Phase A: read-only join census of a palace on disk + # (classifies mined / diary / agent-authored, self-verifies + # ids, emits --json manifest). Reads, never writes. contrib/ # systemd / launchd / cron templates for scheduling feeders extensions/ pi/ # pi↔mempalace MCP bridge (bridge-only; pi's own config is in the pi-toolkit repo) diff --git a/README.md b/README.md index 176e8d2..59a6726 100644 --- a/README.md +++ b/README.md @@ -7,6 +7,7 @@ Producer-side tooling for [MemPalace](https://github.com/MemPalace/mempalace) - `bin/mempalace-session` — exports [opencode](https://github.com/anomalyco/opencode) session history from its local SQLite DB to Claude Code JSONL, then mines it via `mempalace mine --mode convos`. - `bin/mempalace-pi-session` — the same idea for the [pi coding-agent](https://github.com/earendil-works/pi): exports its native JSONL session transcripts and mines them the same way. Unlike `mempalace-session`, this one is normally invoked *for you* — the pi bridge extension (below) runs it automatically on `session_shutdown` and a debounced `agent_settled`, so most pi machines never need the `contrib/` scheduling templates at all. See [`mempalace-pi-session`](#mempalace-pi-session) below and [`extensions/pi/README.md` § Automatic transcript feeding](extensions/pi/README.md#automatic-transcript-feeding). - `bin/mempalace-docs` — mines project directories into MemPalace while excluding source code, keeping the palace signal-dense. +- `bin/mempalace-census` — read-only census of a palace **on disk**: classifies every drawer as mined / diary / agent-authored, so you know what a cross-machine join would actually move (and what dedupes it) before writing anything. Implements [RFC 002](docs/rfc-002-joiner.md) Phase A. Never writes — safe against a live palace. - [`ARCHITECTURE.md`](ARCHITECTURE.md) — **canonical spec**: architecture diagram, component details, setup recipe, operational notes, upstream-retirement roadmap. - [`SKILL.md`](SKILL.md) — the companion agent skill, symlinked into `~/.agents/skills/opencode-mempalace-bridge/` on install. - [`extensions/pi/`](extensions/pi/) — the pi↔mempalace MCP bridge (a TypeScript extension symlinked into `~/.pi/agent/extensions/`). Pi's own base config (keybindings, env loader, settings template) is in the sibling [`pi-toolkit`](https://gitea.jordbo.se/joakimp/pi-toolkit) repo — split out 2026-05-05 so `opencode-devbox` can build slim containers without mempalace. @@ -408,6 +409,33 @@ mempalace-docs --help --- +## `mempalace-census` + +Read-only join census of a palace **on local disk**. Answers *"what would joining this palace into the primary actually move, and what dedupes it?"* — [RFC 002](docs/rfc-002-joiner.md) Phase A. + +```bash +mempalace-census # default palace, human report +mempalace-census --palace /mnt/tor-ms22/palace # a palace rsynced from another machine +mempalace-census --kg /path/knowledge_graph.sqlite3 +mempalace-census --json > manifest.json # machine-readable, feeds Phase B/C +mempalace-census --no-verify-ids # skip id recompute (faster on huge palaces) +mempalace-census --help +``` + +**Classification**, by descending signal strength: + +| Class | Identified by | Join action | +| --- | --- | --- | +| `DIARY` | metadata `type='diary_entry'` | replay + RFC 001 §7.6 suffix skip | +| `MINED` | truthy `source_file` | **re-mine on the target, never replay** — ids are path-derived, so replay duplicates | +| `AGENT-AUTHORED` | neither | replay, idempotent by content id | + +**It self-verifies rather than trusting the docs.** For every replayable drawer it reassembles content from chunks and recomputes the upstream id, comparing against the stored one — checking the id recipe, the chunk reassembly order and the classifier in a single pass. On the reference palace: 176/176 accounted for. This check is what caught three wrong assumptions now written up in [RFC 002 §2.1](docs/rfc-002-joiner.md) — most usefully that `id_recipe` is *not* a mined-only marker, and that `update_drawer` preserves a drawer's id while rewriting content, so 15% of agent-authored drawers no longer reproduce their own content hash (`edited_since_filing` in the manifest). + +**Safety:** opens every sqlite file with `mode=ro` and never writes — no `-wal`/`-shm` files are created, so it is safe to run while `mempalace-serve` is live. It reads **local disk only** and deliberately ignores `MEMPALACE_REMOTE_URL` (it warns if that is set, so you don't mistake a local census for a remote one). + +--- + ## `mempalace-session` Opencode → MemPalace session bridge. Reads `~/.local/share/opencode/opencode.db`, transforms each session into Claude Code JSONL, and files via `mempalace mine --mode convos`. diff --git a/bin/mempalace-census b/bin/mempalace-census new file mode 100755 index 0000000..7ef69d5 --- /dev/null +++ b/bin/mempalace-census @@ -0,0 +1,332 @@ +#!/usr/bin/env bash +# mempalace-census — RFC 002 Phase A: classify a palace by what a join could move. +# +# Answers "what would joining THIS palace into the primary actually move, and +# what dedupes it?" before any writer exists. Read-only: opens every sqlite file +# with `mode=ro` and never writes, so it is safe to run against a live palace +# while `mempalace-serve` is up. +# +# Classification (RFC 002 §2), by descending signal strength: +# DIARY metadata type='diary_entry' → replay + §7.6 suffix skip +# MINED source_file set / id_recipe → RE-MINE on the target, never +# replay (ids are path-derived, +# so replay duplicates) +# AGENT-AUTHORED neither → replay, idempotent by content id +# +# It also self-verifies rather than trusting the docs. For every classified +# drawer it recomputes the upstream ID from reassembled content and compares to +# the stored ID. That checks three things at once: the ID recipe, the chunk +# reassembly order, and the classification. A mismatch rate above ~0 means one +# of those assumptions is wrong for this palace — investigate before joining. +# +# ⚠ Recipe note: upstream's ids.py DOCSTRINGS claim the hash input is +# f"{wing}|{room}|{content}", and ids.py:31 defines _DELIM = "|". Both are +# misleading — _DELIM is dead code and _delimited_sha256() actually +# length-prefixes each part: "".join(f"{len(p)}:{p}"). Verified empirically: +# length-prefixed reproduces real IDs 5/5, pipe-joined 0/5. Diary IDs are +# different again — a PLAIN sha256(entry)[:12], not length-prefixed. +# +# This reads a palace on LOCAL DISK. It is not a client of a remote palace and +# deliberately ignores MEMPALACE_REMOTE_URL — pass --palace to point at a copy +# rsynced from another machine. +# +# Usage: +# mempalace-census # default palace, human report +# mempalace-census --palace /mnt/tor-ms22/palace +# mempalace-census --json > manifest.json # machine-readable, feeds Phase B/C +# mempalace-census --no-verify-ids # skip recompute (faster on huge palaces) + +set -euo pipefail + +# ── Defaults ────────────────────────────────────────────────────────────────── +PALACE="${MEMPALACE_PALACE:-$HOME/.mempalace/palace}" +KG="" +FORMAT="text" +VERIFY="1" + +usage() { + sed -n '2,36p' "$0" | sed 's/^# \{0,1\}//' +} + +# ── Argument parsing ────────────────────────────────────────────────────────── +while [[ $# -gt 0 ]]; do + case "$1" in + -h|--help) usage; exit 0 ;; + --palace) PALACE="${2:?--palace needs a path}"; shift 2 ;; + --kg) KG="${2:?--kg needs a path}"; shift 2 ;; + --json) FORMAT="json"; shift ;; + --no-verify-ids) VERIFY="0"; shift ;; + *) echo "mempalace-census: unknown argument '$1' (try --help)" >&2; exit 2 ;; + esac +done + +# ── Path resolution ─────────────────────────────────────────────────────────── +DB="$PALACE/chroma.sqlite3" +if [[ ! -f "$DB" ]]; then + echo "mempalace-census: no chroma.sqlite3 under '$PALACE'" >&2 + echo " pass --palace /path/to/palace (the dir CONTAINING chroma.sqlite3)" >&2 + exit 2 +fi +# KG lives beside the palace dir, not inside it. +[[ -n "$KG" ]] || KG="$(cd "$(dirname "$PALACE")" && pwd)/knowledge_graph.sqlite3" + +if [[ "$FORMAT" == "text" && -n "${MEMPALACE_REMOTE_URL:-}" ]]; then + echo "note: MEMPALACE_REMOTE_URL is set but ignored — this tool reads local disk." >&2 + echo " censusing: $DB" >&2 +fi + +# ── Census ──────────────────────────────────────────────────────────────────── +PALACE_DB="$DB" KG_DB="$KG" FMT="$FORMAT" VERIFY="$VERIFY" python3 - <<'PY' +import hashlib, json, os, re, sqlite3, sys +from collections import Counter, defaultdict + +DB, KG = os.environ["PALACE_DB"], os.environ["KG_DB"] +FMT, VERIFY = os.environ["FMT"], os.environ["VERIFY"] == "1" +CHUNK_RE = re.compile(r"_chunk_(\d+)$") + +def ro(path): + return sqlite3.connect(f"file:{path}?mode=ro", uri=True) + +# Upstream ids.py::_delimited_sha256 — length-prefixed, NOT delimiter-joined. +def drawer_hash(parts, trunc=24): + key = "".join(f"{len(str(p))}:{p}" for p in parts).encode() + return hashlib.sha256(key).hexdigest()[:trunc] + +con = ro(DB) + +# Both collections share one sqlite file. Filtering by collection is mandatory: +# an embeddings-wide query over-counts by the closet population (~10%). +counts_by_collection = dict( + con.execute( + "SELECT c.name, COUNT(*) FROM embeddings e " + "JOIN segments s ON s.id = e.segment_id " + "JOIN collections c ON c.id = s.collection GROUP BY c.name" + ).fetchall() +) + +rows = defaultdict(dict) +for eid, key, sval, ival in con.execute( + "SELECT e.embedding_id, m.key, m.string_value, m.int_value " + "FROM embeddings e " + "JOIN segments s ON s.id = e.segment_id " + "JOIN collections c ON c.id = s.collection " + "JOIN embedding_metadata m ON m.id = e.id " + "WHERE c.name = 'mempalace_drawers'" +): + # Chroma splits metadata by type across columns — numeric values (chunk_index, + # source_mtime, line_start, normalize_version) land in int_value and leave + # string_value NULL. Reading only string_value silently nulls every numeric + # key, which made the miner-marker cross-check below report 100% conflict. + rows[eid][key] = sval if sval is not None else ival + +# Collapse chunk rows into parent drawers. A chunked drawer has NO parent row +# (verified), so the parent is the id with the _chunk_NNNNNN suffix stripped. +parents = defaultdict(lambda: {"chunks": {}, "meta": None}) +for eid, meta in rows.items(): + m = CHUNK_RE.search(eid) + base = eid[: m.start()] if m else eid + idx = int(m.group(1)) if m else (meta.get("chunk_index") or 0) + p = parents[base] + p["chunks"][idx] = meta.get("chroma:document") or "" + # Keep the lowest-index row's metadata as canonical for the parent. + if p["meta"] is None or idx == 0: + p["meta"] = meta + +def classify(base, meta): + if meta.get("type") == "diary_entry" or base.startswith("diary_"): + return "diary" + # id_recipe is NOT a mined-only marker — the server stamps 'v3' on every + # v3 id, content-hashed ones included. Using it here misclassified all 60 + # agent-authored drawers in the reference palace as mined, which is the + # dangerous direction: Phase C would try to re-mine drawers that have no + # source file and silently drop them. A non-empty source_file is the real + # discriminator (note the miner writes '' rather than omitting the key, so + # presence-of-key is not enough — it must be truthy after strip()). + if (meta.get("source_file") or "").strip(): + return "mined" + return "agent_authored" + +cls = Counter() +wings = Counter() +months = Counter() +machines = Counter() +agents = Counter() +verify = {"checked": 0, "match": 0, "mismatch": 0, "drift": 0, + "samples": [], "drift_samples": []} +manifest = {"agent_authored": [], "diary": []} +signal_conflicts = [] + +for base, p in sorted(parents.items()): + meta = p["meta"] or {} + kind = classify(base, meta) + cls[kind] += 1 + wings[meta.get("wing") or "?"] += 1 + if meta.get("filed_at"): + months[str(meta["filed_at"])[:7]] += 1 + if meta.get("source_machine"): + machines[meta["source_machine"]] += 1 + if meta.get("added_by"): + agents[meta["added_by"]] += 1 + + # Cross-check the classification against the miner's OWN markers + # (source_mtime / normalize_version are written by the miner and by nothing + # else). A split here means this palace has a shape the classifier hasn't + # been taught, and the counts above are soft. + miner_marked = bool(meta.get("source_mtime") or meta.get("normalize_version")) + if miner_marked != (kind == "mined"): + signal_conflicts.append(base) + + if kind == "mined": + continue + + content = "".join(p["chunks"][i] for i in sorted(p["chunks"])) + wing, room = meta.get("wing") or "", meta.get("room") or "" + + if kind == "agent_authored": + expect = f"drawer_{wing}_{room}_{drawer_hash((wing, room, content))}" + drifted = expect != base + if VERIFY: + verify["checked"] += 1 + # A mismatch here is NOT a broken recipe — update_drawer preserves the + # original id while rewriting (and re-chunking) content, so an edited + # drawer's content hash legitimately stops reproducing its id. Named + # separately because it breaks one obvious Phase C strategy: you + # cannot "recompute the content id and check whether the target has + # it" — for drifted drawers that lookup misses and you duplicate. + # Replay by STORED id. + if drifted: + verify["drift"] += 1 + if len(verify["drift_samples"]) < 5: + verify["drift_samples"].append({"stored": base, "recomputed": expect}) + else: + verify["match"] += 1 + manifest["agent_authored"].append( + {"id": base, "wing": wing, "room": room, "chars": len(content), + "chunks": len(p["chunks"]), "filed_at": meta.get("filed_at"), + "added_by": meta.get("added_by"), + "content_id": expect, "edited_since_filing": drifted} + ) + else: # diary — id suffix is a PLAIN sha256(entry)[:12] + suffix = base.rsplit("_", 1)[-1] + recomputed = hashlib.sha256(content.encode()).hexdigest()[:12] + if VERIFY: + verify["checked"] += 1 + if suffix == recomputed: + verify["match"] += 1 + else: + verify["mismatch"] += 1 + if len(verify["samples"]) < 5: + verify["samples"].append({"stored": base, "recomputed_suffix": recomputed}) + manifest["diary"].append( + {"id": base, "wing": wing, "chars": len(content), "chunks": len(p["chunks"]), + "agent": meta.get("agent"), "topic": meta.get("topic"), + "date": meta.get("date"), "dedup_suffix": suffix, + "suffix_verified": suffix == recomputed} + ) + +# ── Knowledge graph ─────────────────────────────────────────────────────────── +kg = {"present": os.path.isfile(KG)} +if kg["present"]: + k = ro(KG) + try: + kg["open_facts"] = k.execute("SELECT COUNT(*) FROM triples WHERE valid_to IS NULL").fetchone()[0] + kg["closed_facts"] = k.execute("SELECT COUNT(*) FROM triples WHERE valid_to IS NOT NULL").fetchone()[0] + kg["entities"] = k.execute("SELECT COUNT(*) FROM entities").fetchone()[0] + kg["predicates"] = dict( + k.execute("SELECT predicate, COUNT(*) FROM triples GROUP BY 1 ORDER BY 2 DESC LIMIT 10").fetchall() + ) + except sqlite3.Error as e: + kg["error"] = str(e) + +report = { + "palace": DB, + "kg": KG, + "rows_by_collection": counts_by_collection, + "parent_drawers": sum(cls.values()), + "classes": dict(cls), + "replay_surface": cls["diary"] + cls["agent_authored"], + "by_wing": dict(wings.most_common()), + "filed_at_by_month": dict(sorted(months.items())), + "source_machine": dict(machines), + "added_by": dict(agents.most_common()), + "id_verification": verify if VERIFY else "skipped", + "signal_conflicts": len(signal_conflicts), + "knowledge_graph": kg, + "manifest": manifest, +} + +if FMT == "json": + print(json.dumps(report, indent=2, sort_keys=False)) + sys.exit(0) + +# ── Human report ────────────────────────────────────────────────────────────── +def bar(n, total, width=28): + return "█" * max(1, round(width * n / total)) if n and total else "" + +print(f"\n palace : {DB}") +print(f" kg : {KG}{'' if kg['present'] else ' (absent)'}") +print("\n ── rows per collection ─────────────────────────────────────") +for name, n in sorted(counts_by_collection.items()): + note = " ← derived at mine time, NOT joinable" if "closet" in name else "" + print(f" {name:<20} {n:>7}{note}") + +total = sum(cls.values()) +print(f"\n ── parent drawers: {total} ───────────────────────────────────") +labels = { + "mined": "MINED re-mine on target, never replay", + "diary": "DIARY replay + §7.6 suffix skip", + "agent_authored": "AGENT-AUTHORED replay, idempotent by content id", +} +for k in ("mined", "diary", "agent_authored"): + n = cls.get(k, 0) + pct = 100.0 * n / total if total else 0 + print(f" {n:>7} {pct:>5.1f}% {labels[k]}") +print(f"\n → REPLAY SURFACE: {report['replay_surface']} records " + f"({100.0 * report['replay_surface'] / total if total else 0:.1f}% of the palace)") + +if VERIFY: + v = verify + state = "OK" if v["mismatch"] == 0 else "⚠ MISMATCH" + print(f"\n ── id recipe / reassembly self-check: {state} ─────────────") + print(f" recomputed {v['checked']} ids — {v['match']} reproduce their stored id, " + f"{v['mismatch']} unexplained") + for s in v["samples"]: + print(f" stored: {s.get('stored')}") + print(f" recomputed: {s.get('recomputed') or s.get('recomputed_suffix')}") + if v["drift"]: + print(f"\n {v['drift']} agent-authored drawers EDITED SINCE FILING " + f"(content hash no longer reproduces the id).") + print(" update_drawer keeps the id and re-chunks, so this is expected — but it") + print(" means Phase C must replay by STORED id. Recomputing the content id and") + print(" probing the target for it would miss these and duplicate them.") + for s in v["drift_samples"][:3]: + print(f" {s['stored']}") +if signal_conflicts: + print(f"\n ⚠ {len(signal_conflicts)} drawers where the class disagrees with the miner's") + print(" own markers (source_mtime / normalize_version) — classifier needs teaching") + +print("\n ── by wing ─────────────────────────────────────────────────") +for w, n in wings.most_common(10): + print(f" {n:>7} {w}") + +if months: + print("\n ── filed_at spread (an MCP replay would flatten all of this) ") + mx = max(months.values()) + for m, n in sorted(months.items()): + print(f" {m} {n:>6} {bar(n, mx)}") + +if machines: + print("\n ── source_machine ──────────────────────────────────────────") + for m, n in machines.most_common(): + print(f" {n:>7} {m}") + +if kg["present"] and "error" not in kg: + print("\n ── knowledge graph ─────────────────────────────────────────") + print(f" {kg['entities']:>7} entities") + print(f" {kg['open_facts']:>7} open facts (server guard dedupes → replay as-is)") + print(f" {kg['closed_facts']:>7} closed facts (NO server guard → client pre-query)") +elif kg.get("error"): + print(f"\n ⚠ knowledge graph unreadable: {kg['error']}") + +print("\n next: --json > manifest.json feeds RFC 002 Phase B/C.\n") +PY diff --git a/docs/rfc-002-joiner.md b/docs/rfc-002-joiner.md index 664e900..f9be2db 100644 --- a/docs/rfc-002-joiner.md +++ b/docs/rfc-002-joiner.md @@ -52,15 +52,47 @@ Verified dedup keys — this supersedes nothing in RFC 001 §4.4 but makes each | Class | How to identify it in the source palace | Dedup key (verified) | Regime A work | Regime B work | | --- | --- | --- | --- | --- | -| **Agent-authored drawers** (`add_drawer`/`checkpoint`) | `source_file` empty | id is `sha256("{wing}\|{room}\|{content}")[:24]` (`ids.py:80`, used at `mcp_server.py:2560`) — **fully content-deterministic** | recompute id, skip if present | none: server probes `[drawer_id, last_chunk_id]` and returns `{"success": True, "reason": "already_exists"}` (`mcp_server.py:2593-2604`) | -| **Mined drawers** | `source_file` set | id is `sha256("{source_file}\|{chunk_index}")[:24]` (`ids.py:67`) — **path-dependent**, so the same content from two machines yields two rows | — | **do not replay.** Re-mine on synlig instead (§4) | -| **Diary entries** | id prefix `diary_`, `room="diary"` | id is `diary_{wing}_{now:%Y%m%d_%H%M%S%f}_{sha256(entry)[:12]}` (`mcp_server.py:3510-3513`). Timestamp is wall-clock, so **full ids never repeat** — match on the 12-hex suffix only | copy row verbatim (id and all) | build the target's suffix set, skip matches — this is RFC 001 §7.6 | +| **Agent-authored drawers** (`add_drawer`/`checkpoint`) | `source_file` empty — note the writer stores `""` rather than omitting the key, so test truthiness, not presence. `id_recipe` is **not** a discriminator: the server stamps `v3` on content ids too (§2.1) | `drawer_{wing}_{room}_{H(wing,room,content)[:24]}`, where `H` **length-prefixes** each part — *not* `\|`-joined (`ids.py:40-53`; `make_drawer_id_from_content` at `ids.py:80`, used at `mcp_server.py:2560`). Content-deterministic **unless the drawer was later edited** (§2.1) | replay by **stored** id — recompute-and-skip is unsafe (§2.1) | none: server probes `[drawer_id, last_chunk_id]` and returns `{"success": True, "reason": "already_exists"}` (`mcp_server.py:2593-2604`) | +| **Mined drawers** | `source_file` non-empty; corroborated by the miner-only keys `source_mtime` / `normalize_version` | `sha256(H(source_file, chunk_index))[:24]` (`ids.py:67`), same length-prefixed helper — **path-dependent**, so the same content from two machines yields two rows | — | **do not replay.** Re-mine on synlig instead (§4) | +| **Diary entries** | metadata `type="diary_entry"`; id prefix `diary_` | id is `diary_{wing}_{now:%Y%m%d_%H%M%S%f}_{sha256(entry)[:12]}` (`mcp_server.py:3510-3513`) — a **plain** sha256 of the entry, *not* the length-prefixed helper. Timestamp is wall-clock, so **full ids never repeat** — match on the 12-hex suffix only | copy row verbatim (id and all) | build the target's suffix set, skip matches — this is RFC 001 §7.6 | | **KG open facts** (`valid_to IS NULL`) | `triples.valid_to IS NULL` | server guard `WHERE subject=? AND predicate=? AND object=? AND valid_to IS NULL` returns the existing id (`knowledge_graph.py:305-311`) | pre-query same key | none: just replay | | **KG closed facts** (`valid_to` set) | `triples.valid_to IS NOT NULL` | **no server guard at all** — the guard above is scoped to open facts, so every replay inserts a fresh row | pre-query `(s,p,o,valid_from,valid_to)` | same pre-query, client-side | | **Entities** | `entities` table | — | needed as FK targets for triples | `kg_add` creates them implicitly | Chunking is deterministic and replay-safe: `DEFAULT_CHUNK_SIZE = 800` (`config.py:274`), chunk ids are -`f"{drawer_id}_chunk_{i:06d}"`, so nothing about chunking needs special handling. +`f"{drawer_id}_chunk_{i:06d}"`, so nothing about chunking needs special handling. One structural note the +census depends on: an oversized drawer has **no parent row** — only `_chunk_NNNNNN` rows — so the parent +id must be recovered by stripping the suffix, and content by concatenating in `chunk_index` order. + +### 2.1 Three corrections this table needed — found by building the census, not by reading + +Every recipe above is now **empirically verified**: `bin/mempalace-census` recomputes each id from +reassembled content and compares it to the stored id (176/176 accounted for on the reference palace). +Building that check falsified three things an earlier docs-only reading of `ids.py` had asserted. + +**(a) The hash input is length-prefixed, not `\|`-joined.** `ids.py:31` defines `_DELIM = "|"` and the +`make_*` docstrings describe the input as `f"{wing}|{room}|{content}"`. Both are misleading: `_DELIM` is +**dead code** (defined, never referenced) and `_delimited_sha256()` actually builds +`"".join(f"{len(part)}:{part}")`. Length-prefixing is the better scheme — it is unambiguous where a bare +delimiter is not — but the docstrings never caught up. Measured on real drawers: length-prefixed +reproduces stored ids **5/5**, pipe-joined **0/5**. *Verify id recipes against the implementation and a +real row; a docstring is a claim, not evidence.* + +**(b) `id_recipe` is not a mined-only marker.** It looked like a clean discriminator (14,586 rows, the +same count as `source_file`), but the server stamps `v3` on **every** v3 id including content-hashed +ones, and the sets differ by exactly the 60 agent-authored drawers. Classifying on +`source_file OR id_recipe` therefore swallowed all 60 into *mined* — the dangerous direction, since +Phase C would try to re-mine drawers that have no source file at all and silently drop them from the +join. The discriminator is a **truthy** `source_file`, cross-checked against the miner-only keys +`source_mtime` / `normalize_version`. The census now reports any disagreement between the two as a +first-class warning rather than trusting one signal. + +**(c) Content ids drift, so "recompute the id and skip if present" is unsafe.** 9 of the 60 +agent-authored drawers (15%) no longer reproduce their own id: `update_drawer` **preserves the id** while +rewriting and re-chunking content, so an edited drawer's content hash stops matching. This is expected +behaviour, but it breaks the obvious Regime A strategy — recomputing the content id and probing the +target for it **misses every drifted drawer and duplicates it**. Phase C must key on the **stored** id. +The census flags these as `edited_since_filing` in the manifest so Phase C can be tested against them. ### Not carried by either regime @@ -87,13 +119,13 @@ re-mined on synlig anyway (§4), this resolves itself for the bulk. ## 3. Phases and deliverables -**Phase A — census (read-only, no writes anywhere).** Point a script at a palace on disk, enumerate every -parent drawer from `chroma.sqlite3` (`embeddings` ⋈ `embedding_metadata`) plus every row of -`knowledge_graph.sqlite3`, classify each into the §2 rows, and emit a manifest JSON + counts. -*Deliverable: "tor-ms22 holds N agent-authored drawers, M diary entries, K closed facts, and X mined -drawers we will re-mine instead."* This is the number that sizes everything else, and it is the piece to -run the moment tor-ms22 is reachable. **Build this first and independently** — it is useful even if the -writer is never written, and it cannot break anything. +**Phase A — census (read-only, no writes anywhere).** ✅ **Built: `bin/mempalace-census`.** Point it at a +palace on disk; it enumerates every parent drawer from `chroma.sqlite3` (`embeddings ⋈ embedding_metadata`, +filtered by collection) plus every row of `knowledge_graph.sqlite3`, classifies each into the §2 rows, and +emits `--json` (a manifest that feeds Phases B/C) or a human report. It opens every file `mode=ro` and is +safe to run against a live palace. It also **self-verifies** — recomputing each id from reassembled content +and comparing to the stored id, which checks the recipe, the chunk reassembly order and the classifier in +one pass. That check is what produced the §2.1 corrections. *Deliverable, realised: see §4.1.* **Phase B — target index.** Given the primary, build the three lookup sets Phase C needs: content-drawer ids, diary `sha256(entry)[:12]` suffixes, and `(s,p,o,valid_from,valid_to)` tuples. Over MCP this is @@ -121,24 +153,34 @@ contract. Two backends behind one interface: ### 4.1 The census, run for real — the replay surface is tiny -Phase A's classifier was run against the frozen EMB-7KJ4VR4G archive (2026-08-15). Since the fleet -shares one devbox image, this is a reasonable prior for what tor-ms22 and MBP-M1-2020 hold: +Produced by `bin/mempalace-census` against the container-local palace (2026-08-15). Since the fleet shares +one devbox image, this is a reasonable prior for what tor-ms22 and MBP-M1-2020 hold. Counts are **parent +drawers in the `mempalace_drawers` collection** — see the correction note below, which is the whole reason +to state the unit: | Class | Count | Share | Joiner action | | --- | --- | --- | --- | -| Mined (`source_file` set) | 15,949 | **98.9%** | re-mine on synlig; **never replay** | -| Diary entries | 116 | 0.7% | replay + §7.6 suffix skip | -| Agent-authored drawers | 60 | 0.4% | replay, idempotent by content id | +| Mined (`source_file` truthy) | 14,389 | **98.8%** | re-mine on synlig; **never replay** | +| Diary entries | 116 | 0.8% | replay + §7.6 suffix skip | +| Agent-authored drawers | 60 | 0.4% | replay by stored id (9 are `edited_since_filing`, §2.1c) | | KG open facts | 34 | — | replay, server guard dedupes | | KG **closed** facts | **0** | — | nothing to do — see below | +**Corrected figure.** An earlier pass reported 15,949 mined / 98.9%. That number was reconstructible +exactly as `16,338 (all embeddings rows) − 192 (diary rows) − 197 (agent rows) = 15,949`: it counted +**rows, not parent drawers**, and it counted them across **both collections**, so it silently absorbed all +1,560 `mempalace_closets` rows into the mined total. Closets are derived at mine time and are not joinable +at all. Two lessons, both now enforced in the census: **always filter by collection** (`embeddings ⋈ +segments ⋈ collections`, since one sqlite file holds both), and **always state whether a count is rows or +parent drawers** — a chunked drawer contributes N rows and no parent row. + **Two consequences that shrink this project sharply.** First, the genuinely replay-only surface is **176 records**, not thousands — so Phase C's writer is a small job, and the §7.6 diary guard that has been treated as the blocker governs *116 records*. Second, **there are zero closed KG facts**, so the unguarded-closed-fact gap (§2) is real in the code but currently empty in the data; it needs handling for correctness, not for this join. -`filed_at` spread in the same archive — 12 records in May, 52 in June, 15,174 in July, 887 in August — is +`filed_at` spread over the same parent drawers — 12 in May, 52 in June, 13,619 in July, 882 in August — is the concrete case for regime A: an MCP replay would restamp all of it to the join date. ## 5. Open decisions for ALC