diff --git a/AGENTS.md b/AGENTS.md index 918d9c1..31288e7 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -15,10 +15,14 @@ README.md # Human-facing quickstart + per-tool usage reference SKILL.md # Agent skill (symlinked into ~/.agents/skills/ on install) docs/ rfc-001-global-palace.md # Design: one primary palace + per-machine local fallback (mempalace-edge) + rfc-002-joiner.md # Design: joining a machine-local palace into the primary bin/ mempalace-docs # Docs-only MemPalace miner (bash wrapper) mempalace-session # Opencode session → MemPalace bridge (bash + inline Python) mempalace-pi-session # pi session → MemPalace bridge (bash + inline Python) + mempalace-census # RFC 002 Phase A: read-only join census of a palace on disk + # (classifies mined / diary / agent-authored, self-verifies + # ids, emits --json manifest). Reads, never writes. contrib/ # systemd / launchd / cron templates for scheduling feeders extensions/ pi/ # pi↔mempalace MCP bridge (bridge-only; pi's own config is in the pi-toolkit repo) diff --git a/README.md b/README.md index 176e8d2..59a6726 100644 --- a/README.md +++ b/README.md @@ -7,6 +7,7 @@ Producer-side tooling for [MemPalace](https://github.com/MemPalace/mempalace) - `bin/mempalace-session` — exports [opencode](https://github.com/anomalyco/opencode) session history from its local SQLite DB to Claude Code JSONL, then mines it via `mempalace mine --mode convos`. - `bin/mempalace-pi-session` — the same idea for the [pi coding-agent](https://github.com/earendil-works/pi): exports its native JSONL session transcripts and mines them the same way. Unlike `mempalace-session`, this one is normally invoked *for you* — the pi bridge extension (below) runs it automatically on `session_shutdown` and a debounced `agent_settled`, so most pi machines never need the `contrib/` scheduling templates at all. See [`mempalace-pi-session`](#mempalace-pi-session) below and [`extensions/pi/README.md` § Automatic transcript feeding](extensions/pi/README.md#automatic-transcript-feeding). - `bin/mempalace-docs` — mines project directories into MemPalace while excluding source code, keeping the palace signal-dense. +- `bin/mempalace-census` — read-only census of a palace **on disk**: classifies every drawer as mined / diary / agent-authored, so you know what a cross-machine join would actually move (and what dedupes it) before writing anything. Implements [RFC 002](docs/rfc-002-joiner.md) Phase A. Never writes — safe against a live palace. - [`ARCHITECTURE.md`](ARCHITECTURE.md) — **canonical spec**: architecture diagram, component details, setup recipe, operational notes, upstream-retirement roadmap. - [`SKILL.md`](SKILL.md) — the companion agent skill, symlinked into `~/.agents/skills/opencode-mempalace-bridge/` on install. - [`extensions/pi/`](extensions/pi/) — the pi↔mempalace MCP bridge (a TypeScript extension symlinked into `~/.pi/agent/extensions/`). Pi's own base config (keybindings, env loader, settings template) is in the sibling [`pi-toolkit`](https://gitea.jordbo.se/joakimp/pi-toolkit) repo — split out 2026-05-05 so `opencode-devbox` can build slim containers without mempalace. @@ -408,6 +409,33 @@ mempalace-docs --help --- +## `mempalace-census` + +Read-only join census of a palace **on local disk**. Answers *"what would joining this palace into the primary actually move, and what dedupes it?"* — [RFC 002](docs/rfc-002-joiner.md) Phase A. + +```bash +mempalace-census # default palace, human report +mempalace-census --palace /mnt/tor-ms22/palace # a palace rsynced from another machine +mempalace-census --kg /path/knowledge_graph.sqlite3 +mempalace-census --json > manifest.json # machine-readable, feeds Phase B/C +mempalace-census --no-verify-ids # skip id recompute (faster on huge palaces) +mempalace-census --help +``` + +**Classification**, by descending signal strength: + +| Class | Identified by | Join action | +| --- | --- | --- | +| `DIARY` | metadata `type='diary_entry'` | replay + RFC 001 §7.6 suffix skip | +| `MINED` | truthy `source_file` | **re-mine on the target, never replay** — ids are path-derived, so replay duplicates | +| `AGENT-AUTHORED` | neither | replay, idempotent by content id | + +**It self-verifies rather than trusting the docs.** For every replayable drawer it reassembles content from chunks and recomputes the upstream id, comparing against the stored one — checking the id recipe, the chunk reassembly order and the classifier in a single pass. On the reference palace: 176/176 accounted for. This check is what caught three wrong assumptions now written up in [RFC 002 §2.1](docs/rfc-002-joiner.md) — most usefully that `id_recipe` is *not* a mined-only marker, and that `update_drawer` preserves a drawer's id while rewriting content, so 15% of agent-authored drawers no longer reproduce their own content hash (`edited_since_filing` in the manifest). + +**Safety:** opens every sqlite file with `mode=ro` and never writes — no `-wal`/`-shm` files are created, so it is safe to run while `mempalace-serve` is live. It reads **local disk only** and deliberately ignores `MEMPALACE_REMOTE_URL` (it warns if that is set, so you don't mistake a local census for a remote one). + +--- + ## `mempalace-session` Opencode → MemPalace session bridge. Reads `~/.local/share/opencode/opencode.db`, transforms each session into Claude Code JSONL, and files via `mempalace mine --mode convos`. diff --git a/bin/mempalace-census b/bin/mempalace-census new file mode 100755 index 0000000..7ef69d5 --- /dev/null +++ b/bin/mempalace-census @@ -0,0 +1,332 @@ +#!/usr/bin/env bash +# mempalace-census — RFC 002 Phase A: classify a palace by what a join could move. +# +# Answers "what would joining THIS palace into the primary actually move, and +# what dedupes it?" before any writer exists. Read-only: opens every sqlite file +# with `mode=ro` and never writes, so it is safe to run against a live palace +# while `mempalace-serve` is up. +# +# Classification (RFC 002 §2), by descending signal strength: +# DIARY metadata type='diary_entry' → replay + §7.6 suffix skip +# MINED source_file set / id_recipe → RE-MINE on the target, never +# replay (ids are path-derived, +# so replay duplicates) +# AGENT-AUTHORED neither → replay, idempotent by content id +# +# It also self-verifies rather than trusting the docs. For every classified +# drawer it recomputes the upstream ID from reassembled content and compares to +# the stored ID. That checks three things at once: the ID recipe, the chunk +# reassembly order, and the classification. A mismatch rate above ~0 means one +# of those assumptions is wrong for this palace — investigate before joining. +# +# ⚠ Recipe note: upstream's ids.py DOCSTRINGS claim the hash input is +# f"{wing}|{room}|{content}", and ids.py:31 defines _DELIM = "|". Both are +# misleading — _DELIM is dead code and _delimited_sha256() actually +# length-prefixes each part: "".join(f"{len(p)}:{p}"). Verified empirically: +# length-prefixed reproduces real IDs 5/5, pipe-joined 0/5. Diary IDs are +# different again — a PLAIN sha256(entry)[:12], not length-prefixed. +# +# This reads a palace on LOCAL DISK. It is not a client of a remote palace and +# deliberately ignores MEMPALACE_REMOTE_URL — pass --palace to point at a copy +# rsynced from another machine. +# +# Usage: +# mempalace-census # default palace, human report +# mempalace-census --palace /mnt/tor-ms22/palace +# mempalace-census --json > manifest.json # machine-readable, feeds Phase B/C +# mempalace-census --no-verify-ids # skip recompute (faster on huge palaces) + +set -euo pipefail + +# ── Defaults ────────────────────────────────────────────────────────────────── +PALACE="${MEMPALACE_PALACE:-$HOME/.mempalace/palace}" +KG="" +FORMAT="text" +VERIFY="1" + +usage() { + sed -n '2,36p' "$0" | sed 's/^# \{0,1\}//' +} + +# ── Argument parsing ────────────────────────────────────────────────────────── +while [[ $# -gt 0 ]]; do + case "$1" in + -h|--help) usage; exit 0 ;; + --palace) PALACE="${2:?--palace needs a path}"; shift 2 ;; + --kg) KG="${2:?--kg needs a path}"; shift 2 ;; + --json) FORMAT="json"; shift ;; + --no-verify-ids) VERIFY="0"; shift ;; + *) echo "mempalace-census: unknown argument '$1' (try --help)" >&2; exit 2 ;; + esac +done + +# ── Path resolution ─────────────────────────────────────────────────────────── +DB="$PALACE/chroma.sqlite3" +if [[ ! -f "$DB" ]]; then + echo "mempalace-census: no chroma.sqlite3 under '$PALACE'" >&2 + echo " pass --palace /path/to/palace (the dir CONTAINING chroma.sqlite3)" >&2 + exit 2 +fi +# KG lives beside the palace dir, not inside it. +[[ -n "$KG" ]] || KG="$(cd "$(dirname "$PALACE")" && pwd)/knowledge_graph.sqlite3" + +if [[ "$FORMAT" == "text" && -n "${MEMPALACE_REMOTE_URL:-}" ]]; then + echo "note: MEMPALACE_REMOTE_URL is set but ignored — this tool reads local disk." >&2 + echo " censusing: $DB" >&2 +fi + +# ── Census ──────────────────────────────────────────────────────────────────── +PALACE_DB="$DB" KG_DB="$KG" FMT="$FORMAT" VERIFY="$VERIFY" python3 - <<'PY' +import hashlib, json, os, re, sqlite3, sys +from collections import Counter, defaultdict + +DB, KG = os.environ["PALACE_DB"], os.environ["KG_DB"] +FMT, VERIFY = os.environ["FMT"], os.environ["VERIFY"] == "1" +CHUNK_RE = re.compile(r"_chunk_(\d+)$") + +def ro(path): + return sqlite3.connect(f"file:{path}?mode=ro", uri=True) + +# Upstream ids.py::_delimited_sha256 — length-prefixed, NOT delimiter-joined. +def drawer_hash(parts, trunc=24): + key = "".join(f"{len(str(p))}:{p}" for p in parts).encode() + return hashlib.sha256(key).hexdigest()[:trunc] + +con = ro(DB) + +# Both collections share one sqlite file. Filtering by collection is mandatory: +# an embeddings-wide query over-counts by the closet population (~10%). +counts_by_collection = dict( + con.execute( + "SELECT c.name, COUNT(*) FROM embeddings e " + "JOIN segments s ON s.id = e.segment_id " + "JOIN collections c ON c.id = s.collection GROUP BY c.name" + ).fetchall() +) + +rows = defaultdict(dict) +for eid, key, sval, ival in con.execute( + "SELECT e.embedding_id, m.key, m.string_value, m.int_value " + "FROM embeddings e " + "JOIN segments s ON s.id = e.segment_id " + "JOIN collections c ON c.id = s.collection " + "JOIN embedding_metadata m ON m.id = e.id " + "WHERE c.name = 'mempalace_drawers'" +): + # Chroma splits metadata by type across columns — numeric values (chunk_index, + # source_mtime, line_start, normalize_version) land in int_value and leave + # string_value NULL. Reading only string_value silently nulls every numeric + # key, which made the miner-marker cross-check below report 100% conflict. + rows[eid][key] = sval if sval is not None else ival + +# Collapse chunk rows into parent drawers. A chunked drawer has NO parent row +# (verified), so the parent is the id with the _chunk_NNNNNN suffix stripped. +parents = defaultdict(lambda: {"chunks": {}, "meta": None}) +for eid, meta in rows.items(): + m = CHUNK_RE.search(eid) + base = eid[: m.start()] if m else eid + idx = int(m.group(1)) if m else (meta.get("chunk_index") or 0) + p = parents[base] + p["chunks"][idx] = meta.get("chroma:document") or "" + # Keep the lowest-index row's metadata as canonical for the parent. + if p["meta"] is None or idx == 0: + p["meta"] = meta + +def classify(base, meta): + if meta.get("type") == "diary_entry" or base.startswith("diary_"): + return "diary" + # id_recipe is NOT a mined-only marker — the server stamps 'v3' on every + # v3 id, content-hashed ones included. Using it here misclassified all 60 + # agent-authored drawers in the reference palace as mined, which is the + # dangerous direction: Phase C would try to re-mine drawers that have no + # source file and silently drop them. A non-empty source_file is the real + # discriminator (note the miner writes '' rather than omitting the key, so + # presence-of-key is not enough — it must be truthy after strip()). + if (meta.get("source_file") or "").strip(): + return "mined" + return "agent_authored" + +cls = Counter() +wings = Counter() +months = Counter() +machines = Counter() +agents = Counter() +verify = {"checked": 0, "match": 0, "mismatch": 0, "drift": 0, + "samples": [], "drift_samples": []} +manifest = {"agent_authored": [], "diary": []} +signal_conflicts = [] + +for base, p in sorted(parents.items()): + meta = p["meta"] or {} + kind = classify(base, meta) + cls[kind] += 1 + wings[meta.get("wing") or "?"] += 1 + if meta.get("filed_at"): + months[str(meta["filed_at"])[:7]] += 1 + if meta.get("source_machine"): + machines[meta["source_machine"]] += 1 + if meta.get("added_by"): + agents[meta["added_by"]] += 1 + + # Cross-check the classification against the miner's OWN markers + # (source_mtime / normalize_version are written by the miner and by nothing + # else). A split here means this palace has a shape the classifier hasn't + # been taught, and the counts above are soft. + miner_marked = bool(meta.get("source_mtime") or meta.get("normalize_version")) + if miner_marked != (kind == "mined"): + signal_conflicts.append(base) + + if kind == "mined": + continue + + content = "".join(p["chunks"][i] for i in sorted(p["chunks"])) + wing, room = meta.get("wing") or "", meta.get("room") or "" + + if kind == "agent_authored": + expect = f"drawer_{wing}_{room}_{drawer_hash((wing, room, content))}" + drifted = expect != base + if VERIFY: + verify["checked"] += 1 + # A mismatch here is NOT a broken recipe — update_drawer preserves the + # original id while rewriting (and re-chunking) content, so an edited + # drawer's content hash legitimately stops reproducing its id. Named + # separately because it breaks one obvious Phase C strategy: you + # cannot "recompute the content id and check whether the target has + # it" — for drifted drawers that lookup misses and you duplicate. + # Replay by STORED id. + if drifted: + verify["drift"] += 1 + if len(verify["drift_samples"]) < 5: + verify["drift_samples"].append({"stored": base, "recomputed": expect}) + else: + verify["match"] += 1 + manifest["agent_authored"].append( + {"id": base, "wing": wing, "room": room, "chars": len(content), + "chunks": len(p["chunks"]), "filed_at": meta.get("filed_at"), + "added_by": meta.get("added_by"), + "content_id": expect, "edited_since_filing": drifted} + ) + else: # diary — id suffix is a PLAIN sha256(entry)[:12] + suffix = base.rsplit("_", 1)[-1] + recomputed = hashlib.sha256(content.encode()).hexdigest()[:12] + if VERIFY: + verify["checked"] += 1 + if suffix == recomputed: + verify["match"] += 1 + else: + verify["mismatch"] += 1 + if len(verify["samples"]) < 5: + verify["samples"].append({"stored": base, "recomputed_suffix": recomputed}) + manifest["diary"].append( + {"id": base, "wing": wing, "chars": len(content), "chunks": len(p["chunks"]), + "agent": meta.get("agent"), "topic": meta.get("topic"), + "date": meta.get("date"), "dedup_suffix": suffix, + "suffix_verified": suffix == recomputed} + ) + +# ── Knowledge graph ─────────────────────────────────────────────────────────── +kg = {"present": os.path.isfile(KG)} +if kg["present"]: + k = ro(KG) + try: + kg["open_facts"] = k.execute("SELECT COUNT(*) FROM triples WHERE valid_to IS NULL").fetchone()[0] + kg["closed_facts"] = k.execute("SELECT COUNT(*) FROM triples WHERE valid_to IS NOT NULL").fetchone()[0] + kg["entities"] = k.execute("SELECT COUNT(*) FROM entities").fetchone()[0] + kg["predicates"] = dict( + k.execute("SELECT predicate, COUNT(*) FROM triples GROUP BY 1 ORDER BY 2 DESC LIMIT 10").fetchall() + ) + except sqlite3.Error as e: + kg["error"] = str(e) + +report = { + "palace": DB, + "kg": KG, + "rows_by_collection": counts_by_collection, + "parent_drawers": sum(cls.values()), + "classes": dict(cls), + "replay_surface": cls["diary"] + cls["agent_authored"], + "by_wing": dict(wings.most_common()), + "filed_at_by_month": dict(sorted(months.items())), + "source_machine": dict(machines), + "added_by": dict(agents.most_common()), + "id_verification": verify if VERIFY else "skipped", + "signal_conflicts": len(signal_conflicts), + "knowledge_graph": kg, + "manifest": manifest, +} + +if FMT == "json": + print(json.dumps(report, indent=2, sort_keys=False)) + sys.exit(0) + +# ── Human report ────────────────────────────────────────────────────────────── +def bar(n, total, width=28): + return "█" * max(1, round(width * n / total)) if n and total else "" + +print(f"\n palace : {DB}") +print(f" kg : {KG}{'' if kg['present'] else ' (absent)'}") +print("\n ── rows per collection ─────────────────────────────────────") +for name, n in sorted(counts_by_collection.items()): + note = " ← derived at mine time, NOT joinable" if "closet" in name else "" + print(f" {name:<20} {n:>7}{note}") + +total = sum(cls.values()) +print(f"\n ── parent drawers: {total} ───────────────────────────────────") +labels = { + "mined": "MINED re-mine on target, never replay", + "diary": "DIARY replay + §7.6 suffix skip", + "agent_authored": "AGENT-AUTHORED replay, idempotent by content id", +} +for k in ("mined", "diary", "agent_authored"): + n = cls.get(k, 0) + pct = 100.0 * n / total if total else 0 + print(f" {n:>7} {pct:>5.1f}% {labels[k]}") +print(f"\n → REPLAY SURFACE: {report['replay_surface']} records " + f"({100.0 * report['replay_surface'] / total if total else 0:.1f}% of the palace)") + +if VERIFY: + v = verify + state = "OK" if v["mismatch"] == 0 else "⚠ MISMATCH" + print(f"\n ── id recipe / reassembly self-check: {state} ─────────────") + print(f" recomputed {v['checked']} ids — {v['match']} reproduce their stored id, " + f"{v['mismatch']} unexplained") + for s in v["samples"]: + print(f" stored: {s.get('stored')}") + print(f" recomputed: {s.get('recomputed') or s.get('recomputed_suffix')}") + if v["drift"]: + print(f"\n {v['drift']} agent-authored drawers EDITED SINCE FILING " + f"(content hash no longer reproduces the id).") + print(" update_drawer keeps the id and re-chunks, so this is expected — but it") + print(" means Phase C must replay by STORED id. Recomputing the content id and") + print(" probing the target for it would miss these and duplicate them.") + for s in v["drift_samples"][:3]: + print(f" {s['stored']}") +if signal_conflicts: + print(f"\n ⚠ {len(signal_conflicts)} drawers where the class disagrees with the miner's") + print(" own markers (source_mtime / normalize_version) — classifier needs teaching") + +print("\n ── by wing ─────────────────────────────────────────────────") +for w, n in wings.most_common(10): + print(f" {n:>7} {w}") + +if months: + print("\n ── filed_at spread (an MCP replay would flatten all of this) ") + mx = max(months.values()) + for m, n in sorted(months.items()): + print(f" {m} {n:>6} {bar(n, mx)}") + +if machines: + print("\n ── source_machine ──────────────────────────────────────────") + for m, n in machines.most_common(): + print(f" {n:>7} {m}") + +if kg["present"] and "error" not in kg: + print("\n ── knowledge graph ─────────────────────────────────────────") + print(f" {kg['entities']:>7} entities") + print(f" {kg['open_facts']:>7} open facts (server guard dedupes → replay as-is)") + print(f" {kg['closed_facts']:>7} closed facts (NO server guard → client pre-query)") +elif kg.get("error"): + print(f"\n ⚠ knowledge graph unreadable: {kg['error']}") + +print("\n next: --json > manifest.json feeds RFC 002 Phase B/C.\n") +PY diff --git a/docs/rfc-002-joiner.md b/docs/rfc-002-joiner.md index 664e900..f9be2db 100644 --- a/docs/rfc-002-joiner.md +++ b/docs/rfc-002-joiner.md @@ -52,15 +52,47 @@ Verified dedup keys — this supersedes nothing in RFC 001 §4.4 but makes each | Class | How to identify it in the source palace | Dedup key (verified) | Regime A work | Regime B work | | --- | --- | --- | --- | --- | -| **Agent-authored drawers** (`add_drawer`/`checkpoint`) | `source_file` empty | id is `sha256("{wing}\|{room}\|{content}")[:24]` (`ids.py:80`, used at `mcp_server.py:2560`) — **fully content-deterministic** | recompute id, skip if present | none: server probes `[drawer_id, last_chunk_id]` and returns `{"success": True, "reason": "already_exists"}` (`mcp_server.py:2593-2604`) | -| **Mined drawers** | `source_file` set | id is `sha256("{source_file}\|{chunk_index}")[:24]` (`ids.py:67`) — **path-dependent**, so the same content from two machines yields two rows | — | **do not replay.** Re-mine on synlig instead (§4) | -| **Diary entries** | id prefix `diary_`, `room="diary"` | id is `diary_{wing}_{now:%Y%m%d_%H%M%S%f}_{sha256(entry)[:12]}` (`mcp_server.py:3510-3513`). Timestamp is wall-clock, so **full ids never repeat** — match on the 12-hex suffix only | copy row verbatim (id and all) | build the target's suffix set, skip matches — this is RFC 001 §7.6 | +| **Agent-authored drawers** (`add_drawer`/`checkpoint`) | `source_file` empty — note the writer stores `""` rather than omitting the key, so test truthiness, not presence. `id_recipe` is **not** a discriminator: the server stamps `v3` on content ids too (§2.1) | `drawer_{wing}_{room}_{H(wing,room,content)[:24]}`, where `H` **length-prefixes** each part — *not* `\|`-joined (`ids.py:40-53`; `make_drawer_id_from_content` at `ids.py:80`, used at `mcp_server.py:2560`). Content-deterministic **unless the drawer was later edited** (§2.1) | replay by **stored** id — recompute-and-skip is unsafe (§2.1) | none: server probes `[drawer_id, last_chunk_id]` and returns `{"success": True, "reason": "already_exists"}` (`mcp_server.py:2593-2604`) | +| **Mined drawers** | `source_file` non-empty; corroborated by the miner-only keys `source_mtime` / `normalize_version` | `sha256(H(source_file, chunk_index))[:24]` (`ids.py:67`), same length-prefixed helper — **path-dependent**, so the same content from two machines yields two rows | — | **do not replay.** Re-mine on synlig instead (§4) | +| **Diary entries** | metadata `type="diary_entry"`; id prefix `diary_` | id is `diary_{wing}_{now:%Y%m%d_%H%M%S%f}_{sha256(entry)[:12]}` (`mcp_server.py:3510-3513`) — a **plain** sha256 of the entry, *not* the length-prefixed helper. Timestamp is wall-clock, so **full ids never repeat** — match on the 12-hex suffix only | copy row verbatim (id and all) | build the target's suffix set, skip matches — this is RFC 001 §7.6 | | **KG open facts** (`valid_to IS NULL`) | `triples.valid_to IS NULL` | server guard `WHERE subject=? AND predicate=? AND object=? AND valid_to IS NULL` returns the existing id (`knowledge_graph.py:305-311`) | pre-query same key | none: just replay | | **KG closed facts** (`valid_to` set) | `triples.valid_to IS NOT NULL` | **no server guard at all** — the guard above is scoped to open facts, so every replay inserts a fresh row | pre-query `(s,p,o,valid_from,valid_to)` | same pre-query, client-side | | **Entities** | `entities` table | — | needed as FK targets for triples | `kg_add` creates them implicitly | Chunking is deterministic and replay-safe: `DEFAULT_CHUNK_SIZE = 800` (`config.py:274`), chunk ids are -`f"{drawer_id}_chunk_{i:06d}"`, so nothing about chunking needs special handling. +`f"{drawer_id}_chunk_{i:06d}"`, so nothing about chunking needs special handling. One structural note the +census depends on: an oversized drawer has **no parent row** — only `_chunk_NNNNNN` rows — so the parent +id must be recovered by stripping the suffix, and content by concatenating in `chunk_index` order. + +### 2.1 Three corrections this table needed — found by building the census, not by reading + +Every recipe above is now **empirically verified**: `bin/mempalace-census` recomputes each id from +reassembled content and compares it to the stored id (176/176 accounted for on the reference palace). +Building that check falsified three things an earlier docs-only reading of `ids.py` had asserted. + +**(a) The hash input is length-prefixed, not `\|`-joined.** `ids.py:31` defines `_DELIM = "|"` and the +`make_*` docstrings describe the input as `f"{wing}|{room}|{content}"`. Both are misleading: `_DELIM` is +**dead code** (defined, never referenced) and `_delimited_sha256()` actually builds +`"".join(f"{len(part)}:{part}")`. Length-prefixing is the better scheme — it is unambiguous where a bare +delimiter is not — but the docstrings never caught up. Measured on real drawers: length-prefixed +reproduces stored ids **5/5**, pipe-joined **0/5**. *Verify id recipes against the implementation and a +real row; a docstring is a claim, not evidence.* + +**(b) `id_recipe` is not a mined-only marker.** It looked like a clean discriminator (14,586 rows, the +same count as `source_file`), but the server stamps `v3` on **every** v3 id including content-hashed +ones, and the sets differ by exactly the 60 agent-authored drawers. Classifying on +`source_file OR id_recipe` therefore swallowed all 60 into *mined* — the dangerous direction, since +Phase C would try to re-mine drawers that have no source file at all and silently drop them from the +join. The discriminator is a **truthy** `source_file`, cross-checked against the miner-only keys +`source_mtime` / `normalize_version`. The census now reports any disagreement between the two as a +first-class warning rather than trusting one signal. + +**(c) Content ids drift, so "recompute the id and skip if present" is unsafe.** 9 of the 60 +agent-authored drawers (15%) no longer reproduce their own id: `update_drawer` **preserves the id** while +rewriting and re-chunking content, so an edited drawer's content hash stops matching. This is expected +behaviour, but it breaks the obvious Regime A strategy — recomputing the content id and probing the +target for it **misses every drifted drawer and duplicates it**. Phase C must key on the **stored** id. +The census flags these as `edited_since_filing` in the manifest so Phase C can be tested against them. ### Not carried by either regime @@ -87,13 +119,13 @@ re-mined on synlig anyway (§4), this resolves itself for the bulk. ## 3. Phases and deliverables -**Phase A — census (read-only, no writes anywhere).** Point a script at a palace on disk, enumerate every -parent drawer from `chroma.sqlite3` (`embeddings` ⋈ `embedding_metadata`) plus every row of -`knowledge_graph.sqlite3`, classify each into the §2 rows, and emit a manifest JSON + counts. -*Deliverable: "tor-ms22 holds N agent-authored drawers, M diary entries, K closed facts, and X mined -drawers we will re-mine instead."* This is the number that sizes everything else, and it is the piece to -run the moment tor-ms22 is reachable. **Build this first and independently** — it is useful even if the -writer is never written, and it cannot break anything. +**Phase A — census (read-only, no writes anywhere).** ✅ **Built: `bin/mempalace-census`.** Point it at a +palace on disk; it enumerates every parent drawer from `chroma.sqlite3` (`embeddings ⋈ embedding_metadata`, +filtered by collection) plus every row of `knowledge_graph.sqlite3`, classifies each into the §2 rows, and +emits `--json` (a manifest that feeds Phases B/C) or a human report. It opens every file `mode=ro` and is +safe to run against a live palace. It also **self-verifies** — recomputing each id from reassembled content +and comparing to the stored id, which checks the recipe, the chunk reassembly order and the classifier in +one pass. That check is what produced the §2.1 corrections. *Deliverable, realised: see §4.1.* **Phase B — target index.** Given the primary, build the three lookup sets Phase C needs: content-drawer ids, diary `sha256(entry)[:12]` suffixes, and `(s,p,o,valid_from,valid_to)` tuples. Over MCP this is @@ -121,24 +153,34 @@ contract. Two backends behind one interface: ### 4.1 The census, run for real — the replay surface is tiny -Phase A's classifier was run against the frozen EMB-7KJ4VR4G archive (2026-08-15). Since the fleet -shares one devbox image, this is a reasonable prior for what tor-ms22 and MBP-M1-2020 hold: +Produced by `bin/mempalace-census` against the container-local palace (2026-08-15). Since the fleet shares +one devbox image, this is a reasonable prior for what tor-ms22 and MBP-M1-2020 hold. Counts are **parent +drawers in the `mempalace_drawers` collection** — see the correction note below, which is the whole reason +to state the unit: | Class | Count | Share | Joiner action | | --- | --- | --- | --- | -| Mined (`source_file` set) | 15,949 | **98.9%** | re-mine on synlig; **never replay** | -| Diary entries | 116 | 0.7% | replay + §7.6 suffix skip | -| Agent-authored drawers | 60 | 0.4% | replay, idempotent by content id | +| Mined (`source_file` truthy) | 14,389 | **98.8%** | re-mine on synlig; **never replay** | +| Diary entries | 116 | 0.8% | replay + §7.6 suffix skip | +| Agent-authored drawers | 60 | 0.4% | replay by stored id (9 are `edited_since_filing`, §2.1c) | | KG open facts | 34 | — | replay, server guard dedupes | | KG **closed** facts | **0** | — | nothing to do — see below | +**Corrected figure.** An earlier pass reported 15,949 mined / 98.9%. That number was reconstructible +exactly as `16,338 (all embeddings rows) − 192 (diary rows) − 197 (agent rows) = 15,949`: it counted +**rows, not parent drawers**, and it counted them across **both collections**, so it silently absorbed all +1,560 `mempalace_closets` rows into the mined total. Closets are derived at mine time and are not joinable +at all. Two lessons, both now enforced in the census: **always filter by collection** (`embeddings ⋈ +segments ⋈ collections`, since one sqlite file holds both), and **always state whether a count is rows or +parent drawers** — a chunked drawer contributes N rows and no parent row. + **Two consequences that shrink this project sharply.** First, the genuinely replay-only surface is **176 records**, not thousands — so Phase C's writer is a small job, and the §7.6 diary guard that has been treated as the blocker governs *116 records*. Second, **there are zero closed KG facts**, so the unguarded-closed-fact gap (§2) is real in the code but currently empty in the data; it needs handling for correctness, not for this join. -`filed_at` spread in the same archive — 12 records in May, 52 in June, 15,174 in July, 887 in August — is +`filed_at` spread over the same parent drawers — 12 in May, 52 in June, 13,619 in July, 882 in August — is the concrete case for regime A: an MCP replay would restamp all of it to the join date. ## 5. Open decisions for ALC