joakimp
d4d8bb6109
RFC 003 §9.1 — retention is settled in direction (rotate old traffic out of
the way, logrotate-style, moved aside rather than destroyed) and the sketch
records the parts that are not obvious:
- Tier artifacts before events. Events are a few KB; artifacts are capped at
4 MiB and stored in-row, so moving artifact CONTENT cold while keeping the
kind/sha256/size/created_by stub reclaims nearly all the space and keeps the
audit trail ("what was handed over, by whom, verified how") intact.
- If events rotate at all, the unit is the correlation_id THREAD whose latest
event is terminal — never the row. Archiving an ask while leaving its reply
(or the reverse) breaks the owed-set join, and both failure modes are bad:
an ask that can never be cleared resurfaces as owed forever, or a reply is
orphaned from what it answered.
- Never rotate an event that is still owed. Owed-ness is DERIVED at read time,
so an unanswered ask is indistinguishable from a stale one except by that
derivation — a purely time-based sweep would discard the live obligations of
a machine that has merely been offline for a month, which is precisely the
case this log exists to serve.
- Rotation invalidates held cursors: since_event_id RAISES on an unknown id
(§7.5), so archiving an event a watcher holds as its resume point turns its
next poll into an error. Either announce rotation ahead of live cursors, or
teach the anchor lookup to fall back to created_at/hlc.
- Archive → verify (row counts, artifact sha256) → only then DELETE + VACUUM,
with a --dry-run that reports in thread units.
RFC 002 §5: "Open decisions for ALC" → "Open decisions". RFC 001 never names
the operator anywhere; impersonal is the mature precedent and ALC was never a
real identifier in the first place (it is the AAAK spec's illustrative code for
"Alice", copy-forwarded into ~700 diary entries without verification).
Same edit also removes two device names and a hostname from §5.3, which is
host inventory and belongs in the private fleet repo, not a public one. The
mechanism it teaches — a palace in a Docker named volume dies on the next
container recreate, so census before flipping — is unchanged and is the part
that mattered.
225 lines
17 KiB
Markdown
225 lines
17 KiB
Markdown
# RFC 002 — The joiner: replaying a second palace into the shared primary
|
||
|
||
**Status:** scoping. No code written yet.
|
||
**Author:** pi (agent), 2026-08-15.
|
||
**Context:** RFC 001 §4.4 designs a join as "idempotent replay of local history" but no replay tool
|
||
exists. The first two joins were whole-palace *file copies* (§4.4 deviation note), which work only for a
|
||
single source and cannot merge. tor-ms22 and MBP-M1-2020 each hold a substantial local palace whose
|
||
content should reach the primary. This document scopes the tool that does that.
|
||
|
||
> **Every mechanism below was read out of mempalace 3.6.0's own source** at
|
||
> `/opt/uv-tools/mempalace/lib/python3.13/site-packages/mempalace/`, with file:line in the appendix.
|
||
> An earlier attempt to gather these facts via a delegated subagent returned confident, fabricated code
|
||
> for a package path that does not exist on this machine. **Do not trust any claim in this document that
|
||
> the appendix does not cite.**
|
||
|
||
---
|
||
|
||
## 1. The finding that drives the design
|
||
|
||
**MCP replay cannot preserve `filed_at`.** `add_drawer` stamps
|
||
`"filed_at": datetime.now().isoformat()` server-side (`mcp_server.py:2580`) and exposes no override
|
||
parameter. `diary_write` is the same: it builds its own `now`-based id (`mcp_server.py:3510-3513`).
|
||
|
||
That is not a detail. This palace's value *is* its chronology — the primary's history runs from
|
||
2026-05-04, `list_drawers` filters on `since`/`before` against `filed_at`, and the diary is read in
|
||
order. A pure MCP replay of tor-ms22's palace would stamp **every** record with the join date,
|
||
collapsing months of history into one instant. RFC 001 §4.4 does not mention this, and it is the single
|
||
most important thing to decide before writing code.
|
||
|
||
`kg_add` is the exception: it accepts `valid_from`/`valid_to`, so **fact validity windows survive** even
|
||
though a triple's own id embeds `recorded_at`.
|
||
|
||
### Two write regimes
|
||
|
||
| | **A — direct disk write** (run on synlig) | **B — MCP replay** (run anywhere) |
|
||
| --- | --- | --- |
|
||
| Preserves `filed_at` / original ids | **Yes** — `col.add(ids=…, documents=…, metadatas=…)` | **No** — always `now()` |
|
||
| Server-side dedup guards | **Bypassed** — joiner owns all dedup | **Available** (see §2) |
|
||
| Requires quiescing `mempalace-serve` | Yes (palace is single-writer) | No |
|
||
| Requires source palace present on synlig | Yes (rsync it first) | No |
|
||
| Precedent in-tree | **`migrate.py`** already does exactly this: reads drawers+metadata straight from the palace's sqlite, then re-adds them into a fresh palace preserving ids, documents and metadata (`migrate.py:326-330`) | none |
|
||
|
||
**Recommendation:** regime **A** for the historical bulk (it is the only one that keeps the timeline, and
|
||
`migrate.py` is a working model to copy), regime **B** for small incremental top-ups where flattened
|
||
timestamps are acceptable. Do not build B first and discover the chronology loss afterwards.
|
||
|
||
---
|
||
|
||
## 2. What must move, and what dedupes it
|
||
|
||
Verified dedup keys — this supersedes nothing in RFC 001 §4.4 but makes each key concrete:
|
||
|
||
| Class | How to identify it in the source palace | Dedup key (verified) | Regime A work | Regime B work |
|
||
| --- | --- | --- | --- | --- |
|
||
| **Agent-authored drawers** (`add_drawer`/`checkpoint`) | `source_file` empty — note the writer stores `""` rather than omitting the key, so test truthiness, not presence. `id_recipe` is **not** a discriminator: the server stamps `v3` on content ids too (§2.1) | `drawer_{wing}_{room}_{H(wing,room,content)[:24]}`, where `H` **length-prefixes** each part — *not* `\|`-joined (`ids.py:40-53`; `make_drawer_id_from_content` at `ids.py:80`, used at `mcp_server.py:2560`). Content-deterministic **unless the drawer was later edited** (§2.1) | replay by **stored** id — recompute-and-skip is unsafe (§2.1) | none: server probes `[drawer_id, last_chunk_id]` and returns `{"success": True, "reason": "already_exists"}` (`mcp_server.py:2593-2604`) |
|
||
| **Mined drawers** | `source_file` non-empty; corroborated by the miner-only keys `source_mtime` / `normalize_version` | `sha256(H(source_file, chunk_index))[:24]` (`ids.py:67`), same length-prefixed helper — **path-dependent**, so the same content from two machines yields two rows | — | **do not replay.** Re-mine on synlig instead (§4) |
|
||
| **Diary entries** | metadata `type="diary_entry"`; id prefix `diary_` | id is `diary_{wing}_{now:%Y%m%d_%H%M%S%f}_{sha256(entry)[:12]}` (`mcp_server.py:3510-3513`) — a **plain** sha256 of the entry, *not* the length-prefixed helper. Timestamp is wall-clock, so **full ids never repeat** — match on the 12-hex suffix only | copy row verbatim (id and all) | build the target's suffix set, skip matches — this is RFC 001 §7.6 |
|
||
| **KG open facts** (`valid_to IS NULL`) | `triples.valid_to IS NULL` | server guard `WHERE subject=? AND predicate=? AND object=? AND valid_to IS NULL` returns the existing id (`knowledge_graph.py:305-311`) | pre-query same key | none: just replay |
|
||
| **KG closed facts** (`valid_to` set) | `triples.valid_to IS NOT NULL` | **no server guard at all** — the guard above is scoped to open facts, so every replay inserts a fresh row | pre-query `(s,p,o,valid_from,valid_to)` | same pre-query, client-side |
|
||
| **Entities** | `entities` table | — | needed as FK targets for triples | `kg_add` creates them implicitly |
|
||
|
||
Chunking is deterministic and replay-safe: `DEFAULT_CHUNK_SIZE = 800` (`config.py:274`), chunk ids are
|
||
`f"{drawer_id}_chunk_{i:06d}"`, so nothing about chunking needs special handling. One structural note the
|
||
census depends on: an oversized drawer has **no parent row** — only `_chunk_NNNNNN` rows — so the parent
|
||
id must be recovered by stripping the suffix, and content by concatenating in `chunk_index` order.
|
||
|
||
### 2.1 Three corrections this table needed — found by building the census, not by reading
|
||
|
||
Every recipe above is now **empirically verified**: `bin/mempalace-census` recomputes each id from
|
||
reassembled content and compares it to the stored id (176/176 accounted for on the reference palace).
|
||
Building that check falsified three things an earlier docs-only reading of `ids.py` had asserted.
|
||
|
||
**(a) The hash input is length-prefixed, not `\|`-joined.** `ids.py:31` defines `_DELIM = "|"` and the
|
||
`make_*` docstrings describe the input as `f"{wing}|{room}|{content}"`. Both are misleading: `_DELIM` is
|
||
**dead code** (defined, never referenced) and `_delimited_sha256()` actually builds
|
||
`"".join(f"{len(part)}:{part}")`. Length-prefixing is the better scheme — it is unambiguous where a bare
|
||
delimiter is not — but the docstrings never caught up. Measured on real drawers: length-prefixed
|
||
reproduces stored ids **5/5**, pipe-joined **0/5**. *Verify id recipes against the implementation and a
|
||
real row; a docstring is a claim, not evidence.*
|
||
|
||
**(b) `id_recipe` is not a mined-only marker.** It looked like a clean discriminator (14,586 rows, the
|
||
same count as `source_file`), but the server stamps `v3` on **every** v3 id including content-hashed
|
||
ones, and the sets differ by exactly the 60 agent-authored drawers. Classifying on
|
||
`source_file OR id_recipe` therefore swallowed all 60 into *mined* — the dangerous direction, since
|
||
Phase C would try to re-mine drawers that have no source file at all and silently drop them from the
|
||
join. The discriminator is a **truthy** `source_file`, cross-checked against the miner-only keys
|
||
`source_mtime` / `normalize_version`. The census now reports any disagreement between the two as a
|
||
first-class warning rather than trusting one signal.
|
||
|
||
**(c) Content ids drift, so "recompute the id and skip if present" is unsafe.** 9 of the 60
|
||
agent-authored drawers (15%) no longer reproduce their own id: `update_drawer` **preserves the id** while
|
||
rewriting and re-chunking content, so an edited drawer's content hash stops matching. This is expected
|
||
behaviour, but it breaks the obvious Regime A strategy — recomputing the content id and probing the
|
||
target for it **misses every drifted drawer and duplicates it**. Phase C must key on the **stored** id.
|
||
The census flags these as `edited_since_filing` in the manifest so Phase C can be tested against them.
|
||
|
||
### Not carried by either regime
|
||
|
||
**Hallways, `known_entities.json`, the palace graph, and closets are built at mine time**, not by
|
||
`add_drawer`. Replayed drawers therefore arrive with no hallway/co-occurrence edges and no closets, so
|
||
`list_hallways` and `traverse` will under-report for joined content, and joined drawers get no
|
||
`closet_boost` in search ranking. All are derived artifacts — rebuilt by re-mining, or acceptably
|
||
degraded. Decide which; do not discover it later.
|
||
|
||
**Closets specifically:** `mempalace_closets` is a second Chroma collection (1,560 rows in the archive,
|
||
~10% of the palace), and **there is no MCP tool that writes a closet** — `mcp_server.py` only exposes
|
||
`_purge_source_closets`. `closet_llm.py` states it plainly: *"Regex closets are always created by the
|
||
miner"*, with the LLM path an opt-in regeneration afterwards. So closets cannot be replayed even in
|
||
principle; they come back only by re-mining. Since ~99% of a devbox palace is mined content that gets
|
||
re-mined on synlig anyway (§4), this resolves itself for the bulk.
|
||
|
||
> **Census gotcha — filter by collection.** `chroma.sqlite3` holds *both* collections. A naive
|
||
> `embeddings`-wide query over-counts: the archive yields 16,338 rows total, which is
|
||
> `mempalace_drawers` 14,778 + `mempalace_closets` 1,560. Join through
|
||
> `segments`→`collections` and keep `mempalace_drawers`, or the census inflates by ~10%. (14,778 also
|
||
> reconciles exactly with the 14,777 seeded to the primary plus the one known post-snapshot chunk.)
|
||
|
||
---
|
||
|
||
## 3. Phases and deliverables
|
||
|
||
**Phase A — census (read-only, no writes anywhere).** ✅ **Built: `bin/mempalace-census`.** Point it at a
|
||
palace on disk; it enumerates every parent drawer from `chroma.sqlite3` (`embeddings ⋈ embedding_metadata`,
|
||
filtered by collection) plus every row of `knowledge_graph.sqlite3`, classifies each into the §2 rows, and
|
||
emits `--json` (a manifest that feeds Phases B/C) or a human report. It opens every file `mode=ro` and is
|
||
safe to run against a live palace. It also **self-verifies** — recomputing each id from reassembled content
|
||
and comparing to the stored id, which checks the recipe, the chunk reassembly order and the classifier in
|
||
one pass. That check is what produced the §2.1 corrections. *Deliverable, realised: see §4.1.*
|
||
|
||
**Phase B — target index.** Given the primary, build the three lookup sets Phase C needs: content-drawer
|
||
ids, diary `sha256(entry)[:12]` suffixes, and `(s,p,o,valid_from,valid_to)` tuples. Over MCP this is
|
||
`list_drawers` pagination + `kg_timeline`; on synlig it is two sqlite queries. Cheap either way.
|
||
|
||
**Phase C — writer**, with `--dry-run` as the default and an explicit `--apply`, mirroring `sync`'s
|
||
contract. Two backends behind one interface:
|
||
- `--mode direct` (regime A): stop `mempalace-serve`, `col.add()` with original ids/metadata, restart.
|
||
Model on `migrate.py`. Must refuse to run if the server is up.
|
||
- `--mode mcp` (regime B): `add_drawer`/`diary_write`/`kg_add` over HTTP, accepting `filed_at` loss.
|
||
|
||
**Phase D — verification.** Counts before/after per class, a real `search` against known joined content
|
||
(proves the HNSW index absorbed it — this is how the first seed was verified), `kg_stats`, and a spot
|
||
`get_drawer` on a known id. Plus: re-run the Phase A census against the *target* and diff.
|
||
|
||
---
|
||
|
||
## 4. Explicitly out of scope
|
||
|
||
- **Mined wings.** RFC 001 §5 makes them `local`; their ids are path-dependent, so replay produces
|
||
duplicates rather than dedup. Re-mine on synlig from sources present there.
|
||
- **Merging two palaces into a third.** Every join targets the existing primary.
|
||
- **`mempalace sync` interaction.** See RFC 001 §7.2 — settle the guard separately; a joiner must never
|
||
call it.
|
||
|
||
### 4.1 The census, run for real — the replay surface is tiny
|
||
|
||
Produced by `bin/mempalace-census` against the container-local palace (2026-08-15). Since the fleet shares
|
||
one devbox image, this is a reasonable prior for what tor-ms22 and MBP-M1-2020 hold. Counts are **parent
|
||
drawers in the `mempalace_drawers` collection** — see the correction note below, which is the whole reason
|
||
to state the unit:
|
||
|
||
| Class | Count | Share | Joiner action |
|
||
| --- | --- | --- | --- |
|
||
| Mined (`source_file` truthy) | 14,389 | **98.8%** | re-mine on synlig; **never replay** |
|
||
| Diary entries | 116 | 0.8% | replay + §7.6 suffix skip |
|
||
| Agent-authored drawers | 60 | 0.4% | replay by stored id (9 are `edited_since_filing`, §2.1c) |
|
||
| KG open facts | 34 | — | replay, server guard dedupes |
|
||
| KG **closed** facts | **0** | — | nothing to do — see below |
|
||
|
||
**Corrected figure.** An earlier pass reported 15,949 mined / 98.9%. That number was reconstructible
|
||
exactly as `16,338 (all embeddings rows) − 192 (diary rows) − 197 (agent rows) = 15,949`: it counted
|
||
**rows, not parent drawers**, and it counted them across **both collections**, so it silently absorbed all
|
||
1,560 `mempalace_closets` rows into the mined total. Closets are derived at mine time and are not joinable
|
||
at all. Two lessons, both now enforced in the census: **always filter by collection** (`embeddings ⋈
|
||
segments ⋈ collections`, since one sqlite file holds both), and **always state whether a count is rows or
|
||
parent drawers** — a chunked drawer contributes N rows and no parent row.
|
||
|
||
**Two consequences that shrink this project sharply.** First, the genuinely replay-only surface is
|
||
**176 records**, not thousands — so Phase C's writer is a small job, and the §7.6 diary guard that has
|
||
been treated as the blocker governs *116 records*. Second, **there are zero closed KG facts**, so the
|
||
unguarded-closed-fact gap (§2) is real in the code but currently empty in the data; it needs handling for
|
||
correctness, not for this join.
|
||
|
||
`filed_at` spread over the same parent drawers — 12 in May, 52 in June, 13,619 in July, 882 in August — is
|
||
the concrete case for regime A: an MCP replay would restamp all of it to the join date.
|
||
|
||
## 5. Open decisions
|
||
|
||
1. **Chronology: keep it or flatten it?** Regime A keeps it and costs a service stop plus an rsync of the
|
||
source palace onto the primary host. Regime B is simpler and loses it. This is the fork in the road.
|
||
2. **Hallways for joined content:** re-mine to rebuild, or accept degraded `traverse`?
|
||
3. **Do the secondary machines keep their palaces on persistent storage?** If a palace lives in a Docker named
|
||
volume rather than a bind mount, its un-migrated content dies on the next container recreate — so the
|
||
census (Phase A) is time-sensitive there, and flipping before censusing is risky.
|
||
4. **Does §7.6 get fixed client-side (in the joiner) or upstream (probe-and-skip in `diary_write`)?** The
|
||
joiner needs the suffix skip either way; upstream would fix it for every writer.
|
||
|
||
## 6. Effort
|
||
|
||
Phase A is the bulk of the value and is small — two sqlite readers and a classifier. Phase B is trivial.
|
||
Phase C is where the risk lives, and regime A must be written defensively (refuse on a live server,
|
||
back up first, `--dry-run` default). Phase D is mostly assertions. Nothing here needs new server code,
|
||
which is the point of RFC 001 §4.4's "existence checks available today".
|
||
|
||
---
|
||
|
||
## Appendix — verified source references
|
||
|
||
mempalace 3.6.0, `/opt/uv-tools/mempalace/lib/python3.13/site-packages/mempalace/`:
|
||
|
||
| Claim | Location |
|
||
| --- | --- |
|
||
| Content-addressed drawer id, 24 hex over `wing\|room\|content` | `ids.py:80` (`make_drawer_id_from_content`), `_HASH_TRUNC_DRAWER = 24` at `ids.py:36` |
|
||
| MCP `add_drawer` uses that recipe | `mcp_server.py:2560` |
|
||
| Miner id over `source_file\|chunk_index` | `ids.py:67` (`make_drawer_id_from_chunk`), used `miner.py:1381`, `format_miner.py:645` |
|
||
| Idempotency probe + `already_exists` (probes parent **and** last chunk) | `mcp_server.py:2593-2604` |
|
||
| `filed_at` stamped server-side, no override | `mcp_server.py:2580` |
|
||
| Diary id recipe, suffix = `sha256(entry)[:12]` | `mcp_server.py:3510-3513` |
|
||
| Triple id `t_{s}_{p}_{o}_{sha256(valid_from\|recorded_at)[:12]}` | `ids.py:111-131`, `_HASH_TRUNC_TRIPLE = 12` at `ids.py:37` |
|
||
| `add_triple` guard scoped to open facts | `knowledge_graph.py:305-311` |
|
||
| Chunk size 800 | `config.py:274` |
|
||
| Direct-write precedent preserving ids+metadata | `migrate.py:326-330` |
|
||
| No join/replay/import tooling exists | `cli.py` subcommands; `exporter.py` emits markdown (lossy, not a join primitive); `diary_ingest.py` ingests *daily-summary files*, unrelated to agent diaries; `migrate.py` is single-palace chromadb recovery; `dedup.py` is cosine-similarity pruning within one `source_file` |
|
||
| Closets are miner-derived, no MCP write path | `closet_llm.py:8-10` ("Regex closets are always created by the miner"); `mcp_server.py:2913` exposes only `_purge_source_closets` |
|
||
| Two collections in one sqlite file | `mempalace_drawers` (14,778 rows) and `mempalace_closets` (1,560) in the archive, via `segments`→`collections` |
|