From 052dbb80387085f922081a3ab88adaeca688d17d Mon Sep 17 00:00:00 2001 From: Joakim Persson Date: Sun, 9 Aug 2026 15:40:25 +0200 Subject: [PATCH] docs(rfc-001): provenance belongs to the sync boundary, not the agent MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Reverses the previous §7.3 on review. It said "stamp added_by everywhere, now, because it cannot be backfilled" and was about to become a skill instruction telling agents to do it. Both halves were wrong. Wrong on ownership: provenance answers "which device asserted this?", so only a party that can verify the answer should write it. An agent must shell out to read env, can forget, and will improvise when the values are absent — the worst possible stamper, and its claim is unverifiable by anyone. Under the §4 design every write reaches the primary over an authenticated channel, including offline ones at outbox-flush time, so the primary can stamp the complete set with no client cooperation. Provenance is a property of the sync channel, not of the record's author. §7.3.2 adds the trust ladder; edge-side stamping is demoted to an advisory interim, because 3.6.0's serve takes a single shared bearer token (mcp_server.py:5291-5293) and the package has zero device/origin concept, so authoritative stamping needs the per-device credentials of §6 — Phase 4, not Phase 0. Wrong on backfill: a solitary devbox is a single-origin store by definition, so origin is a property of the whole palace and can be assigned wholesale at import (one --origin-device flag) at the moment it stops being solitary. Bulk attribution is strictly more reliable than per-record stamping since it cannot be partially applied. Per-record provenance is only needed where origins interleave, which is only the primary. Solitary containers therefore stamp nothing and lose nothing — more R1-compliant than the previous draft, which quietly asked users who had opted out to carry metadata for the feature. Multi-harness-on-one-host stays solved by added_by = agent name. Keeps the verified mechanics (fixed metadata schema, argument whitelisting at mcp_server.py:4777 silently dropping unknown fields, the diary agent_name → wing_pi@host trap, kg_add having no provenance slot, added_by absent from search results) and the identity findings (a container cannot discover its host's identity; hostnames are neither unique nor stable; rename splits one device's history in two). §7.3.5 keeps the fail-closed rule for whichever component does stamp. Phase 0 drops its provenance item accordingly, and §6.2 now specifies server-side stamping from the authenticated credential. Also flags in-document that these notes are a poisoning vector: a future agent reading them out of the palace must not conclude it should hand-stamp. --- ARCHITECTURE.md | 2 +- docs/rfc-001-global-palace.md | 135 ++++++++++++++++++++++++++++++++-- 2 files changed, 128 insertions(+), 9 deletions(-) diff --git a/ARCHITECTURE.md b/ARCHITECTURE.md index c359fa3..1b0cab8 100644 --- a/ARCHITECTURE.md +++ b/ARCHITECTURE.md @@ -357,7 +357,7 @@ These gaps should ideally close upstream, making the wrappers thinner or obsolet When #1 merges, retire `mempalace-docs` to a thin shim. When #2 + #3 land together, `mempalace-session` becomes a manual-only fallback (cron / backfill) while hooks handle live saves. -Separately tracked in [`docs/rfc-001-global-palace.md`](docs/rfc-001-global-palace.md): moving from one palace *per machine* to a **single primary palace with per-machine local fallback** (`mempalace-edge`). That RFC also records upstream items of its own — `origin_host` provenance metadata, per-wing ACLs, a `mempalace_kg_supersede` tool-classification fix, and a guard against running `mempalace sync` on a shared palace. +Separately tracked in [`docs/rfc-001-global-palace.md`](docs/rfc-001-global-palace.md): moving from one palace *per machine* to a **single primary palace with per-machine local fallback** (`mempalace-edge`). That RFC also records upstream items of its own — server-side `origin_device` provenance stamped from a per-device credential, per-wing ACLs, a `mempalace_kg_supersede` tool-classification fix, and a guard against running `mempalace sync` on a shared palace. --- diff --git a/docs/rfc-001-global-palace.md b/docs/rfc-001-global-palace.md index e3ae989..911582d 100644 --- a/docs/rfc-001-global-palace.md +++ b/docs/rfc-001-global-palace.md @@ -233,7 +233,7 @@ A TOCTOU-safe, cross-process "at most one active job per key". Our outbox rows: | --- | --- | | `op_id` | `sha256(origin_device, local_seq, tool_name, canonical_payload)` — stable across retries, the primary's idempotency key | | `local_seq` | Per-device monotonic counter → **causal order** for `kg_invalidate`/`supersede` replay | -| `origin_device`, `origin_host` | Provenance (§7.3) | +| `origin_device`, `origin_label` | Provenance, stamped by edge (interim) or the primary (authoritative) — never by the agent (§7.3) | | `tool_name`, `payload_json` | The MCP call to replay | | `state`, `attempts`, `created_at`, `flushed_at`, `error_json` | Retry/audit | @@ -272,7 +272,7 @@ The transport is largely solved; **the gap is authorization, not cryptography.** 1. **Memory poisoning / persistent cross-machine prompt injection — the underrated one.** Palace content is injected into agent context at wake-up. A shared palace means one careless or compromised container can plant instructions that *every other agent in the fleet reads as trusted - memory*. Mitigation: provenance on every record (§7.3), don't auto-inject wake-up content authored + memory*. Mitigation: server-stamped provenance on every synced record (§7.3.2), don't auto-inject wake-up content authored by devices outside a trusted set, keep an admin-only wing for anything instruction-shaped. 2. **Aggregation raises exfiltration impact.** One primary holds every machine's diaries — which already quote internal hostnames, paths and token prefixes. Mitigation: per-wing ACL; don't put @@ -288,7 +288,7 @@ The transport is largely solved; **the gap is authorization, not cryptography.** | **Transport** | TLS via `serve --tls-cert/--tls-key`, or terminate TLS **+ mTLS** in the reverse proxy (cheaper than patching Python's TLS surface). | | **Authentication** | **Per-device bearer tokens**, not the one shared token. Server-side registry `token → {device_id, scopes}`. Store in the existing `.env.age` flow, 0600 on disk. Enables revoking one laptop and rotating without a fleet outage. | | **Authorization** | Per-device read/write **wing globs**. Server-side refusal of `mine`/`sync`/`delete_*` except for an admin device. `--read-only` gives a free observer tier. | -| **Provenance/audit** | `origin_host` + `origin_device` + `op_id` on every record (§7.3); server-side append-only op log. | +| **Provenance/audit** | `origin_device` (+ optional `origin_label`) + `op_id` on every record, stamped **server-side from the authenticated credential** — a client-asserted origin is a hint, not a fact (§7.3.2); server-side append-only op log. | | **Recovery** | Deletes as tombstones; server-side backups with `backups.py` retention (`MEMPALACE_MAX_BACKUPS`). | | **DoS** | Keep the 16 MiB cap; add rate limiting; no remote `mine`. | @@ -334,13 +334,132 @@ data-dependent rather than obvious. From a laptop that lacks the repos, it is a **Action:** edge blocks `mempalace_sync` from ever reaching the primary; document it; consider an upstream `--refuse-shared` guard. -### 7.3 Add provenance metadata now +### 7.3 Provenance belongs to the sync boundary — not to the agent, and not to a solitary container Today every drawer carries exactly `{wing, room, source_file, added_by, filed_at, id_recipe}` (+`chunk_index`, `parent_drawer_id`); diaries add `{hall, topic, type, agent, date}`. `added_by`/`agent` -is the *agent* name (`pi`, `mcp`, `checkpoint`) — **never the machine**. So after any merge you cannot +is the *agent* name (`pi`, `mcp`, `checkpoint`) — **never the machine**. So in a merged store you cannot tell which host wrote a record, cannot audit, and cannot compute per-device high-water marks. -Metadata is free-form, so this is cheap — but it is **impossible to backfill**. Do it first. + +**Earlier drafts of this section said "stamp it everywhere, now, because it cannot be backfilled." That +was wrong on both counts.** See §7.3.3. + +#### 7.3.1 What can actually be stamped (mechanics) + +The metadata schema is **fixed** — there is no free-form field — and `tools/call` **whitelists arguments +to declared schema properties** (`mcp_server.py:4777`, *"Prevents callers from spoofing internal params +like added_by/source_file"*), so an extra `origin_host=…` is **silently dropped, not rejected**. + +| Surface | Provenance slot | Notes | +| --- | --- | --- | +| `add_drawer`, `checkpoint` | **`added_by`** | The only one. Free-form (`strip_lone_surrogates` only, *not* `sanitize_name`, so `/` and `@` are legal). | +| `diary_write` | **none usable** | ⚠️ **Never** put a device in `agent_name`: `:3504` does `wing = f"wing_{agent_name}"` → a separate wing per host, and `diary_read` filters `{"agent": agent_name}` (`:3636`) → `diary_read("pi")` then **misses** those entries. | +| `kg_add` | **none** | Only `source_file`/`source_closet`/`source_drawer_id`. Origin is inferable only via `source_drawer_id` → that drawer's `added_by`. | + +`added_by` is also **write-only today**: absent from `tool_search` results, surfacing only via +`get_drawer` → `_drawer_payload` metadata. → Upstream asks: **surface `added_by` in search results**, and +**give `kg_add` a provenance field**. + +#### 7.3.2 Who should stamp it — a ladder of trust + +Provenance answers "which device asserted this?" Only something that can *verify* the answer should +write it. Ranked by trustworthiness: + +| Stamper | Knows the device? | Verifiable? | Uniform? | Verdict | +| --- | --- | --- | --- | --- | +| **Agent (via skill)** | No — must shell out to read env | No | No — per-call boilerplate, forgettable, improvisable | ❌ **Worst possible place.** Rejected. | +| **Client / `mempalace-edge`** | Yes, from host-supplied `.env` | No — self-asserted | Yes — one line in a proxy | ⚠️ Acceptable **interim** | +| **Primary, from the authenticated credential** | Yes | **Yes** — bound to the token | Yes, for every synced record | ✅ **Correct home** | + +The decisive point: **a client-asserted origin is a hint, not a fact.** The primary is the only party +that can bind a write to an identity it verified. And under the §4 design *every* write reaches the +primary through an authenticated channel — including offline ones, at outbox-flush time — so the server +can stamp the complete set without any client cooperation. **Provenance is a property of the sync +channel, not of the record's author.** + +Blocker for the ✅ row, verified in 3.6.0: `serve` takes a **single shared bearer token** +(`srv.auth_token`, `hmac.compare_digest`, `mcp_server.py:5291-5293`) and the package contains **zero** +occurrences of any device/origin concept. Server-side stamping therefore *requires* the per-device +credentials of §6 — i.e. **Phase 4**, not Phase 1. Hence the phasing: + +- **Phase 2 (edge):** edge may stamp `added_by` from its configured env — self-asserted, **advisory + only**, never load-bearing for authorization or destructive scoping. +- **Phase 4 (authz):** per-device tokens land; the primary stamps authoritatively and the client-supplied + value becomes redundant (and must be treated as untrusted input, not merely ignored). + +#### 7.3.3 Solitary containers should stamp nothing — and lose nothing by it + +A pi-devbox or opencode-devbox running solitarily is a **single-origin store by definition**. Origin is +therefore a property of the *whole palace*, not of each record — so it can be assigned **wholesale at +the moment the store stops being solitary**: one `--origin-device` flag on the import/first-sync path +attributes every record from that palace to that device. + +That dissolves the "impossible to backfill" argument. Per-record stamping is only necessary once records +from *multiple* origins are interleaved in one store, which is exactly and only the primary. So: + +- **Solitary devboxes: no stamping, no config, no skill instruction, no behaviour change.** This is + strictly more compliant with **R1** than the earlier draft, which quietly asked every solitary user to + carry metadata for a feature they had opted out of. +- **Migration is unaffected**: bulk attribution at import is *more* reliable than per-record stamping, + because it cannot be partially applied. +- **Multi-harness on one host stays solved** by `added_by` = agent name (`pi` vs `opencode`) — that is + what the field is for, and it needs no device component. +- **A palace on a shared host bind-mount** (as tor-ms22's compose does) is still single-*device* under + the "host owns the palace" model, so it too imports as one origin. + +The one case bulk attribution cannot fix: a palace that was *already* merged from several devices without +stamps. Preventing that is precisely why the **primary** must stamp from day one of Phase 4 — it is the +only store where interleaving occurs. + +#### 7.3.4 Identity fields, when a stamper does exist + +For the edge (interim) and the primary (authoritative) — never for agents: + +| Field | Source | Rule | +| --- | --- | --- | +| `origin_device` | `uuidgen` **once**, stored in the **host's `.env`**; later, issued with the device's token | Opaque, stable, collision-free by construction. Compared, never parsed. | +| `origin_label` | hostname, same `.env` | Human readability **only**. Never identity, uniqueness or scoping. Free to change. Optional — a hostname can leak an asset tag or username (`EMB-7KJ4VR4G`, `HOST_SSH_USER=ECSJPER`). | + +Why not the obvious sources: **a container cannot discover its host's identity.** `hostname` returns the +*container ID* (`f3bf2a103473`) which changes on **every recreate**; there is no `/etc/machine-id`; and a +baked one would be *worse* — identical for every container from the same image, a guaranteed collision. +**Hostnames are also neither unique nor stable** (`localhost`, `ubuntu`, golden images, two +`MacBook-Pro.local`), and the worse failure is not collision but **rename**, which silently splits one +device's history in two. Hence host-supplied via `.env` (matching R2/R3 and the existing `HOST_SSH_USER` +/ `DEVBOX_HOST_ALIAS` precedent), *not* container-derived. Do **not** persist the id container-side: +`~/.mempalace/device_id` does not survive recreate for solitary users because `devbox-palace` is +commented out by default (§1.2). And don't write `${HOSTNAME}` in compose — bash *sets* but does not +*export* it, so interpolation sees empty. + +Encoding, always three segments so arity is unambiguous: + +``` +added_by = "/