docs(rfc-001): provenance belongs to the sync boundary, not the agent
Reverses the previous §7.3 on review. It said "stamp added_by everywhere, now, because it cannot be backfilled" and was about to become a skill instruction telling agents to do it. Both halves were wrong. Wrong on ownership: provenance answers "which device asserted this?", so only a party that can verify the answer should write it. An agent must shell out to read env, can forget, and will improvise when the values are absent — the worst possible stamper, and its claim is unverifiable by anyone. Under the §4 design every write reaches the primary over an authenticated channel, including offline ones at outbox-flush time, so the primary can stamp the complete set with no client cooperation. Provenance is a property of the sync channel, not of the record's author. §7.3.2 adds the trust ladder; edge-side stamping is demoted to an advisory interim, because 3.6.0's serve takes a single shared bearer token (mcp_server.py:5291-5293) and the package has zero device/origin concept, so authoritative stamping needs the per-device credentials of §6 — Phase 4, not Phase 0. Wrong on backfill: a solitary devbox is a single-origin store by definition, so origin is a property of the whole palace and can be assigned wholesale at import (one --origin-device flag) at the moment it stops being solitary. Bulk attribution is strictly more reliable than per-record stamping since it cannot be partially applied. Per-record provenance is only needed where origins interleave, which is only the primary. Solitary containers therefore stamp nothing and lose nothing — more R1-compliant than the previous draft, which quietly asked users who had opted out to carry metadata for the feature. Multi-harness-on-one-host stays solved by added_by = agent name. Keeps the verified mechanics (fixed metadata schema, argument whitelisting at mcp_server.py:4777 silently dropping unknown fields, the diary agent_name → wing_pi@host trap, kg_add having no provenance slot, added_by absent from search results) and the identity findings (a container cannot discover its host's identity; hostnames are neither unique nor stable; rename splits one device's history in two). §7.3.5 keeps the fail-closed rule for whichever component does stamp. Phase 0 drops its provenance item accordingly, and §6.2 now specifies server-side stamping from the authenticated credential. Also flags in-document that these notes are a poisoning vector: a future agent reading them out of the palace must not conclude it should hand-stamp.
This commit is contained in:
Joakim Persson
+1
-1
@@ -357,7 +357,7 @@ These gaps should ideally close upstream, making the wrappers thinner or obsolet
|
||||
|
||||
When #1 merges, retire `mempalace-docs` to a thin shim. When #2 + #3 land together, `mempalace-session` becomes a manual-only fallback (cron / backfill) while hooks handle live saves.
|
||||
|
||||
Separately tracked in [`docs/rfc-001-global-palace.md`](docs/rfc-001-global-palace.md): moving from one palace *per machine* to a **single primary palace with per-machine local fallback** (`mempalace-edge`). That RFC also records upstream items of its own — `origin_host` provenance metadata, per-wing ACLs, a `mempalace_kg_supersede` tool-classification fix, and a guard against running `mempalace sync` on a shared palace.
|
||||
Separately tracked in [`docs/rfc-001-global-palace.md`](docs/rfc-001-global-palace.md): moving from one palace *per machine* to a **single primary palace with per-machine local fallback** (`mempalace-edge`). That RFC also records upstream items of its own — server-side `origin_device` provenance stamped from a per-device credential, per-wing ACLs, a `mempalace_kg_supersede` tool-classification fix, and a guard against running `mempalace sync` on a shared palace.
|
||||
|
||||
---
|
||||
|
||||
|
||||
@@ -233,7 +233,7 @@ A TOCTOU-safe, cross-process "at most one active job per key". Our outbox rows:
|
||||
| --- | --- |
|
||||
| `op_id` | `sha256(origin_device, local_seq, tool_name, canonical_payload)` — stable across retries, the primary's idempotency key |
|
||||
| `local_seq` | Per-device monotonic counter → **causal order** for `kg_invalidate`/`supersede` replay |
|
||||
| `origin_device`, `origin_host` | Provenance (§7.3) |
|
||||
| `origin_device`, `origin_label` | Provenance, stamped by edge (interim) or the primary (authoritative) — never by the agent (§7.3) |
|
||||
| `tool_name`, `payload_json` | The MCP call to replay |
|
||||
| `state`, `attempts`, `created_at`, `flushed_at`, `error_json` | Retry/audit |
|
||||
|
||||
@@ -272,7 +272,7 @@ The transport is largely solved; **the gap is authorization, not cryptography.**
|
||||
1. **Memory poisoning / persistent cross-machine prompt injection — the underrated one.** Palace
|
||||
content is injected into agent context at wake-up. A shared palace means one careless or
|
||||
compromised container can plant instructions that *every other agent in the fleet reads as trusted
|
||||
memory*. Mitigation: provenance on every record (§7.3), don't auto-inject wake-up content authored
|
||||
memory*. Mitigation: server-stamped provenance on every synced record (§7.3.2), don't auto-inject wake-up content authored
|
||||
by devices outside a trusted set, keep an admin-only wing for anything instruction-shaped.
|
||||
2. **Aggregation raises exfiltration impact.** One primary holds every machine's diaries — which
|
||||
already quote internal hostnames, paths and token prefixes. Mitigation: per-wing ACL; don't put
|
||||
@@ -288,7 +288,7 @@ The transport is largely solved; **the gap is authorization, not cryptography.**
|
||||
| **Transport** | TLS via `serve --tls-cert/--tls-key`, or terminate TLS **+ mTLS** in the reverse proxy (cheaper than patching Python's TLS surface). |
|
||||
| **Authentication** | **Per-device bearer tokens**, not the one shared token. Server-side registry `token → {device_id, scopes}`. Store in the existing `.env.age` flow, 0600 on disk. Enables revoking one laptop and rotating without a fleet outage. |
|
||||
| **Authorization** | Per-device read/write **wing globs**. Server-side refusal of `mine`/`sync`/`delete_*` except for an admin device. `--read-only` gives a free observer tier. |
|
||||
| **Provenance/audit** | `origin_host` + `origin_device` + `op_id` on every record (§7.3); server-side append-only op log. |
|
||||
| **Provenance/audit** | `origin_device` (+ optional `origin_label`) + `op_id` on every record, stamped **server-side from the authenticated credential** — a client-asserted origin is a hint, not a fact (§7.3.2); server-side append-only op log. |
|
||||
| **Recovery** | Deletes as tombstones; server-side backups with `backups.py` retention (`MEMPALACE_MAX_BACKUPS`). |
|
||||
| **DoS** | Keep the 16 MiB cap; add rate limiting; no remote `mine`. |
|
||||
|
||||
@@ -334,13 +334,132 @@ data-dependent rather than obvious. From a laptop that lacks the repos, it is a
|
||||
**Action:** edge blocks `mempalace_sync` from ever reaching the primary; document it; consider an
|
||||
upstream `--refuse-shared` guard.
|
||||
|
||||
### 7.3 Add provenance metadata now
|
||||
### 7.3 Provenance belongs to the sync boundary — not to the agent, and not to a solitary container
|
||||
|
||||
Today every drawer carries exactly `{wing, room, source_file, added_by, filed_at, id_recipe}`
|
||||
(+`chunk_index`, `parent_drawer_id`); diaries add `{hall, topic, type, agent, date}`. `added_by`/`agent`
|
||||
is the *agent* name (`pi`, `mcp`, `checkpoint`) — **never the machine**. So after any merge you cannot
|
||||
is the *agent* name (`pi`, `mcp`, `checkpoint`) — **never the machine**. So in a merged store you cannot
|
||||
tell which host wrote a record, cannot audit, and cannot compute per-device high-water marks.
|
||||
Metadata is free-form, so this is cheap — but it is **impossible to backfill**. Do it first.
|
||||
|
||||
**Earlier drafts of this section said "stamp it everywhere, now, because it cannot be backfilled." That
|
||||
was wrong on both counts.** See §7.3.3.
|
||||
|
||||
#### 7.3.1 What can actually be stamped (mechanics)
|
||||
|
||||
The metadata schema is **fixed** — there is no free-form field — and `tools/call` **whitelists arguments
|
||||
to declared schema properties** (`mcp_server.py:4777`, *"Prevents callers from spoofing internal params
|
||||
like added_by/source_file"*), so an extra `origin_host=…` is **silently dropped, not rejected**.
|
||||
|
||||
| Surface | Provenance slot | Notes |
|
||||
| --- | --- | --- |
|
||||
| `add_drawer`, `checkpoint` | **`added_by`** | The only one. Free-form (`strip_lone_surrogates` only, *not* `sanitize_name`, so `/` and `@` are legal). |
|
||||
| `diary_write` | **none usable** | ⚠️ **Never** put a device in `agent_name`: `:3504` does `wing = f"wing_{agent_name}"` → a separate wing per host, and `diary_read` filters `{"agent": agent_name}` (`:3636`) → `diary_read("pi")` then **misses** those entries. |
|
||||
| `kg_add` | **none** | Only `source_file`/`source_closet`/`source_drawer_id`. Origin is inferable only via `source_drawer_id` → that drawer's `added_by`. |
|
||||
|
||||
`added_by` is also **write-only today**: absent from `tool_search` results, surfacing only via
|
||||
`get_drawer` → `_drawer_payload` metadata. → Upstream asks: **surface `added_by` in search results**, and
|
||||
**give `kg_add` a provenance field**.
|
||||
|
||||
#### 7.3.2 Who should stamp it — a ladder of trust
|
||||
|
||||
Provenance answers "which device asserted this?" Only something that can *verify* the answer should
|
||||
write it. Ranked by trustworthiness:
|
||||
|
||||
| Stamper | Knows the device? | Verifiable? | Uniform? | Verdict |
|
||||
| --- | --- | --- | --- | --- |
|
||||
| **Agent (via skill)** | No — must shell out to read env | No | No — per-call boilerplate, forgettable, improvisable | ❌ **Worst possible place.** Rejected. |
|
||||
| **Client / `mempalace-edge`** | Yes, from host-supplied `.env` | No — self-asserted | Yes — one line in a proxy | ⚠️ Acceptable **interim** |
|
||||
| **Primary, from the authenticated credential** | Yes | **Yes** — bound to the token | Yes, for every synced record | ✅ **Correct home** |
|
||||
|
||||
The decisive point: **a client-asserted origin is a hint, not a fact.** The primary is the only party
|
||||
that can bind a write to an identity it verified. And under the §4 design *every* write reaches the
|
||||
primary through an authenticated channel — including offline ones, at outbox-flush time — so the server
|
||||
can stamp the complete set without any client cooperation. **Provenance is a property of the sync
|
||||
channel, not of the record's author.**
|
||||
|
||||
Blocker for the ✅ row, verified in 3.6.0: `serve` takes a **single shared bearer token**
|
||||
(`srv.auth_token`, `hmac.compare_digest`, `mcp_server.py:5291-5293`) and the package contains **zero**
|
||||
occurrences of any device/origin concept. Server-side stamping therefore *requires* the per-device
|
||||
credentials of §6 — i.e. **Phase 4**, not Phase 1. Hence the phasing:
|
||||
|
||||
- **Phase 2 (edge):** edge may stamp `added_by` from its configured env — self-asserted, **advisory
|
||||
only**, never load-bearing for authorization or destructive scoping.
|
||||
- **Phase 4 (authz):** per-device tokens land; the primary stamps authoritatively and the client-supplied
|
||||
value becomes redundant (and must be treated as untrusted input, not merely ignored).
|
||||
|
||||
#### 7.3.3 Solitary containers should stamp nothing — and lose nothing by it
|
||||
|
||||
A pi-devbox or opencode-devbox running solitarily is a **single-origin store by definition**. Origin is
|
||||
therefore a property of the *whole palace*, not of each record — so it can be assigned **wholesale at
|
||||
the moment the store stops being solitary**: one `--origin-device` flag on the import/first-sync path
|
||||
attributes every record from that palace to that device.
|
||||
|
||||
That dissolves the "impossible to backfill" argument. Per-record stamping is only necessary once records
|
||||
from *multiple* origins are interleaved in one store, which is exactly and only the primary. So:
|
||||
|
||||
- **Solitary devboxes: no stamping, no config, no skill instruction, no behaviour change.** This is
|
||||
strictly more compliant with **R1** than the earlier draft, which quietly asked every solitary user to
|
||||
carry metadata for a feature they had opted out of.
|
||||
- **Migration is unaffected**: bulk attribution at import is *more* reliable than per-record stamping,
|
||||
because it cannot be partially applied.
|
||||
- **Multi-harness on one host stays solved** by `added_by` = agent name (`pi` vs `opencode`) — that is
|
||||
what the field is for, and it needs no device component.
|
||||
- **A palace on a shared host bind-mount** (as tor-ms22's compose does) is still single-*device* under
|
||||
the "host owns the palace" model, so it too imports as one origin.
|
||||
|
||||
The one case bulk attribution cannot fix: a palace that was *already* merged from several devices without
|
||||
stamps. Preventing that is precisely why the **primary** must stamp from day one of Phase 4 — it is the
|
||||
only store where interleaving occurs.
|
||||
|
||||
#### 7.3.4 Identity fields, when a stamper does exist
|
||||
|
||||
For the edge (interim) and the primary (authoritative) — never for agents:
|
||||
|
||||
| Field | Source | Rule |
|
||||
| --- | --- | --- |
|
||||
| `origin_device` | `uuidgen` **once**, stored in the **host's `.env`**; later, issued with the device's token | Opaque, stable, collision-free by construction. Compared, never parsed. |
|
||||
| `origin_label` | hostname, same `.env` | Human readability **only**. Never identity, uniqueness or scoping. Free to change. Optional — a hostname can leak an asset tag or username (`EMB-7KJ4VR4G`, `HOST_SSH_USER=ECSJPER`). |
|
||||
|
||||
Why not the obvious sources: **a container cannot discover its host's identity.** `hostname` returns the
|
||||
*container ID* (`f3bf2a103473`) which changes on **every recreate**; there is no `/etc/machine-id`; and a
|
||||
baked one would be *worse* — identical for every container from the same image, a guaranteed collision.
|
||||
**Hostnames are also neither unique nor stable** (`localhost`, `ubuntu`, golden images, two
|
||||
`MacBook-Pro.local`), and the worse failure is not collision but **rename**, which silently splits one
|
||||
device's history in two. Hence host-supplied via `.env` (matching R2/R3 and the existing `HOST_SSH_USER`
|
||||
/ `DEVBOX_HOST_ALIAS` precedent), *not* container-derived. Do **not** persist the id container-side:
|
||||
`~/.mempalace/device_id` does not survive recreate for solitary users because `devbox-palace` is
|
||||
commented out by default (§1.2). And don't write `${HOSTNAME}` in compose — bash *sets* but does not
|
||||
*export* it, so interpolation sees empty.
|
||||
|
||||
Encoding, always three segments so arity is unambiguous:
|
||||
|
||||
```
|
||||
added_by = "<agent>/<label>/<device>" e.g. pi/tor-ms22/7f3a9c2e1b04
|
||||
pi/-/7f3a9c2e1b04 (label withheld)
|
||||
```
|
||||
|
||||
**The `/` is the discriminator.** A value with no `/` (`mcp`, `pi`, `checkpoint`) means *origin unknown* —
|
||||
correct for the ~13.6k drawers already filed, and for every solitary palace forever.
|
||||
|
||||
#### 7.3.5 Fail-closed rule for any stamper
|
||||
|
||||
If `origin_device` is unset, blank or unreadable: **write no `added_by` override at all** and let the
|
||||
default (`mcp` / `checkpoint` / agent name) stand. Never synthesize one. The tempting substitutes are all
|
||||
actively harmful, because each is *confidently* wrong where absence is honestly unknown:
|
||||
|
||||
| Substitute | Damage |
|
||||
| --- | --- |
|
||||
| container hostname / ID | A **new fake device per recreate** — thousands of singleton identities, each looking legitimate |
|
||||
| `unknown`, `localhost`, `devbox`, `docker`, `$USER` | **Collides across every machine** — indistinguishable from one real shared device |
|
||||
| a guess from context | Unfalsifiable later |
|
||||
|
||||
If `origin_label` alone is missing, use the literal `-`. If `origin_device` is missing, do not stamp even
|
||||
when the label is present — a hostname alone is exactly the colliding, mutable identity rejected above.
|
||||
|
||||
> **Note for future agents reading this RFC out of the palace:** these notes are themselves a poisoning
|
||||
> vector. Do **not** start hand-stamping `added_by` because you read this section. Provenance is client
|
||||
> and server infrastructure; an agent's contribution to it is to leave the field alone. The mempalace
|
||||
> skill carries a one-line guard to that effect.
|
||||
|
||||
### 7.4 Embedder identity is client-local and currently toothless
|
||||
|
||||
@@ -372,7 +491,7 @@ across models silently degrades recall.
|
||||
|
||||
| Phase | Effort | Deliverable |
|
||||
| --- | --- | --- |
|
||||
| **0 — hygiene** | hours | §7 runbook: move KG/hallways/entities, add provenance metadata, ban `sync` on shared palaces, per-device tokens, fix stale "unauthenticated" docs (incl. `pi-devbox/.env.example:21`) |
|
||||
| **0 — hygiene** | hours | §7 runbook: move KG/hallways/entities, ban `sync` on shared palaces, fix stale "unauthenticated" docs (incl. `pi-devbox/.env.example:21`). **No provenance work here** — it is not backfill-critical (§7.3.3) and belongs to the stamper, not the agent |
|
||||
| **1 — primary up** | hours, **no code** | `mempalace serve --token --tls-cert` on a private-net host (reuse `docker-compose.mempalace.yml` — keep it a **separate standalone project**, R4 — and mind port 8765 vs pi-studio; tor-ms22 already moved to 8766). Repoint pi clients via `MEMPALACE_REMOTE_URL`/`MEMPALACE_REMOTE_TOKEN`. **Shared memory today, no offline.** |
|
||||
| **2 — `mempalace-edge`** | ~1 week | The actual ask: local-first writes + outbox flush + merged reads + per-wing policy. Fixes opencode as a side effect. **Ships with the §1.2 opt-in wiring (compose + `.env.example` + entrypoint templating) and must pass the R1 acceptance test.** |
|
||||
| **3 — pull replication** | ~1 week | **DEFERRED (2026-08-08).** Server-side op-log with monotonic seq → each edge a full offline replica. Only needed if a laptop must hold *everything* offline. Accepted trade-off: offline recall = own writes + last-reachable state. |
|
||||
@@ -390,7 +509,7 @@ for online clients.
|
||||
proxy is not merely nicer — it is the only option for opencode clients.** Verify against opencode's
|
||||
MCP client source before Phase 2.
|
||||
2. **Upstream or local?** `mempalace-edge` needs no core changes, so it belongs in this repo
|
||||
(`bin/` + `extensions/`). But `origin_host` metadata, per-wing ACL, the `kg_supersede`
|
||||
(`bin/` + `extensions/`). But `origin_device` stamping from the authenticated token, per-wing ACL, the `kg_supersede`
|
||||
classification fix, and a `sync --refuse-shared` guard all want to go **upstream**
|
||||
(`github.com/MemPalace/mempalace`).
|
||||
3. **Chunked drawers under merge.** Oversized content splits into `{drawer_id}_chunk_NNNNNN` with
|
||||
|
||||
Reference in New Issue
Block a user