mempalace-toolkit/docs/rfc-003-coordination-log.md

RFC 003 — The coordination log (logstream)

Status: implemented and in production use since 3.7.x. This document is a retrospective specification, written after the fact. Author: pi (agent), 2026-08-26. Applies to: mempalace 3.8.0 (logstream.py, 1261 lines), mempalace-toolkit 5b8d78f (the pi-side mailbox). Context: every mempalace_event_* and mempalace_artifact_* tool description cites "RFC 003". logstream.py's module docstring is headed "Agent coordination event log for MemPalace (RFC 003)" and enumerates five "Design constraints (RFC 003)". Inline comments cite "RFC 003 phase 5", "RFC 003 suggested defaults" and "the first RFC 003 dogfood". No such document has ever existed — verified 2026-08-26 by searching both the toolkit repository and the primary host. This RFC transcribes the spec the implementation already believes in, and — more usefully — records what it does not do.

Read §7 before you build anything on this log; it is the part that is not obvious from the tool descriptions. Every mechanical claim below cites file.py:LINE in mempalace 3.8.0, and §10 indexes them so any claim can be re-verified without re-reading 1261 lines. Claims marked measured were executed against the live fleet on the date given; claims without that marker are reads of the source. Where the code and the shipped tool descriptions disagree, §10.1 says so explicitly rather than quietly siding with one.

One scoping note up front. This RFC covers the log: its storage, its append and query semantics, delivery, and the trust model. It does not specify replication, which the source already attributes to a different document (# ── Replication (RFC 004 step 0: logstream multi-master) ──, logstream.py:1096). RFC 004 does not exist either; §8.2 states what it owes.

---

1. The problem

The palace stores what an agent knows: drawers are semantic, retrieved by meaning, and deliberately have no addressee. That shape is wrong for four things a fleet of agents actually needs:

1. Addressing. "This is for the machine that owns the release" cannot be expressed as a drawer. A drawer is found by whoever happens to search for the right words.
2. A reply that closes something. Semantic memory has no notion of an outstanding question. Nothing in a drawer can be owed.
3. Exact payloads. A unified diff must survive byte-for-byte. Drawers are chunked and embedded; that is a feature for prose and a defect for patches.
4. Order. "What happened after this?" needs an append cursor, not a similarity score.

The coordination log adds exactly those four properties and nothing else. It is a second store beside the palace, not a new kind of drawer.

1.1 What it is not

Stated first, because every misuse of this log so far has come from assuming one of these:

• Not a bus. Nothing subscribes by default; nothing is delivered "live" unless a client is holding an SSE connection or polling. There is no delivery window and nothing is lost by being offline when an event is written.
• Not a chat channel. Latency is bounded by when the recipient next runs, which in a fleet of workstations is hours, days or weeks. §3.5 gives the measured numbers for the case where the recipient is awake.
• Not authenticated per agent. from_agent is a routing label with the trust properties of an e-mail From: header (§6).
• Not a queue. Nothing is consumed, acknowledged-and-removed, or retried. Events are permanent (§9.1) and "handled" is a derived property (§3.3).

---

2. Verified starting point (2026-08-26, mempalace 3.8.0)

Piece	State	Evidence
Separate SQLite store, inside the palace dir	✅ logstream.sqlite3 beside chroma.sqlite3; WAL; dir chmod 0700 best-effort	logstream.py:51, mcp_server.py:1108-1117, logstream.py:430-433
events, artifacts, event_artifacts tables	✅ Created idempotently at open	logstream.py:447-497
Hybrid logical clock on every event	✅ hlc populated on every local append	logstream.py:668, hlc.py:1-21
Seven MCP tools (append/list/wait/ack, artifact put/get, patch_submit)	✅	mcp_server.py:4546-4771
Server-sent events push	✅ GET /logstream/stream, auth required, 15 s heartbeat, ≤8 clients	mcp_server.py:7570-7671, 7189-7194
GET /logstream/events	❌ Never implemented. Only /logstream/stream exists	see §7.8
Auto-delivered mailbox (owed-set derivation + injection)	✅ Client-side, not server-side — lives in the toolkit's pi extension	extensions/pi/mempalace.ts (toolkit 5b8d78f)
Idempotency guard on append	❌ Absent. See §7.1	logstream.py:625-716
Retention / TTL / compaction	❌ Absent by design-so-far. See §9.1	no DELETE FROM events in the package
Interaction with mempalace sync	✅ None. sync prunes drawers only	cli.py:1057, sync.py (no logstream references)

Two things that sound like this feature and are not:

• mempalace_mesh_peers is not a fleet roster. It reports peer replicas. A hub-and-spoke deployment — many thin MCP clients of one server — correctly reports peers: [] while every machine in the fleet is actively writing to the same log. Deciding "coordination does not apply to me" from an empty peer list is a measured failure mode, not a hypothetical one.
• origin_replica is not the writing machine. It is a property of the palace directory (§3.2).

---

3. Design

The five constraints in the module docstring (logstream.py:1-18) are the design, quoted verbatim because they are already normative in the implementation:

• No Chroma dependency, no vector index open — plain SQLite only.
• Append-only: events are immutable; corrections are new events that reference prior events.
• Exact payloads: event bodies and artifact content are stored verbatim.
• Safe under concurrent HTTP requests (WAL + per-instance lock, same pattern as knowledge_graph.py).
• Explicit size limits with clear errors, never silent truncation.

The shape, in the same idiom as RFC 001 §4.1:

agent on device A ─┐                        palace directory
agent on device B ─┼── MCP /mcp ──► server ─┬─ chroma.sqlite3          (drawers: what you know)
agent on device C ─┘        │               ├─ knowledge_graph.sqlite3 (facts, temporal)
                            │               └─ logstream.sqlite3       (events + artifacts)
                     GET /logstream/stream        events        ── append-only, permanent
                     (SSE, auth, ≤8 clients)      artifacts     ── verbatim, ≤4 MiB, sha256
                                                  event_artifacts ── join, checked at append

3.1 Data model

CREATE TABLE IF NOT EXISTS events (
    id TEXT PRIMARY KEY,
    type TEXT NOT NULL,
    stream TEXT NOT NULL,
    room TEXT NOT NULL,
    from_agent TEXT NOT NULL,
    to_agent TEXT,
    correlation_id TEXT,
    branch TEXT,
    base_commit TEXT,
    status TEXT,
    body TEXT NOT NULL DEFAULT '',
    created_at TEXT NOT NULL,
    metadata_json TEXT NOT NULL DEFAULT '{}',
    origin_replica TEXT,
    origin_seq INTEGER,
    hlc TEXT
);
CREATE UNIQUE INDEX IF NOT EXISTS events_origin_seq_idx ON events(origin_replica, origin_seq);

(logstream.py:447-497, unique index at 552-556. artifacts and event_artifacts in the same script; artifacts_sha256_idx is not unique — see §3.4.)

Validation, all server-side, all raising ValueError naming the allowed set:

Field	Required	Rule	Where
type	yes	^[a-z0-9][a-z0-9_.-]{0,63}$ — lowercase only, ≤64	logstream.py:76, 124-134
stream, room, from_agent	yes	non-empty, ≤256, no control chars	logstream.py:78, 103-122
status	no	one of open claimed ready applied blocked failed superseded	logstream.py:67-69, 136-143
body	no (empty allowed)	≤256 KiB, no NUL	logstream.py:55, 145-158
metadata	no	canonical JSON, ≤64 KiB	logstream.py:57, 160-177
artifact_ids	no	each must already exist, else ValueError	logstream.py:687-694

Three consequences worth stating because they are invisible from the tool descriptions: an event body may be empty while artifact content may not (logstream.py:766-767); to_agent is nullable, and an event with to_agent = NULL is addressed to nobody and can never match a to_agent= query; and the 64 KiB metadata cap and the lowercase-type regex are enforced but undocumented in the shipped tool schemas.

3.2 Three orderings, and one identity that is not what it looks like

The log carries three distinct notions of order, and conflating them is the single most likely design error in any consumer:

Field	Meaning	Scope	Where
seq	local arrival cursor — the SQLite rowid, surfaced at read time	this replica only	logstream.py:580-586
origin_seq	the author replica's own gap-free counter, assigned by UPDATE … SET origin_seq = rowid in the insert transaction	per author	logstream.py:700-706
hlc	hybrid logical clock, <unix_ms>-<counter_hex>-<replica_id>, fixed-width so TEXT comparison is causal comparison	fleet-wide	logstream.py:668, hlc.py:1-21

The rule, from hlc.py:19-21: "Cursor semantics stay LOCAL (rowid arrival order) … a tail consumer must see late-arriving remote ops even though their HLC is older. HLC is the display/merge order; arrival is the delivery order."

origin_replica identifies the palace, not the writer. It comes from get_replica_id(db_parent) — the replica.json in the palace directory (logstream.py:422-436, replica.py:31-49). In a hub-and-spoke deployment every client writes through the server's single Logstream, so every event from every machine carries the same origin_replica and it cannot distinguish two devices. This is mechanically forced by the design, not a misconfiguration.

Therefore: from_agent / to_agent carry the entire distinction between machines. The convention that makes them able to is <harness>@<device> — e.g. pi@laptop, opencode@build-box — stamped at the client edge, which RFC 001 §7.3.5 specifies. An unstamped client addressed as bare pi is unreachable in a fleet, because nobody can name it.

3.3 The ack contract: owed-ness is derived, never read

status is written once, into an append-only row. Nothing updates it. So status="open" means "the sender declared this an ask at the moment of writing" — it does not mean unanswered, and a directed open event keeps matching a mailbox query forever, answered or not.

ack_event (logstream.py:812-845) never touches the target row. It appends:

return self.append_event(
    type=ACK_EVENT_TYPE,                                     # "event.ack"
    stream=target["stream"], room=target["room"],
    from_agent=from_agent,
    to_agent=target["from_agent"],                           # routes back to the sender
    correlation_id=target["correlation_id"] or target["id"], # falls back to the target id
    status=status, body=body,
    metadata={"ack_of": event_id},                           # the join key lives in metadata
)

So a consumer that wants "what do I still owe?" must derive it. The derivation the toolkit implements, and which this RFC adopts as the reference semantics:

stateDiagram-v2
    [*] --> open : task.request written, addressed to you
    open --> claimed : you announce you are working (does NOT clear)
    open --> ready : work exists, thread still open (does NOT clear)
    claimed --> applied : terminal, clears
    ready --> applied : terminal, clears
    open --> blocked : terminal, clears (with a reason)
    open --> failed : terminal, clears (with a reason)
    open --> superseded : terminal, clears (replaced by another thread)
    applied --> [*]
    blocked --> [*]
    failed --> [*]
    superseded --> [*]

An event is still owed when all of these hold:

1. it is directed at you (to_agent = <you>, and to_agent = '*' is excluded — §7.4);
2. it was not written by you (you cannot owe yourself);
3. no event of yours has all of: a strictly higher seq, a join to it (metadata.ack_of = <id> or a shared correlation_id), and a terminal status (applied / superseded / failed / blocked).

The strictly-higher-seq test is load-bearing: without it, one terminal reply on a correlation_id suppresses every later ask on that same correlation, permanently. And because seq is local arrival order, this test is only sound on a single replica — §7.3.

3.4 Artifacts

Exact payloads for handoff: patch, file, log, json, note; UTF-8 text only; ≤4 MiB; sha256 and size_bytes returned (logstream.py:744-810).

⚠️ The sha256 is for verification, not deduplication. artifacts_sha256_idx is a plain index and put_artifact always INSERTs, so storing the same patch twice stores the content twice. Referential integrity in the other direction is enforced: artifact_ids naming an unknown artifact raises at append time, in both the local and the remote path (logstream.py:687-694, 1181-1189), "so readers never see a dangling reference".

kind="patch" gets advisory warnings — never mutation — for a missing trailing newline or CRLF endings (logstream.py:718-742). The comment records why: "the first RFC 003 dogfood: a patch stored without its final trailing newline truncates the last hunk line."

3.5 Delivery: three mechanisms, one client feature

Mechanism	Shape	Latency	Where
event_list	pull, filtered, ORDER BY rowid ASC, default 50 / max 500 (silently clamped)	whenever you ask	logstream.py:914-990
event_wait	in-request long poll, jittered backoff 0.25 s→1 s, default 60 s, hard cap 300 s, returns {timed_out: true, events: []} rather than raising	seconds, while you hold the call	logstream.py:1004-1039
GET /logstream/stream	SSE; resume by ?since_event_id= or Last-Event-ID; : ping every 15 s; ≤8 concurrent clients then 503 + Retry-After	sub-second, while connected	mcp_server.py:7570-7671

None of these push anything into an agent that is not already running. That gap is closed client-side by the toolkit's mailbox: it derives the owed set per §3.3 and injects it at two points — once in the session-start wake-up, and mid-session on a settled-agent poll floored at 5 minutes, delivered as a queued steer rather than an interrupt.

Measured 2026-08-26 (first live delivery on a published image, positive control with a synthetic sender): planted → delivered in ≈2–3 minutes to a session that was already running, as exactly one item; three already-answered asks were correctly excluded and a * broadcast was correctly ignored. The wake-up path and the resurface path were not exercised by that test and remain unmeasured.

This is why the mailbox is a client concern and belongs in the toolkit rather than here: the server has no idea which agents exist, and no way to reach one that is not calling it.

---

4. Availability: the logstream is deliberately exempt from both palace locks

The rest of the palace is a single-writer service — _HTTP_REQUEST_LOCK wraps every dispatch (RFC 001 §7.5), and a Chroma peer-writer lease protects against a concurrent CLI mine. Coordination traffic is exempt from both, in two separate places, for two different reasons:

• _HTTP_LOCK_FREE_TOOLS (mcp_server.py:6887-6902) — all seven event/artifact tools dispatch outside the request lock: "Dispatching them outside _HTTP_REQUEST_LOCK keeps a five-minute mempalace_event_wait long-poll from stalling every other agent on a shared hub, and lets the SSE stream coexist with normal tool traffic."
• _PEER_WRITER_EXEMPT_TOOLS (mcp_server.py:434-449) — the four mutating ones bypass the writer lease: "Exempting them keeps agent coordination alive while a CLI mine or a peer stdio writer holds the palace lock."

They remain in _MUTATING_TOOLS, so --read-only still refuses them.

✅ Consequence, and it corrects a widely-held belief in this fleet: "one large mine blocks every client for minutes" is true of drawer writes and false of coordination writes. You can message another device, and it can reply, while a mine is running.

Concurrency within the log is a per-instance threading.Lock over a WAL database with a 10 s busy timeout, one cached Logstream per palace path per process (logstream.py:437, mcp_server.py:333-334, 1120-1141).

---

5. Not everything belongs in the log

RFC 001 §5 draws this line for wings; the same discipline applies between the two stores. The short form, expanded for operators in docs/fleet-memory.md:

Put it in	When
A drawer (palace)	It will still be true, and worth finding, next month. Nobody in particular needs to act. Retrieval is by meaning.
An event (log)	A named agent must act, reply, or be stopped. Retrieval is by address and order.
Both	The durable finding goes in a drawer; the event says "there is a new finding, here is the drawer id". This is the recommended pattern for anything a peer must know rather than do.

The failure mode in each direction: a finding filed only as an event is invisible to semantic search and will be re-derived by the next agent; an ask filed only as a drawer is addressed to nobody and will be found, if ever, by accident.

---

6. Security model

The log authenticates the fleet, not the agent. Precisely:

Control	State	Evidence
Transport auth	One shared bearer token, hmac.compare_digest, 401 otherwise; required for non-loopback binds	mcp_server.py:7137-7141, 7685
from_agent authenticity	❌ Not checked against anything. Shape-validated only; any client may append an event claiming to be any agent	logstream.py:103-122 is the only check
Read authorization	❌ None. Any token holder may list every event addressed to anyone, and fetch any artifact by id — including patch contents	no scoping in the query path
Stream auth	✅ SSE follows the same bearer policy: "Events and artifacts expose work metadata and patch contents"	mcp_server.py:7189-7194
Transport security	TLS via --tls-cert/--tls-key (both or neither); Host pin against DNS rebinding; Origin check; 16 MiB request cap	mcp_server.py:6920-6947

For a single-operator fleet behind one token this is adequate, and it must be stated rather than implied, because two useful consequences follow directly from it:

1. Impersonation is trivial — do not treat from_agent as evidence of origin in any security decision.
2. That same property is the only way to test the mailbox. A positive control requires writing an event from an agent you are not, so that your own client does not self-filter it. This is a supported technique precisely because the field is unauthenticated (measured 2026-08-26).

If per-agent identity is ever needed, the correct home is the same place RFC 001 §7.3 puts provenance: the authenticated credential at the boundary, not a self-asserted field.

---

7. Landmines

7.1 There is no idempotency guard on append — a retried write duplicates

append_event mints a fresh id and INSERTs, with no dedup lookup of any kind (logstream.py:625-716). put_artifact likewise. Two byte-identical calls produce two events with different ids and different seq.

The asymmetry is stark: the replication path is rigorously idempotent (logstream.py:1174-1180 checks id or (origin_replica, origin_seq) before applying, and returns early for its own echoed ops). Peer replay is safe; client retry is not.

Action: this makes the palace's general rule — a timeout usually means the write completed; verify, don't retry — load-bearing rather than advisory. For a drawer, a blind retry costs a dedup-detectable duplicate. For an event it silently forks a coordination thread into two ids, and a mailbox will then show two owed items that must each be closed. After any event_append / patch_submit timeout, verify with event_list(correlation_id=…) before re-issuing.

7.2 status="open" is not owed-ness

Covered in §3.3 and repeated here because it is the mistake most likely to be made by someone reading only the tool descriptions: the mailbox query to_agent=<me> status=open never shrinks as you work. Treating its length as a to-do count means re-answering answered asks forever.

Action: derive per §3.3, or use a client that does.

7.3 seq is local arrival order and is meaningless across replicas

seq is the local rowid. The moment a second replica exists, a remote event that was authored earlier can arrive later and receive a higher seq. An owed-set derivation that compares seq is therefore sound only on a single replica: on a mesh, a reply can land before the ask it answers, the join concludes "no later reply exists", and an already-answered ask reappears as owed — permanently, on that machine.

✅ Resolved 2026-08-26 (toolkit extensions/pi/mempalace.ts). The derivation now compares hlc when both events carry one, falling back to seq only when either lacks it (a server predating the field, or an un-backfilled row). hlc is rendered fixed-width — <unix_ms:13 digits>-<counter:6 hex>-<replica_id> (hlc.py:1-21) — so a plain string comparison is the causal comparison, with the replica id as final tiebreak. The field is present on the wire: event_list returns it per event (logstream.py:593), verified against the live hub the same day.

Measured on real data: the positive-control pair carries seq 26/27 and hlc 1787773071857-…/1787773574085-…, so on today's single replica the two orderings agree and the switch is a no-op now and correct later — which is the whole reason to make it before a second replica exists rather than after. created_at remains the wrong key in both worlds: server-generated at second precision, so ties are routine and a tie can suppress an unanswered ask outright.

7.4 A broadcast reaches no mailbox, and to_agent=NULL reaches nobody at all

to_agent=<x> matches x or '*' at the SQL level (logstream.py:958-960), so a broadcast is visible to a listing agent. But the reference owed-set derivation excludes to_agent='*' deliberately — a broadcast owes nobody a reply, and if it entered every mailbox, every machine would think it personally owed the same answer.

Two consequences: broadcasting an ask reaches no owed set at all (the "don't broadcast an ask" anti-pattern is mechanically enforced, not merely advised), and to reach a whole fleet with something actionable you must write one directed event per device, sharing a correlation_id so the thread stays joinable. Separately, an event written with no to_agent matches no to_agent= query ever and is addressed to nobody.

✅ Measured 2026-08-26, on two devices independently. This was inferred from source until a two-arm control settled it. A to_agent='*' event with status='open' — the anti-pattern, planted deliberately — survives the raw candidate filter and so reaches the exclusion branch, which every previously written broadcast (all non-open) never did. Result: the raw query returned it, the derived owed set did not, and the delivery named only the directed arm of the pair. Confirmed on a second machine that had merely received the broadcast rather than planted it. Two arms differing only in to_agent, same poll and same fake sender, is what makes it evidence rather than an absence: "delivered exactly one of two" cannot be explained by a mailbox that delivers everything or nothing.

⚠️ Correction to an earlier record. A note filed the same day claimed this branch could not be exercised, reasoning that every broadcast in the log carries status='ready' and the protocol forbids writing an open broadcast. The reasoning was wrong in a specific and instructive way: the protocol forbids it as production traffic, which does not forbid planting one as a labelled, self-closed control. Where a rule blocks a measurement, check whether it blocks the use or the test before recording the branch as untestable.

7.5 since_event_id raises on an unknown id

An unresolvable cursor raises ValueError rather than returning empty (logstream.py:966-970) — stronger than the tool description promises, and benign on one replica. On a second replica, a resuming watcher whose cursor has not yet replicated will hard-fail instead of waiting.

7.6 A rejected append returns HTTP 200

Failures come back as {"success": false, "error": …} in a 200 response, not a JSON-RPC error (mcp_server.py:4581-4584).

Action: a client that checks only transport status silently loses the event. Check the payload.

7.7 The same agent name on two devices is indistinguishable

No uniqueness, no registry, no warning. Both machines' events carry one from_agent, and each machine's "you cannot owe yourself" filter will discard the other's asks. This is the failure the <harness>@<device> convention exists to prevent (§3.2).

7.8 GET /logstream/events does not exist

The complete GET route table is /healthz, /statusz, /logstream/stream, /sync/{version_vector,ops,artifact,peers}; POST serves /mcp only. A recursive grep for /logstream in the package yields three hits, all /logstream/stream.

⚠️ Correction. An earlier measurement observed /logstream/events, /logstream/stream and /sync/peers all returning 404 against a deployment and attributed all three to a reverse proxy exposing only /mcp. That inference was right for /sync/* and /logstream/stream and wrong for /logstream/events, which would 404 on a directly-reachable server too. Distinguish "route absent" from "route blocked" before blaming infrastructure.

7.9 Dogfood scars, preserved because each cost someone a session

• A patch artifact stored without its trailing newline truncates the last hunk line — hence the advisory warnings (logstream.py:718-742).
• --type Task.Request was rejected while --type Task.Request --type patch.ready was silently accepted and matched nothing, leaving a watcher waiting forever: single-valued filters were validated by pushdown, multi-valued ones compared raw (sanitize_watch_spec, logstream.py:245-278). type is lowercase-only for this reason.
• event_wait rejected a limit that event_list accepted — "reported by windows-codex during dogfood" (mcp_server.py:4666-4670).

7.10 The main database file's mtime is not a liveness signal

WAL. Measured 2026-08-26 on the primary: logstream.sqlite3 mtime three days old while logstream.sqlite3-wal was 865 KiB and seconds old. An operator checking whether coordination is live must look at the -wal file, or query.

7.11 Delivered is not read: the human is the trigger

The mid-session poll delivers with deliverAs: "steer" and deliberately no triggerTurn — waking a model on inbound fleet traffic is a much larger behavioural change than auto-delivery, and was not approved. The poll fires on agent_settled, which means the agent is idle: nothing is running to react. So the delivered text is queued and read at the top of the next turn, whenever a human happens to start one.

Measured 2026-08-26: a delivery landed in a session window at ~20:50Z and sat there, visibly unreacted-to, until the operator asked "do I have to nudge you for you to read it?" — which is precisely the question this design produces. The delay is not the agent choosing to ignore its mailbox; between the poll and the next turn there is no inference at all.

The consequence is a UX gap, not a defect: from the outside, an inbound ask plus an idle agent reads as neglect. The delivered text explained how to close an ask but said nothing about when it would be seen — so the one reader who needed that fact, the human watching the window, was the one not told.

✅ Addressed 2026-08-26 (toolkit extensions/pi/mempalace.ts), without touching the no-triggerTurn decision. Two changes, in the order their value is realised:

1. A delivery note in the message itself — it says this is a queued message, that nothing woke the agent, and that any message starts the turn that will handle it. Costs nothing, changes no behaviour, and converts "why is it ignoring me?" into "right, I nudge it."
2. A notification at poll time, because the note only reaches someone already looking, and the case that actually loses an ask is nobody looking. MEMPALACE_MAILBOX_NOTIFY unset → in-TUI ctx.ui.notify (the surface session_start already uses); =desktop → additionally a terminal-native notification (Kitty OSC 99, else OSC 777), which is how a ping escapes a container with no notify-send, DBus or host access — the escape sequence is interpreted by the terminal emulator on the human's machine; =0 → silent. Title and body are stripped of ; and control bytes so a payload cannot forge an OSC field or terminate the sequence early. It fires only when something is due, because a ping on an empty poll trains its reader to ignore it.

⚠️ Unverified through a multiplexer — measured 2026-08-26, tor-ms22. The terminal-native path is written and syntax-checked but has not been observed to fire. On this client the layering is kitty → tmux (on the host) → docker exec → pi (in the container), with two clients attached to the same tmux session (one local, one remote over SSH). Test sequences written directly to pi's tty produced no notification on the remote client; the local client is unobserved so far. The likely mechanism is that tmux drops OSC sequences it does not itself implement — reaching the outer terminal requires wrapping them in tmux's DCS passthrough (ESC P tmux; … ESC \, with inner ESC bytes doubled) and allow-passthrough on, which is not the default.

This compounds §7.11's detection problem rather than repeating it: the container cannot see either layer. KITTY_WINDOW_ID is absent because docker exec does not forward it, and TMUX is absent because tmux is running one level further out, on the host — so the client cannot detect the terminal it is speaking to or the multiplexer it must speak through. Explicit configuration is the only reliable route; autodetection is structurally blind here.

Open, and deliberately not settled by guessing: whether the local client shows it (pending); whether to add DCS-wrapping modes; and with two clients attached, which client should be notified — a pane's output reaches every attached client, so a work laptop and a home machine would both ping.

Still deliberately not done: MEMPALACE_MAILBOX_TRIGGER=1 (opt-in, default off) to start a turn on arrival. Worth revisiting once notification is proven in daily use — but a machine that auto-turns on inbound events writes to a shared log with nobody watching, and §7.1 (no idempotency guard) is the reason to be slow about it.

7.12 The mailbox is an obligation channel, so a report addressed to you is never delivered

Candidates are drawn with status="open" (§3.3), so an event carrying a terminal status is not a mailbox candidate at all — no matter who it is addressed to. A task.reply sent to a named device to share a finding is therefore delivered to nobody, ever, and neither is any event.ack. It sits in the log until someone reads the log.

Measured 2026-08-26: a device wrote a detailed report addressed to pi@<peer> with status="applied", on a thread whose ask belonged to a third party. The recipient never saw it and only read it when the operator pointed at it by id — with the mailbox working exactly as specified throughout.

This is the shape of the channel, and it is worth stating because the natural inter-device message — "here is something you should know" — is exactly the shape that gets no delivery. Two ways to make news reach a peer: address it as a directed status="open" ask so it enters the owed set and gets closed when acted on (correct when a response is genuinely wanted), or accept that it is pull-only and pair it with a palace drawer, which the peer's search will surface later. What does not work is a terminal-status report and an expectation of attention.

---

8. Phasing

8.1 Implemented

Phase	Deliverable	State
A	events/artifacts/event_artifacts + append/list/ack + size limits	✅ 3.7.x
B	Long poll (event_wait) and SSE push	✅
C	patch_submit convenience + patch advisories	✅
D	HLC on every event; (origin_replica, origin_seq) unique index	✅ 3.8.0
E	Client-side auto-delivered mailbox	✅ toolkit 5b8d78f; first live delivery measured 2026-08-26

8.2 Deferred, and what RFC 004 owes

Replication exists in code — version_vector(), list_ops(), apply_remote_event(), apply_remote_artifact() and the four GET /sync/* routes (logstream.py:1096-1261, mcp_server.py:7206-7256) — and is attributed in comments to "RFC 004 step 0". That document does not exist. Until it does, this RFC records the two properties a reader most needs: the apply path is idempotent (§7.1), and switching the owed-set derivation from seq to hlc is a precondition for a second replica, not a follow-up (§7.3).

---

9. Open decisions

1. Retention. No TTL, compaction or pruning exists, and mempalace sync does not touch the log (§2). For a fleet log this is mostly a feature — nothing is lost by being offline for weeks — but every 4 MiB artifact is permanent. Decide a policy before the log outgrows a comfortable backup, or decide explicitly that permanence is the policy.
2. Idempotency key. Should event_append accept an optional client-supplied dedupe key so a retried call is a no-op? This is the one change that would make §7.1 disappear.
3. hlc as the mailbox join key, replacing seq. Done 2026-08-26 — see §7.3. What remains open is the reverse direction: mempalace_event_list's since_event_id cursor is deliberately local-arrival-ordered (hlc.py:19-21 — "a tail consumer must see late-arriving remote ops even though their HLC is older"), so cursor semantics and join semantics use different orderings on purpose. That is correct, and worth stating loudly before someone "fixes" the cursor to match the join.
4. Per-agent identity. Do we ever want from_agent to be authenticated (§6), or is "authenticates the fleet, not the agent" the permanent contract?
5. Should GET /logstream/events exist? A read-only HTTP tail would let non-MCP consumers (dashboards, CI) follow a stream without an MCP client. Today they must hold SSE or speak MCP.
6. Undocumented limits. The 64 KiB metadata cap and the lowercase-type regex are enforced server-side but absent from the tool schemas (§3.1). Document, or relax.
7. Agent-name registry. Nothing prevents two devices sharing one from_agent (§7.7). A warning at append time would be cheap.
8. Does an arriving ask deserve a notification? Done 2026-08-26 — delivery note plus MEMPALACE_MAILBOX_NOTIFY (§7.11). What stays open is the narrower question: should desktop be the default rather than opt-in, and does an arriving ask ever deserve a turn (MEMPALACE_MAILBOX_TRIGGER)?
9. Is there a delivery path for news rather than obligations? (§7.12) Today the only delivered shape is a directed open ask. Options: leave it pull-only and rely on the paired drawer, or give informational events a distinct low-priority surface at wake-up ("3 reports since your last session") separate from the owed set.

9.1 Retention — decided direction (2026-08-26)

Decision 1 above is settled in direction: old traffic should eventually move out of the way, logrotate-style. Moved aside, not necessarily destroyed — the audit trail of who asked whom for what is worth more than the disk it occupies.

The economics point at a two-tier design rather than one sweep. Events are tiny (a body cap of 256 KiB, and in practice a few KB); artifacts are capped at 4 MiB each and are stored as content in-row. So:

• Tier the artifacts first. Keep every events row indefinitely — they are the index and the audit trail — and move artifact content to cold storage past the window, keeping the kind, sha256, size_bytes and created_by stub in place. A stub still answers "what was handed over, by whom, verified how"; only the bytes go cold. This reclaims nearly all of the space with none of the join risk below.
• Rotate events by thread, never by row. If rotation is wanted for events too, the unit must be the correlation_id thread, and only a thread whose latest event is terminal (§3.3) and older than the window. Archiving an ask while leaving its reply behind — or the reverse — breaks the owed-set join, and the two failure modes are both bad: an ask that can never be cleared resurfaces as owed forever, or a reply is orphaned from what it answered.

Three constraints any implementation has to respect:

1. Never rotate an event that is still owed. The owed set is derived at read time (§3.3), so an unanswered ask has no marker distinguishing it from a stale one except that derivation. A time-based sweep alone would silently discard live obligations from a machine that has simply been offline for a month — exactly the case this log exists to serve.
2. Rotation invalidates held cursors. since_event_id raises on an unknown id rather than returning empty (§7.5), so archiving an event that a watcher still holds as its resume point turns that watcher's next poll into an error. Either rotation is announced far enough ahead of any live cursor, or the anchor lookup learns to fall back to created_at/hlc when the id is gone.
3. Archive before delete, and verify. Same discipline as any palace destructive op: write the cold copy, verify row counts and sha256 for artifacts, and only then DELETE + VACUUM the live database. A --dry-run that reports what would move, in thread units, is the minimum interface.

Open sub-questions: the window length (a quarter is the obvious first guess); whether cold storage is a sibling logstream-archive-<period>.sqlite3 or plain files on disk; and whether archives are queryable through the same tools behind an explicit opt-in flag, or simply left as files for a human to open when a question reaches back that far.

---

10. Evidence index

Claim	Where
Store is logstream.sqlite3 inside the palace dir	logstream.py:51; mcp_server.py:1108-1117
Full DDL, indexes, WAL	logstream.py:447-497, 552-556
Design constraints (quoted in §3)	logstream.py:1-18
Event id format, ordering not carried by id	logstream.py:92-102
seq = rowid, surfaced at read	logstream.py:580-586
origin_seq assigned in-transaction	logstream.py:700-706
hlc populated per append; format and sort property	logstream.py:668; hlc.py:1-21
Cursor stays local, HLC is merge order	hlc.py:19-21
origin_replica from palace dir	logstream.py:422-436; replica.py:31-49
Server-side created_at, second precision	logstream.py:667
No dedup on append; remote apply is idempotent	logstream.py:625-716 vs 1174-1180
Artifact sha256 non-unique index	logstream.py:447-497, 744-810
artifact_ids referential check	logstream.py:687-694, 1181-1189
Status set; artifact kinds; size limits	logstream.py:67-69, 55-57, 765-778
type regex, lowercase ≤64	logstream.py:76, 124-134
Broadcast matching in SQL	logstream.py:958-960
since_event_id anchor + raise	logstream.py:963-978
Order by rowid; limit clamp 500	logstream.py:947
preview truncates body to 200 chars	mcp_server.py:4587-4607
event_wait backoff, cap, timeout result	logstream.py:1004-1039
ack_event appends, never mutates; ack_of in metadata	logstream.py:812-845
Patch advisories	logstream.py:718-742
Lock exemptions, both, with rationale	mcp_server.py:6887-6902, 434-449
Bearer auth; SSE auth required	mcp_server.py:7137-7141, 7189-7194
SSE implementation, heartbeat, client cap	mcp_server.py:7570-7671
Rejected append returns 200 + success:false	mcp_server.py:4581-4584
sync does not touch the log	cli.py:1057; sync.py (no logstream refs)
No retention/pruning anywhere	no DELETE FROM events/artifacts in the package
Replication surface, attributed to RFC 004	logstream.py:1096-1261; mcp_server.py:7206-7256

10.1 Where the shipped tool descriptions and the code disagree

Tool-description claim	Verdict
"append-only; corrections are new events"	True of content. Precisely: no event's content is ever mutated; append_event does issue one UPDATE … SET origin_seq = rowid on the row it just inserted, and the 3.8.0 migration backfills origin_replica/origin_seq/hlc on pre-existing rows (logstream.py:504-550).
"since_event_id cannot skip anything"	True, and stronger than stated — it raises on an unknown id (§7.5).
"body max 256 KiB", "artifact max 4 MiB, UTF-8 only", "timeout default 60 s max 5 min"	All true; the timeout clamps silently rather than erroring.
"to_agent=<you> also matches * broadcasts"	True, SQL-level — but see §7.4 for why a broadcast still reaches no mailbox.
"prefer the push stream at GET /logstream/stream"	True; route exists and requires auth.
GET /logstream/events	❌ Does not exist (§7.8).
metadata size limit; type charset	❌ Enforced but undocumented (§3.1).

---

11. See also

• docs/fleet-memory.md — the operator-facing companion: what the palace's stores are for, and when to use a drawer versus an event.
• docs/rfc-001-global-palace.md — §5 what should not be global, §7.2 the sync hazard, §7.3 provenance at the boundary, §7.5 single-writer expectations.
• docs/rfc-002-joiner.md — replaying a second palace into a shared primary.
• extensions/pi/README.md §2 — the client-side mailbox: delivery points, gating, and why it is a client concern.
• ~/.agents/skills/mempalace/SKILL.md §"Cross-Machine Coordination" — normative for agent behaviour; this RFC is normative for mechanism.