diff --git a/docs/rfc-003-coordination-log.md b/docs/rfc-003-coordination-log.md index 2ecc7ea..5b36146 100644 --- a/docs/rfc-003-coordination-log.md +++ b/docs/rfc-003-coordination-log.md @@ -171,10 +171,76 @@ An event is **still owed** when all of these hold: 1. it is directed at you (`to_agent = `, and `to_agent = '*'` is excluded — §7.4); 2. it was not written by you (you cannot owe yourself); -3. **no** event of yours has *all* of: a strictly higher `seq`, a join to it (`metadata.ack_of = ` or a shared `correlation_id`), **and** a terminal status (`applied` / `superseded` / `failed` / `blocked`). +3. **no** event of yours has *all* of: a strictly higher `seq`, a join to it (`metadata.ack_of = ` or a shared `correlation_id`), **and** a terminal status (`applied` / `superseded` / `failed` / `blocked`); +4. **the requester has not explicitly withdrawn it** — see §3.3.1. The strictly-higher-`seq` test is load-bearing: without it, one terminal reply on a `correlation_id` suppresses every *later* ask on that same correlation, permanently. And because `seq` is local arrival order, this test is only sound on a single replica — §7.3. +#### 3.3.1 Withdrawal: the one case where someone else may clear your obligation + +> **Status: implemented in the toolkit, not yet in a released image.** Clause 4 is +> live in `extensions/pi/mempalace.ts` (`isWithdrawn`) with rule tests in +> `scripts/test-owed-withdrawal.sh`. It reaches a device only when an image bakes +> a toolkit revision containing it. Until then clauses 1–3 are the whole rule, and +> **a sender must assume its withdrawal has no effect on the recipient's mailbox.** + +Clause 3 says "no event **of yours**", and that asymmetry is deliberate: owed-ness +is a statement about the *recipient's* accountability, so a requester must not be +able to delete an obligation the recipient genuinely has. It is wrong in exactly +one case — the requester retracting its **own** ask. + +**Measured, 2026-09-08/09.** `pi@mbp-m1-2020` withdrew a v1.8.13 rollout ask to +`pi@tor-ms22`: terminal `superseded`, same `correlation_id`, `metadata.closes` +naming the thread, body "DO NOT SPEND A MINUTE ON v1.8.13". It recorded the +withdrawal as done. The withdrawal had **no effect** — its `from_agent` is mbp, so +it could never satisfy a join that only inspects tor-ms22's own events. tor-ms22's +next wake-up, 8h later and on the first boot of the image shipping this very +derivation, still listed the ask as owed, 41h old, for a release that had been +superseded and never installed there. Closing it cost the recipient a write on a +thread nobody wanted answered. **The failure is invisible from the sender's side:** +mbp did everything a sender is told to do and got a result indistinguishable from +success, which is why this went unnoticed for 41h rather than being caught at once. + +A withdrawal is honoured only when **all** of these hold. Each is a guard against a +specific way a mailbox could otherwise be emptied silently, and each is pinned by a +mutation test: + +| Requirement | The failure it prevents | +|---|---| +| `from_agent` = the **original requester** | a third party clearing someone else's obligation | +| `to_agent` = the recipient **exactly** (not `'*'`) | one broadcast emptying every machine's mailbox at once (§7.4) | +| terminal status | a `claimed`/`ready` note reading as a release | +| strictly after the ask (`hlc`, §7.3) | retiring an ask the requester sent **later** on the same correlation | +| joins the ask (`ack_of` or shared `correlation_id`) | clearing an unrelated thread | +| **an explicit marker**: `metadata.withdraws` **or** `metadata.closes`, whose value is exactly the ask's `correlation_id` or its event `id` | **the dangerous one** — see below | + +**Why release must be stated rather than inferred.** The tempting rule is "any +terminal event from the requester clears it". That breaches this log's standing +safety direction, which is that every failure stays on the *noisy-but-visible* +side: a spuriously resurfacing item costs one turn of human correction, while a +suppressed unanswered ask is silent and permanent. Under the naive rule a +requester appending `applied` for its own bookkeeping — on the correlation, +addressed to the recipient, before the recipient ever replied — would silently +delete a real obligation. `withdraws` is canonical; `closes` is honoured because it +is already the fleet's de-facto marker (mbp seq 119; emb seq 117 and 118 all use +it). Prose does not count: the value must *name* the thread, so +`closes: "v1813-client-rollout-tor-ms22"` is a withdrawal and +`closes: "v1813-… — from the RECIPIENT side, which is the only side that can"` is +not. + +**This does not break the fixed point in §9.2, and a reader will reach for §9.2 to +object.** That section rejects letting terminal directed events into the owed set, +because then a reply becomes owed by its requester, closing it mints a fresh +obligation, and the loop never terminates. That argument is about **candidates**. +Clause 4 adds a **clearer**. A withdrawal carries a terminal status, so it can never +be a candidate, and nothing new becomes owed: the asserting shape (`open`) and the +clearing shape (terminal) stay disjoint, which is what makes owed-ness terminate. + +**Recipients keep the last word.** A withdrawal suppresses; it does not rewrite +history. The recipient may still append its own terminal reply — worth doing when +there is a finding to carry across, as tor-ms22 did at seq 122, salvaging two +measurements from the retracted thread. + ### 3.4 Artifacts Exact payloads for handoff: `patch`, `file`, `log`, `json`, `note`; UTF-8 text only; ≤4 MiB; `sha256` and `size_bytes` returned (`logstream.py:744-810`). diff --git a/extensions/pi/mempalace.ts b/extensions/pi/mempalace.ts index 0162f72..1371116 100644 --- a/extensions/pi/mempalace.ts +++ b/extensions/pi/mempalace.ts @@ -1068,17 +1068,114 @@ export default async function mempalaceExtension(pi: ExtensionAPI) { return Boolean(m.correlation_id && m.correlation_id === candidate.correlation_id); }); - /** Directed asks addressed to this device with no terminal reply from it. */ + /** + * Has the ORIGINAL REQUESTER explicitly withdrawn this candidate? + * + * RFC 003 §3.3 clause 3 clears an ask only on "no event **of yours**", and the + * skill states the consequence outright: "there is nothing anyone can do about + * it from the other end". That asymmetry is deliberate and mostly right — owed + * ness is a statement about the RECIPIENT's accountability, and a requester + * must not be able to delete an obligation the recipient genuinely has. + * + * It is wrong in exactly one case: the requester withdrawing its OWN ask. + * + * MEASURED COST, 2026-09-08/09. pi@mbp-m1-2020 withdrew a v1.8.13 rollout ask + * to pi@tor-ms22 (evt seq 119: terminal `superseded`, same correlation_id, + * `metadata.closes` naming it, body "DO NOT SPEND A MINUTE ON v1.8.13") and + * recorded the withdrawal as done. It had no effect: seq 119's `from_agent` is + * mbp, so it could never satisfy a join that only looks at tor-ms22's own + * events. tor-ms22's next wake-up — 8h later, on the first boot of the image + * shipping this very derivation — still listed the ask as owed, 41h old, for a + * release that had been superseded and never installed there. It had to spend a + * write closing an ask nobody wanted answered. The asymmetry is also INVISIBLE + * from the sender's side: mbp did everything a sender is told to do and got a + * result indistinguishable from success. + * + * WHY AN EXPLICIT MARKER AND NOT "ANY TERMINAL EVENT FROM THE REQUESTER". + * This file's standing rule is that every failure mode stays on the + * noisy-but-visible side, because a spuriously resurfacing item is one turn of + * human correction whereas a suppressed unanswered ask is silent and permanent. + * Inferring release from any terminal event would breach that: a requester + * appending `applied` for its own bookkeeping — on the correlation, addressed + * to me, before I ever replied — would silently delete a real obligation. + * So release must be STATED, not inferred. `withdraws` is the canonical key; + * `closes` is honoured because it is already this fleet's de-facto marker (mbp + * seq 119, emb seq 117/118 all use it), and either must name this exact ask — + * its `correlation_id` or its event `id`. Prose does not count. + * + * WHY THIS DOES NOT BREAK THE FIXED POINT IN RFC 003 §9.2. §9.2 rejects letting + * terminal directed events into the owed set, because then every closure mints a + * fresh obligation and the loop never terminates. That argument is about + * CANDIDATES. This adds a CLEARER. A withdrawal carries a terminal status, so it + * can never be a candidate, and nothing new becomes owed. The asserting shape + * (`open`) and the clearing shape (terminal) stay disjoint. A reader who reaches + * for §9.2 to object is answering a different question. + */ + const isWithdrawn = (candidate: LogEvent, inbound: LogEvent[]): boolean => { + const requester = candidate.from_agent; + if (!requester) return false; + // Names THIS ask specifically: its correlation or its id. A marker naming + // something else, or carrying prose, is not a withdrawal of this ask. + const names = (v: unknown): boolean => + typeof v === "string" && + v.length > 0 && + ((Boolean(candidate.correlation_id) && v === candidate.correlation_id) || + (Boolean(candidate.id) && v === candidate.id)); + return inbound.some((e) => { + // Only the party that ASKED may retract. A third party writing a terminal + // event on a shared correlation must never clear my obligation. + if (e.from_agent !== requester) return false; + // Addressed to the party being released. `to_agent=` also matches '*' + // at the SQL level (RFC 003 §7.4), and a broadcast must not be able to + // quietly empty every machine's mailbox at once. + if (e.to_agent !== mailboxAddress) return false; + if (!TERMINAL_STATUS.has((e.status ?? "").toLowerCase())) return false; + // Same ordering guard, same reason as isAnswered: a withdrawal must not + // retire an ask the requester sent LATER on the same correlation. + if (!isStrictlyAfter(e, candidate)) return false; + const ackOf = e.metadata?.ack_of; + const joins = + (typeof ackOf === "string" && Boolean(candidate.id) && ackOf === candidate.id) || + Boolean(e.correlation_id && e.correlation_id === candidate.correlation_id); + if (!joins) return false; + return names(e.metadata?.withdraws) || names(e.metadata?.closes); + }); + }; + + /** + * Directed asks addressed to this device with no terminal reply from it, and + * not explicitly withdrawn by whoever sent them (see isWithdrawn). + */ async function deriveOwed(): Promise { if (!mailboxEnabled || !available) return []; try { - const [candidatesRaw, mineRaw] = await Promise.all([ + const [candidatesRaw, mineRaw, inboundRaw] = await Promise.all([ client.callTool("mempalace_event_list", { to_agent: mailboxAddress, status: "open", limit: 50, }), - client.callTool("mempalace_event_list", { from_agent: mailboxAddress, limit: 100 }), + // `order: "desc"` is a fix, not a flourish. event_list defaults to `asc` + // (append order), so this asked for the OLDEST 100 events this device + // ever wrote — meaning that once a device passes 100 authored events its + // most RECENT replies fall out of the join window and every ask it just + // answered resurfaces as owed. Latent rather than theoretical: tor-ms22 + // was at ~20 authored events when this was written. The window has to be + // anchored at the newest end for the same reason the join uses `hlc`. + client.callTool("mempalace_event_list", { + from_agent: mailboxAddress, + limit: 100, + order: "desc", + }), + // Inbound with NO status filter, deliberately: a withdrawal carries a + // TERMINAL status and is therefore structurally invisible to the + // `status: "open"` candidate query above — the same omission deriveClosed + // is built on. No amount of polling the open set could ever see one. + client.callTool("mempalace_event_list", { + to_agent: mailboxAddress, + limit: 100, + order: "desc", + }), ]); const candidates = parseEvents(candidatesRaw).filter((c) => { // `to_agent: ` ALSO matches `*` broadcasts, per the tool contract. @@ -1094,7 +1191,8 @@ export default async function mempalaceExtension(pi: ExtensionAPI) { }); if (candidates.length === 0) return []; const mine = parseEvents(mineRaw); - return candidates.filter((c) => !isAnswered(c, mine)); + const inbound = parseEvents(inboundRaw); + return candidates.filter((c) => !isAnswered(c, mine) && !isWithdrawn(c, inbound)); } catch { // Fail silent and open: a mailbox read must never break a session. return []; diff --git a/scripts/test-owed-withdrawal.sh b/scripts/test-owed-withdrawal.sh new file mode 100755 index 0000000..9c9cb5b --- /dev/null +++ b/scripts/test-owed-withdrawal.sh @@ -0,0 +1,271 @@ +#!/usr/bin/env bash +# test-owed-withdrawal.sh — rule tests for the owed-set derivation in +# extensions/pi/mempalace.ts: isStrictlyAfter, isAnswered, isWithdrawn. +# +# WHY THIS EXISTS +# isWithdrawn changes who is allowed to clear an obligation, which is the one +# thing in this extension that can go wrong SILENTLY. A wrong rule here does +# not throw and does not show up in a log — it makes a real unanswered ask +# vanish from a mailbox forever. So the rules get pinned down before they ship. +# +# WHY IT EXTRACTS THE PREDICATES FROM THE SOURCE INSTEAD OF RESTATING THEM +# These are closures inside createExtension(), so they cannot be imported. The +# tempting shortcut is to paste a copy of the logic into the test — which tests +# the copy. This repo has already paid for a divergent second copy (the +# pi-extensions skill mirror, 9579 B behind for weeks; the shell-lint logic +# extracted to one file for the same reason). So the harness cuts the actual +# declaration text out of mempalace.ts by brace matching and runs THAT. Change +# the source and this test follows; paraphrase the source and it cannot. +# +# FIXTURES ARE VERBATIM REAL EVENTS from the fleet log (project/pi-devbox), not +# invented shapes — including the exact incident that motivated isWithdrawn: +# mbp-m1-2020 withdrew a v1.8.13 rollout ask to tor-ms22 at seq 119 and the +# withdrawal had no effect, so tor-ms22 was still being told it owed a reply 41h +# later for a release it never installed. +# +# Usage: scripts/test-owed-withdrawal.sh [source.ts] (exit 0 = all rules behave) +# +# The optional argument exists so the suite can be pointed at a deliberately +# MUTATED copy of the source to prove it is sensitive — a suite that has never +# been observed to fail is not evidence. See the mutation check in the CHANGELOG +# entry that introduced isWithdrawn. +set -euo pipefail + +REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" +SRC="${1:-$REPO_ROOT/extensions/pi/mempalace.ts}" +WORK="$(mktemp -d)" +trap 'rm -rf "$WORK"' EXIT + +[ -r "$SRC" ] || { echo "FAIL: cannot read $SRC" >&2; exit 2; } + +# A gate that cannot run must not pass — the standing rule in this repo. +node --experimental-strip-types --check "$SRC" \ + || { echo "FAIL: $SRC does not parse" >&2; exit 2; } + +# ---------------------------------------------------------------- extractor --- +cat >"$WORK/extract.mjs" <<'EXTRACT' +import { readFileSync, writeFileSync } from "node:fs"; + +const src = readFileSync(process.argv[2], "utf8"); + +/** + * Cut `const = ...;` out of the source by matching delimiters, so the + * test runs the shipped text. Returns the declaration verbatim. + */ +function decl(name) { + const start = src.indexOf(`const ${name} =`); + if (start < 0) throw new Error(`declaration not found in source: ${name}`); + let depth = 0; + let inStr = null; + for (let i = start; i < src.length; i++) { + const c = src[i]; + const prev = src[i - 1]; + if (inStr) { + if (c === inStr && prev !== "\\") inStr = null; + continue; + } + if (c === '"' || c === "'" || c === "`") { inStr = c; continue; } + if (c === "/" && src[i + 1] === "/") { i = src.indexOf("\n", i); if (i < 0) break; continue; } + if (c === "{" || c === "(" || c === "[") depth++; + else if (c === "}" || c === ")" || c === "]") depth--; + else if (c === ";" && depth === 0) return src.slice(start, i + 1); + } + throw new Error(`unterminated declaration: ${name}`); +} + +const parts = ["TERMINAL_STATUS", "isStrictlyAfter", "isAnswered", "isWithdrawn"].map(decl); + +// Sanity: the extractor must have found real bodies, not empty matches. A +// silently-empty extraction would make every assertion below pass vacuously. +for (const [i, p] of parts.entries()) { + if (p.length < 40) throw new Error(`extracted declaration ${i} is implausibly short: ${p}`); +} +if (!parts[3].includes("withdraws")) throw new Error("isWithdrawn does not mention its marker key"); + +writeFileSync(process.argv[3], parts.join("\n\n")); +EXTRACT + +node "$WORK/extract.mjs" "$SRC" "$WORK/extracted.ts" +echo "[extract] pulled 4 declarations from $(basename "$SRC") ($(wc -c <"$WORK/extracted.ts") bytes)" + +# ------------------------------------------------------------------ harness --- +{ + cat <<'HEAD' +type LogEvent = { + id?: string; + seq?: number; + hlc?: string; + type?: string; + status?: string; + from_agent?: string; + to_agent?: string; + correlation_id?: string | null; + created_at?: string; + body?: string; + metadata?: Record | null; +}; + +const mailboxAddress = "pi@tor-ms22"; + +HEAD + cat "$WORK/extracted.ts" + cat <<'TAIL' + +// ---- VERBATIM fixtures from the real fleet log, stream project/pi-devbox ---- +const REP = "rep_d344e349ba276d6fc11997cd552f6937"; + +/** seq 112 — mbp's v1.8.13 rollout ask to tor-ms22. The obligation. */ +const ask112: LogEvent = { + id: "evt_20260907T125110_2bef6f9b07a8", + seq: 112, + hlc: `1788785470242-000000-${REP}`, + type: "task.request", + status: "open", + from_agent: "pi@mbp-m1-2020", + to_agent: "pi@tor-ms22", + correlation_id: "v1813-client-rollout-tor-ms22", +}; + +/** seq 119 — mbp's withdrawal of its OWN ask. Terminal, directed, marker present. */ +const withdraw119: LogEvent = { + id: "evt_20260908T223949_8f5b9bec7c92", + seq: 119, + hlc: `1788907189286-000000-${REP}`, + type: "task.reply", + status: "superseded", + from_agent: "pi@mbp-m1-2020", + to_agent: "pi@tor-ms22", + correlation_id: "v1813-client-rollout-tor-ms22", + metadata: { + closes: "v1813-client-rollout-tor-ms22", + nothing_owed: "no reply required to this event", + replacement: "v1814-client-rollout-tor-ms22", + }, +}; + +/** seq 120 — the LIVE v1.8.14 ask. Must survive the v1813 withdrawal. */ +const ask120: LogEvent = { + id: "evt_20260908T224118_808de43d6982", + seq: 120, + hlc: `1788907278075-000000-${REP}`, + type: "task.request", + status: "open", + from_agent: "pi@mbp-m1-2020", + to_agent: "pi@tor-ms22", + correlation_id: "v1814-client-rollout-tor-ms22", +}; + +/** seq 122 — tor-ms22's OWN terminal reply, which is what actually cleared 112. */ +const myReply122: LogEvent = { + id: "evt_20260909T061851_cf9bd18800db", + seq: 122, + hlc: `1788934731146-000000-${REP}`, + type: "task.reply", + status: "superseded", + from_agent: "pi@tor-ms22", + to_agent: "pi@mbp-m1-2020", + correlation_id: "v1813-client-rollout-tor-ms22", + metadata: { closes: "v1813-client-rollout-tor-ms22 — from the RECIPIENT side, which is the only side that can" }, +}; + +const clone = (e: LogEvent, over: Partial): LogEvent => ({ ...e, ...over }); + +let failed = 0; +const check = (name: string, expected: boolean, actual: boolean): void => { + const ok = expected === actual; + if (!ok) failed++; + console.log(`${ok ? " ok " : " FAIL"} ${name} (expected ${expected}, got ${actual})`); +}; + +console.log("\nisWithdrawn — the requester retracting its own ask"); +// THE INCIDENT. Before this rule existed the answer was false and tor-ms22 was +// told it owed a reply for a release that no longer existed. +check("real seq 119 withdraws real seq 112", true, isWithdrawn(ask112, [withdraw119])); +check("withdrawal does NOT touch the live v1814 ask", false, isWithdrawn(ask120, [withdraw119])); + +console.log("\nisWithdrawn — the ways a mailbox must NOT be silently emptied"); +check( + "no marker: a bare terminal event from the requester is not a withdrawal", + false, + isWithdrawn(ask112, [clone(withdraw119, { metadata: { nothing_owed: "no reply required" } })]), +); +check( + "marker naming a DIFFERENT thread does not clear this one", + false, + isWithdrawn(ask112, [clone(withdraw119, { metadata: { closes: "v1814-client-rollout-tor-ms22" } })]), +); +check( + "marker carrying PROSE does not count (tor-ms22's own seq 122 shape)", + false, + isWithdrawn(ask112, [ + clone(withdraw119, { + metadata: { closes: "v1813-client-rollout-tor-ms22 — from the RECIPIENT side, which is the only side that can" }, + }), + ]), +); +check( + "a THIRD PARTY cannot retract someone else's ask", + false, + isWithdrawn(ask112, [clone(withdraw119, { from_agent: "pi@emb-7kj4vr4g" })]), +); +check( + "a BROADCAST cannot empty every machine's mailbox at once", + false, + isWithdrawn(ask112, [clone(withdraw119, { to_agent: "*" })]), +); +check( + "a NON-TERMINAL status is not a withdrawal", + false, + isWithdrawn(ask112, [clone(withdraw119, { status: "open" })]), +); +check( + "an event addressed to a DIFFERENT device does not clear my ask", + false, + isWithdrawn(ask112, [clone(withdraw119, { to_agent: "pi@emb-7kj4vr4g" })]), +); +check( + "ORDERING: a withdrawal cannot retire an ask the requester sent LATER", + false, + isWithdrawn(clone(ask112, { seq: 121, hlc: `1788907300000-000000-${REP}` }), [withdraw119]), +); + +console.log("\nisWithdrawn — accepted marker spellings"); +check( + "canonical `withdraws` naming the correlation", + true, + isWithdrawn(ask112, [clone(withdraw119, { metadata: { withdraws: "v1813-client-rollout-tor-ms22" } })]), +); +check( + "marker naming the ask's EVENT ID instead of its correlation", + true, + isWithdrawn(ask112, [clone(withdraw119, { metadata: { withdraws: ask112.id } })]), +); + +console.log("\nisAnswered — regression guard, unchanged behaviour"); +check("my own terminal reply still closes my own ask", true, isAnswered(ask112, [myReply122])); +check("my reply on one thread does not close another", false, isAnswered(ask120, [myReply122])); +check( + "ORDERING still load-bearing: my reply cannot pre-close a LATER ask", + false, + isAnswered(clone(ask112, { seq: 130, hlc: `1788999999999-000000-${REP}` }), [myReply122]), +); + +console.log("\nEND TO END — the derived owed set for tor-ms22 on 2026-09-09"); +// The behaviour change, stated as the derivation states it. `mine` deliberately +// EXCLUDES seq 122 so this measures the new rule rather than the reply that +// happened to be written first. +const candidates = [ask112, ask120]; +const mine: LogEvent[] = []; +const inbound = [withdraw119]; +const owed = candidates.filter((c) => !isAnswered(c, mine) && !isWithdrawn(c, inbound)); +const owedIds = owed.map((e) => e.correlation_id).join(", "); +check("exactly one ask remains owed", true, owed.length === 1); +check("and it is the v1814 one, not the withdrawn v1813", true, owedIds === "v1814-client-rollout-tor-ms22"); +console.log(` derived owed set: [${owedIds}]`); + +console.log(failed === 0 ? "\n=== PASSED ===" : `\n=== FAILED: ${failed} ===`); +process.exit(failed === 0 ? 0 : 1); +TAIL +} >"$WORK/harness.ts" + +node --experimental-strip-types "$WORK/harness.ts"