mailbox: join the owed set on hlc, not seq, before a second replica exists

deriveOwed decides "is this ask still owed?" by asking whether one of my own
terminal replies is LATER than the ask. It compared `seq` — this database's
arrival rowid. On a single hub that is global order, so it was correct; the
comment above it already said hlc was the durable key "once mesh_peers reports
actual peers". Making the switch now, while one replica means the two orderings
agree, costs nothing; making it later means changing the rule while two machines
already disagree about order.

The bug being pre-empted is specific: with a second replica the same event gets
a different `seq` in each database, because arrival order is not authorship
order. A reply authored after its ask can arrive first and take the lower seq;
the join then concludes "no later reply exists" and an already-answered ask
reappears as owed — permanently, on that machine.

isStrictlyAfter() prefers `hlc` when both events carry one and falls back to
`seq` otherwise (a server predating the field, or an un-backfilled row). hlc is
rendered fixed-width, <unix_ms:13 digits>-<counter:6 hex>-<replica_id>, so a
plain string comparison IS the causal comparison, with the replica id as final
tiebreak. Verified before writing the code that the field is actually on the
wire — event_list returns it per event (logstream.py:593) — because a fallback
that never fires would have made this a no-op dressed as a fix.

`created_at` stays rejected, and the reason is now written down where the
decision is: it is server-generated at second precision, so ties are routine,
and a tie can suppress an UNANSWERED ask. Both remaining failure modes are on
the noisy-but-visible side — an answered item resurfacing is annoying, an
unanswered ask going silent defeats the mailbox.

Tested: 18 cases against the extracted comparator — hlc later/earlier/equal,
same-ms counter ties in hex (0x10 vs 0x9, which is where a non-padded format
would break), cross-replica tiebreak, both mesh reorderings, every seq-fallback
path, and degenerate input (null seq, numeric hlc, empty strings, no keys) which
must never claim "answered". All pass. Real-data check on the positive-control
pair: seq 26/27 carry hlc ...3071857/...3574085, so the orderings agree today
and the switch is a no-op now and correct later.

The cursor keeps the opposite ordering ON PURPOSE — since_event_id is
local-arrival ordered so a tail consumer still sees late-arriving remote ops
whose hlc is older (hlc.py:19-21). RFC 003 §9.3 now says so explicitly, because
that asymmetry looks like a bug worth "fixing" and is not.
This commit is contained in:
2026-08-26 23:30:09 +02:00
parent d2764bf78e
commit bfe9c5cd4f
3 changed files with 53 additions and 16 deletions
+11 -4
View File
@@ -320,12 +320,19 @@ Owed-ness is **derived, never read off a field**, because `event_ack` appends an
`status` is written once: a directed `open` event matches the mailbox query
*forever*, answered or not. Two calls (`to_agent=<me> status=open`, and
`from_agent=<me>`), then a candidate counts as answered only when one of this
device's own events has a **strictly higher `seq`**, joins via
device's own events is **strictly later**, joins via
`metadata.ack_of` or a shared `correlation_id`, *and* carries a terminal status
(`applied`/`superseded`/`failed`/`blocked`). The `seq` test is load-bearing:
(`applied`/`superseded`/`failed`/`blocked`). The ordering test is load-bearing:
without it one terminal reply suppresses every later ask on that correlation
forever. `seq` is replica-local, so `hlc` is the correct key once `mesh_peers`
reports actual peers.
forever.
"Strictly later" means `hlc` when both events carry one — a hybrid logical clock
rendered fixed-width, so a string comparison is a causal comparison across
replicas — falling back to `seq` only when either side lacks an `hlc`. `seq` is
this database's arrival `rowid`, so on a mesh the same event has a different
`seq` per replica and a reply can arrive before its ask. Never `created_at`: it
is second-precision, and a tie there can suppress an *unanswered* ask, which is
the one failure this derivation exists to prevent.
`*` broadcasts are excluded even though `to_agent=<me>` matches them, because the
protocol says a broadcast owes nobody a reply — which also means broadcasting an