ship: don't trust the mtime the exporter deliberately backdates

rsync --update skips a file whose mtime is not strictly newer than the
receiver's. The stage file's mtime IS the source transcript's mtime
(os.utime() at :903, "preserve session mtime for dedup stability"), so
re-exporting a session that has not been appended to since its last
ship produces a mtime that is not newer than what's already at the
receiver — exactly the case a redactor upgrade needs to ship, because
content differs while mtime does not. --update reports success and
sends nothing.

Reported and patched by pi@mbp-m1-2020 (evt_20260827T211925_9674a31da0b4,
artifact art_20260827T211839_fa52af563105, sha256 f217e47e…), measured
live: a scrubbed re-export of a dormant session (pi_01a03022-…f542)
sat unshipped in the palace host's inbox while every local signal
reported a clean stage, saved only because a host-side sweep happened
to rewrite the remote copy independently that same day.

--checksum compares content and ignores size/mtime entirely. Dropping
--update outright was considered and rejected: rsync's default quick
check already transfers on a SIZE difference alone, which is why the
observed case (33-byte placeholder vs a 43-byte token) would have been
masked as "fixed" by a change that only works until a redaction whose
placeholder happens to match the secret's length. os.utime() at :903
is untouched — its backdating is a separate, load-bearing design call
for dedup stability, out of scope for this fix.

Added scripts/test-rsync-ship-idempotency.sh: ships a file, rewrites
its content to an EQUAL-LENGTH string while restoring the original
mtime (what os.utime() does), ships again, asserts the receiver's
sha256 changed. Equal length is deliberate, not cosmetic — mismatched
lengths would pass via the quick check alone and prove nothing about
--checksum specifically; this is the same reasoning that ruled out
dropping --update. Verified the test discriminates: passes against
today's --checksum, fails against --update (checked by temporarily
substituting the flag in a copy, not committed).

Runs offline — a local rsync destination path exercises the same
size/mtime/checksum comparison as the ssh transfer, no palace or
network needed.
This commit is contained in:
2026-08-27 23:29:30 +02:00
parent b2b50afcc1
commit a361b71c40
2 changed files with 101 additions and 2 deletions
+13 -2
View File
@@ -974,15 +974,26 @@ fi
# ── Ship to the palace host (remote mode only) ───────────────────────
# mempalace_mine expands its source path in the SERVER process, so in remote
# mode the exports have to physically exist over there. rsync --update is the
# mode the exports have to physically exist over there. --checksum is the
# idempotent half; the mine is the other half.
#
# NOT --update: the stage file's mtime is deliberately the SOURCE transcript's
# mtime (see the os.utime() in the exporter, "preserve session mtime for dedup
# stability"), so a re-export of a session that has not been appended to since
# the last ship carries an mtime that is NOT newer than the receiver copy. With
# --update rsync then SKIPS it silently -- which is exactly the case that must
# ship after a redactor change, because the content differs while the mtime does
# not. Measured on mbp-m1-2020 2026-08-27: a scrubbed re-export of a dormant
# session was skipped and unscrubbed bytes stayed in the palace host's inbox,
# while every local signal reported a clean stage. --checksum compares content
# and keeps the ship idempotent without trusting timestamps.
MINE_SOURCE="$STAGE"
if [[ "$MODE" == "remote" ]]; then
ssh_cmd="ssh"
[[ -n "$SSH_CONFIG" ]] && ssh_cmd="ssh -F $SSH_CONFIG"
echo ""
echo "Shipping stage to ${SSH_TARGET%/}/$DEVICE/ ..."
if ! rsync -a --update --no-owner --no-group \
if ! rsync -a --checksum --no-owner --no-group \
-e "$ssh_cmd" \
--include='*.jsonl' --exclude='*' \
"$STAGE/" "${SSH_TARGET%/}/$DEVICE/"; then