docs: CHANGELOG for v1.8.2 — the silent transcript-feed failure and its guard

First boot of the 2026-08-15 image on EMB-7KJ4VR4G shipped 7 transcripts to the
palace host and filed none. Two causes, neither visible in the log:

- MEMPALACE_PI_REMOTE_PATH was unset, so the feeder used its /data/feed default,
  which assumes a CONTAINERIZED palace server. That fleet's primary runs
  natively (systemd user unit + uv tool), so it only sees host paths and the
  mine died with "source directory not found". rsync had already succeeded.
- The feeder decided success with `'"error"' in body`, but MCP escapes the
  tool's JSON inside result.content[].text, so the check was blind and the
  catch-up log said "Done. Wing updated." Fixed in mempalace-toolkit 6e1f4f3,
  which ships `--self-test` with fixtures pinning that exact response body.

- smoke-test.sh: run `mempalace-pi-session --self-test` against the BAKED
  toolkit, so a stale or reverted MEMPALACE_TOOLKIT_REF cannot reintroduce a
  feeder that mines nothing while reporting success.
- .env.example: spell out that MEMPALACE_PI_REMOTE_PATH is the path the SERVER
  PROCESS can open — container path for a dockerized server, and identical to
  the ssh-target path for a native one — and that a mismatch fails quietly.

.env.example

@@ -57,9 +57,18 @@ SSH_KEY_PATH=~/.ssh
 # the server to mine its own local copy. Without MEMPALACE_PI_SSH_TARGET the
 # feeder is skipped (a remote palace with no inbox has nothing to mine).
 #   MEMPALACE_PI_SSH_TARGET  where to rsync to, as user@host:path
-#   MEMPALACE_PI_REMOTE_PATH what that inbox is called ON THE SERVER — must be
-#                            the container path if the server runs in Docker
-#                            (see docker-compose.mempalace.yml)
+#   MEMPALACE_PI_REMOTE_PATH what that inbox is called ON THE SERVER — i.e. the
+#                            path the SERVER PROCESS can open. If the palace
+#                            server runs in Docker, that is the container path
+#                            (see docker-compose.mempalace.yml). If it runs
+#                            NATIVELY (systemd unit / uv tool / plain
+#                            `mempalace serve`), it sees host paths, so this
+#                            must equal the path half of
+#                            MEMPALACE_PI_SSH_TARGET. Getting this wrong is
+#                            quiet: rsync still succeeds and only the mine
+#                            fails with "source directory not found", so
+#                            transcripts ship and are filed nowhere. The feeder
+#                            warns in preflight when the two paths disagree.
 #   MEMPALACE_PI_DEVICE      inbox subdirectory for this machine (default: hostname)
 # MEMPALACE_PI_SSH_TARGET=user@palace-host:/srv/mempalace-feed
 # MEMPALACE_PI_REMOTE_PATH=/data/feed

.gitea/workflows/docker-publish.yml

@@ -18,6 +18,14 @@ name: Publish Docker Image
 #   5. build-variant    multi-arch push of latest + vX.Y.Z tags.
 #   6. promote-base-latest  re-tag base-<hash> → base-latest with `crane copy`.
 #   7. update-description   patch Docker Hub description.
+#
+# Note the trigger: `push: tags: v*` (plus workflow_dispatch). Nothing here runs
+# on a push to main, so a smoke assertion added outside a release is UNVALIDATED
+# until the next tag — which is exactly how v1.8.0 shipped a broken assertion
+# written three days earlier (it asserted a literal /home/developer stage path,
+# while `run` executes `docker run --entrypoint=""` as root with HOME=/root).
+# The `smoke_only` dispatch input exists to close that gap: it runs steps 1-4
+# against HEAD and stops before anything is published.
 
 on:
   push:
@@ -33,6 +41,10 @@ on:
         description: 'Update latest aliases (default true for tag-push, false for manual test runs)'
         required: false
         default: 'false'
+      smoke_only:
+        description: 'Build base + run both smoke jobs against HEAD, then stop. Publishes nothing. Use to validate smoke assertions without cutting a tag.'
+        required: false
+        default: 'false'
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.ref }}
@@ -165,6 +177,40 @@ jobs:
             fi
           }
 
+          # Read a commit SHA from Gitea, surviving a bad build token.
+          #
+          # These repos are public (see the note at the call sites), so auth is
+          # a convenience, not a requirement — but Gitea REJECTS an invalid
+          # token (401) rather than ignoring it, so a revoked or malformed
+          # GITEA_BUILD_TOKEN could fail an entire release on reads that work
+          # fine anonymously. An ABSENT secret was always safe (Gitea ignores an
+          # empty `token ` value and serves the request, 200); a STALE one was
+          # not. So: try authed, and on 401/403 retry anonymously.
+          #
+          # A non-200 after that emits nothing and returns 0 deliberately, so
+          # require_sha raises the loud explicit abort rather than this helper
+          # inventing a fallback ref.
+          #
+          # Messages go to STDERR, not as ::warning:: annotations: this
+          # function's stdout IS the SHA, so anything written there would be
+          # captured into the ref by the command substitution.
+          gitea_sha() { # $1=repo
+            local repo="$1" url resp code
+            url="https://gitea.jordbo.se/api/v1/repos/joakimp/${repo}/commits?limit=1&sha=main"
+            resp=$(curl -s -w '\n%{http_code}' -H "$AUTH_HEADER" "$url" || printf '\n000')
+            code=${resp##*$'\n'}
+            if [ "$code" = "401" ] || [ "$code" = "403" ]; then
+              printf 'WARNING: Gitea rejected the build token for %s (HTTP %s); retrying anonymously. The read should succeed (public repo), but GITEA_BUILD_TOKEN is stale or malformed and should be rotated.\n' "$repo" "$code" >&2
+              resp=$(curl -s -w '\n%{http_code}' "$url" || printf '\n000')
+              code=${resp##*$'\n'}
+            fi
+            if [ "$code" != "200" ]; then
+              printf 'WARNING: Gitea commit lookup for %s returned HTTP %s\n' "$repo" "$code" >&2
+              return 0
+            fi
+            printf '%s' "${resp%$'\n'*}" | jq -r '.[0].sha // empty' 2>/dev/null || true
+          }
+
           # ── pi version: from the PIN, not from npm `latest` ───────────
           # Until v1.7.0 this followed npm `latest`, which meant every release
           # silently adopted whatever pi had shipped that morning — unaudited —
@@ -226,15 +272,27 @@ jobs:
           echo "atelier_ref=${ATELIER_REF}" >> "$GITHUB_OUTPUT"
           echo "atelier_tag=${ATELIER_TAG}" >> "$GITHUB_OUTPUT"
 
-          # pi-toolkit / pi-extensions (Gitea) → commit SHAs. Gitea API
-          # requires auth even for public-repo commit listing.
-          TOOLKIT_REF=$(curl -sf -H "$AUTH_HEADER" \
-            "https://gitea.jordbo.se/api/v1/repos/joakimp/pi-toolkit/commits?limit=1&sha=main" \
-            | jq -r '.[0].sha // empty' 2>/dev/null || true)
+          # pi-toolkit / pi-extensions (Gitea) → commit SHAs. All three Gitea
+          # repos read in this step are PUBLIC: an unauthenticated GET of these
+          # commit endpoints returns 200 with the IDENTICAL sha (verified
+          # 2026-08-15 for pi-toolkit, pi-extensions and mempalace-toolkit).
+          # The comment that used to sit here claimed the Gitea API "requires
+          # auth even for public-repo commit listing" — it does not. Only
+          # /api/v1/repos/*/actions/* refuses anonymous reads (401), which is
+          # what that claim was almost certainly generalised from.
+          #
+          # The header is still passed on purpose: it keeps working if a repo is
+          # ever flipped private, and an ABSENT secret degrades cleanly, because
+          # Gitea ignores an empty `token ` value and serves the request
+          # anonymously (200). The real hazard is the opposite one — a REVOKED or
+          # malformed token returns 401 where anonymous would have returned 200,
+          # so a stale GITEA_BUILD_TOKEN turns a healthy public read into a
+          # require_sha failure that reads like an API or network fault. If this
+          # step ever fails on a repo you can browse anonymously, suspect the
+          # token before you suspect Gitea.
+          TOOLKIT_REF=$(gitea_sha pi-toolkit)
           require_sha PI_TOOLKIT_REF "$TOOLKIT_REF"
-          EXTENSIONS_REF=$(curl -sf -H "$AUTH_HEADER" \
-            "https://gitea.jordbo.se/api/v1/repos/joakimp/pi-extensions/commits?limit=1&sha=main" \
-            | jq -r '.[0].sha // empty' 2>/dev/null || true)
+          EXTENSIONS_REF=$(gitea_sha pi-extensions)
           require_sha PI_EXTENSIONS_REF "$EXTENSIONS_REF"
           echo "toolkit_ref=${TOOLKIT_REF}"        >> "$GITHUB_OUTPUT"
           echo "extensions_ref=${EXTENSIONS_REF}"  >> "$GITHUB_OUTPUT"
@@ -244,9 +302,7 @@ jobs:
           # into the base-decide hash (see that job) to force a base rebuild
           # when the toolkit moves — otherwise a toolkit-only fix silently
           # fails to land unless Dockerfile.base itself changes.
-          MEMPALACE_TOOLKIT_REF=$(curl -sf -H "$AUTH_HEADER" \
-            "https://gitea.jordbo.se/api/v1/repos/joakimp/mempalace-toolkit/commits?limit=1&sha=main" \
-            | jq -r '.[0].sha // empty' 2>/dev/null || true)
+          MEMPALACE_TOOLKIT_REF=$(gitea_sha mempalace-toolkit)
           require_sha MEMPALACE_TOOLKIT_REF "$MEMPALACE_TOOLKIT_REF"
           echo "mempalace_toolkit_ref=${MEMPALACE_TOOLKIT_REF}" >> "$GITHUB_OUTPUT"
 
@@ -482,6 +538,14 @@ jobs:
   # ── Phase 4: multi-arch publish ─────────────────────────────────────
   build-variant:
     needs: [base-decide, smoke, resolve-versions]
+    # A `smoke_only` dispatch stops the pipeline here: base is probed/built and
+    # both smoke jobs run, but nothing is published. Deliberately NOT wrapped in
+    # always() — specifying `if:` keeps the implicit "all needs succeeded" gate,
+    # so a failing smoke still blocks the release. On a tag push `inputs` is
+    # unset, and `null != 'true'` is true, so releases are unaffected.
+    # promote-base-latest and update-description need build-variant to have
+    # succeeded, so they skip on their own — no extra guard required.
+    if: inputs.smoke_only != 'true'
     runs-on: ubuntu-latest
     container:
       image: catthehacker/ubuntu:act-latest
@@ -574,6 +638,7 @@ jobs:
   # or fail independently of the core release.
   build-variant-studio:
     needs: [base-decide, smoke-studio, resolve-versions]
+    if: inputs.smoke_only != 'true'
     runs-on: ubuntu-latest
     container:
       image: catthehacker/ubuntu:act-latest

CHANGELOG.md

@@ -11,6 +11,143 @@ Pre-v1.0.0 tags followed the pi npm version (`v{pi_version}[letter]`).
 
 ---
 
+## v1.8.2 — 2026-08-16
+
+Patch release. **Ships the fix for a silent transcript-feed failure**, plus the
+smoke assertion that stops it coming back. No image pins changed from v1.8.1
+(pi `0.84.2`, pi-atelier `v0.8.1`); what moves is the baked `mempalace-toolkit`
+ref and one new smoke check.
+
+**The bug this closes** (found on the first boot of the v1.8.1 image, on
+EMB-7KJ4VR4G, 2026-08-15): the container-start catch-up rsynced seven pi session
+transcripts to the palace host correctly, then asked the server to mine
+`/data/feed/<device>` — the feeder's default `MEMPALACE_PI_REMOTE_PATH`, which
+assumes a *containerized* palace server. That fleet's primary runs **natively**
+(a systemd user unit + uv tool), so it only ever sees host paths and the mine
+died with `source directory not found`. rsync had already succeeded, so the
+inbox looked healthy.
+
+It stayed invisible because of the second half: the feeder decided success with
+`'"error"' in body`. MCP answers a hard tool failure with HTTP 200 and a
+JSON-RPC *result* whose `content[].text` carries the tool's own JSON as an
+**escaped string** — the bytes are `\"error\"`, so the substring could never
+match. `~/.pi/agent/mempalace-catchup.log` printed
+`Done. Wing 'wing_conversations' updated.` directly beneath the error JSON and
+exited 0. A feeder whose only artifact claims success is worse than one that
+crashes: nothing in the container disagreed with it.
+
+Shipped here:
+
+- **`mempalace-toolkit` ≥ `b609cf5`** baked (CI resolves the ref at build time):
+  `classify()` parses the MCP envelope instead of grepping it (JSON-RPC error,
+  MCP `isError`, inner `success=false`/`error`), and separates "verified ok"
+  from "unverified: no JSON tool payload" rather than assuming the good case.
+  A preflight warning fires when the rsync destination and
+  `MEMPALACE_PI_REMOTE_PATH` disagree — in *preflight*, so `--dry-run` and
+  `--prepare` surface it too. Remote mode also stops previewing NEW/SKIP from
+  the *local* palace, which had been reporting "6 already filed" about a palace
+  it was not feeding; the tags are now `[?]` and the summary names who decides.
+- **New smoke assertion** — `mempalace-pi-session --self-test` run against the
+  **baked** toolkit. It replays six recorded MCP responses (fixture 1 is the
+  verbatim 2026-08-15 failure body) plus a regression guard asserting the old
+  substring check is blind to it. A stale or reverted `MEMPALACE_TOOLKIT_REF`
+  can therefore no longer ship a feeder that mines nothing while reporting
+  success.
+- **`.env.example`** now spells out that `MEMPALACE_PI_REMOTE_PATH` is the path
+  the *server process* can open — the container path for a dockerized server,
+  identical to the ssh-target path for a native one — and that a mismatch fails
+  quietly, with rsync succeeding and only the mine failing.
+
+**The `--self-test` assertion is deliberately bare** (`mempalace-pi-session
+--self-test`, no `HOME=…` prefix). `run()` invokes
+`docker run --entrypoint="" $IMAGE sh -c …` and no Dockerfile sets `USER` or
+`ENV HOME`, so it executes with **no `HOME` at all** — the same condition that
+made v1.8.0's stage assertion unsatisfiable. The feeder is `set -u` with
+HOME-anchored defaults, so it used to die with `HOME: unbound variable` there;
+`b609cf5` derives `HOME` from the passwd database (what python's `expanduser()`
+falls back to) instead. Keeping the call bare means smoke also proves the feeder
+runs in a bare container, rather than papering over it with an env prefix.
+
+---
+
+## v1.8.1 — 2026-08-15
+
+Patch release. **Unblocks v1.8.0, which never shipped.** Its `smoke` and
+`smoke-studio` jobs each failed exactly one assertion (67/68 and 70/71 passed),
+so `build-variant` and everything downstream skipped: no `v1.8.0` tag reached
+Docker Hub and `latest` stayed on v1.7.0 from 2026-08-07. Image content is
+unchanged from what v1.8.0 intended — the pins here are identical (pi `0.84.2`,
+pi-atelier `v0.8.1`).
+
+The failing assertion was `pi stage defaults next to the palace (not a cache
+dir)`, added three days earlier in 7c00dd6. **It was a test bug, not a product
+regression.** It asserted a literal path:
+
+```sh
+echo "$out" | grep -q "stage=/home/developer/.mempalace/pi-stage/"
+```
+
+but the `run` helper invokes `docker run --rm --entrypoint="" $IMAGE sh -c …`,
+and neither `Dockerfile.base` nor `Dockerfile.variant` sets `USER` or `ENV HOME`
+(the published base image config carries no `HOME` at all — `HOME` is normally
+set by `entrypoint-user.sh`, which `--entrypoint=""` deliberately skips). So the
+assertion ran as **root with `HOME=/root`**, `mempalace-pi-session` correctly
+resolved `stage=/root/.mempalace/pi-stage/…` (it is `$HOME`-relative by design:
+`$MEMPALACE_PALACE_PATH` → `$MEMPAL_PALACE_PATH` → `~/.mempalace/config.json` →
+`~/.mempalace/palace`), and the literal grep could never match under any
+circumstances. The tell was one line below it in the log: the sibling assertion
+`pi stage follows MEMPALACE_PALACE_PATH` **passed**, because it sets the variable
+explicitly and so never consults `HOME`. Default fails while explicit passes is
+the signature of a wrong `HOME`, not of broken staging.
+
+Fixed by asserting the invariant that was actually meant — the stage sits beside
+the resolved palace, sharing its lifetime — which is user-independent:
+
+```sh
+case "$stage" in
+  "stage=$HOME/.mempalace/pi-stage/"*) exit 0 ;;
+  *) exit 1 ;;
+esac
+```
+
+`$HOME` is expanded by the container's own shell, so this holds as root, as
+`developer`, or under any future user, while a cache-dir default — the
+regression the assertion exists to catch — still fails it (verified against all
+three cases plus a simulated `MEMPALACE_PI_STAGE` cache pin). A second
+assertion, `pi stage is palace-adjacent for the developer user`, now covers the
+deployment-specific path properly, by *supplying* `HOME=/home/developer` instead
+of assuming it.
+
+### Why it took a release to notice — and the `smoke_only` input
+
+`docker-publish.yml` triggers on `push: tags: v*` only. 7c00dd6 was a push to
+**main**, so only `lint.yml` ran; v1.8.0 was the first tag afterwards and
+therefore the assertion's **first execution ever**. Any smoke assertion written
+outside a release was unvalidated until the next release consumed it — the
+worst possible moment to discover it.
+
+New `workflow_dispatch` input **`smoke_only`** closes that: it probes/builds the
+base and runs both smoke jobs against HEAD, then stops before publishing
+anything. Implemented as `if: inputs.smoke_only != 'true'` on `build-variant`
+and `build-variant-studio`, deliberately *without* `always()` so the implicit
+"needs succeeded" gate survives and a red smoke still blocks a release;
+`promote-base-latest` and `update-description` already require `build-variant`
+success and so skip on their own. On a tag push `inputs` is unset and
+`null != 'true'` is true, so releases behave exactly as before. This release was
+validated with a `smoke_only` dispatch before the tag was cut.
+
+### Smoke failures now explain themselves
+
+`run` discarded all output (`>/dev/null 2>&1`), so a red ❌ carried zero
+diagnostic weight — explaining this one-line failure took a CI-log dig plus a
+registry image-config inspection, when the container had already printed the
+answer and thrown it away. It now captures output and prints the last few lines
+under a failed assertion only. Assertions that want a diagnostic echo it to
+stderr (the stage checks now report the resolved stage and the `HOME` they saw),
+which stays invisible while they pass.
+
+---
+
 ## v1.8.0 — 2026-08-15
 
 Minor release. Headline: **pi sessions now feed MemPalace by themselves.** The

scripts/smoke-test.sh

@@ -43,12 +43,23 @@ PASS=0; FAIL=0
 # catching an unexpected +GB regression.
 SIZE_THRESHOLD_MB=3800
 
+# On failure, surface the last few lines the command produced. This used to
+# discard output entirely (`>/dev/null 2>&1`), which made a red ❌ carry zero
+# diagnostic weight: explaining the single v1.8.0 stage-default failure took a
+# full CI-log dig plus a registry-config inspection, when the container had
+# already printed the answer and thrown it away. Assertions that want a
+# diagnostic just echo it to stderr — it stays hidden while they pass.
 run() {
   local label="$1"; local cmd="$2"
-  if docker run --rm --entrypoint="" "$IMAGE" sh -c "$cmd" >/dev/null 2>&1; then
+  local out
+  if out=$(docker run --rm --entrypoint="" "$IMAGE" sh -c "$cmd" 2>&1); then
     printf "  ✅  %s\n" "$label"; PASS=$((PASS+1))
   else
     printf "  ❌  %s\n" "$label"; FAIL=$((FAIL+1))
+    # `if`, not `&&` — a trailing false under `set -e` would abort the script.
+    if [ -n "$out" ]; then
+      printf "        └─ %s\n" "$(printf '%s' "$out" | tail -3 | tr '\n' ' ' | cut -c1-300)"
+    fi
   fi
 }
 
@@ -102,8 +113,37 @@ run "mempalace-pi-session on PATH" "mempalace-pi-session --help"
 # default-staged run at a populated dir would export whatever transcripts it
 # finds into the real stage, which is how a synthetic test session ends up
 # staged for mining as if it were a real conversation.
+#
+# Asserted $HOME-RELATIVE, not against a literal /home/developer. `run` invokes
+# `docker run --entrypoint=""`, and neither Dockerfile sets USER or ENV HOME
+# (HOME is set by entrypoint-user.sh, which --entrypoint="" deliberately skips),
+# so these assertions execute as root with HOME=/root. The original literal
+# /home/developer form could therefore never match and failed the v1.8.0
+# release — a test bug, not a product one: the stage resolution was correct all
+# along, it just follows $HOME. The invariant under test ("the stage sits beside
+# the palace, sharing its lifetime") is user-independent, so pinning the user
+# was never part of it. A cache-dir default still fails the pattern below, which
+# is the regression this guards.
+#
+# It went unnoticed for three days because this workflow only triggers on
+# `push: tags: v*` — the assertion was added on a main push, so v1.8.0 was its
+# first execution ever. Use the `smoke_only` workflow_dispatch input to run
+# smoke against HEAD without cutting a tag.
 run "pi stage defaults next to the palace (not a cache dir)" '
   out=$(mempalace-pi-session --dry-run --reason smoke --sessions-dir "$(mktemp -d)" 2>&1) || true
+  stage=$(echo "$out" | grep -oE "stage=[^ ]+" | head -1)
+  echo "resolved ${stage:-<no stage= line>} with HOME=$HOME" >&2
+  case "$stage" in
+    "stage=$HOME/.mempalace/pi-stage/"*) exit 0 ;;
+    *) exit 1 ;;
+  esac
+'
+# Companion to the above: the deployment-specific case the literal assertion was
+# reaching for, done properly by supplying the HOME the container actually runs
+# with instead of assuming it.
+run "pi stage is palace-adjacent for the developer user" '
+  out=$(HOME=/home/developer mempalace-pi-session --dry-run --reason smoke --sessions-dir "$(mktemp -d)" 2>&1) || true
+  echo "$out" | grep -oE "stage=[^ ]+" | head -1 >&2
   echo "$out" | grep -q "stage=/home/developer/.mempalace/pi-stage/"
 '
 run "pi stage follows MEMPALACE_PALACE_PATH" '
@@ -155,6 +195,16 @@ run_expect "remote-palace-without-inbox skip is announced, not silent" \
     "MemPalace catch-up skipped"
 run "...and the skip notice names the variable that fixes it" \
     "grep -A6 'MemPalace catch-up skipped' /usr/local/bin/entrypoint-user.sh | grep -q 'MEMPALACE_PI_SSH_TARGET'"
+# A remote mine that FAILS must not report success. MCP answers a hard tool
+# failure with HTTP 200 and the tool's own JSON escaped inside
+# result.content[].text, so the feeder's old `'\"error\"' in body` check could
+# never see it: on 2026-08-15 a mine that died with "source directory not found:
+# '/data/feed/...'" logged "Done. Wing updated." and exited 0, and this
+# container's transcripts were filed nowhere for a whole session. The feeder
+# carries fixtures for that exact body; run them against the baked toolkit so a
+# stale/reverted toolkit ref can't reintroduce a silent feed.
+run "baked feeder detects a failed remote mine (no silent false success)" \
+    "mempalace-pi-session --self-test"
 # v1.0.0 base additions — verify presence and basic functionality.
 run "pandoc"          "pandoc --version"
 run "typst"           "typst --version"