From b609cf5a69f47890bc33cc39fd6c72451b0cee5e Mon Sep 17 00:00:00 2001 From: Joakim Persson Date: Sun, 16 Aug 2026 00:52:27 +0200 Subject: [PATCH] docs(synlig): the transcript inbox is the primary's third moving part; and survive an unset HOME MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Runbook gaps found while fixing the 2026-08-15 feed failure: - §2.5 still titled "written but not installed" and still asserting "Not installed, not enabled" — false since 2026-08-12. A reader landing there got a flat contradiction of §4 item 3. Retitled, with the verified-2026-08-16 process line, and it now states the fact §2.6 depends on: the server is a NATIVE process (no mempalace container on synlig), so it sees host paths. - New §2.6 documents ~/mempalace-feed//: why transcripts cannot travel over the HTTPS leg at all, the three client variables, and why MEMPALACE_PI_REMOTE_PATH is the trap (its /data/feed default assumes a containerized server; here it must equal the ssh-target path, and a mismatch fails with rsync succeeding and only the mine failing). Plus the operational notes that cost time: dedup keys on the absolute path so the inbox path is load-bearing, grown sessions are purged+refiled by mtime, and a client-side MCP timeout is NOT a failed mine. - Header status: counts refreshed with an explicit "treat counts as timestamps". Also, bin/mempalace-pi-session: default HOME from the passwd database when it is unset. `docker run --entrypoint="" ` inherits no HOME when the image config declares none, and every default is HOME-anchored under `set -u`, so the script — including the palace-free --self-test — died with "HOME: unbound variable" in exactly the environment pi-devbox's smoke suite uses. pi-devbox v1.8.0 lost a release to the same assumption from the other side. --- bin/mempalace-pi-session | 11 ++++++ docs/synlig-primary-runbook.md | 68 +++++++++++++++++++++++++++++++++- 2 files changed, 78 insertions(+), 1 deletion(-) diff --git a/bin/mempalace-pi-session b/bin/mempalace-pi-session index 177a55e..b24724c 100755 --- a/bin/mempalace-pi-session +++ b/bin/mempalace-pi-session @@ -123,6 +123,17 @@ set -euo pipefail +# HOME can legitimately be unset: `docker run --entrypoint="" ` inherits +# no HOME when the image config declares none (pi-devbox's does not — HOME is +# normally set by its entrypoint, which --entrypoint="" skips), and every +# default below is HOME-anchored under `set -u`, so the script died at line 1 of +# real work with "HOME: unbound variable". Derive it from the passwd database — +# exactly what python's expanduser() falls back to — so the script, and +# especially the palace-free --self-test, runs in a bare container too. +# pi-devbox v1.8.0 lost a release to this same "the image sets HOME" assumption. +: "${HOME:=$(python3 -c 'import os, pwd; print(pwd.getpwuid(os.getuid()).pw_dir)' 2>/dev/null || echo /tmp)}" +export HOME + # ── Defaults ───────────────────────────────────────────────────────── AGENT="${USER:-mempalace}" WING="wing_conversations" diff --git a/docs/synlig-primary-runbook.md b/docs/synlig-primary-runbook.md index 4a77c7e..68b5052 100644 --- a/docs/synlig-primary-runbook.md +++ b/docs/synlig-primary-runbook.md @@ -10,6 +10,12 @@ on the primary, with verified evidence, so the next session (or the next machine > now 14,803 drawers. One client (EMB-7KJ4VR4G's pi-devbox container) is flipped and verified > end-to-end. Both Phase 0 blockers below are cleared. > +> **Update 2026-08-16 00:20 — the transcript feed is live too, and it is the primary's third moving +> part** (alongside the HTTPS tunnel and the palace itself). See §2.6: transcripts arrive over SSH into +> `~/mempalace-feed//` and are mined *by this host's own server process*. 15,478 drawers as of +> that check — but treat every count in this document as a timestamp, not a fact: `status` counts chunk +> rows, and §4 item 7 explains why counts adjudicate nothing. +> > **Read §3 "Deliberately NOT done" as a record of the 2026-08-10 state, not of today's** — every > item in it has since been done. And before running anything in §5 Rollback, read the warning at the > top of it: `~/.mempalace` on synlig is no longer disposable. @@ -132,7 +138,15 @@ Ran on synlig, loopback and docker0 binds, then stopped. **11/11 as predicted:** Test script kept at `/tmp/synlig-phase0-test.sh` on this container (ephemeral — re-create from the table above if needed; it starts, probes and stops the server, and asserts nothing is left listening). -### 2.5 A start unit, written but not installed +### 2.5 A start unit — written, staged, and (since 2026-08-12) installed and running + +> **This subsection describes 2026-08-10. The unit is now live.** Verified 2026-08-16 00:15: +> `systemctl --user list-units` shows `mempalace-serve.service … loaded active running`, and the +> process is +> `~/.local/share/uv/tools/mempalace/bin/python -m mempalace.mcp_server --transport http +> --host 172.17.0.1 --port 8765 --palace /home/ecsjper/.mempalace/palace`. +> Note what that means and §2.6 depends on: **the server is a NATIVE process, not a container** — it +> sees synlig's real filesystem paths, and `docker ps` on synlig lists no mempalace container. `contrib/systemd/mempalace-serve.service` — user unit, follows the existing `contrib/systemd/` style, carries the bind rationale inline so nobody "fixes" it back to loopback. A copy is already staged on synlig @@ -140,6 +154,58 @@ at `~/.config/systemd/user/mempalace-serve.service.staged` — **the `.staged` s systemd only reads `*.service`, so the file cannot be activated by accident, not even by a stray `daemon-reload`. **Not** installed, **not** enabled: it needs one `sudo loginctl enable-linger`, and standing up a network-reachable service while you were asleep was not mine to decide. +(Both were done on 2026-08-12 — §4 item 3 has the exact commands that were run.) + +### 2.6 The transcript inbox — `~/mempalace-feed//` (added 2026-08-16) + +Flipped clients write drawers over HTTPS, but their **session transcripts** cannot travel that way: +`mempalace_mine` resolves its `source` path *in the server process*, so the server cannot see a +client's staged exports. `mempalace-pi-session --mode remote` therefore rsyncs each client's stage into +a per-device inbox here and then asks the server to mine its own local path: + +```sh +ls ~/mempalace-feed/ # one dir per device, e.g. emb-7kj4vr4g/ +ls ~/mempalace-feed/emb-7kj4vr4g/ # pi_.jsonl, mtimes preserved +``` + +Client side, that needs three variables — and **the third one is the trap**: + +| Variable | Value for this fleet | Why | +| --- | --- | --- | +| `MEMPALACE_PI_SSH_TARGET` | `ecsjper@synlig:/home/ecsjper/mempalace-feed` | where rsync puts the files | +| `MEMPALACE_PI_DEVICE` | e.g. `emb-7kj4vr4g` | inbox subdirectory per machine | +| `MEMPALACE_PI_REMOTE_PATH` | `/home/ecsjper/mempalace-feed` | the inbox **as the server process sees it** | + +The feeder's default for the third is `/data/feed`, which assumes a *containerized* palace server with +the inbox bind-mounted there. **This primary is native (§2.5), so it only ever sees host paths and the +value must equal the path half of the SSH target.** Get it wrong and the failure is quiet in the worst +way: rsync succeeds, the files are all present here, and only the mine fails with +`source directory not found: '/data/feed/'`. + +That is exactly what happened on 2026-08-15, and it went unnoticed for a session because the feeder +decided success with `'"error"' in body` — MCP returns HTTP 200 with the tool's own JSON **escaped** +inside `result.content[].text`, so those bytes are `\"error\"`, the substring never matched, and +`~/.pi/agent/mempalace-catchup.log` printed `Done. Wing 'wing_conversations' updated.` directly under the +error. Fixed in `6e1f4f3`: the envelope is parsed, `--self-test` pins that exact response body, and a +preflight warning fires whenever the ship path and `MEMPALACE_PI_REMOTE_PATH` disagree. + +**Operational notes for this inbox:** + +- Dedup keys on the **absolute source path**, so the inbox path is load-bearing: it must stay stable, or + every transcript re-files under its new name. Migrating it on 2026-08-15 (from the clients' old + container-local stage paths, which arrived with the seed) cost a full re-mine plus + `mempalace_delete_by_source` on 6 old paths — 651 drawers purged, 1243 re-filed. `wing_conversations` + is now keyed entirely on `/home/ecsjper/mempalace-feed//…`. +- A **grown** session is purged and re-filed for the same path (mtime-based), so re-feeding a live + session refreshes it instead of duplicating it. That is why the inbox keeps whole transcripts rather + than deltas — do not "tidy" it by deleting files the palace still references. +- `mempalace_mine` over MCP can **exceed a client's request timeout while the server keeps working and + finishes normally**. A client-side timeout is not a failed mine: check + `SELECT COUNT(*) FROM embedding_metadata WHERE key='source_file' AND string_value LIKE '%'` + (read-only, `file:…?mode=ro`) before retrying anything. +- Health check after any client recreate, from the client: the tail of + `~/.pi/agent/mempalace-catchup.log` should end in `Done. Wing … updated.` with no `error:` line above + it. With the fixed feeder a broken run exits 5 and names the reason. ## 3. Deliberately NOT done