diff --git a/CHANGELOG.md b/CHANGELOG.md index 75c753f..07b85af 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -14,9 +14,10 @@ Pre-v1.0.0 tags followed the pi npm version (`v{pi_version}[letter]`). ## Unreleased Three small fixes found by the v1.9.4 first-boot acceptance and the CI base -hash prediction, none release-worthy on its own. Nothing here changes a pin. -`entrypoint-user.sh` is in the base hash, so the next tag rebuilds the base -(~64 min) regardless of what else it carries. +hash prediction, none release-worthy on its own, plus one boot-time wiring fix +that stops a recurring `git push` failure inside the container. Nothing here +changes a pin. `entrypoint-user.sh` is in the base hash, so the next tag +rebuilds the base (~64 min) regardless of what else it carries. ### Components that move with the next build @@ -26,6 +27,37 @@ hash prediction, none release-worthy on its own. Nothing here changes a pin. ### Fixed +- **`git push` from inside the container dies on a read-only ControlPath, and no + amount of documentation was fixing it** — `entrypoint-user.sh` now sets + `core.sshCommand` to `ssh -F $HOME/.ssh-local/config` when that sidecar exists. + `~/.ssh` is commonly bind-mounted **read-only** from the host, so a per-host + `ControlPath ~/.ssh/cm/%r@%h:%p` — the standard CGNAT multiplexing recipe, and + correct *on the host* — resolves inside an unwritable directory here. Every + push then dies `unix_listener: cannot bind to path ~/.ssh/cm/...: Read-only + file system`, hidden behind git's misleading `Please make sure you have the + correct access rights`, which sends the reader hunting for a key problem that + does not exist. `setup-lan-access.sh` already wrote the writable sidecar, whose + leading `Host *` block overrides `ControlPath` into `~/.ssh-local/cm` and only + *then* `Include`s the user's own config, so `-F` repairs the socket path while + keeping every per-host `User`/`Port`/`IdentityFile`. Measured on tor-ms22 + (2026-09-22): `ssh -G -F` yields `user gitea`, `port 22`, `egl_ed25519` + inherited from the host's own block, with a writable ControlPath; a master + socket is created and multiplexing is preserved, which is the whole point of + the host's setting. + + The reason this is code and not a doc line: the remedy was already in the + global `AGENTS.md`, in `pi-devbox-environment` SKILL.md §3, in 24 MemPalace + drawers from three devices, **and printed verbatim by + `recreate-sanity-check.sh`** — and an agent that had run that script two hours + earlier still hit the failure and reinvented a `/tmp/sshcm` workaround. A fifth + copy of the text was not the missing piece; removing the need to know was. + The `[ -r ]` guard is load-bearing: `setup-lan-access.sh` writes no sidecar on + native Linux Docker, and `-F` at a missing file would break *every* + git-over-ssh call instead of fixing one. An existing `core.sshCommand` is left + alone, matching the first-wins convention of the three git settings above it. + Host configs need no change — the same line that is right on the host is simply + unusable through a read-only mount, so the fix belongs in the container layer. + - **Boot log: `WARN: pi-extensions install.sh failed (continuing)` on every start of every device** — pi-extensions `143a214`. `/opt/pi-extensions` is root-owned and `install.sh` runs as `developer`; git refuses the repo ("dubious @@ -75,6 +107,34 @@ hash prediction, none release-worthy on its own. Nothing here changes a pin. scoped to that one `sort`, not to the shell, so it leaves `sv_SE.UTF-8` everywhere else alone. +- **`smoke-test.sh` + `recreate-sanity-check.sh` assert the sidecar wiring, and + the permanent ControlPath warning now says whether it matters** — three + assertions, each placed where it can actually pass. `smoke-test.sh` gets two + **static** greps against the image's `entrypoint-user.sh` (the wiring line, and + the `[ -r ]` guard around it), because `run` executes + `docker run --entrypoint=""`, so the entrypoint never runs there and asserting + the runtime *value* would repeat the v1.8.0 mistake of an assertion that cannot + pass, unvalidated until the next tag. Its runtime-deployment phase gets a + **biconditional**: sidecar present ⇒ `core.sshCommand` must route through it; + sidecar absent ⇒ it must be unset. Both arms are real, and the absent arm is + the one CI exercises, since a Gitea runner is native Linux Docker where + `setup-lan-access.sh` writes no sidecar — so "is set" would have failed CI for a + correct image. + + `recreate-sanity-check.sh` gets the runtime assertion proper, with an explicit + `fail` for the inverted state (`core.sshCommand` set while the sidecar is + missing — which breaks every git-over-ssh call). The existing + "default ssh precedence" warning is unchanged in severity but now states its own + scope: it is **structural and can never reach zero** while a bind-mounted + config pins `ControlPath` inside the read-only `~/.ssh`, and it reports whether + git is wired, not wired, or has no sidecar to point at. That matters because the + script's own comment already warned that "a check that fires benignly every time + is one you learn to ignore" — and on 2026-09-22 that is exactly what happened, + with the remedy sitting inside the dismissed line. All five arms were exercised + against the real script before commit; doing so caught a defect in the first + draft, which reported "git IS wired … unaffected" about a state where the + sidecar was gone and every git-over-ssh call failed. + ### Still open - **pi 0.87.x + pi-observational-memory** — unchanged from v1.9.4: blocked on diff --git a/entrypoint-user.sh b/entrypoint-user.sh index 5235049..6e1e482 100755 --- a/entrypoint-user.sh +++ b/entrypoint-user.sh @@ -319,6 +319,35 @@ fi if [ -f "$HOME/.gitignore_global" ] && ! git config --global core.excludesFile &>/dev/null; then git config --global core.excludesFile "$HOME/.gitignore_global" fi +# Route git-over-ssh through the WRITABLE ssh sidecar. ~/.ssh is commonly +# bind-mounted read-only from the host, and a per-host +# ControlPath ~/.ssh/cm/%r@%h:%p +# inherited from that config (the standard CGNAT multiplexing recipe) kills every +# push with +# unix_listener: cannot bind to path ~/.ssh/cm/...: Read-only file system +# hidden behind git's misleading "Please make sure you have the correct access +# rights", which sends the reader hunting for a key problem that does not exist. +# setup-lan-access.sh (run near the top of this script) already wrote +# ~/.ssh-local/config, whose leading `Host *` block overrides ControlPath into +# the writable ~/.ssh-local/cm and only THEN `Include`s the user's own config — +# so -F repairs the socket path while keeping every per-host User/Port/ +# IdentityFile. Wiring it here means no caller has to know any of that. +# +# WHY THIS IS NOT LEFT TO DOCUMENTATION: measured 2026-09-22 on tor-ms22, an +# agent with the remedy in its system prompt, in a loaded skill, in 24 palace +# drawers, AND printed verbatim by recreate-sanity-check.sh two hours earlier +# still hit this and reinvented a /tmp/sshcm workaround. The knowledge was +# available four times over, so a fifth copy is not the fix — removing the need +# to know is. +# +# The [ -r ] guard is load-bearing, not decoration: setup-lan-access.sh only +# writes the sidecar on VM-backed hosts (OrbStack / Docker Desktop). On native +# Linux Docker there is none, and pointing -F at a missing file would break EVERY +# git-over-ssh operation instead of fixing one. Respect a value the user already +# set — same first-wins convention as the three settings above. +if [ -r "$HOME/.ssh-local/config" ] && ! git config --global core.sshCommand &>/dev/null; then + git config --global core.sshCommand "ssh -F $HOME/.ssh-local/config" +fi # ── pi: deploy toolkit + extensions + mempalace bridge ───────────── # pi is always installed in pi-devbox; no INSTALL_PI guard needed. diff --git a/scripts/recreate-sanity-check.sh b/scripts/recreate-sanity-check.sh index 7aa049d..4d28b00 100755 --- a/scripts/recreate-sanity-check.sh +++ b/scripts/recreate-sanity-check.sh @@ -502,13 +502,33 @@ _ssh_cm_probe() { elif [ "${#bad[@]}" -eq 0 ]; then pass "$label: $n ControlMaster host(s), every ControlPath dir exists and is writable" elif [ "$sev" = "warn" ]; then - warn "$label: ${#bad[@]}/$n host(s) resolve ControlPath to a missing or unwritable dir [$shown] — expected when ~/.ssh/config pins ControlPath inside the read-only ~/.ssh; use 'ssh -F ~/.ssh-local/config' (see Dockerfile.base CAVEAT)" + warn "$label: ${#bad[@]}/$n host(s) resolve ControlPath to a missing or unwritable dir [$shown] — STRUCTURAL and permanent while ~/.ssh/config pins ControlPath inside the read-only ~/.ssh, so this line can never reach zero and is not a to-do; $_git_ssh_note (see Dockerfile.base CAVEAT)" else fail "$label: ${#bad[@]}/$n host(s) resolve ControlPath to a missing or unwritable dir [$shown] — ssh dies rc=255 'unix_listener: cannot bind to path' and the remote command never runs" fi } if command -v ssh >/dev/null 2>&1; then + # Whether git-over-ssh already routes through the sidecar decides how much the + # permanent default-route warning below actually matters, so state it IN that + # message rather than leaving each reader to work it out. Asserted properly as + # its own pass/fail in the next section. + # + # THREE states, not two, and the [ -r ] test is why: core.sshCommand NAMING the + # sidecar does not mean the sidecar EXISTS. Without that test, the arm where the + # path is wired but the file is gone printed "git IS wired ... unaffected" about + # a state in which every single git-over-ssh call fails. Found by exercising all + # five arms of this check rather than only the healthy one. + if [ ! -r "$HOME/.ssh-local/config" ]; then + _git_ssh_note="there is no ssh sidecar on this host, so nothing for -F to point at and these hosts cannot multiplex at all — see the git-over-ssh check below" + elif command -v git >/dev/null 2>&1 && + git config --global --get core.sshCommand 2>/dev/null | + grep -qF -- "-F $HOME/.ssh-local/config"; then + _git_ssh_note="git IS wired to the sidecar (core.sshCommand), so git push/fetch is unaffected; bare 'ssh' to these hosts still needs 'ssh -F ~/.ssh-local/config'" + else + _git_ssh_note="git is NOT wired to the sidecar (see the git-over-ssh check below), so both git and bare 'ssh' need 'ssh -F ~/.ssh-local/config'" + fi + # Concrete Host aliases only: patterns (*, ?) and negations (!) are not # connectable targets, so `ssh -G` on them proves nothing. _cm_cfgs=() @@ -535,6 +555,46 @@ else warn "ssh not on PATH — effective ControlPath not verified" fi +echo +echo "-- git-over-ssh routed through the writable ssh sidecar --" +# The one question in this area that is BINARY, FIXABLE, and therefore worth a +# check that can reach zero and stay there. +# +# The ControlPath warning above cannot: while a bind-mounted ~/.ssh/config pins +# ControlPath inside the read-only ~/.ssh, the default route will ALWAYS resolve +# to an unwritable dir, so that line fires benignly on every run of every device +# forever — and the script's own comment above says why that is dangerous: it is +# a warning you learn to skip. MEASURED 2026-09-22, exactly that: an agent ran +# this script, recorded "two by-design warnings", then two hours later hit +# `unix_listener: cannot bind ... Read-only file system` on `git push`, failed to +# connect it to the warning it had already read, and reinvented a /tmp/sshcm +# workaround — while the remedy string sat inside the dismissed warning. +# entrypoint-user.sh now wires core.sshCommand to the sidecar so nobody has to +# know; this section asserts the wiring actually happened, which is the part a +# reader can act on. +if ! command -v git >/dev/null 2>&1; then + warn "git not on PATH — sidecar wiring not verified" +elif [ ! -r "$HOME/.ssh-local/config" ]; then + # No sidecar (native Linux Docker, where setup-lan-access.sh writes none). The + # correct state is UNSET: -F pointing at a missing file breaks every + # git-over-ssh call, which is worse than the problem being solved. + if [ -z "$(git config --global --get core.sshCommand 2>/dev/null || true)" ]; then + pass "no ssh sidecar on this host and core.sshCommand correctly unset (guard holds)" + else + fail "core.sshCommand is set but ~/.ssh-local/config does not exist — every git-over-ssh call dies on a missing -F file; entrypoint-user.sh's [ -r ] guard did not hold" + fi +else + _git_ssh_cmd=$(git config --global --get core.sshCommand 2>/dev/null || true) + case "$_git_ssh_cmd" in + *"-F $HOME/.ssh-local/config"*) + pass "git core.sshCommand routes through the sidecar ($_git_ssh_cmd)" ;; + '') + fail "a sidecar exists but git core.sshCommand is unset — 'git push' to any host whose config pins ControlPath inside the read-only ~/.ssh dies rc=255 behind git's misleading 'correct access rights'. entrypoint-user.sh should set it; expected on images built before that wiring landed, where the fix is: git config --global core.sshCommand \"ssh -F \$HOME/.ssh-local/config\"" ;; + *) + warn "git core.sshCommand set to something else and left alone (first-wins, deliberate): $_git_ssh_cmd" ;; + esac +fi + echo echo "-- Shell defaults re-seeded from /etc/skel-devbox --" if [ -f "$HOME/.bash_aliases" ]; then diff --git a/scripts/smoke-test.sh b/scripts/smoke-test.sh index 12d272e..bba8bb1 100755 --- a/scripts/smoke-test.sh +++ b/scripts/smoke-test.sh @@ -250,6 +250,22 @@ run_expect "remote-palace-without-inbox skip is announced, not silent" \ "MemPalace catch-up skipped" run "...and the skip notice names the variable that fixes it" \ "grep -A6 'MemPalace catch-up skipped' /usr/local/bin/entrypoint-user.sh | grep -q 'MEMPALACE_PI_SSH_TARGET'" +# git-over-ssh must be wired to the writable ssh sidecar, because ~/.ssh is +# commonly bind-mounted READ-ONLY and a per-host `ControlPath ~/.ssh/cm/...` +# inherited from it kills every push with `unix_listener: cannot bind ...: +# Read-only file system` behind git's misleading "correct access rights". +# +# STATIC assertion, deliberately. `run` executes `docker run --entrypoint=""`, so +# entrypoint-user.sh never runs here and `git config --global core.sshCommand` is +# necessarily unset — asserting the VALUE in this harness would repeat the v1.8.0 +# mistake (an assertion that cannot pass, unvalidated until the next tag). What +# IS checkable at build time is that the wiring code shipped in the image. The +# runtime value is asserted below in the Runtime deployment section, where the +# real entrypoint chain has run. +run "entrypoint wires git core.sshCommand to the ssh sidecar" \ + "grep -q 'core.sshCommand \"ssh -F' /usr/local/bin/entrypoint-user.sh" +run "...and guards it on the sidecar existing (native Linux has none)" \ + "grep -B2 'core.sshCommand \"ssh -F' /usr/local/bin/entrypoint-user.sh | grep -q '\\[ -r \"\$HOME/.ssh-local/config\" \\]'" # A remote mine that FAILS must not report success. MCP answers a hard tool # failure with HTTP 200 and the tool's own JSON escaped inside # result.content[].text, so the feeder's old `'\"error\"' in body` check could @@ -659,6 +675,23 @@ exec_test "settings.json bootstrapped" 'test -f $HOME/.pi/agent/sett exec_test "pi-devbox-environment skill linked" 'test -L $HOME/.agents/skills/pi-devbox-environment && test -f $HOME/.agents/skills/pi-devbox-environment/SKILL.md && echo ok' exec_test "pi-extensions skill linked (fallback)" 'test -L $HOME/.agents/skills/pi-extensions && test -f $HOME/.agents/skills/pi-extensions/SKILL.md && echo ok' exec_test "mempalace skill linked (fallback)" 'test -L $HOME/.agents/skills/mempalace && test -f $HOME/.agents/skills/mempalace/SKILL.md && echo ok' +# git-over-ssh sidecar wiring, asserted as a BICONDITIONAL rather than "is set". +# setup-lan-access.sh writes ~/.ssh-local/config only on VM-backed hosts, and a +# CI runner is native Linux Docker — so "core.sshCommand is set" would fail here +# for a correct image, which is precisely the v1.8.0 trap (an assertion whose +# environment was never checked, unvalidated until the next tag). Both arms are +# real: sidecar present => must route through it; sidecar absent => must be UNSET, +# because -F pointing at a missing file breaks every git-over-ssh call and is +# worse than the problem being fixed. This arm is the one CI actually exercises, +# so CI validates the guard; the other is covered by the static greps above and +# by scripts/recreate-sanity-check.sh on a real device. +exec_test "git core.sshCommand matches sidecar presence" ' + cmd=$(git config --global --get core.sshCommand 2>/dev/null || true) + if [ -r "$HOME/.ssh-local/config" ]; then + case "$cmd" in *"-F $HOME/.ssh-local/config"*) echo "wired: $cmd" ;; *) exit 1 ;; esac + else + [ -z "$cmd" ] || exit 1; echo "no sidecar, correctly unset" + fi' # The vendored mempalace snapshot is refreshed MANUALLY per release (see # rootfs/usr/local/share/pi-devbox/skills/VENDORED.md). Through v1.8.4 it also # silently SHADOWED the live skillset copy, so staleness was invisible — and the