fix(ssh): wire git core.sshCommand to the writable sidecar, and assert it

~/.ssh is commonly bind-mounted READ-ONLY from the host, so a per-host
`ControlPath ~/.ssh/cm/%r@%h:%p` — the standard CGNAT multiplexing recipe, and
correct on the host — resolves inside an unwritable dir in the container. Every
push dies `unix_listener: cannot bind to path ...: Read-only file system`,
behind git's misleading "make sure you have the correct access rights".

setup-lan-access.sh already writes the fix: ~/.ssh-local/config overrides
ControlPath into the writable ~/.ssh-local/cm BEFORE `Include ~/.ssh/config`,
so -F repairs the socket path and keeps every per-host User/Port/IdentityFile.
entrypoint-user.sh now points git at it, guarded on the sidecar existing —
setup-lan-access.sh writes none on native Linux Docker, where -F at a missing
file would break every git-over-ssh call instead of fixing one. An existing
core.sshCommand is left alone (first-wins, as for the three git settings above).

Why code and not another doc line: the remedy was already in the global
AGENTS.md, in pi-devbox-environment SKILL.md §3, in 24 MemPalace drawers from
three devices, and printed verbatim by recreate-sanity-check.sh — and an agent
that had run that script two hours earlier still hit the failure and reinvented
a /tmp/sshcm workaround. A fifth copy was not the missing piece.

Assertions, each where it can actually pass:
- smoke-test.sh: two STATIC greps (wiring line + its [ -r ] guard). `run` uses
  --entrypoint="", so asserting the runtime value there would repeat the v1.8.0
  mistake of an assertion that cannot pass, unvalidated until the next tag.
- smoke-test.sh runtime phase: a BICONDITIONAL — sidecar present => must route
  through it; absent => must be unset. The absent arm is the one CI exercises
  (native Linux runner), so "is set" would have failed CI for a correct image.
- recreate-sanity-check.sh: the runtime assertion, plus an explicit fail for the
  inverted state (set while the sidecar is missing). The permanent "default ssh
  precedence" warning keeps its severity but now states that it is structural and
  can never reach zero, and whether git is wired, unwired, or has no sidecar.

All five arms exercised against the real script before commit; that caught a
defect in the first draft, which reported "git IS wired ... unaffected" about a
state where the sidecar was gone and every git-over-ssh call failed.

Host ~/.ssh/config needs no change: the same line is right on the host and
unusable through a read-only mount, so the fix belongs in the container layer.

CHANGELOG.md

@@ -14,9 +14,10 @@ Pre-v1.0.0 tags followed the pi npm version (`v{pi_version}[letter]`).
 ## Unreleased
 
 Three small fixes found by the v1.9.4 first-boot acceptance and the CI base
-hash prediction, none release-worthy on its own. Nothing here changes a pin.
+hash prediction, none release-worthy on its own, plus one boot-time wiring fix
-`entrypoint-user.sh` is in the base hash, so the next tag rebuilds the base
+that stops a recurring `git push` failure inside the container. Nothing here
-(~64 min) regardless of what else it carries.
+changes a pin. `entrypoint-user.sh` is in the base hash, so the next tag
+rebuilds the base (~64 min) regardless of what else it carries.
 
 ### Components that move with the next build
 
@@ -26,6 +27,37 @@ hash prediction, none release-worthy on its own. Nothing here changes a pin.
 
 ### Fixed
 
+- **`git push` from inside the container dies on a read-only ControlPath, and no
+  amount of documentation was fixing it** — `entrypoint-user.sh` now sets
+  `core.sshCommand` to `ssh -F $HOME/.ssh-local/config` when that sidecar exists.
+  `~/.ssh` is commonly bind-mounted **read-only** from the host, so a per-host
+  `ControlPath ~/.ssh/cm/%r@%h:%p` — the standard CGNAT multiplexing recipe, and
+  correct *on the host* — resolves inside an unwritable directory here. Every
+  push then dies `unix_listener: cannot bind to path ~/.ssh/cm/...: Read-only
+  file system`, hidden behind git's misleading `Please make sure you have the
+  correct access rights`, which sends the reader hunting for a key problem that
+  does not exist. `setup-lan-access.sh` already wrote the writable sidecar, whose
+  leading `Host *` block overrides `ControlPath` into `~/.ssh-local/cm` and only
+  *then* `Include`s the user's own config, so `-F` repairs the socket path while
+  keeping every per-host `User`/`Port`/`IdentityFile`. Measured on tor-ms22
+  (2026-09-22): `ssh -G -F` yields `user gitea`, `port 22`, `egl_ed25519`
+  inherited from the host's own block, with a writable ControlPath; a master
+  socket is created and multiplexing is preserved, which is the whole point of
+  the host's setting.
+
+  The reason this is code and not a doc line: the remedy was already in the
+  global `AGENTS.md`, in `pi-devbox-environment` SKILL.md §3, in 24 MemPalace
+  drawers from three devices, **and printed verbatim by
+  `recreate-sanity-check.sh`** — and an agent that had run that script two hours
+  earlier still hit the failure and reinvented a `/tmp/sshcm` workaround. A fifth
+  copy of the text was not the missing piece; removing the need to know was.
+  The `[ -r ]` guard is load-bearing: `setup-lan-access.sh` writes no sidecar on
+  native Linux Docker, and `-F` at a missing file would break *every*
+  git-over-ssh call instead of fixing one. An existing `core.sshCommand` is left
+  alone, matching the first-wins convention of the three git settings above it.
+  Host configs need no change — the same line that is right on the host is simply
+  unusable through a read-only mount, so the fix belongs in the container layer.
+
 - **Boot log: `WARN: pi-extensions install.sh failed (continuing)` on every start
   of every device** — pi-extensions `143a214`. `/opt/pi-extensions` is root-owned
   and `install.sh` runs as `developer`; git refuses the repo ("dubious
@@ -75,6 +107,34 @@ hash prediction, none release-worthy on its own. Nothing here changes a pin.
   scoped to that one `sort`, not to the shell, so it leaves `sv_SE.UTF-8`
   everywhere else alone.
 
+- **`smoke-test.sh` + `recreate-sanity-check.sh` assert the sidecar wiring, and
+  the permanent ControlPath warning now says whether it matters** — three
+  assertions, each placed where it can actually pass. `smoke-test.sh` gets two
+  **static** greps against the image's `entrypoint-user.sh` (the wiring line, and
+  the `[ -r ]` guard around it), because `run` executes
+  `docker run --entrypoint=""`, so the entrypoint never runs there and asserting
+  the runtime *value* would repeat the v1.8.0 mistake of an assertion that cannot
+  pass, unvalidated until the next tag. Its runtime-deployment phase gets a
+  **biconditional**: sidecar present ⇒ `core.sshCommand` must route through it;
+  sidecar absent ⇒ it must be unset. Both arms are real, and the absent arm is
+  the one CI exercises, since a Gitea runner is native Linux Docker where
+  `setup-lan-access.sh` writes no sidecar — so "is set" would have failed CI for a
+  correct image.
+
+  `recreate-sanity-check.sh` gets the runtime assertion proper, with an explicit
+  `fail` for the inverted state (`core.sshCommand` set while the sidecar is
+  missing — which breaks every git-over-ssh call). The existing
+  "default ssh precedence" warning is unchanged in severity but now states its own
+  scope: it is **structural and can never reach zero** while a bind-mounted
+  config pins `ControlPath` inside the read-only `~/.ssh`, and it reports whether
+  git is wired, not wired, or has no sidecar to point at. That matters because the
+  script's own comment already warned that "a check that fires benignly every time
+  is one you learn to ignore" — and on 2026-09-22 that is exactly what happened,
+  with the remedy sitting inside the dismissed line. All five arms were exercised
+  against the real script before commit; doing so caught a defect in the first
+  draft, which reported "git IS wired … unaffected" about a state where the
+  sidecar was gone and every git-over-ssh call failed.
+
 ### Still open
 
 - **pi 0.87.x + pi-observational-memory** — unchanged from v1.9.4: blocked on

entrypoint-user.sh

@@ -319,6 +319,35 @@ fi
 if [ -f "$HOME/.gitignore_global" ] && ! git config --global core.excludesFile &>/dev/null; then
   git config --global core.excludesFile "$HOME/.gitignore_global"
 fi
+# Route git-over-ssh through the WRITABLE ssh sidecar. ~/.ssh is commonly
+# bind-mounted read-only from the host, and a per-host
+#   ControlPath ~/.ssh/cm/%r@%h:%p
+# inherited from that config (the standard CGNAT multiplexing recipe) kills every
+# push with
+#   unix_listener: cannot bind to path ~/.ssh/cm/...: Read-only file system
+# hidden behind git's misleading "Please make sure you have the correct access
+# rights", which sends the reader hunting for a key problem that does not exist.
+# setup-lan-access.sh (run near the top of this script) already wrote
+# ~/.ssh-local/config, whose leading `Host *` block overrides ControlPath into
+# the writable ~/.ssh-local/cm and only THEN `Include`s the user's own config —
+# so -F repairs the socket path while keeping every per-host User/Port/
+# IdentityFile. Wiring it here means no caller has to know any of that.
+#
+# WHY THIS IS NOT LEFT TO DOCUMENTATION: measured 2026-09-22 on tor-ms22, an
+# agent with the remedy in its system prompt, in a loaded skill, in 24 palace
+# drawers, AND printed verbatim by recreate-sanity-check.sh two hours earlier
+# still hit this and reinvented a /tmp/sshcm workaround. The knowledge was
+# available four times over, so a fifth copy is not the fix — removing the need
+# to know is.
+#
+# The [ -r ] guard is load-bearing, not decoration: setup-lan-access.sh only
+# writes the sidecar on VM-backed hosts (OrbStack / Docker Desktop). On native
+# Linux Docker there is none, and pointing -F at a missing file would break EVERY
+# git-over-ssh operation instead of fixing one. Respect a value the user already
+# set — same first-wins convention as the three settings above.
+if [ -r "$HOME/.ssh-local/config" ] && ! git config --global core.sshCommand &>/dev/null; then
+  git config --global core.sshCommand "ssh -F $HOME/.ssh-local/config"
+fi
 
 # ── pi: deploy toolkit + extensions + mempalace bridge ─────────────
 # pi is always installed in pi-devbox; no INSTALL_PI guard needed.

scripts/recreate-sanity-check.sh

@@ -502,13 +502,33 @@ _ssh_cm_probe() {
   elif [ "${#bad[@]}" -eq 0 ]; then
     pass "$label: $n ControlMaster host(s), every ControlPath dir exists and is writable"
   elif [ "$sev" = "warn" ]; then
-    warn "$label: ${#bad[@]}/$n host(s) resolve ControlPath to a missing or unwritable dir [$shown] — expected when ~/.ssh/config pins ControlPath inside the read-only ~/.ssh; use 'ssh -F ~/.ssh-local/config' (see Dockerfile.base CAVEAT)"
+    warn "$label: ${#bad[@]}/$n host(s) resolve ControlPath to a missing or unwritable dir [$shown] — STRUCTURAL and permanent while ~/.ssh/config pins ControlPath inside the read-only ~/.ssh, so this line can never reach zero and is not a to-do; $_git_ssh_note (see Dockerfile.base CAVEAT)"
   else
     fail "$label: ${#bad[@]}/$n host(s) resolve ControlPath to a missing or unwritable dir [$shown] — ssh dies rc=255 'unix_listener: cannot bind to path' and the remote command never runs"
   fi
 }
 
 if command -v ssh >/dev/null 2>&1; then
+  # Whether git-over-ssh already routes through the sidecar decides how much the
+  # permanent default-route warning below actually matters, so state it IN that
+  # message rather than leaving each reader to work it out. Asserted properly as
+  # its own pass/fail in the next section.
+  #
+  # THREE states, not two, and the [ -r ] test is why: core.sshCommand NAMING the
+  # sidecar does not mean the sidecar EXISTS. Without that test, the arm where the
+  # path is wired but the file is gone printed "git IS wired ... unaffected" about
+  # a state in which every single git-over-ssh call fails. Found by exercising all
+  # five arms of this check rather than only the healthy one.
+  if [ ! -r "$HOME/.ssh-local/config" ]; then
+    _git_ssh_note="there is no ssh sidecar on this host, so nothing for -F to point at and these hosts cannot multiplex at all — see the git-over-ssh check below"
+  elif command -v git >/dev/null 2>&1 &&
+       git config --global --get core.sshCommand 2>/dev/null |
+         grep -qF -- "-F $HOME/.ssh-local/config"; then
+    _git_ssh_note="git IS wired to the sidecar (core.sshCommand), so git push/fetch is unaffected; bare 'ssh' to these hosts still needs 'ssh -F ~/.ssh-local/config'"
+  else
+    _git_ssh_note="git is NOT wired to the sidecar (see the git-over-ssh check below), so both git and bare 'ssh' need 'ssh -F ~/.ssh-local/config'"
+  fi
+
   # Concrete Host aliases only: patterns (*, ?) and negations (!) are not
   # connectable targets, so `ssh -G` on them proves nothing.
   _cm_cfgs=()
@@ -535,6 +555,46 @@ else
   warn "ssh not on PATH — effective ControlPath not verified"
 fi
 
+echo
+echo "-- git-over-ssh routed through the writable ssh sidecar --"
+# The one question in this area that is BINARY, FIXABLE, and therefore worth a
+# check that can reach zero and stay there.
+#
+# The ControlPath warning above cannot: while a bind-mounted ~/.ssh/config pins
+# ControlPath inside the read-only ~/.ssh, the default route will ALWAYS resolve
+# to an unwritable dir, so that line fires benignly on every run of every device
+# forever — and the script's own comment above says why that is dangerous: it is
+# a warning you learn to skip. MEASURED 2026-09-22, exactly that: an agent ran
+# this script, recorded "two by-design warnings", then two hours later hit
+# `unix_listener: cannot bind ... Read-only file system` on `git push`, failed to
+# connect it to the warning it had already read, and reinvented a /tmp/sshcm
+# workaround — while the remedy string sat inside the dismissed warning.
+# entrypoint-user.sh now wires core.sshCommand to the sidecar so nobody has to
+# know; this section asserts the wiring actually happened, which is the part a
+# reader can act on.
+if ! command -v git >/dev/null 2>&1; then
+  warn "git not on PATH — sidecar wiring not verified"
+elif [ ! -r "$HOME/.ssh-local/config" ]; then
+  # No sidecar (native Linux Docker, where setup-lan-access.sh writes none). The
+  # correct state is UNSET: -F pointing at a missing file breaks every
+  # git-over-ssh call, which is worse than the problem being solved.
+  if [ -z "$(git config --global --get core.sshCommand 2>/dev/null || true)" ]; then
+    pass "no ssh sidecar on this host and core.sshCommand correctly unset (guard holds)"
+  else
+    fail "core.sshCommand is set but ~/.ssh-local/config does not exist — every git-over-ssh call dies on a missing -F file; entrypoint-user.sh's [ -r ] guard did not hold"
+  fi
+else
+  _git_ssh_cmd=$(git config --global --get core.sshCommand 2>/dev/null || true)
+  case "$_git_ssh_cmd" in
+    *"-F $HOME/.ssh-local/config"*)
+      pass "git core.sshCommand routes through the sidecar ($_git_ssh_cmd)" ;;
+    '')
+      fail "a sidecar exists but git core.sshCommand is unset — 'git push' to any host whose config pins ControlPath inside the read-only ~/.ssh dies rc=255 behind git's misleading 'correct access rights'. entrypoint-user.sh should set it; expected on images built before that wiring landed, where the fix is: git config --global core.sshCommand \"ssh -F \$HOME/.ssh-local/config\"" ;;
+    *)
+      warn "git core.sshCommand set to something else and left alone (first-wins, deliberate): $_git_ssh_cmd" ;;
+  esac
+fi
+
 echo
 echo "-- Shell defaults re-seeded from /etc/skel-devbox --"
 if [ -f "$HOME/.bash_aliases" ]; then

scripts/smoke-test.sh

@@ -250,6 +250,22 @@ run_expect "remote-palace-without-inbox skip is announced, not silent" \
     "MemPalace catch-up skipped"
 run "...and the skip notice names the variable that fixes it" \
     "grep -A6 'MemPalace catch-up skipped' /usr/local/bin/entrypoint-user.sh | grep -q 'MEMPALACE_PI_SSH_TARGET'"
+# git-over-ssh must be wired to the writable ssh sidecar, because ~/.ssh is
+# commonly bind-mounted READ-ONLY and a per-host `ControlPath ~/.ssh/cm/...`
+# inherited from it kills every push with `unix_listener: cannot bind ...:
+# Read-only file system` behind git's misleading "correct access rights".
+#
+# STATIC assertion, deliberately. `run` executes `docker run --entrypoint=""`, so
+# entrypoint-user.sh never runs here and `git config --global core.sshCommand` is
+# necessarily unset — asserting the VALUE in this harness would repeat the v1.8.0
+# mistake (an assertion that cannot pass, unvalidated until the next tag). What
+# IS checkable at build time is that the wiring code shipped in the image. The
+# runtime value is asserted below in the Runtime deployment section, where the
+# real entrypoint chain has run.
+run "entrypoint wires git core.sshCommand to the ssh sidecar" \
+    "grep -q 'core.sshCommand \"ssh -F' /usr/local/bin/entrypoint-user.sh"
+run "...and guards it on the sidecar existing (native Linux has none)" \
+    "grep -B2 'core.sshCommand \"ssh -F' /usr/local/bin/entrypoint-user.sh | grep -q '\\[ -r \"\$HOME/.ssh-local/config\" \\]'"
 # A remote mine that FAILS must not report success. MCP answers a hard tool
 # failure with HTTP 200 and the tool's own JSON escaped inside
 # result.content[].text, so the feeder's old `'\"error\"' in body` check could
@@ -659,6 +675,23 @@ exec_test "settings.json bootstrapped"             'test -f $HOME/.pi/agent/sett
 exec_test "pi-devbox-environment skill linked"     'test -L $HOME/.agents/skills/pi-devbox-environment && test -f $HOME/.agents/skills/pi-devbox-environment/SKILL.md && echo ok'
 exec_test "pi-extensions skill linked (fallback)"  'test -L $HOME/.agents/skills/pi-extensions && test -f $HOME/.agents/skills/pi-extensions/SKILL.md && echo ok'
 exec_test "mempalace skill linked (fallback)"      'test -L $HOME/.agents/skills/mempalace && test -f $HOME/.agents/skills/mempalace/SKILL.md && echo ok'
+# git-over-ssh sidecar wiring, asserted as a BICONDITIONAL rather than "is set".
+# setup-lan-access.sh writes ~/.ssh-local/config only on VM-backed hosts, and a
+# CI runner is native Linux Docker — so "core.sshCommand is set" would fail here
+# for a correct image, which is precisely the v1.8.0 trap (an assertion whose
+# environment was never checked, unvalidated until the next tag). Both arms are
+# real: sidecar present => must route through it; sidecar absent => must be UNSET,
+# because -F pointing at a missing file breaks every git-over-ssh call and is
+# worse than the problem being fixed. This arm is the one CI actually exercises,
+# so CI validates the guard; the other is covered by the static greps above and
+# by scripts/recreate-sanity-check.sh on a real device.
+exec_test "git core.sshCommand matches sidecar presence" '
+  cmd=$(git config --global --get core.sshCommand 2>/dev/null || true)
+  if [ -r "$HOME/.ssh-local/config" ]; then
+    case "$cmd" in *"-F $HOME/.ssh-local/config"*) echo "wired: $cmd" ;; *) exit 1 ;; esac
+  else
+    [ -z "$cmd" ] || exit 1; echo "no sidecar, correctly unset"
+  fi'
 # The vendored mempalace snapshot is refreshed MANUALLY per release (see
 # rootfs/usr/local/share/pi-devbox/skills/VENDORED.md). Through v1.8.4 it also
 # silently SHADOWED the live skillset copy, so staleness was invisible — and the