diff --git a/CHANGELOG.md b/CHANGELOG.md index abd126d..4b03798 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,67 @@ Pre-v1.0.0 tags followed the pi npm version (`v{pi_version}[letter]`). --- +## Unreleased + +### Added + +- **The SSH sidecar now defaults to connection multiplexing, without overriding + anyone's explicit choice.** `~/.ssh-local/config` already forced `ControlPath` + into the writable sidecar dir, but nothing supplied `ControlMaster` for targets + coming from the user's own bind-mounted `~/.ssh/config`. An entry that never + mentioned it therefore opened a **fresh TCP connection per `ssh` call** — and + an agent doing a dozen calls in a few minutes is exactly the traffic shape that + trips fail2ban or a CGNAT flow-table cap. Observed 2026-08-25 on this fleet: + ~12 connections to one host in 15 minutes, after which port 22 stopped + answering while HTTPS to the same estate stayed healthy in 0.44 s (that + asymmetry is the tell for rate-limiting rather than an outage). + + **The fix is where the block sits, not what it says.** `ssh_config` is + first-value-wins, so position encodes intent, and the two settings need + opposite treatment: + + | Setting | Position | Meaning | Why | + |---|---|---|---| + | `ControlPath` | **before** `Include ~/.ssh/config` | override | the user's value points at read-only `~/.ssh`; it cannot work here, so it must lose | + | `ControlMaster auto` + `ControlPersist 10m` | **after** the `Include` | default | an explicit per-host `ControlMaster no` must keep winning; we only supply an opinion where the user expressed none | + + *Force what is broken, default what is merely absent.* The first draft of this + put both in the leading block, which would have silently overridden an explicit + `ControlMaster no` — the counterfactual is in the test below. + + Verified with `ssh -G` (the resolved-config oracle) rather than by reading the + man page, against a fixture with one host set to `no`, one silent, one set to + `auto`: the explicit `no` resolves to `controlmaster false` **and** still gets + the writable `ControlPath`, the silent host resolves to `auto`, and the same + fixture under the rejected layout flips the `no` host to `auto` — so the test + discriminates the *position*, not merely the presence of the block. Then + end-to-end: the real script rendered in a sandbox `HOME`, block last, `bash -n` + clean, `shellcheck -S error` clean (the gate added in v1.8.7). + + Measured effect on the author's own config (41 host aliases): **22 were silent + about `ControlMaster` and gain `auto` + 10 m persist; 0 are overridden**, since + the fleet contains no explicit `no`. Worth noting *how* the one deliberate + exception is written — `proxmox002-vpn` carries `# No ControlMaster — VPN means + direct route, no CGNAT flow cap`, i.e. the intent is expressed as **absence + plus a comment**, which `ssh` cannot distinguish from "no opinion". That host + does now get multiplexing; its comment says multiplexing is *unnecessary* + there, not harmful. Anything that must stay unmultiplexed needs a literal + `ControlMaster no`. + + Why the ordering matters beyond this one config: `~/.ssh/config` is + **per-machine**, differs across the fleet, and future machines' versions do not + exist yet to be audited. A default-not-override design is correct without + needing to inspect any of them. + + `ControlPersist` is deliberately short (10 m idle, and each new session resets + the idle timer — long enough to collapse an agent's burst, short enough that an + abandoned socket ages out). A per-host entry that sets its own value keeps it: + hosts already specifying `ControlPersist 4h` still resolve to 4 h. The known + cost of multiplexing is the **stale master** — socket present, daemon gone, + after a suspend or network change — which makes every later `ssh` to that host + hang; recovery is `ssh -F ~/.ssh-local/config -O exit `, now documented + in the `pi-devbox-environment` skill along with `-O check`. + ## v1.8.7 — 2026-08-25 Patch release, and the fastest turnaround in the series (~9 h after v1.8.6) for diff --git a/rootfs/usr/local/lib/pi-devbox/setup-lan-access.sh b/rootfs/usr/local/lib/pi-devbox/setup-lan-access.sh index 06ef5cd..ae847c7 100755 --- a/rootfs/usr/local/lib/pi-devbox/setup-lan-access.sh +++ b/rootfs/usr/local/lib/pi-devbox/setup-lan-access.sh @@ -197,6 +197,45 @@ EOF ) fi +# ── Multiplexing default, deliberately LAST ─────────────────────────── +# Why this block exists: ControlPath above is forced, but ControlMaster is not +# set anywhere for targets that come from the user's own ~/.ssh/config. A target +# whose entry omits ControlMaster therefore opens a NEW TCP connection per ssh +# call, and an agent doing a dozen calls in a few minutes can trip fail2ban or a +# CGNAT flow-table cap on the far end — observed 2026-08-25: ~12 connections in +# 15 min and port 22 stopped answering while HTTPS to the same estate stayed fine. +# +# WHY IT IS AT THE BOTTOM, and ControlPath is at the top. ssh_config is +# first-value-wins, so position encodes intent: +# * BEFORE the Include = an OVERRIDE. Correct for ControlPath, whose value in +# the user's config points at read-only ~/.ssh and simply cannot work here. +# * AFTER the Include = a DEFAULT. Correct for ControlMaster, because an +# explicit per-host 'ControlMaster no' (or 'auto', or any value) in the +# user's own config must keep winning. We are supplying an opinion only +# where the user expressed none. +# That asymmetry is the whole design: force what is broken, default what is +# merely absent. It also means this needs no audit of anyone's ~/.ssh/config — +# which matters because that file is per-machine, differs across the fleet, and +# future machines' versions do not exist yet to be audited. +# +# Caveat worth knowing (and documented in the pi-devbox-environment skill): a +# stale master socket — file present, daemon gone, e.g. after the host suspends +# or changes network — makes every later ssh to that host hang. Recovery is +# 'ssh -F ~/.ssh-local/config -O exit '. ControlPersist is deliberately +# short (10m idle, and each new session resets the idle timer) so an abandoned +# socket ages out on its own rather than lingering for hours. +MULTIPLEX_DEFAULT_BLOCK=$(cat <<'EOF' + +# Multiplexing DEFAULT — intentionally after the Include above, so any explicit +# per-host ControlMaster in your own ~/.ssh/config still wins (first-value-wins). +# Applies only to targets that never mentioned ControlMaster at all. +# Stale socket after a suspend/network change? ssh -O exit . +Host * + ControlMaster auto + ControlPersist 10m +EOF +) + cat > "$CONFIG" < @@ -216,6 +255,7 @@ ${JUMP_BLOCK} ${LAN_CONF_BLOCK} ${AUTOJUMP_BLOCK} ${INCLUDE_BLOCK} +${MULTIPLEX_DEFAULT_BLOCK} EOF chmod 600 "$CONFIG" 2>/dev/null || true diff --git a/rootfs/usr/local/share/pi-devbox/skills/pi-devbox-environment/SKILL.md b/rootfs/usr/local/share/pi-devbox/skills/pi-devbox-environment/SKILL.md index c7e79ff..d4eb836 100644 --- a/rootfs/usr/local/share/pi-devbox/skills/pi-devbox-environment/SKILL.md +++ b/rootfs/usr/local/share/pi-devbox/skills/pi-devbox-environment/SKILL.md @@ -185,6 +185,16 @@ entrypoint's `setup-lan-access.sh` writes a **writable SSH sidecar** at - A `Host *` block redirecting `ControlPath` into the writable `~/.ssh-local/cm` (because `~/.ssh` is typically bind-mounted **read-only**, so a master socket can't be created under it), plus `Include ~/.ssh/config`. +- A **trailing** `Host *` block supplying `ControlMaster auto` + `ControlPersist + 10m` as a *default*. Position is the design: `ControlPath` sits **before** the + `Include` (an override — the value in your own config points at read-only + `~/.ssh` and cannot work here), while `ControlMaster` sits **after** it (a + default — an explicit per-host `ControlMaster no`/`auto` in your own config + still wins, because ssh_config is first-value-wins). **Force what is broken, + default what is merely absent.** Without this, a target whose entry never + mentioned `ControlMaster` opens a fresh TCP connection per `ssh` call, and an + agent making a dozen calls in a few minutes can trip fail2ban or a CGNAT + flow-table cap on the far end. - Aliases **`host` / `mac`** → `host.docker.internal` (user comes from `HOST_SSH_USER`) — i.e. SSH back into the Docker host. - On VM-backed hosts only: an **SSH-jump-via-host** block so the container can @@ -199,6 +209,25 @@ ssh -F "$HOME/.ssh-local/config" mac 'hostname; whoami' # reach the host ssh -F "$HOME/.ssh-local/config" '…' # reach a LAN peer (if configured) ``` +**Always go through the sidecar, never `-F ~/.ssh/config`.** This is the single +easiest way to break SSH from inside the container, and the failure actively +misleads: the read-only path makes the master socket uncreatable, so +multiplexing appears *impossible* rather than misconfigured. What follows is a +burst of fresh connections and, on a rate-limiting peer, a block that looks like +an outage. The tell that it is rate-limiting and not an outage: HTTPS to the same +estate keeps working while port 22 stops answering. (Recorded 2026-08-25 — an +agent hit exactly this, concluded "ControlMaster is impossible here", disabled +multiplexing, and filed that as a lesson. The sidecar had solved it since v1.4.) + +If every `ssh` to one host suddenly hangs, suspect a **stale master** — socket +file present, daemon gone, typically after the host suspended or changed +network. Check and clear it: + +```sh +ssh -F "$HOME/.ssh-local/config" -O check # "Master running (pid=…)" or no master +ssh -F "$HOME/.ssh-local/config" -O exit # tear down a stale one +``` + Two related mechanisms (don't reinvent them): - **ControlMaster multiplexing** is preconfigured (`/tmp/sshcm/`) to survive