From 657b1ad856848276cfeade84bcdf6590b3ae94e6 Mon Sep 17 00:00:00 2001 From: pi Date: Tue, 25 Aug 2026 23:09:46 +0200 Subject: [PATCH] ssh sidecar: default to multiplexing, as a default and not an override MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit A target whose ~/.ssh/config entry never mentioned ControlMaster got no multiplexing from the sidecar (only ControlPath was supplied), so every ssh call opened a fresh TCP connection. On 2026-08-25 that produced ~12 connections to one host in 15 min and a fail2ban block that looked like an outage — the tell being that HTTPS to the same estate stayed healthy. The correctness of this depends entirely on WHERE the block goes. ssh_config is first-value-wins: ControlPath before the Include -> override (the user's value points at read-only ~/.ssh and cannot work in the container) ControlMaster after the Include -> default (an explicit per-host 'ControlMaster no' must keep winning) Force what is broken, default what is merely absent. The first draft put both in the leading block and would have silently overridden an explicit 'no'. Verified with ssh -G rather than from the man page, including the counterfactual: under the shipped layout an explicit 'no' resolves to controlmaster false while a silent host resolves to auto; under the rejected layout the 'no' host flips to auto. So the test discriminates position, not presence. Plus a sandbox render of the real script, bash -n, and shellcheck -S error (the v1.8.7 gate) clean. Effect measured on 41 real host aliases: 22 silent entries gain auto+10m, 0 overridden. Note the fleet's one deliberate opt-out is written as absence plus a comment ('# No ControlMaster — VPN means direct route'), which ssh cannot distinguish from no opinion; that host now multiplexes, which its own comment says is unnecessary rather than harmful. Skill documents the sidecar-vs-~/.ssh trap (the failure misleads: read-only ControlPath makes multiplexing look impossible rather than misconfigured) and the stale-master recovery, ssh -O check / -O exit. --- CHANGELOG.md | 61 +++++++++++++++++++ .../local/lib/pi-devbox/setup-lan-access.sh | 40 ++++++++++++ .../skills/pi-devbox-environment/SKILL.md | 29 +++++++++ 3 files changed, 130 insertions(+) diff --git a/CHANGELOG.md b/CHANGELOG.md index abd126d..4b03798 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -11,6 +11,67 @@ Pre-v1.0.0 tags followed the pi npm version (`v{pi_version}[letter]`). --- +## Unreleased + +### Added + +- **The SSH sidecar now defaults to connection multiplexing, without overriding + anyone's explicit choice.** `~/.ssh-local/config` already forced `ControlPath` + into the writable sidecar dir, but nothing supplied `ControlMaster` for targets + coming from the user's own bind-mounted `~/.ssh/config`. An entry that never + mentioned it therefore opened a **fresh TCP connection per `ssh` call** — and + an agent doing a dozen calls in a few minutes is exactly the traffic shape that + trips fail2ban or a CGNAT flow-table cap. Observed 2026-08-25 on this fleet: + ~12 connections to one host in 15 minutes, after which port 22 stopped + answering while HTTPS to the same estate stayed healthy in 0.44 s (that + asymmetry is the tell for rate-limiting rather than an outage). + + **The fix is where the block sits, not what it says.** `ssh_config` is + first-value-wins, so position encodes intent, and the two settings need + opposite treatment: + + | Setting | Position | Meaning | Why | + |---|---|---|---| + | `ControlPath` | **before** `Include ~/.ssh/config` | override | the user's value points at read-only `~/.ssh`; it cannot work here, so it must lose | + | `ControlMaster auto` + `ControlPersist 10m` | **after** the `Include` | default | an explicit per-host `ControlMaster no` must keep winning; we only supply an opinion where the user expressed none | + + *Force what is broken, default what is merely absent.* The first draft of this + put both in the leading block, which would have silently overridden an explicit + `ControlMaster no` — the counterfactual is in the test below. + + Verified with `ssh -G` (the resolved-config oracle) rather than by reading the + man page, against a fixture with one host set to `no`, one silent, one set to + `auto`: the explicit `no` resolves to `controlmaster false` **and** still gets + the writable `ControlPath`, the silent host resolves to `auto`, and the same + fixture under the rejected layout flips the `no` host to `auto` — so the test + discriminates the *position*, not merely the presence of the block. Then + end-to-end: the real script rendered in a sandbox `HOME`, block last, `bash -n` + clean, `shellcheck -S error` clean (the gate added in v1.8.7). + + Measured effect on the author's own config (41 host aliases): **22 were silent + about `ControlMaster` and gain `auto` + 10 m persist; 0 are overridden**, since + the fleet contains no explicit `no`. Worth noting *how* the one deliberate + exception is written — `proxmox002-vpn` carries `# No ControlMaster — VPN means + direct route, no CGNAT flow cap`, i.e. the intent is expressed as **absence + plus a comment**, which `ssh` cannot distinguish from "no opinion". That host + does now get multiplexing; its comment says multiplexing is *unnecessary* + there, not harmful. Anything that must stay unmultiplexed needs a literal + `ControlMaster no`. + + Why the ordering matters beyond this one config: `~/.ssh/config` is + **per-machine**, differs across the fleet, and future machines' versions do not + exist yet to be audited. A default-not-override design is correct without + needing to inspect any of them. + + `ControlPersist` is deliberately short (10 m idle, and each new session resets + the idle timer — long enough to collapse an agent's burst, short enough that an + abandoned socket ages out). A per-host entry that sets its own value keeps it: + hosts already specifying `ControlPersist 4h` still resolve to 4 h. The known + cost of multiplexing is the **stale master** — socket present, daemon gone, + after a suspend or network change — which makes every later `ssh` to that host + hang; recovery is `ssh -F ~/.ssh-local/config -O exit `, now documented + in the `pi-devbox-environment` skill along with `-O check`. + ## v1.8.7 — 2026-08-25 Patch release, and the fastest turnaround in the series (~9 h after v1.8.6) for diff --git a/rootfs/usr/local/lib/pi-devbox/setup-lan-access.sh b/rootfs/usr/local/lib/pi-devbox/setup-lan-access.sh index 06ef5cd..ae847c7 100755 --- a/rootfs/usr/local/lib/pi-devbox/setup-lan-access.sh +++ b/rootfs/usr/local/lib/pi-devbox/setup-lan-access.sh @@ -197,6 +197,45 @@ EOF ) fi +# ── Multiplexing default, deliberately LAST ─────────────────────────── +# Why this block exists: ControlPath above is forced, but ControlMaster is not +# set anywhere for targets that come from the user's own ~/.ssh/config. A target +# whose entry omits ControlMaster therefore opens a NEW TCP connection per ssh +# call, and an agent doing a dozen calls in a few minutes can trip fail2ban or a +# CGNAT flow-table cap on the far end — observed 2026-08-25: ~12 connections in +# 15 min and port 22 stopped answering while HTTPS to the same estate stayed fine. +# +# WHY IT IS AT THE BOTTOM, and ControlPath is at the top. ssh_config is +# first-value-wins, so position encodes intent: +# * BEFORE the Include = an OVERRIDE. Correct for ControlPath, whose value in +# the user's config points at read-only ~/.ssh and simply cannot work here. +# * AFTER the Include = a DEFAULT. Correct for ControlMaster, because an +# explicit per-host 'ControlMaster no' (or 'auto', or any value) in the +# user's own config must keep winning. We are supplying an opinion only +# where the user expressed none. +# That asymmetry is the whole design: force what is broken, default what is +# merely absent. It also means this needs no audit of anyone's ~/.ssh/config — +# which matters because that file is per-machine, differs across the fleet, and +# future machines' versions do not exist yet to be audited. +# +# Caveat worth knowing (and documented in the pi-devbox-environment skill): a +# stale master socket — file present, daemon gone, e.g. after the host suspends +# or changes network — makes every later ssh to that host hang. Recovery is +# 'ssh -F ~/.ssh-local/config -O exit '. ControlPersist is deliberately +# short (10m idle, and each new session resets the idle timer) so an abandoned +# socket ages out on its own rather than lingering for hours. +MULTIPLEX_DEFAULT_BLOCK=$(cat <<'EOF' + +# Multiplexing DEFAULT — intentionally after the Include above, so any explicit +# per-host ControlMaster in your own ~/.ssh/config still wins (first-value-wins). +# Applies only to targets that never mentioned ControlMaster at all. +# Stale socket after a suspend/network change? ssh -O exit . +Host * + ControlMaster auto + ControlPersist 10m +EOF +) + cat > "$CONFIG" < @@ -216,6 +255,7 @@ ${JUMP_BLOCK} ${LAN_CONF_BLOCK} ${AUTOJUMP_BLOCK} ${INCLUDE_BLOCK} +${MULTIPLEX_DEFAULT_BLOCK} EOF chmod 600 "$CONFIG" 2>/dev/null || true diff --git a/rootfs/usr/local/share/pi-devbox/skills/pi-devbox-environment/SKILL.md b/rootfs/usr/local/share/pi-devbox/skills/pi-devbox-environment/SKILL.md index c7e79ff..d4eb836 100644 --- a/rootfs/usr/local/share/pi-devbox/skills/pi-devbox-environment/SKILL.md +++ b/rootfs/usr/local/share/pi-devbox/skills/pi-devbox-environment/SKILL.md @@ -185,6 +185,16 @@ entrypoint's `setup-lan-access.sh` writes a **writable SSH sidecar** at - A `Host *` block redirecting `ControlPath` into the writable `~/.ssh-local/cm` (because `~/.ssh` is typically bind-mounted **read-only**, so a master socket can't be created under it), plus `Include ~/.ssh/config`. +- A **trailing** `Host *` block supplying `ControlMaster auto` + `ControlPersist + 10m` as a *default*. Position is the design: `ControlPath` sits **before** the + `Include` (an override — the value in your own config points at read-only + `~/.ssh` and cannot work here), while `ControlMaster` sits **after** it (a + default — an explicit per-host `ControlMaster no`/`auto` in your own config + still wins, because ssh_config is first-value-wins). **Force what is broken, + default what is merely absent.** Without this, a target whose entry never + mentioned `ControlMaster` opens a fresh TCP connection per `ssh` call, and an + agent making a dozen calls in a few minutes can trip fail2ban or a CGNAT + flow-table cap on the far end. - Aliases **`host` / `mac`** → `host.docker.internal` (user comes from `HOST_SSH_USER`) — i.e. SSH back into the Docker host. - On VM-backed hosts only: an **SSH-jump-via-host** block so the container can @@ -199,6 +209,25 @@ ssh -F "$HOME/.ssh-local/config" mac 'hostname; whoami' # reach the host ssh -F "$HOME/.ssh-local/config" '…' # reach a LAN peer (if configured) ``` +**Always go through the sidecar, never `-F ~/.ssh/config`.** This is the single +easiest way to break SSH from inside the container, and the failure actively +misleads: the read-only path makes the master socket uncreatable, so +multiplexing appears *impossible* rather than misconfigured. What follows is a +burst of fresh connections and, on a rate-limiting peer, a block that looks like +an outage. The tell that it is rate-limiting and not an outage: HTTPS to the same +estate keeps working while port 22 stops answering. (Recorded 2026-08-25 — an +agent hit exactly this, concluded "ControlMaster is impossible here", disabled +multiplexing, and filed that as a lesson. The sidecar had solved it since v1.4.) + +If every `ssh` to one host suddenly hangs, suspect a **stale master** — socket +file present, daemon gone, typically after the host suspended or changed +network. Check and clear it: + +```sh +ssh -F "$HOME/.ssh-local/config" -O check # "Master running (pid=…)" or no master +ssh -F "$HOME/.ssh-local/config" -O exit # tear down a stale one +``` + Two related mechanisms (don't reinvent them): - **ControlMaster multiplexing** is preconfigured (`/tmp/sshcm/`) to survive