v1.14.50c: tag-only retag to recover v1.14.50b's missing variants

CHANGELOG entry for v1.14.50c with full postmortem on the v1.14.50/50b
runner-fleet incident (AVX shadowing + containerd race + Proxmox CPU
default + base-latest auto-promote gap). No container-side code changes
\u2014 the rebuild on the now-healthy fleet is sufficient.

.gitea/workflows/validate.yml

@@ -4,6 +4,22 @@ name: Validate
 # runs the smoke test, and checks image size — without pushing anything
 # to Docker Hub. Tag pushes are handled by docker-publish-split.yml which
 # does the full multi-arch split-base build-and-push.
+#
+# Trade-off: variant builds here use the published `base-latest` image
+# from Docker Hub as their parent, NOT a locally-built base. This is
+# because `docker/build-push-action@v7` runs each invocation in its own
+# buildx container context, so an image loaded into the host docker
+# daemon by step N is not visible to step N+1's buildx invocation.
+# Building base + variant in the same job would require either pushing
+# the base to a registry or sharing a buildx instance across steps — both
+# significantly more complex than just using the published base.
+#
+# Consequence: PRs/pushes that change Dockerfile.base, rootfs/, or
+# entrypoint*.sh are NOT exercised by this workflow. The release path
+# (docker-publish-split.yml on tag push) does build the new base, so
+# release tags are the gate that fully validates base-image changes.
+# The base-change-warning job below surfaces a runtime warning when this
+# blind-spot applies.
 
 on:
   push:
@@ -34,6 +50,33 @@ jobs:
         run: |
           python3 scripts/generate-dockerhub-md.py --check
 
+  base-change-warning:
+    # Surfaces a warning when this commit changes base-image inputs
+    # (Dockerfile.base, rootfs/, entrypoint*.sh). validate.yml uses
+    # Hub's base-latest as the parent for variant builds, so changes to
+    # those files are NOT exercised here — only release tags rebuild the
+    # base via docker-publish-split.yml.
+    runs-on: ubuntu-latest
+    container:
+      image: catthehacker/ubuntu:act-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+        with:
+          fetch-depth: 2
+
+      - name: Detect base-input changes
+        run: |
+          set -e
+          if ! git diff --name-only HEAD~1 HEAD 2>/dev/null \
+              | grep -qE '^(Dockerfile\.base|rootfs/|entrypoint.*\.sh)$'; then
+            echo "No base-image inputs changed in this commit — validate.yml fully exercises the published base-latest."
+            exit 0
+          fi
+          echo "::warning::This commit changes base-image inputs (Dockerfile.base, rootfs/, or entrypoint*.sh). validate.yml uses Hub's base-latest as the parent for variant builds, so the new base is NOT exercised by this workflow. Cut a release tag, or run a workflow_dispatch of docker-publish-split.yml against a test tag (e.g. v0.0.0-base-test, promote_latest=false) for end-to-end validation of the new base."
+          echo "Changed base-input files:"
+          git diff --name-only HEAD~1 HEAD | grep -E '^(Dockerfile\.base|rootfs/|entrypoint.*\.sh)$'
+
   validate-base:
     runs-on: ubuntu-latest
     container:
@@ -83,9 +126,12 @@ jobs:
         uses: docker/build-push-action@v7
         with:
           context: .
+          file: Dockerfile.variant
           platforms: linux/amd64
           push: false
           load: true
+          build-args: |
+            BASE_IMAGE=joakimp/opencode-devbox:base-latest
           tags: opencode-devbox:ci-base
 
       - name: Smoke test
@@ -137,10 +183,12 @@ jobs:
         uses: docker/build-push-action@v7
         with:
           context: .
+          file: Dockerfile.variant
           platforms: linux/amd64
           push: false
           load: true
           build-args: |
+            BASE_IMAGE=joakimp/opencode-devbox:base-latest
             INSTALL_OMOS=true
           tags: opencode-devbox:ci-omos
 
@@ -193,10 +241,12 @@ jobs:
         uses: docker/build-push-action@v7
         with:
           context: .
+          file: Dockerfile.variant
           platforms: linux/amd64
           push: false
           load: true
           build-args: |
+            BASE_IMAGE=joakimp/opencode-devbox:base-latest
             INSTALL_PI=true
           tags: opencode-devbox:ci-with-pi
 
@@ -249,10 +299,12 @@ jobs:
         uses: docker/build-push-action@v7
         with:
           context: .
+          file: Dockerfile.variant
           platforms: linux/amd64
           push: false
           load: true
           build-args: |
+            BASE_IMAGE=joakimp/opencode-devbox:base-latest
             INSTALL_OMOS=true
             INSTALL_PI=true
           tags: opencode-devbox:ci-omos-with-pi

CHANGELOG.md

@@ -8,6 +8,40 @@ Tags follow `v{opencode_version}[letter]` — bare tag for the first build on a
 
 ## Unreleased
 
+## v1.14.50c — 2026-05-14
+
+Recovery release for v1.14.50b's missing variants. v1.14.50b shipped only the `base` variant; `omos`, `with-pi`, and `omos-with-pi` were lost to a runner-fleet incident (see postmortem below).
+
+No container-side changes. This is a tag-only retag to re-run the build on a now-healthy runner fleet. Same `base-35ee5fe7861a` from v1.14.50b is reused via hash-cache hit; only the four variant deltas are rebuilt and published.
+
+### Postmortem: v1.14.50 / v1.14.50b runner-fleet incident
+
+Two orthogonal runner-host issues compounded across runs 285–291:
+
+1. **AVX-less runner shadowing the new fleet.** A pre-migration `act_runner` container on `nyvaken` (Sandy Bridge E3-12xx, has AVX but no AVX2; 4 weeks old, name `act_runner-runner-1`) collided with the orchestrator's freshly deployed `runner-1` VM (Broadwell-EP host, fully AVX2-capable). Gitea scheduled jobs to both. Jobs landing on the nyvaken container `npm install -g opencode-ai@1.14.50` succeeded, then ran `opencode --version` postinstall → the bundled Bun (v1.3.13 baseline) emitted `CPU lacks AVX support`, panicked, and SIGILLed (exit code 132).
+2. **Containerd shared-state race at `capacity: 2`.** The new VM-based runners initially ran `act_runner` with `capacity: 2`, scheduling two concurrent jobs on a single host. Both jobs would invoke `docker/setup-buildx-action@v4`, which pulls `moby/buildkit:buildx-stable-1`. Containerd's content store raced on identical sha256 ingestion, surfacing as `commit failed: rename .../ingest/.../data .../blobs/sha256/...: no such file or directory` or `failed to extract layer: failed to Lchown ...`.
+
+A secondary issue surfaced: **Proxmox VM `cpu:` field defaults mask AVX**. The newly-cloned runner VMs had no explicit `cpu:` line in `qm config` and inherited Proxmox's recent default `x86-64-v2-AES`, which excludes AVX even though the Broadwell-EP host silicon has full `avx2`. Fix: `qm set <vmid> --cpu x86-64-v3` (or `host` for full passthrough), then `qm shutdown` + `qm start` (live reboot is not enough). Verified inside guest with `grep -m1 -oE 'avx[2]?' /proc/cpuinfo`.
+
+Additionally, when `promote-base-latest`'s `needs:` graph requires *all four* `build-variant-*` jobs to succeed, partial publishes leave the `base-latest` Hub alias never advancing. Workaround used during recovery: manually re-tag the new base hash via Docker Hub registry manifest API (`PUT /v2/<repo>/manifests/base-latest` with the body of `GET /v2/<repo>/manifests/base-<sha>`) using a granular Hub PAT. No blob copy needed since blobs are content-addressed.
+
+### Recovery actions taken (orchestrator + this repo)
+
+- Orchestrator (cloud-init + ansible repos): set explicit `cpu_type: x86-64-v3` in all runner host yaml files; provision.sh now applies `qm set --cpu` after clone; added runner-3 on proxmox003 for anti-affinity (one runner per Proxmox node); dropped `capacity: 2 → 1` on all runners; bumped `act_runner` 0.3.1 → 0.6.1 across the fleet; documented the CPU-type gotcha as gotcha #9 in cloud-init AGENTS.md and a section in proxmox-guide.md.
+- User: retired the legacy `act_runner-runner-1` container on nyvaken; cleaned up stale runner registrations in Gitea Site Admin → Actions → Runners.
+- This repo: no changes needed in Dockerfile.base / Dockerfile.variant; v1.14.50c is a tag-only retag.
+
+### Fleet state at v1.14.50c
+
+3 runners (runner-1@proxmox001, runner-2@proxmox002, runner-3@proxmox003), all `act_runner` v0.6.1, all `capacity: 1`, all expose AVX + AVX2 to the guest. No name collisions. Estimated wall clock for v1.14.50c (cache-hit base, 4 variant deltas across 3 runners with capacity:1): ~40–50 min.
+
+## v1.14.50b — 2026-05-14
+
+Rebuild of v1.14.50 with two fixes — the v1.14.50 release was incomplete (smokes failed under containerd contention; build-variant jobs skipped; base-latest never promoted to Docker Hub).
+
+- **Force fresh base rebuild.** Added a `BASE_REBUILD_DATE` comment header to `Dockerfile.base` to invalidate the content hash and trigger a full base rebuild. Picks up ~5 days of Debian trixie security updates and other apt-tracked packages. The comment also documents the pattern for future intentional base-rebuilds without other code changes (recommended cadence: once per release).
+- **First publish of `base-latest` alias.** `promote-base-latest` runs unconditionally on tag push (`PROMOTE_LATEST=true`), so this release is the first to put `joakimp/opencode-devbox:base-latest` on Docker Hub. Required before pi-devbox (and any other downstream image FROMing the base) can build.
+
 ## v1.14.50 — 2026-05-14
 
 opencode 1.14.44 → 1.14.50 bump. First release on the split-base build pipeline.

Dockerfile.base

@@ -11,6 +11,13 @@
 # changes (rootfs/, entrypoint*.sh). Version bumps to OPENCODE_VERSION,
 # OMOS_VERSION, PI_VERSION, etc. do NOT trigger a base rebuild.
 #
+# To force a base rebuild for fresh apt packages without other code
+# changes, bump the BASE_REBUILD_DATE comment below. The hash is
+# content-addressed over this file, so any byte change invalidates the
+# cache. Recommended cadence: once per release for security updates.
+#
+# BASE_REBUILD_DATE: 2026-05-14 (v1.14.50b — fresh apt + first promote-base-latest)
+#
 # See the project README's "Build pipeline" section for the rationale.
 
 ARG DEBIAN_VERSION=trixie-slim