v1.14.50c: tag-only retag to recover v1.14.50b's missing variants

CHANGELOG entry for v1.14.50c with full postmortem on the v1.14.50/50b
runner-fleet incident (AVX shadowing + containerd race + Proxmox CPU
default + base-latest auto-promote gap). No container-side code changes
\u2014 the rebuild on the now-healthy fleet is sufficient.

CHANGELOG.md

@@ -8,6 +8,33 @@ Tags follow `v{opencode_version}[letter]` — bare tag for the first build on a
 
 ## Unreleased
 
+## v1.14.50c — 2026-05-14
+
+Recovery release for v1.14.50b's missing variants. v1.14.50b shipped only the `base` variant; `omos`, `with-pi`, and `omos-with-pi` were lost to a runner-fleet incident (see postmortem below).
+
+No container-side changes. This is a tag-only retag to re-run the build on a now-healthy runner fleet. Same `base-35ee5fe7861a` from v1.14.50b is reused via hash-cache hit; only the four variant deltas are rebuilt and published.
+
+### Postmortem: v1.14.50 / v1.14.50b runner-fleet incident
+
+Two orthogonal runner-host issues compounded across runs 285–291:
+
+1. **AVX-less runner shadowing the new fleet.** A pre-migration `act_runner` container on `nyvaken` (Sandy Bridge E3-12xx, has AVX but no AVX2; 4 weeks old, name `act_runner-runner-1`) collided with the orchestrator's freshly deployed `runner-1` VM (Broadwell-EP host, fully AVX2-capable). Gitea scheduled jobs to both. Jobs landing on the nyvaken container `npm install -g opencode-ai@1.14.50` succeeded, then ran `opencode --version` postinstall → the bundled Bun (v1.3.13 baseline) emitted `CPU lacks AVX support`, panicked, and SIGILLed (exit code 132).
+2. **Containerd shared-state race at `capacity: 2`.** The new VM-based runners initially ran `act_runner` with `capacity: 2`, scheduling two concurrent jobs on a single host. Both jobs would invoke `docker/setup-buildx-action@v4`, which pulls `moby/buildkit:buildx-stable-1`. Containerd's content store raced on identical sha256 ingestion, surfacing as `commit failed: rename .../ingest/.../data .../blobs/sha256/...: no such file or directory` or `failed to extract layer: failed to Lchown ...`.
+
+A secondary issue surfaced: **Proxmox VM `cpu:` field defaults mask AVX**. The newly-cloned runner VMs had no explicit `cpu:` line in `qm config` and inherited Proxmox's recent default `x86-64-v2-AES`, which excludes AVX even though the Broadwell-EP host silicon has full `avx2`. Fix: `qm set <vmid> --cpu x86-64-v3` (or `host` for full passthrough), then `qm shutdown` + `qm start` (live reboot is not enough). Verified inside guest with `grep -m1 -oE 'avx[2]?' /proc/cpuinfo`.
+
+Additionally, when `promote-base-latest`'s `needs:` graph requires *all four* `build-variant-*` jobs to succeed, partial publishes leave the `base-latest` Hub alias never advancing. Workaround used during recovery: manually re-tag the new base hash via Docker Hub registry manifest API (`PUT /v2/<repo>/manifests/base-latest` with the body of `GET /v2/<repo>/manifests/base-<sha>`) using a granular Hub PAT. No blob copy needed since blobs are content-addressed.
+
+### Recovery actions taken (orchestrator + this repo)
+
+- Orchestrator (cloud-init + ansible repos): set explicit `cpu_type: x86-64-v3` in all runner host yaml files; provision.sh now applies `qm set --cpu` after clone; added runner-3 on proxmox003 for anti-affinity (one runner per Proxmox node); dropped `capacity: 2 → 1` on all runners; bumped `act_runner` 0.3.1 → 0.6.1 across the fleet; documented the CPU-type gotcha as gotcha #9 in cloud-init AGENTS.md and a section in proxmox-guide.md.
+- User: retired the legacy `act_runner-runner-1` container on nyvaken; cleaned up stale runner registrations in Gitea Site Admin → Actions → Runners.
+- This repo: no changes needed in Dockerfile.base / Dockerfile.variant; v1.14.50c is a tag-only retag.
+
+### Fleet state at v1.14.50c
+
+3 runners (runner-1@proxmox001, runner-2@proxmox002, runner-3@proxmox003), all `act_runner` v0.6.1, all `capacity: 1`, all expose AVX + AVX2 to the guest. No name collisions. Estimated wall clock for v1.14.50c (cache-hit base, 4 variant deltas across 3 runners with capacity:1): ~40–50 min.
+
 ## v1.14.50b — 2026-05-14
 
 Rebuild of v1.14.50 with two fixes — the v1.14.50 release was incomplete (smokes failed under containerd contention; build-variant jobs skipped; base-latest never promoted to Docker Hub).