joakimp/opencode-devbox
Public Access
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

v1.14.50c: tag-only retag to recover v1.14.50b's missing variants
Publish Docker Image / base-decide (push) Successful in 1m12s
Details
Publish Docker Image / build-base (push) Has been skipped
Details
Publish Docker Image / smoke-omos (push) Successful in 4m39s
Details
Publish Docker Image / smoke-with-pi (push) Successful in 7m22s
Details
Publish Docker Image / smoke-base (push) Successful in 8m1s
Details
Publish Docker Image / smoke-omos-with-pi (push) Successful in 10m56s
Details
Publish Docker Image / build-variant-omos (push) Failing after 12m49s
Details
Publish Docker Image / build-variant-with-pi (push) Successful in 17m0s
Details
Publish Docker Image / build-variant-base (push) Successful in 17m33s
Details
Publish Docker Image / build-variant-omos-with-pi (push) Successful in 26m46s
Details
Publish Docker Image / promote-base-latest (push) Has been skipped
Details
Publish Docker Image / update-description (push) Has been skipped
Details

Browse Source 
CHANGELOG entry for v1.14.50c with full postmortem on the v1.14.50/50b
runner-fleet incident (AVX shadowing + containerd race + Proxmox CPU
default + base-latest auto-promote gap). No container-side code changes
\u2014 the rebuild on the now-healthy fleet is sufficient.
This commit is contained in:
Joakim Persson
2026-05-14 23:32:46 +02:00
parent dba05da7d1
commit f06a70a3bc
1 changed files with 27 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CHANGELOG.md
+27
View File
@@ -8,6 +8,33 @@ Tags follow `v{opencode_version}[letter]` — bare tag for the first build on a
## Unreleased ## Unreleased
## v1.14.50c — 2026-05-14
Recovery release for v1.14.50b's missing variants. v1.14.50b shipped only the `base` variant; `omos`, `with-pi`, and `omos-with-pi` were lost to a runner-fleet incident (see postmortem below).
No container-side changes. This is a tag-only retag to re-run the build on a now-healthy runner fleet. Same `base-35ee5fe7861a` from v1.14.50b is reused via hash-cache hit; only the four variant deltas are rebuilt and published.
### Postmortem: v1.14.50 / v1.14.50b runner-fleet incident
Two orthogonal runner-host issues compounded across runs 285–291:
1. **AVX-less runner shadowing the new fleet.** A pre-migration `act_runner` container on `nyvaken` (Sandy Bridge E3-12xx, has AVX but no AVX2; 4 weeks old, name `act_runner-runner-1`) collided with the orchestrator's freshly deployed `runner-1` VM (Broadwell-EP host, fully AVX2-capable). Gitea scheduled jobs to both. Jobs landing on the nyvaken container `npm install -g opencode-ai@1.14.50` succeeded, then ran `opencode --version` postinstall → the bundled Bun (v1.3.13 baseline) emitted `CPU lacks AVX support`, panicked, and SIGILLed (exit code 132).
2. **Containerd shared-state race at `capacity: 2`.** The new VM-based runners initially ran `act_runner` with `capacity: 2`, scheduling two concurrent jobs on a single host. Both jobs would invoke `docker/setup-buildx-action@v4`, which pulls `moby/buildkit:buildx-stable-1`. Containerd's content store raced on identical sha256 ingestion, surfacing as `commit failed: rename .../ingest/.../data .../blobs/sha256/...: no such file or directory` or `failed to extract layer: failed to Lchown ...`.
A secondary issue surfaced: **Proxmox VM `cpu:` field defaults mask AVX**. The newly-cloned runner VMs had no explicit `cpu:` line in `qm config` and inherited Proxmox's recent default `x86-64-v2-AES`, which excludes AVX even though the Broadwell-EP host silicon has full `avx2`. Fix: `qm set <vmid> --cpu x86-64-v3` (or `host` for full passthrough), then `qm shutdown` + `qm start` (live reboot is not enough). Verified inside guest with `grep -m1 -oE 'avx[2]?' /proc/cpuinfo`.
Additionally, when `promote-base-latest`'s `needs:` graph requires *all four* `build-variant-*` jobs to succeed, partial publishes leave the `base-latest` Hub alias never advancing. Workaround used during recovery: manually re-tag the new base hash via Docker Hub registry manifest API (`PUT /v2/<repo>/manifests/base-latest` with the body of `GET /v2/<repo>/manifests/base-<sha>`) using a granular Hub PAT. No blob copy needed since blobs are content-addressed.
### Recovery actions taken (orchestrator + this repo)
- Orchestrator (cloud-init + ansible repos): set explicit `cpu_type: x86-64-v3` in all runner host yaml files; provision.sh now applies `qm set --cpu` after clone; added runner-3 on proxmox003 for anti-affinity (one runner per Proxmox node); dropped `capacity: 2 → 1` on all runners; bumped `act_runner` 0.3.1 → 0.6.1 across the fleet; documented the CPU-type gotcha as gotcha #9 in cloud-init AGENTS.md and a section in proxmox-guide.md.
- User: retired the legacy `act_runner-runner-1` container on nyvaken; cleaned up stale runner registrations in Gitea Site Admin → Actions → Runners.
- This repo: no changes needed in Dockerfile.base / Dockerfile.variant; v1.14.50c is a tag-only retag.
### Fleet state at v1.14.50c
3 runners (runner-1@proxmox001, runner-2@proxmox002, runner-3@proxmox003), all `act_runner` v0.6.1, all `capacity: 1`, all expose AVX + AVX2 to the guest. No name collisions. Estimated wall clock for v1.14.50c (cache-hit base, 4 variant deltas across 3 runners with capacity:1): ~40–50 min.
## v1.14.50b — 2026-05-14 ## v1.14.50b — 2026-05-14
Rebuild of v1.14.50 with two fixes — the v1.14.50 release was incomplete (smokes failed under containerd contention; build-variant jobs skipped; base-latest never promoted to Docker Hub). Rebuild of v1.14.50 with two fixes — the v1.14.50 release was incomplete (smokes failed under containerd contention; build-variant jobs skipped; base-latest never promoted to Docker Hub).