feat(pi-task): optional model fallback when the primary fails auth before acting

A tier's model can be unusable for reasons that have nothing to do with the
task, typically an expired AWS SSO session behind an amazon-bedrock profile.
pi reports that as a normal run: rc 0, agent_end, willRetry=false. The failure
shows only as stopReason="error" + errorMessage on the assistant message
(measured 2026-10-12). pi-task turned it into "no fenced json envelope",
which points at the wrong layer.

- Always surface it, first in the problem list:
  "child model error (<provider>/<id>): ...".
- New, optional: pi-task.fallbackProfiles.<effort> = {provider, id[, thinking]}
  in ~/.pi/agent/settings.json. It is tried ONCE, and only when the primary
  ended in an auth-class error, made zero tool calls, left the boundary
  diff empty, and at least 30 s of wall_s remain. A write task that failed
  after acting is never re-run on a half-edited tree.
- A fallback without `thinking` inherits the primary's rather than "off".
  A fallback identical to the primary is ignored, with a warning.
- The key lives in pi-task's own namespace because the fork tool does not
  fall back. pi-fork's parser silently drops unknown profile keys.
- Reporting: "[FALLBACK]" on the verdict line plus a warning line.
  result.json gains fallback{} and attempts[]; metrics.model/provider name
  the model that produced the verdict. Audit keeps raw.primary.ndjson,
  argv.fallback.json and boundary-after.primary.json. The second attempt
  gets only the remaining wall budget, so the task tool's (wall_s+90) s kill
  still holds.
- Unchanged: exit 0 = PASS / 1 = FAIL, and the closing "audit:" line.

selftest: 14 -> 29 checks, two-sided for every gate (fixtures mirror the
measured event shape; the SSO string is verbatim from a real expired
session). Each gate was mutation-checked: removing it turns its line BAD.

End-to-end, emb-7kj4vr4g, with fake AWS keys in the environment:
- the old pi-task gives FAIL "no fenced json envelope";
- the new one, per tier, gives PASS [FALLBACK] with the correct answer and
  real tool calls;
- with real credentials: PASS on the primary, fallback configured but unused;
- with no fallback configured: FAIL, model error listed first.
This commit is contained in:
Joakim Persson
2026-10-12 11:07:32 +02:00
parent 9c87ee843f
commit 3760d90998
3 changed files with 313 additions and 44 deletions
+50 -2
View File
@@ -82,7 +82,7 @@ wait out.
| machine-checkable result | the child must emit a fenced `json` envelope (`status`/`deliverable`/`evidence`/`unsure`/`did_not_do`). **If it does not parse, the task FAILED**, however fluent the prose |
| claims carry pointers | every `evidence[]` entry needs a `pointer`; the parent is told to spot-check them |
| post-hoc boundary diff | git `HEAD` + `status --porcelain --ignored` (or a sha256 manifest) of every `roots[]` entry, before and after. `roots` is the WATCHED set; `write_allowed` is the CHANGEABLE subset. Any delta outside `write_allowed` FAILS the task. `--ignored` is load-bearing — see limit 1 |
| audit trail | `~/.pi/agent/pi-task/<stamp>-<id>/` keeps `spec.json`, `prompt.txt`, `argv.json`, `raw.ndjson`, `result.json`, both boundary snapshots, and the child's session |
| audit trail | `~/.pi/agent/pi-task/<stamp>-<id>/` keeps `spec.json`, `prompt.txt`, `argv.json`, `raw.ndjson`, `result.json`, both boundary snapshots, and the child's session; after a model fallback also `argv.fallback.json`, `raw.primary.ndjson`, `boundary-after.primary.json` and `sess-fallback/` |
| budgets | `budget.wall_s` (hard kill) and `budget.usd` (post-hoc, summed from `agent_end.messages[].usage.cost.total`) |
```bash
@@ -110,7 +110,55 @@ then `$PATH`.
`selftest` is not decoration: it feeds the validator one known-good and six
known-bad envelopes plus a two-sided boundary check, and **aborts** if any pair
fails to discriminate. A validator that has only ever returned PASS has not been
shown to validate anything.
shown to validate anything. It also covers both sides of every gate of the model
fallback below, and each gate was mutation-checked: deleting it turns a selftest
line `BAD`.
### Model fallback (optional)
A tier's model can be unusable for reasons unrelated to the task. The typical
case is an expired AWS SSO session behind an `amazon-bedrock` profile. pi reports
this as a normal run: rc `0`, an `agent_end`, `willRetry: false`. The failure
shows ONLY as `stopReason: "error"` plus `errorMessage` on the assistant message
(measured 2026-10-12). pi-task now always surfaces that as
`child model error (<provider>/<id>): …`. Before, it showed up as the misleading
`no fenced json envelope`.
To retry on another model, add a per-effort fallback in pi-task's own namespace
of `~/.pi/agent/settings.json`:
```json
"pi-task": {
"fallbackProfiles": {
"fast": { "provider": "<provider>", "id": "<model>", "thinking": "low" },
"balanced": { "provider": "<provider>", "id": "<model>", "thinking": "medium" },
"deep": { "provider": "<provider>", "id": "<model>", "thinking": "high" }
}
}
```
The fallback runs **once**, and only if ALL of these hold for the primary attempt:
- it ended in an **auth-class** error (expired/invalid token, `aws sso login`, 401, invalid API key, …);
- it made **zero** tool calls;
- the boundary diff is **empty**;
- at least 30 s of `wall_s` remain.
If any of them fails, the primary's failure stands. This is deliberate: a write
task that failed after acting must not be re-run on a half-edited tree.
When the fallback is used, the report line carries `[FALLBACK]` and a
`⚠ fallback: <from> -> <to> — <reason>` line. `result.json` records
`fallback.{configured,used,reason,from,to}` and an `attempts[]` entry per model.
`metrics.model` / `metrics.provider` name the model that produced the verdict.
The second attempt gets only the *remaining* wall budget, so the task tool's
`(wall_s + 90) s` kill still holds.
Notes:
- A fallback that omits `thinking` inherits the primary's rather than `off`.
- A fallback identical to the primary is ignored, with a warning.
- The key sits under `pi-task`, not inside `pi-fork.effortProfiles`, because the
`fork` tool does **not** fall back. pi-fork's profile parser drops unknown keys
without a word, so a fallback placed there would look configured and do nothing.
### Honest limits — read before trusting it