feat(pi-task): optional model fallback when the primary fails auth before acting
A tier's model can be unusable for reasons that have nothing to do with the
task, typically an expired AWS SSO session behind an amazon-bedrock profile.
pi reports that as a normal run: rc 0, agent_end, willRetry=false. The failure
shows only as stopReason="error" + errorMessage on the assistant message
(measured 2026-10-12). pi-task turned it into "no fenced json envelope",
which points at the wrong layer.
- Always surface it, first in the problem list:
"child model error (<provider>/<id>): ...".
- New, optional: pi-task.fallbackProfiles.<effort> = {provider, id[, thinking]}
in ~/.pi/agent/settings.json. It is tried ONCE, and only when the primary
ended in an auth-class error, made zero tool calls, left the boundary
diff empty, and at least 30 s of wall_s remain. A write task that failed
after acting is never re-run on a half-edited tree.
- A fallback without `thinking` inherits the primary's rather than "off".
A fallback identical to the primary is ignored, with a warning.
- The key lives in pi-task's own namespace because the fork tool does not
fall back. pi-fork's parser silently drops unknown profile keys.
- Reporting: "[FALLBACK]" on the verdict line plus a warning line.
result.json gains fallback{} and attempts[]; metrics.model/provider name
the model that produced the verdict. Audit keeps raw.primary.ndjson,
argv.fallback.json and boundary-after.primary.json. The second attempt
gets only the remaining wall budget, so the task tool's (wall_s+90) s kill
still holds.
- Unchanged: exit 0 = PASS / 1 = FAIL, and the closing "audit:" line.
selftest: 14 -> 29 checks, two-sided for every gate (fixtures mirror the
measured event shape; the SSO string is verbatim from a real expired
session). Each gate was mutation-checked: removing it turns its line BAD.
End-to-end, emb-7kj4vr4g, with fake AWS keys in the environment:
- the old pi-task gives FAIL "no fenced json envelope";
- the new one, per tier, gives PASS [FALLBACK] with the correct answer and
real tool calls;
- with real credentials: PASS on the primary, fallback configured but unused;
- with no fallback configured: FAIL, model error listed first.
This commit is contained in:
@@ -82,7 +82,7 @@ wait out.
|
||||
| machine-checkable result | the child must emit a fenced `json` envelope (`status`/`deliverable`/`evidence`/`unsure`/`did_not_do`). **If it does not parse, the task FAILED**, however fluent the prose |
|
||||
| claims carry pointers | every `evidence[]` entry needs a `pointer`; the parent is told to spot-check them |
|
||||
| post-hoc boundary diff | git `HEAD` + `status --porcelain --ignored` (or a sha256 manifest) of every `roots[]` entry, before and after. `roots` is the WATCHED set; `write_allowed` is the CHANGEABLE subset. Any delta outside `write_allowed` FAILS the task. `--ignored` is load-bearing — see limit 1 |
|
||||
| audit trail | `~/.pi/agent/pi-task/<stamp>-<id>/` keeps `spec.json`, `prompt.txt`, `argv.json`, `raw.ndjson`, `result.json`, both boundary snapshots, and the child's session |
|
||||
| audit trail | `~/.pi/agent/pi-task/<stamp>-<id>/` keeps `spec.json`, `prompt.txt`, `argv.json`, `raw.ndjson`, `result.json`, both boundary snapshots, and the child's session; after a model fallback also `argv.fallback.json`, `raw.primary.ndjson`, `boundary-after.primary.json` and `sess-fallback/` |
|
||||
| budgets | `budget.wall_s` (hard kill) and `budget.usd` (post-hoc, summed from `agent_end.messages[].usage.cost.total`) |
|
||||
|
||||
```bash
|
||||
@@ -110,7 +110,55 @@ then `$PATH`.
|
||||
`selftest` is not decoration: it feeds the validator one known-good and six
|
||||
known-bad envelopes plus a two-sided boundary check, and **aborts** if any pair
|
||||
fails to discriminate. A validator that has only ever returned PASS has not been
|
||||
shown to validate anything.
|
||||
shown to validate anything. It also covers both sides of every gate of the model
|
||||
fallback below, and each gate was mutation-checked: deleting it turns a selftest
|
||||
line `BAD`.
|
||||
|
||||
### Model fallback (optional)
|
||||
|
||||
A tier's model can be unusable for reasons unrelated to the task. The typical
|
||||
case is an expired AWS SSO session behind an `amazon-bedrock` profile. pi reports
|
||||
this as a normal run: rc `0`, an `agent_end`, `willRetry: false`. The failure
|
||||
shows ONLY as `stopReason: "error"` plus `errorMessage` on the assistant message
|
||||
(measured 2026-10-12). pi-task now always surfaces that as
|
||||
`child model error (<provider>/<id>): …`. Before, it showed up as the misleading
|
||||
`no fenced json envelope`.
|
||||
|
||||
To retry on another model, add a per-effort fallback in pi-task's own namespace
|
||||
of `~/.pi/agent/settings.json`:
|
||||
|
||||
```json
|
||||
"pi-task": {
|
||||
"fallbackProfiles": {
|
||||
"fast": { "provider": "<provider>", "id": "<model>", "thinking": "low" },
|
||||
"balanced": { "provider": "<provider>", "id": "<model>", "thinking": "medium" },
|
||||
"deep": { "provider": "<provider>", "id": "<model>", "thinking": "high" }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
The fallback runs **once**, and only if ALL of these hold for the primary attempt:
|
||||
- it ended in an **auth-class** error (expired/invalid token, `aws sso login`, 401, invalid API key, …);
|
||||
- it made **zero** tool calls;
|
||||
- the boundary diff is **empty**;
|
||||
- at least 30 s of `wall_s` remain.
|
||||
|
||||
If any of them fails, the primary's failure stands. This is deliberate: a write
|
||||
task that failed after acting must not be re-run on a half-edited tree.
|
||||
|
||||
When the fallback is used, the report line carries `[FALLBACK]` and a
|
||||
`⚠ fallback: <from> -> <to> — <reason>` line. `result.json` records
|
||||
`fallback.{configured,used,reason,from,to}` and an `attempts[]` entry per model.
|
||||
`metrics.model` / `metrics.provider` name the model that produced the verdict.
|
||||
The second attempt gets only the *remaining* wall budget, so the task tool's
|
||||
`(wall_s + 90) s` kill still holds.
|
||||
|
||||
Notes:
|
||||
- A fallback that omits `thinking` inherits the primary's rather than `off`.
|
||||
- A fallback identical to the primary is ignored, with a warning.
|
||||
- The key sits under `pi-task`, not inside `pi-fork.effortProfiles`, because the
|
||||
`fork` tool does **not** fall back. pi-fork's profile parser drops unknown keys
|
||||
without a word, so a fallback placed there would look configured and do nothing.
|
||||
|
||||
### Honest limits — read before trusting it
|
||||
|
||||
|
||||
Reference in New Issue
Block a user