joakimp
25c1265681
The rule was correct and written down twice (global AGENTS.md, this skill)
and was still violated by agents that had just read it: on 2026-09-17 all
five fork briefs in one session carried "do not", one returned confident
verbatim quotes that did not exist, and four had disjoint write boundaries
fork cannot enforce (re-run as pi-task, they passed). Three mechanisms,
none of them wording:
1. fork is a TOOL — its self-recommending description ("implementation,
testing, review…") is in the model's face every turn; pi-task was a CLI
to be remembered and reached through bash with a hand-written JSON.
2. The skill is gone after the first compaction; the tool list never is.
The asymmetry widens in exactly the long sessions where fork is worst.
3. Friction: one string vs a spec file + bash + reading result.json.
extensions/task.ts registers pi-task as the `task` tool. Flat parameters
build the spec; the decision rule sits in the description and in
promptGuidelines (appended to the system prompt, so compaction cannot remove
it). Before spending a model run it rejects the two spec errors that make a
boundary violation certain — write_allowed not an exact subset of roots, and
a writable root nested in a watched-only root (the parent's porcelain would
change every time; pi-task keys deltas by root string) — and it serialises
sibling tasks whose roots overlap (parallel siblings saw each other's writes
as violations, 2026-09-17). Returns the CLI's own parent-facing report; a
FAIL verdict is a result, only a CLI refusal is an error.
extensions/fork-gate.ts is a tool_call hook that BLOCKS a fork whose brief
contains a prohibition, a write boundary, or a clause-initial file-changing
imperative, and returns as the reason the exact task(...) to make instead.
Wording, not intent — the message says so and how to rephrase a genuinely
read-only brief. PI_FORK_GATE=off logs instead; /ext disables.
Evidence: test/fork-gate.test.mjs is two-sided (15 must-block incl. the real
shapes, 10 must-pass incl. "Write a summary…", "Report which files were
modified…", "Give me an update…"); the classifier redirects 5/5 of the real
briefs from the motivating session. test/task.test.mjs pins root overlap and
the pre-launch validation. Live in `pi -p`: the fork was intercepted before
any child spawned (no /tmp/pi-fork-* dir) and the model received the
redirect; task returned PASS with an evidence pointer and audit dir, a
budget-overrun returned a FAIL result (isError=false), and a nested-root spec
was rejected with no audit dir created.
skill/SKILL.md: Part 1 now opens with "decide the rung before the brief"
(the table, the three-question pre-flight, the roots contract, overlap,
what isolation does not fix); the ladder section and quick reference no
longer say pi-task "will never appear in your tool list". package.json gains
"type": "module" and a test script.
77 lines
4.0 KiB
JavaScript
77 lines
4.0 KiB
JavaScript
// Two-sided test of the fork-gate classifier: briefs that MUST be redirected
|
|
// to pi-task and briefs that MUST stay forkable. A gate that only ever blocks
|
|
// has not been shown to discriminate. Runs the shipped .ts directly (node >= 23
|
|
// strips types natively; the only import is type-only).
|
|
import { test } from "node:test";
|
|
import assert from "node:assert/strict";
|
|
import { classifyForkBrief, blockReason, GATE_RULES } from "../extensions/fork-gate.ts";
|
|
|
|
// Real shapes from the sessions that motivated the gate (2026-09-17, tor-ms22).
|
|
const MUST_BLOCK = [
|
|
["prohibition", "Migrate docs/level-01-basics. Do NOT modify files outside docs/level-01-basics/ or tutorials/01-basics/."],
|
|
["prohibition", "Read the repo and report. Don't change anything."],
|
|
["prohibition", "You must not commit."],
|
|
["prohibition", "Never touch the inventory."],
|
|
["prohibition", "Please refrain from editing mkdocs.yml"],
|
|
["boundary", "Only touch the three files listed above."],
|
|
["boundary", "Edit only the README; nothing else."],
|
|
["boundary", "This is a read-only review of the role."],
|
|
["boundary", "Stay within roles/static_site/ for all changes."],
|
|
["boundary", "Changes are limited to the docs directory"],
|
|
["mutation", "Implement the FQCN sweep across docs/ and run mkdocs --strict."],
|
|
["mutation", "Fix the eleven failing playbooks under solutions/."],
|
|
["mutation", "Review the playbooks, then commit the result on develop."],
|
|
["mutation", "Rewrite Exercise 5 to use the inventory plugin."],
|
|
["mutation", "Please update docs/getting-started/structure.md from the real tree."],
|
|
];
|
|
|
|
// Fork's legitimate uses: read-only exploration against this conversation, and
|
|
// opinions. Includes phrasings that a naive verb list would misfire on.
|
|
const MUST_PASS = [
|
|
"Find where the runner config is written to disk and report file:line.",
|
|
"Which function creates the temp files for the session? Report the call chain.",
|
|
"Compare the three staging options we discussed and give an independent opinion.",
|
|
"Summarize the architecture of the static_site role in 200 words.",
|
|
"Write a summary of what the last five commits changed.",
|
|
"Give me an update on how the mailbox derivation handles ready-status events.",
|
|
"Report which files were modified by CI run 23 according to the logs.",
|
|
"Explain the difference between compose and keyed_groups in the openstack inventory plugin.",
|
|
"List every place the 60000 ms default is documented; return paths and line numbers.",
|
|
"Is the fixed deadline applied in the feed path? Quote the code with a pointer.",
|
|
];
|
|
|
|
test("every hazardous brief is blocked with the expected class", () => {
|
|
for (const [kind, brief] of MUST_BLOCK) {
|
|
const v = classifyForkBrief(brief);
|
|
assert.equal(v.block, true, `should block: ${brief}`);
|
|
assert.equal(v.kind, kind, `wrong class for: ${brief} (matched "${v.matched}")`);
|
|
assert.ok(v.matched && v.matched.length > 0, "reports what matched");
|
|
}
|
|
});
|
|
|
|
test("every legitimate exploration brief passes", () => {
|
|
for (const brief of MUST_PASS) {
|
|
const v = classifyForkBrief(brief);
|
|
assert.equal(v.block, false, `false positive on: ${brief} (${v.kind}: "${v.matched}")`);
|
|
}
|
|
});
|
|
|
|
test("non-string / empty briefs never block (gate must not break fork on odd input)", () => {
|
|
for (const x of [undefined, null, 42, "", {}, []]) {
|
|
assert.equal(classifyForkBrief(x).block, false);
|
|
}
|
|
});
|
|
|
|
test("block reason carries the redirect: task(...) call, CLI fallback, sequential-roots rule", () => {
|
|
const r = blockReason(classifyForkBrief(MUST_BLOCK[0][1]));
|
|
for (const needle of ["task(id=", "write_allowed=", "/opt/pi-toolkit/bin/pi-task run", "overlap", "matches wording, not intent"]) {
|
|
assert.ok(r.includes(needle), `reason lacks "${needle}"`);
|
|
}
|
|
assert.ok(r.includes('"Do NOT"'), "reason names the words that fired");
|
|
});
|
|
|
|
test("rules are ordered prohibition → boundary → mutation and each has a why", () => {
|
|
assert.deepEqual(GATE_RULES.map((r) => r.kind), ["prohibition", "boundary", "mutation"]);
|
|
for (const r of GATE_RULES) assert.ok(r.why.length > 20);
|
|
});
|