ci: gate baked skill frontmatter against the loader's own limits
Found while adding pi-devbox-firstboot, whose description was drafted at 1056 characters against pi's MAX_DESCRIPTION_LENGTH of 1024. pi does not reject it: dist/core/skills.js pushes a diagnostic and then loads the skill anyway -- "Still load the skill even with warnings, unless description is missing or empty". So the skill works, the violation is real, and the only thing that would ever report it is a diagnostic inside a live pi session. Nothing in CI starts one, so the warning is invisible here by construction. A spec violation that still works is the most durable kind of rot: nothing forces the fix. Unlike check-skill-floor.sh, this was NOT free to add. Measured headroom on the five baked skills: credential-incident-response 1022 (2 chars spare), pi-extensions 1003 (21), pi-devbox-firstboot 952, pi-devbox-environment 923, mempalace 254. Two of five are within one sentence of the limit, so "nobody would write a description that long" is already false here. It also catches the sibling failure that is a silent DROP rather than a warning: a missing or empty description returns skill: null, so the file ships, the symlink resolves, smoke-test's `test -f` passes, pi-devbox-version lists it -- and no agent can ever see it. Name rules are checked too, including against the directory, because the loader falls back to parentDirName and a mismatch silently renames the skill while every path-based pointer (the managed AGENTS block included) keeps naming the directory. Truth table, each mutation verified to differ materially from the shipped tree: shipped 0; over-long description 1; missing description 1 (reported as fatal, not a mere violation); name != directory 1; no frontmatter 1; directory with no SKILL.md 1; absent skills tree 2. Exit codes match lint-shell.sh, check-skill-floor.sh and check-doc-drift.sh -- a gate that cannot run must not pass. python3-yaml rather than grep/awk because `description: >-` is a folded block scalar whose value spans lines and joins with spaces, so a line-oriented length check measures the wrong string. Same package the actionlint job already installs, same parser check-doc-drift.sh already uses. The vendored copies (pi-extensions, mempalace) are deliberately in scope: an upstream edit over the limit turns this red on a file this repo does not author, which is precedented by check-skill-floor.sh's coupling and correct because the image ships those bytes. The fix in that case is to refresh the snapshot, not to exempt the file.
This commit is contained in:
@@ -249,3 +249,49 @@ jobs:
|
||||
|
||||
- name: Doc claims match the build files
|
||||
run: bash scripts/check-doc-drift.sh
|
||||
|
||||
skill-frontmatter:
|
||||
# Gate the image-baked skills' frontmatter against the spec pi's own loader
|
||||
# measures. Its own job, for the same reason as skill-floor and doc-drift:
|
||||
# "a shipped skill violates the spec" should be a distinct red name, not a
|
||||
# line buried in a job about workflow syntax.
|
||||
#
|
||||
# The gap it closes, measured 2026-10-23 while adding pi-devbox-firstboot:
|
||||
# that skill's description was drafted at 1056 chars against the loader's
|
||||
# MAX_DESCRIPTION_LENGTH of 1024. pi does NOT reject it — dist/core/skills.js
|
||||
# pushes a diagnostic and then loads the skill anyway ("Still load the skill
|
||||
# even with warnings, unless description is missing or empty"). So the only
|
||||
# thing that would ever report the violation is a diagnostic inside a live pi
|
||||
# session, and nothing in CI starts one. A spec violation that still works is
|
||||
# the most durable kind of rot: nothing forces the fix.
|
||||
#
|
||||
# Not free in the way skill-floor was free: two of the five baked skills are
|
||||
# already within ~20 chars of the limit (credential-incident-response 1022,
|
||||
# pi-extensions 1003), so this fires on a plausible next edit rather than on
|
||||
# a hypothetical one.
|
||||
#
|
||||
# The sibling failure it also catches is a silent DROP rather than a warning:
|
||||
# a missing/empty description returns skill: null, so the file ships, the
|
||||
# symlink resolves, smoke-test's `test -f` passes, pi-devbox-version lists
|
||||
# it — and no agent can ever see it.
|
||||
#
|
||||
# Exit codes 0 valid / 1 violation / 2 cannot-run, matching lint-shell.sh,
|
||||
# check-skill-floor.sh and check-doc-drift.sh. A missing skills tree or a
|
||||
# python3 without the yaml module is a red 2, not a green tick.
|
||||
runs-on: ubuntu-latest
|
||||
container:
|
||||
image: catthehacker/ubuntu:act-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
|
||||
# python3-yaml because `description: >-` is a folded block scalar: the
|
||||
# value spans lines and joins with spaces, so a line-oriented length
|
||||
# check measures the wrong string. Same package the actionlint job
|
||||
# installs, same parser check-doc-drift.sh already relies on.
|
||||
- name: Install python3-yaml
|
||||
run: |
|
||||
apt-get update
|
||||
apt-get install -y --no-install-recommends python3-yaml
|
||||
|
||||
- name: Baked skill frontmatter is valid
|
||||
run: bash scripts/check-skill-frontmatter.sh
|
||||
|
||||
Reference in New Issue
Block a user