How to validate an AI Agent Skill

The hard part of skill validation is not the YAML. It is that the four worst failures are all silent.

An Agent Skill is two files and a promise: SKILL.md carries the instructions, the references/ beside it carry the detail, and the promise is that the agent will read the second when the first tells it to. Nothing enforces the promise. When it breaks, no exception is raised and no warning is printed - the agent simply skips a step, and the work comes out slightly worse than it did last week.

So “validate the skill” has to mean four different checks, because they break at four different moments.

1. Will anything load it

Start with the mechanical layer, because it is cheap and absolute. The Agent Skills specification asks for frontmatter with name and description, a name of lowercase letters, digits and single inner hyphens, at most 64 characters, equal to the folder name, and a description at most 1024 characters.

python scripts/sqs.py spec ./my-skill

The trap here is not strictness, it is inconsistency between readers. A lenient client warns about a mismatched name and loads the skill anyway; a strict reference validator and the publication path reject it. That is the worst possible split: the skill works on the machine where it was written and fails the first time somebody else touches it.

The same shape repeats for an XML tag in a description, for a vendor word in a name, and for a block-scalar description (description: >-) that is perfectly good YAML and is flagged non-portable by the vendor-neutral validators.

Fix these first. They are the only findings with exactly one right answer, which is why a fixer can apply them for you:

$ sqs.py fix ./my-skill --apply
fixed  renamed-folder  SP004  name: some-other-name -> renamed-folder

2. Does every pointer still resolve

This is the check that earns the tool.

A skill loads in two stages, and the second stage is a set of relative links inside the first. Rename references/format.md to references/layout.md and the link in SKILL.md becomes a pointer to nothing. The agent activates, reads SKILL.md, reaches the line that says “open the house format”, finds nothing, and carries on with what it already knows.

Nobody gets an error. The only symptom is that the output no longer follows the house format, and the obvious explanation - “the model is having a bad day” - is wrong.

python scripts/sqs.py structure ./my-skill

Run this immediately after any rename. That single moment is when almost every defect in this class is created:

And the quieter cousin: budgets. SKILL.md is loaded on every activation, so a 20 KB one costs tokens on every single turn whether or not the task needed them (ST006). A reference is opened whole, so a 30 KB reference cancels the point of two-stage loading (ST007).

3. Do the instructions read like instructions

A skill can pass every check above and still be a bad skill. Two failures dominate.

The description says what, never when. The description is the entire triggering mechanism: it is the only part of the skill the agent sees before deciding whether to load the rest. “An assistant for spreadsheets that reads tables and computes totals” is a fine sentence about a skill and contains no condition under which to reach for it.

$ sqs.py check ./what-not-when
⚠️ what-not-when
     ⚠️  QL002 description says what the skill is and never when to reach for it (SKILL.md)

The steps have no completion criterion. “Read the diff carefully and be thorough” cannot be finished. The agent cannot tell done from not-done, so it stops when it feels like stopping, and the run-to-run variation gets blamed on the model:

     ·  QL006 be thorough, where appropriate - lines 8, 9 (SKILL.md:8)

Rewrite each step to end on something checkable: “stop when every modified model has an entry”, not “make sure it is complete”.

Two more in this class are worth knowing because they are invisible from the inside: a description written about itself (“This skill helps you…”) rather than as an instruction about when to act, and a bundled script that waits for input - agents run in non-interactive shells, so input() hangs the run until something kills it.

These rules are heuristics, and a heuristic that cannot point at a line does not belong in a linter. Every one of them names a file and a line, and each carries a detection confidence and a false-positive risk so you know how much of the judgement is the machine’s:

$ sqs.py explain QL003
QL003  One branch written twice
severity warning
detection confidence low · false positives high

--min-confidence high keeps the filesystem and parse facts and drops the prose heuristics. That is the setting to leave on in CI; the heuristics are for the pass you do by hand.

4. Will it work anywhere but here

The check your own machine cannot make for you. A hard-coded ~/.claude/skills/other-skill/references/x.md works perfectly where it was written and resolves to nothing on any other agent - silently, again, because a missing reference is a skipped step rather than an error.

python scripts/sqs.py compat ./my-skill --harness all

The important discipline in this layer is what it refuses to say. Where a harness’s own documentation is silent, the verdict is UNKNOWN, not NOT_SUPPORTED. An invented incompatibility reads exactly like a real one, and one of them makes every other row worth less.

The order that works

sqs.py fix   ./my-skill --apply          # the mechanical ones
sqs.py check ./my-skill                  # structure, spec, quality, compat, security
sqs.py check ./my-skill --format board   # one line per layer, for deciding
sqs.py all   ./my-skill --strict         # before it leaves your machine

Then the part no validator does: read the skill against a rubric, and if it matters, run it. sqs.py eval --trigger measures how often the skill actually loads on the wordings a human would type, and --runtime runs the same task with the skill and without it. A skill that validates perfectly, fires reliably and changes nothing about the output is still load the agent pays for.

In CI, so it stays true

- uses: letsloose501/sqs-skills@v2
  with:
    path: .
    strict: "true"
    harness: all
    upload-sarif: "true"

A skill repository with this gate cannot acquire a broken pointer without somebody noticing, which is the whole point: the defect class this page is about is the one that nothing else notices.

See also