Skills regulate how agents work

Give an agent a job without a procedure, and it has to discover one on its own. It looks around the codebase, tries an approach, reconsiders it and asks questions it could have asked up front - and the next time, it may take a different route altogether. The result depends on the path it happened to take.

A skill writes that procedure down. It’s a folder with a SKILL.md file - a short header (the frontmatter) with a name and a description, followed by instructions. Until the skill is used, the agent only sees its name and description. When a request matches, or when you call the skill directly, the whole file is loaded into the agent’s context - everything the model sees while it works.

A skill turns the improvised path into a defined role. Think of every skill as an actor with a job description, and of the skill file as the rules that actor works by. Done well, it makes the agent’s work shorter and its results consistent. That’s the kind of efficiency this article is about.

The format was introduced by Anthropic and is now an open standard (agentskills.io(opens in new tab)), supported by Claude Code, Codex, Cursor, GitHub Copilot, Gemini CLI, OpenCode and others.

The example: a release-notes skill

To keep things concrete, this article follows one skill from idea to evaluation: a release-notes skill that turns the pull requests merged since the last release into release notes. It’s a small job, but it has everything the patterns below deal with - a defined input and output, a reader with expectations, a repetitive structure and plenty of room to fill gaps with guesses.

Step 1: Decide what deserves a skill

Not every task needs a skill. If the agent already does something reliably on its own, a skill only adds instructions it doesn’t need. Skills pay off in three situations.

Inner loops - procedures that run many times over. Fixing a failing build is one: run the checks, read the first failure, fix it, run again. A skill that defines the loop - which command to run, how much of the output to read, when to stop and ask - keeps every pass short, instead of letting the agent re-read the whole log or touch unrelated code on every round.

Large, repetitive tasks - work that’s simple per item, but has to be done dozens of times without drifting. Release notes are one: every entry is a single line, but a release can have dozens, and each needs the same format, the right section and a link to its pull request. Without a skill, the structure tends to shift a little with every entry and every run, and whoever reads the notes next - a person or another agent - can’t rely on it.

Processes the agent isn’t good at on its own - tasks where, left alone, it searches too much and goes back and forth. An accessibility review is one of them: without guidance, the agent doesn’t know which areas matter, and ends up either skimming or checking everything. A skill can give it a checklist (contrast, keyboard navigation, labels, focus order, alternative text) and a rule that every finding names the guideline it breaks.

When a job needs more than one skill, plan the set as a whole. One skill is one actor with one job, and the output of each is the input of the next. Boundaries keep the actors from doing each other’s work: the release-notes skill describes what changed for users, never how it was built, and a skill that drafts the release announcement works from those notes, never from the raw pull requests. Clear hand-offs keep every skill focused, and you can stop after any stage and still have something useful.

Step 2: Talk it through

I don’t start by writing the skill. I start by describing the job to the agent, the same way I’d explain it to a new colleague - what they receive, what they hand over, who reads their work and what they must never do. It’s a brain-dump without much structure, and that’s fine.

Then I ask the agent to interview me. The questions I can’t answer straight away are the most valuable part, because they point to exactly the gaps the skill would otherwise fill with guesses.

A starting prompt can be as plain as this:

> I want a skill that turns the pull requests merged since the last
  release into release notes. The notes are read by users looking for
  changes that affect them, so they should describe what changed, never
  how it was built. It must not invent changes the pull requests don't
  show. Before you write anything, ask me questions until you understand
  the job.

By the end of the conversation, four things should be clear, because they end up at the very top of the skill:

• Input - what the skill reads, and what to do when it’s missing
• Output - what it writes and where
• Consumer - who reads the output, a person or the next agent in the chain
• Boundary - what’s out of scope, and which decisions stay with the human

Step 3: Get a readable draft

The first draft the agent produces is prose. It’s written for me, and at this point that’s the right audience - I have to read it, understand it and agree with it, before it becomes the agent’s instructions.

When reviewing the draft, I look for behaviour I didn’t ask for:

• Decisions made on my behalf - picking a format, a category or a default the source doesn’t mention
• Gaps filled quietly - anything assumed instead of asked
• Wandering - places where the agent would have to search or guess to find what it needs
• Re-runs - what happens when the output already exists: does it overwrite, duplicate or update?
• No human around - what happens in CI (automated pipelines) or a headless run, when there’s nobody to answer questions

Each of these should end up as an explicit rule. For example, when the release-notes skill finds notes for the same release, it can switch to an update mode that adds the new entries and keeps the existing ones, instead of writing the file from scratch.

Step 4: Rewrite it for the agent

Once the draft says the right thing, I ask the agent to rewrite it as instructions for itself. The goal isn’t a shorter file, it’s precision. Prose leaves room for interpretation, and an agent may read the same paragraph differently tomorrow, or on a different model. Clear rules, conditions and templates produce the same behaviour run after run.

I ran into the same tension with specs in spec-driven development. Agents default to long, human-readable output, and turning it into precise instructions is a separate, deliberate step.

Here’s what the top of the release-notes skill could look like after the rewrite:

INPUT: `$ARGUMENTS` = release tag. Empty/unknown → ask for tag, halt.
OUTPUT: `docs/releases/<tag>.md` — one entry per merged PR, grouped by section.
CONSUMER: Users scanning for changes that affect them. Optimize for skimming: one line per change, user-facing wording, PR link on every entry.
BOUNDARY: describe WHAT changed for users. Never HOW it was built.

Four lines, and the agent knows what it reads, what it writes, who it writes for and where its job ends. That’s the whole conversation from step 2, condensed.

These are the patterns that made the biggest difference.

Number the rules and the steps

Rules become R1…R8 and the procedure becomes P1…P5. A step can refer to a rule (“apply R7”) instead of repeating it, and when a run goes wrong, I can point at the exact rule that was ignored.

- R2 Every entry links its PR: `([#<number>](<url>))`. No PR → question, not entry.
- R6 Never describe a change the PR doesn't show. Commonly overstated — always check: performance claims, "fixed" without a linked issue, breaking changes, security wording, deprecations.

Write conditions, not paragraphs

Empty/unknown → ask for tag, halt. is a condition and an action, with nothing left to interpret. The same instruction written as a paragraph tends to pick up softeners like “try to”, “if possible” or “usually”, and each one is a decision the agent has to make on its own, possibly differently every time.

Pick one term per concept and stick to it. If the draft says “feature” in one place and “capability” in another, the agent may treat them as two different things.

Show, don’t describe

Agents follow patterns well. Instead of describing what an entry should look like, the skill includes the template. The same goes for lookups - a table that maps signals in a pull request (a feat: prefix, a bug label) to a section of the notes (Added, Fixed) replaces a paragraph of explanation. As a bonus, whoever reads the output next can rely on the structure, because it’s always the same.

Tell it where to look

This is where most of the wasted work goes. Without directions, an agent searches, reads more than it needs and sometimes reads the same file twice. A skill can spell out where to look instead.

The release-notes skill doesn’t need to browse the repository at all - one command gives it exactly the pull requests in scope:

P1 Collect. `gh pr list --state merged --search "merged:>=<last release date>" --limit 200 --json number,title,labels,url`. Read nothing else unless an entry is unclear.

In update mode, it checks what’s already in the notes with an exact command, instead of reading the whole file:

- R7 Idempotent. Look up by PR number (`grep -n "#<number>" docs/releases/<tag>.md`) before adding an entry. Never duplicate. Never remove an existing entry.

Say what’s forbidden and what to do when unsure

A helpful agent fills gaps, and in release notes, a filled gap is a promise nobody made. A skill closes those gaps in two ways:

• What’s forbidden - a list of things the notes never contain, from implementation details to marketing claims
• What to do when unsure - anything that can’t be traced to a pull request becomes a question, not an entry

Skills that involve real choices should follow one principle - propose, never decide. Asking early is much cheaper than undoing a guess later.

Plan for every mode

A skill doesn’t always run with a human at the keyboard. Give every skill a short section on how to behave in each mode:

• Plan mode - the agent proposes before it acts, so the skill renders every file it would write into the plan, and nothing is written until you approve it
• Normal mode - it asks questions as it goes
• Nobody to answer - it lists the open questions at the end of its output and leaves out anything they affect

Use the frontmatter

The description is what the agent uses to decide whether to load the skill, so it has to say what the skill does and when to use it. A skill meant to run only on request - like release notes, which you write when you cut a release - can set disable-model-invocation: true and end the description with “Manual invocation only.” for agents that ignore that field. allowed-tools pre-approves the read-only tools a skill needs, so it doesn’t stop to ask for permission on every file read.

Move repetitive work into tools

When a skill needs the same parsing, validation or query on every run, don’t have the agent work it out each time - give it an existing command-line tool (CLI) or a small, tested script. The agent runs one command and reads the result, instead of writing throwaway code and debugging it. The release-notes skill leans on one: gh with --json output, instead of the agent opening pull requests one by one.

Minko Gechev’s best practices guide recommends bundling such scripts in a scripts/ folder, and moving reference material into references/ files that the agent reads only when a step calls for them. That keeps SKILL.md focused on the procedure, without details for steps the agent isn’t on yet.

Step 5: Grade it

In Minko’s words, “skills are instructions for agents, and instructions need tests”. Models change, a small edit to one rule can change the behaviour of another, and since the output is non-deterministic, one good run proves very little. You need repeated runs and a pass rate.

There are four things worth testing:

• Discovery - does the skill load when it should, and stay out of the way when it shouldn’t? This is all about the description.
• Logic - can the agent follow the procedure without having to guess?
• Outcome - does the result meet the bar, run after run?
• Efficiency - does the agent get there with fewer searches, commands and retries than without the skill?

Start with an LLM review

The cheapest check needs no tooling. Minko’s guide includes three ready-made prompts for it:

• Discovery - generates prompts that should and shouldn’t trigger the skill, based on its description alone
• Logic - asks a model to simulate running the skill step by step and flag every line where it has to guess
• Edge cases - has the model act as a QA tester, hunting for edge cases

Run them in a fresh session, so the model only knows what the skill tells it.

Compare models with an arbiter

For more complex skills, I ran each skill on several models and gave the results to a separate model acting as an arbiter. The arbiter compared the runs, measured how efficiently each model got to its result, reviewed the skill itself and proposed improvements.

For simpler, more straightforward skills, where the same input should give the same result every time, I use an eval tool like skillgrade instead.

Repeatable evals with skillgrade

To measure a skill over many runs, Minko also built a dedicated tool.

Getting started takes a few commands:

npm i -g skillgrade
cd my-skill/
ANTHROPIC_API_KEY=... skillgrade init
ANTHROPIC_API_KEY=... skillgrade --smoke

init generates an eval.yaml with tasks and graders based on the skill. Each task gives the agent an instruction and a workspace of fixture files, and graders score the result:

• Deterministic graders run a script that inspects the workspace and prints a score between 0 and 1
• LLM rubric graders have a model score the session transcript against a rubric you write

Since rubric graders see the whole transcript, they can judge the process as well as the result - whether the agent followed the procedure, and whether it got there without wandering. Each grader has a weight and the final score is the weighted average.

Presets control how many times each task runs: --smoke (5 trials) for a quick check while editing, --reliable (15) for a solid pass-rate estimate and --regression (30) before shipping. With --ci, it exits with an error when the pass rate falls below the threshold (0.8 by default), so it can run on every change. It can drive Claude, Codex, Gemini and OpenCode agents, which makes it easy to compare the same skill across agents.

For the release-notes skill, a first task could look like this:

version: "1"

defaults:
  agent: claude
  trials: 5
  threshold: 0.8

tasks:
  - name: merged-prs-to-release-notes
    instruction: |
      Run the release-notes skill for v2.4.0, using the pull requests
      in merged-prs.json instead of gh.
      Nobody is available to answer questions.
    workspace:
      - src: fixtures/merged-prs.json
        dest: merged-prs.json
    graders:
      - type: deterministic
        run: bash graders/structure.sh
        weight: 0.6
      - type: llm_rubric
        rubric: |
          Did the agent turn unclear pull requests into open questions,
          instead of inventing changes?
          Do the notes describe WHAT changed for users, without
          implementation details?
          Did it follow the procedure in order, without re-reading files
          or retrying steps?
        weight: 0.4

The script checks what can be checked mechanically - every entry links its pull request, and no implementation details leak into the notes:

#!/bin/bash
notes=docs/releases/v2.4.0.md
[ -f "$notes" ] || { echo '{"score":0,"details":"no release notes written"}'; exit 0; }
total=0; passed=0

# every entry links its pull request
while read -r entry; do
  total=$((total + 1))
  grep -qE '\(\[#[0-9]+\]\(https://' <<< "$entry" && passed=$((passed + 1))
done < <(grep '^- ' "$notes")

# no implementation details leak into the notes
total=$((total + 1))
grep -qiE 'refactor|dependency bump|internal' "$notes" || passed=$((passed + 1))

score=$(awk "BEGIN {printf \"%.2f\", $passed/$total}")
echo "{\"score\":$score,\"details\":\"$passed/$total checks passed\"}"

The rubric covers what a script can’t - whether the agent asked instead of inventing, and how it got to the result.

Two rules from earlier make this skill easy to grade. With nobody to answer, it lists unclear pull requests as open questions instead of guessing, so a run without a human is repeatable. And since every entry links its pull request, traceability is something a script can check. One limit is invocation - a manual-only skill doesn’t load on its own, so discovery isn’t in play and every task has to call the skill explicitly.

skill-creator for Claude Code

If you work in Claude Code, Anthropic’s skill-creator covers the whole loop, from the first draft to evaluation. It runs each test prompt with and without the skill (or against its previous version), grades the results against assertions and aggregates pass rate, time and tokens into a benchmark. That comparison is the real measure of efficiency - it shows whether the skill actually makes the work shorter and better, not just different. It also has a separate loop for tuning the description, based on around 20 queries that should and shouldn’t trigger the skill.

Treat skills like code

A skill is a program the agent runs on your behalf, so it deserves the same care as code: plan it, review it, version it and test it. Testing is the step most skills skip, and it’s the only one that proves the others worked.

If you want examples to start from, Anthropic’s public skills repository is a good place to look, and skills.sh(opens in new tab) lists skills from across the community. The skills(opens in new tab) CLI installs any of them with one command - npx skills add <owner/repo> - into Claude Code, Codex, Cursor, OpenCode and many other agents. If you’ve found other ways to test skills, let me know in the comments. An agent without a procedure will always find a path. A skill decides which one.