Test plugins with evals changedplugin-evals
Nearest release: v2.1.300, published an hour before upstream edited the page. Shown because the two are within 24 hours of each other. Nothing here says the release caused the edit.
Upstream edited this page at 14 Oct 2026 15:54 UTC, give or take a minute or two: the time comes from Anthropic’s own sitemap rather than from a commit. This site recorded the change at 20 Oct 2026 18:37 UTC.
Upstream edited
Recorded here
Lines+81added
Lines−119removed
From line
14
where the diff opens
First seen
11 Sep 2026
this site's first read of the page
Recorded edits19to this page, all time
## Write and refine cases ### Set run limits and tools in prompt.md ### Choose and weight graders ### Score against the no-plugin baseline ### Seed the workspace or conversation ### Mock MCP servers ### Run evals in CI ## What a run can access ### How runs are isolated ### prompt.md frontmatter ### case.yaml fields ### Mock files #### Regex patterns in expect ### "is too old for claude plugin eval" ### "cannot be evaluated where sessions load it" ### `--json output path must end in .json` ### "ended on an API error and scored 0 without grading" ### "mock call budget exceeded"
The whole hunk
from line 14, old and new numbered
/
from line 14
1414
1515This page is for plugin and skill authors who have a working plugin and want to test its behavior, and for teams that gate plugin changes in CI. For iterating on one skill inside a Claude Code conversation, the [skill-creator plugin](/docs/en/skills#run-evals-with-skill-creator) runs a similar comparison with its own `evals/evals.json` format, and neither tool reads the other's case files. To create a plugin, see [Create a plugin](/docs/en/plugins/create); to check a plugin's files for syntax and schema errors rather than its behavior, use [`claude plugin validate`](/docs/en/plugins/cli-reference#plugin-validate).
1616
17<Note>
18 Every eval run and every judge grader is a real model call on your account, counted against your plan's usage or your API bill, so check the [requirements](#requirements) first. Then [create your first eval suite](#create-your-first-eval-suite), or go to [Run evals in CI](#run-evals-in-ci) if you already have one.
19</Note>
17> [!NOTE]
18>
19> Every eval run and every judge grader is a real model call on your account, counted against your plan's usage or your API bill, so check the [requirements](#requirements) first. Then [create your first eval suite](#create-your-first-eval-suite), or go to [Run evals in CI](#run-evals-in-ci) if you already have one.
2020
2121## Requirements
2222
from line 55
5555* A terminal open at your plugin's root directory, the one containing `plugin.json` or `.claude-plugin/plugin.json`
5656* One skill in the plugin you want to test, and a request a user would type that should trigger it
5757
58<Steps>
59 <Step title="Create the cases">
60 From the plugin root, run:
58**1. Create the cases**
6159
62 ```bash theme={null}
63 claude plugin eval init
64 ```
60From the plugin root, run:
6561
66 If Claude Code doesn't already trust this directory it first asks `Trust this plugin directory?`; answer `y`.
62```bash theme={null}
63claude plugin eval init
64```
6765
68 An interactive Claude Code session then opens. Claude reads your plugin and asks you what a good result looks like, proposes prompts that should and shouldn't trigger the plugin, designs graders for each, runs them once as a trial to check they behave, and writes one case directory per prompt under `evals/`, each named after its prompt.
66If Claude Code doesn't already trust this directory it first asks `Trust this plugin directory?`; answer `y`.
6967
70 When Claude tells you the suite is ready, exit that session with `/exit` or Ctrl+D to return to your shell.
68An interactive Claude Code session then opens. Claude reads your plugin and asks you what a good result looks like, proposes prompts that should and shouldn't trigger the plugin, designs graders for each, runs them once as a trial to check they behave, and writes one case directory per prompt under `evals/`, each named after its prompt.
7169
72 If you already have a Claude Code session open at the plugin root, you can instead ask Claude there to run `claude plugin eval init`. Claude runs the command and then asks you the same questions in that conversation.
70When Claude tells you the suite is ready, exit that session with `/exit` or Ctrl+D to return to your shell.
7371
74 If you'd rather write a case yourself to see exactly what the files contain, follow [Write a case manually](#write-a-case-manually) and come back here to run it.
75 </Step>
72If you already have a Claude Code session open at the plugin root, you can instead ask Claude there to run `claude plugin eval init`. Claude runs the command and then asks you the same questions in that conversation.
7673
77 <Step title="Run the suite">
78 Back at your shell in the plugin root, run every case under `evals/`:
74If you'd rather write a case yourself to see exactly what the files contain, follow [Write a case manually](#write-a-case-manually) and come back here to run it.
7975
80 ```bash theme={null}
81 claude plugin eval .
82 ```
76**2. Run the suite**
8377
84 You already trusted this directory during step 1, so the run starts immediately. If you wrote the case manually instead, the run first asks `Trust this plugin directory? [y/N]`; answer `y`. [What a run can access](#security) explains what you're agreeing to.
78Back at your shell in the plugin root, run every case under `evals/`:
8579
86 Each case runs three times with your plugin and three times without it, so one case is six runs. A progress line prints as each run finishes, with that run's score and each grader's verdict.
87 </Step>
80```bash theme={null}
81claude plugin eval .
82```
8883
89 <Step title="Read the summary">
90 When the suite finishes you see a summary table, followed by where the report went:
84You already trusted this directory during step 1, so the run starts immediately. If you wrote the case manually instead, the run first asks `Trust this plugin directory? [y/N]`; answer `y`. [What a run can access](#security) explains what you're agreeing to.
9185
92 ```text theme={null}
93 CASE WITH W/OUT Δ RUNS COST NOTES
94 first-case 1.00 0.33 +0.67 6 $0.41
86Each case runs three times with your plugin and three times without it, so one case is six runs. A progress line prints as each run finishes, with that run's score and each grader's verdict.
9587
96 1 case(s) · mean Δ +0.67 · 74s · $0.41
97 Report: /Users/you/my-plugin/evals/results/2026-09-10T17-02-11-482Z/report.html
98 Published: https://claude.ai/... · keep local next time with --no-publish
99 ```
88**3. Read the summary**
10089
101 `WITH` is the case's score with your plugin loaded, `W/OUT` is the score without it, and a positive `Δ` means the plugin raised the score. `COST` is a list-price estimate of the model calls, and `NOTES` shows the highest-weight failing grader's explanation, or the run's error, from the with-arm.
102 </Step>
90When the suite finishes you see a summary table, followed by where the report went:
10391
104 <Step title="Open the report and iterate">
105 Open the `Published:` URL, or the `Report:` path when no `Published:` line appears, to see each grader's verdict and explanation for every run, and for `llm` graders the judge's votes and the excerpt it judged. The `Published:` line appears only when your account can [publish reports](#html-report).
92```text theme={null}
93CASE WITH W/OUT Δ RUNS COST NOTES
94first-case 1.00 0.33 +0.67 6 $0.41
10695
107 The most common first finding is a `Δ` near zero with the case's `tool_used: Skill` grader failing, which means Claude isn't choosing your skill on natural phrasing. Adjust the skill's [`description`](/docs/en/skills#frontmatter-reference), run `claude plugin eval .` again, and compare.
961 case(s) · mean Δ +0.67 · 74s · $0.41
97Report: /Users/you/my-plugin/evals/results/2026-09-10T17-02-11-482Z/report.html
98Published: https://claude.ai/... · keep local next time with --no-publish
99```
108100
109 To iterate on one case with fewer runs, run a single arm once. A single run is noisy, so confirm any change at the default three runs before you trust it. With one arm the table shows `SCORE` and `PASS%` columns instead of `WITH`, `W/OUT`, and `Δ`:
101`WITH` is the case's score with your plugin loaded, `W/OUT` is the score without it, and a positive `Δ` means the plugin raised the score. `COST` is a list-price estimate of the model calls, and `NOTES` shows the highest-weight failing grader's explanation, or the run's error, from the with-arm.
110102
111 ```bash theme={null}
112 claude plugin eval . --case <case-name> --runs 1 --ablation none
113 ```
103**4. Open the report and iterate**
114104
115 Replace `<case-name>` with one of the directory names under `evals/`.
116 </Step>
117</Steps>
105Open the `Published:` URL, or the `Report:` path when no `Published:` line appears, to see each grader's verdict and explanation for every run, and for `llm` graders the judge's votes and the excerpt it judged. The `Published:` line appears only when your account can [publish reports](#html-report).
118106
119<h2 id="write-and-refine-cases">
120 Write and refine cases
121</h2>
107The most common first finding is a `Δ` near zero with the case's `tool_used: Skill` grader failing, which means Claude isn't choosing your skill on natural phrasing. Adjust the skill's [`description`](/docs/en/skills#frontmatter-reference), run `claude plugin eval .` again, and compare.
122108
109To iterate on one case with fewer runs, run a single arm once. A single run is noisy, so confirm any change at the default three runs before you trust it. With one arm the table shows `SCORE` and `PASS%` columns instead of `WITH`, `W/OUT`, and `Δ`:
110
111```bash theme={null}
112claude plugin eval . --case <case-name> --runs 1 --ablation none
113```
114
115Replace `<case-name>` with one of the directory names under `evals/`.
116
117## Write and refine cases
118
123119The cases `claude plugin eval init` writes are plain files you can open, change, and add to. A case is a directory under the plugin's eval directory that contains a `prompt.md`, a `case.yaml`, or both. Give each case at least one grader, as a `graders/<name>.md` file or a `graders:` entry in `case.yaml`, because a case without one fails to load. To group cases, nest them under a directory that isn't itself a case; anything inside a case directory, such as `graders/` and fixture files, belongs to that case.
124120
125121This is the layout `claude plugin eval init` writes and the one to use for new suites. The [eval suite reference](#eval-suite-reference) has the complete tree, including mocks and results:
from line 193
197193
198194With both files saved, run the case the way the [quickstart](#create-your-first-eval-suite) does, with `claude plugin eval .` from the plugin root.
199195
200<h3 id="set-run-limits-and-tools-in-prompt-md">
201 Set run limits and tools in prompt.md
202</h3>
196### Set run limits and tools in prompt.md
203197
204198Set a case's `max_turns`, `timeout_seconds`, `model`, `tags`, and the `allowed_tools` it may use in `prompt.md` frontmatter; the [prompt.md frontmatter](#prompt-md-fields) reference lists every field and its default.
205199
206200Claude receives the body exactly as you wrote it. `@path` mentions in it aren't expanded into file attachments, so if Claude needs to read a file, grant a tool for it in `allowed_tools`.
207201
208<h3 id="grade-the-result">
209 Choose and weight graders
210</h3>
202### Choose and weight graders
211203
212204A grader's frontmatter sets its `type`, and optionally a `weight` that makes it count for more of the run's score and an [`arm`](#compare-against-a-no-plugin-baseline) that controls how it's scored against the baseline. Of the six types, `regex`, `tool_used`, `tool_order`, and `file_exists` are computed from the transcript and files and cost nothing, while `llm` and `baseline` call a judge model and add to the run's cost.
213205
from line 218
226218* If a case's `tool_used: Skill` grader passes but `Δ` is negative, suspect the judge before the plugin. A small judge model can mark a correct answer wrong because it's formatted differently from what the rubric describes. Re-run with `--judge-model sonnet`, and tighten the rubric so formatting doesn't decide the verdict.
227219* To check that a build or test passed inside the run, have the prompt ask Claude to run it and write the outcome to a file, grade that file, and assert the command ran with a `tool_used` grader whose `input_match` names the command.
228220
229<h3 id="compare-against-a-no-plugin-baseline">
230 Score against the no-plugin baseline
231</h3>
221### Score against the no-plugin baseline
232222
233223When a plugin is under test, a case normally runs in two arms. The with-arm is its runs with the plugin loaded, and the without-arm is the same number of runs with no plugin at all. The summary and report show both scores and `Δ`, the with-arm score minus the without-arm score.
234224
from line 253
263253
264254A case can need more than a prompt: files or a git repository in the workspace, an earlier conversation to continue, or answers from the MCP servers your plugin connects to. Each of those is set up beside the case so runs stay repeatable.
265255
266<h3 id="add-setup-or-history-with-case-yaml">
267 Seed the workspace or conversation
268</h3>
256### Seed the workspace or conversation
269257
270258Each run starts in an empty workspace. When a case needs more than the prompt, add a `case.yaml` beside `prompt.md` with a `context` block:
271259
from line 276
288276
289277A scaffold script starts in the empty workspace with a small fixed environment: your shell's `PATH`, `HOME` set to the run's temporary home directory, `TMPDIR`, and a few constants such as `TERM=dumb`. Nothing else from your shell reaches it, and neither do the case's `EVAL_*` variables. If the script exits non-zero or runs longer than 120 seconds, that run scores 0 with a `scaffold failed` error. Use the script for files and git state only, since project configuration it writes [isn't loaded](#how-runs-are-isolated).
290278
291<h3 id="mock-mcp-servers">
292 Mock MCP servers
293</h3>
279### Mock MCP servers
294280
295281You can evaluate a plugin whose skills call MCP tools without the real service behind them. Put one Markdown file per tool under `evals/mocks/<server>/<tool>.md` for the whole suite, or under a case's own `mocks/` directory for one case, where `<server>` is the server's name in your plugin's [MCP configuration](/docs/en/plugins/components#mcp-servers).
296282
from line 374
388374| `--publish-report` | | Publish the report even where it would stay local by default, such as a run a Claude Code session started |
389375| `--keep-temp` | Off | Keep every run's sandbox directory and print its path, for debugging what Claude produced |
390376
391<h3 id="run-evals-in-ci">
392 Run evals in CI
393</h3>
377### Run evals in CI
394378
395379In your CI job, run the suite with `--json` to write the result for archiving, and fail the build on the exit code. Pass `--trust-plugin` so the job never waits at the [first-run trust prompt](#security), pin both models so scores are comparable over time, keep the report local, and set a cost ceiling as an upper limit:
396380
from line 419
435419
436420`report.html` is a single self-contained file that makes no external requests, so you can attach it to a CI job or open it from disk. This example is the top of a report for a three-case suite run with `--threshold 0.8`; the cost shown is a list-price estimate and varies with the model and the number of cases:
437421
438<img src="https://mintcdn.com/claude-code/qq7LHDi_F0aeFHgk/images/plugin-eval-report.png?fit=max&auto=format&n=qq7LHDi_F0aeFHgk&q=85&s=106eb6e6a70a6565f891ea3a4564f87d" alt="Top of an eval report: a verdict line reading "Plugin effect: +33.3 pts vs baseline, improved 2, flat 1, regressed 0 of 3 cases", five summary tiles for suite score, ablation delta, baseline score, cases passing the threshold, and perfect runs, then the first case with its delta, score bar, and one run whose two graders both show pass" width="1360" height="1032" data-path="images/plugin-eval-report.png" />
422
439423
440424Read it from the top down:
441425
from line 452
468452| `cases[].arms.with[].skippedPaidGraders` | `true` when the cost ceiling skipped this run's judge graders, so its score isn't comparable |
469453| `costUsd`, `durationSeconds`, `claudeVersion` | Estimated cost at list price including judge calls, wall-clock seconds, and the Claude Code version that ran the suite |
470454
471<h2 id="security">
472 What a run can access
473</h2>
455## What a run can access
474456
475457`claude plugin eval` loads the target plugin's skills, hooks, and agents and runs its eval suite on your machine, as you. Pointing it at a plugin is the same trust decision as `claude --plugin-dir`, so only evaluate plugins you trust.
476458
from line 472
490472
491473When the plugin includes hooks you didn't write, or you start its real MCP servers, treat its scores as advisory unless you ran it in an isolated environment such as a container or CI runner, since hooks and servers run outside the agent's sandbox and could modify the files the graders read.
492474
493<h3 id="how-runs-are-isolated">
494 How runs are isolated
495</h3>
475### How runs are isolated
496476
497477Each run gets a temporary home directory, working directory, and Claude Code configuration, and the agent under test runs there as a `claude -p` child process with only your plugin loaded. Keep these consequences in mind when you write cases:
498478
from line 508
528508 └── mock-recordings/ # agent-mock answers from clean runs, with ADOPT.txt
529509```
530510
531<h3 id="prompt-md-fields">
532 prompt.md frontmatter
533</h3>
511### prompt.md frontmatter
534512
535513`prompt.md` frontmatter accepts these fields. An unknown key is an error:
536514
from line 528
550528| `append_system_prompt` | | Text appended to the child session's system prompt |
551529| `env` | `{}` | Extra environment variables for the child session. Keys must match `EVAL_[A-Z0-9_]*`; any other key fails the run. The run inherits only an allowlist from your shell: basics such as `PATH` and locale, proxy and certificate settings, the variables that select and authenticate your model provider, most `ANTHROPIC_*` and `CLAUDE_CODE_*` configuration, and `EVAL_*`. To pass the plugin anything else, such as a toolchain setting, export it as an `EVAL_*` variable |
552530
553<h3 id="case-yaml-fields">
554 case.yaml fields
555</h3>
531### case.yaml fields
556532
557533`case.yaml` is an alternative or companion to `prompt.md`: it describes a case in YAML and adds the fields that point at other files. It requires `schema_version: "1.1"` and `name`. The `prompt.md` fields `description`, `tags`, `plugins`, `runs`, and `expected_outcome` go at the top level; `model`, `max_turns`, `timeout_seconds`, `allowed_tools`, `append_system_prompt`, and `env` go under `execution:`. When both files exist, `prompt.md` frontmatter overrides the matching `case.yaml` fields, the `prompt.md` body is the prompt, and `graders/*.md` are added after any graders listed in `case.yaml`.
558534
from line 577
601577| `llm` | `criteria`, `focus` | A judge model votes PASS on the rubric in at least two of three votes. In the `.md` layout the file body is the criteria |
602578| `baseline` | `baseline_file`, `criteria` | A judge finds the run satisfies the criteria at least as well as the reference transcript at `baseline_file`, a `.jsonl` in the case directory |
603579
604<h3 id="mock-files">
605 Mock files
606</h3>
580### Mock files
607581
608582A `<tool>.md` file under `mocks/<server>/` answers one tool. Its body is the tool result, with `{{input.<field>}}` and `{{file:fixtures/<name>}}` substitutions. Its frontmatter accepts these keys:
609583
from line 595
621595
622596A case's own `mocks/` directory uses the same layout and overrides the suite's mocks file by file.
623597
624<h4 id="expect-patterns">
625 Regex patterns in expect
626</h4>
598#### Regex patterns in expect
627599
628600A `/regex/` value in `expect:` uses a small dialect that Claude Code checks when it loads the suite:
629601
from line 624
652624
653625This is the first run against a directory Claude Code doesn't trust yet, and it can't ask you because stdin or stdout isn't a terminal or you passed `--json`. Run `claude plugin eval <dir>` once in a terminal and answer the prompt, or pass `--trust-plugin` if you trust the plugin's code and suite. See [What a run can access](#security).
654626
655<h3 id="git-is-too-old-for-claude-plugin-eval">
656 "is too old for claude plugin eval"
657</h3>
627### "is too old for claude plugin eval"
658628
659629The `git` on your `PATH` is older than 2.31, so `claude plugin eval` stopped before running any case and exited 1 with a message naming your version:
660630
from line 636
666636
667637Before v2.1.283, `claude plugin eval` didn't check the git version, and on an older git the suite ran with those programs left on.
668638
669<h3 id="cannot-be-evaluated-where-sessions-load-it">
670 "cannot be evaluated where sessions load it"
671</h3>
639### "cannot be evaluated where sessions load it"
672640
673641You ran `claude plugin eval` in your shell with the name of a plugin installed from a marketplace you [added from a local path](/docs/en/plugins/loading#in-place-and-copied-plugins). No session could load the plugin from that marketplace now, so the command refuses with this message and exits with code 1.
674642
from line 688
720688
721689The default `--threshold` is 1.0, so the command exits 1 when any case scores below perfect. Set a threshold that matches the score you require.
722690
723<h3 id="json-output-path-must-end-in-json">
724 `--json output path must end in .json`
725</h3>
691### `--json output path must end in .json`
726692
727693You put the target after `--json`, so it was read as the output path. Put the target first, as in `claude plugin eval . --json`, or give `--json` an explicit `.json` path.
728694
from line 696
730696
731697That grader is excluded from the score by design in a two-arm run, and its `scored` field is `false`. See [Score against the no-plugin baseline](#compare-against-a-no-plugin-baseline).
732698
733<h3 id="runs-ended-on-an-api-error">
734 "ended on an API error and scored 0 without grading"
735</h3>
699### "ended on an API error and scored 0 without grading"
736700
737701This stderr warning means the API answered at least one run's last request with an HTTP error, such as a rate limit or a plan usage limit. Such a run scores 0 with no grader results, and the command exits non-zero whatever the scores and `--threshold` are.
738702
from line 713
749713
750714Runs that end with a usage-limit or rate-limit error [score 0 without grading](#runs-ended-on-an-api-error), and the command exits non-zero. The suite still finishes and isn't marked `partial`. Re-run after the limit resets, with `--runs 1` or a `--case` filter if you need to stay under it.
751715
752<h3 id="mock-call-budget-exceeded">
753 "mock call budget exceeded"
754</h3>
716### "mock call budget exceeded"
755717
756718Every `type: agent` [mock](#mock-mcp-servers) in a run draws on one call budget of four times the case's `max_turns`, which is 40 calls at the default of 10. Calls answered from `.replay/` recordings count too, and the case's `mock budget` progress line prints the budget. A call past it aborts the run with score 0 and this reason, so raise `max_turns` in the case for a skill that makes many calls to agent mocks.
757719
No line in this hunk matches that.