Test plugins with evals changedplugin-evals
Nearest release: v2.1.283, published 4 hours before upstream edited the page. Shown because the two are within 24 hours of each other. Nothing here says the release caused the edit.
Upstream edited this page at 25 Sep 2026 23:00 UTC, give or take a minute or two: the time comes from Anthropic’s own sitemap rather than from a commit. This site recorded the change at 25 Sep 2026 23:07 UTC.
Upstream edited
Recorded here
Lines+15added
Lines−0removed
From line
23
where the diff opens
First seen
11 Sep 2026
this site's first read of the page
Recorded edits7to this page, all time
The whole hunk
from line 23, old and new numbered
/
from line 23
2323To run plugin evals you need:
2424
2525* Claude Code v2.1.269 or later. Run `claude --version` to check and `claude update` to upgrade.
26* Git 2.31 or later, if git is installed. Run `git --version` to check. With an older git, `claude plugin eval` [stops before running any case](#git-is-too-old-for-claude-plugin-eval). Without git, it runs normally.
2627* A plugin directory with a `plugin.json` or `.claude-plugin/plugin.json` manifest, or a [skills-directory plugin](/docs/en/plugins/loading#plugins-shared-through-a-repository).
2728* The same authentication and model provider your normal Claude Code sessions use. Eval runs, judge-scored graders, and `claude plugin eval init` call the model with your credentials, so they count against your plan's usage limits or your API bill. When the command reports a cost, the figure is a [list-price estimate](/docs/en/costs) of those calls.
2829
from line 630
629630### "is not a trusted plugin directory, and this run cannot stop to ask you about it"
630631
631632This is the first run against a directory Claude Code doesn't trust yet, and it can't ask you because stdin or stdout isn't a terminal, you passed `--json`, or the `CI` environment variable is set to a true value such as `true`. Run `claude plugin eval <dir>` once in a terminal and answer the prompt, or pass `--trust-plugin` if you trust the plugin's code and suite. See [What a run can access](#security).
633
634<h3 id="git-is-too-old-for-claude-plugin-eval">
635 "is too old for claude plugin eval"
636</h3>
637
638The `git` on your `PATH` is older than 2.31, so `claude plugin eval` stopped before running any case and exited 1 with a message naming your version:
639
640```text theme={null}
641git 2.30 is too old for claude plugin eval: it ignores the environment configuration (GIT_CONFIG_COUNT, added in git 2.31) that switches off the repository's git hooks and helper programs for the run. Install git 2.31 or newer.
642```
643
644For each run, Claude Code switches off git hooks, credential helpers, and other programs a repository's git configuration can start. It does so through environment configuration that git reads only from version 2.31. An older git ignores that configuration, so the suite stops rather than scoring runs where those programs could execute. Install git 2.31 or later and run the suite again.
645
646Before v2.1.283, `claude plugin eval` didn't check the git version, and on an older git the suite ran with those programs left on.
632647
633648### "No eval cases found"
634649
No line in this hunk matches that.