claude plugin eval runs a plugin against a set of test cases and scores the results. Before a run it now checks git --version and stops with a message that git is too old if the version is below 2.31. Older git ignores GIT_CONFIG_COUNT, a way of passing git settings through the environment, and eval relies on it to switch off the repository's hooks and helper programs.
Eval also sets git so that it prints output without a pager, never opens an editor, and uses no credential helper (core.pager=cat, core.editor=true and an empty credential.helper).
This keeps a repository's own scripts from running during an eval, but eval can now fail on machines with an older git that it used to run on.
The `claude plugin eval` shell command runs your [plugin](/docs/en/plugins/overview) against a suite of test cases and scores the results. Each case is a realistic prompt plus one or more graders. A grader is a pass/fail check on what Clau…see the edit
Two tools automate that comparison. For a skill that ships in a [plugin](/docs/en/plugins/overview), [`claude plugin eval`](/docs/en/plugin-evals) runs each prompt in an isolated session with and without the plugin, scores it with graders …see the edit
If the subagent ships in a [plugin](/docs/en/plugins/overview), you can measure how reliably Claude delegates to it across realistic prompts instead of checking one at a time: [`claude plugin eval`](/docs/en/plugin-evals) runs each prompt …see the edit
If the skill is part of a plugin, you can also test the whole plugin from the Claude Code command line with [`claude plugin eval`](https://code.claude.com/docs/en/plugin-evals), which grades eval cases you write and compares against a run …see the edit
The entry above is what we published on the day. These lines were added later, as Anthropic's own pages caught up, and they sit beside the original rather than replacing it.
| `GIT_CONFIG_COUNT`, `GIT_CONFIG_KEY_n`, `GIT_CONFIG_VALUE_n` | Git settings the runner fixes for the git your hook runs. [Git configuration inside lifecycle hooks](#git-configuration-inside-lifecycle-hooks) describes them. Requires Claud…self-hosted-environments-configuration see the edit
The first time you run `claude plugin eval` against a directory, Claude Code asks `Trust this plugin directory?` before it loads anything from it, unless you already accepted the trust prompt there in an interactive `claude` session. Insid…plugin-evals see the edit
| `experimental.evals` | Path, or array of paths | Directory that holds the plugin's [eval cases](/docs/en/plugin-evals#use-a-different-eval-directory) when it isn't the default `evals/`. `claude plugin eval --eval-dir` overrides it |plugins/manifest-reference see the edit
claude plugin eval on Test plugins with evals
Fixed installed_plugins.json being rewritten, losing records, when it holds a record this version cannot read; claude plugin commands now…