plugin eval and plugin eval init now accept an --eval-dir flag, which flows into the generated interview prompt and the child-process invocation, is verified with an error message if it can't be, and is recorded as out_dir in the run's metadata. The scored HTML report generated by eval runs now also renders per-case advisories as warning notes.
Names in the bundle--eval-dir
Documented inclaude-code/plugin-evals
The entry above is what we published on the day. These lines were added later, as Anthropic's own pages caught up, and they sit beside the original rather than replacing it.
Confirmed since
Anthropic's documentation has since written up --eval-dir, on Plugins reference.
| `experimental.evals` | string\|array | Directory below the plugin root that holds the plugin's [eval cases](/docs/en/plugin-evals#use-a-different-eval-directory), when it isn't the default `evals/`. `claude plugin eval --eval-dir` overri…plugins-reference see the edit
Added since
A small documentation edit on Create and distribute a plugin marketplace touched a line naming plugin eval after this was published.
Test your marketplace before sharing. Validation checks file structure; to test whether a plugin changes what Claude does on realistic prompts, run its eval suite with [`claude plugin eval`](/docs/en/plugin-evals) before you publish a new …plugin-marketplaces see the edit
Confirmed since
Anthropic's documentation has since written up plugin eval init, on Error reference.
You ran [`claude plugin eval`](/docs/en/plugin-evals) or `claude plugin eval init` and it exited 1 with one of these messages before doing anything:errors see the edit
Confirmed since
Anthropic's documentation has since written up plugin eval, on Claude Code changelog.
* Added `claude plugin eval`: run a plugin's eval suite against Claude Code and get scored, reproducible results (JSON + HTML report); see `claude plugin eval --help`changelog see the edit
Two sources agreeTwo things we can check say the same as this entry.
Anthropic's documentation agrees
Anthropic's documentation has since written up plugin eval, on Claude Code changelog.
The name it cites is new in this build
New in this build: --eval-dir