claude plugin eval now checks which plugin is actually loaded before running
The eval command resolves the target, the plugin root, and any untrusted or adopted manifest directory up front, and prints explicit notices when a nearby plugin is not loaded: because it sits above the working directory, falls outside the scope consulted, or its ownership can't be verified. That way a case running without a plugin loaded runs against baseline Claude instead of silently picking up someone's plugin. --eval-dir is now validated and vetted with its own refusal codes, eval_dir_refused and eval_dir_unvettable, the eval directory can be supplied by the plugin manifest, and the empty-result help output now names the directory and manifest it consulted.
The entry above is what we published on the day. These lines were added later, as Anthropic's own pages caught up, and they sit beside the original rather than replacing it.
| `experimental.evals` | string\|array | Directory below the plugin root that holds the plugin's [eval cases](/docs/en/plugin-evals#use-a-different-eval-directory), when it isn't the default `evals/`. `claude plugin eval --eval-dir` overri…plugins-reference see the edit
Test your marketplace before sharing. Validation checks file structure; to test whether a plugin changes what Claude does on realistic prompts, run its eval suite with [`claude plugin eval`](/docs/en/plugin-evals) before you publish a new …plugin-marketplaces see the edit
* Added `claude plugin eval`: run a plugin's eval suite against Claude Code and get scored, reproducible results (JSON + HTML report); see `claude plugin eval --help`changelog see the edit
Anthropic's documentation has since written up plugin eval, on Claude Code changelog.
New in this build: --eval-dir