You'll noticeTier: how much it should matter to you
3Useful: my rating, 1 to 5
2Signal: worth watching, 1 to 5
Plugin EvalArea: what it touches
ImprovementsKind: in v2.1.235,
ImprovementsSection of the release
The eval command resolves the target, the plugin root, and any untrusted or adopted manifest directory up front, and prints explicit notices when a nearby plugin is not loaded: because it sits above the working directory, falls outside the scope consulted, or its ownership can't be verified. That way a case running without a plugin loaded runs against baseline Claude instead of silently picking up someone's plugin. --eval-dir is now validated and vetted with its own refusal codes, eval_dir_refused and eval_dir_unvettable, the eval directory can be supplied by the plugin manifest, and the empty-result help output now names the directory and manifest it consulted.
The entry above is what we published on the day. These lines were added later, as Anthropic's own pages caught up, and they sit beside the original rather than replacing it.
Confirmed sinceAnthropic's documentation has since written up --eval-dir, on Plugins reference.| `experimental.evals` | string\|array | Directory below the plugin root that holds the plugin's [eval cases](/docs/en/plugin-evals#use-a-different-eval-directory), when it isn't the default `evals/`. `claude plugin eval --eval-dir` overri…plugins-referencesee the edit
Added sinceA small documentation edit on Create and distribute a plugin marketplace touched a line naming plugin eval after this was published.Test your marketplace before sharing. Validation checks file structure; to test whether a plugin changes what Claude does on realistic prompts, run its eval suite with [`claude plugin eval`](/docs/en/plugin-evals) before you publish a new …plugin-marketplacessee the edit
Confirmed sinceAnthropic's documentation has since written up plugin eval, on Claude Code changelog.* Added `claude plugin eval`: run a plugin's eval suite against Claude Code and get scored, reproducible results (JSON + HTML report); see `claude plugin eval --help`changelogsee the edit
How sure we are
Two sources agreeTwo things we can check say the same as this entry.
Anthropic's documentation agreesAnthropic's documentation has since written up plugin eval, on Claude Code changelog.
The name it cites is new in this buildNew in this build: --eval-dir