What
The bundled reference for claude plugin eval describes a redesigned sandbox: the working directory sits inside the sandbox home, an empty git repository at the sandbox home root stops git from walking up into your real repo, and the child process is started with --setting-sources user and the prompt passed on stdin instead of the command line. Enterprise managed settings are called out as the one policy that still applies inside a run.
Details
- Adds exit code 143 for a run killed by SIGTERM, and an
auth_failedreason for a partial run. - Per-plugin failure codes appear under the suite's plugin list, and individual cases can carry advisories.
- Image and binary grading moved from "upcoming" to shipped.
- This is documentation text; the eval harness itself remains early-access gated.
Evidence
--setting-sources user
Strings lifted out of the shipped bundle, so the claim above can be checked against them.
Names in the bundleclaude plugin eval
The entry above is what we published on the day. These lines were added later, as Anthropic's own pages caught up, and they sit beside the original rather than replacing it.
Added since
A small documentation edit on Create and distribute a plugin marketplace touched a line naming claude plugin eval after this was published.
Test your marketplace before sharing. Validation checks file structure; to test whether a plugin changes what Claude does on realistic prompts, run its eval suite with [`claude plugin eval`](/docs/en/plugin-evals) before you publish a new …plugin-marketplaces see the edit
Confirmed since
Anthropic's documentation has since written up claude plugin eval, on Claude Code changelog.
* Added `claude plugin eval`: run a plugin's eval suite against Claude Code and get scored, reproducible results (JSON + HTML report); see `claude plugin eval --help`changelog see the edit
One source agreesOne thing we can check says the same as this entry.
Anthropic's documentation agrees
Anthropic's documentation has since written up claude plugin eval, on Claude Code changelog.