What probably matters to youSection of the release
What
The claude plugin eval command now accepts an allowRealServers option and a concurrency option, both passed through to the eval runner. It also now tracks harness failures (errors in the eval tooling itself) separately from case failures, and fails the run distinctly, recording a harness_failure event, when the harness itself errors rather than when individual test cases fail.
Why
Separating harness failures from case failures makes it clearer whether an eval run failed because of the code being tested or because of a problem in the eval tooling itself. The new options give more control over whether real (non-mocked) servers can be used and how many cases run concurrently.
The entry above is what we published on the day. These lines were added later, as Anthropic's own pages caught up, and they sit beside the original rather than replacing it.
Confirmed sinceAnthropic's documentation has since written up --concurrency, on Test plugins with evals.| `-j`, `--concurrency <n>` | `1` | Run up to this many agent runs at once, from 1 to 8. They share your account's rate limit, so this shortens wall-clock time rather than raising throughput past that limit. Results keep case order |plugin-evalssee the edit
Confirmed sinceAnthropic's documentation has since written up --allow-real-servers, on Test plugins with evals.When a case asked for a tool you didn't grant, the progress output lists it as `not granted`. Tools on a [mocked](#mock-mcp-servers) MCP server need no grant. Tools on a real plugin MCP server need both the server started, with `--allow-re…plugin-evalssee the edit
How sure we are
Two sources agreeTwo things we can check say the same as this entry.
Anthropic's documentation agreesAnthropic's documentation has since written up --allow-real-servers, on Test plugins with evals.
Anthropic's release notes agreeAdded claude plugin eval: run a plugin's eval suite against Claude Code and get scored, reproducible results (JSON + HTML report); see…
The name it cites is new in this buildNew in this build: --allow-real-servers
