'claude plugin eval' adds --allow-real-servers and concurrency options, plus separate harness-failure tracking
The claude plugin eval command now accepts an allowRealServers option and a concurrency option, both passed through to the eval runner. It also now tracks harness failures (errors in the eval tooling itself) separately from case failures, and fails the run distinctly, recording a harness_failure event, when the harness itself errors rather than when individual test cases fail.
Separating harness failures from case failures makes it clearer whether an eval run failed because of the code being tested or because of a problem in the eval tooling itself. The new options give more control over whether real (non-mocked) servers can be used and how many cases run concurrently.
Added claude plugin eval: run a plugin's eval suite against Claude Code and get scored, reproducible results (JSON + HTML report); see…
New in this build: --allow-real-servers