claude plugin eval gains a --concurrency flag to run up to 8 test cases at once
The claude plugin eval command (which runs a plugin's test suite and produces a scored report) now supports a -j, --concurrency <n> option. It accepts a whole number from 1 to 8 (default 1) and controls how many agent runs execute at the same time. All of these runs share your account's rate limit, progress lines may interleave in the output, but results and the final report still keep the original case order.
The JSON report also now includes a concurrency field recording the value used for that run.
This lets larger eval suites finish faster by running multiple cases in parallel instead of one at a time, while keeping results easy to read in their original order.
* Added `claude plugin eval`: run a plugin's eval suite against Claude Code and get scored, reproducible results (JSON + HTML report); see `claude plugin eval --help`see the edit
You ran [`claude plugin eval`](/docs/en/plugin-evals) or `claude plugin eval init` and it exited 1 with one of these messages before doing anything:see the edit
| `experimental.evals` | string\|array | Directory below the plugin root that holds the plugin's [eval cases](/docs/en/plugin-evals#use-a-different-eval-directory), when it isn't the default `evals/`. `claude plugin eval --eval-dir` overri…see the edit
| `-j, --concurrency <n>` | Agent sessions to run at once, 1 to 8. They share your rate limit | `1` |see the edit
You don't have to write the suite manually. `claude plugin eval init` asks you about your plugin, proposes the cases and graders, tries them, and writes the files. You can also ask Claude to do the same from a session you already have open.see the edit
Test your marketplace before sharing. Validation checks file structure; to test whether a plugin changes what Claude does on realistic prompts, run its eval suite with [`claude plugin eval`](/docs/en/plugin-evals) before you publish a new …see the edit
Trying the plugin with `--plugin-dir` tells you it can work. To find out how often Claude actually reaches for it and gets the right result, run it against a set of test prompts with [`claude plugin eval`](/docs/en/plugin-evals). Each prom…see the edit
Two tools automate that comparison. For a skill that ships in a [plugin](/docs/en/plugins), [`claude plugin eval`](/docs/en/plugin-evals) runs each prompt in an isolated session with and without the plugin, scores it with graders you defin…see the edit
claude plugin eval on Claude Code changelog
Added claude plugin eval: run a plugin's eval suite against Claude Code and get scored, reproducible results (JSON + HTML report); see…
New in this build: --concurrency