plugin eval gets a configurable case directory via --eval-dir or a manifest key, still gated as early access
The eval harness no longer hard-codes evals/ as the case directory. A new --eval-dir <dir> flag, or a new plugin.json key experimental.evals (a single path, or a list whose first entry wins), now selects it, and results are written to <plugin>/<dir>/results/.
Both plugin eval and plugin eval init remain marked as early access, gated by tengu_walnut_spire (falling back to false) or the env var CLAUDE_CODE_WALNUT_SPIRE. Users without access get a "currently in early access" message rather than an unrecognized-command error. The manifest's experimental.evals value is validated strictly: it must be relative, contain no .., stay under 200 characters and 8 path segments, use plain segment names, not be Windows-hostile, and not point inside node_modules, .claude, or results; a misplaced top-level "evals" key is flagged rather than silently accepted.
tengu_walnut_spire Off in both readingsThe flag server returned off for the account this site reads and for the anonymous baseline. A reading of off cannot rule out a rollout these two readings sit outside of.
This account: off · anonymous baseline: off · compiled default in v2.1.235: off
Read once, for one account on one subscription tier, against v2.1.235. It isn't a statement about your account. What a flag value here can and cannot tell you
The entry above is what we published on the day. These lines were added later, as Anthropic's own pages caught up, and they sit beside the original rather than replacing it.
| `experimental.evals` | string\|array | Directory below the plugin root that holds the plugin's [eval cases](/docs/en/plugin-evals#use-a-different-eval-directory), when it isn't the default `evals/`. `claude plugin eval --eval-dir` overri…plugins-reference see the edit
Test your marketplace before sharing. Validation checks file structure; to test whether a plugin changes what Claude does on realistic prompts, run its eval suite with [`claude plugin eval`](/docs/en/plugin-evals) before you publish a new …plugin-marketplaces see the edit
You ran [`claude plugin eval`](/docs/en/plugin-evals) or `claude plugin eval init` and it exited 1 with one of these messages before doing anything:errors see the edit
* Added `claude plugin eval`: run a plugin's eval suite against Claude Code and get scored, reproducible results (JSON + HTML report); see `claude plugin eval --help`changelog see the edit
A reading is one sample. Claude Code evaluates its flags remotely, so no client sees the targeting rule behind a value and this says nothing about your account.
A reading is one sample. Claude Code evaluates its flags remotely, so no client sees the targeting rule behind a value and this says nothing about your account.
Anthropic's documentation has since written up plugin eval, on Claude Code changelog.
New in this build: --eval-dir