The claude plugin eval init path now accepts an eval directory either from the --eval-dir flag or from the plugin manifest's experimental.evals key. It refuses to consult a manifest sitting in a directory it cannot vouch for, refuses to write cases or results into a symlinked or foreign-owned location, and cleans up any directory it created if the run fails partway through.
The bundled plugin-eval reference docs were updated to match: earlier notes calling --eval-dir an upcoming feature are gone, and the flag is now listed among the documented options.
Names in the bundle--eval-dir
Documented inclaude-code/plugin-evals
The entry above is what we published on the day. These lines were added later, as Anthropic's own pages caught up, and they sit beside the original rather than replacing it.
Confirmed since
Anthropic's documentation has since written up --eval-dir, on Plugins reference.
| `experimental.evals` | string\|array | Directory below the plugin root that holds the plugin's [eval cases](/docs/en/plugin-evals#use-a-different-eval-directory), when it isn't the default `evals/`. `claude plugin eval --eval-dir` overri…plugins-reference see the edit
Two sources agreeTwo things we can check say the same as this entry.
Anthropic's documentation agrees
Anthropic's documentation has since written up --eval-dir, on Plugins reference.
The name it cites is new in this build
New in this build: --eval-dir