You'll noticeTier: how much it should matter to you
2Useful: my rating, 1 to 5
1Signal: worth watching, 1 to 5
Plugin EvalArea: what it touches
ImprovementsKind: in v2.1.235,
ImprovementsSection of the release
Before running, plugin eval now reads the plugin manifest's declared component paths — commands, agents, skills, outputStyles, themes, workflows, monitors, mcpServers, lspServers, and hooks — follows symlinks into the real tree, and checks whether the eval directory overlaps any of them, since files written there could otherwise be loaded as plugin components.
When the directory came from --eval-dir, an overlap is a hard error unless the run doesn't write into the plugin. When it came from the manifest or the default location, it's downgraded to a warning that suggests --output-dir or running from the plugin root instead.
Since it was published
The entry above is what we published on the day. These lines were added later, as Anthropic's own pages caught up, and they sit beside the original rather than replacing it.
Confirmed sinceAnthropic's documentation has since written up --eval-dir, on Plugins reference.| `experimental.evals` | string\|array | Directory below the plugin root that holds the plugin's [eval cases](/docs/en/plugin-evals#use-a-different-eval-directory), when it isn't the default `evals/`. `claude plugin eval --eval-dir` overri…plugins-referencesee the edit
How sure we are
One source agreesOne thing we can check says the same as this entry.
Anthropic's documentation agreesAnthropic's documentation has since written up --eval-dir, on Plugins reference.
