What probably matters to youSection of the release
What
The plugin eval runner tests a plugin or skill by running Claude on a set of test cases and grading the results. You can now remove built-in tools from Claude during those runs in two ways:
--disallow-tools on the command line removes the listed tools in every case.
execution.disallowed_tools inside a single case removes them for that case only.
Both accept only bare built-in tool names, such as Agent or TaskStop. A pattern like Bash(rm:*) is rejected with a message explaining why, and so is an MCP tool (a tool supplied by an add-on server). Tools that are accepted are listed at the start of the run as "Removed from Claude's tools in every case (--disallow-tools)".
If a grader needs a tool you removed, the runner reports that the grader cannot pass. A run that ends on an API error now scores 0, is marked with ended_on_api_error: true, and is left out of the comparison between runs with and without the plugin.
Why
You can now check how a plugin or skill behaves when Claude cannot use certain tools. Failed API calls no longer distort the with-and-without comparison.
Two sources agreeTwo things we can check say the same as this entry.
Anthropic's release notes agreeAdded --disallow-tools and a disallowed_tools case field to claude plugin eval to keep specific built-in tools, such as Agent, out of eval…
The name it cites is new in this buildNew in this build: --disallow-tools
