{"version":"2.1.299","anchor":"eval-harness-gains-disallow-tools-and-per-case-disallowed","canonical_anchor":"eval-harness-gains-disallow-tools-and-per-case-disallowed","heading":"Plugin evals can now take tools away from Claude with --disallow-tools","tier":"use","area":"Plugin Evals","scope":"individual","heads_up":false,"url":"https:\/\/changelogs.core-directive.com\/v\/2.1.299\/e\/eval-harness-gains-disallow-tools-and-per-case-disallowed","release_url":"https:\/\/changelogs.core-directive.com\/v\/2.1.299","markdown":"### Plugin evals can now take tools away from Claude with --disallow-tools\n\nThe plugin eval runner accepts `--disallow-tools` and a per-case `disallowed_tools` list, and scores runs that end on an API error as 0\n\n**What**\n\nThe plugin eval runner tests a plugin or skill by running Claude on a set of test cases and grading the results. You can now remove built-in tools from Claude during those runs in two ways:\n\n- `--disallow-tools` on the command line removes the listed tools in every case.\n\n- `execution.disallowed_tools` inside a single case removes them for that case only.\n\nBoth accept only bare built-in tool names, such as `Agent` or `TaskStop`. A pattern like `Bash(rm:*)` is rejected with a message explaining why, and so is an MCP tool (a tool supplied by an add-on server). Tools that are accepted are listed at the start of the run as \"Removed from Claude's tools in every case (--disallow-tools)\".\n\nIf a grader needs a tool you removed, the runner reports that the grader cannot pass. A run that ends on an API error now scores 0, is marked with `ended_on_api_error: true`, and is left out of the comparison between runs with and without the plugin.\n\n**Why**\n\nYou can now check how a plugin or skill behaves when Claude cannot use certain tools. Failed API calls no longer distort the with-and-without comparison.\n\n- Area: Plugin Evals\n- Names: `--disallow-tools`\n- Tier: Use it now\n- Useful: 4\/5\n- Signal: 2\/5\n- Scope: individual\n- Heads-up: no"}