What
The skill that helps you write plugin evals now states a hard rule: a grader that implies a side effect only passes if the case allows the tool that produces it. A grader checking a file exists needs Write or Edit, a grader checking a tool was used needs that tool, and a task that runs a scan, build or test needs Bash. Authors are told to size timeout_seconds and max_turns to the actual task rather than copying the template.
Details
- The eval runner backs this up with a warning when a file-based grader is paired with a case that has no tool capable of creating the file.
- The calibration check tightened: instead of only requiring a non-empty plugin list, an entry must carry no
manifest_invalid,disabled_by_defaultorwill_not_loadproblem.identity_unverifiedandarchive_not_probedare explicitly non-blocking.
Evidence
cannot pass with the granted tools
Strings lifted out of the shipped bundle, so the claim above can be checked against them.
Names in the bundletimeout_secondsmax_turns
The entry above is what we published on the day. These lines were added later, as Anthropic's own pages caught up, and they sit beside the original rather than replacing it.
Confirmed since
Anthropic's documentation has since written up max_turns, on migration.
| `max_turns` | Count turns client-side. |managed-agents/migration see the edit
One source agreesOne thing we can check says the same as this entry.
Anthropic's documentation agrees
Anthropic's documentation has since written up max_turns, on migration.