Use it now
Useful3 Signal2
Plugin eval cases must now allow the tools their graders imply, and size timeouts to the real task.
What's wrong with this entry?
Anonymous. No account, no email.
timeout_secondsmax_turns
What
The skill that helps you write plugin evals now states a hard rule: a grader that implies a side effect only passes if the case allows the tool that produces it. A grader checking a file exists needs Write or Edit, a grader checking a tool was used needs that tool, and a task that runs a scan, build or test needs Bash. Authors are told to size timeout_seconds and max_turns to the actual task rather than copying the template.
Details
- The eval runner backs this up with a warning when a file-based grader is paired with a case that has no tool capable of creating the file.
- The calibration check tightened: instead of only requiring a non-empty plugin list, an entry must carry no
manifest_invalid,disabled_by_defaultorwill_not_loadproblem.identity_unverifiedandarchive_not_probedare explicitly non-blocking.
Evidence
cannot pass with the granted tools
Strings lifted out of the shipped bundle, so the claim above can be checked against them.