{"version":"2.1.234","anchor":"plugin-eval-cases-must-grant-the-tools-their-graders-imply","canonical_anchor":"plugin-eval-cases-must-grant-the-tools-their-graders-imply","heading":"Plugin eval cases must grant the tools their graders imply","tier":"use","area":"Plugin Evals","scope":null,"heads_up":null,"url":"https:\/\/changelogs.core-directive.com\/v\/2.1.234\/e\/plugin-eval-cases-must-grant-the-tools-their-graders-imply","release_url":"https:\/\/changelogs.core-directive.com\/v\/2.1.234","markdown":"### Plugin eval cases must grant the tools their graders imply\n\nPlugin eval cases must now allow the tools their graders imply, and size timeouts to the real task.\n\n**What**\n\nThe skill that helps you write plugin evals now states a hard rule: a grader that implies a side effect only passes if the case allows the tool that produces it. A grader checking a file exists needs Write or Edit, a grader checking a tool was used needs that tool, and a task that runs a scan, build or test needs Bash. Authors are told to size `timeout_seconds` and `max_turns` to the actual task rather than copying the template.\n\n**Details**\n\n- The eval runner backs this up with a warning when a file-based grader is paired with a case that has no tool capable of creating the file.\n- The calibration check tightened: instead of only requiring a non-empty plugin list, an entry must carry no `manifest_invalid`, `disabled_by_default` or `will_not_load` problem. `identity_unverified` and `archive_not_probed` are explicitly non-blocking.\n\n**Evidence**\n\n`cannot pass with the granted tools`\n\n- Area: Plugin Evals\n- Names: `timeout_seconds`, `max_turns`\n- Tier: Use it now\n- Useful: 3\/5\n- Signal: 2\/5"}