{"version":"2.1.242","anchor":"eval-run-analysis-refuses-to-grade-a-run-that-bypassed-its-m","canonical_anchor":"eval-run-analysis-refuses-to-grade-a-run-that-bypassed-its-m","heading":"Eval run analysis refuses to grade a run that bypassed its mocks","tier":"internal","area":"Plugin Eval","scope":null,"heads_up":null,"url":"https:\/\/changelogs.core-directive.com\/v\/2.1.242\/e\/eval-run-analysis-refuses-to-grade-a-run-that-bypassed-its-m","release_url":"https:\/\/changelogs.core-directive.com\/v\/2.1.242","markdown":"### Eval run analysis refuses to grade a run that bypassed its mocks\n\nEval grading now refuses to score a run whose tool calls bypassed the mock servers.\n\n**What**\n\nThe code that reads an eval run's transcript was rewritten. It now verifies each stand-in server connected under the expected name and matched a nonce-marked call log, aborts on an expectation violation, and marks the run ungraded when the transcript shows more tool calls than the stand-ins recorded.\n\n**Details**\n\n- Per-tool-call output and error status are now recorded, and permission denials from the result message are honoured.\n- Cost is estimated from assistant token usage when no final result message arrives.\n- Results carry mock call records, a tally, an aborted record and a mock setup failure field.\n\n**Evidence**\n\n`refusing to run against what may be the real server`\n\n- Area: Plugin Eval\n- Tier: Under the hood\n- Useful: 2\/5\n- Signal: 3\/5"}