{"version":"2.1.251","anchor":"plugin-eval-cross-checks-mocked-tool-calls-against-what-actu","canonical_anchor":"plugin-eval-cross-checks-mocked-tool-calls-against-what-actu","heading":"Plugin eval cross-checks mocked tool calls against what actually ran","tier":"internal","area":"Plugin Eval","scope":null,"heads_up":null,"url":"https:\/\/changelogs.core-directive.com\/v\/2.1.251\/e\/plugin-eval-cross-checks-mocked-tool-calls-against-what-actu","release_url":"https:\/\/changelogs.core-directive.com\/v\/2.1.251","markdown":"### Plugin eval cross-checks mocked tool calls against what actually ran\n\nPlugin eval refuses to grade a run when mocked tool calls don't match what actually ran.\n\n**What**\n\nThe tooling behind `claude eval`, used to test plugins against mocked MCP tools, now compares every mocked tool call with the harness's own record of what was dispatched, and refuses to grade a run when the two do not line up.\n\n**Details**\n\n- Flags results altered after the mock returned them, for example by a PostToolUse hook, and calls the child agent denied itself.\n- Flags mock answers lost because the child was killed while they were in flight.\n- In the ambiguous cases it reports that the run is not graded rather than producing a score. Reported under `cli_plugin_eval_mocks` telemetry.\n\n**Evidence**\n\n`mocks: an agent mock call was in flight when the child was killed`\n\n- Area: Plugin Eval\n- Tier: Under the hood\n- Useful: 2\/5\n- Signal: 3\/5"}