Under the hoodTier: how much it should matter to you
2Useful: my rating, 1 to 5
3Signal: worth watching, 1 to 5
Plugin EvalArea: what it touches
Internal ChangesKind: in v2.1.251,
Internal ChangesSection of the release
What
The tooling behind claude eval, used to test plugins against mocked MCP tools, now compares every mocked tool call with the harness's own record of what was dispatched, and refuses to grade a run when the two do not line up.
Details
Flags results altered after the mock returned them, for example by a PostToolUse hook, and calls the child agent denied itself.
Flags mock answers lost because the child was killed while they were in flight.
In the ambiguous cases it reports that the run is not graded rather than producing a score. Reported under cli_plugin_eval_mocks telemetry.
Evidence
mocks: an agent mock call was in flight when the child was killed
Strings lifted out of the shipped bundle, so the claim above can be checked against them.
