Source Intelligence
Follow Discord

DisclaimerUnofficial, and not affiliated with Anthropic. Nearly all of this is read straight out of what ships: npm bundles, captured prompts, published docs. Anthropic's own notes go in verbatim, marked as theirs. The rest is my reading, and every entry carries the strings behind it. If one looks wrong, vote it down and say why.

← All of v2.1.251 Home All releases ←olderv2.1.250
Claude Code v2.1.251

Plugin eval cross-checks mocked tool calls against what actually ran

Under the hood
Useful2 Signal3
Plugin Eval

Plugin eval refuses to grade a run when mocked tool calls don't match what actually ran.

What

The tooling behind claude eval, used to test plugins against mocked MCP tools, now compares every mocked tool call with the harness's own record of what was dispatched, and refuses to grade a run when the two do not line up.

Details
  • Flags results altered after the mock returned them, for example by a PostToolUse hook, and calls the child agent denied itself.
  • Flags mock answers lost because the child was killed while they were in flight.
  • In the ambiguous cases it reports that the run is not graded rather than producing a score. Reported under cli_plugin_eval_mocks telemetry.
Evidence

mocks: an agent mock call was in flight when the child was killed

Strings lifted out of the shipped bundle, so the claim above can be checked against them.

Related

Other releases about the same thing. Found by shared names or similar wording; neither means one caused the other.

See this entry in the whole of v2.1.251 →