Under the hoodTier: how much it should matter to you
1Useful: my rating, 1 to 5
2Signal: worth watching, 1 to 5
Plugin EvalArea: what it touches
In DevelopmentKind: in v2.1.242,
In DevelopmentSection of the release
What
An eval result now includes a mocks block listing each stand-in server's kind and per-tool responder, defaulting to "fixed", plus a tally that counts calls the mocks did not cover as unmocked.
Details
A run that aborted, or whose mocks could not be prepared, returns a zero score carrying the server, tool and reason, plus any auth_rejected flag.
Mocked tools are announced per case and excluded from the check that a case exercises a real tool.
Mock call logs are read back into the result as call records and a tally.
A separate standalone eval config schema, covering schema version, graders and config overrides, was dropped from this part of the bundle.
Evidence
mocks: could not be prepared for this case, mockSetupFailure, "Mock stand-ins for MCP servers, from <eval dir>/mocks/ (record | off; default: record — off spawns the real servers, gated by --allow-tools as usual)"
Strings lifted out of the shipped bundle, so the claim above can be checked against them.
