Eval results now list which stand-in servers answered and count calls nothing covered.
What's wrong with this entry?
An eval result now includes a mocks block listing each stand-in server's kind and per-tool responder, defaulting to "fixed", plus a tally that counts calls the mocks did not cover as unmocked.
- A run that aborted, or whose mocks could not be prepared, returns a zero score carrying the server, tool and reason, plus any
auth_rejectedflag. - Mocked tools are announced per case and excluded from the check that a case exercises a real tool.
- Mock call logs are read back into the result as call records and a tally.
- A separate standalone eval config schema, covering schema version, graders and config overrides, was dropped from this part of the bundle.
mocks: could not be prepared for this case, mockSetupFailure, "Mock stand-ins for MCP servers, from <eval dir>/mocks/ (record | off; default: record — off spawns the real servers, gated by --allow-tools as usual)"
Strings lifted out of the shipped bundle, so the claim above can be checked against them.
Related
Other releases about the same thing. Found by shared names or similar wording; neither means one caused the other.
-
v2.1.234
plugin evalno longer swallows piped inputBoth mention eval
-
v2.1.234
Plugin eval reference documents image grading and new exit codes
Both mention eval
-
v2.1.234
claude plugin evalhandles termination signals and cleans up its outputBoth mention eval