Under the hood
Useful2 Signal3
Eval grading now refuses to score a run whose tool calls bypassed the mock servers.
What's wrong with this entry?
Anonymous. No account, no email.
What
The code that reads an eval run's transcript was rewritten. It now verifies each stand-in server connected under the expected name and matched a nonce-marked call log, aborts on an expectation violation, and marks the run ungraded when the transcript shows more tool calls than the stand-ins recorded.
Details
- Per-tool-call output and error status are now recorded, and permission denials from the result message are honoured.
- Cost is estimated from assistant token usage when no final result message arrives.
- Results carry mock call records, a tally, an aborted record and a mock setup failure field.
Evidence
refusing to run against what may be the real server
Strings lifted out of the shipped bundle, so the claim above can be checked against them.
Related
Other releases about the same thing. Found by shared names or similar wording; neither means one caused the other.
-
v2.1.234
plugin evalno longer swallows piped inputBoth mention eval
-
v2.1.234
Plugin eval reference documents image grading and new exit codes
Both mention eval
-
v2.1.234
claude plugin evalhandles termination signals and cleans up its outputBoth mention eval