Under the hoodTier: how much it should matter to you
2Useful: my rating, 1 to 5
3Signal: worth watching, 1 to 5
Plugin EvalArea: what it touches
In DevelopmentKind: in v2.1.242,
In DevelopmentSection of the release
What
The code that reads an eval run's transcript was rewritten. It now verifies each stand-in server connected under the expected name and matched a nonce-marked call log, aborts on an expectation violation, and marks the run ungraded when the transcript shows more tool calls than the stand-ins recorded.
Details
Per-tool-call output and error status are now recorded, and permission denials from the result message are honoured.
Cost is estimated from assistant token usage when no final result message arrives.
Results carry mock call records, a tally, an aborted record and a mock setup failure field.
Evidence
refusing to run against what may be the real server
Strings lifted out of the shipped bundle, so the claim above can be checked against them.
