You'll noticeTier: how much it should matter to you
2Useful: my rating, 1 to 5
2Signal: worth watching, 1 to 5
ElsewhereArea: what it touches
ImprovementsKind: in v2.1.299,
ImprovementsSection of the release
What
The eval harness is the tool that runs tests scoring how well Claude does on a set of tasks. It now counts runs that ended on an API error, which is a failed request to the model service. Those runs scored 0 without being graded. The harness prints a warning and exits with a failure code, even when the scores would otherwise pass.
Why
Automated checks that use the eval suite can now fail where they passed before, so an API error no longer looks like a clean pass.
