{"version":"2.1.299","anchor":"eval-suite-fails-runs-that-ended-on-an-api-error","canonical_anchor":"eval-suite-fails-runs-that-ended-on-an-api-error","heading":"Eval runs that end on an API error now fail the suite","tier":"notice","area":"Elsewhere","scope":"individual","heads_up":true,"url":"https:\/\/changelogs.core-directive.com\/v\/2.1.299\/e\/eval-suite-fails-runs-that-ended-on-an-api-error","release_url":"https:\/\/changelogs.core-directive.com\/v\/2.1.299","markdown":"### Eval runs that end on an API error now fail the suite\n\nThe eval harness now fails when any run ended on an API error, even if all the scores pass\n\n**What**\n\nThe eval harness is the tool that runs tests scoring how well Claude does on a set of tasks. It now counts runs that ended on an API error, which is a failed request to the model service. Those runs scored 0 without being graded. The harness prints a warning and exits with a failure code, even when the scores would otherwise pass.\n\n**Why**\n\nAutomated checks that use the eval suite can now fail where they passed before, so an API error no longer looks like a clean pass.\n\n- Area: Elsewhere\n- Tier: You'll notice\n- Useful: 2\/5\n- Signal: 2\/5\n- Scope: individual\n- Heads-up: yes"}