{"version":"2.1.269","anchor":"eval-cli-now-distinguishes-harness-failures-from-scoring-fai","canonical_anchor":"eval-cli-now-distinguishes-harness-failures-from-scoring-fai","heading":"Eval CLI now distinguishes harness failures from scoring failures and reports them","tier":"notice","area":"Eval CLI","scope":null,"heads_up":null,"url":"https:\/\/changelogs.core-directive.com\/v\/2.1.269\/e\/eval-cli-now-distinguishes-harness-failures-from-scoring-fai","release_url":"https:\/\/changelogs.core-directive.com\/v\/2.1.269","markdown":"### Eval CLI now distinguishes harness failures from scoring failures and reports them\n\nclaude eval now separately reports runs that crashed the harness versus runs that were graded and scored\n\n**What**\n\nThe eval runner behind `claude eval` (and the related plugin eval tooling) now tracks each test case as a \"plan\" with results per run, and counts \"harness failures\" \u2014 runs that could not even start \u2014 separately from runs that did start and were scored. If any harness failures occur, it prints a new warning and the command exits with a non-zero exit code even if every run that did get scored passed.\n\n**Why**\n\nPreviously a run that failed to start at all could get lost among scored results; now a broken harness run is called out explicitly and fails the eval, so a passing score summary can't mask setup or infrastructure problems.\n\n- Area: Eval CLI\n- Tier: You'll notice\n- Useful: 3\/5\n- Signal: 2\/5"}