You'll noticeTier: how much it should matter to you
3Useful: my rating, 1 to 5
2Signal: worth watching, 1 to 5
Eval CLIArea: what it touches
ImprovementsKind: in v2.1.269,
ImprovementsSection of the release
What
The eval runner behind claude eval (and the related plugin eval tooling) now tracks each test case as a "plan" with results per run, and counts "harness failures" — runs that could not even start — separately from runs that did start and were scored. If any harness failures occur, it prints a new warning and the command exits with a non-zero exit code even if every run that did get scored passed.
Why
Previously a run that failed to start at all could get lost among scored results; now a broken harness run is called out explicitly and fails the eval, so a passing score summary can't mask setup or infrastructure problems.
