You'll notice
No documentation found
claude eval now separately reports runs that crashed the harness versus runs that were graded and scored
What
The eval runner behind claude eval (and the related plugin eval tooling) now tracks each test case as a "plan" with results per run, and counts "harness failures" — runs that could not even start — separately from runs that did start and were scored. If any harness failures occur, it prints a new warning and the command exits with a non-zero exit code even if every run that did get scored passed.
Why
Previously a run that failed to start at all could get lost among scored results; now a broken harness run is called out explicitly and fails the eval, so a passing score summary can't mask setup or infrastructure problems.