Source Intelligence
Follow Discord

DisclaimerUnofficial, and not affiliated with Anthropic. Nearly all of this is read straight out of what ships: npm bundles, captured prompts, published docs. Anthropic's own notes go in verbatim, marked as theirs. The rest is my reading, and every entry carries the strings behind it. If one looks wrong, vote it down and say why.

← All of v2.1.233 Home All releases ←olderv2.1.232 v2.1.234newer→
Claude Code v2.1.233

Eval suites stop at the first authentication failure

You'll notice
Useful2 Signal1
Plugins

An eval suite stops immediately when the first run is rejected for bad credentials.

What

If the very first run in an eval suite is rejected for bad credentials, the suite halts immediately rather than working through every remaining case and failing each one the same way.

Details
  • The stop is reported as auth_failed, with a new auth_preflight_failed alongside it; previously partial runs were always attributed to cost_ceiling.
  • The printed message points you at the credential error above and asks you to re-run.
Evidence

every remaining run would fail the same way, so the suite stops here. Fix the credential above and re-run.

Strings lifted out of the shipped bundle, so the claim above can be checked against them.

See this entry in the whole of v2.1.233 →