You'll noticeTier: how much it should matter to you
2Useful: my rating, 1 to 5
1Signal: worth watching, 1 to 5
AuthArea: what it touches
Bug FixesKind: in v2.1.305,
ImprovementsSection of the release
Unclear Not settled which runs use this credential lookup, so it is unclear which sessions now pick the API key over a login.
What
claude plugin eval runs tests on a plugin. It now handles an API key (a secret key used instead of signing in) for those test runs.
Picking a credential: when a key is available and OAuth sign-in is not forced, the key is used. It is also used as a fallback when the OAuth token is missing or cannot be refreshed. Before, the eval path returned nothing when an API key file descriptor or other key source was present.
Windows: there it stops with "claude plugin eval can't pass this session's API key to its test runs on Windows; set ANTHROPIC_API_KEY and run it again".
If you run evals with an API key, you no longer have to export the key yourself, except on Windows where you still need to set ANTHROPIC_API_KEY. Which credential your eval runs use may also differ from before.