An auth preflight for claude plugin eval, behind the early-access gate.
claude plugin eval now runs an authentication preflight up front. If it fails, the whole suite aborts with exit code 2 and a report status of auth_failed rather than burning every case on a broken credential. A preflight problem that is not fatal is printed as a notice and the run continues.
- The result object carries an
authPreflightFailedflag alongside the report status. - Plugin eval as a whole is early access, gated on
tengu_walnut_spireorCLAUDE_CODE_WALNUT_SPIRE.
authentication check failed before running any case
Strings lifted out of the shipped bundle, so the claim above can be checked against them.
tengu_walnut_spire Off in both readingsThe flag server returned off for the account this site reads and for the anonymous baseline. A reading of off cannot rule out a rollout these two readings sit outside of.
This account: off · anonymous baseline: off · compiled default in v2.1.233: off
These values were read against a different version of Claude Code, so treat them as the nearest reading available instead of one taken on this release.
Read once, for one account on one subscription tier, against v2.1.233. It isn't a statement about your account. What a flag value here can and cannot tell you
The entry above is what we published on the day. These lines were added later, as Anthropic's own pages caught up, and they sit beside the original rather than replacing it.
Test your marketplace before sharing. Validation checks file structure; to test whether a plugin changes what Claude does on realistic prompts, run its eval suite with [`claude plugin eval`](/docs/en/plugin-evals) before you publish a new …plugin-marketplaces see the edit
* Added `claude plugin eval`: run a plugin's eval suite against Claude Code and get scored, reproducible results (JSON + HTML report); see `claude plugin eval --help`changelog see the edit
A reading is one sample. Claude Code evaluates its flags remotely, so no client sees the targeting rule behind a value and this says nothing about your account.
A reading is one sample. Claude Code evaluates its flags remotely, so no client sees the targeting rule behind a value and this says nothing about your account.
Anthropic's documentation has since written up claude plugin eval, on Claude Code changelog.