The plugin eval CLI subcommand (for evaluating plugins against test cases) was previously hard-gated behind tengu_walnut_spire. The gate is still present but is now also unlockable via the CLAUDE_CODE_WALNUT_SPIRE environment variable.
- When
mvc()returnsfalse(bothtengu_walnut_spireflag andCLAUDE_CODE_WALNUT_SPIREenv var absent), invokingclaude plugin evalorclaude plugin eval initimmediately aborts with the message "plugin evalis currently in early access" — no command logic runs. claude plugin eval [target]discovers eval cases fromevals//case.yamlorevals//prompt.md+graders/*.mdunder the target directory. The target may be a file-system path, an installed plugin name, or aplugin@marketplaceid; skills-dir plugins also resolve and automatically add a no-plugin baseline arm for comparison.- The command exposes a full suite of options:
--case <glob>(filter cases by name),--tag <tag...>(filter by tag, repeatable),--runs <n>(default fromcase.runs ?? 3),--model(override model),--judge-model(LLM grader model, default Haiku),--max-cost-usd(hard budget ceiling — breaching it exits with code 2 and reports partial results; the overrunning agent skips paid graders but free graders still score),--output-dir(default./evals/results/<timestamp>/),--json(emitaggregate-result.jsonto stdout for CI),--threshold <0..1>(exit 1 if any case score falls below this value, default 1.0),--allow-tools <tools...>(operator grant for gated tools:Bash,Write,Edit,WebFetch,mcp__; supportsTool(pattern:)syntax),--scaffold/--no-scaffold(runscaffold_scriptfrom case files, off by default),--ablation(none|with-without, automaticallywith-withoutwhen targeting a plugin by name),--keep-temp(preserve scaffold dirs for debugging),--verbose(stream trace as it runs). claude plugin eval init [name]authors an eval suite underevals/via an interactive interview (--interactive/--interview) or writes a blank single-case template (--bare).- Model-trigger evaluation (the "would the model route to this skill?" check) is still incomplete: when the model callback returns
null, the case receives verdictskippedwith reason "Model evaluation not yet wired up — tracks the plugin evaluation framework." All other eval machinery (grading, scoring, reporting) does execute.
Feature-flagged via tengu_walnut_spire; also enabled by CLAUDE_CODE_WALNUT_SPIRE env var.
Gate helper mvc() (line 184017) with check if (!mvc()) ms(...) at lines 780193–780194 (eval) and line 780275 (eval init); full command definition lines 780196–780284; partial-results skipped verdict at lines 672882–672889 (search for "CLAUDE_CODE_WALNUT_SPIRE" and "Model evaluation not yet wired up")
Strings lifted out of the shipped bundle, so the claim above can be checked against them.
tengu_walnut_spire Off in both readingsThe flag server returned off for the account this site reads and for the anonymous baseline. A reading of off cannot rule out a rollout these two readings sit outside of.
This account: off · anonymous baseline: off · compiled default in v2.1.207: off
These values were read against a different version of Claude Code, so treat them as the nearest reading available instead of one taken on this release.
Read once, for one account on one subscription tier, against v2.1.207. It isn't a statement about your account. What a flag value here can and cannot tell you
A reading is one sample. Claude Code evaluates its flags remotely, so no client sees the targeting rule behind a value and this says nothing about your account.
A reading is one sample. Claude Code evaluates its flags remotely, so no client sees the targeting rule behind a value and this says nothing about your account.