{"version":"2.1.207","anchor":"walnut-spire-plugin-eval-general-availability-in-development","canonical_anchor":"walnut-spire-plugin-eval-general-availability-in-development","heading":"Walnut Spire ( plugin eval General Availability) [In Development]","tier":null,"area":null,"scope":null,"heads_up":null,"url":"https:\/\/changelogs.core-directive.com\/v\/2.1.207\/e\/walnut-spire-plugin-eval-general-availability-in-development","release_url":"https:\/\/changelogs.core-directive.com\/v\/2.1.207","markdown":"### Walnut Spire (`plugin eval` General Availability) [In Development]\n\n**What**\n\nThe `plugin eval` CLI subcommand (for evaluating plugins against test cases) was previously hard-gated behind `tengu_walnut_spire`. The gate is still present but is now also unlockable via the `CLAUDE_CODE_WALNUT_SPIRE` environment variable.\n\n**Details**\n\n- When `mvc()` returns `false` (both `tengu_walnut_spire` flag and `CLAUDE_CODE_WALNUT_SPIRE` env var absent), invoking `claude plugin eval` or `claude plugin eval init` immediately aborts with the message \"`plugin eval` is currently in early access\" \u2014 no command logic runs.\n- `claude plugin eval [target]` discovers eval cases from `evals\/**\/case.yaml` or `evals\/**\/prompt.md` + `graders\/*.md` under the target directory. The target may be a file-system path, an installed plugin name, or a `plugin@marketplace` id; skills-dir plugins also resolve and automatically add a no-plugin baseline arm for comparison.\n- The command exposes a full suite of options: `--case <glob>` (filter cases by name), `--tag <tag...>` (filter by tag, repeatable), `--runs <n>` (default from `case.runs ?? 3`), `--model` (override model), `--judge-model` (LLM grader model, default Haiku), `--max-cost-usd` (hard budget ceiling \u2014 breaching it exits with code 2 and reports partial results; the overrunning agent skips paid graders but free graders still score), `--output-dir` (default `.\/evals\/results\/<timestamp>\/`), `--json` (emit `aggregate-result.json` to stdout for CI), `--threshold <0..1>` (exit 1 if any case score falls below this value, default 1.0), `--allow-tools <tools...>` (operator grant for gated tools: `Bash`, `Write`, `Edit`, `WebFetch`, `mcp__*`; supports `Tool(pattern:*)` syntax), `--scaffold`\/`--no-scaffold` (run `scaffold_script` from case files, off by default), `--ablation` (`none` | `with-without`, automatically `with-without` when targeting a plugin by name), `--keep-temp` (preserve scaffold dirs for debugging), `--verbose` (stream trace as it runs).\n- `claude plugin eval init [name]` authors an eval suite under `evals\/` via an interactive interview (`--interactive`\/`--interview`) or writes a blank single-case template (`--bare`).\n- Model-trigger evaluation (the \"would the model route to this skill?\" check) is still incomplete: when the model callback returns `null`, the case receives verdict `skipped` with reason \"Model evaluation not yet wired up \u2014 tracks the plugin evaluation framework.\" All other eval machinery (grading, scoring, reporting) does execute.\n\n**Status**\n\nFeature-flagged via `tengu_walnut_spire`; also enabled by `CLAUDE_CODE_WALNUT_SPIRE` env var.\n\n**Evidence**\n\nGate helper `mvc()` (line 184017) with check `if (!mvc()) ms(...)` at lines 780193\u2013780194 (eval) and line 780275 (eval init); full command definition lines 780196\u2013780284; partial-results `skipped` verdict at lines 672882\u2013672889 (search for `\"CLAUDE_CODE_WALNUT_SPIRE\"` and `\"Model evaluation not yet wired up\"`)\n\n- Flag `tengu_walnut_spire`: Off in both readings (read for one account on one subscription tier against v2.1.207; this account: off, anonymous baseline: off, compiled default: off) These values were read against a different version of Claude Code, so treat them as the nearest reading available instead of one taken on this release."}