Plugin Eval Command [Early Access] / Claude Code v2.1.198

What

A new claude plugin eval CLI subcommand that runs structured evaluation suites against a plugin and scores the results.

Usage

# Interactive interview to author a new eval suite
claude plugin eval init

# Write a blank case template without the interview
claude plugin eval init --bare <name>

# Run eval cases against a plugin
claude plugin eval <target> [options]

Details

Cases live under evals/ as case.yaml or prompt.md files with companion graders in graders/*.md
Grader types: regex, tool_order, tool_used, file_exists, llm, baseline
LLM grader uses haiku by default; override with --judge-model <model>
Scoring options: --threshold <0..1>, --max-cost-usd <usd>, --runs <n>, --ablation <none|with-without>
Passes --scaffold to execute each case's scaffold_script (author-only; off by default)
Emits aggregate-result.json; use --output-dir <dir> to control placement or --json for stdout
Filter cases with --tag <tag> (repeatable) or a name glob
Cost ceiling support: if --max-cost-usd is hit, remaining paid graders are skipped while free graders still run
Early access: gated behind an invitation; the eval suite format and CLI flags may change

Evidence

CLI help text (search for "Run eval cases (evals//case.yaml or evals//prompt.md") and interview system ("Let's set up evals for this plugin.")

Strings lifted out of the shipped bundle, so the claim above can be checked against them.

Other releases about the same thing. Found by shared names or similar wording; neither means one caused the other.

v2.1.207 Walnut Spire (plugin eval General Availability) [In Development]

Plugin Eval Command [Early Access]

Related