Claude Code v2.1.198 · 1 Jul 2026
Plugin Eval Command [Early Access]
Entry Kind: in v2.1.198, 01 Jul 26
New Features Section of the release
What
A new claude plugin eval CLI subcommand that runs structured evaluation suites against a plugin and scores the results.
Usage
# Interactive interview to author a new eval suite
claude plugin eval init
# Write a blank case template without the interview
claude plugin eval init --bare <name>
# Run eval cases against a plugin
claude plugin eval <target> [options]Details
Cases live under evals/ as case.yaml prompt.md graders/*.md Grader types: regex tool_order, tool_used, file_exists, llm baseline LLM grader uses haiku by default; override with --judge-model <model> Scoring options: --threshold <0..1>--max-cost-usd <usd>--runs <n>--ablation <none|with-without> Passes --scaffold scaffold_script (author-only; off by default) Emits aggregate-result.json; use --output-dir <dir>--json Filter cases with --tag <tag> Cost ceiling support: if --max-cost-usd is hit, remaining paid graders are skipped while free graders still run Early access: gated behind an invitation; the eval suite format and CLI flags may change Evidence
CLI help text (search for "Run eval cases (evals//case.yaml or evals/ /prompt.md") and interview system ("Let's set up evals for this plugin.")
Strings lifted out of the shipped bundle, so the claim above can be checked against them.
See this entry in the whole of v2.1.198 →
Names in this entry
setting case.yaml plugin eval warns when writing results inside a plugin's own directory could get picked up as plugin components
setting prompt.md plugin eval warns when writing results inside a plugin's own directory could get picked up as plugin components
setting regex Eval graders can now judge images, so rendered slides and charts get scored on appearance.
setting llm Eval graders can now judge images, so rendered slides and charts get scored on appearance.
cli --judge-model --judge-model <model> is an option of claude plugin eval [target], which runs a plugin's eval cases and reports scored results. It sets the model used by the LLM grader, default haiku.
cli --threshold Option of claude plugin eval, a number from 0 to 1 (default 1). A case passes when its score (the mean of its runs) is at or above it, and the command exits 1 if any case scores below it. A value outside 0..1 exits 1 with Error: --threshold must be between 0 and 1.
cli --max-cost-usd Help text for --max-cost-usd now explains its cost ceiling is checked before each run, not per single run
cli --runs Option of claude plugin eval. Number of runs per case, overriding each case's own runs value for every case in the suite. Must be a positive integer, otherwise the command prints Error: --runs must be a positive integer and exits 1.
cli --ablation Plugin eval prompts now go over stdin and runs ignore project settings, so long prompts survive intact.
cli --scaffold Eval cases with a scaffold_script now warn explicitly when --scaffold wasn't passed instead of silently skipping it
cli --output-dir plugin eval warns when writing results inside a plugin's own directory could get picked up as plugin components
cli --json Plugin eval JSON output is validated before printing, and suppressed with a stderr warning if invalid.
cli --tag Option of claude plugin eval, repeatable. Filters the suite to cases whose tags include at least one of the given tags; it combines with --case, so a case must also match that glob.
Is this right?
0 0 Feedback
What's wrong with this entry?
Hard to understand Too vague Looks wrong Not relevant to me Too long Missing the detail I wanted Duplicate of another card
Clear Learned something new I can act on this Good catch
Related
Other releases about the same thing. Found by shared names or similar wording; neither means one caused the other.
In v2.1.198 New Features release-page order
Next → · 2 of 26
Auto-Mode Setup
