{"version":"2.1.198","anchor":"plugin-eval-command-early-access","canonical_anchor":"plugin-eval-command-early-access","heading":"Plugin Eval Command [Early Access]","tier":null,"area":null,"scope":null,"heads_up":null,"url":"https:\/\/changelogs.core-directive.com\/v\/2.1.198\/e\/plugin-eval-command-early-access","release_url":"https:\/\/changelogs.core-directive.com\/v\/2.1.198","markdown":"### Plugin Eval Command [Early Access]\n\n**What**\n\nA new `claude plugin eval` CLI subcommand that runs structured evaluation suites against a plugin and scores the results.\n\n**Usage**\n\n`# Interactive interview to author a new eval suite claude plugin eval init # Write a blank case template without the interview claude plugin eval init --bare <name> # Run eval cases against a plugin claude plugin eval <target> [options]` **Details** - Cases live under `evals\/` as `case.yaml` or `prompt.md` files with companion graders in `graders\/*.md` - Grader types: `regex`, `tool_order`, `tool_used`, `file_exists`, `llm`, `baseline` - LLM grader uses haiku by default; override with `--judge-model <model>` - Scoring options: `--threshold <0..1>`, `--max-cost-usd <usd>`, `--runs <n>`, `--ablation <none|with-without>` - Passes `--scaffold` to execute each case's `scaffold_script` (author-only; off by default) - Emits `aggregate-result.json`; use `--output-dir <dir>` to control placement or `--json` for stdout - Filter cases with `--tag <tag>` (repeatable) or a name glob - Cost ceiling support: if `--max-cost-usd` is hit, remaining paid graders are skipped while free graders still run - Early access: gated behind an invitation; the eval suite format and CLI flags may change **Evidence** CLI help text (search for `\"Run eval cases (evals\/**\/case.yaml or evals\/**\/prompt.md\"`) and interview system (`\"Let's set up evals for this plugin.\"`)"}