Plugin Eval Command [Early Access]#
What's wrong with this entry?
A new claude plugin eval CLI subcommand that runs structured evaluation suites against a plugin and scores the results.
# Interactive interview to author a new eval suite
claude plugin eval init
# Write a blank case template without the interview
claude plugin eval init --bare <name>
# Run eval cases against a plugin
claude plugin eval <target> [options]- Cases live under
evals/ascase.yamlorprompt.mdfiles with companion graders ingraders/*.md - Grader types:
regex,tool_order,tool_used,file_exists,llm,baseline - LLM grader uses haiku by default; override with
--judge-model <model> - Scoring options:
--threshold <0..1>,--max-cost-usd <usd>,--runs <n>,--ablation <none|with-without> - Passes
--scaffoldto execute each case'sscaffold_script(author-only; off by default) - Emits
aggregate-result.json; use--output-dir <dir>to control placement or--jsonfor stdout - Filter cases with
--tag <tag>(repeatable) or a name glob - Cost ceiling support: if
--max-cost-usdis hit, remaining paid graders are skipped while free graders still run - Early access: gated behind an invitation; the eval suite format and CLI flags may change
CLI help text (search for "Run eval cases (evals//case.yaml or evals//prompt.md") and interview system ("Let's set up evals for this plugin.")
Strings lifted out of the shipped bundle, so the claim above can be checked against them.
Related
Other releases about the same thing. Found by shared names or similar wording; neither means one caused the other.