Source Intelligence
Follow Discord

DisclaimerUnofficial, and not affiliated with Anthropic. Nearly all of this is read straight out of what ships: npm bundles, captured prompts, published docs. Anthropic's own notes go in verbatim, marked as theirs. The rest is my reading, and every entry carries the strings behind it. If one looks wrong, vote it down and say why.

← All of v2.1.224 Home All releases ←olderv2.1.223 v2.1.225newer→
Claude Code v2.1.224

Plugin eval can validate and report its own JSON output

You'll notice
Useful2 Signal2
Plugin Eval

Plugin eval JSON output is validated before printing, and suppressed with a stderr warning if invalid.

claude plugin eval--json
What

claude plugin eval --json now checks its result against the expected shape before printing. If validation fails the JSON is not printed at all and a warning goes to stderr instead; if it merely differs from the expected shape a warning is printed and the result is still emitted.

Details
  • On drift, a result_schema_drift counter fires and the result is still printed.
  • On outright validation failure, output is withheld entirely and stderr carries the explanation.
  • New outcome telemetry records pass and fail counts, how many cases hit run errors, and separates exits caused by a cost ceiling from exits caused by a case that failed to load.
Evidence

warning: --json result withheld because it failed schema validation

Strings lifted out of the shipped bundle, so the claim above can be checked against them.

Related

Other releases about the same thing. Found by shared names or similar wording; neither means one caused the other.

See this entry in the whole of v2.1.224 →