You'll noticeTier: how much it should matter to you
2Useful: my rating, 1 to 5
2Signal: worth watching, 1 to 5
Plugin EvalArea: what it touches
New FeaturesKind: in v2.1.224,
New FeaturesSection of the release
What
claude plugin eval --json now checks its result against the expected shape before printing. If validation fails the JSON is not printed at all and a warning goes to stderr instead; if it merely differs from the expected shape a warning is printed and the result is still emitted.
Details
On drift, a result_schema_drift counter fires and the result is still printed.
On outright validation failure, output is withheld entirely and stderr carries the explanation.
New outcome telemetry records pass and fail counts, how many cases hit run errors, and separates exits caused by a cost ceiling from exits caused by a case that failed to load.
Evidence
warning: --json result withheld because it failed schema validation
Strings lifted out of the shipped bundle, so the claim above can be checked against them.
