Eval suites can only publish artifacts or override feature flags if you explicitly allow it.
What's wrong with this entry?
Two environment variables fence what an eval suite can do to the machine running it, both off by default. A suite asking to publish artifacts only does so if the operator passes the flag or sets CLAUDE_CODE_EVAL_ALLOW_ARTIFACT_PUBLISH, and a suite's feature-flag overrides are filtered against an allowlist in CLAUDE_CODE_EVAL_ALLOW_FLAG_OVERRIDES, with anything unlisted dropped and reported on the console.
- Publishing outcomes are recorded as
cli_plugin_eval_artifact_publishwith resultssuccess,run_erroredorno_publish, and only for the arm that has the plugin installed. - The eval job definition gained
artifact_publishandgrowthbook_overrides, the latter a map of boolean, string, number or null values. Overrides are only seeded for runs that were granted artifact publishing. CLAUDE_CODE_EVAL_ALLOW_FLAG_OVERRIDESwas added to the scrubbed and forwarded environment lists.- A suite that has a history file now runs only the with-plugin arm instead of both arms.
- The HTML eval report lists each plugin with a chip when it has a recorded problem.
growthbook_overrides dropped (operator allowlist required — CLAUDE_CODE_EVAL_ALLOW_FLAG_OVERRIDES), CLAUDE_CODE_EVAL_ALLOW_FLAG_OVERRIDES
Strings lifted out of the shipped bundle, so the claim above can be checked against them.
Related
Other releases about the same thing. Found by shared names or similar wording; neither means one caused the other.
-
v2.1.233
Eval report publishing prints a working link and a way to opt out
Both mention eval publish