What
Two environment variables fence what an eval suite can do to the machine running it, both off by default. A suite asking to publish artifacts only does so if the operator passes the flag or sets CLAUDE_CODE_EVAL_ALLOW_ARTIFACT_PUBLISH, and a suite's feature-flag overrides are filtered against an allowlist in CLAUDE_CODE_EVAL_ALLOW_FLAG_OVERRIDES, with anything unlisted dropped and reported on the console.
Details
- Publishing outcomes are recorded as
cli_plugin_eval_artifact_publishwith resultssuccess,run_erroredorno_publish, and only for the arm that has the plugin installed. - The eval job definition gained
artifact_publishandgrowthbook_overrides, the latter a map of boolean, string, number or null values. Overrides are only seeded for runs that were granted artifact publishing. CLAUDE_CODE_EVAL_ALLOW_FLAG_OVERRIDESwas added to the scrubbed and forwarded environment lists.- A suite that has a history file now runs only the with-plugin arm instead of both arms.
- The HTML eval report lists each plugin with a chip when it has a recorded problem.
Evidence
growthbook_overrides dropped (operator allowlist required — CLAUDE_CODE_EVAL_ALLOW_FLAG_OVERRIDES), CLAUDE_CODE_EVAL_ALLOW_FLAG_OVERRIDES
Strings lifted out of the shipped bundle, so the claim above can be checked against them.
Names in the bundleCLAUDE_CODE_EVAL_ALLOW_ARTIFACT_PUBLISHCLAUDE_CODE_EVAL_ALLOW_FLAG_OVERRIDES