{"version":"2.1.234","anchor":"plugin-eval-suites-need-operator-opt-in-to-publish-artifacts","canonical_anchor":"plugin-eval-suites-need-operator-opt-in-to-publish-artifacts","heading":"Plugin eval suites need operator opt-in to publish artifacts or override feature flags","tier":"use","area":"Plugin Evals","scope":null,"heads_up":null,"url":"https:\/\/changelogs.core-directive.com\/v\/2.1.234\/e\/plugin-eval-suites-need-operator-opt-in-to-publish-artifacts","release_url":"https:\/\/changelogs.core-directive.com\/v\/2.1.234","markdown":"### Plugin eval suites need operator opt-in to publish artifacts or override feature flags\n\nEval suites can only publish artifacts or override feature flags if you explicitly allow it.\n\n**What**\n\nTwo environment variables fence what an eval suite can do to the machine running it, both off by default. A suite asking to publish artifacts only does so if the operator passes the flag or sets `CLAUDE_CODE_EVAL_ALLOW_ARTIFACT_PUBLISH`, and a suite's feature-flag overrides are filtered against an allowlist in `CLAUDE_CODE_EVAL_ALLOW_FLAG_OVERRIDES`, with anything unlisted dropped and reported on the console.\n\n**Details**\n\n- Publishing outcomes are recorded as `cli_plugin_eval_artifact_publish` with results `success`, `run_errored` or `no_publish`, and only for the arm that has the plugin installed.\n- The eval job definition gained `artifact_publish` and `growthbook_overrides`, the latter a map of boolean, string, number or null values. Overrides are only seeded for runs that were granted artifact publishing.\n- `CLAUDE_CODE_EVAL_ALLOW_FLAG_OVERRIDES` was added to the scrubbed and forwarded environment lists.\n- A suite that has a history file now runs only the with-plugin arm instead of both arms.\n- The HTML eval report lists each plugin with a chip when it has a recorded problem.\n\n**Evidence**\n\n`growthbook_overrides dropped (operator allowlist required \u2014 CLAUDE_CODE_EVAL_ALLOW_FLAG_OVERRIDES)`, `CLAUDE_CODE_EVAL_ALLOW_FLAG_OVERRIDES`\n\n- Area: Plugin Evals\n- Names: `CLAUDE_CODE_EVAL_ALLOW_ARTIFACT_PUBLISH`, `CLAUDE_CODE_EVAL_ALLOW_FLAG_OVERRIDES`\n- Tier: Use it now\n- Useful: 2\/5\n- Signal: 2\/5"}