{"version":"2.1.269","anchor":"claude-plugin-eval-gains-allow-real-servers-and-concurre","canonical_anchor":"claude-plugin-eval-gains-allow-real-servers-and-concurre","heading":"'claude plugin eval' gains --allow-real-servers and concurrency options, plus harness-failure tracking","tier":"use","area":"Plugin Eval","scope":null,"heads_up":null,"url":"https:\/\/changelogs.core-directive.com\/v\/2.1.269\/e\/claude-plugin-eval-gains-allow-real-servers-and-concurre","release_url":"https:\/\/changelogs.core-directive.com\/v\/2.1.269","markdown":"### 'claude plugin eval' gains --allow-real-servers and concurrency options, plus harness-failure tracking\n\n'claude plugin eval' adds --allow-real-servers and concurrency options, plus separate harness-failure tracking\n\n**What**\n\nThe `claude plugin eval` command now accepts an `allowRealServers` option and a concurrency option, both passed through to the eval runner. It also now tracks harness failures (errors in the eval tooling itself) separately from case failures, and fails the run distinctly, recording a `harness_failure` event, when the harness itself errors rather than when individual test cases fail.\n\n**Why**\n\nSeparating harness failures from case failures makes it clearer whether an eval run failed because of the code being tested or because of a problem in the eval tooling itself. The new options give more control over whether real (non-mocked) servers can be used and how many cases run concurrently.\n\n- Area: Plugin Eval\n- Names: `--allow-real-servers`, `--concurrency`\n- Tier: Use it now\n- Useful: 4\/5\n- Signal: 2\/5"}