Test plugins with evals changedplugin-evals
Nearest release: v2.1.300, published an hour before upstream edited the page. Shown because the two are within 24 hours of each other. Nothing here says the release caused the edit.
Upstream edited this page at 14 Oct 2026 15:54 UTC, give or take a minute or two: the time comes from Anthropic’s own sitemap rather than from a commit. This site recorded the change at 14 Oct 2026 16:07 UTC.
Upstream edited
Recorded here
Lines+22added
Lines−5removed
From line
409
where the diff opens
First seen
11 Sep 2026
this site's first read of the page
Recorded edits19to this page, all time
The whole hunk
from line 409, old and new numbered
/
from line 409
409409
410410| Exit code | Meaning |
411411| :- | :- |
412| 0 | Every case scored at or above `--threshold` and every case file loaded |
413| 1 | A case scored below the threshold, a case file failed to load, no cases were found, a run couldn't be started, the plugin directory isn't trusted and `--trust-plugin` wasn't passed, or an option was invalid |
412| 0 | The suite passed: every case scored at or above `--threshold` and nothing else failed the run |
413| 1 | A case scored below the threshold, a case file failed to load, no cases were found, a run couldn't be started, or a run [ended on an API error](#runs-ended-on-an-api-error). Also returned when the plugin directory isn't trusted and `--trust-plugin` wasn't passed, an option was invalid, or the `--json` result couldn't be written |
414414| 2 | Partial run: the `--max-cost-usd` ceiling was hit, or your credential was rejected before or at the first run. `results.json` is still written with `partial: true` and the reason |
415415| 130 | Interrupted. Partial results are written |
416416| 143 | Terminated, such as by a CI timeout |
from line 463
463463| `cases[].name` | Case name |
464464| `cases[].aggregates.score` | Mean with-arm run score for the case |
465465| `cases[].aggregates.delta` | With-arm score minus without-arm score. Omitted when the case ran one arm or the arms aren't comparable |
466| `cases[].arms.with[].error` | `null`, or why a run ended abnormally, such as `timed out after 300s`. A run that started but ended badly is still graded on what it produced, so a non-null error doesn't imply score 0 |
466| `cases[].arms.with[].error` | `null`, or why a run ended abnormally, such as `timed out after 300s`. Some of these runs are still graded on what they produced, so a non-null error doesn't imply score 0. A run that [ended on an API error](#runs-ended-on-an-api-error) scores 0 without grading |
467467| `cases[].arms.with[].aborted` | Present when a [mock](#mock-mcp-servers) stopped the run through `expect:`, `abort_when`, or the [agent-mock call budget](#mock-call-budget-exceeded), with `server`, `tool`, and `reason`. The run scores 0 and `error` stays `null` |
468468| `cases[].arms.with[].skippedPaidGraders` | `true` when the cost ceiling skipped this run's judge graders, so its score isn't comparable |
469469| `costUsd`, `durationSeconds`, `claudeVersion` | Estimated cost at list price including judge calls, wall-clock seconds, and the Claude Code version that ran the suite |
from line 716
716716
717717### The run exits 1 but the results look fine
718718
719The default `--threshold` is 1.0, so the command exits 1 when any case scores below perfect. Set a threshold that matches the score you require. Exit 1 also covers a case file that failed to load, which is reported on stderr above the table.
719When every case scored at or above `--threshold` and the command still exits 1, a line on stderr names the cause, even under `--json`. For example, a case file failed to load, or runs [ended on an API error](#runs-ended-on-an-api-error). For the other causes, see the [exit codes](#run-evals-in-ci).
720720
721The default `--threshold` is 1.0, so the command exits 1 when any case scores below perfect. Set a threshold that matches the score you require.
722
721723<h3 id="json-output-path-must-end-in-json">
722724 `--json output path must end in .json`
723725</h3>
from line 730
728730
729731That grader is excluded from the score by design in a two-arm run, and its `scored` field is `false`. See [Score against the no-plugin baseline](#compare-against-a-no-plugin-baseline).
730732
733<h3 id="runs-ended-on-an-api-error">
734 "ended on an API error and scored 0 without grading"
735</h3>
736
737This stderr warning means the API answered at least one run's last request with an HTTP error, such as a rate limit or a plan usage limit. Such a run scores 0 with no grader results, and the command exits non-zero whatever the scores and `--threshold` are.
738
739To fix it, find each affected run's error and resolve the cause it reports, then run the suite again:
740
741* **Without `--json`**: the error is at the end of the run's progress line, after `error:`
742* **Under `--json`**: the error is in the [JSON result](#json-result), at `cases[].arms.with[].error` or `cases[].arms.without[].error`
743
744The count includes [baseline](#the-no-plugin-baseline) runs too. Their scores never decide whether a case passes, so every case can pass while the command still fails. In a two-arm run, the affected case's `W/OUT` and `Δ` print as `—`. For a usage or rate limit, see [Runs fail with a usage-limit or rate-limit error](#runs-fail-with-a-usage-limit-or-rate-limit-error-partway-through).
745
746Before v2.1.299, such a run was graded on what it had produced.
747
731748### Runs fail with a usage-limit or rate-limit error partway through
732749
733If your account reaches its plan's usage limit or an API rate limit while a suite is running, each later run ends with that error, is graded on what it produced, and usually scores 0. The suite still finishes and isn't marked `partial`, so the result can look like a regression. Check the `NOTES` column or `cases[].arms.with[].error` in the JSON for the limit message before trusting the scores, then re-run after the limit resets, with `--runs 1` or a `--case` filter if you need to stay under it.
750Runs that end with a usage-limit or rate-limit error [score 0 without grading](#runs-ended-on-an-api-error), and the command exits non-zero. The suite still finishes and isn't marked `partial`. Re-run after the limit resets, with `--runs 1` or a `--case` filter if you need to stay under it.
734751
735752<h3 id="mock-call-budget-exceeded">
736753 "mock call budget exceeded"
No line in this hunk matches that.