One read of Claude Code CLIclaude-code-20261014T160701Z
3 pages moved out of 222 read.
Pages moved
3
significant first
Pages read
222
in this capture
Captured
16:07 UTC
Corpus hash
62713cf25aeb
corpus-hash
What this read moved
1-3 of 3plugin-evals Changed · +22 / -5 lines
from line 409
409409
410410| Exit code | Meaning |
411411| :- | :- |
412| 0 | Every case scored at or above `--threshold` and every case file loaded |
413| 1 | A case scored below the threshold, a case file failed to load, no cases were found, a run couldn't be started, the plugin directory isn't trusted and `--trust-plugin` wasn't passed, or an option was invalid |
412| 0 | The suite passed: every case scored at or above `--threshold` and nothing else failed the run |
413| 1 | A case scored below the threshold, a case file failed to load, no cases were found, a run couldn't be started, or a run [ended on an API error](#runs-ended-on-an-api-error). Also returned when the plugin directory isn't trusted and `--trust-plugin` wasn't passed, an option was invalid, or the `--json` result couldn't be written |
414414| 2 | Partial run: the `--max-cost-usd` ceiling was hit, or your credential was rejected before or at the first run. `results.json` is still written with `partial: true` and the reason |
415415| 130 | Interrupted. Partial results are written |
416416| 143 | Terminated, such as by a CI timeout |
from line 463
463463| `cases[].name` | Case name |
464464| `cases[].aggregates.score` | Mean with-arm run score for the case |
465465| `cases[].aggregates.delta` | With-arm score minus without-arm score. Omitted when the case ran one arm or the arms aren't comparable |
466| `cases[].arms.with[].error` | `null`, or why a run ended abnormally, such as `timed out after 300s`. A run that started but ended badly is still graded on what it produced, so a non-null error doesn't imply score 0 |
466| `cases[].arms.with[].error` | `null`, or why a run ended abnormally, such as `timed out after 300s`. Some of these runs are still graded on what they produced, so a non-null error doesn't imply score 0. A run that [ended on an API error](#runs-ended-on-an-api-error) scores 0 without grading |
467467| `cases[].arms.with[].aborted` | Present when a [mock](#mock-mcp-servers) stopped the run through `expect:`, `abort_when`, or the [agent-mock call budget](#mock-call-budget-exceeded), with `server`, `tool`, and `reason`. The run scores 0 and `error` stays `null` |
468468| `cases[].arms.with[].skippedPaidGraders` | `true` when the cost ceiling skipped this run's judge graders, so its score isn't comparable |
469469| `costUsd`, `durationSeconds`, `claudeVersion` | Estimated cost at list price including judge calls, wall-clock seconds, and the Claude Code version that ran the suite |
from line 716
716716
717717### The run exits 1 but the results look fine
718718
719The default `--threshold` is 1.0, so the command exits 1 when any case scores below perfect. Set a threshold that matches the score you require. Exit 1 also covers a case file that failed to load, which is reported on stderr above the table.
719When every case scored at or above `--threshold` and the command still exits 1, a line on stderr names the cause, even under `--json`. For example, a case file failed to load, or runs [ended on an API error](#runs-ended-on-an-api-error). For the other causes, see the [exit codes](#run-evals-in-ci).
720720
721The default `--threshold` is 1.0, so the command exits 1 when any case scores below perfect. Set a threshold that matches the score you require.
722
721723<h3 id="json-output-path-must-end-in-json">
722724 `--json output path must end in .json`
723725</h3>
from line 730
728730
729731That grader is excluded from the score by design in a two-arm run, and its `scored` field is `false`. See [Score against the no-plugin baseline](#compare-against-a-no-plugin-baseline).
730732
733<h3 id="runs-ended-on-an-api-error">
734 "ended on an API error and scored 0 without grading"
735</h3>
736
737This stderr warning means the API answered at least one run's last request with an HTTP error, such as a rate limit or a plan usage limit. Such a run scores 0 with no grader results, and the command exits non-zero whatever the scores and `--threshold` are.
738
739To fix it, find each affected run's error and resolve the cause it reports, then run the suite again:
740
741* **Without `--json`**: the error is at the end of the run's progress line, after `error:`
742* **Under `--json`**: the error is in the [JSON result](#json-result), at `cases[].arms.with[].error` or `cases[].arms.without[].error`
743
744The count includes [baseline](#the-no-plugin-baseline) runs too. Their scores never decide whether a case passes, so every case can pass while the command still fails. In a two-arm run, the affected case's `W/OUT` and `Δ` print as `—`. For a usage or rate limit, see [Runs fail with a usage-limit or rate-limit error](#runs-fail-with-a-usage-limit-or-rate-limit-error-partway-through).
745
746Before v2.1.299, such a run was graded on what it had produced.
747
731748### Runs fail with a usage-limit or rate-limit error partway through
732749
733If your account reaches its plan's usage limit or an API rate limit while a suite is running, each later run ends with that error, is graded on what it produced, and usually scores 0. The suite still finishes and isn't marked `partial`, so the result can look like a regression. Check the `NOTES` column or `cases[].arms.with[].error` in the JSON for the limit message before trusting the scores, then re-run after the limit resets, with `--runs 1` or a `--case` filter if you need to stay under it.
750Runs that end with a usage-limit or rate-limit error [score 0 without grading](#runs-ended-on-an-api-error), and the command exits non-zero. The suite still finishes and isn't marked `partial`. Re-run after the limit resets, with `--runs 1` or a `--case` filter if you need to stay under it.
734751
735752<h3 id="mock-call-budget-exceeded">
736753 "mock call budget exceeded"
claude-tag Changed · +1 / -1 lines
This page is larger than the 256 KiB this site keeps, so one side of the diff below stops where the stored text does.
Nothing in the body moved in this read. What changed is above.
plugins/cli-reference Changed · +2 / -2 lines
from line 481
481481
482482| Exit code | Meaning |
483483| :- | :- |
484| `0` | Every case meets the threshold |
485| `1` | A failing case, a load error, or an untrusted plugin directory |
484| `0` | The suite passed: every case meets the threshold and nothing else failed the run |
485| `1` | A failing case, a load error, an untrusted plugin directory, a run that [ended on an API error](/docs/en/plugin-evals#runs-ended-on-an-api-error), or another cause listed under [Run evals in CI](/docs/en/plugin-evals#run-evals-in-ci) |
486486| `2` | A partial run |
487487| `130` | Interrupted |
488488| `143` | Terminated |