### plugin eval ### plugin eval init
The whole hunk
from line 456, old and new numbered
/
lines
from line 456
456456 "lspServers": "./.lsp.json",
457457 "experimental": {
458458 "themes": "./themes/",
459 "monitors": "./monitors.json"
459 "monitors": "./monitors.json",
460 "evals": "quality/evals"
460461 },
461462 "dependencies": [
462463 "helper-lib",
from line 533
532533
533534### Component path fields
534535
535| Field | Type | Description | Example |
536| :---------------------- | :-------------------- | :---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | :--------------------------------------------------- |
537| `skills` | string\|array | Custom skill directories containing `<name>/SKILL.md`. Adds to the default `skills/` scan. See [Path behavior rules](#path-behavior-rules) for the marketplace-root exception | `"./custom/skills/"` |
538| `commands` | string\|array | Custom flat `.md` skill files or directories (replaces default `commands/`) | `"./custom/cmd.md"` or `["./cmd1.md"]` |
539| `agents` | string\|array | Custom agent files (replaces default `agents/`) | `"./custom/agents/reviewer.md"` |
540| `workflows` | string\|array | Custom [workflow](/docs/en/workflows) script files or directories (replaces default `workflows/`) | `"./custom/workflows/"` |
541| `hooks` | string\|array\|object | Hook config paths or inline config | `"./my-extra-hooks.json"` |
542| `mcpServers` | string\|array\|object | MCP config paths or inline config | `"./my-extra-mcp-config.json"` |
543| `outputStyles` | string\|array | Custom output style files/directories (replaces default `output-styles/`) | `"./styles/"` |
544| `lspServers` | string\|array\|object | [Language Server Protocol](https://microsoft.github.io/language-server-protocol/) configs for code intelligence (go to definition, find references, etc.) | `"./.lsp.json"` |
545| `experimental.themes` | string\|array | Color theme files/directories (replaces default `themes/`). See [Themes](#themes) | `"./themes/"` |
546| `experimental.monitors` | string\|array | Background [Monitor](/docs/en/tools-reference#monitor-tool) configurations that start automatically when the plugin is active. See [Monitors](#monitors) | `"./monitors.json"` |
547| `userConfig` | object | User-configurable values prompted at enable time. See [User configuration](#user-configuration) | See below |
548| `channels` | array | Channel declarations for message injection (Telegram, Slack, Discord style). See [Channels](#channels) | See below |
549| `dependencies` | array | Other plugins this plugin requires, optionally with semver version constraints. See [Constrain plugin dependency versions](/docs/en/plugin-dependencies) | `[{ "name": "secrets-vault", "version": "~2.1.0" }]` |
536| Field | Type | Description | Example |
537| :---------------------- | :-------------------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | :--------------------------------------------------- |
538| `skills` | string\|array | Custom skill directories containing `<name>/SKILL.md`. Adds to the default `skills/` scan. See [Path behavior rules](#path-behavior-rules) for the marketplace-root exception | `"./custom/skills/"` |
539| `commands` | string\|array | Custom flat `.md` skill files or directories (replaces default `commands/`) | `"./custom/cmd.md"` or `["./cmd1.md"]` |
540| `agents` | string\|array | Custom agent files (replaces default `agents/`) | `"./custom/agents/reviewer.md"` |
541| `workflows` | string\|array | Custom [workflow](/docs/en/workflows) script files or directories (replaces default `workflows/`) | `"./custom/workflows/"` |
542| `hooks` | string\|array\|object | Hook config paths or inline config | `"./my-extra-hooks.json"` |
543| `mcpServers` | string\|array\|object | MCP config paths or inline config | `"./my-extra-mcp-config.json"` |
544| `outputStyles` | string\|array | Custom output style files/directories (replaces default `output-styles/`) | `"./styles/"` |
545| `lspServers` | string\|array\|object | [Language Server Protocol](https://microsoft.github.io/language-server-protocol/) configs for code intelligence (go to definition, find references, etc.) | `"./.lsp.json"` |
546| `experimental.themes` | string\|array | Color theme files/directories (replaces default `themes/`). See [Themes](#themes) | `"./themes/"` |
547| `experimental.monitors` | string\|array | Background [Monitor](/docs/en/tools-reference#monitor-tool) configurations that start automatically when the plugin is active. See [Monitors](#monitors) | `"./monitors.json"` |
548| `experimental.evals` | string\|array | Directory below the plugin root that holds the plugin's [eval cases](/docs/en/plugin-evals#use-a-different-eval-directory), when it isn't the default `evals/`. `claude plugin eval --eval-dir` overrides it | `"quality/evals"` |
549| `userConfig` | object | User-configurable values prompted at enable time. See [User configuration](#user-configuration) | See below |
550| `channels` | array | Channel declarations for message injection (Telegram, Slack, Discord style). See [Channels](#channels) | See below |
551| `dependencies` | array | Other plugins this plugin requires, optionally with semver version constraints. See [Constrain plugin dependency versions](/docs/en/plugin-dependencies) | `[{ "name": "secrets-vault", "version": "~2.1.0" }]` |
550552
551553### Experimental components
552554
from line 1256
12541256On exit 2, the command writes nothing to stdout; the error message goes to stderr.
12551257
12561258Within an interactive session, `/plugin validate <path>` runs the same checks inline.
1259
1260### plugin eval
1261
1262Run a plugin's [eval cases](/docs/en/plugin-evals) and report scored results. Requires Claude Code v2.1.269 or later. Each case is a prompt plus graders; Claude Code runs it several times in an isolated session with only the target plugin loaded, and by default also without the plugin so the report shows the difference. See [Test plugins with evals](/docs/en/plugin-evals) for the case format, graders, results, and CI usage.
1263
1264```bash theme={null}
1265claude plugin eval [target] [options]
1266```
1267
1268The optional `target` is a plugin directory, a single `prompt.md` or `case.yaml` file, an installed plugin as `name` or `name@marketplace`, or `name@skills-dir`, and defaults to the current directory. Put it before `--tag`, `--allow-tools`, and `--json`.
1269
1270This table lists the options most runs use. Run `claude plugin eval --help` for the complete set, including `--case`, `--tag`, `--output-dir`, `--report`, `--allow-real-servers`, `--keep-temp`, and `--verbose`.
1271
1272| Option | Description | Default |
1273| :------------------------- | :-------------------------------------------------------------------------------------------------------------------------------------------------------------- | :----------------------------------------------------------------------------- |
1274| `--runs <n>` | Runs per case per arm | Each case's `runs`, else 3 |
1275| `-j, --concurrency <n>` | Agent sessions to run at once, 1 to 8. They share your rate limit | `1` |
1276| `--model <model>` | Model for the agent under test | Each case's `model`, else `ANTHROPIC_MODEL` if set, else Claude Code's default |
1277| `--judge-model <model>` | Model for `llm` and `baseline` graders | A small fast model |
1278| `--ablation <mode>` | `none` or `with-without`. See [Compare against a no-plugin baseline](/docs/en/plugin-evals#compare-against-a-no-plugin-baseline) | `with-without` when a plugin resolves, else `none` |
1279| `--threshold <0..1>` | Exit 1 if any case scores below this | `1.0` |
1280| `--max-cost-usd <usd>` | Stop before the next run once spend reaches this, exit 2, and report partial results | No ceiling |
1281| `--allow-tools <tools...>` | Grant tools beyond the read-only set, such as `Bash`, `Write`, `Edit`, or `"mcp__plugin_<plugin>_<server>__*"`. See [Grant tools](/docs/en/plugin-evals#grant-tools) | |
1282| `--scaffold` | Run each case's [`scaffold_script`](/docs/en/plugin-evals#add-setup-or-history-with-case-yaml) | Off |
1283| `--trust-plugin` | Skip the first-run trust prompt, for CI. See [What a run can access](/docs/en/plugin-evals#security) | Off |
1284| `--mocks <mode>` | `record` or `off`. See [Mock MCP servers](/docs/en/plugin-evals#mock-mcp-servers) | `record` |
1285| `--eval-dir <dir>` | Directory below the plugin that holds the cases | The manifest's `experimental.evals`, else `evals` |
1286| `--json [path]` | Print the [result document](/docs/en/plugin-evals#json-result) to stdout, or write it to a `.json` path | |
1287| `--no-publish` | Keep the HTML report local | |
1288| `-h, --help` | Display help for command | |
1289
1290The command exits 0 when every case meets the threshold, 1 on a failing case, a load error, or an untrusted plugin directory, 2 on a partial run, 130 when interrupted, and 143 when terminated. See [Run evals in CI](/docs/en/plugin-evals#run-evals-in-ci).
1291
1292### plugin eval init
1293
1294Create an eval suite for the plugin in the current directory. Requires Claude Code v2.1.269 or later. In a terminal this starts an authoring interview that reads the plugin, proposes cases and graders, pilots them, and writes the files. With `--bare`, or without a terminal, it writes a blank single-case template instead. Run from inside an interactive Claude Code session, it prints the interview instructions for that session to follow rather than writing a template. See [Create your first eval suite](/docs/en/plugin-evals#create-your-first-eval-suite).
1295
1296```bash theme={null}
1297claude plugin eval init [name] [options]
1298```
1299
1300The optional `name` is a case name: the interview doesn't need one, while `--bare` and the no-terminal template path require it. It accepts these options:
1301
1302| Option | Description | Default |
1303| :------------------ | :------------------------------------------------------------------------------------------------ | :------------------------------------------------ |
1304| `--bare` | Write a blank `prompt.md` and `graders/criteria.md` for `<name>` instead of running the interview | |
1305| `-i, --interactive` | Require the interview. Fails without a terminal instead of writing a template | |
1306| `--eval-dir <dir>` | Directory below the current directory to write cases into | The manifest's `experimental.evals`, else `evals` |
1307| `-h, --help` | Display help for command | |
12571308
12581309### plugin tag
12591310