Follow Discord
Sweep 25 Sep 2026 · 19:33Z Build v2.1.283 504 read Stable v2.1.274 Latest v2.1.283 Next v2.1.283 Feeds RSS JSON llms.txt llms-full.txt Unofficial
HomeDocsClaude Code CLI 24 Sep 2026
One change · claude-code

Test plugins with evals changedplugin-evals

Nearest release: v2.1.282, published an hour before upstream edited the page. Shown because the two are within 24 hours of each other. Nothing here says the release caused the edit.

Upstream edited this page at 24 Sep 2026 17:00 UTC, give or take a minute or two: the time comes from Anthropic’s own sitemap rather than from a commit. This site recorded the change at 24 Sep 2026 17:07 UTC.

Upstream edited
Recorded here
Lines+9added
Lines−1removed
From line 421 where the diff opens
First seen 11 Sep 2026 this site's first read of the page
Recorded edits7to this page, all time

### "Agent type '...' not found" for one of your plugin's agents

The whole hunk

from line 421, old and new numbered
/
lines
from line 421
421421 What a run can access
422422</h2>
423423 
424`claude plugin eval` loads the target plugin's skills and hooks and runs its eval suite on your machine, as you. Pointing it at a plugin is the same trust decision as `claude --plugin-dir`, so only evaluate plugins you trust. The isolation described in this section limits what the agent under test can reach; it isn't a boundary against the plugin's own code, and a suite that passes says nothing about whether the plugin is safe.
424`claude plugin eval` loads the target plugin's skills, hooks, and agents and runs its eval suite on your machine, as you. Pointing it at a plugin is the same trust decision as `claude --plugin-dir`, so only evaluate plugins you trust. The isolation described in this section limits what the agent under test can reach; it isn't a boundary against the plugin's own code, and a suite that passes says nothing about whether the plugin is safe.
425425 
426426### Trust the plugin directory
427427 
from line 585
585585If the summary has no `W/OUT` column, or the case fails with "ablation requested but no plugin resolved", no plugin was found for the case. Add `plugins: ["../.."]` to the case, giving the path from the case directory to the plugin directory.
586586 
587587If the plugin did load and `Δ` is still near zero with your `tool_used: Skill` grader failing, that's usually a real finding, meaning the skill's `description` doesn't trigger on the prompt's phrasing. Adjust the description and re-run the same suite.
588 
589### "Agent type '...' not found" for one of your plugin's agents
590 
591By default each case runs both with your plugin and without it, and the runs without it are the [no-plugin baseline](#the-no-plugin-baseline). When Claude dispatches one of your plugin's agents in a baseline run, the Agent tool call fails with `Agent type '<plugin>:<agent-name>' not found. Available agents: ...`. The list names only agents that exist without the plugin, such as the [built-in subagents](/docs/en/sub-agents#built-in-subagents).
592 
593The error is expected, since `Δ` compares your plugin's runs against the baseline. In the JSON result, the baseline runs are under `cases[].arms.without`.
594 
595In runs with your plugin loaded, a case that lists `Agent` in `allowed_tools` can dispatch one of your plugin's agents by its namespaced name, such as `my-plugin:code-reviewer` for the `code-reviewer` agent in a plugin named `my-plugin`. To skip the baseline runs, pass `--ablation none`.
588596 
589597### Everything scores zero although the right files were produced
590598 
Feedback