Evaluation runs of a plugin can now swap that plugin's real MCP servers, the external tool servers a plugin declares, for fake ones defined as files under <eval dir>/mocks/, so an eval exercises tool calls without starting anything real. The mode is controlled by a new --mocks flag that defaults to record, and the bundled quick reference for plugin eval now carries a Mocks section describing the whole layout. The commands remain early access.
--mocks record(the default) serves canned responses frommocks/<server>/<tool>.md;--mocks offspawns the real servers instead; any other value exits with "Error: --mocks must be record or off".- A responder file may carry an
expect:guard on the input, which aborts the run with a record of the server, tool and reason when the call does not match. - Responders also support
error: trueto return a failure, andtype: agentwith its own abort conditions. mocks/<server>/_tools.jsonsupplies the saved tool list; without it a tool is served with a permissive schema and no description._server.mdlets a single responder answer several tools for that server.- Directory and tool names must be valid tool-name segments,
__is rejected because it separates server name from tool name, and a directory is capped at 256 entries. - Mocks can shadow a server that the plugins under test declare, or stand alone with no real counterpart.
- Grading gains
mock_callsas a source, so a grader can assert on which mock tools were called, and the LLM judge can now grade image files. - The reference synopsis also lists
--report,--publish-reportand--no-publish. - Enabled per organization: when your organization is not enabled, the commands print an early-access line and exit with status 1.
Mock stand-ins for MCP servers, from <eval dir>/mocks/ (record | off; default: record, --mocks record|off
Strings lifted out of the shipped bundle, so the claim above can be checked against them.
The entry above is what we published on the day. These lines were added later, as Anthropic's own pages caught up, and they sit beside the original rather than replacing it.
| `--mocks <mode>` | `record` or `off`. See [Mock MCP servers](/docs/en/plugin-evals#mock-mcp-servers) | `record` |plugins-reference see the edit
Test your marketplace before sharing. Validation checks file structure; to test whether a plugin changes what Claude does on realistic prompts, run its eval suite with [`claude plugin eval`](/docs/en/plugin-evals) before you publish a new …plugin-marketplaces see the edit
* Added `claude plugin eval`: run a plugin's eval suite against Claude Code and get scored, reproducible results (JSON + HTML report); see `claude plugin eval --help`changelog see the edit
Anthropic's documentation has since written up plugin eval, on Claude Code changelog.
New in this build: --mocks