What probably matters to youSection of the release
What
The stand-in servers take the place of the plugin's own MCP servers under the same name, auto-allow the tools that have mock files, and deny every other tool on that server. A new grader target, mock_calls, grades what the plugin asked the server to do rather than what it said.
Details
A case can carry its own mocks/ directory, and mocks/ becomes a reserved name inside eval directories.
Frontmatter supports expect: input assertions that abort the run on violation, and error: true to return a tool error.
Each run's JSON gains a mocks block with per-server responder kinds, a call tally of total/errors/unmocked, and warnings, plus an aborted record.
Failures are reported through telemetry key cli_plugin_eval_mocks with outcomes standin_registration, standin_identity, standin_tools_missing, standin_integrity and aborted_by_mock.
claude plugin eval is itself early access, enabled per organisation or by an env var, and prints that it is unavailable otherwise.
Evidence
mock_calls, --mocks <mode>, Mock stand-ins for MCP servers, from <eval dir>/mocks/
Strings lifted out of the shipped bundle, so the claim above can be checked against them.
