Use it now
Useful3 Signal3
Mock servers replace real ones by name, allow mocked tools and deny the rest; a mock_calls grader is added.
What's wrong with this entry?
Anonymous. No account, no email.
mock_calls
What
The stand-in servers take the place of the plugin's own MCP servers under the same name, auto-allow the tools that have mock files, and deny every other tool on that server. A new grader target, mock_calls, grades what the plugin asked the server to do rather than what it said.
Details
- A case can carry its own
mocks/directory, andmocks/becomes a reserved name inside eval directories. - Frontmatter supports
expect:input assertions that abort the run on violation, anderror: trueto return a tool error. - Each run's JSON gains a
mocksblock with per-server responder kinds, a call tally of total/errors/unmocked, and warnings, plus anabortedrecord. - Failures are reported through telemetry key
cli_plugin_eval_mockswith outcomes standin_registration, standin_identity, standin_tools_missing, standin_integrity and aborted_by_mock. claude plugin evalis itself early access, enabled per organisation or by an env var, and prints that it is unavailable otherwise.
Evidence
mock_calls, --mocks <mode>, Mock stand-ins for MCP servers, from <eval dir>/mocks/
Strings lifted out of the shipped bundle, so the claim above can be checked against them.