{"version":"2.1.242","anchor":"plugin-eval-can-mock-a-plugin-s-mcp-servers-with-docs-to-mat","canonical_anchor":"plugin-eval-can-mock-a-plugin-s-mcp-servers-with-docs-to-mat","heading":"plugin eval can mock a plugin's MCP servers, with docs to match","tier":"use","area":"Plugin Eval","scope":null,"heads_up":null,"url":"https:\/\/changelogs.core-directive.com\/v\/2.1.242\/e\/plugin-eval-can-mock-a-plugin-s-mcp-servers-with-docs-to-mat","release_url":"https:\/\/changelogs.core-directive.com\/v\/2.1.242","markdown":"### `plugin eval` can mock a plugin's MCP servers, with docs to match\n\nPlugin evals can swap a plugin's MCP servers for fakes defined in a mocks folder.\n\n**What**\n\nEvaluation runs of a plugin can now swap that plugin's real MCP servers, the external tool servers a plugin declares, for fake ones defined as files under `<eval dir>\/mocks\/`, so an eval exercises tool calls without starting anything real. The mode is controlled by a new `--mocks` flag that defaults to `record`, and the bundled quick reference for `plugin eval` now carries a Mocks section describing the whole layout. The commands remain early access.\n\n**Details**\n\n- `--mocks record` (the default) serves canned responses from `mocks\/<server>\/<tool>.md`; `--mocks off` spawns the real servers instead; any other value exits with \"Error: --mocks must be record or off\".\n- A responder file may carry an `expect:` guard on the input, which aborts the run with a record of the server, tool and reason when the call does not match.\n- Responders also support `error: true` to return a failure, and `type: agent` with its own abort conditions.\n- `mocks\/<server>\/_tools.json` supplies the saved tool list; without it a tool is served with a permissive schema and no description.\n- `_server.md` lets a single responder answer several tools for that server.\n- Directory and tool names must be valid tool-name segments, `__` is rejected because it separates server name from tool name, and a directory is capped at 256 entries.\n- Mocks can shadow a server that the plugins under test declare, or stand alone with no real counterpart.\n- Grading gains `mock_calls` as a source, so a grader can assert on which mock tools were called, and the LLM judge can now grade image files.\n- The reference synopsis also lists `--report`, `--publish-report` and `--no-publish`.\n- Enabled per organization: when your organization is not enabled, the commands print an early-access line and exit with status 1.\n\n**Evidence**\n\n`Mock stand-ins for MCP servers, from <eval dir>\/mocks\/ (record | off; default: record`, `--mocks record|off`\n\n- Area: Plugin Eval\n- Names: `--mocks`, `plugin eval`\n- Tier: Use it now\n- Useful: 4\/5\n- Signal: 4\/5"}