What
A grader can now check the tool calls that hit mock stand-ins during an eval run. If no mocks applied, the grader gets an explicit reason instead of empty data. The note about plugins in an eval directory was also tightened.
Details
- Requires
--mocks; without it, or when nomocks/directory applies to the case, the grader is told mock stand-ins were not active in that run. - The eval directory note now always says the plugin is not loaded unless you name its directory, and states that naming a directory counts as consent to load it.
- Resolving a suite path also records the directory that consent was granted for.
Evidence
no mock stand-ins were active in this run (--mocks off, or no mocks/ directory applies to this case)
Strings lifted out of the shipped bundle, so the claim above can be checked against them.
Names in the bundle--mocks
The entry above is what we published on the day. These lines were added later, as Anthropic's own pages caught up, and they sit beside the original rather than replacing it.
Confirmed since
Anthropic's documentation has since written up --mocks, on Plugins reference.
| `--mocks <mode>` | `record` or `off`. See [Mock MCP servers](/docs/en/plugin-evals#mock-mcp-servers) | `record` |plugins-reference see the edit
Two sources agreeTwo things we can check say the same as this entry.
Anthropic's documentation agrees
Anthropic's documentation has since written up --mocks, on Plugins reference.
The name it cites is new in this build
New in this build: --mocks