You'll notice
Useful1 Signal1
Eval graders can inspect the calls that hit mock tools during a run.
What's wrong with this entry?
Anonymous. No account, no email.
--mocks
What
A grader can now check the tool calls that hit mock stand-ins during an eval run. If no mocks applied, the grader gets an explicit reason instead of empty data. The note about plugins in an eval directory was also tightened.
Details
- Requires
--mocks; without it, or when nomocks/directory applies to the case, the grader is told mock stand-ins were not active in that run. - The eval directory note now always says the plugin is not loaded unless you name its directory, and states that naming a directory counts as consent to load it.
- Resolving a suite path also records the directory that consent was granted for.
Evidence
no mock stand-ins were active in this run (--mocks off, or no mocks/ directory applies to this case)
Strings lifted out of the shipped bundle, so the claim above can be checked against them.