What
The eval sandbox documentation now warns that the harness judges a run from the child process's output and files, which a shell-granted process running as you can also reach. Scores from an untrusted suite run with --allow-tools Bash, or any other grant that can execute code, should be treated as advisory unless the run had operating-system-level isolation.
Details
- Text only; the sandbox itself behaves as before.
Evidence
treat scores from an untrusted suite run
Strings lifted out of the shipped bundle, so the claim above can be checked against them.
Names in the bundle--allow-tools Bash
The entry above is what we published on the day. These lines were added later, as Anthropic's own pages caught up, and they sit beside the original rather than replacing it.
Added since
A small documentation edit on Test plugins with evals touched a line naming --allow-tools after this was published.
The allowlist is the read-only tools the case lists in `allowed_tools`, from `Read`, `Glob`, `Grep`, `NotebookRead`, `Skill`, `Agent`, `TodoWrite`, and the task tools `TaskCreate`, `TaskGet`, `TaskList`, `TaskUpdate`, and `TaskStop`, plus …plugin-evals see the edit
Confirmed since
Anthropic's documentation has since written up --allow-tools, on Plugins reference.
The optional `target` is a plugin directory, a single `prompt.md` or `case.yaml` file, an installed plugin as `name` or `name@marketplace`, or `name@skills-dir`, and defaults to the current directory. Put it before `--tag`, `--allow-tools`…plugins-reference see the edit
One source agreesOne thing we can check says the same as this entry.
Anthropic's documentation agrees
Anthropic's documentation has since written up --allow-tools, on Plugins reference.