{"version":"2.1.235","anchor":"plugin-eval-gains-eval-dir-and-a-manifest-eval-directory","canonical_anchor":"plugin-eval-gains-eval-dir-and-a-manifest-eval-directory","heading":"plugin eval gains --eval-dir and a manifest eval directory, still early access","tier":"soon","area":"Plugin Eval","scope":null,"heads_up":null,"url":"https:\/\/changelogs.core-directive.com\/v\/2.1.235\/e\/plugin-eval-gains-eval-dir-and-a-manifest-eval-directory","release_url":"https:\/\/changelogs.core-directive.com\/v\/2.1.235","markdown":"### `plugin eval` gains --eval-dir and a manifest eval directory, still early access\n\nplugin eval gets a configurable case directory via --eval-dir or a manifest key, still gated as early access\n\nThe eval harness no longer hard-codes `evals\/` as the case directory. A new `--eval-dir <dir>` flag, or a new plugin.json key `experimental.evals` (a single path, or a list whose first entry wins), now selects it, and results are written to `<plugin>\/<dir>\/results\/`.\n\nBoth `plugin eval` and `plugin eval init` remain marked as early access, gated by `tengu_walnut_spire` (falling back to false) or the env var `CLAUDE_CODE_WALNUT_SPIRE`. Users without access get a \"currently in early access\" message rather than an unrecognized-command error. The manifest's `experimental.evals` value is validated strictly: it must be relative, contain no `..`, stay under 200 characters and 8 path segments, use plain segment names, not be Windows-hostile, and not point inside `node_modules`, `.claude`, or `results`; a misplaced top-level `\"evals\"` key is flagged rather than silently accepted.\n\n- Flag `tengu_walnut_spire`: Off in both readings (read for one account on one subscription tier against v2.1.235; this account: off, anonymous baseline: off, compiled default: off)\n- Area: Plugin Eval\n- Names: `--eval-dir`\n- Tier: Nothing to try yet\n- Useful: 4\/5\n- Signal: 4\/5\n- Present in the build but not switched on"}